gomojimoji

README

(go) mojimoji

This is a port of the excellent mojimoji library written in Python to Golang.

It provides two functions:

• HanToZen - half-width to full-width character conversion.
• ZenToHan - half-width to full-width character conversion.

Each of the functions allow the following options:

• ASCII - enable or disable ASCII translation.
• Digits - enable or disable Digits translation.
• Kana - enable or disable Kana translation.

All options are enabled by default, see examples on their usage.

Logic is implemented as of commit aca2661.

Examples

HanToZen

fmt.Println(HanToZen("ﾆｭｰｼﾞｰﾗﾝﾄﾞ"))
fmt.Println(HanToZen("ﾆｭｰｼﾞｰﾗﾝﾄﾞ Auckland 6012", ASCII(true), Digits(false), Kana(false)))

// Output:
// ニュージーランド
// ﾆｭｰｼﾞｰﾗﾝﾄﾞ　Ａｕｃｋｌａｎｄ　6012

ZenToHan

fmt.Println(ZenToHan("ニュージーランド"))
fmt.Println(ZenToHan("ニュージーランド Ａｕｃｋｌａｎｄ ０１２３", Kana(false), Digits(true)))

// Output:
// ﾆｭｰｼﾞｰﾗﾝﾄﾞ
// ニュージーランド Auckland 0123

Benchmark

Original library etc.

Original mojimoji, zenhan and unicodedata on my system, for comparison:

In [4]: s = u'ＡＢＣＤＥＦＧ０１２３４５' * 10

In [5]: %time for n in range(1000000): mojimoji.zen_to_han(s)
CPU times: user 3.24 s, sys: 1.28 ms, total: 3.24 s
Wall time: 3.24 s

In [6]: %time for n in range(1000000): zenhan.z2h(s)
CPU times: user 26.2 s, sys: 16.3 ms, total: 26.2 s
Wall time: 26.2 s

In [7]: %time for n in range(1000000): unicodedata.normalize('NFKC', s)
CPU times: user 3.12 s, sys: 15.4 ms, total: 3.13 s
Wall time: 3.14 s

This library

ZenToHan and HanToZen use different approaches:

• ZenToHan uses string.Builder, which is simpler to implement.
• HanToZen uses direct slice operations to allow for seeking when needed.

ZenToHan:

mojimoji (master)> go test -bench=BenchmarkZenToHanConv
goos: darwin
goarch: amd64
pkg: github.com/rusq/gomojimoji
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkZenToHanConv-16               1        2880823810 ns/op
--- BENCH: BenchmarkZenToHanConv-16
    mojimoji_test.go:98: 2.88079814s
PASS
ok      github.com/rusq/gomojimoji      2.977s

HanToZen:

mojimoji (master)> go test -bench=BenchmarkHanToZen    
goos: darwin
goarch: amd64
pkg: github.com/rusq/gomojimoji
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkHanToZenConv-16               1        2712209539 ns/op
--- BENCH: BenchmarkHanToZenConv-16
    mojimoji_test.go:107: 2.712166151s
PASS
ok      github.com/rusq/gomojimoji      2.804s

Documentation

Overview

Package mojimoji is a port of mojimoji package to Go. Original: https://github.com/studio-ousia/mojimoji

Index

• Variables
• func HanToZen(text string, opt ...Option) string
• func ZenToHan(text string, opt ...Option) string
• type Option
  • func ASCII(enable bool) Option
  • func Digits(enable bool) Option
  • func Kana(enable bool) Option

Examples

• HanToZen
• ZenToHan

Variables

var (
	ASCII_ZENKAKU_CHARS = []rune{'ａ', 'ｂ', 'ｃ', 'ｄ', 'ｅ', 'ｆ', 'ｇ', 'ｈ', 'ｉ', 'ｊ', 'ｋ', 'ｌ', 'ｍ', 'ｎ', 'ｏ', 'ｐ', 'ｑ', 'ｒ', 'ｓ', 'ｔ', 'ｕ', 'ｖ', 'ｗ', 'ｘ', 'ｙ', 'ｚ', 'Ａ', 'Ｂ', 'Ｃ', 'Ｄ', 'Ｅ', 'Ｆ', 'Ｇ', 'Ｈ', 'Ｉ', 'Ｊ', 'Ｋ', 'Ｌ', 'Ｍ', 'Ｎ', 'Ｏ', 'Ｐ', 'Ｑ', 'Ｒ', 'Ｓ', 'Ｔ', 'Ｕ', 'Ｖ', 'Ｗ', 'Ｘ', 'Ｙ', 'Ｚ', '！', '”', '＃', '＄', '％', '＆', '’', '（', '）', '＊', '＋', '，', '－', '．', '／', '：', '；', '＜', '＝', '＞', '？', '＠', '［', '￥', '］', '＾', '＿', '‘', '｛', '｜', '｝', '～', '\u3000'}
	ASCII_HANKAKU_CHARS = []rune{'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', '!', '"', '#', '$', '%', '&', '\'', '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '=', '>', '?', '@', '[', '¥', ']', '^', '_', '`', '{', '|', '}', '~', ' '}

	KANA_ZENKAKU_CHARS = []rune{'ア', 'イ', 'ウ', 'エ', 'オ', 'カ', 'キ', 'ク', 'ケ', 'コ', 'サ', 'シ', 'ス', 'セ', 'ソ', 'タ', 'チ', 'ツ', 'テ', 'ト', 'ナ', 'ニ', 'ヌ', 'ネ', 'ノ', 'ハ', 'ヒ', 'フ', 'ヘ', 'ホ', 'マ', 'ミ', 'ム', 'メ', 'モ', 'ヤ', 'ユ', 'ヨ', 'ラ', 'リ', 'ル', 'レ', 'ロ', 'ワ', 'ヲ', 'ン', 'ァ', 'ィ', 'ゥ', 'ェ', 'ォ', 'ッ', 'ャ', 'ュ', 'ョ', '。', '、', '・', '゛', '゜', '「', '」', 'ー'}
	KANA_HANKAKU_CHARS = []rune{'ｱ', 'ｲ', 'ｳ', 'ｴ', 'ｵ', 'ｶ', 'ｷ', 'ｸ', 'ｹ', 'ｺ', 'ｻ', 'ｼ', 'ｽ', 'ｾ', 'ｿ', 'ﾀ', 'ﾁ', 'ﾂ', 'ﾃ', 'ﾄ', 'ﾅ', 'ﾆ', 'ﾇ', 'ﾈ', 'ﾉ', 'ﾊ', 'ﾋ', 'ﾌ', 'ﾍ', 'ﾎ', 'ﾏ', 'ﾐ', 'ﾑ', 'ﾒ', 'ﾓ', 'ﾔ', 'ﾕ', 'ﾖ', 'ﾗ', 'ﾘ', 'ﾙ', 'ﾚ', 'ﾛ', 'ﾜ', 'ｦ', 'ﾝ', 'ｧ', 'ｨ', 'ｩ', 'ｪ', 'ｫ', 'ｯ', 'ｬ', 'ｭ', 'ｮ', '｡', '､', '･', 'ﾞ', 'ﾟ', '｢', '｣', 'ｰ'}

	DIGIT_ZENKAKU_CHARS = []rune{'０', '１', '２', '３', '４', '５', '６', '７', '８', '９'}
	DIGIT_HANKAKU_CHARS = []rune{'0', '1', '2', '3', '4', '5', '6', '7', '8', '9'}

	KANA_TEN_MAP = map[rune]rune{
		'ガ': 'ｶ', 'ギ': 'ｷ', 'グ': 'ｸ', 'ゲ': 'ｹ', 'ゴ': 'ｺ',
		'ザ': 'ｻ', 'ジ': 'ｼ', 'ズ': 'ｽ', 'ゼ': 'ｾ', 'ゾ': 'ｿ',
		'ダ': 'ﾀ', 'ヂ': 'ﾁ', 'ヅ': 'ﾂ', 'デ': 'ﾃ', 'ド': 'ﾄ',
		'バ': 'ﾊ', 'ビ': 'ﾋ', 'ブ': 'ﾌ', 'ベ': 'ﾍ', 'ボ': 'ﾎ',
		'ヴ': 'ｳ',
	}

	KANA_MARU_MAP = map[rune]rune{
		'パ': 'ﾊ', 'ピ': 'ﾋ', 'プ': 'ﾌ', 'ペ': 'ﾍ', 'ポ': 'ﾎ',
	}
)

Character tables are taken from the original mojimoji python library.

Functions

func HanToZen

func HanToZen(text string, opt ...Option) string

HanToZen converts text to full-width runes. By default all runes are converted, optionally caller can switch off rune-set by passing Option.

Example

fmt.Println(HanToZen("ﾆｭｰｼﾞｰﾗﾝﾄﾞ"))
fmt.Println(HanToZen("ﾆｭｰｼﾞｰﾗﾝﾄﾞ Auckland 6012", ASCII(true), Digits(false), Kana(false)))

Output:

ニュージーランド
ﾆｭｰｼﾞｰﾗﾝﾄﾞ　Ａｕｃｋｌａｎｄ　6012

func ZenToHan

func ZenToHan(text string, opt ...Option) string

ZenToHan converts text to half-width runes. By default all runes are converted, optionally caller can switch off rune-set by passing Option.

Example

fmt.Println(ZenToHan("ニュージーランド"))
fmt.Println(ZenToHan("ニュージーランド Ａｕｃｋｌａｎｄ ０１２３", Kana(false), Digits(true)))

Output:

ﾆｭｰｼﾞｰﾗﾝﾄﾞ
ニュージーランド Auckland 0123

Types

type Option

type Option func(*options)

Option is the option function signature.

func ASCII

func ASCII(enable bool) Option

ASCII enables or disables conversion of ASCII runes (A-Za-z).

func Digits

func Digits(enable bool) Option

Digits enables or disables conversion of digit runes (0-9).

func Kana

func Kana(enable bool) Option

Kana enables or disables conversion of Kana runes.