hangulize

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 31, 2018 License: MIT Imports: 14 Imported by: 0

README

Hangulize

GoDoc Go Report Card Build Status

(WIP: 아직 개발 중, API가 임의로 바뀔 수 있어요!)

한글라이즈

외국어의 한글 표기 체계가 제대로 서려면 일반인이 외국어를 한글로 표기하고 싶을 때 바로바로 쉽게 용례를 찾을 수 있어야 한다. 정기적으로 회의를 열어 용례를 정하는 것으로는 한계가 있다. 외래어 표기 심의 방식이 자동화되어 한글로 표기하고 싶은 외국어를 입력하자마자 한글 표기가 나와야 한다. 이미 용례가 정해진 것은 그것을 따르고 용례에 없는 것이라도 각 언어의 표기 규칙에 따라 권장 표기를 표시해야 한다. 프로그래머들과 언어학자들이 손잡고 연구한다면 이게 공상으로만 그치지 않을 것이다.

Brian Jongseong Park (http://iceager.egloos.com/2610028)

Hangulize는 외래어를 한글로 변환해주는 도구입니다.

$ go get -u github.com/hangulize/hangulize

import "github.com/hangulize/hangulize"

hangulize.Hangulize("ita", "gloria")
// output: "글로리아"

리부트

Hangulize 프로젝트는 2010년에 Python으로 처음 구현되었고, 웹 상에서 누구나 쉽게 사용할 수 있도록 hangulize.org 서비스를 제공해왔습니다. 이 Python 구현의 소스코드와 관련 자료는 sublee/hangulize에서 확인할 수 있습니다.

시간이 흘러 2018년, 기존 Hangulize의 기능은 모두 계승하면서 성능을 높이고 코드의 유지보수 가능성과 규칙 설계 시 생산성을 높이기 위해 리부트를 결정했습니다. 리부트한 Hangulize를 구현하는 데에는 Go를 사용했습니다.

이 프로젝트는 처음에 "Hangulize 2"로 명명했으나 기존 Hangulize를 충분히 대체할 수 있을 것으로 보여 숫자 2를 뗀 "Hangulize"로 이름을 바꿨습니다.

목표

  • 기존 Hangulize(Python 구현)의 기능을 모두 계승
  • 규칙 설계에 정적 파일(.hgl) 사용
  • 간편한 규칙 설계환경
  • 규칙 설계법 꼼꼼히 문서화
  • hangulize.org 개편

지원하는 언어

LANG     STAGE    ENG                      KOR
aze      draft    Azerbaijani              아제르바이잔어
bel      draft    Belarusian               벨라루스어
bul      draft    Bulgarian                불가리아어
cat      draft    Catalan                  카탈로니아어
ces      draft    Czech                    체코어
cym      draft    Welsh                    웨일스어
deu      draft    German                   독일어
ell      draft    Greek                    그리스어
epo      draft    Esperanto                에스페란토어
est      draft    Estonian                 에스토니아어
fin      draft    Finnish                  핀란드어
grc      draft    Ancient Greek            고대 그리스어
hbs      draft    Serbo-Croatian           세르보크로아트어
hun      draft    Hungarian                헝가리어
isl      draft    Icelandic                아이슬란드어
ita      draft    Italian                  이탈리아어
jpn      draft    Japanese                 일본어
kat-1    draft    Georgian (1st scheme)    조지아어(제1안)
kat-2    draft    Georgian (2nd scheme)    조지아어(제2안)
lat      draft    Latin                    라틴어
lav      draft    Latvian                  라트비아어
lit      draft    Lithuanian               리투아니아어
mkd      draft    Macedonian               마케도니아어
nld      draft    Dutch                    네덜란드어
pol      draft    Polish                   폴란드어
por      draft    Portuguese               포르투갈어
por-br   draft    Brazilian Portuguese     브라질 포르투갈어
ron      draft    Romanian                 루마니아어
rus      draft    Russian                  러시아어
slk      draft    Slovak                   슬로바키아어
slv      draft    Slovenian                슬로베니아어
spa      draft    Spanish                  스페인어
sqi      draft    Albanian                 알바니아어
swe      draft    Swedish                  스웨덴어
tur      draft    Turkish                  터키어
ukr      draft    Ukrainian                우크라이나어
vie      draft    Vietnamese               베트남어
wlm      draft    Middle Welsh             웨일스어(중세)

만든이

라이선스

Hangulize는 MIT 라이선스 하에 공개되어 있습니다. 소스코드를 사용할 경우 라이선스 내용을 준수해주세요. 라이선스 전문은 LICENSE 파일에서 확인하실 수 있습니다.

Documentation

Overview

Package hangulize transcribes non-Korean words into Hangul. 

"Hello!" -> "헬로!"

Hangulize was inspired by Brian Jongseong Park (http://iceager.egloos.com/2610028). Based on this idea, the original Hangulize was developed in Python and went out in 2010 (https://github.com/sublee/hangulize). Since then, serving as a web application on http://hangulize.org/, it has been of great help for Korean translators.

This Go re-implementation is a reboot of Hangulize with feature improvements.

Pipeline

Hangulize transcribes with 5 steps. These steps include "Normalize", "Group", "Rewrite", "Transcribe", and "Compose". To clarify these concepts, let's consider an imaginary example of "Hello!" in English into "헬로!" (actually, English is not supported yet).

First, Hangulize normalizes letter cases: 

"Hello" -> "hello!"

And then, it groups letters by meanings: 

"hello!" -> "hello", "!"

After that, grouped chunks are rewritten as source language-specific rules. This step is usually for minimizing the differences between pronunciation and spelling: 

"hello", "!" -> "heˈlō", "!"

And it transcribes rewritten chunks into Hangul Jamo phonemes. 

"heˈlō", "!" -> "ㅎㅔ-ㄹㄹㅗ", "!"

Finally, it composes Jamo phonemes to Hangul syllables and joins all groups. 

"ㅎㅔ-ㄹㄹㅗ", "!" -> "헬로!"

Spec

A spec is written by the HGL format which is a configuration DSL for Hangulize 2. One spec is for one language transcription system. So we need to describe about the language at the first: 

lang:
    id      = "ita"
    codes   = "it", "ita" # ISO 639-1 and 3 codes
    english = "Italian"
    korean  = "이탈리아어"
    script  = "roman"

Then write about yourself and the stage of this spec: 

config:
    author = "John Doe <john@example.com>"
    stage  = "draft"

We will write many patterns in rewrite/transcribe rules soon. Some expressions may appear many times annoyingly. To not repeat ourselves, we can use variables and macros.

A variable is a combination of letters. Variable in pattern will match with one of the letters. Variable "foo" can be referenced with "<foo>" in the patterns. 

vars:
    "vowels" = "a", "e", "i", "o", "u"

A macro expression is replaced with the target before parsing the patterns. "@" is the common macro for "<vowels>" variable: 

macros:
    "@" = "<vowels>"

Now we can write "rewrite" rules. There are Pattern and RPattern. Pattern matches with letters in a word. RPattern represents how the matched letters should be replaced. A replaced word by a rule would become as the input for the next rule: 

rewrite:
    "^gli$"   -> "li"
    "^gli{@}" -> "li"
    "{@}gli"  -> "li"
    "gn{@}"   -> "nJ"

Pattern is based on Regular Expression but it has it's own custom syntax. We call it "HRE" which means "Hangulize-specific Regular Expression". For the detail, see the documentation of Pattern.

"transcribe" rules are exactly same with "rewrite" rules. But it's RPatterns represent Hangul Jamo phonemes. In contrast to "rewrite", a replaced word won't become as the input for the next rules: 

transcribe:
    "b" -> "ㅂ"
    "d" -> "ㄷ"
    "f" -> "ㅍ"
    "g" -> "ㄱ"

Finally, we should write expected transcription examples. They are used for unit testing. Verify your spec yourself: 

test:
    "allegretto" -> "알레그레토"
    "gita"       -> "지타"
    "bisnonno"   -> "비스논노"
    "Pinocchio"  -> "피노키오"
Example 
// Person names from http://iceager.egloos.com/2610028
fmt.Println(Hangulize("ron", "Cătălin Moroşanu"))
fmt.Println(Hangulize("nld", "Jerrel Venetiaan"))
fmt.Println(Hangulize("por", "Vítor Constâncio"))

Output:
커털린 모로샤누
예럴 페네티안
비토르 콘스탄시우

Index

Examples

Constants

View Source 
const Version = "0.2.0"

Version is the version number of Hangulize package. The version follows Semantic Versioning 2.0.0.

Variables

This section is empty.

Functions

func ComposeHangul 

func ComposeHangul(word string) string

ComposeHangul converts decomposed Jamo phonemes to composed Hangul syllables.

Decomposed Jamo phonemes look like "ㅎㅏ-ㄴㄱㅡ-ㄹㄹㅏㅇㅣㅈㅡ". A Jaeum after a hyphen ("-ㄴ") means that it is a Jongseong (tail).

Example (Interpolation) 
fmt.Println(ComposeHangul("ㅗㅈ"))

Output:
오즈
Example (Perfect) 
fmt.Println(ComposeHangul("ㅎㅏ-ㄴㄱㅡ-ㄹㄹㅏㅇㅣㅈㅡ"))

Output:
한글라이즈

func Hangulize 

func Hangulize(lang string, word string) string

Hangulize is the most simple and useful API of thie package. It transcribes a non-Korean word into Hangul, which is the Korean alphabet. For example, it will transcribe "Владивосто́к" in Russian into "블라디보스토크".

Example (Gloria) 
fmt.Println(Hangulize("ita", "gloria"))

Output:
글로리아
Example (Nietzsche) 
fmt.Println(Hangulize("deu", "Friedrich Wilhelm Nietzsche"))

Output:
프리드리히 빌헬름 니체
Example (ShinkaiMakoto) 
// import "github.com/hangulize/hangulize/pronounce/furigana"
// UsePronouncer(&furigana.P)

fmt.Println(Hangulize("jpn", "新海誠"))

Output:
신카이 마코토

func ListLangs 

func ListLangs() []string

ListLangs returns the language name list of bundled specs. The bundled spec can be loaded by LoadSpec.

Example

Here're all supported languages. 

for _, lang := range ListLangs() {
	fmt.Println(lang)
}

Output:
aze
bel
bul
cat
ces
cym
deu
ell
epo
est
fin
grc
hbs
hun
isl
ita
jpn
kat-1
kat-2
lat
lav
lit
mkd
nld
pol
por
por-br
ron
rus
slk
slv
spa
sqi
swe
tur
ukr
vie
wlm

func UnusePronouncer added in v0.2.0 

func UnusePronouncer(id string) bool

UnusePronouncer discards an imported pronouncer.

func UsePronouncer added in v0.2.0 

func UsePronouncer(p Pronouncer) bool

UsePronouncer keeps a pronouncer for ready to use.

Types

type Config 

type Config struct {
	Authors []string
	Stage   string
}

Config keeps some configurations for a transactiption specification.

type Hangulizer 

type Hangulizer struct {
	// contains filtered or unexported fields
}

Hangulizer provides the transcription logic for the underlying spec.

func NewHangulizer 

func NewHangulizer(spec *Spec) *Hangulizer

NewHangulizer creates a Hangulizer for a spec.

Example 
spec, _ := LoadSpec("nld")
h := NewHangulizer(spec)

fmt.Println(h.Hangulize("Vincent van Gogh"))

Output:
빈센트 반고흐

func (*Hangulizer) Hangulize 

func (h *Hangulizer) Hangulize(word string) string

Hangulize transcribes a loanword into Hangul.

func (*Hangulizer) HangulizeTrace 

func (h *Hangulizer) HangulizeTrace(word string) (string, []Trace)

HangulizeTrace transcribes a loanword into Hangul and returns the traced internal events too.

func (*Hangulizer) Spec 

func (h *Hangulizer) Spec() *Spec

Spec returns the underlying spec.

type Language 

type Language struct {
	ID        string    // Arbitrary, but identifiable language ID.
	Codes     [2]string // [0]: ISO 639-1 code, [1]: ISO 639-3 code
	English   string    // The language name in English.
	Korean    string    // The language name in Korean.
	Script    string
	Pronounce string
}

Language identifies a natural language.

func (*Language) String 

func (l *Language) String() string

type Pattern 

type Pattern struct {
	// contains filtered or unexported fields
}

Pattern represents an HRE (Hangulize-specific Regular Expression) pattern.

The transcription logic includes several rewriting rules. A rule has a Pattern and an RPattern. A sub-word which is matched with the Pattern, will be rewritten by the RPattern. 

rewrite:
    "'"        -> ""
    "^gli$"    -> "li"
    "^glia$"   -> "g.lia"
    "^glioma$" -> "g.lioma"
    "^gli{@}"  -> "li"
    "{@}gli"   -> "li"
    "gn{@}"    -> "nJ"
    "gn"       -> "n"

Some expressions in Pattern have special meaning: 

"^"      // start of chunk
"^^"     // start of string
"$"      // end of chunk
"$$"     // end of string
"{...}"  // zero-width match
"{~...}" // zero-width negative match
"{}"     // zero-width space
"<var>"  // one of var values (defined in spec)

func (*Pattern) Explain 

func (p *Pattern) Explain() string

Explain shows the HRE expression with the underlying standard regexp patterns.

func (*Pattern) Find 

func (p *Pattern) Find(word string, n int) [][]int

Find searches up to n matches in the word. If n is -1, it will search all matches. The result is an array of submatch locations.

Example 
p, _ := newPattern("^he(l+o){,}", nil, nil)
fmt.Println(p.Find("hello, helo, hellllo", -1))

Output:
[[0 5 2 5] [7 11 9 11]]

func (*Pattern) Letters 

func (p *Pattern) Letters() []string

Letters returns the set of natural letters used in the expression in ascending order.

Example 
p, _ := newPattern("^hello{,}", nil, nil)
fmt.Println(p.Letters())

Output:
[, e h l o]

func (*Pattern) String 

func (p *Pattern) String() string

type Pronouncer added in v0.2.0 

type Pronouncer interface {
	ID() string
	Pronounce(string) string
}

Pronouncer is an interface to guess pronunciation from spelling based on lexical analysis.

The lexical analysis may require large size of dictionary data. To keep Hangulize lightweight, pronouncers are implemented out of this package.

For example, there is the pronouncer for Furigana of Japanese in a separate package. 

import "github.com/hangulize/hangulize"
import "github.com/hangulize/hangulize/pronounce/furigana"

hangulize.UsePronouncer(&furigana.P)
fmt.Println(hangulize.Hangulize("jpn", "日本語"))

func GetPronouncer added in v0.2.0 

func GetPronouncer(id string) (Pronouncer, bool)

GetPronouncer returns the imported pronouncer by the ID.

type RPattern 

type RPattern struct {
	// contains filtered or unexported fields
}

RPattern is a dynamic replacement pattern.

Some expressions in RPattern have special meaning: 

"{}"    // zero-width space
"<var>" // ...

"R" in the name means "replacement" or "right-side".

func (*RPattern) Interpolate 

func (rp *RPattern) Interpolate(p *Pattern, word string, m []int) string

Interpolate determines the final replacement based on the matched Pattern.

func (*RPattern) Letters 

func (rp *RPattern) Letters() []string

Letters returns the set of natural letters used in the expression in ascending order.

func (*RPattern) String 

func (rp *RPattern) String() string

type Rule 

type Rule struct {
	From *Pattern
	To   *RPattern
}

Rule is a pair of Pattern and RPattern.

func (*Rule) String 

func (r *Rule) String() string

type Spec 

type Spec struct {
	// Meta information sections
	Lang   Language
	Config Config

	// Helper setting sections
	Macros    map[string]string
	Vars      map[string][]string
	Normalize map[string][]string

	// Rewrite/Transcribe
	Rewrite    []*Rule
	Transcribe []*Rule

	// Test examples
	Test [][2]string

	// Source code
	Source string
	// contains filtered or unexported fields
}

Spec represents a transactiption specification for a language.

func LoadSpec 

func LoadSpec(lang string) (*Spec, bool)

LoadSpec finds a bundled spec by the given language name. Once it loads a spec, it will cache the spec.

func ParseSpec 

func ParseSpec(r io.Reader) (*Spec, error)

ParseSpec parses a Spec from an HGL source.

func (*Spec) String 

func (s *Spec) String() string

type Trace 

type Trace struct {
	Step string
	Why  string
	Word string
}

Trace is emitted when a replacement occurs. It is used for tracing of Hangulize pipeline internal.

func (*Trace) String 

func (t *Trace) String() string

Source Files

View all Source files

Directories

Path Synopsis
pronounce
furigana
Package furigana implements the hangulize.Pronouncer interface for Japanese Kanji.
 Package furigana implements the hangulize.Pronouncer interface for Japanese Kanji.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.