gomanize

README

Gomanize

A Go library and CLI that romanizes Devanagari (Hindi) into readable Latin script, built for song lyrics and other colloquial text. Rule-based engine with optional embedded learned components; no runtime dependencies.

नमस्ते दुनिया  →  namaste duniya

Try it in your browser → budhash.com/gomanize — the full engine, compiled to WebAssembly, runs entirely client-side (no server; no text leaves your machine).

It does romanization — spelling Hindi the way it sounds (नमस्ते → namaste) — rather than the strict, reversible transliteration of schemes like IAST or ISO 15919. ("Transliteration" is the common colloquial term for the same task, and the API keeps the Translit name.)

Romanization has no single correct answer — जनता is validly janata, janta, or janataa — so this project measures itself against all human-attested variants. On its primary use case, romanizing verse, its character error rate is at the level where human romanizers disagree with each other. Details, methodology, and known limitations: docs/RESEARCH.md.

Install

# Library
go get github.com/budhash/gomanize

# CLI from source
git clone https://github.com/budhash/gomanize
cd gomanize
make build

For JavaScript/TypeScript, the engine is published to npm as @budhash/gomanize — the same engine compiled to WebAssembly (not a reimplementation), so output is identical:

import { load } from "@budhash/gomanize";
const g = await load();
g.translit("नमस्ते दुनिया");            // "namaste duniya"
g.translit("गाना", { longVowels: true }); // "gaanaa"

Usage

CLI

./gomanize "नमस्ते दुनिया"       # namaste duniya
echo "हिंदी गाना" | ./gomanize   # hindi gana

Flag	Effect	Example
(default)	Colloquial rules	जनता → janta
--keep-medial-schwa	Retain medial schwa	जनता → janata
--long-vowels	aa for every ā	गाना → gaanaa
--simple-nasals	Simplified nasal endings	करें → karen
--schwa-model	Learned schwa classifier	जनता → janta
--lexicon	Attested spellings for 8,367 known words	अंकल → uncle
--rerank	Character-LM picks best of rules/schwa-model outputs	see Accuracy below
--list-rules, --debug	Inspect and trace the rule engine

Library

import gomanize "github.com/budhash/gomanize"

g, err := gomanize.New("hindi")
if err != nil {
    panic(err)
}
fmt.Println(g.Translit("नमस्ते दुनिया")) // "namaste duniya"

Options mirror the CLI flags via gomanize.NewWithOptions.

Accuracy

Scored against all human-attested romanization variants (see docs/RESEARCH.md for methodology, datasets, and the full result set including negative results):

Benchmark	Result
Curated Dakshina	86.1% exact / 92.8% any attested variant (94.7% with --rerank)
Naturally-typed Hindi (COMI-LINGUA), token-weighted	78.9% (85.7% with --lexicon)
Held-out unseen words	69.0% (70.4% with --rerank)
Song lyrics, line-level character error	0.049, or 0.039 with --lexicon (human agreement floor is about 0.054)

Known limitations:

• Vowel-length spelling (ee/oo vs i/u) is a lexical convention, not rule-governed; roughly half of remaining errors are this class. Positional rules for it measured net-negative and were rejected (docs/RESEARCH.md, §5).
• Named entities and loanwords are weak without --lexicon (names have conventional spellings no phonetic rule can produce: दीक्षित → dixit).
• Different annotation efforts use different conventions; scores against corpora with other conventions (e.g. Aksharantar's frequent-word slice) are substantially lower than against Dakshina-style references.

Documentation

Document	Contents
docs/RESEARCH.md	The problem, literature, datasets and licenses, evaluation methodology, all results including negatives
docs/DESIGN.md	Engine architecture, rule system, character mappings, learned components, future directions
docs/ROADMAP.md	Post-1.0 directions (convention schemes, more languages) with tradeoffs
web/README.md	The browser demo (budhash.com/gomanize): make wasm / make wasm-serve and the Pages deploy
CLAUDE.md	Development workflow, commands, repository conventions
docs/PROCESS.md	Task tracking, PR discipline, accuracy reporting rules
docs/reviews/	Dated decision records for every result, including failures

Development

make init     # First-time setup (deps + pre-commit hooks)
make ci       # Full pipeline: format check, lint, build, tests, benchmarks
make help     # All commands

Contributions follow docs/PROCESS.md: feature branches, make ci before PRs, accuracy changes must show before/after on the benchmark suite.

License

MIT. Copyright (c) 2023-2026 Budhaditya (budhash@gmail.com).

Benchmark data derives from Dakshina (CC BY-SA 4.0), Aksharantar (CC-BY 4.0), COMI-LINGUA (CC-BY 4.0), and Shabd (CC0); see docs/RESEARCH.md for full attribution.

Documentation

Overview

Package gomanize transliterates Devanagari (Hindi) text into readable Latin script, using a rule-based engine with optional embedded learned components (a schwa classifier, an attested-spelling lexicon, and a character-LM re-ranker). It targets colloquial, diacritic-free romanization of the kind used for song lyrics.

Basic use:

g, err := gomanize.New("hindi")
out := g.Translit("नमस्ते दुनिया") // "namaste duniya"

A Gomanize instance is safe for concurrent Translit calls. TranslitDebug and runtime rule toggling (DisableRule/EnableRule) mutate engine state and must not run concurrently with other calls on the same instance. TranslitDebug returns nil debug info when the Lexicon or Rerank options short-circuit the rule pipeline.

Index

• Variables
• type DebugInfo
• type EngineOption
• type ExtendedRomanizer
• type Gomanize
  • func New(language string) (*Gomanize, error)
  • func NewWithOptions(language string, opts Options, engineOpts ...EngineOption) (*Gomanize, error)
  • func (g *Gomanize) DisableRule(pattern string) int
  • func (g *Gomanize) EnableRule(pattern string) int
  • func (g *Gomanize) GetOptions() Options
  • func (g *Gomanize) ListRules(pattern string) []RuleStatus
  • func (g *Gomanize) SetOptions(opts Options)
  • func (g Gomanize) Test()
  • func (g Gomanize) Translit(sentence string) string
  • func (g Gomanize) TranslitDebug(word string) (string, *DebugInfo)
• type Options
  • func NewOptions() Options
• type Romanizer
• type RuleStatus

Variables

var WithDisabledRules = core.WithDisabledRules

WithDisabledRules returns an option that disables rules matching the given patterns. Patterns can be exact names or glob patterns (e.g., "schwa.*", "vowel.long-aa.*").

var WithEnabledRules = core.WithEnabledRules

WithEnabledRules returns an option that enables rules matching the given patterns. Useful for enabling rules that are disabled by default.

Types

type DebugInfo

type DebugInfo = core.DebugInfo

DebugInfo contains debugging information from transliteration.

type EngineOption

type EngineOption = core.EngineOption

EngineOption configures engine creation.

type ExtendedRomanizer

type ExtendedRomanizer interface {
	Romanizer
	// ListRules returns all rules with their enabled/disabled status.
	// If pattern is empty, returns all rules.
	ListRules(pattern string) []RuleStatus
	// DisableRule disables rules matching the given pattern.
	DisableRule(pattern string) int
	// EnableRule enables rules matching the given pattern.
	EnableRule(pattern string) int
}

ExtendedRomanizer extends Romanizer with rule management capabilities.

type Gomanize

type Gomanize struct {
	// contains filtered or unexported fields
}

func New

func New(language string) (*Gomanize, error)

New creates a Gomanize instance with default options.

func NewWithOptions

func NewWithOptions(language string, opts Options, engineOpts ...EngineOption) (*Gomanize, error)

NewWithOptions creates a Gomanize instance with custom options.

func (*Gomanize) DisableRule

func (g *Gomanize) DisableRule(pattern string) int

DisableRule disables rules matching the given pattern. Returns the number of rules disabled, or 0 if not supported.

func (*Gomanize) EnableRule

func (g *Gomanize) EnableRule(pattern string) int

EnableRule enables rules matching the given pattern. Returns the number of rules enabled, or 0 if not supported.

func (*Gomanize) GetOptions

func (g *Gomanize) GetOptions() Options

GetOptions returns the current options.

func (*Gomanize) ListRules

func (g *Gomanize) ListRules(pattern string) []RuleStatus

ListRules returns all rules with their enabled/disabled status. If pattern is empty, returns all rules. Supports glob patterns (e.g., "schwa.*"). Returns nil if the romanizer doesn't support rule listing.

func (*Gomanize) SetOptions

func (g *Gomanize) SetOptions(opts Options)

SetOptions updates the options for this instance.

func (Gomanize) Test

func (g Gomanize) Test()

func (Gomanize) Translit

func (g Gomanize) Translit(sentence string) string

Translit transliterates text using the configured options. Words are segmented on ALL whitespace (spaces, tabs, newlines) AND punctuation — both preserved verbatim — so multi-line input romanizes line by line and attached punctuation (danda: "जीत।") no longer defeats word-final rules. Devanagari combining marks (matras, halant, nukta) are marks, not punctuation, so words are never split internally.

func (Gomanize) TranslitDebug

func (g Gomanize) TranslitDebug(word string) (string, *DebugInfo)

TranslitDebug transliterates a word and returns debug information.

type Options

type Options = core.Options

Options configures transliteration behavior. Use NewOptions() to get defaults, then modify as needed.

func NewOptions

func NewOptions() Options

NewOptions returns Options with default values.

type Romanizer

type Romanizer interface {
	Name() string
	Transliterate(word string) string
	TransliterateWithOptions(word string, opts Options) string
	TransliterateDebug(word string, opts Options) (string, *DebugInfo)
	Info()
}

type RuleStatus

type RuleStatus = core.RuleStatus

RuleStatus represents a rule and its current enabled/disabled state.