confusables

README

go-confusables

Unicode Confusable Characters Library for Go.

This library provides utilities to detect and normalize Unicode "confusable" characters (homoglyphs), which are often used to evade text-based security filters. It uses the authoritative data from Unicode TR39.

Installation

go get github.com/disciplinedware/go-confusables

Features

• Authoritative Data: Embedded latest confusables.txt from unicode.org.
• Fast: In-memory lookups, zero network/FS calls at runtime.
• Normalize to ASCII: Easily replace visually similar characters with their ASCII equivalents.
• Skeleton Algorithm: Full implementation of the TR39 skeleton algorithm for secure string comparison.
• CLI Tool: Included confusables-gen to update data from the latest Unicode version.

Usage

Simple ASCII Normalization

import "github.com/disciplinedware/go-confusables"

db := confusables.Default()

// Cyrillic 'а' and 'о' look like Latin 'a' and 'o'
text := "hеllо" 
normalized := db.ToASCII(text)
fmt.Println(normalized) // "hello"

Secure String Comparison (Skeleton)

The skeleton algorithm is recommended for detecting spoofed strings (e.g., usernames).

db := confusables.Default()

user1 := "apple"
user2 := "аррle" // Uses Cyrillic 'а' and 'р'

if db.IsConfusable(user1, user2) {
    fmt.Println("Warning: Usernames are confusable!")
}

Metadata Access

Access information about the embedded Unicode dataset:

fmt.Printf("Unicode Version: %s\n", db.UnicodeVersion())
fmt.Printf("Source Date: %s\n", db.SourceDate())

CLI Tool: Updating Data

You can regenerate the embedded data using the provided CLI tool:

go run ./cmd/confusables-gen

For reproducible builds or targeting specific versions, you can use optional flags:

go run ./cmd/confusables-gen [--version latest] [--generated-at 2026-02-18T12:00:00Z]

Development

A Makefile is provided for common development tasks:

• make test: Run all tests with the race detector.
• make lint: Run the linter.
• make build: Build the library and the generator tool.
• make generate: Update the embedded data from Unicode.org.
• make clean: Remove build artifacts.

License

MIT

Documentation

Overview

Package confusables provides utilities to detect and normalize Unicode confusable characters.

Index

• type DB
  • func Default() *DB
  • func Load(jsonData []byte) (*DB, error)
  • func (db *DB) GeneratedAt() time.Time
  • func (db *DB) IsConfusable(a, b string) bool
  • func (db *DB) Lookup(r rune) []rune
  • func (db *DB) LookupASCII(r rune) (rune, bool)
  • func (db *DB) Skeleton(s string) string
  • func (db *DB) SourceDate() string
  • func (db *DB) SourceURL() string
  • func (db *DB) ToASCII(s string) string
  • func (db *DB) UnicodeVersion() string
• type Mapping

Types

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB is the confusables database. Thread-safe after initialization.

func Default

func Default() *DB

Default returns the embedded database (loaded once via sync.Once).

func Load

func Load(jsonData []byte) (*DB, error)

Load creates a DB from a JSON file (for custom/updated data).

func (*DB) GeneratedAt

func (db *DB) GeneratedAt() time.Time

GeneratedAt returns the timestamp when the database was generated.

func (*DB) IsConfusable

func (db *DB) IsConfusable(a, b string) bool

IsConfusable checks if two strings would produce the same skeleton.

func (*DB) Lookup

func (db *DB) Lookup(r rune) []rune

Lookup returns all target runes for a confusable character. Returns a defensive copy of the internal slice. Returns nil if the character has no mapping.

func (*DB) LookupASCII

func (db *DB) LookupASCII(r rune) (rune, bool)

LookupASCII returns the ASCII equivalent of a rune, if one exists. Returns (replacement, true) for single-char ASCII mappings. Returns (0, false) for characters with no mapping or multi-char/non-ASCII targets. Note: If the input rune is already ASCII (r < 0x80), it returns (0, false) to prevent remapping characters that are already valid ASCII.

func (*DB) Skeleton

func (db *DB) Skeleton(s string) string

Skeleton returns the TR39 skeleton of the string. Maps all confusable characters through the database, regardless of target length. Result is NOT suitable for display — use only for comparison. Implementation: NFD → map → NFD

func (*DB) SourceDate

func (db *DB) SourceDate() string

SourceDate returns the date of the source data.

func (*DB) SourceURL

func (db *DB) SourceURL() string

SourceURL returns the URL of the source data.

func (*DB) ToASCII

func (db *DB) ToASCII(s string) string

ToASCII replaces confusable characters with their ASCII equivalents. Only replaces characters that map to a SINGLE ASCII char (0x20-0x7E). Characters with multi-char targets or non-ASCII targets are kept as-is. Already-ASCII characters (0x00-0x7F) are always returned unchanged.

func (*DB) UnicodeVersion

func (db *DB) UnicodeVersion() string

UnicodeVersion returns the Unicode version of the database.

type Mapping

type Mapping struct {
	Source     int    `json:"source"`
	Target     []int  `json:"target"`
	SourceName string `json:"source_name"`
	TargetName string `json:"target_name"`
}

Mapping represents a single confusable mapping.