ichkil

README

ichkil-go — Arabic Tashkeel for Go

Vocalize bare Arabic text with a tiny, self-contained ONNX model (≈5 MB, CPU-only, no GPU, no tokenizer, no server). One call, fully offline after the first run.

ichkil-go downloads its model from Hugging Face on first use, verifies its SHA-256, and runs inference with ONNX Runtime. After that it is 100 % offline.

got, err := ichkil.Diacritize(ctx, "محمد قرأ الكتاب")
// got == "مُحَمَّدٌ قَرَأَ الْكِتَابِ"

Install

go get github.com/Ichkil/ichkil-go

Requires Go ≥ 1.21. The only dependency is yalue/onnxruntime_go, a thin cgo wrapper that loads the ONNX Runtime shared library at runtime — you need ONNX Runtime ≥ 1.29 installed on the machine (see below).

ONNX Runtime on your machine

The Go library loads onnxruntime from the system at startup; it never downloads or bundles the runtime binary.

• Linux (glibc): install a package, or download the release build — e.g. libonnxruntime.so.1.x on LD_LIBRARY_PATH, or set ICHKIL_ONNXRUNTIME_LIB=/path/to/libonnxruntime.so (or libonnxruntime.so.1.x) to point at an exact file.

• macOS: libonnxruntime.dylib — same override variable works.

• Windows: onnxruntime.dll on PATH — or ICHKIL_ONNXRUNTIME_LIB=C:\path\to\onnxruntime.dll.

• Easiest everywhere: pip install onnxruntime and point the env var at it:

  ICHKIL_ONNXRUNTIME_LIB=$(python -c "import onnxruntime; print(onnxruntime.get_library_path())")
  

The library resolves in this order: $ICHKIL_ONNXRUNTIME_LIB (explicit path), then the standard system lookup for the platform's ONNX Runtime library.

Quickstart

package main

import (
    "context"
    "log"

    "github.com/Ichkil/ichkil-go"
)

func main() {
    ctx := context.Background()

    // one-liner (lazy model download on first call, then cached)
    got, err := ichkil.Diacritize(ctx, "العلم نور والجهل ظلمة")
    if err != nil {
        log.Fatal(err)
    }
    log.Println(got) // الْعِلْمِ نُورٌ وَالْجَهْلُ ظُلْمَةُ

    // explicit control
    d, err := ichkil.New(ctx) // default repository: ichkil/ichkil@main
    if err != nil {
        log.Fatal(err)
    }
    defer d.Close()

    text, _ := d.Diacritize("محمد قرأ الكتاب")
    labels, _ := d.PredictLabels("كتاب") // one label id per character, 0..12

    // fully offline: point at local files
    d2, err := ichkil.NewFromFiles("./model.onnx")
    _ = d2
}

A Diacritizer is safe for concurrent use; Close releases the ONNX Runtime session. Inference is synchronous CPU work — the context.Context is used for model resolution (network) only.

Command line

go run github.com/Ichkil/ichkil-go/cmd/ichkil "محمد قرأ الكتاب"
# مُحَمَّدٌ قَرَأَ الْكِتَابِ

go run github.com/Ichkil/ichkil-go/cmd/ichkil --json "العلم نور"
# {"input":"العلم نور","output":"الْعِلْمِ نُور"}

cat text.txt | ichkil                    # read from stdin
ichkil --model ./model.onnx "محمد"      # offline mode

API

Item	Description
ichkil.Diacritize(ctx, text) (string, error)	Vocalize text (module-level convenience, lazy default model).
ichkil.PredictLabels(ctx, text) ([]int, error)	Label id (0–12) per character.
ichkil.New(ctx, opts...) (*Diacritizer, error)	Resolve the model from Hugging Face and open an ONNX session.
ichkil.NewFromFiles(modelPath, opts...) (*Diacritizer, error)	Build from a local model.onnx (+ config.json beside it).
(*Diacritizer).Diacritize(text) / .PredictLabels(text)	Instance methods (same behavior).
Options	WithRepoID, WithRevision, WithCacheDir, WithToken, WithNumThreads, WithoutChecksumVerification, WithBaseURL, WithConfigPath.
ichkil.SetDefault(d) / ResetDefault()	Manage the shared default instance used by the module-level API.
ichkil.RuntimeVersion()	Version of the loaded ONNX Runtime library.
Pipeline	StripDiacritics, Encode, Decode, Attach, Argmax, IsArabicLetter, label tables.
Errors	Sentinels: ErrChecksumMismatch, ErrModelLoad, ErrInference (match with errors.Is).
Env	HF_TOKEN, HF_ENDPOINT, ICHKIL_CACHE_DIR, ICHKIL_ONNXRUNTIME_LIB.

Label contract

The model predicts exactly one label per input character:

id	meaning	id	meaning
0	non-letter token / padding	7	shadda + fatha (ّـَ)
1	letter, no diacritic	8	shadda + damma (ّـُ)
2	fatha (ـَ)	9	shadda + kasra (ّـِ)
3	damma (ـُ)	10	tanwin fatha (ـً)
4	kasra (ـِ)	11	tanwin damma (ـٌ)
5	sukun (ـْ)	12	tanwin kasra (ـٍ)
6	shadda (ـّ)

Already-vocalized input is stripped first, so Diacritize output is canonical and re-applying it changes nothing. Non-Arabic characters (Latin letters, digits, punctuation, spaces) pass through unchanged.

Model provenance

• Repository: ichkil/ichkil (model.onnx, config.json, SHA256SUMS)
• SHA-256 of model.onnx: 9055816b214346b0a8044a4a8deceb36e818dffd0b04699715ea23883bf54744
• Contract: input_ids int64 [batch, seq] → logits float32 [batch, seq, 13]; sequence length fully dynamic (up to max_seq_len = 1024 in the golden contract)
• Quality: DER 2.80 % on the reference evaluation set; ONNX output is bit-identical to the PyTorch reference (equivalence verified)
• Cache layout: $ICHKIL_CACHE_DIR (default ~/.cache/ichkil) under models--ichkil--ichkil/<revision>/, atomic downloads (.part → rename)

Development

go build ./...
go test ./...          # pure pipeline + download tests (no network, no ONNX Runtime)

# integration tests (download model once, ~5 MB) — need the ONNX Runtime lib:
ICHKIL_ONNXRUNTIME_LIB=$(python -c "import onnxruntime; print(onnxruntime.get_library_path())") \
  go test -v ./...

gofmt -l .             # must print nothing
go vet ./...

Publishing

Go modules are consumed through version tags — there is no separate package registry step. The Publish workflow fires on v* tags, builds and tests the tagged commit, verifies the tag is a valid semver, and creates a GitHub Release. From that moment go get github.com/Ichkil/ichkil-go@vX.Y.Z works for every developer (via proxy.golang.org) and the module appears on pkg.go.dev.

Sibling libraries

• ichkil-python — PyPI package
• ichkil-js — npm package (Node.js + browser)
• Core repo — training, evaluation, model spec, golden vectors

License

MIT © 2026 Maaouia BenHamed

Documentation

Overview

Package ichkil provides Arabic Tashkeel (diacritization): it assigns harakat, shadda and tanwin to bare Arabic text.

The engine is a tiny, self-contained ONNX model (≈5 MB, CPU-only) that is downloaded from Hugging Face on first use and cached locally; afterwards everything works offline.

Quick start:

// One-liner — downloads + caches the model on first use.
got, err := ichkil.Diacritize(ctx, "محمد قرأ الكتاب")
// got == "مُحَمَّدٌ قَرَأَ الْكِتَابِ"

Or, for explicit control:

d, err := ichkil.New(ctx)                    // default repository
defer d.Close()
got, err := d.Diacritize("الكتاب على الطاولة")

d, err := ichkil.NewFromFiles("model.onnx")  // fully offline

A Diacritizer is safe for concurrent use. Close releases the underlying ONNX Runtime session; it must not be called concurrently with other methods.

Index

• Constants
• Variables
• func Argmax(values []float32) int
• func Attach(base string, labels []int) string
• func ComboToLabel(combo string) int
• func Decode(base string, logits []float32, numLabels int) string
• func Diacritize(ctx context.Context, text string) (string, error)
• func Encode(text string, sym2ID map[string]int, unkID int) []int
• func IsArabicLetter(r rune) bool
• func LabelToCombo(label int) string
• func PredictLabels(ctx context.Context, text string) ([]int, error)
• func ResetDefault()
• func RuntimeVersion() string
• func SHA256OfFile(path string) (string, error)
• func SetDefault(d *Diacritizer)
• func StripDiacritics(text string) string
• func VerifyChecksum(path, expected string) error
• type Config
  • func LoadConfig(path string) (Config, error)
• type Diacritizer
  • func New(ctx context.Context, opts ...Option) (*Diacritizer, error)
  • func NewFromFiles(modelPath string, opts ...Option) (*Diacritizer, error)
  • func (d *Diacritizer) Close() error
  • func (d *Diacritizer) Diacritize(text string) (string, error)
  • func (d *Diacritizer) MaxSeqLen() int
  • func (d *Diacritizer) NumLabels() int
  • func (d *Diacritizer) PredictLabels(text string) ([]int, error)
  • func (d *Diacritizer) RepoID() string
  • func (d *Diacritizer) Revision() string
  • func (d *Diacritizer) String() string
• type ModelBundle
  • func ResolveModel(ctx context.Context, cfg *config) (ModelBundle, error)
• type Option
  • func WithBaseURL(baseURL string) Option
  • func WithCacheDir(dir string) Option
  • func WithConfigPath(path string) Option
  • func WithNumThreads(n int) Option
  • func WithRepoID(repoID string) Option
  • func WithRevision(revision string) Option
  • func WithToken(token string) Option
  • func WithoutChecksumVerification() Option

Constants

const (
	// DefaultRepoID is the Hugging Face model repository.
	DefaultRepoID = "ichkil/ichkil"
	// DefaultRevision is the branch fetched by default.
	DefaultRevision = "main"
)

Defaults for model resolution.

const (
	// LabelNonLetter is the label for non-letter tokens (spaces, punctuation)
	// or padding.
	LabelNonLetter = 0
	// LabelNone is the label for a letter that carries no diacritic.
	LabelNone = 1
	// NumLabels is the number of distinct labels the model can emit.
	NumLabels = 13
)

Label constants.

const Version = "1.0.0"

Version is the semantic version of the ichkil Go library.

Variables

var (
	// ErrChecksumMismatch is returned when the SHA-256 of model.onnx does not
	// match the expected digest (from SHA256SUMS or config.sha256).
	ErrChecksumMismatch = errors.New("ichkil: model checksum mismatch")

	// ErrModelLoad is returned when the model artifact or its config is
	// missing, invalid or unreadable (including ONNX Runtime environment
	// initialization failures).
	ErrModelLoad = errors.New("ichkil: model load error")

	// ErrInference is returned when an ONNX Runtime inference fails.
	ErrInference = errors.New("ichkil: inference error")
)

Sentinel errors returned by the ichkil package. Use errors.Is to test for them, e.g. errors.Is(err, ichkil.ErrChecksumMismatch).

Functions

func Argmax

func Argmax(values []float32) int

Argmax returns the index of the maximum value (first one wins ties), matching ONNX Runtime.

func Attach

func Attach(base string, labels []int) string

Attach appends predicted diacritics onto the bare letters of base. labels is one entry per character of base (a shorter slice leaves the remaining characters undecorated). Non-letter characters pass through unchanged.

func ComboToLabel

func ComboToLabel(combo string) int

ComboToLabel maps a diacritic combination (or "") to its label id (0 if unknown).

func Decode

func Decode(base string, logits []float32, numLabels int) string

Decode attaches predicted diacritics onto base from flat logits. logits is the model output row for base in row-major order (len(logits) == len(base) * numLabels). Non-letter characters pass through unchanged.

func Diacritize

func Diacritize(ctx context.Context, text string) (string, error)

Diacritize vocalizes bare Arabic text using a shared default Diacritizer (created lazily on first use — it downloads the model once, then works offline).

func Encode

func Encode(text string, sym2ID map[string]int, unkID int) []int

Encode maps each rune to its vocabulary id (out-of-vocabulary -> unkID).

func IsArabicLetter

func IsArabicLetter(r rune) bool

IsArabicLetter reports whether r is a single Arabic letter (U+0621..U+064A).

func LabelToCombo

func LabelToCombo(label int) string

LabelToCombo maps a label id to its diacritic combination ("" for unknown ids).

func PredictLabels

func PredictLabels(ctx context.Context, text string) ([]int, error)

PredictLabels returns per-character labels for text using a shared default Diacritizer.

func ResetDefault

func ResetDefault()

ResetDefault discards the shared default Diacritizer (if any), forcing a fresh model resolution + session on the next convenience call.

func RuntimeVersion

func RuntimeVersion() string

RuntimeVersion returns the version of the loaded ONNX Runtime shared library (empty string when it is not available).

func SHA256OfFile

func SHA256OfFile(path string) (string, error)

SHA256OfFile returns the hex SHA-256 digest of the file at path.

func SetDefault

func SetDefault(d *Diacritizer)

SetDefault installs d as the shared default Diacritizer used by Diacritize and PredictLabels.

func StripDiacritics

func StripDiacritics(text string) string

StripDiacritics returns text with all diacritics and tatweel removed. Letters, spaces and punctuation are preserved so the result stays aligned with the original character positions.

func VerifyChecksum

func VerifyChecksum(path, expected string) error

VerifyChecksum returns an error wrapping ErrChecksumMismatch if the file's SHA-256 differs from expected (compared case-insensitively).

Types

type Config

type Config struct {
	// Sym2ID maps every vocabulary symbol to its token id.
	Sym2ID map[string]int `json:"sym2id"`
	// NumTags is the number of output labels (13).
	NumTags int `json:"num_tags"`
	// MaxSeqLen is the maximum sequence length of the model (1024).
	MaxSeqLen int `json:"max_seq_len"`
	// SHA256 is the optional expected digest of model.onnx (fallback when the
	// repository has no SHA256SUMS file).
	SHA256 string `json:"sha256"`
}

Config is the runtime configuration loaded from the model's config.json.

func LoadConfig

func LoadConfig(path string) (Config, error)

LoadConfig loads and minimally validates a config.json.

type Diacritizer

type Diacritizer struct {
	// contains filtered or unexported fields
}

Diacritizer is a ready-to-use Arabic diacritizer backed by ONNX Runtime.

func New

func New(ctx context.Context, opts ...Option) (*Diacritizer, error)

New creates a Diacritizer by resolving the model from Hugging Face (cached after the first download) and opening an ONNX Runtime session.

func NewFromFiles

func NewFromFiles(modelPath string, opts ...Option) (*Diacritizer, error)

NewFromFiles creates a Diacritizer from local model.onnx + config.json files (fully offline). The config defaults to config.json next to the model; use WithConfigPath to point elsewhere.

func (*Diacritizer) Close

func (d *Diacritizer) Close() error

Close releases the ONNX Runtime session. The diacritizer must not be used after Close; Close is idempotent. The shared ONNX Runtime environment (process-wide) is left intact.

func (*Diacritizer) Diacritize

func (d *Diacritizer) Diacritize(text string) (string, error)

Diacritize turns bare (or already vocalized) Arabic into fully-vocalized text. Diacritics are stripped first, so the result is canonical and idempotent; non-Arabic characters pass through unchanged.

Diacritize("محمد قرأ الكتاب") -> "مُحَمَّدٌ قَرَأَ الْكِتَابِ"

func (*Diacritizer) MaxSeqLen

func (d *Diacritizer) MaxSeqLen() int

MaxSeqLen returns the model's maximum sequence length (1024).

func (*Diacritizer) NumLabels

func (d *Diacritizer) NumLabels() int

NumLabels returns the number of distinct labels (13).

func (*Diacritizer) PredictLabels

func (d *Diacritizer) PredictLabels(text string) ([]int, error)

PredictLabels returns the predicted label id (0..NumLabels-1) for every character of text (diacritics stripped first). Empty input yields a nil slice.

func (*Diacritizer) RepoID

func (d *Diacritizer) RepoID() string

RepoID returns the Hugging Face repository the model came from ("<local>" for NewFromFiles).

func (*Diacritizer) Revision

func (d *Diacritizer) Revision() string

Revision returns the branch / tag / commit (or model file name) the model came from.

func (*Diacritizer) String

func (d *Diacritizer) String() string

type ModelBundle

type ModelBundle struct {
	// OnnxPath is the absolute path to the local model.onnx.
	OnnxPath string
	// Config is the parsed runtime configuration.
	Config Config
	// RepoID is the Hugging Face repository the bundle came from.
	RepoID string
	// Revision is the branch / tag / commit the bundle was fetched from.
	Revision string
}

ModelBundle is a resolved, verified local model artifact plus its config.

func ResolveModel

func ResolveModel(ctx context.Context, cfg *config) (ModelBundle, error)

ResolveModel downloads (or reuses the cached) model files from Hugging Face: model.onnx, config.json and — when checksum verification is enabled — SHA256SUMS. Everything is cached under $ICHKIL_CACHE_DIR (or ~/.cache/ichkil by default), so only the first call per revision touches the network.

type Option

type Option func(*config)

Option customizes model resolution and the ONNX Runtime session.

func WithBaseURL

func WithBaseURL(baseURL string) Option

WithBaseURL sets the Hugging Face base URL (default: $HF_ENDPOINT or https://huggingface.co).

func WithCacheDir

func WithCacheDir(dir string) Option

WithCacheDir sets the model cache directory (default $ICHKIL_CACHE_DIR or ~/.cache/ichkil).

func WithConfigPath

func WithConfigPath(path string) Option

WithConfigPath sets the config.json used by NewFromFiles (default: config.json next to the model).

func WithNumThreads

func WithNumThreads(n int) Option

WithNumThreads sets the ONNX Runtime intra-op thread count (0 = ORT heuristic, the default).

func WithRepoID

func WithRepoID(repoID string) Option

WithRepoID sets the Hugging Face model repository (default "ichkil/ichkil").

func WithRevision

func WithRevision(revision string) Option

WithRevision sets the branch, tag or commit to fetch (default "main").

func WithToken

func WithToken(token string) Option

WithToken sets the Hugging Face token used for gated / private repositories (default: $HF_TOKEN).

func WithoutChecksumVerification

func WithoutChecksumVerification() Option

WithoutChecksumVerification disables the SHA-256 verification of model.onnx against SHA256SUMS (enabled by default).