bidi

README

bidi

A pure-Go, CGO_ENABLED=0, standard-library-only implementation of the Unicode Bidirectional Algorithm (UAX #9) for laying out mixed left-to-right / right-to-left text.

Unlike most Go bidi implementations, go-opentype/bidi does not depend on golang.org/x/text. The Bidi_Class and paired-bracket Unicode properties are compiled into small generated lookup tables (see cmd/genbidi), so the package builds anywhere the standard library does.

Install

go get github.com/go-opentype/bidi

Usage

package main

import (
	"fmt"

	"github.com/go-opentype/bidi"
)

func main() {
	// A logical-order string mixing English and Hebrew.
	s := "abc אבג def"

	// Visual (left-to-right) order for display.
	fmt.Println(bidi.VisualOrder(s, bidi.Auto))

	// Or work with levels directly.
	runes := []rune(s)
	levels := bidi.ResolveLevels(runes, bidi.LeftToRight)
	order := bidi.Reorder(runes, levels) // visual-order permutation of indices
	fmt.Println(levels, order)

	// Inspect a single rune's Bidi_Class.
	fmt.Println(bidi.ClassOf('א')) // R
}

See example_test.go for runnable examples of each of the functions above, and go doc github.com/go-opentype/bidi for the full reference.

API

Symbol	Purpose
ClassOf(r rune) Class	Bidi_Class of a rune
Class enum	L R AL EN ES ET AN CS NSM BN B S WS ON LRE RLE LRO RLO PDF LRI RLI FSI PDI
ResolveLevels(text []rune, base Direction) []Level	resolved embedding level per rune
BaseLevel(text []rune, base Direction) Level	paragraph level (rules P2/P3)
Reorder(text []rune, levels []Level) []int	rule L2 visual-order permutation
ReorderWithMarks(text []rune, levels []Level) []int	L2 + rule L3 combining-mark reorder
VisualOrder(text string, base Direction) string	resolve + reorder convenience
Paragraphs(text []rune) [][]rune	rule P1 paragraph splitting
VisualParagraphs(text string, base Direction) []string	full display pipeline per paragraph
Mirror(r rune) rune	rule L4 mirrored glyph of a rune
MirrorRunes(text []rune, levels []Level) []rune	mirror characters at RTL levels
JoinForms(text []rune) []JoinForm	Arabic cursive form per character
PresentationForm(r rune, form JoinForm) rune	Arabic Presentation Forms-B fallback
JoinForm	Isolated, Initial, Medial, Final
Direction	LeftToRight, RightToLeft, Auto
Level	embedding level (even = LTR, odd = RTL)

Implemented vs deferred

Implemented — the core algorithm runs through rule L2, the full extent covered by the Unicode conformance file BidiCharacterTest.txt, with P1, L3, L4 and Arabic joining layered on top for display:

• P1 paragraph splitting on Paragraph_Separator (Paragraphs), with CR+LF treated as a single separator.
• P2, P3 base paragraph level from the first strong character.
• X1–X8 explicit embeddings and isolates (with overflow handling and the directional status stack); X9 removal of the deprecated formatting characters and BN; X10 isolating run sequences with sos/eos.
• W1–W7 weak types.
• N0 paired-bracket resolution (BD16, incl. the U+2329/U+232A canonical equivalence), N1–N2 neutral types.
• I1, I2 implicit levels.
• L1 separator / trailing-whitespace reset, L2 reordering.
• L3 combining marks kept adjacent to their base after reordering (ReorderWithMarks; plain Reorder stays L2-only to match the conformance data).
• L4 glyph mirroring for characters at right-to-left levels (Mirror, MirrorRunes).
• Arabic cursive joining at the Unicode level (JoinForms), plus a static Arabic Presentation Forms-B fallback (PresentationForm).

Deferred:

• Full contextual shaping — JoinForms resolves each letter's isolated/initial/medial/final form, but real rendering needs the font's GSUB init/medi/fina/isol features and contextual ligatures (such as the mandatory LAM+ALEF ligature). PresentationForm is only a per-letter fallback for the common letters, not a shaper.

Conformance

The package is validated against the entire BidiCharacterTest.txt (all cases pass: paragraph level, per-character levels and visual order). A curated representative subset is embedded under testdata and run by TestConformance. CI enforces exactly 100% statement coverage, go vet, gofmt, and cross-compilation for the six 64-bit architectures plus js/wasm, darwin/arm64 and windows/amd64.

Regenerating the tables

go run ./cmd/genbidi .

This fetches the latest DerivedBidiClass.txt, BidiBrackets.txt, BidiMirroring.txt and ArabicShaping.txt from the Unicode Character Database and rewrites bidiclass_table.go, bidibrackets_table.go, bidimirror_table.go and joining_table.go.

Part of the go-opentype pure-Go text stack

go-opentype/bidi is the Unicode Bidirectional Algorithm (UBA) layer of a dependency-free text stack:

• opentype — the parsing, GSUB/GPOS shaping and rasterising engine.
• bidi (this repo) — orders mixed left-to-right/right-to-left text into visual order before it is shaped.
• shape — a HarfBuzz-lite complex-script shaper (Arabic, Indic, Hangul, USE, Egyptian hieroglyphs, ...) built on opentype's GSUB/GPOS engine; it consumes this package's join forms and reordering for right-to-left scripts.
• fonts — 36 bundled OFL/BSD font families, per-family lazily go:embed-ed, ready to feed to opentype.Parse.

License

BSD-3-Clause. See LICENSE.

Documentation

Overview

Package bidi is a pure-Go, CGO=0, standard-library-only implementation of the Unicode Bidirectional Algorithm (UAX #9) for laying out mixed left-to-right / right-to-left text.

It has no dependency on golang.org/x/text: the Bidi_Class property and the paired-bracket property are shipped as generated lookup tables (see cmd/genbidi), so the package builds anywhere the standard library does.

Public API

• ClassOf reports the Bidi_Class of a rune, and the Class enum enumerates every Bidi_Class value.
• ResolveLevels runs the algorithm over a paragraph and returns the resolved embedding Level of each rune.
• BaseLevel reports the paragraph embedding level chosen by rules P2/P3.
• Reorder applies rule L2 to a level slice, returning the visual-order index permutation; ReorderWithMarks additionally applies the rule L3 combining-mark refinement.
• VisualOrder is a convenience that resolves and reorders a string in one call.
• Paragraphs splits text into paragraphs (rule P1), and VisualParagraphs runs the whole display pipeline per paragraph.
• Mirror and MirrorRunes apply glyph mirroring (rule L4).
• JoinForms resolves Arabic cursive presentation forms, and PresentationForm maps a letter to the Arabic Presentation Forms-B block.
• Direction selects the base direction: LeftToRight, RightToLeft or Auto.

Implemented rules

The core algorithm runs through rule L2, the full extent verified by the Unicode conformance file BidiCharacterTest.txt, with rules L3, L4 and P1 and Arabic joining layered on top for display:

• P1: splitting text into paragraphs on Paragraph_Separator (Paragraphs).
• P2, P3: paragraph embedding level from the first strong character.
• X1–X8: explicit embeddings (LRE/RLE/LRO/RLO/PDF) and isolates (LRI/RLI/FSI/PDI), including overflow handling and the directional status stack, with FSI resolved per X5c.
• X9: the deprecated embedding/override/PDF formatting characters and BN are removed (reported as carrying the preceding character's level).
• X10: isolating run sequences with their sos/eos boundary types.
• W1–W7: the weak-type rules.
• N0: paired-bracket resolution (BD16), including the U+2329/U+232A ~ U+3008/U+3009 canonical equivalence.
• N1, N2: the neutral-type rules.
• I1, I2: the implicit level rules.
• L1: resetting separators and trailing whitespace to the paragraph level.
• L2: reordering to visual order.
• L3: keeping combining marks adjacent to their base after reordering (ReorderWithMarks); the plain Reorder applies L2 only, matching the conformance data.
• L4: substituting mirrored glyphs for characters at right-to-left levels (Mirror, MirrorRunes).

It also provides the Unicode-level Arabic cursive joining algorithm (JoinForms) and a static fallback to the Arabic Presentation Forms-B block (PresentationForm).

The package is validated against the full BidiCharacterTest.txt (all cases pass); a curated subset is embedded under testdata for the committed test suite.

Deferred

• Full contextual shaping: JoinForms resolves the isolated/initial/ medial/final form of each Arabic letter, but real display requires the font's GSUB init/medi/fina/isol features and ligatures (such as the mandatory LAM+ALEF ligature). PresentationForm is only a static per-letter fallback for the common letters, not a shaper.

Index

• func Mirror(r rune) rune
• func MirrorRunes(text []rune, levels []Level) []rune
• func Paragraphs(text []rune) [][]rune
• func PresentationForm(r rune, form JoinForm) rune
• func Reorder(text []rune, levels []Level) []int
• func ReorderWithMarks(text []rune, levels []Level) []int
• func VisualOrder(text string, base Direction) string
• func VisualParagraphs(text string, base Direction) []string
• type Class
  • func ClassOf(r rune) Class
  • func (c Class) String() string
• type Direction
• type JoinForm
  • func JoinForms(text []rune) []JoinForm
• type JoiningType
• type Level
  • func BaseLevel(text []rune, base Direction) Level
  • func ResolveLevels(text []rune, base Direction) []Level

Examples

• ClassOf
• JoinForms
• Mirror
• ResolveLevels
• VisualOrder
• VisualParagraphs

Functions

func Mirror

func Mirror(r rune) rune

Mirror implements the character half of rule L4: it returns the code point of the character whose glyph mirrors r (the Bidi_Mirroring_Glyph property), or r itself when r has no mirrored counterpart.

Mirror is unconditional; it does not consult an embedding level. Callers that only want to mirror characters resolved to a right-to-left level should use MirrorRunes, or guard the call with the character's Level.

Example

ExampleMirror applies rule L4: a paired punctuation mark gets its mirrored glyph when it will be displayed at a right-to-left level, while a letter like Hebrew alef (which has no mirrored counterpart) is returned unchanged.

package main

import (
	"fmt"

	"github.com/go-opentype/bidi"
)

func main() {
	fmt.Printf("%c\n", bidi.Mirror('('))
	fmt.Println(bidi.Mirror('א') == 'א')
}

Output:
)
true

func MirrorRunes

func MirrorRunes(text []rune, levels []Level) []rune

MirrorRunes implements rule L4 over a paragraph: it returns a copy of text in which every character resolved to an odd (right-to-left) embedding level is replaced by its mirrored glyph (see Mirror); characters at even levels, and characters without a mirror, are copied unchanged.

levels must be the per-rune levels of text (as returned by ResolveLevels); a rune with no corresponding level is treated as left-to-right.

func Paragraphs

func Paragraphs(text []rune) [][]rune

Paragraphs implements rule P1: it splits text into paragraphs on characters of Bidi_Class B (Paragraph_Separator), returning the paragraphs in order.

Each Paragraph_Separator is kept at the end of the paragraph it terminates, and a CR+LF pair (U+000D U+000A) is treated as a single separator. Text after the final separator forms a last paragraph; when text ends with a separator no trailing empty paragraph is produced. Each returned paragraph can be fed independently to ResolveLevels, BaseLevel or VisualOrder.

func PresentationForm

func PresentationForm(r rune, form JoinForm) rune

PresentationForm returns the Arabic Presentation Forms-B code point for the letter r in cursive form (as produced by JoinForms). It is a static fallback for common letters only: when r has no mapping, or no glyph for the requested form, r is returned unchanged.

This is the Unicode-level fallback. Correct rendering requires the font's GSUB init/medi/fina/isol features and contextual ligatures, which this package does not perform.

func Reorder

func Reorder(text []rune, levels []Level) []int

Reorder implements rule L2. Given the per-rune levels (as returned by ResolveLevels for the same text), it returns the visual left-to-right order as a permutation of indices into text.

func ReorderWithMarks

func ReorderWithMarks(text []rune, levels []Level) []int

ReorderWithMarks runs rule L2 and then the rule L3 combining-mark refinement, returning the visual left-to-right order as a permutation of indices into text. It is the mark-aware counterpart of Reorder: the plain Reorder applies L2 only, matching the extent of the Unicode conformance data.

func VisualOrder

func VisualOrder(text string, base Direction) string

VisualOrder is a convenience that resolves levels for text and returns the characters in visual (left-to-right) order. Characters removed by rule X9 are dropped. Glyph mirroring (rule L4) is not applied; see the package documentation.

Example

ExampleVisualOrder reorders a logical-order string mixing English (L) and Hebrew (R) into left-to-right visual order, the one-call convenience most callers want.

package main

import (
	"fmt"

	"github.com/go-opentype/bidi"
)

func main() {
	s := "abc אבג def"
	fmt.Println(bidi.VisualOrder(s, bidi.Auto))
}

Output:
abc גבא def

func VisualParagraphs

func VisualParagraphs(text string, base Direction) []string

VisualParagraphs runs the full display pipeline on multi-paragraph text: it splits text into paragraphs (rule P1), then for each paragraph resolves levels, reorders to visual order (L2), keeps combining marks with their base (L3) and mirrors right-to-left glyphs (L4). It returns one visual (left-to-right) string per paragraph, with characters removed by rule X9 dropped.

Example

ExampleVisualParagraphs runs the full display pipeline (split into paragraphs, resolve, reorder) over multi-paragraph text in one call.

package main

import (
	"fmt"

	"github.com/go-opentype/bidi"
)

func main() {
	for _, p := range bidi.VisualParagraphs("abc\nאבג", bidi.Auto) {
		fmt.Printf("%q\n", p)
	}
}

Output:
"abc\n"
"גבא"

Types

type Class

type Class uint8

Class is a Unicode Bidi_Class property value, as defined by UAX #9 and UAX #44. Every rune has exactly one Bidi_Class.

const (
	// Strong types.
	L  Class = iota // Left_To_Right
	R               // Right_To_Left
	AL              // Arabic_Letter

	// Weak types.
	EN  // European_Number
	ES  // European_Separator
	ET  // European_Terminator
	AN  // Arabic_Number
	CS  // Common_Separator
	NSM // Nonspacing_Mark
	BN  // Boundary_Neutral

	// Neutral types.
	B  // Paragraph_Separator
	S  // Segment_Separator
	WS // White_Space
	ON // Other_Neutral

	// Explicit formatting types.
	LRE // Left_To_Right_Embedding
	RLE // Right_To_Left_Embedding
	LRO // Left_To_Right_Override
	RLO // Right_To_Left_Override
	PDF // Pop_Directional_Format
	LRI // Left_To_Right_Isolate
	RLI // Right_To_Left_Isolate
	FSI // First_Strong_Isolate
	PDI // Pop_Directional_Isolate

)

The Bidi_Class values. The ordering matches the enumeration used throughout the Unicode Bidirectional Algorithm; do not rely on the numeric values other than for equality.

func ClassOf

func ClassOf(r rune) Class

ClassOf returns the Bidi_Class of r. Runes outside the assigned ranges follow the Unicode defaults declared by the @missing lines of DerivedBidiClass.txt (L in general, R/AL in unassigned right-to-left blocks, ET in the currency block, BN for default-ignorable and non-characters).

It is named ClassOf rather than Class because Go does not allow a function to share the name of the Class type it returns.

Example

ExampleClassOf reports the Unicode Bidi_Class of a few runes: a Hebrew letter is strong right-to-left (R), a Latin letter is strong left-to-right (L), and a digit is a European number (EN).

package main

import (
	"fmt"

	"github.com/go-opentype/bidi"
)

func main() {
	fmt.Println(bidi.ClassOf('א'))
	fmt.Println(bidi.ClassOf('A'))
	fmt.Println(bidi.ClassOf('1'))
}

Output:
R
L
EN

func (Class) String

func (c Class) String() string

String returns the short Bidi_Class abbreviation (for example "AL"). An out-of-range Class value renders as "Class(N)".

type Direction

type Direction int

Direction selects the base paragraph direction passed to the algorithm.

const (
	// LeftToRight forces a base paragraph embedding level of 0.
	LeftToRight Direction = iota
	// RightToLeft forces a base paragraph embedding level of 1.
	RightToLeft
	// Auto derives the base level from the first strong character (rules P2
	// and P3), defaulting to left-to-right.
	Auto
)

type JoinForm

type JoinForm int

JoinForm is the cursive presentation form an Arabic-script letter takes, according to its neighbours.

const (
	// Isolated is the standalone form (no cursive connection).
	Isolated JoinForm = iota
	// Initial connects only to the following letter.
	Initial
	// Medial connects to both neighbours.
	Medial
	// Final connects only to the preceding letter.
	Final
)

func JoinForms

func JoinForms(text []rune) []JoinForm

JoinForms resolves the cursive presentation form of every character in text per the Arabic joining algorithm. Transparent characters (combining marks) are reported as Isolated and are skipped when joining their neighbours; a letter joins to an adjacent letter only when both are able to connect on the shared side.

The result is the Unicode-level joining form; a full shaper additionally applies the font's GSUB init/medi/fina/isol features and ligatures (such as the mandatory LAM+ALEF ligature). See PresentationForm for a static fallback mapping to the Arabic Presentation Forms-B block.

Example

ExampleJoinForms resolves the Arabic cursive presentation form of each letter in a word: an initial-joining letter, a dual-joining letter taking its medial form, a right-joining-only letter taking its final form, and a letter that starts a fresh join (isolated) because its predecessor does not connect forward.

package main

import (
	"fmt"

	"github.com/go-opentype/bidi"
)

func main() {
	names := map[bidi.JoinForm]string{
		bidi.Isolated: "Isolated",
		bidi.Initial:  "Initial",
		bidi.Medial:   "Medial",
		bidi.Final:    "Final",
	}
	word := []rune("سلام") // "peace"
	for i, form := range bidi.JoinForms(word) {
		fmt.Printf("%c: %s\n", word[i], names[form])
	}
}

Output:
س: Initial
ل: Medial
ا: Final
م: Isolated

type JoiningType

type JoiningType uint8

JoiningType is the Arabic Joining_Type property of a character (UAX #9 / Section 9.2 of the Unicode Standard), which drives cursive joining.

type Level

type Level int

Level is a bidirectional embedding level as defined by UAX #9. Even levels are left-to-right; odd levels are right-to-left.

func BaseLevel

func BaseLevel(text []rune, base Direction) Level

BaseLevel returns the resolved paragraph embedding level chosen for text under base (rules P2/P3): 0 for left-to-right, 1 for right-to-left.

func ResolveLevels

func ResolveLevels(text []rune, base Direction) []Level

ResolveLevels runs the Unicode Bidirectional Algorithm over text and returns the resolved embedding level of every rune. Even levels are left-to-right, odd levels right-to-left. Characters removed by rule X9 (the deprecated embedding/override formatting characters and BN) carry the level of the preceding retained character.

Example

ExampleResolveLevels shows the lower-level API VisualOrder builds on: ResolveLevels assigns an embedding Level to every rune (even = LTR, odd = RTL), and Reorder turns those levels into a visual-order index permutation.

package main

import (
	"fmt"

	"github.com/go-opentype/bidi"
)

func main() {
	text := []rune("abc אבג def")
	levels := bidi.ResolveLevels(text, bidi.LeftToRight)
	order := bidi.Reorder(text, levels)

	fmt.Println(levels)
	fmt.Println(order)
}

Output:
[0 0 0 0 1 1 1 0 0 0 0]
[0 1 2 3 6 5 4 7 8 9 10]