README
¶
khatool
khatool — KittyTK's Hebrew & Arabic Text Output & Ordering Library ("חתול" — cat)
If you use this, please support me on ko-fi: https://ko-fi.com/jeffday
Hebrew and Arabic for renderers that draw one cell at a time. It answers two
questions — what order the runes of a line are drawn in, and which glyphs
to draw — and nothing else: it has no opinion about fonts, cells, terminals or
documents, and its only dependency is golang.org/x/text.
It's a sibling to the rest of the line: Mew (text editor), PurfecTerm (terminal emulator), PawScript (language), KittyTK (UI toolkit) and ifitfits (viewport tiling). The name is a pun on חתול — khatul, cat.
Model
- Ordering is the Unicode bidirectional algorithm (UAX #9, resolved by
x/text) with the base direction forced: a caller states the direction a line begins in, and it holds regardless of the line's first strong character. RTL runs reverse, numbers inside an RTL region keep their own digit order while the region mirrors around them, and the result comes back as a permutation — visual slot to logical index. Positions stay logical, so a caller's cursor, selection and hit-testing arithmetic is untouched. - Output is presentation-form substitution, one per script: Arabic letters
become their contextual joining forms (
Shape, including the mandatory lam-alef ligature), and Hebrew points fold into the letter that carries them (PrecomposeCluster). Both hand a renderer a single standalone glyph where it would otherwise have to shape or compose — which is exactly what a cell renderer, a bitmap font, or a terminal cannot be trusted to do. - Cell width is the caller's business. Where the ordering has to know which
runes share a cell — so a mark travels with its base when a run turns over —
it asks, through a
Ridesfunction the caller supplies. Two answers to that question means two disagreeing pictures of one line, so the answer belongs to whoever will draw it.niltakesDefaultRides.
Use
import "github.com/phroun/khatool"
runes := []rune(line)
lay := khatool.Order(runes, baseRTL, nil) // nil: the default cluster rule
if lay == nil {
draw(runes) // visual order is logical order — the common case
return
}
for _, i := range lay.Perm { // left to right on the screen
r := runes[i]
if lay.Glyph != nil { // Arabic shaped in logical order
if r = lay.Glyph[i]; r == khatool.LigatureAbsorbed {
continue // lam-alef: the pair took one cell
}
}
if lay.RTL[i] {
r = khatool.Mirror(r) // brackets face the way the run reads
}
drawCell(r)
}
A caller that draws ill-formed marks or control characters as something visible gives them cells of their own, and says so with a rule:
lay := khatool.Order(runes, baseRTL, func(runes []rune, i int) bool {
return myWidth(runes[i]) == 0 && !khatool.DefectiveMark(khatool.PrevBase(runes, i), runes[i])
})
Test
go test ./...
License
MIT — see LICENSE.
Documentation
¶
Overview ¶
Hebrew point folding: the points that Unicode also encodes as part of a single Alphabetic-Presentation-Form glyph are folded into that glyph, so a renderer that mispositions a free-standing point -- drawing a dagesh, dot or rafe a cell off its base -- has nothing left to misposition.
It is the Hebrew counterpart of Shape: both substitute a presentation form so that a renderer with no shaper of its own is handed one glyph where it would otherwise have to compose two.
Package khatool turns a line of Hebrew or Arabic text into the order and the glyphs a renderer needs when it can only draw one cell at a time: the visual ordering of the Unicode bidirectional algorithm (UAX #9, via golang.org/x/text), and the presentation forms that stand in for shaping.
ORDERING. A line starts in a base direction the caller states — forced, not guessed from the first strong character. Runs of RTL text are resolved by the full bidi classification and reversed; numbers inside an RTL region keep their own left-to-right digit order while the region as a whole mirrors (the L2 reordering step applied over the resolved runs). Positions in the text stay strictly LOGICAL: only the layout is visual, so a caller's own cursor, selection and hit-testing arithmetic is left alone.
OUTPUT. Arabic letters become their contextual presentation forms (Shape) and Hebrew points fold into theirs (Precompose): standalone glyphs a renderer can reverse, clip and colour cell by cell without a shaper of its own, and which a terminal cannot misplace.
The library holds no opinion about how wide a glyph is. Where it has to know which runes share a cell — to keep a mark with its base through a reversal — it asks the caller (see Rides).
Index ¶
- Constants
- func ComposedBase(runes []rune) (rune, bool)
- func DefaultRides(runes []rune, i int) bool
- func DefectiveMark(prev, r rune) bool
- func FlipRuns(bases []rune, wordwise bool) (order, style []int, mirror []bool)
- func FoldRidingMarks(runes []rune, folding bool, zeroWidth func(rune) bool) []rune
- func Folds(r rune) bool
- func HasZeroWidthAfterFold(runes []rune, folding bool, zeroWidth func(rune) bool) bool
- func IsControl(r rune) bool
- func IsDirectionControl(r rune) bool
- func IsMark(r rune) bool
- func IsStrongRTL(r rune) bool
- func MarkForm(r rune) string
- func Mirror(r rune) rune
- func PrecomposeCluster(runes []rune) ([]rune, bool)
- func PrevBase(runes []rune, i int) rune
- func RTLAt(runes []rune, idx int, baseRTL bool) bool
- func Shape(runes []rune) []rune
- func Substitute(r rune) string
- func UnplaceableFillAfterFold(runes []rune, folding bool, zeroWidth func(rune) bool, visual []int) []bool
- type Layout
- type Marks
- type Rides
Constants ¶
const ( MarkerLTR = -1 // ">" — an LTR fragment begins here (reading rightward) MarkerRTL = -2 // "<" — an RTL fragment begins here (reading leftward) MarkerEnd = -3 // "|" — the fragment's reading end )
Marker slot values that may appear in Layout.Perm when the layout was computed with direction markers: a negative entry is a synthetic one-column glyph at a fragment's edge rather than a real rune. The LTR/RTL markers point at a fragment's leading (reading-start) edge; MarkerEnd sits at the same fragment's trailing edge, showing where the foreign-direction run has ended.
const LigatureAbsorbed rune = -1
LigatureAbsorbed marks the second logical position of a mandatory ligature (the alef of a lam-alef pair): the glyph lives on the first position and this one takes no cell of its own. Selecting either position highlights the shared cell; deleting either code point breaks the ligature apart, since the remaining letter re-shapes on its own.
const MarkAnchor = '◌' // DOTTED CIRCLE
MarkAnchor is the base character an isolated mark is composed onto.
const MarkDropped rune = -2
MarkDropped stands where a combining mark was, for a caller told not to show the marks that ride a right-to-left letter. It is the same idea as LigatureAbsorbed: the position is still there, so a caller's own indexing survives, but nothing is drawn for it.
Variables ¶
This section is empty.
Functions ¶
func ComposedBase ¶
ComposedBase folds base + its folding points into the single presentation-form glyph, ignoring any vowels. It is PrecomposeCluster's base rune alone — for callers that fold the base but handle the vowels themselves (e.g. drift, which moves the vowels to another cell). Returns the base unchanged, ok=false, when nothing folds.
func DefaultRides ¶
DefaultRides is the rule for a caller with no width model to offer: a well-formed combining mark or a format character rides the cell before it, and everything else takes a cell of its own.
A caller that paints a control or an ill-formed mark as something VISIBLE -- a hex substitute, a dotted-circle anchor -- has given that rune a cell, and must say so with a rule of its own; this one assumes the plain reading.
func DefectiveMark ¶
DefectiveMark reports whether the combining mark r is ill-formed after the base character prev. Two cases:
- No base at all (prev == 0): the mark opens the line with nothing to anchor onto.
- The mark is SCRIPT-SPECIFIC and the base belongs to a different script: a Hebrew accent over a CJK ideograph, niqqud on a Latin letter, an NKo tone on punctuation. Unicode calls such a sequence ill-formed, and no shaper will compose it.
Both this library and wcwidth call every combining mark zero-width, which is a promise about what a renderer will do: paint the mark INTO the preceding cell and advance nothing. A mark with nothing to compose onto, or a base a shaper refuses to attach it to, breaks that promise: the fallback is a SPACING glyph - .notdef, or the shaper's own dotted-circle plus mark - which advances a column nobody budgeted for.
The test is about the SEQUENCE, never about which glyphs happen to be installed. Font inventory is deliberately not consulted: the glyph comes from whatever font the renderer ends up using, so any answer derived from one font's coverage would be about the wrong font. That also means legitimate text in a script the caller ships no face for - an NKo mark on an NKo letter - is left alone.
General diacritics (script=Inherited/Common - the U+0300..U+036F block, the Arabic vowel marks, the kana voicing marks) belong to no script and legitimately attach to any base, so they are never defective on this rule.
func FlipRuns ¶ added in v0.1.1
FlipRuns is the order to emit a row of cells in, for a host that applies its own bidi to what it receives.
bases is one rune per cell, left to right, as the renderer laid the row out; a cell with nothing in it takes zero. Three answers come back, one entry per emitted slot:
- order is which cell's GLYPH goes in that slot.
- style is which cell's ATTRIBUTES go there.
- mirror is whether that glyph goes out as the character it mirrors.
wordwise selects the run boundary AND the attribute mapping, to match how the host segments:
- Off: a run is a maximal right-to-left span with interior neutrals -- the spaces between words -- absorbed as long as another right-to-left cell follows. The whole span reverses, glyph and attributes together, because the host reverses the parsed cell, colour and all, as a unit.
- On: a space is a boundary, so each whitespace-separated right-to-left word is its own run and reverses in place. That host reverses the GLYPHS but paints attributes at the physical column, so the glyphs reverse while the attributes stay in the order they were laid out -- and each one then lands on the letter its glyph settled on.
func FoldRidingMarks ¶ added in v0.1.1
FoldRidingMarks says what to draw for each rune when the combining marks that ride a right-to-left letter are not to be shown. The answer is one rune per input rune, so a caller that has already worked out where each of them goes keeps its own indexing: the rune itself where nothing changes, the folded base where a cluster folds, and MarkDropped where a mark is not drawn at all.
It is what a display gives up when it wants its colour back. A terminal that reorders what it is sent miscounts a background fill over any line still carrying zero-width marks, and the marks are the only thing on such a line that can be given up -- so an application that would rather keep its selection bars, its highlights and its gutter than its vowels drops them, and pointed Hebrew renders one codepoint per cell the way pre-shaped Arabic does.
folding is what makes that bearable. A point with a presentation form (the dagesh, the shin and sin dots, the rafe, the holam-haser) folds INTO its letter, so it survives as one glyph and only the vowels and accents go. With folding off there is nothing to fold into and every riding mark is dropped.
A mark on a left-to-right base is left alone: it is not what the reordering miscounts, and this is not a rule about combining marks in general.
zeroWidth is the caller's own width model, for the reason it is everywhere else here: two answers about which runes take a cell is two disagreeing pictures of one line. nil takes every non-spacing mark.
func Folds ¶
Folds reports whether r is a Hebrew point that folds into its base's presentation form: the dagesh/mapiq, shin dot, sin dot, rafe, or the holam-haser-for-vav. Vowels and accents do not fold.
func HasZeroWidthAfterFold ¶ added in v0.1.1
HasZeroWidthAfterFold reports whether the runes still carry a ZERO-WIDTH one once each cluster has been folded into its presentation form.
It answers a question about what a renderer can safely paint OVER. A terminal that runs its own bidi counts codepoints where a grid counts cells, so a background fill -- a selection bar, a highlight -- over a line holding combining marks lands on the wrong cells and half-vanishes. Foreground colour and weight ride each glyph through that reordering intact, so a caller with such a line reaches for those instead.
Folding is what makes it worth asking per line rather than per script. A point that folds into its base no longer inflates the codepoint count, so a line of pointed consonants comes out even and keeps the ordinary fill; only marks that survive the fold -- vowels, accents, points with no form to fold into -- force the other treatment.
zeroWidth is the caller's own width model, for the reason Rides is: two answers about which runes take a cell means two disagreeing pictures of one line. nil takes every non-spacing mark, which is the plain reading.
func IsControl ¶
IsControl reports whether r is a C0 or C1 control character. A control has no legitimate glyph, so whatever a caller draws for one takes a cell.
func IsDirectionControl ¶
IsDirectionControl reports whether r is an explicit Unicode direction control (LRM, RLM, ALM, the embedding/override controls, or the isolate controls). Under showBidi these render as a visible one-column marker; a fragment led by one gets no additional synthetic marker — the character represents the transition itself.
func IsMark ¶
IsMark reports whether r is a combining mark - a codepoint that carries no cell of its own and paints into the preceding one.
func IsStrongRTL ¶
IsStrongRTL reports whether a rune is a strong right-to-left character (Hebrew, Arabic, and their presentation forms — bidi classes R and AL). It is the cheap question a caller asks of a finished row to find its RTL runs.
func MarkForm ¶ added in v0.1.1
MarkForm is what a caller draws for a mark DefectiveMark rejects: a dotted circle supplying the base it has none of, with the mark composed onto it.
This is the Unicode convention for showing an isolated combining mark, and it is what a shaper already does for a defective cluster -- so the reader sees the actual mark rather than a number, and it costs the circle's cell plus whatever the mark itself advances (nothing, for a non-spacing mark).
It matters that the pair is drawn rather than left as a bare mark. A mark with no base it can attach to is corruption, not text, and it cannot be painted as zero-width: a renderer whose shaper rejects the pairing falls back to a SPACING glyph that advances a cell nobody budgeted, sliding the rest of the line along.
func Mirror ¶
Mirror returns the paired counterpart of a bracket rune (for painting brackets inside RTL runs), or the rune unchanged.
func PrecomposeCluster ¶
PrecomposeCluster folds a Hebrew cluster — a base rune followed by its combining marks — for a terminal that mishandles free-standing points. It returns the runes to emit: the base with its folding points folded into one presentation-form glyph, followed by the vowels/accents that ride normally; and ok=true when a fold happened.
An isolated point anchored on a dotted circle (◌ + shin/sin dot, or ◌ + holam-haser) is shown on its faux base — the shin-with-dot or vav-with-holam glyph — since those points have no meaningful isolated rendering. A dagesh whose letter has no presentation form is dropped. ok=false when nothing folds (no such point, or a non-Hebrew base), and the caller emits the cluster as-is.
func PrevBase ¶
PrevBase returns the cluster base for the rune at index i: the nearest preceding rune that is not itself a combining mark, or 0 when there is none. It is what DefectiveMark wants for prev - a mark rides the last real character, not the mark in front of it.
func RTLAt ¶
RTLAt reports whether the rune at logical index idx sits in an RTL run (the registered rtl command). For an index at or past the end of the line the direction of the last rune applies; an empty line reports the base direction.
func Shape ¶
Shape returns a copy of runes with each Arabic letter replaced by its contextual presentation form, or nil if the line contains no Arabic letter (the common fast path). Combining marks and every non-Arabic rune are copied through unchanged, so the result stays one rune per input rune.
func Substitute ¶ added in v0.1.1
Substitute is the visible stand-in for a rune that must not reach the renderer as itself: the C0 controls and DEL as the caret forms every terminal user knows (^@, ^I, ^[), and anything else as its codepoint in hex.
C1 is the range that makes this more than a convenience. U+0080..U+009F arrives as ordinary two-byte UTF-8 and no width table calls it special, but a terminal decoding UTF-8 honours those codepoints as controls, and the range holds the string introducers -- DCS, SOS, CSI, ST, OSC, PM, APC. One of them emitted from a binary file makes the terminal swallow everything after it as a control string, so the rest of the line vanishes until a terminator that may never come.
The form is always plain ASCII, so its cell count is its rune count and a caller's width model cannot disagree with what gets drawn.
func UnplaceableFillAfterFold ¶ added in v0.1.1
func UnplaceableFillAfterFold(runes []rune, folding bool, zeroWidth func(rune) bool, visual []int) []bool
UnplaceableFillAfterFold marks, for each rune, whether a background fill on its cell can still be placed by a terminal that reorders what it is sent.
It is HasZeroWidthAfterFold's question asked per POSITION rather than of a whole line, because such a terminal misplaces the fill from one point onwards rather than everywhere. The damage starts at the first right-to-left run still carrying a zero-width mark after folding, and runs to the end of the line: that run's fill slides off the cells it was meant for, and everything after it slides with it. What comes BEFORE is placed correctly and keeps its fill, which is the whole point of asking -- a row of chrome and English with one pointed word in it has no business losing the fill on the half that would have been right.
The trailing half matters as much as the run itself. A lone selected full stop after a pointed word is in no run of its own, and left with a fill it lands somewhere else entirely: one stray cell of colour behind a letter that was never selected.
A run is what such a terminal takes as that unit: a span opened by a strong right-to-left rune, carrying the marks that ride its letters, and absorbing what is neither strong nor a mark -- the spaces between words -- as long as another strong right-to-left rune follows before any strong left-to-right one. Runes outside every run are false: there is nothing to reorder and the fill lands where it was put.
visual is the order the cells go out in: visual[slot] is the logical index of the rune drawn in that slot, which is what a caller that has already ordered the line holds. It matters because "after" means after in the STREAM, and on a line whose base direction is right-to-left that is the opposite end of the rune array -- mark the array's tail there and the wrong half of the line gives up its fill. nil says the two orders are the same, which they are on a line with no ordering to do.
The answer comes back indexed by LOGICAL rune either way, since that is what a caller asks its questions in.
zeroWidth is the caller's own width model, as it is for HasZeroWidthAfterFold and for Rides, so one line does not get two disagreeing pictures of itself. nil takes every non-spacing mark.
Types ¶
type Layout ¶
type Layout struct {
// Perm maps visual slot -> logical rune index: painting the line means
// drawing runes[Perm[0]], runes[Perm[1]], ... left to right. When Marked,
// negative MarkerLTR/MarkerRTL entries are synthetic direction-marker
// cells (one column each) at fragment leading edges.
Perm []int
// RTL reports, per LOGICAL rune index, whether that rune lives in a
// right-to-left run (used for bracket mirroring and the rtl command).
RTL []bool
// Marked records that the layout was computed with direction markers:
// Perm may hold marker slots, and explicit direction-control characters
// (normally zero-width) render one column wide as their own marker.
Marked bool
// Glyph holds the shaped glyph per LOGICAL rune index (Arabic cursive
// presentation forms and lam-alef ligatures), or nil when the line has no
// Arabic. A LigatureAbsorbed entry takes no cell (see Shape). Consumers
// paint Glyph[i] and treat an absorbed entry as zero-width.
Glyph []rune
}
Layout is the computed visual arrangement of one line.
func Order ¶
Order returns the visual layout of a line under the base direction, or nil when visual order equals logical order (the common pure-LTR case — the caller should use its ordinary sequential path).
rides says which runes share a cell with the one before them, so a mark stays with its base when a run is turned over; nil takes DefaultRides.
Reordering applies UAX #9's L2 step over the resolved runs:
- LTR base: maximal regions of RTL runs (absorbing numeric runs flanked by RTL on both sides) mirror as a whole — run sequence reversed, RTL run contents reversed, numeric runs kept digit-order.
- RTL base: the entire run sequence is reversed (the line reads from the right), RTL run contents reversed, LTR/numeric run contents kept.
func OrderMarked ¶
OrderMarked is Order with direction markers: a synthetic one-column marker slot is injected at the leading edge of every fragment except the line-initial fragment when it is in the natural (base) direction — and except fragments led by an explicit direction-control character, which represents the transition itself (rendered one column wide by consumers when Layout.Marked is set). It is how a caller shows the reader where the direction turned.
type Marks ¶ added in v0.1.1
type Marks struct {
// LineEnd draws the closing bar at the LINE's own reading end -- past the
// last fragment, when that fragment reads the way the line does.
//
// As notation it says nothing the line has not: the reading stops at the
// end of the line whether or not a bar marks it. As a CELL it is somewhere
// to put the caret's last position, which on a line whose reading ends at
// the left sits out past the leftmost character with no character of its
// own to stand on. A caller that reserves its own room there leaves this
// off; one that has nowhere else to put that caret keeps it.
LineEnd bool
}
Marks says which of a marked line's marks are drawn.
type Rides ¶
Rides reports whether the rune at index i is drawn INTO the cell before it rather than taking one of its own -- a well-formed combining mark, a joiner, a zero-width space.
This is the CLUSTER rule, and it belongs to the caller because it has to be the same question the caller's own width model answers: two answers means two disagreeing pictures of one line, and a reversal made against the wrong one puts marks on the wrong cells. A caller with no width model of its own can pass nil and take DefaultRides.
Source Files