displaywidth

README

displaywidth

A high-performance Go package for measuring the monospace display width of strings, UTF-8 bytes, and runes.

Install

go get github.com/clipperhouse/displaywidth

Usage

package main

import (
    "fmt"
    "github.com/clipperhouse/displaywidth"
)

func main() {
    width := displaywidth.String("Hello, 世界!")
    fmt.Println(width)

    width = displaywidth.Bytes([]byte("🌍"))
    fmt.Println(width)

    width = displaywidth.Rune('🌍')
    fmt.Println(width)
}

For most purposes, you should use the String or Bytes methods. They sum the widths of grapheme clusters in the string or byte slice.

Note: in your application, iterating over runes to measure width is likely incorrect; the smallest unit of display is a grapheme, not a rune.

Iterating over graphemes

If you need the individual graphemes:

import (
    "fmt"
    "github.com/clipperhouse/displaywidth"
)

func main() {
    g := displaywidth.StringGraphemes("Hello, 世界!")
    for g.Next() {
        width := g.Width()
        value := g.Value()
        // do something with the width or value
    }
}

Options

There is one option, displaywidth.Options.EastAsianWidth, which defines how East Asian Ambiguous characters are treated.

When false (default), East Asian Ambiguous characters are treated as width 1. When true, they are treated as width 2.

You may wish to configure this based on environment variables or locale. go-runewidth, for example, does so during package initialization.

displaywidth does not do this automatically, we prefer to leave it to you. You might do something like:

var width displaywidth.Options // zero value is default

func init() {
    if os.Getenv("EAST_ASIAN_WIDTH") == "true" {
        width = displaywidth.Options{EastAsianWidth: true}
    }
    // or check locale, or any other logic you want
}

// use it in your logic
func myApp() {
    fmt.Println(width.String("Hello, 世界!"))
}

Technical standards and compatibility

This package implements the Unicode East Asian Width standard (UAX #11), and handles version selectors, and regional indicator pairs (flags). We implement Unicode TR51. We are keeping an eye on emerging standards.

clipperhouse/displaywidth, mattn/go-runewidth, and rivo/uniseg will give the same outputs for most real-world text. Extensive details are in the compatibility analysis.

If you wish to investigate the core logic, see the lookupProperties and width functions in width.go. The essential trie generation logic is in buildPropertyBitmap in unicode.go.

Prior Art

mattn/go-runewidth

rivo/uniseg

x/text/width

x/text/internal/triegen

Benchmarks

cd comparison
go test -bench=. -benchmem

goos: darwin
goarch: arm64
pkg: github.com/clipperhouse/displaywidth/comparison
cpu: Apple M2

BenchmarkString_Mixed/clipperhouse/displaywidth-8             10400 ns/op	  162.21 MB/s	      0 B/op	     0 allocs/op
BenchmarkString_Mixed/mattn/go-runewidth-8                    14296 ns/op	  118.00 MB/s	      0 B/op	     0 allocs/op
BenchmarkString_Mixed/rivo/uniseg-8                           19770 ns/op	   85.33 MB/s	      0 B/op	     0 allocs/op

BenchmarkString_EastAsian/clipperhouse/displaywidth-8         10593 ns/op	  159.26 MB/s	      0 B/op	     0 allocs/op
BenchmarkString_EastAsian/mattn/go-runewidth-8                23980 ns/op	   70.35 MB/s	      0 B/op	     0 allocs/op
BenchmarkString_EastAsian/rivo/uniseg-8                       19777 ns/op	   85.30 MB/s	      0 B/op	     0 allocs/op

BenchmarkString_ASCII/clipperhouse/displaywidth-8              1032 ns/op	  124.09 MB/s	      0 B/op	     0 allocs/op
BenchmarkString_ASCII/mattn/go-runewidth-8                     1162 ns/op	  110.16 MB/s	      0 B/op	     0 allocs/op
BenchmarkString_ASCII/rivo/uniseg-8                            1586 ns/op	   80.69 MB/s	      0 B/op	     0 allocs/op

BenchmarkString_Emoji/clipperhouse/displaywidth-8              3017 ns/op	  240.01 MB/s	      0 B/op	     0 allocs/op
BenchmarkString_Emoji/mattn/go-runewidth-8                     4745 ns/op	  152.58 MB/s	      0 B/op	     0 allocs/op
BenchmarkString_Emoji/rivo/uniseg-8                            6745 ns/op	  107.34 MB/s	      0 B/op	     0 allocs/op

BenchmarkRune_Mixed/clipperhouse/displaywidth-8                3381 ns/op	  498.90 MB/s	      0 B/op	     0 allocs/op
BenchmarkRune_Mixed/mattn/go-runewidth-8                       5383 ns/op	  313.41 MB/s	      0 B/op	     0 allocs/op

BenchmarkRune_EastAsian/clipperhouse/displaywidth-8            3395 ns/op	  496.96 MB/s	      0 B/op	     0 allocs/op
BenchmarkRune_EastAsian/mattn/go-runewidth-8                  15645 ns/op	  107.83 MB/s	      0 B/op	     0 allocs/op

BenchmarkRune_ASCII/clipperhouse/displaywidth-8                 257.8 ns/op	  496.57 MB/s	      0 B/op	     0 allocs/op
BenchmarkRune_ASCII/mattn/go-runewidth-8                        267.3 ns/op	  478.89 MB/s	      0 B/op	     0 allocs/op

BenchmarkRune_Emoji/clipperhouse/displaywidth-8                1338 ns/op	  541.24 MB/s	      0 B/op	     0 allocs/op
BenchmarkRune_Emoji/mattn/go-runewidth-8                       2287 ns/op	  316.58 MB/s	      0 B/op	     0 allocs/op

BenchmarkTruncateWithTail/clipperhouse/displaywidth-8          3689 ns/op	   47.98 MB/s	    192 B/op	    14 allocs/op
BenchmarkTruncateWithTail/mattn/go-runewidth-8                 8069 ns/op	   21.93 MB/s	    192 B/op	    14 allocs/op

BenchmarkTruncateWithoutTail/clipperhouse/displaywidth-8       3457 ns/op	   66.24 MB/s	      0 B/op	     0 allocs/op
BenchmarkTruncateWithoutTail/mattn/go-runewidth-8             10441 ns/op	   21.93 MB/s	      0 B/op	     0 allocs/op

Here are some notes on how to make Unicode things fast.

Documentation

Index

• Variables
• func Bytes(s []byte) int
• func Rune(r rune) int
• func String(s string) int
• func TruncateBytes(s []byte, maxWidth int, tail []byte) []byte
• func TruncateString(s string, maxWidth int, tail string) string
• type Graphemes
  • func BytesGraphemes(s []byte) Graphemes[[]byte]
  • func StringGraphemes(s string) Graphemes[string]
  • func (g *Graphemes[T]) Next() bool
  • func (g *Graphemes[T]) Value() T
  • func (g *Graphemes[T]) Width() int
• type Options
  • func (options Options) Bytes(s []byte) int
  • func (options Options) BytesGraphemes(s []byte) Graphemes[[]byte]
  • func (options Options) Rune(r rune) int
  • func (options Options) String(s string) int
  • func (options Options) StringGraphemes(s string) Graphemes[string]
  • func (options Options) TruncateBytes(s []byte, maxWidth int, tail []byte) []byte
  • func (options Options) TruncateString(s string, maxWidth int, tail string) string

Variables

var DefaultOptions = Options{EastAsianWidth: false}

DefaultOptions is the default options for the display width calculation, which is EastAsianWidth: false.

Functions

func Bytes

func Bytes(s []byte) int

Bytes calculates the display width of a []byte, by iterating over grapheme clusters in the byte slice and summing their widths.

func Rune

func Rune(r rune) int

Rune calculates the display width of a rune. You should almost certainly use String or Bytes for most purposes.

The smallest unit of display width is a grapheme cluster, not a rune. Iterating over runes to measure width is incorrect in many cases.

func String

func String(s string) int

String calculates the display width of a string, by iterating over grapheme clusters in the string and summing their widths.

func TruncateBytes

func TruncateBytes(s []byte, maxWidth int, tail []byte) []byte

TruncateBytes truncates a []byte to the given maxWidth, and appends the given tail if the []byte is truncated.

It ensures the total width, including the width of the tail, is less than or equal to maxWidth.

func TruncateString

func TruncateString(s string, maxWidth int, tail string) string

TruncateString truncates a string to the given maxWidth, and appends the given tail if the string is truncated.

It ensures the total width, including the width of the tail, is less than or equal to maxWidth.

Types

type Graphemes

type Graphemes[T stringish.Interface] struct {
	// contains filtered or unexported fields
}

Graphemes is an iterator over grapheme clusters.

Iterate using the Next method, and get the width of the current grapheme using the Width method.

func BytesGraphemes

func BytesGraphemes(s []byte) Graphemes[[]byte]

BytesGraphemes returns an iterator over grapheme clusters for the given []byte.

Iterate using the Next method, and get the width of the current grapheme using the Width method.

func StringGraphemes

func StringGraphemes(s string) Graphemes[string]

StringGraphemes returns an iterator over grapheme clusters for the given string.

Iterate using the Next method, and get the width of the current grapheme using the Width method.

func (*Graphemes[T]) Next

func (g *Graphemes[T]) Next() bool

Next advances the iterator to the next grapheme cluster.

func (*Graphemes[T]) Value

func (g *Graphemes[T]) Value() T

Value returns the current grapheme cluster.

func (*Graphemes[T]) Width

func (g *Graphemes[T]) Width() int

Width returns the display width of the current grapheme cluster.

type Options

type Options struct {
	EastAsianWidth bool
}

Options allows you to specify the treatment of ambiguous East Asian characters. When EastAsianWidth is false (default), ambiguous East Asian characters are treated as width 1. When EastAsianWidth is true, ambiguous East Asian characters are treated as width 2.

func (Options) Bytes

func (options Options) Bytes(s []byte) int

Bytes calculates the display width of a []byte, for the given options, by iterating over grapheme clusters in the slice and summing their widths.

func (Options) BytesGraphemes

func (options Options) BytesGraphemes(s []byte) Graphemes[[]byte]

BytesGraphemes returns an iterator over grapheme clusters for the given []byte, with the given options.

Iterate using the Next method, and get the width of the current grapheme using the Width method.

func (Options) Rune

func (options Options) Rune(r rune) int

Rune calculates the display width of a rune, for the given options.

You should almost certainly use String or Bytes for most purposes.

The smallest unit of display width is a grapheme cluster, not a rune. Iterating over runes to measure width is incorrect in many cases.

func (Options) String

func (options Options) String(s string) int

String calculates the display width of a string, for the given options, by iterating over grapheme clusters in the string and summing their widths.

func (Options) StringGraphemes

func (options Options) StringGraphemes(s string) Graphemes[string]

StringGraphemes returns an iterator over grapheme clusters for the given string, with the given options.

Iterate using the Next method, and get the width of the current grapheme using the Width method.

func (Options) TruncateBytes

func (options Options) TruncateBytes(s []byte, maxWidth int, tail []byte) []byte

TruncateBytes truncates a []byte to the given maxWidth, and appends the given tail if the []byte is truncated.

It ensures the total width, including the width of the tail, is less than or equal to maxWidth.

func (Options) TruncateString

func (options Options) TruncateString(s string, maxWidth int, tail string) string

TruncateString truncates a string to the given maxWidth, and appends the given tail if the string is truncated.

It ensures the total width, including the width of the tail, is less than or equal to maxWidth.