twemoji

README

twemoji

Twemoji emoji graphics as embedded SVG, and the :shortcode: table of markdown-it-emoji, for pure-Go programs (CGO_ENABLED=0, no network).

import "github.com/go-icons/twemoji"

svg, ok := twemoji.SVG("👩‍💻")          // the SVG document, however FE0F is placed
seq, _ := twemoji.Shortcode("smile")  // "😄"
n := twemoji.Cut("😄 bonjour")        // 4: the emoji at the start of the text, in bytes
start, end := twemoji.Find(text, 0)   // the next emoji in text, or -1
stem := twemoji.Stem("🇫🇷")           // "1f1eb-1f1f7", the file name Twemoji gives it

It is a data package: it draws nothing. A renderer (go-gfx's SVG rasteriser, a browser, <img src="data:image/svg+xml;base64,…">) turns the SVG into pixels.

Graphics	4,009 SVG, Twemoji 17.0.3 (Emoji 17.0)
Shortcodes	1,903, markdown-it-emoji 3.0.0
Embedded size	2.1 MB for 10.1 MB of SVG (deflate, in 48 KB blocks of neighbouring emoji)
Start-up cost	none: the archive is parsed, and the lookup tables built, on first use
Lookup	SVG ≈ 38 ns (one 48 KB block inflated on a miss, 8 recently used kept)
Text scan	Find ≈ 300 MB/s on prose, no allocation

Attribution

The graphics are CC-BY 4.0: whoever shows them must credit them. Use twemoji.Attribution:

Twemoji, Copyright 2020 Twitter, Inc and other contributors (CC-BY 4.0), continued by the jdecked/twemoji project

The code of this module is BSD-3-Clause (LICENSE). The third-party files keep their licences: TWEMOJI-LICENSE-GRAPHICS (CC-BY 4.0, the graphics), TWEMOJI-LICENSE (MIT, Twemoji's code), MARKDOWN-IT-EMOJI-LICENSE (MIT, the shortcode table, itself derived from gemoji).

Names

A graphic's stem is its file name in Twemoji: the code points in lower-case hexadecimal joined with -, with U+FE0F dropped unless the sequence has a ZWJ: 😄 is 1f604, 🇫🇷 is 1f1eb-1f1f7, 👩‍💻 is 1f469-200d-1f4bb, 1️⃣ is 31-20e3. Stem computes it exactly as twemoji.js does; Lookup finds the stem the set actually holds, ignoring U+FE0F.

Reading text the way Marp Core does

Cut and Find find emoji in text with the rules of Marp Core (emoji-regex, then Twemoji's parser). They were compared with Marp Core 4.4.0 on all 6,380 spellings of the set (each sequence as Twemoji names it, and with U+FE0F after its first character):

• the longest sequence the set has a graphic for;
• a character whose default presentation is text (© ™ ↔ ☺ ✔ ▶, 207 of them: TextDefault) is an emoji only with U+FE0F after it: ✔ stays a glyph, ✔️ becomes an image. The exception is ❤, which emoji-regex takes bare;
• a match followed by U+FE0E (text selector) is left alone;
• a lone regional indicator is not a flag; a bare digit, # or * is not a keycap.

Where it differs from Marp Core, it is to convert more: ☝️ ✌️ ✍️ ⛹️ 🏋️ 🏌️ 🕴️ 🕵️ 🖐️ ⛷️ (Marp Core leaves them as text), and the few sequences Twemoji draws that Unicode does not list (👨‍🎄). Marp Core also keeps a U+FE0F after about thirty BMP emoji (⚽ ☕ ♈) inside the match; here it stays in the text, where it is invisible.

Regenerating

git clone --depth 1 --branch v17.0.3 https://github.com/jdecked/twemoji
git clone --depth 1 --branch 3.0.0 https://github.com/markdown-it/markdown-it-emoji
# emoji-sequences.txt: unicode-org/unicodetools, unicodetools/data/emoji/17.0/
go run ./cmd/gen --twemoji ./twemoji --markdown-it-emoji ./markdown-it-emoji \
    --emoji-sequences emoji-sequences.txt

The archive format (internal/pack) sorts the documents by key and deflates blocks of about 48 KB: skin-tone and gender variants of one emoji are neighbours and share most of their markup, so a block compresses far better than the files one by one. The reader checks every length and offset before using it, bounds a block at 4 MB, and refuses an archive that disagrees with itself.

Documentation

Overview

Package twemoji serves the Twemoji emoji graphics (jdecked/twemoji, the community continuation of Twitter's set; graphics CC-BY 4.0 — see TWEMOJI-LICENSE-GRAPHICS) as SVG documents keyed by emoji sequence, with the :shortcode: table of markdown-it-emoji (MIT — see MARKDOWN-IT-EMOJI-LICENSE).

It is a DATA package: it embeds the SVGs, compressed, and hands one back; it draws nothing. The archive is inflated lazily, a block at a time, and the lookup tables are built on first use, so importing the package costs a few kilobytes of address space and no start-up time.

Attribution

The graphics are licensed CC-BY 4.0: whoever shows them must credit "Twemoji, Copyright 2020 Twitter, Inc and other contributors (CC-BY 4.0), continued by the jdecked/twemoji project". Attribution is the one condition this package cannot meet for its users.

Names

An emoji's STEM is the file name Twemoji gives its graphic: the code points in lower-case hexadecimal joined with "-", with U+FE0F (the emoji presentation selector) dropped unless the sequence has a ZWJ: 😄 is "1f604", 🇫🇷 is "1f1eb-1f1f7", 👩‍💻 is "1f469-200d-1f4bb", 1️⃣ is "31-20e3". Lookups here ignore U+FE0F altogether, so a sequence finds its graphic however the text qualified it.

Index

• Constants
• func Cut(s string) int
• func Err() error
• func Find(s string, from int) (start, end int)
• func Len() int
• func Lookup(seq string) string
• func Replace(s string, f func(name, seq string) string) string
• func SVG(seq string) (string, bool)
• func SVGByStem(stem string) (string, bool)
• func Shortcode(name string) (string, bool)
• func Shortcodes() []string
• func Stem(seq string) string
• func Stems() []string
• func TextDefault(r rune) bool

Constants

const (
	Version           = "17.0.3"
	ShortcodesVersion = "3.0.0"
)

Version is the Twemoji release the embedded graphics come from, and ShortcodesVersion the markdown-it-emoji release of the :shortcode: table.

const Attribution = "Twemoji, Copyright 2020 Twitter, Inc and other contributors (CC-BY 4.0), continued by the jdecked/twemoji project"

Attribution is the credit CC-BY 4.0 asks for.

Functions

func Cut

func Cut(s string) int

Cut returns the byte length of the emoji at the start of s, or 0 when s does not start with one. It reads text the way Marp Core does (the emoji-regex it matches with, then Twemoji's parser):

• the match is the longest sequence the set has a graphic for, with or without U+FE0F inside it;
• a character whose default presentation is text (©, ™, ☺, ✔, ▶: see TextDefault) is an emoji only when U+FE0F follows it — except ❤ (U+2764), which is an emoji bare;
• a U+FE0F after a character that is already an emoji by default (😄) is not part of the match; after a text-default one it is (Marp Core also takes it after about thirty BMP emoji such as ⚽ ☕ ♈; the difference is an invisible U+FE0F left in the text or not);
• a match followed by U+FE0E (text presentation selector) is not one;
• a lone regional indicator is not a flag, a bare digit, # or * is not a keycap (they are with U+20E3), and nothing in the private use area is an emoji.

func Err

func Err() error

Err reports why the embedded data could not be read: nil unless the package was built from a damaged archive, which its tests rule out.

func Find

func Find(s string, from int) (start, end int)

Find returns the byte range of the first emoji of s at or after from, or start == -1.

func Len

func Len() int

Len is how many graphics the set holds.

func Lookup

func Lookup(seq string) string

Lookup returns the stem of the embedded graphic for seq, ignoring U+FE0F, or "" when the set has none.

func Replace

func Replace(s string, f func(name, seq string) string) string

Replace is the :shortcode: pass of markdown-it-emoji over text: each :name: that names a shortcode becomes f(name, sequence). Matches do not overlap; ":a:smile:" yields the smile, since ":a:" names nothing. Text without a colon is returned as is.

func SVG

func SVG(seq string) (string, bool)

SVG returns the SVG document of the graphic for seq (a sequence in any qualification: with or without U+FE0F), and whether the set has one.

func SVGByStem

func SVGByStem(stem string) (string, bool)

SVGByStem is SVG for a stem as Stem or Lookup return it.

func Shortcode

func Shortcode(name string) (string, bool)

Shortcode returns the sequence of the :name: shortcode (name without the colons: "smile", "+1", "100"), and whether it is defined.

func Shortcodes

func Shortcodes() []string

Shortcodes lists every shortcode name, sorted.

func Stem

func Stem(seq string) string

Stem is the name Twemoji gives the graphic of seq, computed the way twemoji.js does: lower-case hexadecimal code points joined with "-", U+FE0F dropped unless seq contains a ZWJ. It does not look at the embedded set; see Lookup.

func Stems

func Stems() []string

Stems lists the stem of every graphic, sorted.

func TextDefault

func TextDefault(r rune) bool

TextDefault reports whether r is an emoji whose default presentation is text: a plain glyph unless U+FE0F follows it (© ™ ↔ ☺ ✔ ▶ and about 200 more, from Unicode's emoji-sequences.txt).