chroma

package module
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 2, 2018 License: MIT Imports: 11 Imported by: 0

README

Chroma - A general purpose syntax highlighter in pure Go Build Status Gitter chat

NOTE: As Chroma has just been released, its API is still in flux. That said, the high-level interface should not change significantly.

Chroma takes source code and other structured text and converts it into syntax highlighted HTML, ANSI-coloured text, etc.

Chroma is based heavily on Pygments, and includes translators for Pygments lexers and styles.

Table of Contents

Supported languages

ABNF, ANTLR, APL, ActionScript, ActionScript 3, Ada, Angular2, ApacheConf, AppleScript, Awk, BNF, Base Makefile, Bash, Batchfile, BlitzBasic, Brainfuck, C, C#, C++, CFEngine3, CMake, COBOL, CSS, Cap'n Proto, Ceylon, ChaiScript, Cheetah, Clojure, CoffeeScript, Common Lisp, Coq, Crystal, Cython, DTD, Dart, Diff, Django/Jinja, Docker, EBNF, Elixir, Elm, EmacsLisp, Erlang, FSharp, Factor, Fish, Forth, Fortran, GAS, GDScript, GLSL, Genshi, Genshi HTML, Genshi Text, Gnuplot, Go, Groovy, HTML, Handlebars, Haskell, Haxe, Hexdump, Hy, INI, Idris, Io, JSON, Java, JavaScript, Julia, Kotlin, LLVM, Lighttpd configuration file, Lua, Mako, Mason, Mathematica, MiniZinc, Modula-2, MySQL, Myghty, NASM, Newspeak, Nginx configuration file, Nim, OCaml, Octave, PHP, PL/pgSQL, POVRay, PacmanConf, Perl, Pig, PkgConfig, PostScript, PostgreSQL SQL dialect, PowerShell, Prolog, Protocol Buffer, Puppet, Python, Python 3, QBasic, R, Racket, Ragel, Rexx, Ruby, Rust, SPARQL, SQL, Sass, Scala, Scheme, Scilab, Smalltalk, Smarty, Snobol, SquidConf, SVG, Swift, TASM, Tcl, Tcsh, Termcap, Terminfo, Terraform, Thrift, Transact-SQL, Turtle, Twig, TypeScript, TypoScript, TypoScriptCssData, TypoScriptHtmlData, VHDL, VimL, XML, Xorg, YAML, cfstatement, markdown, reStructuredText, reg, systemverilog, verilog

I will attempt to keep this section up to date, but an authoritative list can be displayed with chroma --list.

Using the library

Chroma, like Pygments, has the concepts of lexers, formatters and styles.

Lexers convert source text into a stream of tokens, styles specify how token types are mapped to colours, and formatters convert tokens and styles into formatted output.

A package exists for each of these, containing a global Registry variable with all of the registered implementations. There are also helper functions for using the registry in each package, such as looking up lexers by name or matching filenames, etc.

In all cases, if a lexer, formatter or style can not be determined, nil will be returned. In this situation you may want to default to the Fallback value in each respective package, which provides sane defaults.

Quick start

A convenience function exists that can be used to simply format some source text, without any effort:

err := quick.Highlight(os.Stdout, someSourceCode, "go", "html", "monokai")
Identifying the language

To highlight code, you'll first have to identify what language the code is written in. There are three primary ways to do that:

  1. Detect the language from its filename.

    lexer := lexers.Match("foo.go")
    
  2. Explicitly specify the language by its Chroma syntax ID (a full list is available from lexers.Names()).

    lexer := lexers.Get("go")
    
  3. Detect the language from its content.

    lexer := lexers.Analyse("package main\n\nfunc main()\n{\n}\n")
    

In all cases, nil will be returned if the language can not be identified.

if lexer == nil {
  lexer = lexers.Fallback
}

At this point, it should be noted that some lexers can be extremely chatty. To mitigate this, you can use the coalescing lexer to coalesce runs of identical token types into a single token:

lexer = chroma.Coalesce(lexer)
Formatting the output

Once a language is identified you will need to pick a formatter and a style (theme).

style := styles.Get("swapoff")
if style == nil {
  style = styles.Fallback
}
formatter := formatters.Get("html")
if formatter == nil {
  formatter = formatters.Fallback
}

Then obtain an iterator over the tokens:

contents, err := ioutil.ReadAll(r)
iterator, err := lexer.Tokenise(nil, string(contents))

And finally, format the tokens from the iterator:

err := formatter.Format(w, style, iterator)
The HTML formatter

By default the html registered formatter generates standalone HTML with embedded CSS. More flexibility is available through the lexers/html package.

Firstly, the output generated by the formatter can be customised with the following constructor options:

  • Standalone() - generate standalone HTML with embedded CSS.
  • WithClasses() - use classes rather than inlined style attributes.
  • ClassPrefix(prefix) - prefix each generated CSS class.
  • TabWidth(width) - Set the rendered tab width, in characters.
  • WithLineNumbers() - Render line numbers (style with LineNumbers).
  • HighlightLines(ranges) - Highlight lines in these ranges (style with LineHighlight).

If WithClasses() is used, the corresponding CSS can be obtained from the formatter with:

formatter := html.New(html.WithClasses())
err := formatter.WriteCSS(w, style)

More detail

Lexers

See the Pygments documentation for details on implementing lexers. Most concepts apply directly to Chroma, but see existing lexer implementations for real examples.

In many cases lexers can be automatically converted directly from Pygments by using the included Python 3 script pygments2chroma.py. I use something like the following:

python3 ~/Projects/chroma/_tools/pygments2chroma.py \
  pygments.lexers.jvm.KotlinLexer \
  > ~/Projects/chroma/lexers/kotlin.go \
  && gofmt -s -w ~/Projects/chroma/lexers/*.go

See notes in pygments-lexers.go for a list of lexers, and notes on some of the issues importing them.

Formatters

Chroma supports HTML output, as well as terminal output in 8 colour, 256 colour, and true-colour.

A noop formatter is included that outputs the token text only, and a tokens formatter outputs raw tokens. The latter is useful for debugging lexers.

Styles

Chroma styles use the same syntax as Pygments.

All Pygments styles have been converted to Chroma using the _tools/style.py script.

Command-line interface

A command-line interface to Chroma is included. It can be installed with:

go get -u github.com/alecthomas/chroma/cmd/chroma

What's missing compared to Pygments?

  • Quite a few lexers, for various reasons (pull-requests welcome):
    • Pygments lexers for complex languages often include custom code to handle certain aspects, such as Perl6's ability to nest code inside regular expressions. These require time and effort to convert.
    • I mostly only converted languages I had heard of, to reduce the porting cost.
  • Some more esoteric features of Pygments are omitted for simplicity.
  • Though the Chroma API supports content detection, very few languages support them. I have plans to implement a statistical analyser at some point, but not enough time.

Documentation

Overview

Package chroma takes source code and other structured text and converts it into syntax highlighted HTML, ANSI- coloured text, etc.

Chroma is based heavily on Pygments, and includes translators for Pygments lexers and styles.

For more information, go here: https://github.com/alecthomas/chroma

Index

Constants

View Source
const (
	Whitespace = TextWhitespace

	Date = LiteralDate

	String          = LiteralString
	StringAffix     = LiteralStringAffix
	StringBacktick  = LiteralStringBacktick
	StringChar      = LiteralStringChar
	StringDelimiter = LiteralStringDelimiter
	StringDoc       = LiteralStringDoc
	StringDouble    = LiteralStringDouble
	StringEscape    = LiteralStringEscape
	StringHeredoc   = LiteralStringHeredoc
	StringInterpol  = LiteralStringInterpol
	StringOther     = LiteralStringOther
	StringRegex     = LiteralStringRegex
	StringSingle    = LiteralStringSingle
	StringSymbol    = LiteralStringSymbol

	Number            = LiteralNumber
	NumberBin         = LiteralNumberBin
	NumberFloat       = LiteralNumberFloat
	NumberHex         = LiteralNumberHex
	NumberInteger     = LiteralNumberInteger
	NumberIntegerLong = LiteralNumberIntegerLong
	NumberOct         = LiteralNumberOct
)

Aliases.

Variables

View Source
var ANSI2RGB = map[string]string{
	"#ansiblack":     "000000",
	"#ansidarkred":   "7f0000",
	"#ansidarkgreen": "007f00",
	"#ansibrown":     "7f7fe0",
	"#ansidarkblue":  "00007f",
	"#ansipurple":    "7f007f",
	"#ansiteal":      "007f7f",
	"#ansilightgray": "e5e5e5",

	"#ansidarkgray":  "555555",
	"#ansired":       "ff0000",
	"#ansigreen":     "00ff00",
	"#ansiyellow":    "ffff00",
	"#ansiblue":      "0000ff",
	"#ansifuchsia":   "ff00ff",
	"#ansiturquoise": "00ffff",
	"#ansiwhite":     "ffffff",

	"#black":     "000000",
	"#darkred":   "7f0000",
	"#darkgreen": "007f00",
	"#brown":     "7f7fe0",
	"#darkblue":  "00007f",
	"#purple":    "7f007f",
	"#teal":      "007f7f",
	"#lightgray": "e5e5e5",

	"#darkgray":  "555555",
	"#red":       "ff0000",
	"#green":     "00ff00",
	"#yellow":    "ffff00",
	"#blue":      "0000ff",
	"#fuchsia":   "ff00ff",
	"#turquoise": "00ffff",
	"#white":     "ffffff",
}

ANSI2RGB maps ANSI colour names, as supported by Chroma, to hex RGB values.

View Source
var (
	StandardTypes = map[TokenType]string{
		Background:       "chroma",
		LineNumbers:      "ln",
		LineNumbersTable: "lnt",
		LineHighlight:    "hl",
		LineTable:        "lntable",
		LineTableTD:      "lntd",
		Text:             "",
		Whitespace:       "w",
		Error:            "err",
		Other:            "x",

		Keyword:            "k",
		KeywordConstant:    "kc",
		KeywordDeclaration: "kd",
		KeywordNamespace:   "kn",
		KeywordPseudo:      "kp",
		KeywordReserved:    "kr",
		KeywordType:        "kt",

		Name:                 "n",
		NameAttribute:        "na",
		NameBuiltin:          "nb",
		NameBuiltinPseudo:    "bp",
		NameClass:            "nc",
		NameConstant:         "no",
		NameDecorator:        "nd",
		NameEntity:           "ni",
		NameException:        "ne",
		NameFunction:         "nf",
		NameFunctionMagic:    "fm",
		NameProperty:         "py",
		NameLabel:            "nl",
		NameNamespace:        "nn",
		NameOther:            "nx",
		NameTag:              "nt",
		NameVariable:         "nv",
		NameVariableClass:    "vc",
		NameVariableGlobal:   "vg",
		NameVariableInstance: "vi",
		NameVariableMagic:    "vm",

		Literal:     "l",
		LiteralDate: "ld",

		String:          "s",
		StringAffix:     "sa",
		StringBacktick:  "sb",
		StringChar:      "sc",
		StringDelimiter: "dl",
		StringDoc:       "sd",
		StringDouble:    "s2",
		StringEscape:    "se",
		StringHeredoc:   "sh",
		StringInterpol:  "si",
		StringOther:     "sx",
		StringRegex:     "sr",
		StringSingle:    "s1",
		StringSymbol:    "ss",

		Number:            "m",
		NumberBin:         "mb",
		NumberFloat:       "mf",
		NumberHex:         "mh",
		NumberInteger:     "mi",
		NumberIntegerLong: "il",
		NumberOct:         "mo",

		Operator:     "o",
		OperatorWord: "ow",

		Punctuation: "p",

		Comment:            "c",
		CommentHashbang:    "ch",
		CommentMultiline:   "cm",
		CommentPreproc:     "cp",
		CommentPreprocFile: "cpf",
		CommentSingle:      "c1",
		CommentSpecial:     "cs",

		Generic:           "g",
		GenericDeleted:    "gd",
		GenericEmph:       "ge",
		GenericError:      "gr",
		GenericHeading:    "gh",
		GenericInserted:   "gi",
		GenericOutput:     "go",
		GenericPrompt:     "gp",
		GenericStrong:     "gs",
		GenericSubheading: "gu",
		GenericTraceback:  "gt",
	}
)

Functions

func Words

func Words(prefix, suffix string, words ...string) string

Words creates a regex that matches any of the given literal words.

Types

type Analyser

type Analyser interface {
	AnalyseText(text string) float32
}

Analyser determines how appropriate this lexer is for the given text.

type Colour

type Colour int32

Colour represents an RGB colour.

func MustParseColour

func MustParseColour(colour string) Colour

MustParseColour is like ParseColour except it panics if the colour is invalid.

Will panic if colour is in an invalid format.

func NewColour

func NewColour(r, g, b uint8) Colour

NewColour creates a Colour directly from RGB values.

func ParseColour

func ParseColour(colour string) Colour

ParseColour in the forms #rgb, #rrggbb, #ansi<colour>, or #<colour>. Will return an "unset" colour if invalid.

func (Colour) Blue

func (c Colour) Blue() uint8

Blue component of colour.

func (Colour) Brighten

func (c Colour) Brighten(factor float64) Colour

Brighten returns a copy of this colour with its brightness adjusted.

If factor is negative, the colour is darkened.

Uses approach described here (http://www.pvladov.com/2012/09/make-color-lighter-or-darker.html).

func (Colour) Brightness

func (c Colour) Brightness() float64

Brightness of the colour (roughly) in the range 0.0 to 1.0

func (Colour) Distance

func (c Colour) Distance(e2 Colour) float64

Distance between this colour and another.

This uses the approach described here (https://www.compuphase.com/cmetric.htm). This is not as accurate as LAB, et. al. but is *vastly* simpler and sufficient for our needs.

func (Colour) GoString

func (c Colour) GoString() string

func (Colour) Green

func (c Colour) Green() uint8

Green component of colour.

func (Colour) IsSet

func (c Colour) IsSet() bool

func (Colour) Red

func (c Colour) Red() uint8

Red component of colour.

func (Colour) String

func (c Colour) String() string

type Colours

type Colours []Colour

Colours is an orderable set of colours.

func (Colours) Len

func (c Colours) Len() int

func (Colours) Less

func (c Colours) Less(i, j int) bool

func (Colours) Swap

func (c Colours) Swap(i, j int)

type CompiledRule

type CompiledRule struct {
	Rule
	Regexp *regexp2.Regexp
	// contains filtered or unexported fields
}

A CompiledRule is a Rule with a pre-compiled regex.

Note that regular expressions are lazily compiled on first use of the lexer.

type CompiledRules

type CompiledRules map[string][]*CompiledRule

type Config

type Config struct {
	// Name of the lexer.
	Name string

	// Shortcuts for the lexer
	Aliases []string

	// File name globs
	Filenames []string

	// Secondary file name globs
	AliasFilenames []string

	// MIME types
	MimeTypes []string

	// Regex matching is case-insensitive.
	CaseInsensitive bool

	// Regex matches all characters.
	DotAll bool

	// Regex does not match across lines ($ matches EOL).
	//
	// Defaults to multiline.
	NotMultiline bool

	// Make sure that the input ends with a newline. This
	// is required for some lexers that consume input linewise.
	EnsureNL bool

	// Priority of lexer.
	//
	// If this is 0 it will be treated as a default of 1.
	Priority float32
}

Config for a lexer.

type Emitter

type Emitter interface {
	// Emit tokens for the given regex groups.
	Emit(groups []string, lexer Lexer) Iterator
}

An Emitter takes group matches and returns tokens.

func ByGroups

func ByGroups(emitters ...Emitter) Emitter

ByGroups emits a token for each matching group in the rule's regex.

func Using

func Using(lexer Lexer) Emitter

Using returns an Emitter that uses a given Lexer for parsing and emitting.

func UsingSelf

func UsingSelf(state string) Emitter

UsingSelf is like Using, but uses the current Lexer.

type EmitterFunc

type EmitterFunc func(groups []string, lexer Lexer) Iterator

EmitterFunc is a function that is an Emitter.

func (EmitterFunc) Emit

func (e EmitterFunc) Emit(groups []string, lexer Lexer) Iterator

Emit tokens for groups.

type Formatter

type Formatter interface {
	// Format returns a formatting function for tokens.
	//
	// If the iterator panics, the Formatter should recover.
	Format(w io.Writer, style *Style, iterator Iterator) error
}

A Formatter for Chroma lexers.

func RecoveringFormatter

func RecoveringFormatter(formatter Formatter) Formatter

RecoveringFormatter wraps a formatter with panic recovery.

type FormatterFunc

type FormatterFunc func(w io.Writer, style *Style, iterator Iterator) error

A FormatterFunc is a Formatter implemented as a function.

Guards against iterator panics.

func (FormatterFunc) Format

func (f FormatterFunc) Format(w io.Writer, s *Style, it Iterator) (err error)

type Iterator

type Iterator func() *Token

An Iterator across tokens.

nil will be returned at the end of the Token stream.

If an error occurs within an Iterator, it may propagate this in a panic. Formatters should recover.

func Concaterator

func Concaterator(iterators ...Iterator) Iterator

Concaterator concatenates tokens from a series of iterators.

func Literator

func Literator(tokens ...*Token) Iterator

Literator converts a sequence of literal Tokens into an Iterator.

func (Iterator) Tokens

func (i Iterator) Tokens() []*Token

Tokens consumes all tokens from the iterator and returns them as a slice.

type Lexer

type Lexer interface {
	// Config describing the features of the Lexer.
	Config() *Config
	// Tokenise returns an Iterator over tokens in text.
	Tokenise(options *TokeniseOptions, text string) (Iterator, error)
}

A Lexer for tokenising source code.

func Coalesce

func Coalesce(lexer Lexer) Lexer

Coalesce is a Lexer interceptor that collapses runs of common types into a single token.

func RemappingLexer added in v0.2.0

func RemappingLexer(lexer Lexer, mapper func(*Token) []*Token) Lexer

RemappingLexer remaps a token to a set of, potentially empty, tokens.

func TypeRemappingLexer added in v0.2.0

func TypeRemappingLexer(lexer Lexer, mapping TypeMapping) Lexer

TypeRemappingLexer remaps types of tokens coming from a parent Lexer.

eg. Map "defvaralias" tokens of type NameVariable to NameFunction:

mapping := TypeMapping{
	{NameVariable, NameFunction, []string{"defvaralias"},
}
lexer = TypeRemappingLexer(lexer, mapping)

type LexerMutator

type LexerMutator interface {
	// Rules are the lexer rules, state is the state key for the rule the mutator is associated with.
	MutateLexer(rules CompiledRules, state string, rule int) error
}

A LexerMutator is an additional interface that a Mutator can implement to modify the lexer when it is compiled.

type LexerState

type LexerState struct {
	Lexer *RegexLexer
	Text  []rune
	Pos   int
	Rules CompiledRules
	Stack []string
	State string
	Rule  int
	// Group matches.
	Groups []string
	// Custum context for mutators.
	MutatorContext map[interface{}]interface{}
	// contains filtered or unexported fields
}

func (*LexerState) Get

func (l *LexerState) Get(key interface{}) interface{}

func (*LexerState) Iterator

func (l *LexerState) Iterator() *Token

func (*LexerState) Set

func (l *LexerState) Set(key interface{}, value interface{})

type Lexers

type Lexers []Lexer

Lexers is a slice of lexers sortable by name.

func (Lexers) Len

func (l Lexers) Len() int

func (Lexers) Less

func (l Lexers) Less(i, j int) bool

func (Lexers) Swap

func (l Lexers) Swap(i, j int)

type Mutator

type Mutator interface {
	// Mutate the lexer state machine as it is processing.
	Mutate(state *LexerState) error
}

A Mutator modifies the behaviour of the lexer.

func Combined

func Combined(states ...string) Mutator

Combined creates a new anonymous state from the given states, and pushes that state.

type MutatorFunc

type MutatorFunc func(state *LexerState) error

A MutatorFunc is a Mutator that mutates the lexer state machine as it is processing.

func Mutators

func Mutators(modifiers ...Mutator) MutatorFunc

Mutators applies a set of Mutators in order.

func Pop

func Pop(n int) MutatorFunc

Pop state from the stack when rule matches.

func Push

func Push(states ...string) MutatorFunc

Push states onto the stack.

func (MutatorFunc) Mutate

func (m MutatorFunc) Mutate(state *LexerState) error

type PrioritisedLexers added in v0.2.0

type PrioritisedLexers []Lexer

PrioritisedLexers is a slice of lexers sortable by priority.

func (PrioritisedLexers) Len added in v0.2.0

func (l PrioritisedLexers) Len() int

func (PrioritisedLexers) Less added in v0.2.0

func (l PrioritisedLexers) Less(i, j int) bool

func (PrioritisedLexers) Swap added in v0.2.0

func (l PrioritisedLexers) Swap(i, j int)

type RegexLexer

type RegexLexer struct {
	// contains filtered or unexported fields
}

func MustNewLexer

func MustNewLexer(config *Config, rules Rules) *RegexLexer

MustNewLexer creates a new Lexer or panics.

func NewLexer

func NewLexer(config *Config, rules Rules) (*RegexLexer, error)

NewLexer creates a new regex-based Lexer.

"rules" is a state machine transitition map. Each key is a state. Values are sets of rules that match input, optionally modify lexer state, and output tokens.

func (*RegexLexer) AnalyseText

func (r *RegexLexer) AnalyseText(text string) float32

func (*RegexLexer) Config

func (r *RegexLexer) Config() *Config

func (*RegexLexer) SetAnalyser

func (r *RegexLexer) SetAnalyser(analyser func(text string) float32) *RegexLexer

SetAnalyser sets the analyser function used to perform content inspection.

func (*RegexLexer) Tokenise

func (r *RegexLexer) Tokenise(options *TokeniseOptions, text string) (Iterator, error)

func (*RegexLexer) Trace

func (r *RegexLexer) Trace(trace bool) *RegexLexer

type Rule

type Rule struct {
	Pattern string
	Type    Emitter
	Mutator Mutator
}

func Default

func Default(mutators ...Mutator) Rule

Default returns a Rule that applies a set of Mutators.

func Include

func Include(state string) Rule

Include the given state.

type Rules

type Rules map[string][]Rule

Rules maps from state to a sequence of Rules.

func (Rules) Clone added in v0.3.0

func (r Rules) Clone() Rules

type Style

type Style struct {
	Name string
	// contains filtered or unexported fields
}

A Style definition.

See http://pygments.org/docs/styles/ for details. Semantics are intended to be identical.

func MustNewStyle

func MustNewStyle(name string, entries StyleEntries) *Style

MustNewStyle creates a new style or panics.

func NewStyle

func NewStyle(name string, entries StyleEntries) (*Style, error)

NewStyle creates a new style definition.

func (*Style) Builder

func (s *Style) Builder() *StyleBuilder

Builder creates a mutable builder from this Style.

The builder can then be safely modified. This is a cheap operation.

func (*Style) Get

func (s *Style) Get(ttype TokenType) StyleEntry

Get a style entry. Will try sub-category or category if an exact match is not found, and finally return the Background.

func (*Style) Has

func (s *Style) Has(ttype TokenType) bool

Has checks if an exact style entry match exists for a token type.

This is distinct from Get() which will merge parent tokens.

func (*Style) Types

func (s *Style) Types() []TokenType

Types that are styled.

type StyleBuilder

type StyleBuilder struct {
	// contains filtered or unexported fields
}

A StyleBuilder is a mutable structure for building styles.

Once built, a Style is immutable.

func NewStyleBuilder

func NewStyleBuilder(name string) *StyleBuilder

func (*StyleBuilder) Add

func (s *StyleBuilder) Add(ttype TokenType, entry string) *StyleBuilder

Add an entry to the Style map.

See http://pygments.org/docs/styles/#style-rules for details.

func (*StyleBuilder) AddAll

func (s *StyleBuilder) AddAll(entries StyleEntries) *StyleBuilder

func (*StyleBuilder) AddEntry

func (s *StyleBuilder) AddEntry(ttype TokenType, entry StyleEntry) *StyleBuilder

func (*StyleBuilder) Build

func (s *StyleBuilder) Build() (*Style, error)

func (*StyleBuilder) Get

func (s *StyleBuilder) Get(ttype TokenType) StyleEntry

type StyleEntries

type StyleEntries map[TokenType]string

StyleEntries mapping TokenType to colour definition.

type StyleEntry

type StyleEntry struct {
	// Hex colours.
	Colour     Colour
	Background Colour
	Border     Colour

	Bold      Trilean
	Italic    Trilean
	Underline Trilean
	NoInherit bool
}

A StyleEntry in the Style map.

func ParseStyleEntry

func ParseStyleEntry(entry string) (StyleEntry, error)

ParseStyleEntry parses a Pygments style entry.

func (StyleEntry) Inherit

func (s StyleEntry) Inherit(ancestors ...StyleEntry) StyleEntry

Inherit styles from ancestors.

Ancestors should be provided from oldest to newest.

func (StyleEntry) IsZero

func (s StyleEntry) IsZero() bool

func (StyleEntry) String

func (s StyleEntry) String() string

func (StyleEntry) Sub

func (s StyleEntry) Sub(e StyleEntry) StyleEntry

type Token

type Token struct {
	Type  TokenType `json:"type"`
	Value string    `json:"value"`
}

Token output to formatter.

func Tokenise

func Tokenise(lexer Lexer, options *TokeniseOptions, text string) ([]*Token, error)

Tokenise text using lexer, returning tokens as a slice.

func (*Token) Clone

func (t *Token) Clone() *Token

func (*Token) GoString

func (t *Token) GoString() string

func (*Token) String

func (t *Token) String() string

type TokenType

type TokenType int

TokenType is the type of token to highlight.

It is also an Emitter, emitting a single token of itself

const (
	// Default background style.
	Background TokenType = -1 - iota
	// Line numbers in output.
	LineNumbers
	// Line numbers in output when in table.
	LineNumbersTable
	// Line higlight style.
	LineHighlight
	// Line numbers table wrapper style.
	LineTable
	// Line numbers table TD wrapper style.
	LineTableTD
	// Input that could not be tokenised.
	Error
	// Other is used by the Delegate lexer to indicate which tokens should be handled by the delegate.
	Other
	// No highlighting.
	None
)

Meta token types.

const (
	Keyword TokenType = 1000 + iota
	KeywordConstant
	KeywordDeclaration
	KeywordNamespace
	KeywordPseudo
	KeywordReserved
	KeywordType
)

Keywords.

const (
	Name TokenType = 2000 + iota
	NameAttribute
	NameBuiltin
	NameBuiltinPseudo
	NameClass
	NameConstant
	NameDecorator
	NameEntity
	NameException
	NameFunction
	NameFunctionMagic
	NameKeyword
	NameLabel
	NameNamespace
	NameOperator
	NameOther
	NamePseudo
	NameProperty
	NameTag
	NameVariable
	NameVariableAnonymous
	NameVariableClass
	NameVariableGlobal
	NameVariableInstance
	NameVariableMagic
)

Names.

const (
	Literal TokenType = 3000 + iota
	LiteralDate
	LiteralOther
)

Literals.

const (
	LiteralString TokenType = 3100 + iota
	LiteralStringAffix
	LiteralStringAtom
	LiteralStringBacktick
	LiteralStringBoolean
	LiteralStringChar
	LiteralStringDelimiter
	LiteralStringDoc
	LiteralStringDouble
	LiteralStringEscape
	LiteralStringHeredoc
	LiteralStringInterpol
	LiteralStringName
	LiteralStringOther
	LiteralStringRegex
	LiteralStringSingle
	LiteralStringSymbol
)

Strings.

const (
	LiteralNumber TokenType = 3200 + iota
	LiteralNumberBin
	LiteralNumberFloat
	LiteralNumberHex
	LiteralNumberInteger
	LiteralNumberIntegerLong
	LiteralNumberOct
)

Literals.

const (
	Operator TokenType = 4000 + iota
	OperatorWord
)

Operators.

const (
	Comment TokenType = 6000 + iota
	CommentHashbang
	CommentMultiline
	CommentSingle
	CommentSpecial
)

Comments.

const (
	CommentPreproc TokenType = 6100 + iota
	CommentPreprocFile
)

Preprocessor "comments".

const (
	Generic TokenType = 7000 + iota
	GenericDeleted
	GenericEmph
	GenericError
	GenericHeading
	GenericInserted
	GenericOutput
	GenericPrompt
	GenericStrong
	GenericSubheading
	GenericTraceback
	GenericUnderline
)

Generic tokens.

const (
	Text TokenType = 8000 + iota
	TextWhitespace
	TextSymbol
	TextPunctuation
)

Text.

const (
	Punctuation TokenType = 5000 + iota
)

Punctuation.

func (TokenType) Category

func (t TokenType) Category() TokenType

func (TokenType) Emit

func (t TokenType) Emit(groups []string, lexer Lexer) Iterator

func (TokenType) InCategory

func (t TokenType) InCategory(other TokenType) bool

func (TokenType) InSubCategory

func (t TokenType) InSubCategory(other TokenType) bool

func (*TokenType) MarshalJSON added in v0.2.1

func (t *TokenType) MarshalJSON() ([]byte, error)

func (TokenType) Parent added in v0.1.1

func (t TokenType) Parent() TokenType

func (TokenType) String

func (i TokenType) String() string

func (TokenType) SubCategory

func (t TokenType) SubCategory() TokenType

func (*TokenType) UnmarshalJSON added in v0.2.1

func (t *TokenType) UnmarshalJSON(data []byte) error

type TokeniseOptions

type TokeniseOptions struct {
	// State to start tokenisation in. Defaults to "root".
	State string
	// Nested tokenisation.
	Nested bool
}

type Trilean

type Trilean uint8

Trilean value for StyleEntry value inheritance.

const (
	Pass Trilean = iota
	Yes
	No
)

func (Trilean) Prefix

func (t Trilean) Prefix(s string) string

func (Trilean) String

func (t Trilean) String() string

type TypeMapping added in v0.2.0

type TypeMapping []struct {
	From, To TokenType
	Words    []string
}

Directories

Path Synopsis
_tools
cmd
Package lexers contains the registry of all lexers.
Package lexers contains the registry of all lexers.
a
b
c
d
e
f
g
h
i
j
k
l
m
n
o
p
q
r
s
t
v
w
x
y
Package quick provides simple, no-configuration source code highlighting.
Package quick provides simple, no-configuration source code highlighting.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL