token

Documentation

Index

• Variables
• type ID
  • func (i ID) IDNumber() ID
  • func (i ID) String() string
• type Item
  • func (i Item) MarshalJSON() ([]byte, error)
  • func (i *Item) String() string
• type Lexer
  • func Lex(name string, input []byte, logConf log.Config) (l *Lexer, err error)
  • func (l *Lexer) NextItem() *Item
• type Type
  • func (t Type) String() string
  • func (t *Type) UnmarshalJSON(data []byte) error

Variables

var EOL rune = utf8.RuneError

EOL is denoted by a utf8.RuneError

Types

type ID

type ID int

ID is a consecutive number for identication of a lexed item and parsed item. Primarily for the purpose of debugging lexer and parser output when compared to the JSON encoded tests.

func (ID) IDNumber

func (i ID) IDNumber() ID

IDNumber returns the ID from an item.

func (ID) String

func (i ID) String() string

String implements Stringer and returns ID as a string.

type Item

type Item struct {
	ID            `json:"id"`
	Type          Type   `json:"type"`
	Text          string `json:"text,omitempty"`
	Line          int    `json:"line"`
	StartPosition int    `json:"startPosition"`
	Length        int    `json:"length,omitempty"`
}

Struct for tokens emitted by the scanning process

func (Item) MarshalJSON

func (i Item) MarshalJSON() ([]byte, error)

MarshalJSON satisfies the Marshaler interface.

func (*Item) String

func (i *Item) String() string

String satisfies the Stringer interface.

type Lexer

type Lexer struct {
	Name string // The name of the current lexer

	log.Logger
	// contains filtered or unexported fields
}

The Lexer struct tracks the state of the Lexer

func Lex

func Lex(name string, input []byte, logConf log.Config) (l *Lexer, err error)

lex is the entry point of the lexer. Name should be any name that signifies the purporse of the lexer. It is mostly used to identify the lexing process in debugging.

func (*Lexer) NextItem

func (l *Lexer) NextItem() *Item

nextItem returns the next item from the input.

type Type

type Type int

Type are the types that are emitted by the lexer.

const (
	EOF Type = iota
	Error
	Title
	SectionAdornment
	Text
	BlockQuote
	LiteralBlock
	SystemMessage
	Space
	BlankLine
	Transition
	CommentMark
	EnumListAffix
	EnumListArabic
	HyperlinkTargetStart
	HyperlinkTargetPrefix
	HyperlinkTargetQuote
	HyperlinkTargetName
	HyperlinkTargetSuffix
	HyperlinkTargetURI
	InlineStrongOpen
	InlineStrong
	InlineStrongClose
	InlineEmphasisOpen
	InlineEmphasis
	InlineEmphasisClose
	InlineLiteralOpen
	InlineLiteral
	InlineLiteralClose
	InlineInterpretedTextOpen
	InlineInterpretedText
	InlineInterpretedTextClose
	InlineInterpretedTextRoleOpen
	InlineInterpretedTextRole
	InlineInterpretedTextRoleClose
	InlineReferenceOpen
	InlineReferenceText
	InlineReferenceClose
	DefinitionTerm
	DefinitionText
	Bullet
	Escape
)

func (Type) String

func (t Type) String() string

String implements the Stringer interface for printing Type types.

func (*Type) UnmarshalJSON

func (t *Type) UnmarshalJSON(data []byte) error