tokenizer

Documentation

Index

• type Tokenizer
  • func NewDefaultTokenizer() *Tokenizer
  • func NewTokenizer(config model.EDIFACTConfig) *Tokenizer
  • func (t *Tokenizer) ExtractSegmentTag(segmentTokens []model.Token) string
  • func (t *Tokenizer) GroupTokensIntoSegments(tokens []model.Token) ([][]model.Token, error)
  • func (t *Tokenizer) IsSegmentStart(token model.Token) bool
  • func (t *Tokenizer) Tokenize(input string) ([]model.Token, error)

Types

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer handles the tokenization of EDIFACT strings

func NewDefaultTokenizer

func NewDefaultTokenizer() *Tokenizer

NewDefaultTokenizer creates a new tokenizer with default EDIFACT configuration

func NewTokenizer

func NewTokenizer(config model.EDIFACTConfig) *Tokenizer

NewTokenizer creates a new tokenizer with the given configuration

func (*Tokenizer) ExtractSegmentTag

func (t *Tokenizer) ExtractSegmentTag(segmentTokens []model.Token) string

ExtractSegmentTag extracts the segment tag from the first element of a segment

func (*Tokenizer) GroupTokensIntoSegments

func (t *Tokenizer) GroupTokensIntoSegments(tokens []model.Token) ([][]model.Token, error)

GroupTokensIntoSegments groups tokens into segments

func (*Tokenizer) IsSegmentStart

func (t *Tokenizer) IsSegmentStart(token model.Token) bool

IsSegmentStart checks if the given token represents the start of a new segment

func (*Tokenizer) Tokenize

func (t *Tokenizer) Tokenize(input string) ([]model.Token, error)

Tokenize converts an EDIFACT string into a slice of tokens