Documentation

Overview

    Package halfpike provides a lexer/parser framework library that can simplify lexing and parsing by using a very limited subset of the regexp syntax. This prevents many of the common errors encountered when trying to parse output from devices where the complete language syntax is unknown and can change between releases. Routers and other devices with human readable output or badly mangled formats within a standard (such as XML or JSON).

    Called halfpike, because this solution is a mixture of Rob Pike's lexer talk and the use of regex's within a single line of output to do captures in order to store a value within a struct type.

    A similar method replaced complex regex captures at a large search company's network group to prevent accidental empty matches and other bad behavior from regexes that led to issues in automation stacks. It allowed precise diagnosis of problems and readable code (complex regexes are not easily readable).

    Example (Long)
    Output:
    
    [{PeerIP:     10.10.10.2,
      PeerPort:   179,
      PeerAS:     22,
      LocalIP:    10.10.10.1,
      LocalPort:  65406,
      LocalAS:    22,
      Type:       1,
      State:      3,
      LastState:  5,
      HoldTime:   90000000000,
      Preference: 170,
      PeerID:     10.10.10.2,
      LocalID:    10.10.10.1,
      InetStats:  {0: {ID:                 0,
                       Bit:                10000,
                       RIBState:           2,
                       SendState:          1,
                       ActivePrefixes:     0,
                       RecvPrefixes:       0,
                       AcceptPrefixes:     0,
                       SurpressedPrefixes: 2,
                       AdvertisedPrefixes: 0}}},
     {PeerIP:     10.10.10.6,
      PeerPort:   54781,
      PeerAS:     22,
      LocalIP:    10.10.10.5,
      LocalPort:  179,
      LocalAS:    22,
      Type:       1,
      State:      3,
      LastState:  5,
      HoldTime:   90000000000,
      Preference: 170,
      PeerID:     10.10.10.6,
      LocalID:    10.10.10.1,
      InetStats:  {0: {ID:                 0,
                       Bit:                10000,
                       RIBState:           2,
                       SendState:          1,
                       ActivePrefixes:     0,
                       RecvPrefixes:       0,
                       AcceptPrefixes:     0,
                       SurpressedPrefixes: 0,
                       AdvertisedPrefixes: 0}}}]
    
    Example (Short)
    Output:
    
    [{VendorDesc: "ge-3/0/2",
      Blade:      3,
      Pic:        0,
      Port:       2,
      State:      1,
      Status:     1,
      LinkLevel:  1,
      MTU:        1522,
      Speed:      1000000000},
     {VendorDesc: "ge-3/0/3",
      Blade:      3,
      Pic:        0,
      Port:       3,
      State:      1,
      Status:     1,
      LinkLevel:  2,
      MTU:        1522,
      Speed:      1000000000}]
    

    Index

    Examples

    Constants

    View Source
    const Skip = "$.<skip>.$"

      Any provides a special string for FindStart that will skip an item.

      Variables

      This section is empty.

      Functions

      func ItemJoin

      func ItemJoin(line Line, start, end int) string

        ItemJoin takes a line, the inclusive beginning index and the non-inclusive ending index and joins all the values with a single space between them. -1 for start or end means from the absolute begin or end of the line slice. This will automatically remove the carriage return or EOF items.

        func Match

        func Match(re *regexp.Regexp, s string) (map[string]string, error)

          Match returns matches of the regex with keys set to the submatch names. If these are not named submatches (aka `(?P<name>regex)`) this will probably panic. A match that is empty string will cause an error to return.

          func Parse

          func Parse(ctx context.Context, p *Parser, start ParseFn) error

            Parse begins parsing content from the underlying input to Parser. Parsing starts with the "start" ParseFn until a returned ParseFn == nil. Calling p.HasError() will return an error if there was one. If err == nil, the Validator object passed to Parser should have .Validate() called to ensure all data is correct.

            Types

            type Item

            type Item struct {
            	// Type is the type of item that is stored in .Val.
            	Type ItemType
            	// Val is the value of the item that was in the text output.
            	Val string
            	// contains filtered or unexported fields
            }

              Item represents a token created by the Lexer.

              func (Item) IsZero

              func (i Item) IsZero() bool

                IsZero indicates the Item is the zero value.

                func (Item) ToFloat

                func (i Item) ToFloat() (float64, error)

                  ToFloat returns the value as a float64 type. if the Item.Type is not itemFloat, this will panic.

                  func (Item) ToInt

                  func (i Item) ToInt() (int, error)

                    ToInt returns the value as an int type. If the Item.Type is not ItemInt, this will panic.

                    type ItemType

                    type ItemType int

                      ItemType describes the type of item being emitted by the Lexer. There are predefined ItemType(s) and the rest are defined by the user.

                      const (
                      	// ItemUnknown indicates that the Item is an unknown. This should only happen on
                      	// a Item that is the zero type.
                      	ItemUnknown ItemType = iota
                      	// ItemEOF indicates that the end of input is reached. No further tokens will be sent.
                      	ItemEOF
                      	// ItemText indicates that it is a block of text separated by some type of space (including tabs).
                      	// This may contain numbers, but if it is not a pure number it is contained in here.
                      	ItemText
                      	// ItemInt indicates that an integer was found.
                      	ItemInt
                      	// ItemFloat indicates that a float was found.
                      	ItemFloat
                      	// ItemEOL indicates the end of a line was reached.
                      	ItemEOL
                      )

                      type Line

                      type Line struct {
                      	// Items are the Item(s) that make up a line.
                      	Items []Item
                      	// LineNum is the line number in the content this represents, starting at 1.
                      	LineNum int
                      	// Raw is the actual raw string that made up the line.
                      	Raw string
                      }

                        Line represents a line in the input.

                        type ParseFn

                        type ParseFn func(ctx context.Context, p *Parser) ParseFn

                          ParseFn handles parsing items provided by a lexer into an object that implements the Validator interface.

                          type Parser

                          type Parser struct {
                          	Validator Validator
                          	// contains filtered or unexported fields
                          }

                            Parser parses items coming from the Lexer and puts the values into *struct that must satisfy the Validator interface. It provides helper methods for recording an Item directory to a field handling text conversions. More complex types such as conversion to time.Time or custom objects are not covered.

                            func NewParser

                            func NewParser(input string, val Validator) (*Parser, error)

                              NewParser is the constructor for Parser.

                              func (*Parser) Backup

                              func (p *Parser) Backup() Line

                                Backup undoes a Next() call and returns the items in the previous line.

                                func (*Parser) Close

                                func (p *Parser) Close()

                                  TODO(johnsiilver): Implement this or we are going to leak goroutines. TODO(johnsiilver): Update the reset to also shut down the lexer so that we stop leaking.

                                  func (*Parser) EOF

                                  func (p *Parser) EOF(line Line) bool

                                    EOF returns true if the last Item in []Item is a ItemEOF.

                                    func (*Parser) Errorf

                                    func (p *Parser) Errorf(str string, args ...interface{}) ParseFn

                                      Errorf records an error in parsing. The ParseFn should immediately return nil. Errorf will always return a nil ParseFn.

                                      func (*Parser) FindREStart

                                      func (p *Parser) FindREStart(find []*regexp.Regexp) (Line, error)

                                        FindREStart looks for a match of [n]*regexp.Regexp against [n]Item.Val continuing to call .Next() until a match is found or EOF is reached. Once this is found, Line is returned. This is done from the current position.

                                        func (*Parser) FindStart

                                        func (p *Parser) FindStart(find []string) (Line, error)

                                          FindStart looks for an exact match of starting items in a line represented by Line continuing to call .Next() until a match is found or EOF is reached. Once this is found, Line is returned. This is done from the current position.

                                          func (*Parser) FindUntil

                                          func (p *Parser) FindUntil(find []string, until []string) (matchFound Line, untilFound bool, err error)

                                            FindUntil searches a Line until it matches "find", matches "until" or reaches the EOF. If "find" is matched, we return the Line. If "until" is matched, we call .Backup() and return true. This is useful when you wish to discover a line that represent a sub-entry of a record (find) but wish to stop searching if you find the beginning of the next record (until).

                                            func (*Parser) HasError

                                            func (p *Parser) HasError() error

                                              HasError returns if the Parser encountered an error.

                                              func (*Parser) IsAtStart

                                              func (p *Parser) IsAtStart(line Line, find []string) bool

                                                IsAtStart checks to see that "find" is at the beginning of "line".

                                                func (*Parser) IsREStart

                                                func (p *Parser) IsREStart(line Line, find []*regexp.Regexp) bool

                                                  IsREStart checks to see that matches to "find" is at the beginning of "line".

                                                  func (*Parser) Next

                                                  func (p *Parser) Next() Line

                                                    Next moves to the next Line sent from the Lexer. That Line is returned. If we haven't received the next Line, the Parser will block until that Line has been received.

                                                    func (*Parser) Peek

                                                    func (p *Parser) Peek() Line

                                                      Peek returns the item in the next position, but does not change the current position.

                                                      func (*Parser) Reset

                                                      func (p *Parser) Reset(s string, val Validator) error

                                                        Reset will reset the Parsers internal attributes for parsing new input "s" into "val".

                                                        type Validator

                                                        type Validator interface {
                                                        	// Validate indicates if the type validates or not.
                                                        	Validate() error
                                                        }

                                                          Validator provides methods to validate that a data type is okay.