vslparser

README

vslparser

Simple and efficient parser of varnishlog output written in Go.

Description

vslparser is a Go module which provides basic functionality required for further processing of varnishlog output.

varnishlog output is made up of entries, such as Request or BeReq. The provided Parse function parses a single such entry and returns a convenient Entry struct. Little processing is required to obtain the Entry, only line splitting and basic sanity checks are performed.

The Entry provides a range of convenience methods which further parse the entry. This way, only the fields which actually need to be parsed are ever processed. The parsing process is therefore about as efficient as it gets, and the API is easy to use at the same time.

Example

Compile this minimal example to start "varnishlog" and use vslparser to parse its standard output.

package main

import (
	"fmt"
	"github.com/Showmax/vslparser"
	"log"
	"os/exec"
)

func main() {
	cmd := exec.Command("varnishlog") // Add "-b" to only process back-end requests.
	stdout, err := cmd.StdoutPipe()
	if err != nil {
		log.Fatal(err)
	}
	if err := cmd.Start(); err != nil {
		log.Fatal(err)
	}
	scanner := bufio.NewScanner(stdout)
	for {
		entry, err := vslparser.Parse(scanner)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Println(entry)
	}
}

Contributing

Contributions are welcome. Open a PR and we'll get to you soon.

License

Apache 2.0.

Documentation

Index

• Constants
• type Entry
  • func Parse(scanner *bufio.Scanner) (*Entry, error)
  • func (e *Entry) Field(key string) ([]string, error)
  • func (e *Entry) HeadersField(key string) (http.Header, error)
  • func (e *Entry) IntField(key string) (int, error)
  • func (e *Entry) NamedField(key, name string) (string, error)
  • func (e *Entry) NamedFieldParts(key, name string) ([]string, error)
  • func (e *Entry) Timestamp(name string) (*Timestamp, error)
  • func (e *Entry) TryField(key string) string
  • func (e *Entry) URLField(key string) (*url.URL, error)
• type Fields
• type Timestamp

Constants

const (
	Request = "Request"
	BeReq   = "BeReq"
)

Types

type Entry

type Entry struct {
	Kind   string
	VXID   int
	Fields Fields
}

Entry holds a single log entry. An entry consists mostly of a collection of log fields.

func Parse

func Parse(scanner *bufio.Scanner) (*Entry, error)

Parse will attempt to produce a single Entry from the log, which it reads using the given scanner.

The process is fairly efficient, as the individual fields (e.g. time-stamps) aren't converted to any special representation. Instead, the parsed entry is kept mostly in its textual form. Only basic processing, such as splitting lines into fields with a key and a value, are performed. The Entry struct provides various convenience methods which perform the subsequent parsing.

func (*Entry) Field

func (e *Entry) Field(key string) ([]string, error)

Field returns a fields of the log field with the given key. For example, in "- Foo: bar", "Foo" is the key of the field and "bar" is one of the values returned.

func (*Entry) HeadersField

func (e *Entry) HeadersField(key string) (http.Header, error)

Headers returns the values of the field with the given key parsed as a list of HTTP headers if possible. Each value of the field has to be a non-empty header name immediately followed by a colon, followed by the optional value of the header. Headers can appear multiple times.

func (*Entry) IntField

func (e *Entry) IntField(key string) (int, error)

IntField returns a field with the given key, converted into an integer if possible.

func (*Entry) NamedField

func (e *Entry) NamedField(key, name string) (string, error)

NamedField returns a structured field with the given key, whose name component is the given name, if it exists. Names are compared case-insensitive to accommodate for HTTP headers.

A structured field is any field whose value is made of an additional name and value components, for example "- BerespHeader: X-Header: foo", where "BerespHeader" is the key, "X-Header" is the name and "foo" is the value returned.

func (*Entry) NamedFieldParts

func (e *Entry) NamedFieldParts(key, name string) ([]string, error)

NamedFieldParts returns the same value as a corresponding NamedFields call, except that the string value is split by whitespace using strings.Fields.

func (*Entry) Timestamp

func (e *Entry) Timestamp(name string) (*Timestamp, error)

Timestamp parses and returns a Timestamp with the given name, if the log entry contains such a timestamp.

func (*Entry) TryField

func (e *Entry) TryField(key string) string

TryField returns the first value of a log field with the given key if present, or an empty string otherwise.

func (*Entry) URLField

func (e *Entry) URLField(key string) (*url.URL, error)

URLField attempts to parse a URL from the field with the given key.

type Fields

type Fields map[string][]string

Fields are a collection of log fields. A single log field consists of a key and a list of values which appeared with this key. Consider this fragment of varnishlog output:

- Foo Bar Baz - Foo Foobar - Bar

There are two fields, "Foo" with values "Bar Baz" and "Foobar" and the field "Bar" with an empty string as a sole value.

type Timestamp

type Timestamp struct {
	AbsTime     time.Time // Absolute time component.
	UsSinceUnit int       // Number of microseconds since the start of work unit.
	UsSincePrev int       // Number of microseconds since previous stamp.
}

Timestamp represents a parsed time-stamp.