nxparse

README

NXParse

The best GoLang package to parse NGINX access.log entries!

This package was created for the GNU Open Source software Logslate. A defensive security log analyser CLI.

How to use

This package has a single public function, so you really can't go wrong.

nxparse.Parse(format, line)

As detailed in nxparse.go, the format is an almost exact copy of your nginx.conf log_format directive. If you're using the default NGINX combined log_format you can use nxparse.Combined as the format argument. If you're not specifying your own, that's what you're using. If you copy/paste, make sure to remove new lines and escape " with \".

The package supports any format. If it's working in your nginx.conf, it should work here.

Log Entry

The nxparse.Entry struct contains all possible existing values that can be specified in the nginx.conf log_format option. HTTP headers are stored in entry.HTTP["header"]. See parse.go.

Format cache

You can use any number of formats. They will be cached once parsed. The Combined format is pre-cached.

Error handling

All errors are wrapped, are in the public API, and start with Err. You can use errors.Is(err, ErrWhatever) to check for specific errors.

Test coverage

Test coverage is 100% and is enforced by pre-release checks.

Documentation

Overview

Package nxparse (NGINX parse) provides an API to parse NGINX access logs.

The API respects https://nginx.org/en/docs/http/ngx_http_log_module.html#log_format

Index

• Constants
• Variables
• type Entry
  • func Parse(format, line string) (Entry, error)

Constants

const (
	TimeLayoutIso8601 = time.RFC3339
	TimeLayoutLocal   = "02/Jan/2006:15:04:05 -0700"
)

const (
	Combined = "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\""
)

Default `combined` `log_format`.

Variables

var (
	ErrNoCachedFormat = errors.New("cached format not found")
	ErrOverwriteCache = errors.New("cannot overwrite cache")
)

Cache handling errors.

var (
	ErrNilVarName   = errors.New("format variable nil after first position")
	ErrNilSeparator = errors.New("format separator nil before last position")
	ErrParseFailed  = errors.New("parsing log line failed")
	ErrSepNotFound  = errors.New("separator not found")
	ErrVarNotExist  = errors.New("log variable does not exist")
	ErrConverFail   = errors.New("value conversion failed")
	ErrIpConvFail   = errors.New("IP address conversion failed")
)

Entry parsing errors.

Types

type Entry

type Entry struct {
	BodyBytesSent      int
	BytesSent          int
	Connection         string
	ConnectionRequests int
	Http               map[string]string
	Msec               int
	Pipe               bool
	RemoteAddr         net.IP
	RemoteUser         string
	Request            string
	RequestLength      int
	RequestTime        int
	Status             int
	TimeIso8601        time.Time
	TimeLocal          time.Time
}

An Entry represents a line from the NGINX access log. Typically, you'll use Parse() to populate this.

func Parse

func Parse(format, line string) (Entry, error)

Parse parses a single line of NGINX access log and returns a structured representation of the entry.

The format is the same (almost) string as you have in the nginx.conf `log_format` directive. You need to escape " characters with \ to avoid breaking GoLang strings. The format doesn't support new lines either. So, don't use those. By default, NGINX uses the `combined` log format, so you'll typically just use the nxparse.Combined constant which defines this format. If you use anything else, you'll have to use your format.

The line must be the entry from the access log including the first and last characters of the line.