cflogparser

README

cflogparser

Description

A parser for Amazon Web Services (AWS) CloudFront log implemented in Go

Example

package main

import (
	"fmt"
	"os"
	"strings"

	"github.com/Maki-Daisuke/cflogparser"
	"github.com/mattn/go-forlines"
)

func main() {
	cnt := map[string]int{} // Count URI

	// Read from Stdin, parse each line and count accesses to each URI
	forlines.Do(os.Stdin, func(line string) error {
		if strings.HasPrefix(line, "#") {
			// Ignore leading comment lines for meta-information
			return nil
		}
		log, err := cflogparser.ParseLineWeb(line)  // ParseLineWeb returns *WebLog
		if err != nil {
			fmt.Fprintln(os.Stderr, err)
			return nil
		}
		cnt[log.URI]++
		return nil
	})

	for k, v := range cnt {
		fmt.Printf("%d\t%s\n", v, k)
	}
}

Supported Formtats

• CloudFront Access log
  • Web Distribution Log File Format: Version 1.0
  • RTMP Distribution Log File Format: Version 1.0

API

See Godoc.

Rationale

CloudFront log format is very simple and looks easy to parse. It is essentially URL-encoded values delimitted by '\t' (tab) character. But, there are putfalls.

For example, it escape some characters twice, such as escaping ' ' (space) to "%2520". (See this AWS Discussion Forums post for the historical background.)

Another example is that it records "-" when there is no value to record even though that field expect an integer value.

We needed something that strictly complies with the specification.

License

MIT License

Author

Daisuke (yet another) Maki

Documentation

Index

• func MustUnescape(s string) string
• func Unescape(s string) (string, error)
• type RTMPLog
  • func ParseLineRTMP(line string) (l *RTMPLog, err error)
• type WebLog
  • func ParseLineWeb(line string) (l *WebLog, err error)

Functions

func MustUnescape

func MustUnescape(s string) string

MustUnescape works as the same as Unescape, but it panics instead of returning error when an error occurs.

func Unescape

func Unescape(s string) (string, error)

Unescape unescapes field value of log. This is essentially a URL unescaping like QueryUnescape of net/url, but slightly dirrerent. Because, CloudFront escapes some characters twice, for example, escaping ' ' (space) into "%2520". See CloudFront doc for the detailed specification: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html Also, please see AWS Discussion Forums for the background: https://forums.aws.amazon.com/thread.jspa?threadID=134017

Types

type RTMPLog

type RTMPLog struct {
	Time          time.Time `json:"time"`
	Location      string    `json:"location"`
	RequestIP     net.IP    `json:"request_ip"`
	EventType     string    `json:"event_type"`
	Bytes         uint64    `json:"bytes"`
	Status        string    `json:"status"`
	ClientID      string    `json:"client_id"`
	URI           string    `json:"uri"`
	QueryString   string    `json:"query_string"`
	Referrer      string    `json:"referrer"`
	PageURL       string    `json:"page_url"`
	UserAgent     string    `json:"user_agent"`
	StreamName    string    `json:"stream_name"`
	StreamQuery   string    `json:"stream_query"`
	StreamFileExt string    `json:"stream_file_ext"`
	StreamID      uint32    `json:"stream_id"`
}

RTMPLog represents a record of RTMP distribution log.

func ParseLineRTMP

func ParseLineRTMP(line string) (l *RTMPLog, err error)

ParseLineRTMP parses a line of log for a RTMP distribution.

type WebLog

type WebLog struct {
	Time               time.Time `json:"time"`
	Location           string    `json:"location"`
	Bytes              uint64    `json:"bytes"`
	RequestIP          net.IP    `json:"request_ip"`
	Method             string    `json:"method"`
	Host               string    `json:"host"`
	URI                string    `json:"uri"`
	Status             uint16    `json:"status"`
	Referrer           string    `json:"referrer"`
	UserAgent          string    `json:"user_agent"`
	QueryString        string    `json:"query_string"`
	Cookie             string    `json:"cookie"`
	ResultType         string    `json:"result_type"`
	RequestID          string    `json:"request_id"`
	HostHeader         string    `json:"host_header"`
	RequestProtocol    string    `json:"request_protocol"`
	RequestBytes       uint64    `json:"request_bytes"`
	TimeTaken          float32   `json:"time_taken"`
	XforwardedFor      string    `json:"xforwarded_for"`
	SslProtocol        string    `json:"ssl_protocol"`
	SslCipher          string    `json:"ssl_cipher"`
	ResponseResultType string    `json:"response_result_type"`
	HTTPVersion        string    `json:"http_version"`
	FleStatus          string    `json:"fle_status"`
	FleEncryptedFields uint32    `json:"fle_encrypted_fields"`
}

WebLog represents a record of CloudFront's Web distribution log. Naming convention is borrowed from this article: https://docs.aws.amazon.com/athena/latest/ug/cloudfront-logs.html

func ParseLineWeb

func ParseLineWeb(line string) (l *WebLog, err error)

ParseLineWeb parses a line of log for a web distribution.