httplog

package module
Version: v0.9.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 31, 2021 License: MIT Imports: 19 Imported by: 1

README

httplog - HTTP Request and Response Logging

!!!!WARNING!!!! - This package works, but is something I wrote a long time ago and really needs to be updated. I logged Issue #8 to some day address this.

Installation

go get -u github.com/gilcrest/httplog

GoDoc

https://pkg.go.dev/github.com/gilcrest/httplog

External Dependencies

httplog has two external libraries that it depends on, listed below.

  • github.com/pkg/errors
  • github.com/rs/zerolog
  • github.com/rs/xid

If you plan to use the Database Logging feature of httplog, you will need to extract the httplogDDL.sql file included in the workspace and run this on your own PostgreSQL database. This script is pretty raw right now and will be made better if there is interest.

Overview

httplog logs http requests and responses. It’s highly configurable, e.g. in production, log all response and requests, but don’t log the body or headers, in your dev environment log everything and so on. httplog also has different ways to log depending on your preference — structured logging via JSON, relational database logging or just plain standard library logging.

httplog has logic to turn on/off logging based on options you can either pass in to the middleware handler or from a JSON input file included with the library.

httplog offers three middleware choices, each of which adhere to fairly common middleware patterns: a simple HandlerFunc (LogHandlerFunc), a function (LogHandler) which takes a handler and returns a handler (often used with alice) and finally, a function (LogAdapter), which returns an Adapter type (based on Mat Ryer’s post). An httplog.Adapt function and httplog.Adapter type are provided for the latter.

Beyond logging request and response elements, httplog creates a unique id for each incoming request (using xid) and sets it (and a few other key request elements) into the request context. You can access these context items using provided helper functions, including a function that returns an httplog.Audit struct which bundles all these items for response payloads to provide clients with helpful information for support.

Features

Middleware

Each middleware takes a minimum of three parameters:

  • log - an instance of zerolog.logger

  • db - a pointer to a sql database (PostgreSQL)

    • You can set this parameter to nil if you are not planning to log to PostgreSQL
  • o - an httplog.Opts struct which has the all of the logging configurations

    • You can set this parameter to nil and httplog will use options from the httpLogOpt.json file
    • If you prefer not to use the httpLogOpt.json file, simply initialize the httplog.Opts struct and all values are set to false (the whole struct is boolean flags and in Go, a boolean's zero value is false). You can then pick and choose which flags to turn on via code.
Middleware Examples

The below examples are taken from go-API-template. It uses all three httplog middlewares for example sake. You obviously would choose 1 pattern and stick to that (for the most part).

package app

import (
    "github.com/gilcrest/go-API-template/datastore"
    "github.com/gilcrest/httplog"
    "github.com/justinas/alice"
)

// routes registers handlers to the router
func (s *server) routes() error {

    // Get a logger instance from the server struct
    log := s.logger

    // Get pointer to logging database to pass into httplog
    // Only need this if you plan to use the PostgreSQL
    // logging style of httplog
    logdb, err := s.ds.DB(datastore.LogDB)
    if err != nil {
        return err
    }

    // httplog.NewOpts gets a new httplog.Opts struct
    // (with all flags set to false)
    opts := httplog.NewOpts()

    // For the examples below, I chose to turn on db logging only
    // Log the request headers only (body has password on this api!)
    // Log both the response headers and body
    opts.Log2DB.Enable = true
    opts.Log2DB.Request.Header = true
    opts.Log2DB.Response.Header = true
    opts.Log2DB.Response.Body = true

    // HandlerFunc middleware example
    // function takes an http.HandlerFunc and returns an http.HandlerFunc
    // Also, match only POST requests with Content-Type header = application/json
    s.router.HandleFunc("/v1/handlefunc/user",
        httplog.LogHandlerFunc(s.handleUserCreate(), log, logdb, opts)).
        Methods("POST").
        Headers("Content-Type", "application/json")

    // function (`LogHandler`) that takes a handler and returns a handler (aka Constructor)
    // (`func (http.Handler) http.Handler`)    - used with alice
    // Also, match only POST requests with Content-Type header = application/json
    s.router.Handle("/v1/alice/user",
        alice.New(httplog.LogHandler(log, logdb, opts)).
            ThenFunc(s.handleUserCreate())).
        Methods("POST").
        Headers("Content-Type", "application/json")

    // Adapter Type middleware example
    // Also, match only POST requests with Content-Type header = application/json
    s.router.Handle("/v1/adapter/user",
        httplog.Adapt(s.handleUserCreate(),
            httplog.LogAdapter(log, logdb, opts))).
        Methods("POST").
        Headers("Content-Type", "application/json")

    return nil
}

Configurable Logging

The boolean fields found within the Opts struct type drive the rules for what logging features are turned on. You can have one to three log styles turned on using this file (or none, if you so choose). Below are all the boolean options in the struct.

Note: Right now, this struct for options serves its purpose and is pretty simple. I have read Dave Cheney's great post on Functional Options for Friendly APIs and it's great - I may switch to this style later.

// Log2StdOut
opts.Log2StdOut.Request.Enable
opts.Log2StdOut.Request.Options.Header
opts.Log2StdOut.Request.Options.Body
opts.Log2StdOut.Response.Enable
opts.Log2StdOut.Response.Options.Header
opts.Log2StdOut.Response.Options.Body

// Log2DB
opts.Log2DB.Enable
opts.Log2DB.Request.Header
opts.Log2DB.Request.Body
opts.Log2DB.Response.Header
opts.Log2DB.Response.Body

// DumpRequest
opts.HTTPUtil.DumpRequest.Body
opts.HTTPUtil.DumpRequest.Body
  1. Pass an Opts struct when using one of the given middleware functions. httplog.NewOpts will return an Opts struct with all logging turned off. You can then set whichever logging style and option you like.
  2. If you do not pass an Opts struct to one of the provided middlewares, there is code in each that will import/marshal the httpLogOpt.json file found in the root of the httplog library into the Opts struct type. You can change log configuration by altering the boolean values present in this file.
Log Style 1: Structured via JSON
JSON Request Logging

Set log_json.Request.enable in the HTTP Log Config File or opts.Log2StdOut.Request.Enable to true in the httplog.Opts struct to enable http request logging as JSON (so long as you have properly "chained" the middleware). The output for a request looks something like:

{"time":1517970302,"level":"info","request_id":"b9t66vma6806ln8iak8g","header_json":"{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Content-Length\":[\"129\"],\"Content-Type\":[\"application/json\"],\"Postman-Token\":[\"9949f5e5-b406-4e22-aff3-ab6ba6e7d841\"],\"User-Agent\":[\"PostmanRuntime/7.1.1\"]}","body":"{\"username\": \"repoMan\",\"mobile_ID\": \"1-800-repoman\",\"email\":\"repoman@alwaysintense.com\",\"First_Name\":\"Otto\",\"Last_Name\":\"Maddox\"}","method":"POST","scheme":"http","host":"127.0.0.1","port":"8080","path":"/api/v1/appuser","protocol":"HTTP/1.1","proto_major":1,"proto_minor":1,"Content Length":129,"Transfer-Encoding":"","Close":false,"RemoteAddr":"127.0.0.1:58689","RequestURI":"/api/v1/appuser","message":"Request received"}

NOTE - the HTTP header key:value pairs and json from the body are represented as escaped JSON within the actual message. If you don't want this data, set these fields to false in the JSON config file or httplog.Opts struct.

JSON Response Logging

Set log_json.Response.enable in the HTTP Log Config File or opts.Log2StdOut.Response.Enable to true to enable http response logging as JSON. The response output will look something like:

{"time":1517970302,"level":"info","request_id":"b9t66vma6806ln8iak8g","response_code":200,"response_header":"{\"Content-Type\":[\"text/plain; charset=utf-8\"],\"Request-Id\":[\"b9t66vma6806ln8iak8g\"]}","response_body":"{\"username\":\"repoMan\",\"mobile_id\":\"1-800-repoman\",\"email\":\"repoman@alwaysintense.com\",\"first_name\":\"Otto\",\"last_name\":\"Maddox\",\"create_user_id\":\"gilcrest\",\"create_date\":\"2018-02-06T21:25:02.538322Z\",\"update_user_id\":\"\",\"update_date\":\"0001-01-01T00:00:00Z\"}\n{\"username\":\"repoMan\",\"mobile_id\":\"1-800-repoman\",\"email\":\"repoman@alwaysintense.com\",\"first_name\":\"Otto\",\"last_name\":\"Maddox\",\"create_user_id\":\"gilcrest\",\"create_date\":\"2018-02-06T21:25:02.538322Z\",\"update_user_id\":\"\",\"update_date\":\"0001-01-01T00:00:00Z\"}\n","message":"Response Sent"}

NOTE - same as request - the HTTP header key:value pairs and json from the body are represented as escaped JSON within the actual message. If you don't want this data, set these fields to false in the JSON config file (httpLogOpt.json) or httplog.Opts struct.

Log Style 2: Relational DB Logging via PostgreSQL

Set log_2DB.enable to true in the HTTP Log Config File to enable Database logging to a PostgreSQL database. The DDL is provided within the ddl directory (httplogDDL.sql) and consists of one table and one stored function. Once enabled, Request and Response information will be logged as one transaction to the database. You can optionally choose to log request and response headers using the Options fields within the HTTP Log Config File or httplog.Opts struct.

Database Log

Logging Database Table

In total 20 fields are logged as part of the database transaction.

Column Name Datatype Description
request_id VARCHAR(100) Unique Request ID
client_id VARCHAR(100) API Client ID
request_timestamp TIMESTAMP UTC time request received
response_code INTEGER HTTP Response Code
response_timestamp TIMESTAMP UTC time response sent
duration_in_millis BIGINT Response time duration in milliseconds
protocol VARCHAR(20) HTTP protocol version, e.g. HTTP/1.1
protocol_major INTEGER HTTP protocol major version
protocol_minor INTEGER HTTP protocol minor version
request_method VARCHAR(10) HTTP method (GET, POST, PUT, etc.)
scheme VARCHAR(100) URL scheme (http, https, etc.)
host VARCHAR(100) URL host
port VARCHAR(100) URL port
path VARCHAR(4000) URL path
remote_address VARCHAR(100) Network address which sent request
request_content_length BIGINT Request content length
request_header JSONB Key:Value pairs from HTTP request in JSON format
request_body TEXT Request body content
response_header JSONB Key:Value pairs from HTTP response in JSON format
response_body TEXT Response body content
Log Style 3: httputil DumpRequest or DumpResponse
httputil.DumpRequest

Set httputil.DumpRequest.enable in the HTTP Log Config File or opts.HTTPUtil.DumpRequest.Enable in httplog.Opts to true to enable logging the request via the httputil.DumpRequest method. Nothing special here, really - just providing an easy way to turn this on or off. Output typically looks like:

httputil.DumpRequest output:
POST /api/v1/appuser HTTP/1.1
Host: 127.0.0.1:8080
Accept: */*
Accept-Encoding: gzip, deflate
Cache-Control: no-cache
Connection: keep-alive
Content-Length: 129
Content-Type: application/json
Postman-Token: 6d1b2461-59e2-4c87-baf5-d8e64a93c55b
User-Agent: PostmanRuntime/7.1.1

{"username": "repoMan","mobile_ID": "1-800-repoman","email":"repoman@alwaysintense.com","First_Name":"Otto","Last_Name":"Maddox"}

NOTE - in order to log the body, set httputil.DumpRequest.body in httplogOpt.json or opts.HTTPUtil.DumpRequest.Enable in httplog.Opts to true. If you don't want this data, set the appropriate field to false in the JSON config file (httpLogOpt.json) or httplog.Opts struct (depending on which method you chose).

Add Unique ID and Key Request Elements to Context

httplog middleware creates a unique ID to track each request. In addition, it adds several request elements to the request context that can be accessed with helper functions later.

Unique Request ID

Each request is given a 20 character Unique Request ID generated by xid. This unique ID is populated throughout each log type for easy tracking. This ID is also meant to be sent back to the client of your API either in the response header or response body (see below for further help on including httplog context items in a response body).

Other Request Elements added to Context

In addition to the generated Unique ID, httplog also adds the following request elements to the context:

  • Host
  • Port
  • Path
  • Raw Query
  • Fragment
Retrieve Unique ID and Key Request Elements from Context

In order to retrieve particular key:value pairs from the request context, the following helper functions are provided:

// RequestID gets the Request ID from the context.
func RequestID(ctx context.Context) string {
// RequestHost gets the request host from the context
func RequestHost(ctx context.Context) string {
// RequestPort gets the request port from the context
func RequestPort(ctx context.Context) string {
// RequestPath gets the request URL from the context
func RequestPath(ctx context.Context) string {
// RequestRawQuery gets the request Query string details from the context
func RequestRawQuery(ctx context.Context) string {
// RequestFragment gets the request Fragment details from the context
func RequestFragment(ctx context.Context) string {
Audit Struct for Response Payload

Some APIs may find it helpful to echo back certain request elements or helpful contextual information in the response payload. httplog provides httplog.Audit for just this purpose. Use constructor function httplog.NewAudit to initialize this struct. The unique Request ID will always be sent back as part of the struct -- the other request elements are optional and can be turned on/off using the httplog.AuditOpts config struct. Below is a sample response with the audit struct included to give an idea of how it can be used. The example below is from the go-API-template repository which has examples of this audit struct in use.

{
    "username": "15",
    "mobile_id": "1-800-repoman",
    "email": "repoman@alwaysintense.com",
    "first_name": "Otto",
    "last_name": "Maddox",
    "update_user_id": "chillcrest",
    "created": 1539138260,
    "audit": {
        "id": "beum5l708qml02e3hvag",
        "url": {
            "host": "127.0.0.1",
            "port": "8080",
            "path": "/api/v1/adapter/user",
            "query": "qskey1=fake&qskey2=test"
        }
    }
}

A snippet from the handleUserCreate handler function within go-API-template shows how to setup the AuditOpts struct and turn on a few options as well as plugging httplog.Audit into the response.

    // create new AuditOpts struct and set options to true that you
    // want to see in the response body (Request ID is always present)
    aopt := new(httplog.AuditOpts)
    aopt.Host = true
    aopt.Port = true
    aopt.Path = true
    aopt.Query = true

    // get a new httplog.Audit struct from NewAudit using the
    // above set options and request context
    aud, err := httplog.NewAudit(ctx, aopt)
    if err != nil {
        err = HTTPErr{
            Code: http.StatusInternalServerError,
            Kind: errs.Other,
            Err:  err,
        }
        httpError(w, err)
        return
    }

    // create a new response struct and set Audit and other
    // relevant elements
    resp := new(response)
    resp.Audit = aud
    resp.Username = usr.Username()
    resp.MobileID = usr.MobileID()
    resp.Email = usr.Email()
    resp.FirstName = usr.FirstName()
    resp.LastName = usr.LastName()
    resp.UpdateUserID = usr.UpdateUserID()
    resp.UpdateUnixTime = usr.UpdateTimestamp().Unix()

    // Encode response struct to JSON for the response body
    json.NewEncoder(w).Encode(*resp)

Helpful Resources I've used in this library (outside of the standard, yet amazing blog.golang.org and golang.org/doc/, etc.)

websites/youtube

Books

Blog/Medium Posts

Documentation

Overview

Package httplog logs http requests and responses. It’s highly configurable, e.g. in production, log all response and requests, but don’t log the body or headers, in your dev environment log everything and so on. httplog also has different ways to log depending on your preference — structured logging via JSON, relational database logging or just plain standard library logging. httplog has logic to turn on/off logging based on options you can either pass in to the middleware handler or from a JSON input file included with the library. httplog offers three middleware choices, each of which adhere to fairly common middleware patterns: a simple HandlerFunc (`LogHandlerFunc`), a function (`LogHandler`) that takes a handler and returns a handler (aka Constructor) (`func (http.Handler) http.Handler`) often used with alice (https://github.com/justinas/alice) and finally a function (`LogAdapter`) that returns an Adapter type. An `httplog.Adapt` function and `httplog.Adapter` type are provided. Beyond logging request and response elements, httplog creates a unique id for each incoming request (using xid (https://github.com/rs/xid)) and sets it (and a few other key request elements) into the request context. You can access these context items using provided helper functions, including a function that returns an audit struct you can add to response payloads that provide clients with helpful information for support.

!!!!WARNING!!!! - This package works, but is something I wrote a long time ago and really needs to be updated. I logged Issue #8 to some day address this.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Adapt

func Adapt(h http.Handler, adapters ...Adapter) http.Handler

Adapt function takes the handler you want to adapt, and a list of our Adapter types. The result of any wrapper function should be an acceptable Adapter. Our Adapt function will simply iterate over all adapters, calling them one by one (in reverse order) in a chained manner, returning the result of the first adapter. - Mat Ryer @matryer

func Log2Database added in v0.5.0

func Log2Database(enable bool, reqHdr bool, reqBody bool, respHdr bool, respBody bool) option

Log2Database sets the options for logging to the database. enable turns on the functionality - if this is set to false, the parameters afterward are irrelevant as nothing will log. reqHdr logs http request headers reqBody logs the http request body respHdr logs http response headers respBody logs the http response body

func LogHandler

func LogHandler(logger zerolog.Logger, db *sql.DB, o *Opts) (mw func(http.Handler) http.Handler)

LogHandler records and logs as much as possible about an incoming HTTP request and response

func LogHandlerFunc

func LogHandlerFunc(next http.HandlerFunc, logger zerolog.Logger, db *sql.DB, o *Opts) http.HandlerFunc

LogHandlerFunc middleware records and logs as much as possible about an incoming HTTP request and response

func LogRequest2Stdout added in v0.5.0

func LogRequest2Stdout(enable bool, header bool, body bool) option

LogRequest2Stdout sets the options for logging http requests to Standard Output (stdout). enable turns on the functionality header logs http request headers body logs the http request body

func LogRequestViaHTTPUtil added in v0.5.0

func LogRequestViaHTTPUtil(enable bool, body bool) option

LogRequestViaHTTPUtil sets the options for logging requests using the standard HTTPUtil package enable turns on the functionality body logs the http request body This is just wrapper functionality and in hindsight is really kinda silly to include, but alas...

func LogResponse2Stdout added in v0.5.0

func LogResponse2Stdout(enable bool, header bool, body bool) option

LogResponse2Stdout sets the options for logging http responses to Standard Output (stdout). enable turns on the functionality header logs http response headers body logs the http response body

func RequestFragment

func RequestFragment(ctx context.Context) (string, error)

RequestFragment gets the request Fragment details from the context

func RequestHost

func RequestHost(ctx context.Context) (string, error)

RequestHost gets the request host from the context

func RequestID

func RequestID(ctx context.Context) (string, error)

RequestID gets the Request ID from the context.

func RequestPath

func RequestPath(ctx context.Context) (string, error)

RequestPath gets the request URL from the context

func RequestPort

func RequestPort(ctx context.Context) (string, error)

RequestPort gets the request port from the context

func RequestRawQuery

func RequestRawQuery(ctx context.Context) (string, error)

RequestRawQuery gets the request Query string details from the context

Types

type Adapter

type Adapter func(http.Handler) http.Handler

Adapter type (it gets its name from the adapter pattern — also known as the decorator pattern) above is a function that both takes in and returns an http.Handler. This is the essence of the wrapper; we will pass in an existing http.Handler, the Adapter will adapt it, and return a new (probably wrapped) http.Handler for us to use in its place. So far this is not much different from just wrapping http.HandlerFunc types, however, now, we can instead write functions that themselves return an Adapter. - Mat Ryer @matryer

func LogAdapter

func LogAdapter(logger zerolog.Logger, db *sql.DB, o *Opts) Adapter

LogAdapter records and logs as much as possible about an incoming HTTP request and response using the Adapter pattern Found adapter pattern in a Mat Ryer post

type Audit

type Audit struct {
	RequestID string   `json:"id"`
	URL       AuditURL `json:"url,omitempty"`
}

Audit struct can be included as part of an http response body. This struct sends back the Unique ID generated upon receiving a request as well as echoes back the URL information to help with debugging.

func NewAudit

func NewAudit(ctx context.Context) (*Audit, error)

NewAudit is a constructor for the Audit struct. Elements added to the context through the provided middleware functions can be retrieved through the various helper functions or if you prefer in this one convenient struct.

type AuditURL

type AuditURL struct {
	RequestHost     string `json:"host,omitempty"`
	RequestPort     string `json:"port,omitempty"`
	RequestPath     string `json:"path,omitempty"`
	RequestRawQuery string `json:"query,omitempty"`
	RequestFragment string `json:"fragment,omitempty"`
}

AuditURL has URL info which can be included as part of an http response body

type DumpRequest

type DumpRequest struct {
	Enable bool `json:"enable"`
	Body   bool `json:"body"`
}

DumpRequest holds the options for the net/http/httputil.DumpRequest function

type HTTPUtil

type HTTPUtil struct {
	DumpRequest DumpRequest
}

HTTPUtil struct hold the options for using the net/http/httputil package

type L2SOpt

type L2SOpt struct {
	Enable  bool `json:"enable"`
	Options ROpt
}

L2SOpt struct holds the log2StdOut options. Enable should be true if you want httplog to write to stdout, set the ROpt Header and Body booleans accordingly if you want to write those

type Log2DB

type Log2DB struct {
	Enable   bool `json:"enable"`
	Request  ROpt
	Response ROpt
}

Log2DB struct holds the options for logging to a database Set Enable to true you want any database logging Set the Request and Response options according to whether you want to log request and/or response to the database Requests/Responses will only be logged if Enable is true

type Log2StdOut

type Log2StdOut struct {
	Request  L2SOpt
	Response L2SOpt
}

Log2StdOut (Log to Standard Output) struct holds the options for logging requests and responses to stdout (using zerolog)

type Opts

type Opts struct {
	Log2StdOut Log2StdOut `json:"log_json"`
	Log2DB     Log2DB     `json:"log_2DB"`
	HTTPUtil   HTTPUtil   `json:"httputil"`
}

Opts represent HTTP Logging Options

func FileOpts

func FileOpts() (*Opts, error)

FileOpts constructs an Opts struct using the httpLogOpt.json file included with the library The idea here is to have a config file that you can swap out on different servers - many enterprises will not let you touch "source code", but allow for manipulation of a config file like this... go figure

func (*Opts) Option added in v0.4.0

func (o *Opts) Option(opts ...option)

Option sets the options specified.

type ROpt

type ROpt struct {
	Header bool `json:"header"`
	Body   bool `json:"body"`
}

ROpt is the http request/response logging options choose whether you want to log the http headers or body by setting either value to true

Directories

Path Synopsis
Package errs is a modified copy of the upspin.io/errors package.
Package errs is a modified copy of the upspin.io/errors package.
Package logger has helpers to setup a zerolog.Logger https://github.com/rs/zerolog
Package logger has helpers to setup a zerolog.Logger https://github.com/rs/zerolog

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL