httplog

package module
v1.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 1, 2023 License: MIT Imports: 15 Imported by: 13

README

Beautiful logger for http

Proudly created and supported by MadAppGang company.

Go Reference Go Report Card codecov

Why?

Every single web framework has a build-in logger already, why do we need on more? The question is simple and the answer is not.

The best logger is structured logger, like Uber Zap. Structure logs are essentials today to help collect and analyze logs with monitoring tools like ElasticSearch or DataDog.

But what about humans? Obviously you can go to your monitoring tool and parse your structured logs. But what about seeing human readable logs in place just in console? Although you can read JSON, it is extremely hard to find most critical information in the large JSON data flow.

Httplog package brings good for humans and log collection systems, you can use tools like zap to create structured logs and see colored beautiful human-friendly output in console in place. Win-win for human and machines 🤖❤️👩🏽‍💻

Nice and clean output is critical for any web framework. Than is why some people use go web frameworks just to get beautiful logs.

This library brings you fantastic http logs to any web framework, even if you use native net/http for that.

But it's better to see once, here the default output you will get with couple of lines of code:

logs screenshot

And actual code looks like this:

  func main() {
    // setup routes
    http.Handle("/happy", httplog.Logger(happyHandler))
    http.Handle("/not_found", httplog.Logger(http.NotFoundHandler()))

    //run server
    _ = http.ListenAndServe(":3333", nil)
  }

All you need is wrap you handler with httplog.Logger and the magic happens.

And this is how the structured logs looks in AWS CloudWatch. As you can see, the color output looks not as good here as in console. But JSON structured logs are awesome 😍

structured logs

Here is a main features:

  • framework agnostic (could be easily integrated with any web framework), you can find examples for:
  • response code using special wrapper
  • response length using special wrapper
  • can copy response body
  • get real user IP for Google App Engine
  • get real user IP for CloudFront
  • get real user IP for other reverse proxy which implements RFC7239
  • customize output format
  • has the list of routes to ignore
  • build in structure logger integration
  • callback function to modify response before write back (add headers or do something)

This framework is highly inspired by Gin logger library, but has not Gin dependencies at all and has some improvements. Httplog has only one dependency at all: github.com/mattn/go-isatty. So it's will not affect your codebase size.

Custom format

You can modify formatter as you want. Now there are two formatter available:

  • DefaultLogFormatter
  • ShortLogFormatter
  • HeadersLogFormatter
  • DefaultLogFormatterWithHeaders
  • BodyLogFormatter
  • DefaultLogFormatterWithHeadersAndBody
  • RequestHeaderLogFormatter
  • DefaultLogFormatterWithRequestHeader
  • RequestBodyLogFormatter
  • DefaultLogFormatterWithRequestHeadersAndBody
  • FullFormatterWithRequestAndResponseHeadersAndBody

And you can combine them using ChainLogFormatter.

Here is an example of formatter in code:

// Short log formatter
shortLoggedHandler := httplog.LoggerWithFormatter(
  httplog.ShortLogFormatter,
  wrappedHandler,
)

You can define your own log format. Log formatter is a function with a set of precalculated parameters:

// Custom log formatter
customLoggedHandler := httplog.LoggerWithFormatter(
  // formatter is a function, you can define your own
  func(param httplog.LogFormatterParams) string {
    statusColor := param.StatusCodeColor()
    resetColor := param.ResetColor()
    boldRedText := "\033[1;31m"

    return fmt.Sprintf("🥑[I am custom router!!!] %s %3d %s| size: %10d bytes | %s %#v %s 🔮👨🏻‍💻\n",
      statusColor, param.StatusCode, resetColor,
      param.BodySize,
      boldRedText, param.Path, resetColor,
    )
  },
  happyHandler,
)
http.Handle("/happy_custom", customLoggedHandler)

For more details and how to capture response body please look in the example app.

params is a type of LogFormatterParams and the following params available for you:

param description
Request http.Request instance
RouterName when you create logger, you can specify router name
Timestamp TimeStamp shows the time after the server returns a response
StatusCode StatusCode is HTTP response code
Latency Latency is how much time the server cost to process a certain request
ClientIP ClientIP calculated real IP of requester, see Proxy for details
Method Method is the HTTP method given to the request
Path Path is a path the client requests
BodySize BodySize is the size of the Response Body
Body Body is a body content, if body is copied

Integrate with structure logger

Good and nice output is good, but as soon as we have so much data about every response it is a good idea to pass it to our application log structured collector.

One of the most popular solution is Uber zap. You can use any structured logger you want, use zap's integration example as a reference.

All you need is create custom log formatter function with your logger integration. This repository has this formatter for zap created and you can use it importing github.com/MadAppGang/httplog/zap:

logger := httplog.LoggerWithConfig(
  httplog.LoggerConfig{
    Formatter:  lzap.DefaultZapLogger(zapLogger, zap.InfoLevel, ""),
  },
  http.HandlerFunc(handler),
)
http.Handle("/happy", logger)

You can find full-featured example in zap integration folder.

Customize log output destination

You can use any output you need, your output must support io.Writer protocol. After that you need init logger:

buffer := new(bytes.Buffer)
logger := LoggerWithWriter(buffer, handler) //all output is written to buffer

Care about secrets. Skipping path and masking headers

Some destinations should not be logged. For that purpose logger config has SkipPaths property with array of strings. Each string is a Regexp for path you want to skip. You can write exact path to skip, which would be valid regexp, or you can use regexp power:


logger := LoggerWithConfig(LoggerConfig{
  SkipPaths: []string{
    "/skipped",
    "/payments/\\w+",
    "/user/[0-9]+",
  },
}, handler)


The other feature to safe your logs from leaking secrets is Header masking. For example you do not want to log Bearer token header, but it is  useful to see it is present and not empty.

For this purpose `LoggerConfig` has field `HideHeaderKeys` which works the same as `SkipPaths`. Just feed an array of case insensitive key names regexps like that:

```go

logger := LoggerWithConfig(LoggerConfig{
  HideHeaderKeys: []string{
    "Bearer",
    "Secret-Key",
    "Cookie",
  },
}, handler)

If regexp is failed to compile, the logger will skip and and it will write the message to the log output destination.

Use GoogleApp Engine or CloudFlare

You application is operating behind load balancers and reverse proxies. That is why origination IP address is changing on every hop. To save the first sender's IP (real user remote IP) reverse proxies should save original IP in request headers. Default headers are X-Forwarded-For and X-Real-IP.

But some clouds have custom headers, like Cloudflare and Google Apps Engine. If you are using those clouds or have custom headers in you environment, you can handle that by using custom Proxy init parameters:

httplog.LoggerWithConfig(
  httplog.LoggerConfig{
    ProxyHandler: NewProxyWithType(httpdlog.ProxyGoogleAppEngine),
  },
http.HandlerFunc(h),

or if you have your custom headers:

logger := httplog.LoggerWithConfig(
  httplog.LoggerConfig{
    ProxyHandler: NewProxyWithTypeAndHeaders(
      httpdlog.ProxyDefaultType,
      []string{"My-HEADER-NAME", "OTHER-HEADER-NAME"}
    ),
  },
  handler,
)

How to save request body and headers

You can capture response data as well. But please use it in dev environments only, as it use extra resources and produce a lot of output in terminal. Example of body output could be found here.

body output

You can use DefaultLogFormatterWithHeaders for headers output or DefaultLogFormatterWithHeadersAndBody to output response body. Don't forget to set CaptureBody in LoggerParams.

You can combine your custom Formatter and HeadersLogFormatter or/and BodyLogFormatter using ChainLogFormatter:

var myFormatter = httplog.ChainLogFormatter(
  MyLogFormatter,
  httplog.HeadersLogFormatter,
  httplog.BodyLogFormatter,
)

Integration examples

Please go to examples folder and see how it's work:

Run demo

Native net/http package

This package is suing canonical golang approach, and it easy to implement it with all net/http package.

http.Handle("/not_found", httplog.Logger(http.NotFoundHandler()))

Full example could be found here.

Alice middleware

Alice is a fantastic lightweight framework to chain and manage middlewares. As Alice is using canonical approach, it is working with httplog out-of-the-box.

You don't need any wrappers and you can user logger directly:


.....
chain := alice.New(httplog.Logger, nosurf.NewPure)
mux.Handle("/happy", chain.Then(happyHandler()))

Full example could be found here.

Chi

Chi is a router which uses the standard approach as Alice package.

You don't need any wrappers and you can user logger directly:


r := chi.NewRouter()
r.Use(httplog.Logger)
r.Get("/happy", happyHandler)
...

Full example [could be found here](https://github.com/MadAppGang/httplog/blob/main/examples/chi/main.go).

Echo

Echo middleware uses internal echo.Context to manage request/responser flow and middleware management.

As a result, echo creates it's own http.ResponseWriter wrapper to catch all written data. That is why httplog could not handle it automatically.

All you need is create wrapper, which set the response status and size manually from echo's response writer wrapper.

func loggerMiddleware(next echo.HandlerFunc) echo.HandlerFunc {
  return func(c echo.Context) error {
    logger := httplog.Logger(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
      if err := next(c); err != nil {
]      c.Error(err)
      }
      // echo is overwriting http.ResponseWriter to proxy data write (c.Response() as echo.Response type)
      // we just need manually bring this data back to our log's response writer (rw variable)
      lrw, _ := rw.(httplog.ResponseWriter)
      lrw.Set(c.Response().Status, int(c.Response().Size))
    }))

    logger.ServeHTTP(c.Response().Writer, c.Request())
    return nil
  }
}

//use this as native echo middleware:
e := echo.New()
e.Use(loggerMiddleware)11
e.GET("/happy", happyHandler)

Full example could be found here.

Gin

Gin has the most beautiful log output. That is why this package has build highly inspired by Gin's logger.

If you missing some features of Gin's native logger and want to use this one, it is still possible.

Gin middleware uses internal gin.Context to manage request/responser flow and middleware management. The same approach as Echo.

That is why we created httplog, to bring this fantastic logger to native golang world :-)

Gin creates it's own http.ResponseWriter wrapper to catch all written data. That is why httplog could not handle it automatically.

All you need is create wrapper, which set the response status and size manually from Gin's response writer wrapper.

// httplog.ResponseWriter is not fully compatible with gin.ResponseWriter
func LoggerMiddleware() gin.HandlerFunc {
  return func(c *gin.Context) {
    l := httplog.Logger(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
      // gin uses wrapped ResponseWriter, we don't want to replace it
      // as gin use custom middleware approach with Next() method and context
      c.Next()
      // set result for ResponseWriter manually
      rwr, _ := rw.(httplog.ResponseWriter)
      rwr.Set(c.Writer.Status(), c.Writer.Size())
    }))
    l.ServeHTTP(c.Writer, c.Request)
  }
}

//Use now this wrapper as native middleware
r := gin.New()
r.Use(LoggerMiddleware())
r.GET("/happy", happyHandler)

Full example could be found here.

Goji

Goji is using canonical middleware approach, that is why it is working with httplog out-of-the-box.

You don't need wrappers or anything like that.

mux := goji.NewMux()
mux.Handle(pat.Get("/happy"), httplog.Logger(happyHandler))

Full example could be found here.

Gorilla

Gorilla mux is one of the most loved muxer/router. Gorilla is using canonical middleware approach, that is why it is working with httplog out-of-the-box.

You don't need to create any wrappers:


r := mux.NewRouter()
r.HandleFunc("/happy", happyHandler)
r.Use(httplog.Logger)

Full example could be found here.

HTTPRouter

To use HTPPRouter you need to create simple wrapper. As HTTPRouter using custom httprouter.Handler and additional argument with params in handler function.

func LoggerMiddleware(h httprouter.Handle) httprouter.Handle {
  return func(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
    logger := httplog.Logger(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
      h(w, r, ps)
    }))
    logger.ServeHTTP(w, r)
  }
}

//use as native middleware
router := httprouter.New()
router.GET("/happy", LoggerMiddleware(happyHandler))

Full example could be found here.

Mojito

Because Go-Mojito uses dynamic handler functions, which include support for net/http types, httplog works out-of-the-box:

mojito.WithMiddleware(httplog.Logger)
mojito.GET("/happy", happyHandler)

Full example could be found here.

Negroni

Negroni uses custom negroni.Handler as middleware. We need to create custom wrapper for that:

var negroniLoggerMiddleware negroni.Handler = negroni.HandlerFunc(func(rw http.ResponseWriter, r *http.Request, next http.HandlerFunc) {
  logger := httplog.Logger(next)
  logger.ServeHTTP(rw, r)
})

//use it as native middleware:
n := negroni.New()
n.Use(negroniLoggerMiddleware)
n.UseHandler(mux)

Full example could be found here.

Documentation

Index

Constants

View Source
const (
	// ProxyGoogleAppEngine when running on Google App Engine. Trust X-Appengine-Remote-Addr
	// for determining the client's IP
	ProxyGoogleAppEngine = "X-Appengine-Remote-Addr"
	// ProxyCloudflare when using Cloudflare's CDN. Trust CF-Connecting-IP for determining
	// the client's IP
	ProxyCloudflare = "CF-Connecting-IP"

	// ProxyDefaultType represents default proxy type
	ProxyDefaultType = ""
)

Supported proxies

Variables

View Source
var DefaultErrorWriter io.Writer = os.Stderr

DefaultErrorWriter is the default io.Writer used by httplog to debug errors

View Source
var DefaultLogFormatterWithRequestHeader = ChainLogFormatter(DefaultLogFormatter, RequestHeaderLogFormatter)

DefaultLogFormatterWithHeaders is a combination of default log formatter and header log formatter

DefaultLogFormatterWithHeadersAndBody is a combination of default log formatter, header log formatter and json body

View Source
var DefaultLogFormatterWithResponseHeader = ChainLogFormatter(DefaultLogFormatter, ResponseHeaderLogFormatter)

DefaultLogFormatterWithHeader is a combination of default log formatter and header log formatter

DefaultLogFormatterWithHeadersAndBody is a combination of default log formatter, header log formatter and json body

View Source
var DefaultRemoteIPHeaders = []string{"X-Forwarded-For", "X-Real-IP"}

Default proxy remote IP headers

View Source
var DefaultWriter io.Writer = os.Stdout

DefaultWriter is the default io.Writer used by httplog for middleware output Logger() and Body(). To support coloring in Windows use:

import "github.com/mattn/go-colorable"
httplog.DefaultWriter = colorable.NewColorableStdout()

Functions

func DefaultLogFormatter

func DefaultLogFormatter(param LogFormatterParams) string

DefaultLogFormatter is the default log format function Logger middleware uses.

func DisableConsoleColor

func DisableConsoleColor()

DisableConsoleColor disables color output in the console.

func ForceConsoleColor

func ForceConsoleColor()

ForceConsoleColor force color output in the console.

func HandlerWithConfig added in v1.3.0

func HandlerWithConfig(conf LoggerConfig, next http.Handler) http.Handler

HandlerWithConfig instance a Logger handler with config and next handler.

func HandlerWithFormatter added in v1.3.0

func HandlerWithFormatter(f LogFormatter, next http.Handler) http.Handler

HandlerWithFormatter instance a Logger handler with the specified log format function and next handler.

func HandlerWithFormatterAndName added in v1.3.0

func HandlerWithFormatterAndName(routerName string, f LogFormatter, next http.Handler) http.Handler

HandlerWithFormatterAndName instance a Logger handler with the specified log format function and next handler.

func HandlerWithName added in v1.3.0

func HandlerWithName(routerName string, next http.Handler) http.Handler

HandlerWithName instance a Logger handler with the specified prefix name and next handler.

func HandlerWithWriter added in v1.3.0

func HandlerWithWriter(out io.Writer, next http.Handler, notlogged ...string) http.Handler

HandlerWithWriter instance a Logger handler with the specified writer buffer and next handler. Example: os.Stdout, a file opened in write mode, a socket...

func Logger

func Logger(next http.Handler) http.Handler

Logger instances a Logger middleware that will write the logs to console. By default, gin.DefaultWriter = os.Stdout.

func LoggerWithConfig

func LoggerWithConfig(conf LoggerConfig) func(next http.Handler) http.Handler

LoggerWithConfig instance a Logger middleware with config.

func LoggerWithFormatter

func LoggerWithFormatter(f LogFormatter) func(next http.Handler) http.Handler

LoggerWithFormatter instance a Logger middleware with the specified log format function.

func LoggerWithFormatterAndName

func LoggerWithFormatterAndName(routerName string, f LogFormatter) func(next http.Handler) http.Handler

LoggerWithFormatterAndName instance a Logger middleware with the specified log format function.

func LoggerWithName

func LoggerWithName(routerName string) func(next http.Handler) http.Handler

LoggerWithName instance a Logger middleware with the specified name prefix.

func LoggerWithWriter

func LoggerWithWriter(out io.Writer, notlogged ...string) func(next http.Handler) http.Handler

LoggerWithWriter instance a Logger middleware with the specified writer buffer. Example: os.Stdout, a file opened in write mode, a socket...

func RemoteIP

func RemoteIP(r *http.Request) string

RemoteIP parses the IP from Request.RemoteAddr, normalizes and returns the IP (without the port).

func RequestBodyLogFormatter added in v1.1.0

func RequestBodyLogFormatter(param LogFormatterParams) string

RequestBodyLogFormatter format function with JSON body output or text

func RequestHeaderLogFormatter added in v1.1.0

func RequestHeaderLogFormatter(param LogFormatterParams) string

HeaderLogFormatter format function with headers output.

func ResponseBodyLogFormatter added in v1.1.0

func ResponseBodyLogFormatter(param LogFormatterParams) string

ResponseBodyLogFormatter format function with JSON body output or text

func ResponseHeaderLogFormatter added in v1.1.0

func ResponseHeaderLogFormatter(param LogFormatterParams) string

ResponseHeaderLogFormatter format function with headers output.

func ShortLogFormatter

func ShortLogFormatter(param LogFormatterParams) string

Types

type FormatterFunction

type FormatterFunction = func(param LogFormatterParams) string

func ChainLogFormatter

func ChainLogFormatter(formatters ...FormatterFunction) FormatterFunction

ChainLogFormatter chain a list of log formatters

type LogFormatter

type LogFormatter func(params LogFormatterParams) string

LogFormatter gives the signature of the formatter function passed to LoggerWithFormatter you can use predefined, like httplog.DefaultLogFormatter or httplog.ShortLogFormatter or you can create your custom

type LogFormatterParams

type LogFormatterParams struct {
	Request *http.Request

	// Router prints router name in the log.
	// If you have more than one router it is useful to get one's name in a console output.
	RouterName string
	// TimeStamp shows the time after the server returns a response.
	TimeStamp time.Time
	// StatusCode is HTTP response code.
	StatusCode int
	// Latency is how much time the server cost to process a certain request.
	Latency time.Duration
	// ClientIP calculated real IP of requester, see Proxy for details.
	ClientIP string
	// Method is the HTTP method given to the request.
	Method string
	// Path is a path the client requests.
	Path string

	// BodySize is the size of the Response Body
	BodySize int
	// Body is a body content, if body is copied
	Body []byte
	// Response header
	ResponseHeader http.Header
	// contains filtered or unexported fields
}

LogFormatterParams is the structure any formatter will be handed when time to log comes

func (*LogFormatterParams) IsOutputColor

func (p *LogFormatterParams) IsOutputColor() bool

IsOutputColor indicates whether can colors be outputted to the log.

func (*LogFormatterParams) MethodColor

func (p *LogFormatterParams) MethodColor() string

MethodColor is the ANSI color for appropriately logging http method to a terminal.

func (*LogFormatterParams) ResetColor

func (p *LogFormatterParams) ResetColor() string

ResetColor resets all escape attributes.

func (*LogFormatterParams) StatusCodeColor

func (p *LogFormatterParams) StatusCodeColor() string

StatusCodeColor is the ANSI color for appropriately logging http status code to a terminal.

type LoggerConfig

type LoggerConfig struct {
	// Optional. Default value is httplog.DefaultLogFormatter
	Formatter LogFormatter

	// Output is a writer where logs are written.
	// Optional. Default value is httplog.DefaultWriter.
	Output io.Writer

	// SkipPaths is an url path array which logs are not written.
	// Could be a regexp like: /user/payment/*
	// Optional.
	SkipPaths []string

	// HideHeader is a header keys array which value should be masked with ****.
	// Optional.
	HideHeaderKeys []string

	// ProxyHandler is a instance of Proxy struct with could get remote IP using proxy data
	// Default is default httplog.NewLogger()
	// If you run you instance on Google App engine or Cloudflare,
	// you need to create explicit Proxy instance with httplog.NewLoggerWithType(...)
	ProxyHandler *Proxy

	// Router prints router name in the log.
	// If you have more than one router it is useful to get one's name in a console output.
	RouterName string

	// CaptureBody saves response body copy for debug  purposes
	CaptureBody bool
}

LoggerConfig defines the config for Logger middleware.

type Proxy

type Proxy struct {
	RemoteIPHeaders []string
	// contains filtered or unexported fields
}

func NewProxy

func NewProxy() *Proxy

NewProxy creates and returns default proxy with default params

func NewProxyWithType

func NewProxyWithType(t ProxyType) *Proxy

NewProxy creates and returns proxy with specific type

func NewProxyWithTypeAndHeaders

func NewProxyWithTypeAndHeaders(t ProxyType, header []string) *Proxy

NewProxy creates and returns proxy with specific type and proxy headers

func (*Proxy) ClientIP

func (p *Proxy) ClientIP(r *http.Request) string

ClientIP implements one best effort algorithm to return the real client IP. It calls c.RemoteIP() under the hood, to check if the remote IP is a trusted proxy or not. If it is it will then try to parse the headers defined in Engine.RemoteIPHeaders (defaulting to [X-Forwarded-For, X-Real-Ip]). If the headers are not syntactically valid OR the remote IP does not correspond to a trusted proxy, the remote IP (coming from Request.RemoteAddr) is returned.

type ProxyType

type ProxyType string

ProxyType is a proxy type we support

func (ProxyType) String

func (pt ProxyType) String() string

type ResponseWriter

type ResponseWriter interface {
	http.ResponseWriter
	http.Hijacker
	http.Flusher
	http.CloseNotifier
	// Status returns the status code of the response or 0 if the response has
	// not been written
	Status() int
	// Written returns whether or not the ResponseWriter has been written.
	Written() bool
	// Size returns the size of the response body.
	Size() int
	// Before allows for a function to be called before the ResponseWriter has been written to. This is
	// useful for setting headers or any other operations that must happen before a response has been written.
	Before(func(ResponseWriter))
	// Optional copy of response body
	// If you need to log or save full response bodies - use it
	// But extra memory and CPU will be used for that
	Body() []byte
	// Manually set Status and Size if you need, written is set as well after call
	Set(status, size int)
}

ResponseWriter is a wrapper around http.ResponseWriter that provides extra information about the response. It is recommended that middleware handlers use this construct to wrap a responsewriter if the functionality calls for it.

func NewResponseWriter

func NewResponseWriter(rw http.ResponseWriter) ResponseWriter

NewResponseWriter creates a ResponseWriter that wraps an http.ResponseWriter

func NewResponseWriterWithBody

func NewResponseWriterWithBody(rw http.ResponseWriter) ResponseWriter

NewResponseWriterWithBody creates a ResponseWriter that wraps an http.ResponseWriter and copy the body of response. The body is not copied if you use ReadFrom for obvious reason

func NewWriter

func NewWriter(rw http.ResponseWriter, captureBody bool) ResponseWriter

NewWriter creates a ResponseWriter that wraps an http.ResponseWriter and optionaly capture body

Directories

Path Synopsis
echolog module
ginlog module
zap module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL