httpstats

README

httpstats - Standard datadog/statsd integration for HTTP services.

Status: Production

• httpstats - Standard datadog/statsd integration for HTTP services.
  • Usage
    • HTTP Service
    • HTTP Client
  • Standard Metrics
    • HTTP Service
      • Tags
    • HTTP Client
      • Tags
  • Contributing
    • License
    • Contributing Agreement

This project contains middleware for HTTP services and clients that uses xstats emit detailed runtime metrics for service operations.

Usage

HTTP Service

The middleware exported is a func(http.Handler) http.Handler and should work with virtually any router/mux implementation that supports middleware.

var middleware = httpstats.NewMiddleware(
  httpstats.MiddlewareOptionUDPSender("statsd:8125", 1<<15, 10*time.Second, "myservice."),
)

The middleware uses xstats as the base. A stat client can be extracted using xstats.FromContext or xstats.FromRequest as needed to emit custom stats. The standard set of stats emitted automatically by the middleware are documented below.

Each service installing the middleware should emit stats to a statsd or datadog agent. The agent location and settings are configurable through the httpstats.MiddlewareOptionUDPSender or MiddlewareOptionUDPGlobalRollupSender option. The global rollup option helps prevent several forms of skew that can arise from statsd and datadog aggregation of data and is described in greater detail below.

HTTP Client

In addition to an HTTP middleware, there is also an http.RoundTripper wrapper included that will properly manage spans for outgoing HTTP requests. To apply:

var client = &http.Client{
  Transport: httpstats.NewTransport(
    httpstats.TransportOptionTag("dependency", "some-other-service"),
  )(http.DefaultTransport),
}

Like the service middleware, the client decorator provides a set of default stats that are detailed below. No other configuration of the client decorator is needed as it will assume any options set in the middleware by nature of using the same stat client from the incoming request context.

Standard Metrics

HTTP Service

• service_bytes_received

  A histogram of the number of bytes received in each incoming request. The name for this can be overridden with httpstats.MiddlewareOptionBytesInName.

• service_bytes_returned

  A histogram of the number of bytes sent in response to each incoming request. The name for this can be overridden with httpstats.MiddlewareOptionBytesOutName.

• service_bytes_total

  A histogram of the number of bytes sent or received as part of each incoming request. The name for this can be overridden with httpstats.MiddlewareOptionBytesTotalName.

• service_time

  A timer of the amount of time spend processing a request. The name for this can be overridden with httpstats.MiddlewareOptionRequestTimeName.

Tags

This package uses the datadog extensions to the statsd line protocol to inject metadata for metrics. All HTTP service metrics will be tagged with:

• server_method

  The HTTP method used in the incoming request.

• server_status_code

  The status code returned by the service.

• server_status

  A string representation of the exit status of the request. This will be ok for 2xx range responses, error for other responses, timeout for cases where the request context deadline was exceeded, and cancelled for cases where the request context is explicitly cancelled.

Additional tags may be injected either statically or on a per-request basis using the httpstats.MiddlewareOptionTag and httpstats.MiddlewareOptionRequestTag options respectively.

HTTP Client

• client_request_bytes_received

  A histogram of the number of bytes read from the response body of an outgoing request. This name can be overridden using httpstats.TransportOptionBytesInName.

• client_request_bytes_sent

  A histogram of the number of bytes sent in the body of an outgoing request. This name can be overridden using httpstats.TransportOptionBytesOutName.

• client_request_bytes_total

  A histogram of the total bytes sent and read as part of an outgoing request. This name can be overridden using httpstats.TransportOptionBytesTotalName.

• client_request_time

  A timer of how long it took for the round trip to complete. This name can be overridden using stidestats.TransportOptionRequestTimeName.

  This metric will be tagged with the following:

  • method

    The HTTP method used in the outgoing request.

  • status_code

    The HTTP status code received in the response.

  • status

  A string representation of the exit status of the request. This will be ok for 2xx range responses, error for other responses, timeout for cases where the request context deadline was exceeded, and cancelled for cases where the request context is explicitly cancelled.

• client_got_connection

  A timer of how long it took for the HTTP client to acquire a TCP connection. This name may be overridden using httpstats.TransportOptionGotConnectionName.

  This metric will be tagged with the following:

  • reused

    true or false to indicate whether or not the connection was reused or created new respectively.

  • idle

    true or false to indicate whether or not the connection was pulled from the idle pool.

• client_connection_idle

  A timer of how long a cached connection spend in the idle pool before it was used again. This name may be overridden using httpstats.TransportOptionConnectionIdleName.

• client_dns

  A timer of how long it took to resolve the DNS name associated with the outgoing request. This name may be overridden using httpstats.TransportOptionDNSName.

  This metric will be tagged with the following:

  • coalesced

    true or false to indicate whether or not the resolution was shared by multiple, concurrent lookups for the same address.

  • error

    true or false to indicate whether or not there was an error in resolving DNS.

• client_tls

  A timer of how long it took to complete the TLS handshake after acquiring a TCP connection to the remote host. This name may be overridden using httpstats.TransportOptionTLSName.

  This metric will be tagged with the following:

  • error

    true or false to indicate whether or not there was an error in performing the TLS handshake.

• client_wrote_headers

  A timer of how long it took between getting a connection and writing the request headers to the stream. This measure should closely represent the total amount of time it took for a request to leave the service. This name may be overridden using httpstats.TransportOptionWroteHeadersName.

• client_first_response_byte

  A timer of how long it took between writing the headers to the stream and getting the first byte of a response. This measure should closely represent the amount of time it took the remote service to process and respond with the latency of the network included in the measure. This name may be overridden using httpstats.TransportOptionFirstResponseByteName.

• client_put_idle

  A counter emitted each time a connection is placed into the idle pool. This name may be overridden using httpstats.TransportOptionPutIdleName.

  This metric will be tagged with the following:

  • error

    true or false to indicate whether or not there was an error in performing the TLS handshake.

Tags

Unlike the service middleware, the application of tags is not universal across all stats. Instead, each client stat comes with tags that are meaningful to that specific operation. Each metric above should list the applicable tags rather than enumerating another table here.

Contributing

License

This project is licensed under Apache 2.0. See LICENSE.txt for details.

Contributing Agreement

Atlassian requires signing a contributor's agreement before we can accept a patch. If you are an individual you can fill out the individual CLA. If you are contributing on behalf of your company then please fill out the corporate CLA.

Documentation

Index

• func NewMiddleware(options ...MiddlewareOption) (func(http.Handler) http.Handler, xstats.XStater, error)
• func NewTransport(options ...TransportOption) func(http.RoundTripper) http.RoundTripper
• type Middleware
  • func (m *Middleware) ServeHTTP(w http.ResponseWriter, r *http.Request)
• type MiddlewareOption
  • func MiddlewareOptionBytesInName(name string) MiddlewareOption
  • func MiddlewareOptionBytesOutName(name string) MiddlewareOption
  • func MiddlewareOptionBytesTotalName(name string) MiddlewareOption
  • func MiddlewareOptionRequestTag(tagger func(*http.Request) (string, string)) MiddlewareOption
  • func MiddlewareOptionRequestTimeName(name string) MiddlewareOption
  • func MiddlewareOptionTag(tagName string, tagValue string) MiddlewareOption
  • func MiddlewareOptionUDPGlobalRollupSender(host string, maxPacketSize int, flushInterval time.Duration, prefix string, ...) MiddlewareOption
  • func MiddlewareOptionUDPSender(host string, maxPacketSize int, flushInterval time.Duration, prefix string) MiddlewareOption
• type Transport
  • func (t *Transport) RoundTrip(r *http.Request) (*http.Response, error)
• type TransportOption
  • func TransportOptionBytesInName(name string) TransportOption
  • func TransportOptionBytesOutName(name string) TransportOption
  • func TransportOptionBytesTotalName(name string) TransportOption
  • func TransportOptionConnectionIdleName(name string) TransportOption
  • func TransportOptionDNSName(name string) TransportOption
  • func TransportOptionFirstResponseByteName(name string) TransportOption
  • func TransportOptionGotConnectionName(name string) TransportOption
  • func TransportOptionPutIdleName(name string) TransportOption
  • func TransportOptionRequestTag(tagger func(*http.Request) (string, string)) TransportOption
  • func TransportOptionRequestTimeName(name string) TransportOption
  • func TransportOptionTLSName(name string) TransportOption
  • func TransportOptionTag(tagName string, tagValue string) TransportOption
  • func TransportOptionWroteHeadersName(name string) TransportOption

Functions

func NewMiddleware

func NewMiddleware(options ...MiddlewareOption) (func(http.Handler) http.Handler, xstats.XStater, error)

NewMiddleware configures and constructs a stat emitting HTTP middleware along with a stat client that can be used to generate metrics outside the scope of an HTTP request.

func NewTransport

func NewTransport(options ...TransportOption) func(http.RoundTripper) http.RoundTripper

NewTransport configures and returns an HTTP Transport middleware.

Types

type Middleware

type Middleware struct {
	// contains filtered or unexported fields
}

Middleware is an http.Handler wrapper that instruments HTTP servers with the standard SecDev metrics.

func (*Middleware) ServeHTTP

func (m *Middleware) ServeHTTP(w http.ResponseWriter, r *http.Request)

type MiddlewareOption

type MiddlewareOption func(*Middleware) (*Middleware, error)

MiddlewareOption is used to configure the HTTP server middleware.

func MiddlewareOptionBytesInName

func MiddlewareOptionBytesInName(name string) MiddlewareOption

MiddlewareOptionBytesInName sets the metric name used to identity the number of bytes read from an incoming HTTP request. The default value is service_bytes_received

func MiddlewareOptionBytesOutName

func MiddlewareOptionBytesOutName(name string) MiddlewareOption

MiddlewareOptionBytesOutName sets the metric name used to identify the number of bytes written as the result of a request. The default value is service_bytes_sent

func MiddlewareOptionBytesTotalName

func MiddlewareOptionBytesTotalName(name string) MiddlewareOption

MiddlewareOptionBytesTotalName sets the metric name used to identify the total number of bytes read or written as the result of handling a request. The default value is service_bytes_total

func MiddlewareOptionRequestTag

func MiddlewareOptionRequestTag(tagger func(*http.Request) (string, string)) MiddlewareOption

MiddlewareOptionRequestTag is a function that is run on every incoming request. The resulting key/value pair emitted is added to the stat sender such that all stats emitted during the lifetime of the request will have the annotations.

func MiddlewareOptionRequestTimeName

func MiddlewareOptionRequestTimeName(name string) MiddlewareOption

MiddlewareOptionRequestTimeName sets the metric name used to indentify the duration of handling requests. The default value is service_time.

func MiddlewareOptionTag

func MiddlewareOptionTag(tagName string, tagValue string) MiddlewareOption

MiddlewareOptionTag applies a static key/value pair to all metrics.

func MiddlewareOptionUDPGlobalRollupSender

func MiddlewareOptionUDPGlobalRollupSender(host string, maxPacketSize int, flushInterval time.Duration, prefix string, rollupTags []string) MiddlewareOption

MiddlewareOptionUDPGlobalRollupSender enables datadog style statsd emissions over UDP but specifically for timers and percentiles which might need global aggregation to prevent host outliers from skewing percentiles.

func MiddlewareOptionUDPSender

func MiddlewareOptionUDPSender(host string, maxPacketSize int, flushInterval time.Duration, prefix string) MiddlewareOption

MiddlewareOptionUDPSender enables datadog style statsd emissions over UDP

type Transport

type Transport struct {
	// contains filtered or unexported fields
}

Transport is an http.RoundTripper wrapper that instruments HTTP clients with the standard SecDev metrics.

func (*Transport) RoundTrip

func (t *Transport) RoundTrip(r *http.Request) (*http.Response, error)

RoundTrip instruments the HTTP request/response cycle with metrics.

type TransportOption

type TransportOption func(*Transport) *Transport

TransportOption is used to configure the HTTP transport middleware.

func TransportOptionBytesInName

func TransportOptionBytesInName(name string) TransportOption

TransportOptionBytesInName sets the name of the metric used to track the number of bytes read as part of the HTTP response from a request. The default value is client_request_bytes_received.

func TransportOptionBytesOutName

func TransportOptionBytesOutName(name string) TransportOption

TransportOptionBytesOutName sets the name of the metric used to track the number of bytes written as part of the HTTP request body. The default value is client_request_bytes_sent.

func TransportOptionBytesTotalName

func TransportOptionBytesTotalName(name string) TransportOption

TransportOptionBytesTotalName sets the name of the metric used to track the number of bytes written and read as part of the HTTP request. The default value is client_request_bytes_total.

func TransportOptionConnectionIdleName

func TransportOptionConnectionIdleName(name string) TransportOption

TransportOptionConnectionIdleName sets the name of the metric used to track the duration a TCP connection spent in the idle pool. The default value is client_connection_idle.

func TransportOptionDNSName

func TransportOptionDNSName(name string) TransportOption

TransportOptionDNSName sets the name of the metric used to track the duration of resolving the target DNS name of hte request. The default value is client_dns.

func TransportOptionFirstResponseByteName

func TransportOptionFirstResponseByteName(name string) TransportOption

TransportOptionFirstResponseByteName sets the name of the metric used to track the duration between the time when the request headers are written to the stream and the first byte of the response being read back. The default value is client_first_response_byte.

func TransportOptionGotConnectionName

func TransportOptionGotConnectionName(name string) TransportOption

TransportOptionGotConnectionName sets the name of the metric used to track the duration of the creating/fetching a TCP connection. The default value is client_got_connection.

func TransportOptionPutIdleName

func TransportOptionPutIdleName(name string) TransportOption

TransportOptionPutIdleName sets the name of the metric used to count the number of connections that are placed in the idle pool. The default value is client_put_idle.

func TransportOptionRequestTag

func TransportOptionRequestTag(tagger func(*http.Request) (string, string)) TransportOption

TransportOptionRequestTag is called on each round trip with the request instance. The key/value pair returned by this function will be used to annotate all metrics emitted by the transport middleware.

func TransportOptionRequestTimeName

func TransportOptionRequestTimeName(name string) TransportOption

TransportOptionRequestTimeName sets the name of the metric used to track the duration of the HTTP request. The default value is client_request_time.

func TransportOptionTLSName

func TransportOptionTLSName(name string) TransportOption

TransportOptionTLSName sets the name of the metric used to track the duration of performing the TLS handshake with the remote server. The default value is client_tls.

func TransportOptionTag

func TransportOptionTag(tagName string, tagValue string) TransportOption

TransportOptionTag adds a static key/value annotation to all metrics emitted by the middleware.

func TransportOptionWroteHeadersName

func TransportOptionWroteHeadersName(name string) TransportOption

TransportOptionWroteHeadersName sets the name of the metric used to track the duration between the time to get a TCP connection and when the request headers are written to the stream. The default value is client_wrote_headers.