agent

README

Bearer.sh Go agent

This module provides a pure Go HTTP (HTTPS, HTTP/2) transport decorator for Go Web API clients. It relies on the Bearer.sh platform to provide metrics observation and anomaly detection.

Getting started

Register on https://login.bearer.sh/login to obtain an account and its secret key.

Then, with that key in hand, just start your program code with a simple snippet:

package main

import bearer `github.com/bearer/go-agent`

func main() {
    agent := bearer.New(`app_50-digits-long-secret-key-from-bearer.sh`)
    defer agent.Close()

    // From then on, all your http.Get and similar calls are instrumented.
}

This will provide instrumentation for any clients using the http.DefaultTransport provided by the Go standard library.

Fully configurable version

If your needs go beyond the default Go transport, you can create your own instrumented transport, and use it in your custom-created HTTP clients:

package main

import (
  "net/http"
  "os"
  "time"

  bearer "github.com/bearer/go-agent"
)

func main() {
  // Step 1: initialize Bearer.
  agent := bearer.New(os.Getenv(bearer.SecretKeyName))
  defer agent.Close()

  // Step 2: prepare your custom transport.
  baseTransport := &http.Transport{
    TLSHandshakeTimeout: 5 * time.Second,
  }

  // Step 3: decorate your transport with the Bearer agent.
  transport := agent.Decorate(baseTransport)

  // Step 4: use your client as usual.
  client := http.Client{Transport: transport}
  response, err := client.Get(`https://some.example.com/path`)

  // ...use the API response and error as usual
}

For even more advanced use cases, the API also allows creating multiple configurations and multiple Bearer agents.

Privacy considerations / GDPR

Since logging API calls may involve sensitive data, you may want to configure custom filters to strip PII from the logs, using the SensitiveKeys and SensitiveRegex options on the agent.

Deployment

On a live system, you will likely apply two best practices:

• Take the secret key from the environment. We suggest calling the variable BEARER_SECRET_KEY, for which the SecretKeyName constant is available in the config package.
• For logging
  • either use the default agent logging, which goes to standard error output (12-factor suggests standard output), whence messages can be picked up,
  • or apply a frequent deviation from 12-factor by injecting a logger of your choice to the configuration in config/NewConfig() and, ideally, also injecting it to the default Go logger using log.SetOutput(myLogger) to ensure logs consistency.

Your firewall will need to allow your application to perform outgoing HTTPS/HTTP2 calls to the Bearer platform, at https://config.bearer.sh and https://logs.bearer.sh.

Prerequisites

• For your applications:
  • Go 1.13 or later
  • Go modules enabled
• To contribute to the agent
  • Go stringer command
  • The make command
  • To rebuild the import graph
    • Godepgraph command
    • Graphviz
  • To check syntax and coding standards:
    • Golint
    • Golangci-lint

Contributing

Running the tests

The run the 540+ tests in the package, you can use go test if you wish, or run the preconfigured go test command from the Makefile:

$ make test

(this runs the tests with the race detector)

Run coding style tests

These tests verify that the code base applies best practices: make lint

This should just show the command being run, and display no warnings.

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

• Versions 2.m.p are the current stable versions.
• Versions 0.m.p and 1.m.p use the original PoC code base and are now obsolete.

When preparing to commit a branch, commit all files, then, run go generate agent.go to generate the agent_sha.go file containing the commit SHA, and add a commit with that information, not squashing it. Users will be able to report that SHA to support to enable them to be sure of the version of the agent actually in use.

Credits / Legal

Authors

• Frédéric G. MARAND - Project development - OSInet
• Manfred TOURON - PoC version - Manfred.life
• Bearer.sh - https://www.bearer.sh

License

This project is published under the Apache 2.0 License - see the LICENSE file for details

Acknowledgments

• The events package is very much inspired by the PSR-14 specification, published under the CC-BY-3.0 UNPORTED license for text and MIT License for code.
• The dependency resolution algorithm in Description.resolveHashes takes inspiration from an article by Ferry Boender
• The well-formed invalid credit card numbers used for sensitive data validation were provided by https://www.freeformatter.com/credit-card-number-generator-validator.html

Documentation

Index

• Constants
• Variables
• type Agent
  • func New(secretKey string, opts ...Option) *Agent
  • func (a *Agent) Close() error
  • func (a *Agent) Decorate(rt http.RoundTripper) http.RoundTripper
  • func (a *Agent) DecorateClientTransports(clients ...*http.Client)
  • func (a *Agent) DefaultTransport() http.RoundTripper
  • func (a *Agent) Error() error
  • func (a *Agent) LogError(msg string, fields map[string]interface{})
  • func (a *Agent) LogTrace(msg string, fields map[string]interface{})
  • func (a *Agent) LogWarn(msg string, fields map[string]interface{})
  • func (a *Agent) Logger() *zerolog.Logger
  • func (*Agent) Provider(e events.Event) []events.Listener
  • func (a *Agent) SetLogger(w io.Writer) *Agent
• type Config
  • func NewConfig(secretKey string, transport http.RoundTripper, version string, opts ...Option) (*Config, error)
  • func (c *Config) DataCollectionRules() []*interception.DataCollectionRule
  • func (c *Config) DisableRemote()
  • func (c *Config) Environment() string
  • func (c *Config) IsDisabled() bool
  • func (c *Config) SecretKey() string
  • func (c *Config) SensitiveKeys() []*regexp.Regexp
  • func (c *Config) SensitiveRegexps() []*regexp.Regexp
  • func (c *Config) UpdateFromDescription(description *config.Description)
• type Option
  • func WithDisabled(value bool) Option
  • func WithEndpoints(fetchEndpoint string, reportEndpoint string) Option
  • func WithEnvironment(rtet string) Option
  • func WithLogger(logger io.Writer) Option
  • func WithSensitiveKeys(keys []string) Option
  • func WithSensitiveRegexps(res []string) Option

Constants

const (
	// ExampleWellFormedInvalidKey is a well-formed key known to be invalid. It may
	// be used for integration test scenarios.
	ExampleWellFormedInvalidKey = `app_12345678901234567890123456789012345678901234567890`

	// SecretKeyName is the environment variable used to hold the Bearer secret key,
	// specific to each client. Fetching the secret key from the environment is a
	// best practice in 12-factor application development.
	SecretKeyName = `BEARER_SECRET_KEY`

	// Version is the semantic agent version.
	Version = `1.0.2`
)

Variables

var BearerSHA = "ea6465ac2a5df90ad7fdf8f3ae8af25b6014f0ad"

Types

type Agent

type Agent struct {
	SecretKey string
	// contains filtered or unexported fields
}

Agent is the type of the Bearer entry point for your programs.

func New

func New(secretKey string, opts ...Option) *Agent

New constructs a new Agent and returns it.

In most usage scenarios, you will only use a single Agent in a given application, and pass a config.WithLogger(some *io.Writer) config.Option.

func (*Agent) Close

func (a *Agent) Close() error

Close shuts down the agent

func (*Agent) Decorate

func (a *Agent) Decorate(rt http.RoundTripper) http.RoundTripper

Decorate wraps a http.RoundTripper with Bearer instrumentation.

func (*Agent) DecorateClientTransports

func (a *Agent) DecorateClientTransports(clients ...*http.Client)

DecorateClientTransports wraps the http.RoundTripper transports in all passed clients with Bearer instrumentation.

func (*Agent) DefaultTransport

func (a *Agent) DefaultTransport() http.RoundTripper

DefaultTransport returns the original implementation of the http.DefaultTransport, even if it was overridden by the Agent in the meantime.

func (*Agent) Error

func (a *Agent) Error() error

Error returns any error that has cause the agent to shutdown. If there has been no error then it returns nil

func (*Agent) LogError

func (a *Agent) LogError(msg string, fields map[string]interface{})

LogError logs an error with the specified message and fields.

func (*Agent) LogTrace

func (a *Agent) LogTrace(msg string, fields map[string]interface{})

LogTrace logs a trace-level debug event with the specified message and fields.

func (*Agent) LogWarn

func (a *Agent) LogWarn(msg string, fields map[string]interface{})

LogWarn logs a warning with the specified message and fields.

func (*Agent) Logger

func (a *Agent) Logger() *zerolog.Logger

Logger returns a valid zerolog.Logger instance for the agent.

func (*Agent) Provider

func (*Agent) Provider(e events.Event) []events.Listener

Provider provides the default agent listeners:

• TopicConnect: RFCListener, validating URL under RFC grammars.
• TopicRequest, TopicResponse, TopicBodies: no.

func (*Agent) SetLogger

func (a *Agent) SetLogger(w io.Writer) *Agent

SetLogger changes the logger with a specific zerolog.Logger.

If the writer is a zerolog.Writer, it is used as such, otherwise a new zerolog.Logger is used to wrap it. If the agent has no current config, an empty config will be added to it to carry the logger.

type Config

type Config struct {
	Rules []interface{} // XXX Agent spec defines the field but no use for it.

	ReportEndpoint    string
	ReportOutstanding uint

	*zerolog.Logger
	sync.Mutex
	// contains filtered or unexported fields
}

Config represents the Agent configuration.

func NewConfig

func NewConfig(secretKey string, transport http.RoundTripper, version string, opts ...Option) (*Config, error)

NewConfig is the default Config constructor: it builds a configuration from the builtin agent defaults, the environment, the Bearer platform configuration and any optional Option values passed by the caller.

func (*Config) DataCollectionRules

func (c *Config) DataCollectionRules() []*interception.DataCollectionRule

DataCollectionRules returns the active DataCollectionRule instances.

func (*Config) DisableRemote

func (c *Config) DisableRemote()

DisableRemote stops the goroutine updating the Agent configuration periodically.

func (*Config) Environment

func (c *Config) Environment() string

Environment is a getter for runtimeEnvironmentType.

func (*Config) IsDisabled

func (c *Config) IsDisabled() bool

IsDisabled is a getter for isDisabled, also checking whether the key is plausible.

func (*Config) SecretKey

func (c *Config) SecretKey() string

SecretKey is a getter for secretKey.

func (*Config) SensitiveKeys

func (c *Config) SensitiveKeys() []*regexp.Regexp

SensitiveKeys is a getter for sensitiveKeys.

func (*Config) SensitiveRegexps

func (c *Config) SensitiveRegexps() []*regexp.Regexp

SensitiveRegexps is a getter for sensitiveRegexps.

func (*Config) UpdateFromDescription

func (c *Config) UpdateFromDescription(description *config.Description)

UpdateFromDescription overrides the Config with configuration generated from a configuration Description.

type Option

type Option func(*Config) error

Option is the type use by functional options for configuration.

func WithDisabled

func WithDisabled(value bool) Option

WithDisabled is a functional Option to disable the agent

func WithEndpoints

func WithEndpoints(fetchEndpoint string, reportEndpoint string) Option

WithEndpoints is an undocumented functional Option used for development purposes.

func WithEnvironment

func WithEnvironment(rtet string) Option

WithEnvironment is a functional Option configuring the runtime environment type.

The environment type is a free-form tag for clients, allowing them to report which type of environment they are running in, like "development", "staging" or "production", for reporting in the Bearer UI.

It allows clients to avoid the issues associated with having development and production metrics grouped together although they have different use profiles.

func WithLogger

func WithLogger(logger io.Writer) Option

WithLogger is a functional Option for the logger.

func WithSensitiveKeys

func WithSensitiveKeys(keys []string) Option

WithSensitiveKeys is a functional Option configuring the sensitive regexps.

It will return an error if any key is empty. Duplicate regexps will be reduced to unique values to limit filtering costs.

func WithSensitiveRegexps

func WithSensitiveRegexps(res []string) Option

WithSensitiveRegexps is a functional Option configuring the sensitive regular expressions.

It will cause an error if any of the regular expressions is invalid.