ddlambda

README

datadog-lambda-go

Datadog's Lambda Go client library enables distributed tracing between serverful and serverless environments.

Installation

go get github.com/DataDog/datadog-lambda-go

You can set the following environment variables via the AWS CLI or Serverless Framework

DD_API_KEY

Your datadog API key

DD_SITE

Which Datadog site to use. Set this to datadoghq.eu to send your data to the Datadog EU site.

DD_LOG_LEVEL

How much logging datadog-lambda-go should do. Set this to "debug" for extensive logs.

DD_FLUSH_TO_LOG

If your Lambda function powers a performance-critical task (e.g., a consumer-facing API). You can avoid the added latencies of metric-submitting API calls, by setting this value to true. Custom metrics will be submitted asynchronously through CloudWatch Logs and the Datadog log forwarder.

Usage

Datadog needs to be able to read headers from the incoming Lambda event. Wrap your Lambda handler function like so:

package main

import (
  "github.com/aws/aws-lambda-go/lambda"
  "github.com/DataDog/datadog-lambda-go"
)

func main() {
  // Wrap your lambda handler like this
  lambda.Start(ddlambda.WrapHandler(myHandler, nil))
  /* OR with manual configuration options
  lambda.Start(ddlambda.WrapHandler(myHandler, &ddlambda.Config{
    BatchInterval: time.Second * 15
    APIKey: "my-api-key",
  }))
  */
}

func myHandler(ctx context.Context, event MyEvent) (string, error) {
  // ...
}

Custom Metrics

Custom metrics can be submitted using the Metric function. The metrics are submitted as distribution metrics.

IMPORTANT NOTE: If you have already been submitting the same custom metric as non-distribution metric (e.g., gauge, count, or histogram) without using the datadog-lambda-go lib, you MUST pick a new metric name to use for ddlambda.Metric. Otherwise that existing metric will be converted to a distribution metric and the historical data prior to the conversion will be no longer queryable.

ddlambda.Metric(
  "coffee_house.order_value", // Metric name
  12.45, // The value
  "product:latte", "order:online" // Associated tags
)

VPC

If your Lambda function is associated with a VPC, you need to ensure it has access to the public internet.

Distributed Tracing

Distributed tracing allows you to propagate a trace context from a service running on a host to a service running on AWS Lambda, and vice versa, so you can see performance end-to-end. Linking is implemented by injecting Datadog trace context into the HTTP request headers.

Distributed tracing headers are language agnostic, e.g., a trace can be propagated between a Java service running on a host to a Lambda function written in Go.

Because the trace context is propagated through HTTP request headers, the Lambda function needs to be triggered by AWS API Gateway or AWS Application Load Balancer.

To enable this feature, make sure any outbound requests have Datadog's tracing headers.

  req, err := http.NewRequest("GET", "http://example.com/status")
  // Use the same Context object given to your lambda handler.
  // If you don't want to pass the context through your call hierarchy, you can use ddlambda.GetContext()
  ddlambda.AddTraceHeaders(ctx, req)

  client := http.Client{}
  client.Do(req)
}

Sampling

The traces for your Lambda function are converted by Datadog from AWS X-Ray traces. X-Ray needs to sample the traces that the Datadog tracing agent decides to sample, in order to collect as many complete traces as possible. You can create X-Ray sampling rules to ensure requests with header x-datadog-sampling-priority:1 or x-datadog-sampling-priority:2 via API Gateway always get sampled by X-Ray.

These rules can be created using the following AWS CLI command.

aws xray create-sampling-rule --cli-input-json file://datadog-sampling-priority-1.json
aws xray create-sampling-rule --cli-input-json file://datadog-sampling-priority-2.json

The file content for datadog-sampling-priority-1.json:

{
  "SamplingRule": {
    "RuleName": "Datadog-Sampling-Priority-1",
    "ResourceARN": "*",
    "Priority": 9998,
    "FixedRate": 1,
    "ReservoirSize": 100,
    "ServiceName": "*",
    "ServiceType": "AWS::APIGateway::Stage",
    "Host": "*",
    "HTTPMethod": "*",
    "URLPath": "*",
    "Version": 1,
    "Attributes": {
      "x-datadog-sampling-priority": "1"
    }
  }
}

The file content for datadog-sampling-priority-2.json:

{
  "SamplingRule": {
    "RuleName": "Datadog-Sampling-Priority-2",
    "ResourceARN": "*",
    "Priority": 9999,
    "FixedRate": 1,
    "ReservoirSize": 100,
    "ServiceName": "*",
    "ServiceType": "AWS::APIGateway::Stage",
    "Host": "*",
    "HTTPMethod": "*",
    "URLPath": "*",
    "Version": 1,
    "Attributes": {
      "x-datadog-sampling-priority": "2"
    }
  }
}

Non-proxy integration

If your Lambda function is triggered by API Gateway via the non-proxy integration, then you have to set up a mapping template, which passes the Datadog trace context from the incoming HTTP request headers to the Lambda function via the event object.

If your Lambda function is deployed by the Serverless Framework, such a mapping template gets created by default.

Opening Issues

If you encounter a bug with this package, we want to hear about it. Before opening a new issue, search the existing issues to avoid duplicates.

When opening an issue, include the Datadog Lambda Layer version, Python version, and stack trace if available. In addition, include the steps to reproduce when appropriate.

You can also open an issue for a feature request.

Contributing

If you find an issue with this package and have a fix, please feel free to open a pull request following the procedures.

License

Unless explicitly stated otherwise all files in this repository are licensed under the Apache License Version 2.0.

This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2019 Datadog, Inc.

Documentation

Index

• Constants
• func AddTraceHeaders(ctx context.Context, req *http.Request)
• func Distribution(metric string, value float64, tags ...string)
• func GetContext() context.Context
• func GetTraceHeaders(ctx context.Context) map[string]string
• func InvokeDryRun(callback func(ctx context.Context), cfg *Config) (interface{}, error)
• func Metric(metric string, value float64, tags ...string)
• func MetricWithTimestamp(metric string, value float64, timestamp time.Time, tags ...string)
• func WrapHandler(handler interface{}, cfg *Config) interface{}
• type Config

Constants

const (
	// DatadogAPIKeyEnvVar is the environment variable that will be used as an API key by default
	DatadogAPIKeyEnvVar = "DD_API_KEY"
	// DatadogKMSAPIKeyEnvVar is the environment variable that will be sent to KMS for decryption, then used as an API key.
	DatadogKMSAPIKeyEnvVar = "DD_KMS_API_KEY"
	// DatadogSiteEnvVar is the environment variable that will be used as the API host.
	DatadogSiteEnvVar = "DD_SITE"
	// DatadogLogLevelEnvVar is the environment variable that will be used to check the log level.
	// if it equals "debug" everything will be logged.
	DatadogLogLevelEnvVar = "DD_LOG_LEVEL"
	// DatadogShouldUseLogForwarderEnvVar is the environment variable that is used to enable log forwarding of metrics.
	DatadogShouldUseLogForwarderEnvVar = "DD_FLUSH_TO_LOG"
	// DefaultSite to send API messages to.
	DefaultSite = "datadoghq.com"
	// EnhancedMetrics is the environment variable used to enable enhanced metrics
	EnhancedMetrics = "DD_ENHANCED_METRICS"
)

Functions

func AddTraceHeaders

func AddTraceHeaders(ctx context.Context, req *http.Request)

AddTraceHeaders adds DataDog trace headers to a HTTP Request

func Distribution

func Distribution(metric string, value float64, tags ...string)

Distribution sends a distribution metric to DataDog Deprecated: Use Metric method instead

func GetContext

func GetContext() context.Context

GetContext retrieves the last created lambda context. Only use this if you aren't manually passing context through your call hierarchy.

func GetTraceHeaders

func GetTraceHeaders(ctx context.Context) map[string]string

GetTraceHeaders reads a map containing the DataDog trace headers from a context object.

func InvokeDryRun

func InvokeDryRun(callback func(ctx context.Context), cfg *Config) (interface{}, error)

InvokeDryRun is a utility to easily run your lambda for testing

func Metric

func Metric(metric string, value float64, tags ...string)

Metric sends a distribution metric to DataDog

func MetricWithTimestamp

func MetricWithTimestamp(metric string, value float64, timestamp time.Time, tags ...string)

MetricWithTimestamp sends a distribution metric to DataDog with a custom timestamp

func WrapHandler

func WrapHandler(handler interface{}, cfg *Config) interface{}

WrapHandler is used to instrument your lambda functions, reading in context from API Gateway. It returns a modified handler that can be passed directly to the lambda.Start function.

Types

type Config

type Config struct {
	// APIKey is your Datadog API key. This is used for sending metrics.
	APIKey string
	// KMSAPIKey is your Datadog API key, encrypted using the AWS KMS service. This is used for sending metrics.
	KMSAPIKey string
	// ShouldRetryOnFailure is used to turn on retry logic when sending metrics via the API. This can negatively effect the performance of your lambda,
	// and should only be turned on if you can't afford to lose metrics data under poor network conditions.
	ShouldRetryOnFailure bool
	// ShouldUseLogForwarder enabled the log forwarding method for sending metrics to Datadog. This approach requires the user to set up a custom lambda
	// function that forwards metrics from cloudwatch to the Datadog api. This approach doesn't have any impact on the performance of your lambda function.
	ShouldUseLogForwarder bool
	// BatchInterval is the period of time which metrics are grouped together for processing to be sent to the API or written to logs.
	// Any pending metrics are flushed at the end of the lambda.
	BatchInterval time.Duration
	// Site is the host to send metrics to. If empty, this value is read from the 'DD_SITE' environment variable, or if that is empty
	// will default to 'datadoghq.com'.
	Site string

	// DebugLogging will turn on extended debug logging.
	DebugLogging bool
	// EnhancedMetrics enables the reporting of enhanced metrics under `aws.lambda.enhanced*` and adds enhanced metric tags
	EnhancedMetrics bool
}

Config gives options for how ddlambda should behave