README
¶
aws-embedded-metrics-golang
A Go library for the AWS CloudWatch Embedded Metric Format (EMF).
EMF lets you publish CloudWatch metrics by writing JSON log lines. CloudWatch Logs reads them and creates the metrics for you, so there are no CloudWatch API calls, credentials or clients in your code, and nothing to mock in tests.
- Simple, chainable API for metrics, dimensions, properties and namespaces
- Lists of values and high-resolution (1 second) metrics
- Output always follows the EMF spec, and is tested against AWS's JSON Schema and fuzzed
- Bad input is skipped and reported, instead of silently breaking the whole log line
- Safe to use from several goroutines at once
- Lambda dimensions and properties added for you
- The library itself only uses the standard library
Forked from prozz/aws-embedded-metrics-golang.
Installation
Requires Go 1.27 or newer.
go get github.com/dangrier/aws-embedded-metrics-golang
Usage
import "github.com/dangrier/aws-embedded-metrics-golang/emf"
emf.New().
Namespace("shop").
Dimension("Service", "checkout").
MetricAs("OrderTotal", 120, emf.Count).
Log()
This writes one line to stdout:
{"OrderTotal":120,"Service":"checkout","_aws":{"Timestamp":1791500000000,"CloudWatchMetrics":[{"Namespace":"shop","Dimensions":[["Service"]],"Metrics":[{"Name":"OrderTotal","Unit":"Count"}]}]}}
In Lambda, stdout already goes to CloudWatch Logs. Elsewhere, use the CloudWatch agent or another log shipper.
There are runnable examples in the package docs.
Metrics
logger := emf.New()
logger.Metric("Requests", 1) // int
logger.MetricFloat("Ratio", 0.75) // float64
logger.MetricAs("Errors", 2, emf.Count) // int with a unit
logger.MetricFloatAs("Latency", 12.5, emf.Milliseconds) // float64 with a unit
logger.Metrics(map[string]int{"Hits": 9, "Misses": 1}) // several at once
logger.MetricsFloatAs(map[string]float64{"P50": 10, "P99": 48}, emf.Milliseconds)
Put and PutValues take options for the unit and resolution:
logger.Put("Latency", 12.5, emf.Unit(emf.Milliseconds))
logger.Put("QueueDepth", 42, emf.Unit(emf.Count), emf.HighResolution())
logger.PutValues("Latency", []float64{12.5, 8.1, 30.2}, emf.Unit(emf.Milliseconds))
Every CloudWatch unit
has a constant, like emf.Seconds, emf.Bytes, emf.Percent and emf.CountSecond.
Lists of values
A metric can have a list of values, such as the latency of each request in a batch. CloudWatch keeps
every value, so statistics like percentiles stay accurate. Use PutValues, or log the same name
more than once and the values are collected:
logger.Put("Latency", 12.5).Put("Latency", 8.1) // logs "Latency":[12.5,8.1]
The same name in two contexts is published in both namespaces, sharing all of its values, even when the contexts are written on separate lines. So a repeated name must use the same unit and resolution everywhere, or the new value is skipped and reported. The spec allows at most 100 values per metric, so longer lists are split across several lines.
High resolution
emf.HighResolution() stores a metric per second instead of per minute, so you can graph and alarm
on it at 1 second intervals. High-resolution metrics cost more.
Dimensions
// Each call adds its own dimension set, so each metric is published once per set.
logger.Dimension("Service", "checkout").Dimension("Region", "ap-southeast-2")
// A set with the same keys as one already added, in any order, isn't added again, but its
// values are updated. This leaves one ["Service"] set, with Service now "cart".
logger.Dimension("Service", "cart")
// One set with both dimensions, so each metric is published once for the combination.
logger.DimensionSet(
emf.NewDimension("Service", "checkout"),
emf.NewDimension("Region", "ap-southeast-2"),
)
A key repeated within one set is written once, with the last value. Repeats are only removed within a context, since each context is a separate directive with its own namespace.
Properties
Properties are logged with the metrics, but don't become metrics or dimensions. They're handy for searching in CloudWatch Logs Insights.
logger.Property("requestId", requestID)
A property shares the log line with dimensions and metrics, so each key can only hold one of them. A property with the key of a metric or dimension is skipped and reported. A dimension or metric added later with a property's key is kept, and replaces the property, which is reported too. A dimension replaces it on every line, and a metric on the lines the metric is written on. The Lambda properties the library adds (see Lambda defaults) can be replaced without a report.
Namespaces and contexts
Metrics go in the aws-embedded-metrics namespace unless you set one. A context adds metrics with
their own namespace and dimensions to the same log line.
logger := emf.New().Namespace("shop").Metric("Orders", 3)
logger.NewContext().
Namespace("payments").
Dimension("Provider", "card").
MetricAs("Declined", 1, emf.Count)
logger.Log()
Each context keeps its own dimension values. Contexts that disagree on a dimension value (the same key with different values) are written on separate lines, so each context's metrics are published under its own value. Contexts that agree still share a line. Properties go on every line.
logger := emf.New()
logger.NewContext().Namespace("payments").Dimension("Provider", "card").Metric("Declined", 1)
logger.NewContext().Namespace("payments").Dimension("Provider", "paypal").Metric("Refunds", 2)
logger.Log() // two lines: one with "Provider":"card", one with "Provider":"paypal"
Logging with defer
Create a new logger for each unit of work, such as a request or a batch, and call Log() once at
the end:
func processOrder(id string) error {
metrics := emf.New().Namespace("api").Dimension("Operation", "ProcessOrder")
defer metrics.Log()
start := time.Now()
// ... process the order ...
metrics.MetricFloatAs("Latency", float64(time.Since(start).Milliseconds()), emf.Milliseconds)
return nil
}
A logger isn't meant to be reused. Each Log() writes every value added since New(), so calling
it again writes the earlier values again. A metric logged once before each of two Log() calls
comes out as "Requests":1, then "Requests":[1,1]. Values keep building up for as long as the
logger lives, and every line keeps the timestamp set when New() was called (or by
WithTimestamp).
In Lambda, create the logger inside the handler, not at package level or in init. A logger shared
across invocations double counts, keeps growing, and stamps every metric with the cold-start time.
Options
emf.New(
emf.WithWriter(os.Stderr), // write somewhere other than stdout
emf.WithTimestamp(time.Now().Add(-time.Hour)), // record metrics for another time
emf.WithoutDimensions(), // leave out the Lambda dimensions and properties
emf.WithLogGroup("my-logs"), // log group for the CloudWatch agent
emf.WithErrorHandler(func(err error) { // hear about skipped input and write errors
log.Println(err)
}),
)
Lambda defaults
When AWS_LAMBDA_FUNCTION_NAME is set, each logger adds:
- a dimension set of
ServiceName(the function name) andServiceType(AWS::Lambda::Function) executionEnvironment,memorySize,functionVersionandlogStreamIdproperties
Use WithoutDimensions() to leave these out. If X-Ray sampled the request, the trace ID is also
added as a traceId property.
Spec compliance and errors
Output always follows the EMF specification. Input that would break it is skipped, and the rest is still logged. This includes NaN or infinite values, invalid names, unknown units, and a name used for both a metric and a dimension.
Some input is reported, but still written:
- Timestamp: CloudWatch only publishes metrics with a timestamp from 14 days in the past to 2 hours in the future. It keeps a line outside that window as a log event, but won't turn it into metrics.
- Size: CloudWatch Logs rejects log lines over 1 MB, usually because of very large properties.
- Replaced property: a dimension or metric with the same key as a property replaces it. See Properties.
Use WithErrorHandler to find out about any of these. The errors wrap emf.ErrInvalid:
emf.New(emf.WithErrorHandler(func(err error) {
if errors.Is(err, emf.ErrInvalid) {
log.Println("metrics:", err)
}
}))
More than 100 metrics or values
A single call to Log() may write more than one line. The spec allows at most 100 metrics per log
event, and 100 values per metric, and CloudWatch drops every metric in an event that has more. So if
you log more than 100 metrics at once (across all contexts that share a line), or a metric with more
than 100 values, they are split across several lines. Contexts that disagree on a dimension value
are also written on separate lines, and each of those is split the same way. Every line keeps the
same timestamp and properties, and each metric keeps its own namespace, dimensions and dimension
values. If your code reads the output (in tests, for example), expect one or more lines per Log().
Concurrency
A Logger and its contexts are safe to use from several goroutines at once. Lines from concurrent
Log() calls are never mixed together. The error handler may be called from several goroutines
too, and it can safely use the logger.
Development
Common tasks use just:
| Command | What it does |
|---|---|
just check |
Build, vet, lint and test, the same as CI |
just test-race |
Tests with the race detector and coverage |
just fuzz [time] |
Fuzz the output against the EMF spec (default 1 minute) |
just bench |
Run the benchmarks |
just bench-compare |
Compare benchmarks with main, and fail on a regression |
just scan |
Scan for known vulnerabilities with Grype and govulncheck |
just fmt |
Format the code |
Contributing
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change. Please make sure to update tests.
License
Directories
¶
| Path | Synopsis |
|---|---|
|
Package emf writes CloudWatch metrics in the AWS Embedded Metric Format (EMF).
|
Package emf writes CloudWatch metrics in the AWS Embedded Metric Format (EMF). |
|
internal
|
|
|
cmd/benchcompare
command
Command benchcompare compares benchmarks between a git ref and the working tree, and fails on a significant performance regression.
|
Command benchcompare compares benchmarks between a git ref and the working tree, and fails on a significant performance regression. |