README
¶
shadow-tool
Shadow testing for Go. Run a new implementation next to the current one on a slice of real traffic, compare the results in the background, and log what differs – without ever changing what the caller gets back.
The typical use case is replacing something risky: a backend service, a query, a whole code path. You keep serving responses from the current implementation while the new one runs "in the shadow" for a configurable percentage of calls. When the two results diverge, the differing fields are logged, and the actual values can be encrypted first so response data doesn't leak into your logs.
Requirements
Go 1.22 or later.
Installation
go get github.com/aaukhatov/shadow-tool
How it works
Compare calls the current flow synchronously and returns its result – always. Then, for the configured percentage of
calls, it runs the new flow in a background goroutine, diffs the two results, and logs the paths of the fields that
differ. A slow, failing, or even panicking shadow flow never affects the main flow: errors and panics are logged, never
propagated.
Both results are normalised through a JSON round-trip before the diff, so you are free to mutate the returned value
immediately and only differences that survive encoding/json are reported - unexported fields and fields tagged
json:"-" are never compared. The shadow flow receives a context derived
with context.WithoutCancel, so it keeps the request's values (trace IDs) but is not canceled together with the
request; it is bounded by a 10-second default timeout so a hung new flow can't hold its concurrency slot forever, and
that timeout can be changed with WithShadowTimeout or removed entirely with WithoutShadowTimeout. At most 100
shadow flows run concurrently by default – sampled calls beyond the cap are skipped, never queued – and the cap is
configurable with WithMaxConcurrentShadows.
Without an encryption service, only the names of the differing fields are logged. If you construct the flow with an encryption service, the old and new values are logged too, encrypted.
Error-path parity is out of scope. If the current flow returns an error, the shadow comparison is skipped
entirely – the new flow is never called, so you can't learn whether it would have failed the same way, failed
differently, or succeeded. If the new flow itself returns an error, that's only logged at Warn level; it is never
compared against the current flow's (successful) result. This library diffs successful outputs on the happy path –
it is not a substitute for monitoring the new flow's own error rate once it's live.
Usage
package main
import (
"context"
"log"
shadowflow "github.com/aaukhatov/shadow-tool"
)
// Payload is the response we want to compare across the two implementations.
type Payload struct {
Id int `json:"id"`
Name string `json:"name"`
Date string `json:"date"`
}
// LegacyBackend is the implementation currently serving traffic.
type LegacyBackend struct{}
func (s *LegacyBackend) GetPayload(ctx context.Context) (*Payload, error) {
return &Payload{Id: 1, Name: "John", Date: "2024-01-01"}, nil
}
// NewBackend is the implementation that should eventually replace it.
type NewBackend struct{}
func (s *NewBackend) GetPayload(ctx context.Context) (*Payload, error) {
return &Payload{Id: 1, Name: "John", Date: "2024-03-01"}, nil
}
func main() {
// Shadow 1% of the calls. In a real application the percentage usually
// comes from configuration, so it can be dialed up gradually.
flow, err := shadowflow.New[Payload]("payload-service", 1)
if err != nil {
log.Fatalf("failed to create the shadow flow: %v", err)
}
legacy := &LegacyBackend{}
candidate := &NewBackend{}
// The caller always gets the legacy result. On sampled calls the new
// backend also runs in the background and differences are logged.
payload, err := flow.Compare(context.Background(), legacy.GetPayload, candidate.GetPayload)
if err != nil {
log.Fatalf("backend call failed: %v", err)
}
log.Printf("got payload: %+v", payload)
// On shutdown, wait for in-flight shadow comparisons to finish so no
// diffs are lost. Stop issuing Compare calls first (e.g. shut down the
// HTTP server), then drain the shadow flows.
flow.Wait()
}
A sampled call with a divergence produces a log line like:
time=2024-03-01T12:00:00.000Z level=INFO msg="differences found" component=shadow-flow instance=payload-service properties=date
For slice-returning flows there is CompareSlices, with the same behavior but flows shaped
func(context.Context) ([]T, error).
Logging
The shadow flow logs through log/slog. By default it uses slog.Default(), so the output lands wherever your
application already sends its logs - the library does not impose a destination or a format of its own. The instance name
is attached as an instance attribute rather than interpolated into the message, so you can filter on it.
Four levels are used:
| Level | Logged |
|---|---|
Debug |
Each sampled call, as the shadow flow starts, and sampled calls skipped because the concurrency cap was reached. Off by default; enable it to confirm sampling is working. |
Info |
The fields that differ, and the encrypted values when an encryption service is configured. |
Warn |
A shadow flow that returned an error or a nil result. |
Error |
A shadow flow that panicked, failures to diff the two responses, and failures to copy the response for comparison. |
To send the output somewhere other than slog.Default(), pass the WithLogger option:
flow, err := shadowflow.New[Payload]("payload-service", 1,
shadowflow.WithLogger(slog.New(slog.NewJSONHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelDebug}))),
)
Backends other than slog work through their slog bridge - zapslog
for zap, or the samber/slog-* family for zerolog, logrus, and others. No
adapter code of your own is needed.
Logging the differing values, encrypted
To see what changed and not just which fields, create the flow with an encryption service:
service := shadowflow.NewNoopEncryptionService()
flow, err := shadowflow.New[Payload]("payload-service", 1, shadowflow.WithEncryptionService(service))
Two implementations ship with the package:
NewNoopEncryptionService()- no encryption at all, it only base64-encodes the values. Fine for local development; don't use it where the logs matter, since base64 is trivially reversible.NewPublicKeyEncryptionService(publicKey)- encrypts with RSA-OAEP (SHA-256) using your*rsa.PublicKey, so only the holder of the private key can read the values. Note that RSA-OAEP caps the message size (about 190 bytes with a 2048-bit key); if a diff is too large to encrypt, the field names are still logged, but the values are dropped rather than logged in plain text.
With an encryption service configured, a divergence produces a log line like:
time=2024-03-01T12:00:00.000Z level=INFO msg="differences found" component=shadow-flow instance=payload-service count=1 encrypted_values="mzHJ..."
NewPublicKeyEncryptionService additionally logs a key_fingerprint attribute - a short, stable identifier for the
configured public key (the first 8 bytes of the SHA-256 hash of its DER encoding) - alongside encrypted_values, so a
log line can be matched to the private key that can decrypt it after the key is rotated:
time=2024-03-01T12:00:00.000Z level=INFO msg="differences found" component=shadow-flow instance=payload-service count=1 encrypted_values="mzHJ..." key_fingerprint=3f9a1c2e7b4d5f60
The differing field paths are not logged in plain text by default: diff paths include map keys, which may themselves
be sensitive (say, a map keyed by e-mail address). The full paths travel inside the encrypted payload. If your responses
carry no sensitive map keys and you want the paths visible for quick triage, opt back in with
shadowflow.WithPlaintextProperties().
You can also implement the one-method EncryptionService interface yourself, for example to use AES-GCM with a key from
your secret manager. Implement the optional KeyFingerprinter interface (KeyFingerprint() string) as well to get the
same key_fingerprint log attribute for your own implementation.
Other options
WithShadowTimeout(d)- cancels the context passed to the shadow flow afterd, instead of the 10-second default.WithoutShadowTimeout()- removes the default timeout, so the shadow flow runs until it returns on its own. A hung new flow then holds its concurrency slot indefinitely, so preferWithShadowTimeoutunless the new flow is already known to be bounded.WithMaxConcurrentShadows(n)- caps the number of shadow flows running at the same time (default 100). Sampled calls beyond the cap are skipped, never queued, so a slow new flow cannot pile up goroutines.WithPlaintextProperties()- logs the differing field paths in plain text next to the encrypted values (see above).
Contributing
Run the tests with the race detector - the whole point of this library is doing work concurrently, so -race is part of
the baseline:
go test -race
With a coverage report:
go test -race -cover
Or an HTML one:
go test -coverprofile=coverage.out && go tool cover -html=coverage.out
Documentation
¶
Overview ¶
Package shadowflow runs a new code path alongside an existing one on a sample of traffic, diffs their results, and logs the differences — optionally encrypting the logged values to avoid leaking sensitive data.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type EncryptionService ¶
EncryptionService encrypts the diff values logged by a ShadowFlow so they don't leak sensitive data in plain text.
Implementations must not embed plainText or key material in the error they return from Encrypt: ShadowFlow logs that error's type on failure, but never trusts its message, since a message like "failed to encrypt %q with key %x" would leak the exact data encryption is meant to protect.
type KeyFingerprinter ¶
type KeyFingerprinter interface {
KeyFingerprint() string
}
KeyFingerprinter is implemented by EncryptionService implementations that have an associated key, so a ShadowFlow can log which key encrypted a value. This lets old log lines be matched to the right private key after key rotation.
type NoopEncryptionService ¶
type NoopEncryptionService struct{}
NoopEncryptionService is a version of the EncryptionService that doesn't perform any encryption, it only encodes the differences as a base64 string as defined in RFC 4648.
func NewNoopEncryptionService ¶
func NewNoopEncryptionService() *NoopEncryptionService
NewNoopEncryptionService creates a NoopEncryptionService.
type Option ¶
type Option func(*config) error
Option configures optional ShadowFlow settings. Pass options to New.
func WithEncryptionService ¶
func WithEncryptionService(encryptionService EncryptionService) Option
WithEncryptionService enables logging of the changed values, encrypted with the given service. Without it only the names of the differing fields are logged.
func WithLogger ¶
WithLogger routes the shadow flow logs to the given logger instead of slog.Default().
func WithMaxConcurrentShadows ¶
WithMaxConcurrentShadows caps the number of shadow flows running at the same time; sampled calls beyond the cap are skipped, never queued, so a slow new flow cannot pile up goroutines. Defaults to 100.
func WithPlaintextProperties ¶
func WithPlaintextProperties() Option
WithPlaintextProperties logs the differing field paths in plain text next to the encrypted values. By default an encryption service suppresses them, because diff paths include map keys, which may themselves be sensitive.
func WithShadowTimeout ¶
WithShadowTimeout bounds each shadow flow call: the context passed to the new flow is cancelled after the given duration. Without it, shadow flows get a default timeout of 10 seconds; use WithoutShadowTimeout to run them unbounded instead.
func WithoutShadowTimeout ¶
func WithoutShadowTimeout() Option
WithoutShadowTimeout disables the default shadow timeout, so the shadow flow runs until it returns on its own. A hung new flow then holds its concurrency slot indefinitely; prefer WithShadowTimeout unless the new flow is already known to be bounded.
type PublicKeyEncryptionService ¶
type PublicKeyEncryptionService struct {
// contains filtered or unexported fields
}
PublicKeyEncryptionService is a struct that represents the EncryptionService for encrypting data using a public key. The encryption process uses SHA-256 as the hash function.
func NewPublicKeyEncryptionService ¶
func NewPublicKeyEncryptionService(publicKey *rsa.PublicKey) (*PublicKeyEncryptionService, error)
NewPublicKeyEncryptionService creates a PublicKeyEncryptionService that encrypts with the given RSA public key. The key must be at least 2048 bits; smaller RSA keys provide inadequate encryption strength.
func (*PublicKeyEncryptionService) Encrypt ¶
func (e *PublicKeyEncryptionService) Encrypt(plainText string) (string, error)
Encrypt encrypts plainText with RSA-OAEP using the configured public key and returns the result base64-encoded.
func (*PublicKeyEncryptionService) KeyFingerprint ¶
func (e *PublicKeyEncryptionService) KeyFingerprint() string
KeyFingerprint returns a short, stable identifier for the configured public key (the first 8 bytes of the SHA-256 hash of its DER encoding), so log lines can be matched to the private key that can decrypt them.
type ShadowFlow ¶
type ShadowFlow[T any] struct { // contains filtered or unexported fields }
ShadowFlow runs a new code path alongside an existing one on a sample of traffic, diffs their results, and logs what changed.
func New ¶
New creates a ShadowFlow for the given instance name, sampling percentage (0-100), and options.
func (*ShadowFlow[T]) Compare ¶
func (s *ShadowFlow[T]) Compare(ctx context.Context, currentFlow, newFlow func(context.Context) (*T, error)) (*T, error)
Compare runs the current flow and, based on a random percentage, may also run the new flow. If the new flow is run, it compares the results of the current and new flows, logs the differences, and optionally encrypts and logs the changed values if an encryption service is provided. It always returns the result of the current flow.
The context is passed to currentFlow as-is. The new flow runs in the background on a context derived with context.WithoutCancel, so it keeps the request's values (trace IDs) but is not cancelled together with the request; it is instead bounded by a default timeout of 10 seconds unless overridden with WithShadowTimeout, or left unbounded with WithoutShadowTimeout.
Both results are normalised through a JSON round-trip before comparison, so the caller may mutate the returned value right away and only differences that survive encoding/json are reported: unexported fields and fields tagged `json:"-"` are never compared.
currentFlow: A function that when called with ctx, returns the result of the current flow. newFlow: A function that when called with ctx, returns the result of the new flow.
If currentFlow returns a nil result with a nil error, the shadow comparison is skipped entirely (newFlow is not called) since there is nothing meaningful to diff against; this is logged at debug level.
Returns: The result of the current flow.
func (*ShadowFlow[T]) CompareSlices ¶
func (s *ShadowFlow[T]) CompareSlices(ctx context.Context, currentFlow, newFlow func(context.Context) ([]T, error)) ([]T, error)
CompareSlices is the slice-returning counterpart to Compare: it runs currentFlow, samples the traffic percentage to decide whether to also run newFlow, and logs the differences between the two slice results.
func (*ShadowFlow[T]) Wait ¶
func (s *ShadowFlow[T]) Wait()
Wait blocks until every in-flight shadow comparison has finished. Call it on graceful shutdown so pending diffs are not lost when the process exits.
Stop issuing Compare calls before calling Wait: sync.WaitGroup requires that an Add starting from a zero counter happens before Wait, so shut down the traffic source (e.g. the HTTP server) first and drain the shadow flows last.
Source Files