README
¶
SPECTRE
SPECTRE (Service Pair Execution Comparator for Traffic Response Equivalence) is a system for validating service migrations against real traffic. It runs a reference implementation and a candidate implementation in parallel, then compares their externally observable behaviour.
The reference implementation remains authoritative for client responses and side effects. The candidate runs in a network sandbox: SPECTRE intercepts its outbound requests, compares them with the reference implementation's requests, and replays the reference results without allowing the candidate to make real writes.
Semantic differences are captured for analysis and can be fed back into an LLM-assisted migration workflow. A candidate is quarantined when its behaviour diverges from the reference implementation.
SPECTRE is primarily intended for language and framework migrations, but the same approach can help validate dependency upgrades, service consolidation, and other behaviour-sensitive changes.
See the design document for the proposed architecture and safety model.
Try it
The builtin sample service uses the normaliser scripts in internal/sample/scripts/. bit sample generates the sample's descriptor set. Then proctor runs the whole exemplar over unix domain sockets under dist/sockets/:
forecasts, a forecast provider that stands in for a downstream dependency.egress, the Spectre egress proxy between the samples and the forecast provider.referenceandcandidate, two sample backends that fetch forecasts through egress.ingress, the Spectre ingress proxy that mirrors client requests to both backends.
$ bit sample
$ proctor
setup ● exit 0
forecasts │ INF Sample forecast provider listening address=.../dist/sockets/forecasts.sock
forecasts ● ready
egress │ INF Proxy listening address=.../dist/sockets/egress-reference.sock
egress │ INF Proxy listening address=.../dist/sockets/egress-candidate.sock
egress │ INF Proxy listening address=.../dist/sockets/egress-health.sock
egress ● ready
reference │ INF Sample Connect server listening address=.../dist/sockets/reference.sock
candidate │ INF Sample Connect server listening address=.../dist/sockets/candidate.sock
reference ● ready
candidate ● ready
ingress │ INF Proxy listening address=.../dist/sockets/ingress.sock
ingress ● ready
In another terminal issue a gRPC request over the ingress socket. grpcurl -unix needs an absolute socket path:
$ grpcurl -unix -plaintext -d '{}' "$(pwd)/dist/sockets/ingress.sock" spectre.sample.v1.UserService/ListUsers
...
The first terminal should then output something like this:
reference │ INF HTTP request method=POST path=/spectre.sample.v1.UserService/ListUsers status=200 duration=941.25µs
reference │ INF HTTP request method=POST path=/grpc.reflection.v1.ServerReflection/ServerReflectionInfo status=200 duration=3.49ms
candidate │ INF HTTP request method=POST path=/spectre.sample.v1.UserService/ListUsers status=200 duration=1.01725ms
candidate │ INF HTTP request method=POST path=/grpc.reflection.v1.ServerReflection/ServerReflectionInfo status=200 duration=3.748417ms
ingress │ INF HTTP request method=POST path=/spectre.sample.v1.UserService/ListUsers status=200 duration=1.13775ms
ingress │ INF HTTP request method=POST path=/grpc.reflection.v1.ServerReflection/ServerReflectionInfo status=200 duration=4.2135ms
ingress │ DBG Response comparison completed path=/grpc.reflection.v1.ServerReflection/ServerReflectionInfo outcome=skipped reason="gRPC namespace is excluded from response comparison"
ingress │ DBG Response comparison skipped reason="gRPC namespace is excluded from response comparison"
ingress │ DBG Payload normaliser completed kind=field target=spectre.sample.v1.User.roles side=reference payload_path=$.users[0].roles
ingress │ DBG Payload normaliser completed kind=field target=spectre.sample.v1.User.roles side=reference payload_path=$.users[1].roles
ingress │ DBG Payload normaliser completed kind=field target=spectre.sample.v1.User.roles side=reference payload_path=$.users[2].roles
ingress │ DBG Payload normaliser completed kind=field target=spectre.sample.v1.ListUsersResponse.generated_at side=reference payload_path=$.generatedAt
ingress │ DBG Payload normalisation completed message=spectre.sample.v1.ListUsersResponse side=reference normalisers=4
ingress │ DBG Payload normaliser completed kind=field target=spectre.sample.v1.User.roles side=candidate payload_path=$.users[0].roles
ingress │ DBG Payload normaliser completed kind=field target=spectre.sample.v1.User.roles side=candidate payload_path=$.users[1].roles
ingress │ DBG Payload normaliser completed kind=field target=spectre.sample.v1.User.roles side=candidate payload_path=$.users[2].roles
ingress │ DBG Payload normaliser completed kind=field target=spectre.sample.v1.ListUsersResponse.generated_at side=candidate payload_path=$.generatedAt
ingress │ DBG Payload normalisation completed message=spectre.sample.v1.ListUsersResponse side=candidate normalisers=4
ingress │ DBG Response comparison completed path=/spectre.sample.v1.UserService/ListUsers outcome=equivalent
This shows the registered normalisers sorting roles and removing the generation timestamp in each backend's response, then structural comparison succeeding.
The sample also serves raw HTTP JSON endpoints modelled on a legacy weather service. They have no RPC paths, so internal/sample/scripts/weather.js maps each one to a method of the synthetic WeatherService in internal/sample/proto/weather.proto. Issue one with curl:
$ curl --unix-socket dist/sockets/ingress.sock 'http://localhost/v2/forecast?location=london'
{"forecast":{"location":"london", ...}, "alerts":[{"id":"alert-1", ...}, ...]}
Each backend fetches the forecast from the forecast provider through egress. internal/sample/scripts/forecasts.js types that dependency call so egress can match the candidate's call with the reference's. Egress forwards only the reference's call to the provider, then replays the recorded response to the candidate. The first terminal should show one provider request, two egress requests with their normalisation, the candidate's request matching the reference's, and the alerts being sorted before comparison. No normalisers are registered for the forecast request, so egress hashes it unchanged:
forecasts │ INF HTTP request method=GET path=/v1/forecasts/london status=200 duration=850.416µs
egress │ INF HTTP request method=GET path=/v1/forecasts/london status=200 duration=4.520709ms
egress │ DBG Payload normalisation completed message=spectre.sample.v1.FetchForecastRequest side=reference normalisers=0
egress │ DBG Payload normalisation completed message=spectre.sample.v1.FetchForecastRequest side=candidate normalisers=0
egress │ DBG Candidate request matched a reference request host=forecasts.example path=/v1/forecasts/london
egress │ INF HTTP request method=GET path=/v1/forecasts/london status=200 duration=5.972042ms
reference │ INF HTTP request method=GET path=/v2/forecast status=200 duration=2.718583ms
candidate │ INF HTTP request method=GET path=/v2/forecast status=200 duration=2.736917ms
ingress │ INF HTTP request method=GET path=/v2/forecast status=200 duration=3.01225ms
ingress │ DBG Payload normaliser completed kind=field target=spectre.sample.v1.GetForecastV2Response.alerts side=reference payload_path=$.alerts
ingress │ DBG Payload normalisation completed message=spectre.sample.v1.GetForecastV2Response side=reference normalisers=1
ingress │ DBG Payload normaliser completed kind=field target=spectre.sample.v1.GetForecastV2Response.alerts side=candidate payload_path=$.alerts
ingress │ DBG Payload normalisation completed message=spectre.sample.v1.GetForecastV2Response side=candidate normalisers=1
ingress │ DBG Response comparison completed path=/v2/forecast outcome=equivalent
If the candidate makes a dependency call that the reference did not, egress answers it with a 502 and stops serving the candidate. The candidate's responses then diverge, and ingress quarantines it.
/api/v1/forecast?location=sydney and /_status are also available.
Documentation
¶
Overview ¶
Package spectre is the public API: a custom binary calls Main with options that add extensions, and inherits every built-in command.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Option ¶
type Option func(registered *extensions)
Option registers an extension with Main.
func WithCommand ¶
WithCommand adds a top-level command. command must be a pointer to a struct, because Kong parses flags into its fields.
func WithKongOptions ¶ added in v0.28.0
WithKongOptions passes options to the parser after the built-in ones, so kong.Vars adds to the built-in variables, such as version.
func WithTemplateRepo ¶ added in v0.38.0
WithTemplateRepo sets the default repository spectre init fetches templates from: a git source such as git+ssh://github.com/org/repo, or a local directory.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
spectre
command
Command spectre runs the supervisor with its embedded proxies and generates the TypeScript schema declarations used for comparison.
|
Command spectre runs the supervisor with its embedded proxies and generates the TypeScript schema declarations used for comparison. |
|
spectre-sample
command
|
|
|
spectre-sample-forecasts
command
|
|
|
cli
Package cli is the built-in command line: its commands and the terminal output they write.
|
Package cli is the built-in command line: its commands and the terminal output they write. |
|
cli/terminal
Package terminal styles command output for terminals.
|
Package terminal styles command output for terminals. |
|
comparison
Package comparison normalises payloads with scripts, then diffs or hashes them.
|
Package comparison normalises payloads with scripts, then diffs or hashes them. |
|
comparison/javascript
Package javascript evaluates Spectre payload normaliser scripts.
|
Package javascript evaluates Spectre payload normaliser scripts. |
|
descriptors
Package descriptors loads the protobuf descriptors that gRPC and Connect wire decoding uses.
|
Package descriptors loads the protobuf descriptors that gRPC and Connect wire decoding uses. |
|
future
Package future provides a value that one goroutine sets once and others wait for.
|
Package future provides a value that one goroutine sets once and others wait for. |
|
httpserver
Package httpserver serves HTTP ingress and egress points for the protocol engine.
|
Package httpserver serves HTTP ingress and egress points for the protocol engine. |
|
httpserver/codec
Package codec decodes gRPC, Connect, and bare protobuf payloads to JSON.
|
Package codec decodes gRPC, Connect, and bare protobuf payloads to JSON. |
|
httpserver/message
Package message decodes HTTP requests and responses into engine messages.
|
Package message decodes HTTP requests and responses into engine messages. |
|
httpserver/middleware/health
Package health provides HTTP liveness and readiness middleware.
|
Package health provides HTTP liveness and readiness middleware. |
|
httpserver/middleware/logging
Package logging provides structured HTTP request logging middleware.
|
Package logging provides structured HTTP request logging middleware. |
|
httpserver/route
Package route matches HTTP requests against net/http.ServeMux patterns.
|
Package route matches HTTP requests against net/http.ServeMux patterns. |
|
jsonschema
Package jsonschema generates TypeScript schema declarations from JSON Schema.
|
Package jsonschema generates TypeScript schema declarations from JSON Schema. |
|
logger
Package logger configures structured logging and carries loggers through contexts.
|
Package logger configures structured logging and carries loggers through contexts. |
|
model
Package model is SPECTRE's data model: the traffic that crosses a service's boundary through its proxies, and why a candidate was quarantined.
|
Package model is SPECTRE's data model: the traffic that crosses a service's boundary through its proxies, and why a candidate was quarantined. |
|
netaddr
Package netaddr parses listen and backend addresses and opens or dials them.
|
Package netaddr parses listen and backend addresses and opens or dials them. |
|
protocol
Package protocol pairs the reference and candidate traffic that protocol servers report, compares ingress responses, and replays egress responses.
|
Package protocol pairs the reference and candidate traffic that protocol servers report, compares ingress responses, and replays egress responses. |
|
proxy
Package proxy admits candidate work and budgets shared memory for every protocol.
|
Package proxy admits candidate work and budgets shared memory for every protocol. |
|
sample
Package sample provides a Connect service for ingress experiments.
|
Package sample provides a Connect service for ingress experiments. |
|
scaffold
Package scaffold sets a service up for SPECTRE by rendering a template from a template repository into the spectre directory of its checkout with scaffolder.
|
Package scaffold sets a service up for SPECTRE by rendering a template from a template repository into the spectre directory of its checkout with scaffolder. |
|
schema
Package schema models the payload types that normalisers and protocols share.
|
Package schema models the payload types that normalisers and protocols share. |
|
server
Package server is the SPECTRE server, which collects the quarantines that mirrors report.
|
Package server is the SPECTRE server, which collects the quarantines that mirrors report. |
|
server/client
Package client calls the SPECTRE server.
|
Package client calls the SPECTRE server. |
|
server/storage
Package storage keeps the divergences that mirrors report to the SPECTRE server, either in memory or in Redis.
|
Package storage keeps the divergences that mirrors report to the SPECTRE server, either in memory or in Redis. |
|
state
Package state describes a service's migration state: the modules a migration moves, the IO each performs, and how far each has got.
|
Package state describes a service's migration state: the modules a migration moves, the IO each performs, and how far each has got. |
|
supervisor
Package supervisor runs a candidate and reference service behind the SPECTRE proxies, sandboxing the candidate's egress and splitting host resources with cgroups.
|
Package supervisor runs a candidate and reference service behind the SPECTRE proxies, sandboxing the candidate's egress and splitting host resources with cgroups. |