README
¶
audit
An audit trail an application owns: one record format, one write path, an immutable archive, and
projections for security, billing and history. An application emits records through a library; the
writer puts them in an Object-Locked bucket, an indexer follows the bucket into Postgres for search,
a notary seals each hour, and audit verify lets an auditor check the archive with read access to
the bucket and nothing else.
Status: stabilizing. A minor release may carry a breaking change with a Breaking: entry in the
CHANGELOG and a migration page under docs/how-to/upgrade/;
a patch never does (policy decision 0012).
flowchart LR
E["emit library<br/>validates, block or async"] -- "ack = durable" --> R["receiver"]
R --> W["writer"] --> S3[("Object-Locked bucket<br/>THE RECORD")]
O["indexer"] -- "follows by cursor" --> S3
O --> PG[("index, rebuildable")]
N["notary, hourly"] -- "seals" --> S3
Q["query service"] --> PG
Who it is for
An application team that already has, or can provision, a Postgres database, an S3-compatible bucket (with Object Lock for a profile that demands it) and a reference clock. It deliberately does not install a central, multi-tenant audit service, its own message bus, or the Audit page, which lives in the application's own console.
The model
An installation is one deployment in one application's namespace. It writes records, validated against the application's catalogue, through a receiver into the archive (an Object-Locked bucket) and an index (Postgres, rebuildable, not itself evidence) that an indexer writes by following the bucket. A query service reads them back behind declared grants.
Install and a worked example
An installation belongs to one application and runs in that application's namespace (0011). It comes in three shapes, each with a tutorial from nothing to working:
| shape | for | start |
|---|---|---|
| Kubernetes, with the chart | an internal service (direct) or a product (stream) | Getting started on Kubernetes |
| AWS Lambda, with the Pulumi library | writer and notary as functions behind an SQS queue | Getting started on AWS Lambda |
| connected to sluis | audit's half of an access-management install | Getting started with a sluis-connected install |
They write the same archive, under the same catalogue rules and bucket layout, and are verified by the same command. The two combine: writer on Lambda, observe and query in Kubernetes.
The worked example is charts/audit/examples/direct.yaml, rendered as a golden fixture, walked through
in Getting started on Kubernetes.
Artifacts
| artifact | where |
|---|---|
chart audit |
oci://ghcr.io/truvity/charts/audit (chart README) |
images audit-writer, audit-query, audit-observe, audit-notary, audit |
ghcr.io/truvity/audit/<name> |
Lambda zips audit-writer-lambda_<version>_linux_arm64.zip, audit-notary-lambda_<version>_linux_arm64.zip |
the GitHub release, with checksums.txt |
| Pulumi library | github.com/truvity/audit/deploy/pulumi (reference) |
| Go SDK: record, catalogue, emitter, sink | github.com/truvity/audit/sdk |
audit CLI (verify, hold, reindex, migrate, ...) |
the GitHub release archives |
@truvity/audit: query client, sentences, React view |
GitHub Packages, at each release tag |
| JSON Schemas of the record, the catalogue and every configuration file | https://truvity.github.io/audit/schemas/ (schemas/) |
Consumers
| repo | surface |
|---|---|
| estates that deploy it | chart audit, or the Pulumi library for AWS Lambda |
| sluis | emits records; see the sluis page |
Neighbours
sluis (access management), OpenBAO (a relying party and, optionally, the key provider) and this component's own records are one trail: audit is the record each of them writes.
Documentation
docs/ is organised by what you are doing: tutorials in
getting-started/, tasks and runbooks in
how-to/, lookups in reference/, the why in
explanation/ and the decisions. The
changelog says what changed; the steps to take are in how-to/upgrade/.
The rule that makes this repository public
Mechanism only: nothing in this repository may name a real organisation, cluster, account, team, person,
incident or internal ticket (CONTRIBUTING). Every chart value that names one is an
input with a neutral default; the consuming estate supplies the particulars from its own private
repository. hack/leak-canary.sh enforces it, and just check runs it. This repository follows the shared
component contract.
Status
Stabilizing (see above). What is built, designed and run live, platform by platform: capabilities.
What it is not
- Not an application log pipeline. Logs keep flowing through the observability stack. This records what happened, who did it, to what, with what outcome, and keeps it as long as a framework requires.
- Not a SIEM. It exports to one.
- Not a central, multi-tenant audit service. An installation belongs to one application.
Development
Tools come from devbox.json through direnv. just check is the gate and needs nothing but this
checkout. CONTRIBUTING has the rest, and
the repository layout says where things are.
Releasing
A pushed v* tag runs release.yaml, the shared release-public
workflow: it builds the toolchain archives, the Lambda zips and, through ko, the images, and pushes the
audit chart to oci://ghcr.io/truvity/charts/audit at the tag's version.
charts/audit/Chart.yaml's version: 0.0.0 / appVersion: "0.0.0" are
placeholders the release stamps over; never bump them by hand. A breaking change links its migration page
from the CHANGELOG (upgrade pages).
Licence
MIT. See LICENSE.
Documentation
¶
Overview ¶
Package audit carries the files this repository publishes as data: the framework presets, the proto and its generated JSON Schema, and the binaries' configuration schemas. The meta-schemas and the common catalogue are the SDK's (github.com/truvity/audit/sdk).
They are embedded so that a deployment gets them from the binary it already runs, rather than from files it has to ship and keep in step.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ConfigSchemas embed.FS
ConfigSchemas are the schemas of each binary's configuration file, under schemas/config/. They are generated by `just config-schemas` from internal/config and committed, so that a deployment can validate its file without this repository's code. Version 1 of each is kept under schemas/config/v1/, for the one minor in which it is still read.
var Generated embed.FS
Generated holds what this build published for readers outside Go: the record's JSON Schema, with the proto's comments carried as descriptions. The writer puts a copy of it in the archive beside the records it describes.
var Presets embed.FS
Presets are the framework presets under presets/.
var Proto embed.FS
Proto is the schema of record itself. The writer copies the proto of a record's major beside the record's JSON Schema on first use, so the archive keeps the meaning of every field and not only its shape.
Functions ¶
This section is empty.
Types ¶
This section is empty.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package authn is the server side of the SDK's auth package: the JWT authenticator, which verifies a bearer token against the issuers a deployment trusts, and the access-roster grants preset.
|
Package authn is the server side of the SDK's auth package: the JWT authenticator, which verifies a bearer token against the issuers a deployment trusts, and the access-roster grants preset. |
|
cmd
|
|
|
audit
command
Command audit is the tool that holds a deployment to the contracts this repository publishes: it validates catalogues and presets, explains what a profile keeps, and checks that the code and the catalogue still agree.
|
Command audit is the tool that holds a deployment to the contracts this repository publishes: it validates catalogues and presets, explains what a profile keeps, and checks that the code and the catalogue still agree. |
|
audit-notary
command
Command audit-notary seals the archive.
|
Command audit-notary seals the archive. |
|
audit-notary-lambda
command
Command audit-notary-lambda is the notary as an AWS Lambda function invoked hourly by an EventBridge Scheduler schedule.
|
Command audit-notary-lambda is the notary as an AWS Lambda function invoked hourly by an EventBridge Scheduler schedule. |
|
audit-observe
command
Command audit-observe is the indexer: it turns the archive into the search index by following the bucket (docs/decisions/0020).
|
Command audit-observe is the indexer: it turns the archive into the search index by following the bucket (docs/decisions/0020). |
|
audit-query
command
Command audit-query answers questions about what was written.
|
Command audit-query answers questions about what was written. |
|
audit-writer
command
Command audit-writer is an installation's front door and its write path, which are one process in a small installation and two in a busy one.
|
Command audit-writer is an installation's front door and its write path, which are one process in a small installation and two in a busy one. |
|
audit-writer-lambda
command
Command audit-writer-lambda is the write path as an AWS Lambda function behind an SQS event source mapping.
|
Command audit-writer-lambda is the write path as an AWS Lambda function behind an SQS event source mapping. |
|
protoc-gen-audit-jsonschema
command
Command protoc-gen-audit-jsonschema is the buf plugin that writes the record's JSON Schema.
|
Command protoc-gen-audit-jsonschema is the buf plugin that writes the record's JSON Schema. |
|
dedupe
|
|
|
dynamodbdedupe
Package dynamodbdedupe is the writer's record of what it has written, kept in a DynamoDB table, for a writer that has no database: a function platform, where the writer is a Lambda and Postgres is somebody else's service.
|
Package dynamodbdedupe is the writer's record of what it has written, kept in a DynamoDB table, for a writer that has no database: a function platform, where the writer is a Lambda and Postgres is somebody else's service. |
|
deploy
|
|
|
pulumi
module
|
|
|
e2e
|
|
|
fixture
Package fixture stands in for the platform on the local kind box (see truvity/policy's hack/kind/README.md and docs/decisions/0005-kind-is-the-gate.md there): a Postgres database and its two roles, a JetStream stream, an S3 bucket — the things a real deployment's platform would already have provisioned before `helm install audit` ever runs.
|
Package fixture stands in for the platform on the local kind box (see truvity/policy's hack/kind/README.md and docs/decisions/0005-kind-is-the-gate.md there): a Postgres database and its two roles, a JetStream stream, an S3 bucket — the things a real deployment's platform would already have provisioned before `helm install audit` ever runs. |
|
fixture/cmd/resolve
command
Command resolve prints e2e/fixture's names as shell variable assignments, so apply.sh and every hack script that installs onto the same names can `eval` this rather than repeating them.
|
Command resolve prints e2e/fixture's names as shell variable assignments, so apply.sh and every hack script that installs onto the same names can `eval` this rather than repeating them. |
|
examples
|
|
|
emit
command
Command emit is an example application that records what it does into the audit trail.
|
Command emit is an example application that records what it does into the audit trail. |
|
read
command
Command read is an example of reading the audit trail from Go: a search, paged to the end with its cursor, and one record read with where it came from.
|
Command read is an example of reading the audit trail from Go: a search, paged to the end with its cursor, and one record read with where it came from. |
|
hack
|
|
|
goreleaser-snapshot-config
command
Command goreleaser-snapshot-config derives a single-architecture copy of .goreleaser.yaml for the local and CI loops that build straight into the kind box's own registry (see hack/example-snapshot.sh).
|
Command goreleaser-snapshot-config derives a single-architecture copy of .goreleaser.yaml for the local and CI loops that build straight into the kind box's own registry (see hack/example-snapshot.sh). |
|
Package index is the projection a search reads.
|
Package index is the projection a search reads. |
|
indextest
Package indextest is the one corpus every searcher is asked about.
|
Package indextest is the one corpus every searcher is asked about. |
|
postgres
Package postgres is the default index and the shared deduplication store.
|
Package postgres is the default index and the shared deduplication store. |
|
s3scan
Package s3scan answers queries by reading the archive, with no index at all.
|
Package s3scan answers queries by reading the archive, with no index at all. |
|
internal
|
|
|
authtest
Package authtest is an identity provider for tests: a discovery document and a key set served over a real listener, so the code under test runs its actual discovery and key fetch rather than being handed a key.
|
Package authtest is an identity provider for tests: a discovery document and a key set served over a real listener, so the code under test runs its actual discovery and key fetch rather than being handed a key. |
|
bucketcontract
Package bucketcontract checks a bucket against the v1 bucket contract, docs/reference/bucket-contract.md: the key grammar, the envelope of every line, the metadata of every object, its sha256, the hash of every record, and the order keys sort in.
|
Package bucketcontract checks a bucket against the v1 bucket contract, docs/reference/bucket-contract.md: the key grammar, the envelope of every line, the metadata of every object, its sha256, the hash of every record, and the order keys sort in. |
|
buildinfo
Package buildinfo says which build this process is.
|
Package buildinfo says which build this process is. |
|
cli
Package cli is the implementation behind cmd/audit.
|
Package cli is the implementation behind cmd/audit. |
|
clock
Package clock measures how far this machine's clock is from UTC.
|
Package clock measures how far this machine's clock is from UTC. |
|
config
Package config is what each binary of this repository is configured with: one typed configuration per binary, read from one file and validated against a JSON Schema before anything starts.
|
Package config is what each binary of this repository is configured with: one typed configuration per binary, read from one file and validated against a JSON Schema before anything starts. |
|
config/gen
command
Command gen writes the configuration schemas of version 2 into schemas/config/, and the chart's values schema, which embeds them, into charts/audit/.
|
Command gen writes the configuration schemas of version 2 into schemas/config/, and the chart's values schema, which embeds them, into charts/audit/. |
|
config/schema
Package schema builds the JSON Schema of each binary's configuration file.
|
Package schema builds the JSON Schema of each binary's configuration file. |
|
corpus
Package corpus reads the example records in testdata/records, for tests that hold a transport or an encoder to them.
|
Package corpus reads the example records in testdata/records, for tests that hold a transport or an encoder to them. |
|
health
Package health serves a process's readiness: whether it can do its work now, as against /healthz, which says only that the process is up and answering.
|
Package health serves a process's readiness: whether it can do its work now, as against /healthz, which says only that the process is up and answering. |
|
hold
Package hold places and releases legal holds over the archive.
|
Package hold places and releases legal holds over the archive. |
|
identity
Package identity is the way back from a pseudonym to the person, for the cases the law requires and only for them.
|
Package identity is the way back from a pseudonym to the person, for the cases the law requires and only for them. |
|
merkle
Package merkle is the Merkle tree of docs/reference/bucket-contract.md: the Merkle Tree Hash of RFC 6962 section 2.1, over SHA-256, and the audit path that proves one leaf is in it (RFC 9162 section 2.1.3 for the checking).
|
Package merkle is the Merkle tree of docs/reference/bucket-contract.md: the Merkle Tree Hash of RFC 6962 section 2.1, over SHA-256, and the audit path that proves one leaf is in it (RFC 9162 section 2.1.3 for the checking). |
|
observe
Package observe is the third part of an installation: it turns the archive into the search index by following the bucket (docs/decisions/0020).
|
Package observe is the third part of an installation: it turns the archive into the search index by following the bucket (docs/decisions/0020). |
|
observe/observetest
Package observetest is what the cursor indexer is held to, whatever it is run over: the same cases against the in-memory archive, a real S3 and a real Postgres, because the claims are about the listing and the transaction and both are the places a double is kinder than the real thing.
|
Package observetest is what the cursor indexer is held to, whatever it is run over: the same cases against the in-memory archive, a real S3 and a real Postgres, because the claims are about the listing and the transaction and both are the places a double is kinder than the real thing. |
|
pgtest
Package pgtest gives a test its own Postgres schema.
|
Package pgtest gives a test its own Postgres schema. |
|
query
Package query is the read service: it turns a request into a closed query, narrows it to what the caller was granted, asks a searcher, and records that the read happened.
|
Package query is the read service: it turns a request into a closed query, narrows it to what the caller was granted, asks a searcher, and records that the read happened. |
|
recobj
Package recobj is the format of a record object: the one object an ingest batch becomes, as docs/reference/bucket-contract.md specifies it.
|
Package recobj is the format of a record object: the one object an ingest batch becomes, as docs/reference/bucket-contract.md specifies it. |
|
registry
Package registry holds the catalogues a deployment has registered.
|
Package registry holds the catalogues a deployment has registered. |
|
s3test
Package s3test gives a test a real S3 to walk.
|
Package s3test gives a test a real S3 to walk. |
|
schemagen
Package schemagen turns the record's protobuf descriptor into a JSON Schema of the form this project writes.
|
Package schemagen turns the record's protobuf descriptor into a JSON Schema of the form this project writes. |
|
seal
Package seal is the format of the three signed statements of docs/reference/bucket-contract.md — seals, delegations and revocations — the keys that sign them, and how a verifier decides whether to believe one.
|
Package seal is the format of the three signed statements of docs/reference/bucket-contract.md — seals, delegations and revocations — the keys that sign them, and how a verifier decides whether to believe one. |
|
telemetry
Package telemetry is the OpenTelemetry metrics and traces this repository's processes publish.
|
Package telemetry is the OpenTelemetry metrics and traces this repository's processes publish. |
|
ulid
Package ulid makes the identifiers that end a record object's key.
|
Package ulid makes the identifiers that end a record object's key. |
|
writer
Package writer turns records into the copies the profiles keep, and puts those copies where they are kept.
|
Package writer turns records into the copies the profiles keep, and puts those copies where they are kept. |
|
Package keys holds the two kinds of key this system needs and the interfaces a deployment plugs its own behind.
|
Package keys holds the two kinds of key this system needs and the interfaces a deployment plugs its own behind. |
|
Package preset reads the framework presets and composes them into profiles.
|
Package preset reads the framework presets and composes them into profiles. |
|
Package query is the query service as a library: search, facets, get, export, tail and resolve over what the writer wrote, behind the grants a caller holds, with every read recorded in the trail it reads.
|
Package query is the query service as a library: search, facets, get, export, tail and resolve over what the writer wrote, behind the grants a caller holds, with every read recorded in the trail it reads. |
|
sdk
module
|
|
|
sink
|
|
|
natssink
Package natssink publishes records to a NATS JetStream stream, and consumes them again on the other side.
|
Package natssink publishes records to a NATS JetStream stream, and consumes them again on the other side. |
|
sqssink
Package sqssink publishes records to an Amazon SQS queue, and consumes them again on the other side.
|
Package sqssink publishes records to an Amazon SQS queue, and consumes them again on the other side. |
|
Package sinkserver is the receiving side of the sink: the SinkService handler the writer mounts, and the Receiver that checks a caller and a batch before handing it to a Sink.
|
Package sinkserver is the receiving side of the sink: the SinkService handler the writer mounts, and the Receiver that checks a caller and a batch before handing it to a Sink. |
|
Package store is the object store the archive lives in.
|
Package store is the object store the archive lives in. |
|
s3store
Package s3store keeps the archive in an S3 bucket.
|
Package s3store keeps the archive in an S3 bucket. |
|
storetest
Package storetest holds the store a test writes to.
|
Package storetest holds the store a test writes to. |
|
Package wire is how this repository's services speak JSON.
|
Package wire is how this repository's services speak JSON. |
|
Package writer is the audit writer as a library: the thing that takes records, splits them into the copies their profiles keep, pseudonymises, and puts them into the locked archive.
|
Package writer is the audit writer as a library: the thing that takes records, splits them into the copies their profiles keep, pseudonymises, and puts them into the locked archive. |