README
¶
A-Novel infrastructure
OpenTofu definitions of A-Novel's production on Google Cloud.
What this is
This repository is A-Novel's production. Every Google Cloud resource the services run on is declared
here, from the projects and the network to each service's database host and Cloud Run workloads. The
code on master is what runs. A change reaches production only through a pull request.
Each service gets its own root, its own identities and its own database. Shared pieces, such as the network and the trust-zone projects, live in foundation, and the CI identities live in bootstrap.
How a change reaches production
Every change is a pull request. CI plans each root with a read-only identity, and a few kinds of
change need a maintainer's label. Merging deploys the change.
deploy.yaml applies the roots, rolls the database hosts during
their planned downtime, and checks production health. Nobody applies by hand, except bootstrap
during a rebuild from nothing. Ops basics
walks through it.
Disabling deploy.yaml freezes production.
Layout
| Path | Contents |
|---|---|
bootstrap/ |
Management project: state bucket, CI identities, secret containers. |
environments/production/foundation |
Projects, VPC, firewall, database hosts, monitoring, budget. |
environments/production/<service> |
One service: its Cloud Run workloads, identities, backup repository and alerts. |
environments/recovery |
Disposable disaster-recovery host, for manual drills. |
modules/service-runtime |
A service's identities, image repositories and alert channel in one project. |
modules/smoke-check |
A private service's smoke job, and the health check every service shares. |
modules/database-runtime |
A database host's boot configuration and host scripts. |
modules/backup-repository |
A service's pgBackRest repository VM and its backup alerts. |
modules/project-shell |
An empty trust-zone project, for foundation. |
cmd/, internal/, builds/ |
The two programs that run on the VMs: the TLS loader and the restore worker. |
.github/actions/, .github/workflows |
The pipeline that plans, deploys and checks production. |
.github/scripts/ |
The pipeline's rules that can stop or misdeploy production, with their tests. |
tests/backup/ |
Offline backup and restore tests inside the real database images. |
Documentation
- Ops basics: the ideas every runbook relies on.
- Architecture: what runs where in Google Cloud.
- Runbooks: every operation that still needs a human.
- Costs: what production pays for.
Contributing
Start with the developer guide, then CONTRIBUTING.md.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
host-credentials
command
Command host-credentials prepares a service's pinned TLS files using its VM identity.
|
Command host-credentials prepares a service's pinned TLS files using its VM identity. |
|
native-restore
command
Command native-restore runs once on an approved disposable recovery host.
|
Command native-restore runs once on an approved disposable recovery host. |
|
internal
|
|
|
hostcredentials
Package hostcredentials loads a host's pinned pgBackRest TLS files from Secret Manager, checks them, and publishes them before PostgreSQL or the pgBackRest server starts on the host.
|
Package hostcredentials loads a host's pinned pgBackRest TLS files from Secret Manager, checks them, and publishes them before PostgreSQL or the pgBackRest server starts on the host. |
|
recovery
Package recovery restores one exact pgBackRest backup set of an approved database, once, on a disposable recovery host, and can then verify the restored data offline at backup consistency.
|
Package recovery restores one exact pgBackRest backup set of an approved database, once, on a disposable recovery host, and can then verify the restored data offline at backup consistency. |