pgbouncer-aurora-operator

README

pgbouncer-aurora-operator

pgbouncer-aurora-operator watches Aurora PostgreSQL topology and runs one PgBouncer per DB instance (1:1). Applications can connect through fixed Writer/Reader Kubernetes Services, or through per-instance Services as a host-list, while the operator keeps Service membership aligned with the current Aurora roles. Its goal is to reduce the Reader connection skew that PgBouncer pooling introduces and to reflect topology changes such as instance scaling and failover into the connection layer automatically.

Status: v0.1.1 (CRD API v1alpha1). A multi-arch image (linux/amd64, linux/arm64) is published on Quay, manifests ship under deploy/, and Helm charts are available from the Quay OCI chart registry. Helm packaging is versioned and released separately from the core manifest/image release.

Table of Contents

• Why it exists
• What it does
• Limitations and operating assumptions
  • Recommended application settings
  • Application connection guide
• Requirements
• Installation
• Configuration
  • Custom Resource
  • Secret
  • Operator
• Status dashboard
• Zone-aware placement
• Writer and Reader Service membership convergence performance
• Reader Service load balancing
• Development
• Contributing
• License

Why it exists

A typical "PgBouncer in front of Aurora" setup has the following limitations.

1. The Aurora Reader endpoint cannot spread a connection pool. The Reader endpoint only picks an instance at connection time (DNS resolution). A long-lived pool connection, once established, is pinned to that instance — so in a pooled environment load skews onto one particular Reader (Reader connection skew).

2. Topology changes are not reflected automatically. If PgBouncer is pointed directly at a single Aurora instance endpoint to avoid problem #1, it does not follow instances that change through scale-out, replacement, or promotion. To use a newly added Aurora instance, you have to define the corresponding PgBouncer deployment by hand each time.

3. The Aurora cluster endpoint is strongly affected by DNS cache, TTL, and refresh behavior. After failover, even if the cluster endpoint points at the new Writer, stale endpoint/IP data can remain in use longer depending on DNS settings.

This operator watches Aurora topology, updates Writer/Reader Kubernetes Service membership per instance through operator-managed Pod labels, and connects each PgBouncer directly to a single Aurora instance (1:1). With this structure, Reader load is spread across per-instance PgBouncer Pods instead of being pinned to one endpoint, and new instances are joined to the Service for their current role after they are ready.

What it does

flowchart LR
  App[Application Pods]
  WriterSvc[Writer Service]
  ReaderSvc[Reader Service]
  PerSvc[Per-instance Services]
  Pods[PgBouncer Deployments\none per Aurora instance]
  Aurora[(Aurora PostgreSQL Cluster)]
  RDS[AWS RDS API\nDescribeDBClusters + DescribeDBInstances]
  CR[PgBouncerAurora CR]
  Operator[pgbouncer-aurora-operator]
  K8s[Kubernetes API]

  App --> WriterSvc
  App --> ReaderSvc
  WriterSvc --> Pods
  ReaderSvc --> Pods
  PerSvc --> Pods
  Pods --> Aurora

  CR --> Operator
  Operator --> K8s
  Operator --> Aurora
  Operator --> RDS
  Operator --> PerSvc
  Operator --> WriterSvc
  Operator --> ReaderSvc
  Operator --> Pods

Key features

• Automatic Aurora topology sync — detects instance add/remove and Writer failover, and reflects them into Writer/Reader Service membership.
• Per-instance 1:1 PgBouncer — resolves Reader connection skew.
• Fast Writer failover detection — once trusted Discovery confirms the new Writer and its PgBouncer Pod is Ready, the Writer Service membership is switched first without waiting for the next Monitor success (fast path).
• Zone-aware placement — places each PgBouncer Pod on a node in the same AZ as its Aurora instance, preferred or required.
• Safety mechanisms — last-known-good retention and freeze keep the last verified healthy traffic state when observations are uncertain.
• Built-in status dashboard — the operator serves a /status web UI and /status.json that show managed CRs, topology, Writer/Reader membership, and conditions at a glance (Status dashboard).

How it works

• Discovery

  • Builds Aurora Writer/Reader cluster endpoints from spec.discovery.clusterName, spec.discovery.domainName, and spec.discovery.port, then queries aurora_replica_status() through the Writer cluster endpoint by default.
  • If the Writer cluster endpoint is briefly unavailable during failover, it immediately retries with the Reader cluster endpoint built in the same discovery tick.
  • Determines role from session_id (MASTER_SESSION_ID = Writer).
  • Instance endpoints are generated deterministically as {instanceName}.{domainName}.
  • RDS metadata is used only for AZ data needed by zone-aware placement and DbiResourceId (physical instance identity) enrichment/debugging. It is not used as the topology decision source or as a Service membership-removal trigger.
  • RDS API failures or timeouts do not affect normal Discovery. RDS metadata is currently referenced only for zone-aware placement and physical identity enrichment/debugging.

• Rendering

  • Creates one ConfigMap, Deployment, and Service per discovered Aurora DB instance.
  • By default, renders a PgBouncer [databases] wildcard route to the corresponding Aurora instance. You can add or override database entries, but each entry's host/port is still managed per instance by the operator.
  • Creates role Services for Writer/Reader traffic.
  • Role Services use operator-managed Pod labels as the Kubernetes Service selector.

• Monitoring

  • Checks PgBouncer Pod readiness first.
  • Always connects directly to the backend DB and checks role/health via pg_is_in_recovery() and transaction_read_only.
  • Applies failure/recovery thresholds before changing healthy Service membership.
  • During Writer failover, Discovery is the role decision source. Monitor is treated as a sanity check rather than the primary gate.

• Topology handling

  • New Aurora instances get new PgBouncer resources.
  • Removed Aurora instances are first dropped from the role Services, and after retention the per-instance resources are deleted.
  • Zone-aware scheduling can prefer (Preferred) or require (Required) the same AZ as the Aurora instance.

Limitations and operating assumptions

This operator's job is to detect topology and update connection paths. Its responsibility boundary is stated clearly here.

Maintaining high availability (HA)

PgBouncer is a lightweight, high-performance connection pooler, not a high-availability proxy that preserves connections without interruption. This operator does not change PgBouncer's role. Instead, it detects Aurora role changes quickly and updates Service membership so applications can keep using fixed Writer/Reader Service addresses.

• Aurora failover behavior: when the Writer changes, existing Writer connections can be dropped. Any in-flight transaction on that connection can fail, and no proxy layer can completely remove that behavior.
• Writer change cleanup: during Aurora failover, the PgBouncer Deployments for the old and new Writer instances are restarted. This drops stale app→PgBouncer and PgBouncer→Aurora server connections and nudges the application/pool to reconnect through the new Writer path.
• PgBouncer pod redundancy: use at least two PgBouncer replicas per Aurora instance. With graceful/rolling restarts, this keeps PgBouncer Pods available while old connections are drained and replaced.
• Reader fallback for availability: when no Reader is available, readerEmptyFallback can temporarily include the Writer in the Reader Service. If the application uses role checks, set Reader connections to targetServerType=preferSecondary or targetServerType=any. This is especially useful for two-member Aurora clusters.

If you need strong zero-downtime HA at the level of preserving connections, consider a layer designed for that purpose, such as Pgpool-II or PgCat.

What the application is responsible for

Detecting disconnects, reconnecting, retrying, and deciding which role (Writer/Reader) to use a connection for are the application's responsibility and out of scope for this project (Non-goals: HA, zero-downtime failover, in-flight transaction recovery, query-time role selection).

This operator only updates Kubernetes Services and PgBouncer workloads when Aurora topology changes; it does not move already-established app→PgBouncer TCP connections to a different Pod. Existing connections keep using the current PgBouncer process until the application or PgBouncer closes them, or until the Writer-change rolling restart drains/terminates the old Pod. Cleanup timing depends on PgBouncer shutdown behavior and spec.pgbouncer.terminationGracePeriodSeconds (30s by default).

Recommended application settings

The values below are conservative starting examples. Real values depend on the driver, pool framework, workload, and DB/PgBouncer timeout policies. Tune them from production observations.

Item	Recommended
Connection validation / keepalive	Prefer the validation, keepalive, or ping features built into your driver/pool framework. If you split Writer/Reader connections, also consider role checks such as targetServerType.
Periodic idle connection check	Periodically validate idle pooled connections so connections already closed by the DB server or network are detected and handled before application code receives them.
Maximum connection lifetime	Use a finite pool connection lifetime (for example, around 15 minutes). Close and reopen connections after they return to the pool.
Idle timeout	Keep the pool/client idle timeout shorter than server-side idle cleanup such as idle_session_timeout (for example, 10-15 minutes).
Reconnect/retry	Enable short backoff retries on connection-acquisition failures. Enable automatic transaction retries only when the operation is safe and idempotent.
Pool size	Keep the number of backend connections fixed and finite. number of application instances × pool size should stay comfortably below the max_connections equivalent limit of the DB/PgBouncer.
DNS cache TTL	Avoid infinite DNS caching and use a short TTL (for example, no later than about 30 seconds).
Connection/login timeout	Keep login or connection timeouts short so temporary database delays fail fast and the application can recover quickly (for example, 2-3 seconds).

Application connection guide

There are three ways for an application to point at the Services the operator creates. With any of them, the operator reflects failover/scale changes by updating each Service's membership.

Method 1 — two Writer/Reader Services as a host-list + targetServerType (driver role check)

postgresql://example-pg-writer.<namespace>.svc.cluster.local:6432,example-pg-reader.<namespace>.svc.cluster.local:6432/db?targetServerType=primary

• The driver takes these two Services as a host-list and chooses a Service that matches the requested role when opening a new connection. Write connections use primary. For read connections, use preferSecondary/any when readerEmptyFallback.enabled=true, or secondary/read-only when it is false.
• The Service names (<cr-name>-writer, <cr-name>-reader) are fixed and the operator updates membership, so Aurora scale/topology changes are reflected automatically.

Method 2 — split read/write connections to the Reader/Writer Service respectively (simplest)

Point write-only connections at the Writer Service and read-only connections at the Reader Service.

# write-only datasource
postgresql://example-pg-writer.<namespace>.svc.cluster.local:6432/db
# read-only datasource
postgresql://example-pg-reader.<namespace>.svc.cluster.local:6432/db

• This is the most natural setup if the application already splits read/write datasources. It does not require driver host-list or targetServerType support.
• Even when the Writer changes due to failover, the operator updates the Writer Service membership to the new Writer, so the application keeps using the same Service address.

Method 3 — individual instance Services as a host-list

postgresql://example-pg-instance-1.<namespace>.svc.cluster.local:6432,example-pg-instance-2.<namespace>.svc.cluster.local:6432/db?targetServerType=primary

• Listing per-instance Services (<cr-name>-<dbInstanceIdentifier>) directly makes the driver iterate over instances to find a valid connection (the same pattern as the AWS docs' "individual DB instance nodes").
• ⚠️ Aurora scale-out/topology changes are not reflected automatically. When a new instance appears, you must manually update the host-list in the connection string. To get the operator's automatic-reflection benefit, use Method 1 or 2.

The port follows the PgBouncer listen_port (default 6432). Connect to the port PgBouncer listens on, not the Aurora port (5432).

Reference: AWS fast failover best practices

Driver-side fast failover settings align with the recommendations in the AWS docs: Best practices with Amazon Aurora PostgreSQL fast failover. Key recommended values (examples):

• Aggressive timeouts: loginTimeout=2, connectTimeout=2, cancelSignalTimeout=1, with a low socketTimeout for short queries (split long queries into separate connections).
• tcpKeepAlive=true, and loadBalanceHosts=true when using a host-list.
• DNS TTL under 30 seconds (JVM: networkaddress.cache.ttl=1, networkaddress.cache.negative.ttl=3).
• Specify the target with targetServerType (primary/secondary/preferSecondary/any).

The AWS docs recommend "listing individual instance nodes in a host-list" for the best failover, while noting its downside: "requires manual update on topology change." Methods 1 and 2 above (pointing at the Writer/Reader Services) automate exactly that manual update via the operator, so the application only needs to point at the Writer/Reader Services.

Requirements

Runtime environment

Component	Version / requirement	Notes
Kubernetes	>= 1.27 recommended	Uses apiextensions.k8s.io/v1, apps/v1, rbac.authorization.k8s.io/v1, and standard Service/Pod APIs. Lower versions may work but are not yet targeted.
Aurora PostgreSQL	Aurora PostgreSQL with aurora_replica_status()	Discovery depends on select server_id, session_id from aurora_replica_status().
PgBouncer	1.25.2 verified	Verified with PgBouncer 1.25.2; provide your own image (see spec.pgbouncer.image). Other versions may work if the config options are compatible.
AWS IAM	rds:DescribeDBClusters, rds:DescribeDBInstances	Needed only for RDS metadata used by zone-aware placement and DbiResourceId enrichment/debugging. RDS API failures or timeouts do not affect normal Discovery. EKS IRSA recommended.
Network	Pod→Aurora connectivity	The operator and PgBouncer Pods must be able to reach the Aurora endpoints/ports.

Build / test tools

Tool	Version / requirement	Notes
Go	1.26	Declared in go.mod and the Docker build image.
Docker Buildx	Recent Docker Desktop / BuildKit	Needed for local and multi-arch image builds.
kubectl	Compatible with the target cluster	Used for manifest dry-run, install, and smoke-test scripts.
Bash	A Bash-capable POSIX shell	Needed for hack/*.sh.

Installation

Use either Manifest or Helm charts to install the published operator runtime. To build and run your own image instead, see Build from source.

Pin the explicit version tag shown below. See Requirements for the cluster, Aurora, IAM, and network prerequisites.

1-A. Install the operator with Manifest

# Set your environment
VERSION=v0.1.1
NAMESPACE=pgbouncer-aurora
OPERATOR_IMAGE=quay.io/case-88/pgbouncer-aurora-operator:${VERSION}
MANIFEST_BASE=https://raw.githubusercontent.com/case-88/pgbouncer-aurora-operator/${VERSION}/deploy

kubectl create namespace ${NAMESPACE}
kubectl apply --server-side -f ${MANIFEST_BASE}/crd.yaml
kubectl apply -f ${MANIFEST_BASE}/serviceaccount.yaml -f ${MANIFEST_BASE}/role.yaml -f ${MANIFEST_BASE}/rolebinding.yaml -f ${MANIFEST_BASE}/operator.yaml

# The bundled deploy/operator.yaml ships a placeholder image, so point the
# Deployment at the published Quay image:
kubectl -n ${NAMESPACE} set image deploy/pgbouncer-aurora-operator manager=${OPERATOR_IMAGE}
kubectl -n ${NAMESPACE} rollout status deploy/pgbouncer-aurora-operator

The bundled manifests are namespaced to pgbouncer-aurora; keep NAMESPACE set to that value, or edit the manifests' namespace: fields to install elsewhere.

1-B. Install with Helm charts

The Helm packaging is split into three charts:

Chart	Installs	Notes
pgbouncer-aurora-crds	PgBouncerAurora CRD only	CRD is rendered from templates/ so helm upgrade can update the schema.
pgbouncer-aurora-operator	Operator ServiceAccount, RBAC, Deployment, metrics/status Service, NetworkPolicy	Does not install CRD or CRs.
pgbouncer-aurora	One PgBouncerAurora CR and related Secret/ExternalSecret resources	Can optionally install a dedicated operator through the operator subchart.

Install the CRD chart first:

NAMESPACE=pgbouncer-aurora
HELM_CHART_REPO=oci://quay.io/case-88/charts
HELM_CHART_VERSION=0.1.1

helm upgrade --install pgbouncer-aurora-crds ${HELM_CHART_REPO}/pgbouncer-aurora-crds \
  --version ${HELM_CHART_VERSION} --namespace ${NAMESPACE} --create-namespace

Then install one shared operator for the namespace:

helm upgrade --install pgbouncer-aurora-operator ${HELM_CHART_REPO}/pgbouncer-aurora-operator \
  --version ${HELM_CHART_VERSION} --namespace ${NAMESPACE} --create-namespace

The default topology is shared mode:

crds chart -> one operator chart -> N pgbouncer-aurora instance chart releases

For dedicated mode, set operator.enabled=true on the instance chart. The operator subchart then watches only that CR by default.

When mixing shared and dedicated operators in one namespace, keep the shared operator on an explicit watch.names include list and exclude dedicated CRs. Per-CR Lease claims prevent duplicate writes, but watch.names is still the intended management boundary.

Helm upgrade:

helm upgrade pgbouncer-aurora-crds ${HELM_CHART_REPO}/pgbouncer-aurora-crds \
  --version ${HELM_CHART_VERSION} --namespace ${NAMESPACE}

helm upgrade pgbouncer-aurora-operator ${HELM_CHART_REPO}/pgbouncer-aurora-operator \
  --version ${HELM_CHART_VERSION} --namespace ${NAMESPACE}

The CRD chart sets helm.sh/resource-policy: keep by default. Keep this default unless you deliberately want helm uninstall to delete the CRD and all PgBouncerAurora custom resources.

Reference: with Helm 3.17.0 or later, use --take-ownership only if you need to take ownership of a CRD that was previously installed through a chart crds/ directory.

helm upgrade pgbouncer-aurora-crds ${HELM_CHART_REPO}/pgbouncer-aurora-crds \
  --version ${HELM_CHART_VERSION} --namespace ${NAMESPACE} --take-ownership

2. Configure credentials and a PgBouncerAurora resource

The Secret and the CR carry environment-specific values — DB user/password, userlist.txt, clusterName/domainName, the PgBouncer image, and so on. Use either raw manifests or the pgbouncer-aurora Helm instance chart for this step.

Manifest path:

VERSION=v0.1.1
MANIFEST_BASE=https://raw.githubusercontent.com/case-88/pgbouncer-aurora-operator/${VERSION}/deploy
curl -fsSLo /tmp/pgbouncer-aurora-secrets.yaml ${MANIFEST_BASE}/secrets.yaml
curl -fsSLo /tmp/pgbouncer-aurora.yaml         ${MANIFEST_BASE}/cr.yaml

# Edit both files:
#   secrets.yaml — DB username/password and the userlist.txt entries
#   cr.yaml      — discovery clusterName/domainName/port, pgbouncer.image, etc.
$EDITOR /tmp/pgbouncer-aurora-secrets.yaml /tmp/pgbouncer-aurora.yaml

kubectl -n ${NAMESPACE} apply -f /tmp/pgbouncer-aurora-secrets.yaml
kubectl -n ${NAMESPACE} apply -f /tmp/pgbouncer-aurora.yaml

Helm instance chart path:

# Prepare Secrets first. By default the chart references existing Secrets named
# pgbouncer-operator-db-auth and pgbouncer-userlist; it also supports
# operatorAuth.create/userlist.create and externalSecret.enabled modes.
kubectl -n ${NAMESPACE} apply -f /tmp/pgbouncer-aurora-secrets.yaml

helm upgrade --install example-pg ${HELM_CHART_REPO}/pgbouncer-aurora \
  --version ${HELM_CHART_VERSION} --namespace ${NAMESPACE} \
  --set discovery.clusterName=example-pg \
  --set discovery.domainName=xxxx.ap-northeast-2.rds.amazonaws.com \
  --set pgbouncer.image=<your-registry>/pgbouncer:1.25.2

ExternalSecret mode is also supported. In that case the chart renders ExternalSecret resources and does not require pre-created Kubernetes Secrets:

helm upgrade --install example-pg ${HELM_CHART_REPO}/pgbouncer-aurora \
  --version ${HELM_CHART_VERSION} --namespace ${NAMESPACE} \
  --set skipCRDCheck=true \
  --set discovery.clusterName=example-pg \
  --set discovery.domainName=xxxx.ap-northeast-2.rds.amazonaws.com \
  --set pgbouncer.image=<your-registry>/pgbouncer:1.25.2 \
  --set operatorAuth.existingSecretName= \
  --set operatorAuth.name=pgbouncer-operator-db-auth \
  --set operatorAuth.externalSecret.enabled=true \
  --set operatorAuth.externalSecret.apiVersion=external-secrets.io/v1 \
  --set operatorAuth.externalSecret.secretStoreRef.kind=ClusterSecretStore \
  --set operatorAuth.externalSecret.secretStoreRef.name=aws-cluster-secret-store \
  --set operatorAuth.externalSecret.remoteRefs.username.key=dev-eks/db/pgba \
  --set operatorAuth.externalSecret.remoteRefs.username.property=username \
  --set operatorAuth.externalSecret.remoteRefs.password.key=dev-eks/db/pgba \
  --set operatorAuth.externalSecret.remoteRefs.password.property=password \
  --set userlist.existingSecretName= \
  --set userlist.name=pgbouncer-userlist \
  --set userlist.externalSecret.enabled=true \
  --set userlist.externalSecret.apiVersion=external-secrets.io/v1 \
  --set userlist.externalSecret.secretStoreRef.kind=ClusterSecretStore \
  --set userlist.externalSecret.secretStoreRef.name=aws-cluster-secret-store \
  --set userlist.externalSecret.remoteRefs.userlist.key=dev-eks/db/pgba/pg-poc

For userlist, the remote secret value is used as the full userlist.txt content; property is not supported for this mode.

See Configuration for every Secret key and CR field.

3. Verify

kubectl -n ${NAMESPACE} get pgba
kubectl -n ${NAMESPACE} get deploy,svc,pod -l pgbouncer-aurora.io/managed-by=pgbouncer-aurora-operator

# Open the built-in status dashboard
kubectl -n ${NAMESPACE} port-forward deploy/pgbouncer-aurora-operator 8080:8080
# then open http://localhost:8080/status

Optional connection check through the generated Writer/Reader Services:

CR_NAME=example-pg
DB_NAME=postgres
DB_USER=test
PGBOUNCER_PORT=6432

kubectl -n ${NAMESPACE} run pgba-psql-writer --rm -i --restart=Never \
  --image=postgres:16 --env PGPASSWORD='<password>' -- \
  psql "host=${CR_NAME}-writer.${NAMESPACE}.svc.cluster.local port=${PGBOUNCER_PORT} dbname=${DB_NAME} user=${DB_USER} sslmode=disable" \
  -c "select inet_server_addr(), pg_is_in_recovery(), current_setting('transaction_read_only')"

kubectl -n ${NAMESPACE} run pgba-psql-reader --rm -i --restart=Never \
  --image=postgres:16 --env PGPASSWORD='<password>' -- \
  psql "host=${CR_NAME}-reader.${NAMESPACE}.svc.cluster.local port=${PGBOUNCER_PORT} dbname=${DB_NAME} user=${DB_USER} sslmode=disable" \
  -c "select inet_server_addr(), pg_is_in_recovery(), current_setting('transaction_read_only')"

The Writer Service should return pg_is_in_recovery = false and transaction_read_only = off. The Reader Service normally returns a read-only backend; when readerEmptyFallback.enabled=true and no Reader is available, it can temporarily return the Writer.

Configuration

An operator deployment is made up of five parts — the Custom Resource that defines what is managed, the Secret that holds DB/PgBouncer credentials, the Operator manager process, and the Role Binding / Service Account that grant permissions. The full manifests are under deploy/.

Custom Resource

One PgBouncerAurora CR manages one Aurora cluster. The minimal form is below; the full example is in deploy/cr.yaml.

apiVersion: pgbouncer-aurora.io/v1alpha1
kind: PgBouncerAurora
metadata:
  name: example-pg
spec:
  discovery:
    clusterName: example-pg
    domainName: xxxx.ap-northeast-2.rds.amazonaws.com
    port: 5432
    authSecretRef:
      name: pgbouncer-operator-db-auth
  pgbouncer:
    image: <your-registry>/pgbouncer:1.25.2
    authFileSecretRef:
      name: pgbouncer-userlist

CRD names

Item	Value
API group	pgbouncer-aurora.io
Version	v1alpha1
Kind	PgBouncerAurora
Plural	pgbouncerauroras
Short name	pgba
Scope	Namespaced

spec.discovery

Option	Required	Default	Unit	Description
spec.discovery.clusterName	Yes	none	string	Aurora DB cluster identifier prefix used to build cluster endpoints. e.g. example-pg.
spec.discovery.domainName	Yes	none	DNS suffix	Common Aurora endpoint suffix. e.g. xxxx.ap-northeast-2.rds.amazonaws.com.
spec.discovery.port	No	5432	TCP port	Port for the generated Writer/Reader/Instance endpoints.
spec.discovery.clusterEndpoints.writer.host	Advanced	generated	DNS name	Explicit Writer endpoint override.
spec.discovery.clusterEndpoints.writer.port	Advanced	spec.discovery.port	TCP port	Explicit Writer endpoint port override.
spec.discovery.clusterEndpoints.reader.host	Advanced	generated	DNS name	Explicit Reader endpoint override.
spec.discovery.clusterEndpoints.reader.port	Advanced	spec.discovery.port	TCP port	Explicit Reader endpoint port override.
spec.discovery.database	No	postgres	database name	Database used for the discovery connection.
spec.discovery.sslMode	No	require	PostgreSQL SSL mode	SSL mode used by Aurora discovery and direct DB monitor probe connections.
spec.discovery.authSecretRef.name	Yes	none	Secret name	Operator DB credential Secret. Used for Aurora discovery and the direct DB monitor probe.
spec.discovery.interval	No	3s	Go duration	Minimum interval for Aurora topology discovery checks. Internal floor 1s.
spec.discovery.timeout	No	3s	Go duration	Timeout for discovery DB queries.
spec.discovery.failureThreshold	No	3	count	Number of consecutive untrusted discoveries allowed before freezing instead of using cached topology.

Endpoint generation rules:

• Writer endpoint: {clusterName}.cluster-{domainName}
• Reader endpoint: {clusterName}.cluster-ro-{domainName}
• Instance endpoint: {dbInstanceIdentifier}.{domainName}

spec.discovery.interval defaults to 3s to follow Aurora failover quickly. This only raises the Aurora topology query frequency. AWS RDS metadata calls for auxiliary AZ/DbiResourceId data are refreshed by the operator process's shared metadata worker. RDS API failures or timeouts do not affect normal Discovery. RDS metadata is currently referenced only for zone-aware placement and physical identity enrichment/debugging, not for traffic membership removal.

The operator does not infer the AWS region from spec.discovery.domainName. RDS metadata lookups use the operator process flag --aws-region, so this region must match the region of the Aurora cluster that domainName points at. For multi-region operation, run a separate operator deployment per region.

spec.monitor

Option	Required	Default	Unit	Description
spec.monitor.interval	No	10s	Go duration	Minimum interval for monitor checks. Internal floor 1s.
spec.monitor.timeout	No	3s	Go duration	Timeout per backend monitor probe.
spec.monitor.failureThreshold	No	3	count	Consecutive failures before treating a healthy backend as unhealthy.
spec.monitor.recoveryThreshold	No	2	count	Consecutive successes before treating an unhealthy backend as healthy.

Monitor uses Pod readiness and an always-on direct DB probe.

• Direct DB probe — connects directly to the Aurora instance to check health/role. The connection DB is spec.discovery.database (default postgres), and the SSL mode follows spec.discovery.sslMode.
  • For sslmode=verify-full, the operator Pod must trust the Aurora CA. With the pgx driver this can be done by mounting the CA bundle into the operator Pod and setting PGSSLROOTCERT to that file path.

TLS from PgBouncer to Aurora can be configured through PgBouncer settings such as server_tls_sslmode and server_tls_ca_file, with the CA file mounted through spec.pgbouncer.volumes and volumeMounts. Client-to-PgBouncer TLS is not managed directly by the operator API; terminate it outside the operator or provide the required PgBouncer files through custom volumes if you choose to manage it yourself.

spec.pgbouncer

Option	Required	Default	Unit	Description
spec.pgbouncer.image	Yes	empty	image reference	Required; no default. This project does not ship a PgBouncer image. Build your own PgBouncer image and host it on a registry your cluster can pull from (e.g. Amazon ECR), or use a third-party hub image.
spec.pgbouncer.replicas	No	1	replicas	Default replica count for the per-instance PgBouncer Deployment.
spec.pgbouncer.terminationGracePeriodSeconds	No	30	seconds	Kubernetes Pod termination grace for PgBouncer Pods. Lower values shorten the upper bound for stale app→PgBouncer connections during Writer-change rollouts, but can cut off graceful drain earlier.
spec.pgbouncer.config	No	{}	PgBouncer config section	Structured PgBouncer config. See below.
spec.pgbouncer.instanceOverrides[]	No	[]	list	Per-Aurora-instance enable/disable, replicas, and config overrides.
spec.pgbouncer.authFileSecretRef.name	Yes	none	Secret name	Secret holding userlist.txt. Mounted at the default auth_file path /etc/pgbouncer/userlist.txt, or at the effective path when global [pgbouncer].auth_file is overridden.
spec.pgbouncer.resources	No	{}	Kubernetes ResourceRequirements	CPU/memory requests/limits for the main PgBouncer container.
spec.pgbouncer.labels	No	{}	map[string]string	Extra labels on the generated PgBouncer Pod template. Operator-managed labels win on conflict.
spec.pgbouncer.annotations	No	{}	map[string]string	Extra annotations on the generated PgBouncer Pod template. Operator-managed annotations win on conflict.
spec.pgbouncer.serviceAccountName	No	empty	ServiceAccount name	ServiceAccount used by the PgBouncer Pod when needed.
spec.pgbouncer.nodeSelector	No	{}	map[string]string	PgBouncer Pod nodeSelector.
spec.pgbouncer.affinity	No	{}	Kubernetes Affinity	node/pod affinity/anti-affinity. Zone-aware affinity is still operator-managed.
spec.pgbouncer.tolerations	No	[]	list	Pod tolerations.
spec.pgbouncer.priorityClassName	No	empty	PriorityClass name	PgBouncer Pod priority class.
spec.pgbouncer.runtimeClassName	No	empty	RuntimeClass name	PgBouncer Pod runtime class.
spec.pgbouncer.podSecurityContext	No	{}	Kubernetes PodSecurityContext	Pod-level security context.
spec.pgbouncer.containerSecurityContext	No	{}	Kubernetes SecurityContext	Security context for the main pgbouncer container.
spec.pgbouncer.livenessProbe	No	config.pgbouncer.listen_port TCP probe	Kubernetes Probe	Main-container liveness probe override. Use with care.
spec.pgbouncer.readinessProbe	No	config.pgbouncer.listen_port TCP probe	Kubernetes Probe	Main-container readiness probe override. Affects Service membership and monitor behavior.
spec.pgbouncer.sidecars	No	[]	container list	Extra sidecar containers.
spec.pgbouncer.volumes	No	[]	volume list	Extra volumes. Operator-managed config/auth-file volumes win on conflict.
spec.pgbouncer.volumeMounts	No	[]	volume mount list	Extra mounts on the main pgbouncer container. Operator-managed mounts win on conflict.
spec.pgbouncer.imagePullSecrets	No	[]	Kubernetes LocalObjectReference[]	Image pull secrets added to the PgBouncer Pod.
spec.pgbouncer.topologySpreadConstraints	No	[]	Kubernetes TopologySpreadConstraint[]	Native Pod topology spread. Use this instead of an operator-specific policy for replica spreading.

The operator intentionally does not expose every Deployment field. Deployment strategy, init containers, lifecycle hooks, and arbitrary main-container command or probe wiring beyond the fields above are not part of the v1alpha1 API.

spec.pgbouncer.config maps directly to PgBouncer ini sections:

Option	Required	Default	Description
spec.pgbouncer.config.pgbouncer	No	{auth_type: md5, auth_file: /etc/pgbouncer/userlist.txt, listen_addr: 0.0.0.0, listen_port: 6432}	Rendered as [pgbouncer]. listen_port, listen_addr, auth_file, and pidfile are global-only settings and cannot be overridden per instance.
spec.pgbouncer.config.databases	No	{"*": {}}	Rendered as [databases]. Each key is a database entry name. host/port are operator-managed per Aurora instance and ignored in both global config and instance overrides.
spec.pgbouncer.config.users	No	{}	Rendered as [users].
spec.pgbouncer.config.peers	No	{}	Rendered as [peers].

Options handled by the operator:

Key	Value	Description
[pgbouncer].listen_addr	spec.pgbouncer.config.pgbouncer.listen_addr or 0.0.0.0	Global config only; not allowed in instance overrides. Set it to an address the Kubernetes Service can reach.
[pgbouncer].listen_port	spec.pgbouncer.config.pgbouncer.listen_port or 6432	Global config only; not allowed in instance overrides. Used by the container port, Service port, and default probes.
[pgbouncer].auth_file	spec.pgbouncer.config.pgbouncer.auth_file or /etc/pgbouncer/userlist.txt	Global config only; not allowed in instance overrides. The authFileSecretRef Secret is automatically mounted at the effective path.
[pgbouncer].pidfile	spec.pgbouncer.config.pgbouncer.pidfile	Global config only; not allowed in instance overrides. Optional PgBouncer key rendered only when set.
[databases].*.host	Aurora instance endpoint	Rendered per instance by the operator. Not overridable in global config or instance overrides.
[databases].*.port	Aurora instance port	Rendered per instance by the operator. Not overridable in global config or instance overrides.

The operator does not validate arbitrary PgBouncer config keys. Operator-handled keys are processed separately; for example, invalid listen_port values fall back to the default port. Ordinary PgBouncer key: value entries are rendered as-is into pgbouncer.ini, so invalid PgBouncer config is expected to fail at PgBouncer startup, not at CR admission time.

[!CAUTION] Changing [pgbouncer].listen_port on an existing CR updates the generated ConfigMaps, Deployments, Pods, probes, and Services. Treat it as an operationally sensitive change in production.

spec.pgbouncer.instanceOverrides[]

Option	Required	Default	Unit	Description
spec.pgbouncer.instanceOverrides[].name	Yes	none	Aurora DB instance identifier	Matches server_id from aurora_replica_status().
spec.pgbouncer.instanceOverrides[].enabled	No	true	boolean	Set to false to intentionally exclude that discovered instance from PgBouncer resources, monitor probes, and Writer/Reader Service membership. The instance remains visible in status as disabled.
spec.pgbouncer.instanceOverrides[].replicas	No	spec.pgbouncer.replicas	replicas	PgBouncer Deployment replica count for that instance.
spec.pgbouncer.instanceOverrides[].config	No	{}	PgBouncer config section	Deep-merged on top of spec.pgbouncer.config for that instance. [pgbouncer].listen_port, [pgbouncer].listen_addr, [pgbouncer].auth_file, [pgbouncer].pidfile, [databases].*.host, and [databases].*.port are operator-handled/global-only keys and cannot be overridden per instance.

spec.services

Option	Required	Default	Unit	Description
spec.services.writer.name	No	writer	DNS label suffix	Name suffix of the Writer traffic role Service. The final name is <cr-name>-<name>.
spec.services.writer.type	No	ClusterIP	Kubernetes Service type	Writer role Service type. One of ClusterIP/NodePort/LoadBalancer.
spec.services.writer.annotations	No	{}	map[string]string	Writer role Service annotations.
spec.services.reader.name	No	reader	DNS label suffix	Name suffix of the Reader traffic role Service. The final name is <cr-name>-<name>.
spec.services.reader.type	No	ClusterIP	Kubernetes Service type	Reader role Service type. One of ClusterIP/NodePort/LoadBalancer.
spec.services.reader.annotations	No	{}	map[string]string	Reader role Service annotations.
spec.services.perInstances.type	No	ClusterIP	Kubernetes Service type	Type of every per-instance PgBouncer Service. One of ClusterIP/NodePort/LoadBalancer.
spec.services.perInstances.annotations	No	{}	map[string]string	Annotations for each per-instance Service.

Because direct per-instance targeting depends on per-instance Services, a per-instance Service is always created for discovered instances.

spec.topologyPolicy

Option	Required	Default	Unit	Description
spec.topologyPolicy.removeAfterMissingCount	No	3	discovery observations	Number of consecutive trusted discovery cycles an instance must be missing before becoming a removal candidate from retained membership.
spec.topologyPolicy.removedInstanceRetention	No	1h	Go duration	How long to retain a removed instance's ConfigMap/Deployment/Service before deletion.
spec.topologyPolicy.readerEmptyFallback.enabled	No	true	boolean	Whether to temporarily include the Writer in the Reader Service when no Reader is confirmed.
spec.topologyPolicy.zoneAware.enabled	No	true	boolean	Whether to add node affinity based on the Aurora instance AZ.
spec.topologyPolicy.zoneAware.enforcement	No	Preferred	enum	Preferred adds preferred node affinity, Required adds required node affinity.
spec.topologyPolicy.zoneAware.topologyKey	No	topology.kubernetes.io/zone	node label key	Node label key used for zone-aware scheduling.
spec.topologyPolicy.zoneAware.conflictPolicy	No	Warn	enum	Policy when user scheduling constraints conflict with the zone-aware intent. One of Warn/Fail/Ignore. Warn records a condition and continues, Fail freezes reconcile for safety, Ignore hides the conflict condition.

[!NOTE] When an Aurora cluster has only one Writer and one Reader, and the application uses driver-side role checks, prefer preferSecondary or any targets for Reader connections. During Aurora failover there can be a short period with no available Reader. If the driver strictly requires secondary/read-only, connections can keep failing until the demoted instance returns as a Reader.

Secret

The operator DB auth Secret referenced by spec.discovery.authSecretRef uses the keys below. The same credentials are used for Aurora discovery and the direct DB monitor probe.

Key	Required	Default	Unit	Description
username or user	Yes	none	string	PostgreSQL user used for discovery and monitor probe queries.
password	Yes	none	string	PostgreSQL password.

The PgBouncer auth file Secret referenced by spec.pgbouncer.authFileSecretRef:

Key	Required	Default	Unit	Description
userlist.txt	Yes	none	file content	Mounted at the effective global [pgbouncer].auth_file path (default /etc/pgbouncer/userlist.txt). Uses PgBouncer-compatible password entries.

Operator

The operator is a Deployment that watches a single namespace; the default manifest uses replicas: 2 and enables leader election because this process is part of the database availability path. The default manifest is deploy/operator.yaml, and the watch target is set by --watch-namespace (defaults to the operator Pod's namespace) and --watch-names (defaults to *, all CRs in that namespace).

[!IMPORTANT] Recommended operator deployment

Internal testing confirmed that one operator can manage up to 50 CRs (=Aurora clusters) and 100 backend instances smoothly. It uses up to 10 workers per CR (WORKERS_PER_CR), and core work such as Discovery, Monitor, and Reconcile was processed without issue.

Operator flags

These are manager process flags, not CR options.

Only the public operational flags are listed below. Discovery/Monitor execution is controlled by per-CR single-flight scheduling and does not expose a global worker pool for user tuning. DB probe throughput is controlled by monitor intervals, per-probe timeout, and --workers-per-cr. The AWS API limiter remains intentionally low as a last-resort guard against accidental hot-loop bugs in the shared metadata worker.

Flag	Default	Unit	Description
--metrics-bind-address	:8080	address	Metrics bind address.
--health-probe-bind-address	:8081	address	Health/readiness probe bind address.
--leader-elect	false	boolean	Enable controller-runtime leader election. The default manifest passes this flag.
--leader-election-id	auto-detected Deployment identity, then built-in fallback	string	Leader election Lease name. Replicas of the same operator Deployment must share this value. Different operator Deployments in the same namespace should use different values.
--operator-id	auto-detected Deployment identity, then leader election ID	string	Stable identity written to CR ownership claim Leases. Same-Deployment replicas must share this value.
--operator-claim-ttl	OPERATOR_CLAIM_TTL or 60s	Go duration	Lease duration for per-CR ownership claims. Another operator may take over after this expires.
--operator-claim-renew-interval	OPERATOR_CLAIM_RENEW_INTERVAL or 10s	Go duration	Renew interval for per-CR ownership claim Leases.
--aws-region	""	AWS region	Single AWS region for RDS metadata lookups. Must match the Aurora region the managed CRs use. The default manifest sets ap-northeast-2.
--rds-metadata-refresh-interval	RDS_METADATA_REFRESH_INTERVAL or 1m	Go duration	Shared RDS metadata refresh interval for AZ/DbiResourceId enrichment. Values below 10s are clamped to 10s.
--aws-api-qps	AWS_API_QPS or 1	requests/sec	Defensive AWS API limiter for the shared metadata worker. Normal load is already bounded by cluster de-duplication and the refresh interval.
--aws-api-burst	AWS_API_BURST or 1	requests	Defensive AWS API limiter burst.
--workers-per-cr	WORKERS_PER_CR or 10	count	Maximum concurrent backend monitor probes within one CR. The monitor creates only as many worker goroutines as the current backend count requires.
--max-concurrent-reconciles	64	count	Maximum concurrent reconciles across different CRs. controller-runtime still serializes the same workqueue key, so this raises cross-CR throughput without allowing concurrent reconciles for the same CR.
--watch-namespace	WATCH_NAMESPACE	namespace	Namespace the manager watches. Required. Cluster-wide watch is not supported. The default manifest uses the operator Pod namespace.
--watch-names	WATCH_NAMES	CR names	Optional PgBouncerAurora name filter. Empty/* watches all CRs in --watch-namespace; a,b,c or repeated --watch-names=a --watch-names=b watches only those CRs.
--reconcile-min-interval	RECONCILE_MIN_INTERVAL or 1s	Go duration	Minimum interval between heavy reconciles for the same CR. This is a per-CR guard, not a global reconcile throttle.
--k8s-api-timeout	K8S_API_TIMEOUT or 10s	Go duration	Timeout for each Kubernetes API request made by the operator.
--status-refresh-min-interval	STATUS_REFRESH_MIN_INTERVAL or 5s	Go duration	Minimum refresh interval for the cached snapshot the /status dashboard shows.
--status-recent-window	STATUS_RECENT_WINDOW or 1m	Go duration	Time window used to highlight recently changed /status items. Values are clamped to 1m–24h.
--zap-devel	false	boolean	Enable development-mode logging.

The operator detects its Deployment identity from POD_NAME/POD_NAMESPACE and the owner chain Pod -> ReplicaSet -> Deployment. That identity is used for the default leader election ID and per-CR ownership claims. If detection is not possible, the built-in fallback remains available and a warning is logged.

Each managed PgBouncerAurora has a namespaced Lease named pgba-claim-<cr-name> that records which operator Deployment owns it. Discovery, monitoring, RDS metadata refresh, status updates, and child-resource writes all check this claim. To manually hand a CR to another operator, stop or exclude the old operator and delete the claim Lease:

kubectl -n <namespace> delete lease pgba-claim-<cr-name>

The status field .status.operatorClaim.operatorId is updated when ownership is acquired or taken over, but heartbeat renewals are stored only in the Lease to avoid noisy CR status writes.

Claim renewal currently piggybacks on reconcile, discovery, monitor, and shared RDS metadata worker activity. Keep the effective discovery/monitor intervals at or below half of --operator-claim-ttl, or increase the claim TTL. If a CR's configured intervals exceed half the TTL, the scheduler logs a warning because a mixed deployment can otherwise see ownership ping-pong. In a handoff, takeover latency is roughly claim TTL + the other operator's next job/reconcile interval; delete the claim Lease manually when immediate handoff is required.

Role Binding

The operator runs only within its own namespace via a namespaced Role (deploy/role.yaml) — not a ClusterRole, and it does not access resources in other namespaces. A RoleBinding (deploy/rolebinding.yaml) binds this Role to the operator ServiceAccount.

Permissions the Role grants:

Resource	Verbs	Purpose
pgbouncerauroras, .../status, .../finalizers	get, list, watch, update, patch	Watch the CR and update status/finalizers.
apps/deployments	get, list, watch, create, update, patch, delete	Manage per-instance PgBouncer Deployments.
apps/replicasets	get	Detect the operator Deployment identity from the running Pod owner chain.
services, configmaps	get, list, watch, create, update, patch, delete	Manage role/instance Services and PgBouncer config.
pods	get, list, watch, patch	Service membership labeling and operator Deployment identity detection.
secrets	get	Read DB auth/userlist Secrets.
coordination.k8s.io/leases	get, list, watch, create, update, patch, delete	Leader election and per-CR ownership claims.
events	create, patch, update	Record reconcile events.

Service Account

The operator Pod runs as the pgbouncer-aurora-operator ServiceAccount (deploy/serviceaccount.yaml) and gets its permissions only through the RoleBinding above. The PgBouncer Pod's ServiceAccount is separate and set, when needed, via spec.pgbouncer.serviceAccountName (unset by default).

Status dashboard

The operator provides a built-in web dashboard to view runtime state at a glance. It is served directly by the operator process, with no separate deployment.

Path	Content
/status	Human-facing HTML dashboard
/status.json	Machine-readable JSON of the same data

• Where it is served — it rides on the metrics server, so it is reachable on --metrics-bind-address (default :8080). For a quick check, run kubectl -n <namespace> port-forward deploy/pgbouncer-aurora-operator 8080:8080 and open http://localhost:8080/status in a browser.
• Refresh interval — the dashboard shows a cached snapshot, whose minimum refresh interval is controlled by --status-refresh-min-interval (default 5s).
• Recent-change highlight — summary cards, Managed CR rows, and Conditions are highlighted when lastAppliedTime or condition lastTransitionTime falls within --status-recent-window (default 1m, clamped to 1m–24h). Periodic discovery/monitor timestamps are not used for this highlight.

What it shows:

• Top summary — number of managed CRs, Writer/Reader Service member counts, and degraded/frozen CR counts.
• Managed CRs — all CRs the operator watches with each one's state (Ready/Degraded/Frozen), generation, and Writer/Reader counts.
• Selected CR detail
  • Lifecycle — last discovery/monitor/applied times, ownership claim owner, Lease name, claim time, consecutive discovery failures, and current Writer/Reader members.
  • Services — candidates/healthy/members/ready counts per Writer/Reader role Service.
  • Instances — role/health/ready/AZ/DbiResourceId/endpoint per Instance.
  • Conditions — DiscoveryTrusted, MonitorSucceeded, Reconciled, Frozen, BackendHealthy, WriterReady, ReaderReady, ReaderFallback, Degraded, and so on.

[!WARNING] /status and /status.json have no authentication and expose topology information such as Aurora endpoints and instance identifiers as-is. Do not expose the metrics/status port to outside the cluster or to untrusted networks; restrict access with a NetworkPolicy or port-forward.

Zone-aware placement

If an Aurora instance and the PgBouncer Pod that serves it are in different AZs, you incur cross-AZ latency and data transfer cost. Zone-aware placement reduces this by putting each PgBouncer Pod on a node in the same AZ as its Aurora instance. It is enabled by default and controlled by spec.topologyPolicy.zoneAware.

• AZ source — aurora_replica_status() has no AZ, so only when zone-aware is on does the operator enrich each instance's AZ via RDS DescribeDBClusters/DescribeDBInstances. This metadata is only a placement hint and physical identity enrichment/debugging data, not the topology decision source or Service membership-removal trigger. A lookup failure does not block instance add/remove itself.
• Enforcement — Preferred (default) generates preferredDuringSchedulingIgnoredDuringExecution nodeAffinity, while Required generates requiredDuringSchedulingIgnoredDuringExecution nodeAffinity. The default is Preferred so that the Pod still schedules and avoids a DB-connection outage even when no same-AZ node exists. Use Required when strongly steering node creation in a specific AZ via Karpenter or similar, but be aware the Pod may stay Pending if the NodePool/subnet/AZ constraints do not align.
• Relationship with user scheduling constraints — zone-aware is additive to existing settings and does not replace user-specified nodeSelector, affinity, or topologySpreadConstraints. The operator detects common cases that may conflict with the zone-aware intent — a strict nodeSelector, a required nodeAffinity, or zone pinning + topology spread on the same key — and handles them per conflictPolicy (Warn/Fail/Ignore). It cannot prove every scheduler outcome, so also monitor the Kubernetes scheduling state.
• Replica spreading — to spread the PgBouncer replicas of the same Aurora instance across nodes, use the native spec.pgbouncer.topologySpreadConstraints (there is no operator-specific replicaSpread option). The roles are split: zone nodeAffinity handles AZ alignment, and topologySpreadConstraints handles replica spreading.

Writer and Reader Service membership convergence performance

Failover recovery performance

This operator's performance metric is "how fast an Aurora role/topology change is reflected into Writer/Reader Service membership." Overall failover performance cannot be described by a single number — role/topology change, Service membership update, and application query recovery are different stages.

This operator focuses on converging the connection path quickly through SQL discovery and Kubernetes Service membership, without waiting for Aurora endpoint DNS propagation:

Aurora Writer/Reader role change
→ Operator discovery
→ Writer/Reader Service membership update
→ EndpointSlice update
→ PgBouncer backend reconnect
→ Application reconnect / pool recycle

These numbers do not mean already-established connections or in-flight transactions are preserved without interruption. Detecting disconnects, reconnecting, and discarding wrong-role connections are the application responsibilities described in Limitations and operating assumptions.

In repeated two-instance Aurora failover tests, the observed sequence was: new Writer restart starts, old Writer restart starts, the Writer endpoint changes around the Completed failover event, the old Writer finishes restart, and the Reader endpoint changes last. The Reader endpoint changed much later than the Writer endpoint and could point at the new Writer during that lag. This is why the operator treats SQL discovery (aurora_replica_status()) as the topology source of truth and does not use Aurora Writer/Reader endpoint DNS membership as topology truth.

The table below shows app-like driver client recovery times measured with Aurora PostgreSQL 17.4, a 2-member cluster (1 Writer, 1 Reader), two PgBouncer replicas per instance, and readerEmptyFallback.enabled=true. Most clients used the Writer/Reader Kubernetes Services as a host-list, with primary for Writer connections and preferSecondary for Reader connections. Connection timeout was 3 seconds, query timeout was 5 seconds, and failed connections were discarded and retried every 1 second.

Values are averages. Writer recovery is measured by role-match recovery; Reader recovery is measured by connection availability. Driver/library versions are from the test image.

[!NOTE] These values are reference numbers from the test clients and driver settings used here. Real application recovery time depends on connection pool behavior, timeout settings, validation queries, maximum connection lifetime, retry policy, and workload shape.

Client	Driver/library version	Writer recovery time	Reader recovery time	Connection handling in test
c-libpq	libpq 15.18	15.4s	5.8s	host-list + libpq target_session_attrs
go-libpq	github.com/lib/pq v1.12.3	14.0s	5.0s	host-list + libpq-style target_session_attrs
go-pgx	github.com/jackc/pgx/v5 v5.5.5	11.0s	3.6s	host-list + pgx target_session_attrs + app-level validation
java-pgjdbc	pgjdbc 42.7.7	17.2s	2.1s	host-list + pgjdbc targetServerType
node-pg	pg 8.22.0	13.0s	2.0s	split datasources + app-level validation
node-pg-native	pg-native 3.8.0	14.8s	4.8s	host-list + libpq target_session_attrs
php-pgsql	PHP 8.2.31 + pdo_pgsql/pgsql	16.6s	4.8s	host-list + libpq target_session_attrs
python-asyncpg	asyncpg 0.31.0	14.2s	5.2s	host-list + target_session_attrs
python-psycopg3	psycopg 3.3.4	16.8s	4.8s	host-list + libpq target_session_attrs
ruby-pg	pg 1.6.3	16.2s	6.0s	host-list + libpq target_session_attrs
rust-tokio-postgres	tokio-postgres 0.7.18	12.0s	4.2s	split datasources + any + app-level validation

The Reader recovery times above use readerEmptyFallback.enabled=true together with preferSecondary/any Reader targets. If you use strict secondary/read-only, see the spec.topologyPolicy.readerEmptyFallback.enabled=false behavior in the configuration section.

With a 3-member Aurora cluster, internal tests showed Reader recovery time converging to 0 seconds because an extra Reader instance not involved in the failover remained available to accept connections.

External exposure reference measurements

When exposing the Services outside the cluster, you can create NLBs through AWS Load Balancer Controller. That path has target deregistration, health-check, and propagation latency, so Aurora topology changes can take 10-20+ seconds longer than the in-cluster Kubernetes Service path. To reduce that overhead, use a short deregistration delay, connection termination on deregistration, cross-zone balancing, and fast TCP health checks:

# spec.services.writer / spec.services.reader
type: LoadBalancer
annotations:
  service.beta.kubernetes.io/aws-load-balancer-type: external
  service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
  service.beta.kubernetes.io/aws-load-balancer-scheme: internal
  service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: TCP
  service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: traffic-port
  service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "2"
  service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "2"
  service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout: "3"
  service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "5"
  service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: deregistration_delay.timeout_seconds=10,deregistration_delay.connection_termination.enabled=true,load_balancing.cross_zone.enabled=true
  service.beta.kubernetes.io/aws-load-balancer-attributes: load_balancing.cross_zone.enabled=true

These are aggressive values intended to speed up NLB target replacement and health-check convergence. In internal repeated tests, they reduced failover recovery time on the external exposure path by more than 50% compared with the default NLB settings. Lower health-check intervals increase target and network probe traffic, so tune them from the actual traffic pattern and target count in production.

Topology change cases

Beyond failover, common operational topology-change cases were measured.

Case	Service reflection	Response time
Writer deletion + failover	Switch the Writer Service to the new Writer, and exclude the deleting instance from the Reader Service as well	about 3.1-3.2s
Reader addition	Include the new Reader in the Reader Service. Until confirmed, fallback treats it as an unconfirmed Reader	about 3.1-3.2s
Reader deletion	Exclude the Reader from the Reader Service according to the missing-count policy	about 3.1s
All Readers removed or unconfirmed	With readerEmptyFallback=true, temporarily include the Writer in the Reader Service	about 0.1-0.2s

The response times above include the default 3-second discovery interval. Fallback cases that can be decided immediately from Monitor results are applied in the same reconcile without waiting for the discovery interval.

In these measurements too, the topology decision source is aurora_replica_status(), and RDS metadata is an optional layer for AZ/DbiResourceId enrichment and debugging. A metadata refresh failure does not make discovery untrusted, and ordinary add/remove proceeds via the discovery/monitor policy rather than RDS instance status.

Reader Service load balancing

This distribution is per connection, not per query. This project's scope is connection distribution across Reader instances; it does not provide per-query load balancing.

The Reader Service delegates connection distribution to the default balancing of the Kubernetes Service (kube-proxy). It is important to understand the characteristics of this distribution.

• Distribution is random, not round-robin. kube-proxy's default iptables mode picks a backend probabilistically at each connection-establishment time (statistic mode random). So the unit of distribution is the connection, and once established a connection is pinned to that Reader instance until it closes.
• Small samples can skew. Random distribution is independent trials, so with few connections (within a few multiples of the instance count) load can skew onto a particular Reader. In a long-lived connection pool, this initial distribution tends to persist.
• In real production, however, it usually distributes evenly. Applications typically run pools of dozens of connections per instance, with dozens to hundreds of such instances running at once. Since the total connection count comfortably exceeds the Reader instance count, random distribution statistically converges to uniform.
• Advanced networking data paths can be considered. The default kube-proxy setup is usually sufficient at operating scale. If you need stricter distribution or higher Service processing performance, consider IPVS schedulers such as rr (round-robin) or lc (least-connection), nftables-based kube-proxy, or a Cilium eBPF kube-proxy replacement.

In repeated tests with three Readers and a simulated connection pool, distribution became even quickly as the connection count increased. The large scenario models 100 application processes, each opening 10 connections.

Scenario	Application processes	Pool size	Total connections	Reader distribution	Balance skew
small	2	10	20	31.0% / 35.0% / 34.0%	2.18x
medium	10	10	100	35.4% / 34.6% / 30.0%	1.28x
large	100	10	1,000	33.4% / 32.9% / 33.7%	1.12x

Reader distribution and Balance skew are averages from repeated tests. Balance skew is the ratio between the most-connected Reader and the least-connected Reader in each run. Values closer to 1.00x indicate more even distribution.

Development

# full local checks
./hack/check.sh

# including the Docker build
DOCKER=true ./hack/check.sh

# race detector
go test -race -count=1 ./...

The smoke script is designed to be safe by default. ./hack/smoke-test.sh only validates the manifests locally unless you pass APPLY=true. For Kubernetes API-server validation without creating resources, use DRY_RUN=server VALIDATE=true ./hack/smoke-test.sh.

Build from source (custom image)

The public install uses the published Quay image, so you do not need to build anything to run the operator. To run your own image — for development or a private registry — build and push a multi-arch image, then point the Deployment at it instead of the Quay image:

docker buildx build --platform linux/amd64,linux/arm64 \
  -t <registry>/pgbouncer-aurora-operator:<tag> \
  --push .

kubectl -n pgbouncer-aurora set image deploy/pgbouncer-aurora-operator \
  manager=<registry>/pgbouncer-aurora-operator:<tag>

Contributing

See CONTRIBUTING.md for contribution, versioning, and release policies.

Contributions are welcome. The process is intentionally simple for now:

1. For non-trivial behavior changes, open an issue first or draft the intended change.
2. Keep changes small and focused.
3. Add/update tests for controller/discovery/monitor/planner/render behavior.
4. Run ./hack/check.sh before a PR.
5. Update the README when CRD options or user-visible behavior change.

License

This project is distributed under the ISC License. See LICENSE.