pgbouncer-aurora-operator

README

pgbouncer-aurora-operator

pgbouncer-aurora-operator watches Aurora PostgreSQL topology and runs one PgBouncer per DB instance (1:1). Applications can connect through fixed Writer/Reader Kubernetes Services, or through per-instance Services as a host-list, while the operator keeps Service membership aligned with the current Aurora roles. Its goal is to reduce the Reader connection skew that PgBouncer pooling introduces and to reflect topology changes such as instance scaling and failover into the connection layer automatically.

Table of Contents

• Why it exists
• What it does
• Limitations and operating assumptions
  • Recommended application settings
  • Application connection guide
• Requirements
• Installation
• Configuration
  • Custom Resource
  • Secret
  • Operator
• Status dashboard
• Writer and Reader Service membership convergence performance
• Reader Service load balancing
• Development
• Contributing
• License

Why it exists

A typical "PgBouncer in front of Aurora" setup has the following limitations.

1. The Aurora Reader endpoint cannot spread a connection pool. The Reader endpoint only picks an instance at connection time (DNS resolution). A long-lived pool connection, once established, is pinned to that instance — so in a pooled environment load skews onto one particular Reader (Reader connection skew).

2. Topology changes are not reflected automatically. If PgBouncer is pointed directly at a single Aurora instance endpoint to avoid problem #1, it does not follow instances that change through scale-out, replacement, or promotion. To use a newly added Aurora instance, you have to define the corresponding PgBouncer deployment by hand each time.

3. The Aurora cluster endpoint is strongly affected by DNS cache, TTL, and refresh behavior. After failover, even if the cluster endpoint points at the new Writer, stale endpoint/IP data can remain in use longer depending on DNS settings.

This operator watches Aurora topology, updates Writer/Reader Kubernetes Service membership per instance through operator-managed Pod labels, and connects each PgBouncer directly to a single Aurora instance (1:1). With this structure, Reader load is spread across per-instance PgBouncer Pods instead of being pinned to one endpoint, and new instances are joined to the Service for their current role after they are ready.

What it does

flowchart LR
  App[Application Pods]
  WriterSvc[Writer Service]
  ReaderSvc[Reader Service]
  PerSvc[Per-instance Services]
  Pods[PgBouncer Deployments\none per Aurora instance]
  Aurora[(Aurora PostgreSQL Cluster)]
  RDS[AWS RDS API\nDescribeDBClusters + DescribeDBInstances]
  CR[PgBouncerAurora CR]
  Operator[pgbouncer-aurora-operator]
  K8s[Kubernetes API]

  App --> WriterSvc
  App --> ReaderSvc
  WriterSvc --> Pods
  ReaderSvc --> Pods
  PerSvc --> Pods
  Pods --> Aurora

  CR --> Operator
  Operator --> K8s
  Operator --> Aurora
  Operator --> RDS
  Operator --> PerSvc
  Operator --> WriterSvc
  Operator --> ReaderSvc
  Operator --> Pods

Key features

• Automatic Aurora topology sync — detects instance add/remove and Writer failover, and reflects them into Writer/Reader Service membership.
• Per-instance 1:1 PgBouncer — resolves Reader connection skew.
• Fast Writer failover response — once trusted Discovery confirms the new Writer and its PgBouncer Pod is Ready, the Writer Service membership is switched first without waiting for the next Monitor success (fast path). Detecting a failover opens a 30-second failover window during which discovery and monitor run on a 100ms fast cadence.
• Safety mechanisms — last-known-good retention and freeze keep the last verified healthy traffic state when observations are uncertain.
• Built-in status dashboard — the operator serves a /status web UI and /status.json that show managed CRs, topology, Writer/Reader membership, and conditions at a glance (Status dashboard).

How it works

• Discovery

  • Builds Aurora Writer/Reader cluster endpoints from spec.discovery.clusterName, spec.discovery.domainName, and spec.discovery.port, then queries aurora_replica_status() through the Writer cluster endpoint by default.
  • If the Writer cluster endpoint is briefly unavailable during failover, it immediately retries with the Reader cluster endpoint built in the same discovery tick.
  • Determines role from session_id (MASTER_SESSION_ID = Writer).
  • Instance endpoints are generated deterministically as {instanceName}.{domainName}.
  • RDS metadata assists SQL discovery: it enriches AZ and DbiResourceId (the physical instance identity) for status, and when fresh metadata confirms a Reader instance is deleting, the operator detaches it from traffic immediately instead of waiting out the debounce. The topology truth is always aurora_replica_status().
  • Topology changes such as failover and reader add/delete pull the metadata refresh forward instead of waiting for the periodic interval, and a completed refresh is consumed in the same decision pass.
  • RDS API failures or timeouts do not affect normal Discovery. Without metadata, the operator keeps going on SQL discovery and the debounce policy alone.

• Rendering

  • Creates one ConfigMap, Deployment, and Service per discovered Aurora DB instance.
  • By default, renders a PgBouncer [databases] wildcard route to the corresponding Aurora instance. You can add or override database entries, but each entry's host/port is still managed per instance by the operator.
  • Creates role Services for Writer/Reader traffic.
  • Role Services select Pods through two operator-managed label axes: member-writer/member-reader say which role a Pod belongs to, and traffic says whether it may be routed to right now. A Service only picks Pods that match both the membership and traffic="true".
  • Every Pod starts with traffic="false" and earns traffic through monitor health and Pod readiness. Deleted, disabled, or retained instances carry traffic="none", which the monitor never wakes up.
  • Every Pod also carries a member-any label, so when no healthy Reader exists the Reader Service can fall back to the Writer by switching only its selector to member-any — no rollout required.

• Monitoring

  • Always connects directly to the backend DB and checks role/health via pg_is_in_recovery() and transaction_read_only. healthy means exactly that the probe succeeded; PgBouncer Pod readiness is observed as a separate signal alongside it.
  • A Reader serves traffic only when both signals hold: the probe has cleared failureThreshold/recoveryThreshold, and all Pods are Ready (a plain readiness gate, not threshold-based).
  • A Writer never loses traffic to a monitor failure alone; only a trusted topology change replaces the Writer.
  • During Writer failover, Discovery is the role decision source. Monitor is treated as a sanity check rather than the primary gate.

• Topology handling

  • New Aurora instances get new PgBouncer resources and join traffic only after proving their health.
  • Instances that vanish from discovery pass the missing debounce (removeAfterMissingCount plus removeAfterMissingSeconds) before losing traffic, and their per-instance resources are deleted after retention. When fresh RDS metadata confirms deleting, the debounce is skipped and traffic is detached immediately.

Limitations and operating assumptions

This operator's job is to detect topology and update connection paths. Its responsibility boundary is stated clearly here.

Maintaining high availability (HA)

PgBouncer is a lightweight, high-performance connection pooler, not a high-availability proxy that preserves connections without interruption. This operator does not change PgBouncer's role. Instead, it detects Aurora role changes quickly and updates Service membership so applications can keep using fixed Writer/Reader Service addresses.

• Aurora failover behavior: when the Writer changes, existing Writer connections can be dropped. Any in-flight transaction on that connection can fail, and no proxy layer can completely remove that behavior.
• Writer change cleanup: during Aurora failover, the PgBouncer Deployments for the old and new Writer instances are restarted. This drops stale app→PgBouncer and PgBouncer→Aurora server connections and nudges the application/pool to reconnect through the new Writer path.
• PgBouncer pod redundancy: use at least two PgBouncer replicas per Aurora instance. With graceful/rolling restarts, this keeps PgBouncer Pods available while old connections are drained and replaced.
• Reader fallback built in: availability comes first in production, so when no Reader is available the operator temporarily includes the Writer in the Reader Service. This behavior is built in and cannot be turned off. If the application uses role checks, set Reader connections to targetServerType=preferSecondary or targetServerType=any. This is especially valuable for two-member Aurora clusters.

If you need strong zero-downtime HA at the level of preserving connections, consider a layer designed for that purpose, such as Pgpool-II or PgCat.

What the application is responsible for

Detecting disconnects, reconnecting, retrying, and deciding which role (Writer/Reader) to use a connection for are the application's responsibility and out of scope for this project (Non-goals: HA, zero-downtime failover, in-flight transaction recovery, query-time role selection).

This operator only updates Kubernetes Services and PgBouncer workloads when Aurora topology changes; it does not move already-established app→PgBouncer TCP connections to a different Pod. Existing connections keep using the current PgBouncer process until the application or PgBouncer closes them, or until the Writer-change rolling restart drains/terminates the old Pod. Cleanup timing depends on PgBouncer shutdown behavior and spec.pgbouncer.terminationGracePeriodSeconds (30s by default).

Recommended application settings

The values below are conservative starting examples. Real values depend on the driver, pool framework, workload, and DB/PgBouncer timeout policies. Tune them from production observations.

Item	Recommended
Connection validation / keepalive	Prefer the validation, keepalive, or ping features built into your driver/pool framework. If you split Writer/Reader connections, also consider role checks such as targetServerType.
Periodic idle connection check	Periodically validate idle pooled connections so connections already closed by the DB server or network are detected and handled before application code receives them.
Maximum connection lifetime	Use a finite pool connection lifetime (for example, around 15 minutes). Close and reopen connections after they return to the pool.
Idle timeout	Keep the pool/client idle timeout shorter than server-side idle cleanup such as idle_session_timeout (for example, 10-15 minutes).
Reconnect/retry	Enable short backoff retries on connection-acquisition failures. Enable automatic transaction retries only when the operation is safe and idempotent.
Pool size	Keep the number of backend connections fixed and finite. number of application instances × pool size should stay comfortably below the max_connections equivalent limit of the DB/PgBouncer.
DNS cache TTL	Avoid infinite DNS caching and use a short TTL (for example, no later than about 30 seconds).
Connection/login timeout	Keep login or connection timeouts short so temporary database delays fail fast and the application can recover quickly (for example, 2-3 seconds).

Application connection guide

There are three ways for an application to point at the Services the operator creates. With any of them, the operator reflects failover/scale changes by updating each Service's membership.

Method 1 — two Writer/Reader Services as a host-list + targetServerType (driver role check)

postgresql://example-pg-writer.<namespace>.svc.cluster.local:6432,example-pg-reader.<namespace>.svc.cluster.local:6432/db?targetServerType=primary

• The driver takes these two Services as a host-list and chooses a Service that matches the requested role when opening a new connection. Write connections use primary. For read connections, use preferSecondary or any so temporary Reader fallback to the Writer remains usable during two-member failover windows.
• The Service names (<cr-name>-writer, <cr-name>-reader) are fixed and the operator updates membership, so Aurora scale/topology changes are reflected automatically.

Method 2 — split read/write connections to the Reader/Writer Service respectively (simplest)

Point write-only connections at the Writer Service and read-only connections at the Reader Service.

# write-only datasource
postgresql://example-pg-writer.<namespace>.svc.cluster.local:6432/db
# read-only datasource
postgresql://example-pg-reader.<namespace>.svc.cluster.local:6432/db

• This is the most natural setup if the application already splits read/write datasources. It does not require driver host-list or targetServerType support.
• Even when the Writer changes due to failover, the operator updates the Writer Service membership to the new Writer, so the application keeps using the same Service address.

Method 3 — individual instance Services as a host-list

postgresql://example-pg-instance-1.<namespace>.svc.cluster.local:6432,example-pg-instance-2.<namespace>.svc.cluster.local:6432/db?targetServerType=primary

• Listing per-instance Services (<cr-name>-<dbInstanceIdentifier>) directly makes the driver iterate over instances to find a valid connection (the same pattern as the AWS docs' "individual DB instance nodes").
• ⚠️ Aurora scale-out/topology changes are not reflected automatically. When a new instance appears, you must manually update the host-list in the connection string. To get the operator's automatic-reflection benefit, use Method 1 or 2.

The port follows the PgBouncer listen_port (default 6432). Connect to the port PgBouncer listens on, not the Aurora port (5432).

Reference: AWS fast failover best practices

Driver-side fast failover settings align with the recommendations in the AWS docs: Best practices with Amazon Aurora PostgreSQL fast failover. Key recommended values (examples):

• Aggressive timeouts: loginTimeout=2, connectTimeout=2, cancelSignalTimeout=1, with a low socketTimeout for short queries (split long queries into separate connections).
• tcpKeepAlive=true, and loadBalanceHosts=true when using a host-list.
• DNS TTL under 30 seconds (JVM: networkaddress.cache.ttl=1, networkaddress.cache.negative.ttl=3).
• Specify the target with targetServerType (primary/secondary/preferSecondary/any).

The AWS docs recommend "listing individual instance nodes in a host-list" for the best failover, while noting its downside: "requires manual update on topology change." Methods 1 and 2 above (pointing at the Writer/Reader Services) automate exactly that manual update via the operator, so the application only needs to point at the Writer/Reader Services.

Requirements

Runtime environment

Component	Version / requirement	Notes
Kubernetes	>= 1.27 recommended	Uses apiextensions.k8s.io/v1, apps/v1, rbac.authorization.k8s.io/v1, and standard Service/Pod APIs. Lower versions may work but are not yet targeted.
Aurora PostgreSQL	Aurora PostgreSQL with aurora_replica_status()	Discovery depends on select server_id, session_id from aurora_replica_status().
PgBouncer	1.25.2 verified	Verified with PgBouncer 1.25.2; provide your own image (see spec.pgbouncer.image). Other versions may work if the config options are compatible.
AWS IAM	rds:DescribeDBClusters, rds:DescribeDBInstances	Needed for RDS metadata: AZ/DbiResourceId enrichment and confirming a Reader is deleting. RDS API failures or timeouts do not affect normal Discovery. EKS IRSA recommended.
Network	Pod→Aurora connectivity	The operator and PgBouncer Pods must be able to reach the Aurora endpoints/ports.

Build / test tools

Tool	Version / requirement	Notes
Go	1.26	Declared in go.mod and the Docker build image.
Docker Buildx	Recent Docker Desktop / BuildKit	Needed for local and multi-arch image builds.
kubectl	Compatible with the target cluster	Used for manifest dry-run, install, and smoke-test scripts.
Bash	A Bash-capable POSIX shell	Needed for hack/*.sh.

Installation

Use either Manifest or Helm charts to install the published operator runtime. To build and run your own image instead, see Build from source.

Pin the explicit version tag shown below. See Requirements for the cluster, Aurora, IAM, and network prerequisites.

1-A. Install the operator with Manifest

# Set your environment
VERSION=v0.2.0
NAMESPACE=pgbouncer-aurora
OPERATOR_IMAGE=quay.io/case-88/pgbouncer-aurora-operator:${VERSION}
MANIFEST_BASE=https://raw.githubusercontent.com/case-88/pgbouncer-aurora-operator/${VERSION}/deploy

kubectl create namespace ${NAMESPACE}
kubectl apply --server-side -f ${MANIFEST_BASE}/crd.yaml
kubectl apply -f ${MANIFEST_BASE}/serviceaccount.yaml -f ${MANIFEST_BASE}/role.yaml -f ${MANIFEST_BASE}/rolebinding.yaml -f ${MANIFEST_BASE}/operator.yaml

# The bundled deploy/operator.yaml ships a placeholder image, so point the
# Deployment at the published Quay image:
kubectl -n ${NAMESPACE} set image deploy/pgbouncer-aurora-operator manager=${OPERATOR_IMAGE}
kubectl -n ${NAMESPACE} rollout status deploy/pgbouncer-aurora-operator

The bundled manifests are namespaced to pgbouncer-aurora; keep NAMESPACE set to that value, or edit the manifests' namespace: fields to install elsewhere.

1-B. Install with Helm charts

The Helm packaging is split into three charts:

Chart	Installs	Notes
pgbouncer-aurora-crds	PgBouncerAurora CRD only	CRD is rendered from templates/ so helm upgrade can update the schema.
pgbouncer-aurora-operator	Operator ServiceAccount, RBAC, Deployment, metrics/status Service, NetworkPolicy	Does not install CRD or CRs.
pgbouncer-aurora	One PgBouncerAurora CR and related Secret/ExternalSecret resources	Can optionally install a dedicated operator through the operator subchart.

Install the CRD chart first:

NAMESPACE=pgbouncer-aurora
HELM_CHART_REPO=oci://quay.io/case-88/charts
HELM_CHART_VERSION=0.2.0

helm upgrade --install pgbouncer-aurora-crds ${HELM_CHART_REPO}/pgbouncer-aurora-crds \
  --version ${HELM_CHART_VERSION} --namespace ${NAMESPACE} --create-namespace

Then install one shared operator for the namespace:

helm upgrade --install pgbouncer-aurora-operator ${HELM_CHART_REPO}/pgbouncer-aurora-operator \
  --version ${HELM_CHART_VERSION} --namespace ${NAMESPACE} --create-namespace

The default topology is shared mode:

crds chart -> one operator chart -> N pgbouncer-aurora instance chart releases

For dedicated mode, set operator.enabled=true on the instance chart. The operator subchart then watches only that CR by default.

When mixing shared and dedicated operators in one namespace, keep the shared operator on an explicit watch.names include list and exclude dedicated CRs. Per-CR Lease claims prevent duplicate writes, but watch.names is still the intended management boundary.

Helm upgrade:

helm upgrade pgbouncer-aurora-crds ${HELM_CHART_REPO}/pgbouncer-aurora-crds \
  --version ${HELM_CHART_VERSION} --namespace ${NAMESPACE}

helm upgrade pgbouncer-aurora-operator ${HELM_CHART_REPO}/pgbouncer-aurora-operator \
  --version ${HELM_CHART_VERSION} --namespace ${NAMESPACE}

The CRD chart sets helm.sh/resource-policy: keep by default. Keep this default unless you deliberately want helm uninstall to delete the CRD and all PgBouncerAurora custom resources.

Reference: with Helm 3.17.0 or later, use --take-ownership only if you need to take ownership of a CRD that was previously installed through a chart crds/ directory.

helm upgrade pgbouncer-aurora-crds ${HELM_CHART_REPO}/pgbouncer-aurora-crds \
  --version ${HELM_CHART_VERSION} --namespace ${NAMESPACE} --take-ownership

2. Configure credentials and a PgBouncerAurora resource

The Secret and the CR carry environment-specific values — DB user/password, userlist.txt, clusterName/domainName, the PgBouncer image, and so on. Use either raw manifests or the pgbouncer-aurora Helm instance chart for this step.

Manifest path:

VERSION=v0.2.0
MANIFEST_BASE=https://raw.githubusercontent.com/case-88/pgbouncer-aurora-operator/${VERSION}/deploy
curl -fsSLo /tmp/pgbouncer-aurora-secrets.yaml ${MANIFEST_BASE}/secrets.yaml
curl -fsSLo /tmp/pgbouncer-aurora.yaml         ${MANIFEST_BASE}/cr.yaml

# Edit both files:
#   secrets.yaml — DB username/password and the userlist.txt entries
#   cr.yaml      — discovery clusterName/domainName/port, pgbouncer.image, etc.
$EDITOR /tmp/pgbouncer-aurora-secrets.yaml /tmp/pgbouncer-aurora.yaml

kubectl -n ${NAMESPACE} apply -f /tmp/pgbouncer-aurora-secrets.yaml
kubectl -n ${NAMESPACE} apply -f /tmp/pgbouncer-aurora.yaml

Helm instance chart path:

# Prepare Secrets first. By default the chart references existing Secrets named
# pgbouncer-operator-db-auth and pgbouncer-userlist; it also supports
# operatorAuth.create/userlist.create and externalSecret.enabled modes.
kubectl -n ${NAMESPACE} apply -f /tmp/pgbouncer-aurora-secrets.yaml

helm upgrade --install example-pg ${HELM_CHART_REPO}/pgbouncer-aurora \
  --version ${HELM_CHART_VERSION} --namespace ${NAMESPACE} \
  --set discovery.clusterName=example-pg \
  --set discovery.domainName=xxxx.ap-northeast-2.rds.amazonaws.com \
  --set pgbouncer.image=<your-registry>/pgbouncer:1.25.2

ExternalSecret mode is also supported. In that case the chart renders ExternalSecret resources and does not require pre-created Kubernetes Secrets:

helm upgrade --install example-pg ${HELM_CHART_REPO}/pgbouncer-aurora \
  --version ${HELM_CHART_VERSION} --namespace ${NAMESPACE} \
  --set skipCRDCheck=true \
  --set discovery.clusterName=example-pg \
  --set discovery.domainName=xxxx.ap-northeast-2.rds.amazonaws.com \
  --set pgbouncer.image=<your-registry>/pgbouncer:1.25.2 \
  --set operatorAuth.existingSecretName= \
  --set operatorAuth.name=pgbouncer-operator-db-auth \
  --set operatorAuth.externalSecret.enabled=true \
  --set operatorAuth.externalSecret.apiVersion=external-secrets.io/v1 \
  --set operatorAuth.externalSecret.secretStoreRef.kind=ClusterSecretStore \
  --set operatorAuth.externalSecret.secretStoreRef.name=aws-cluster-secret-store \
  --set operatorAuth.externalSecret.remoteRefs.username.key=dev-eks/db/pgba \
  --set operatorAuth.externalSecret.remoteRefs.username.property=username \
  --set operatorAuth.externalSecret.remoteRefs.password.key=dev-eks/db/pgba \
  --set operatorAuth.externalSecret.remoteRefs.password.property=password \
  --set userlist.existingSecretName= \
  --set userlist.name=pgbouncer-userlist \
  --set userlist.externalSecret.enabled=true \
  --set userlist.externalSecret.apiVersion=external-secrets.io/v1 \
  --set userlist.externalSecret.secretStoreRef.kind=ClusterSecretStore \
  --set userlist.externalSecret.secretStoreRef.name=aws-cluster-secret-store \
  --set userlist.externalSecret.remoteRefs.userlist.key=dev-eks/db/pgba/pg-poc

For userlist, the remote secret value is used as the full userlist.txt content; property is not supported for this mode.

See Configuration for every Secret key and CR field.

3. Verify

kubectl -n ${NAMESPACE} get pgba
kubectl -n ${NAMESPACE} get deploy,svc,pod -l pgbouncer-aurora.io/managed-by=pgbouncer-aurora-operator

# Open the built-in status dashboard
kubectl -n ${NAMESPACE} port-forward deploy/pgbouncer-aurora-operator 8080:8080
# then open http://localhost:8080/status

Optional connection check through the generated Writer/Reader Services:

CR_NAME=example-pg
DB_NAME=postgres
DB_USER=test
PGBOUNCER_PORT=6432

kubectl -n ${NAMESPACE} run pgba-psql-writer --rm -i --restart=Never \
  --image=postgres:16 --env PGPASSWORD='<password>' -- \
  psql "host=${CR_NAME}-writer.${NAMESPACE}.svc.cluster.local port=${PGBOUNCER_PORT} dbname=${DB_NAME} user=${DB_USER} sslmode=disable" \
  -c "select inet_server_addr(), pg_is_in_recovery(), current_setting('transaction_read_only')"

kubectl -n ${NAMESPACE} run pgba-psql-reader --rm -i --restart=Never \
  --image=postgres:16 --env PGPASSWORD='<password>' -- \
  psql "host=${CR_NAME}-reader.${NAMESPACE}.svc.cluster.local port=${PGBOUNCER_PORT} dbname=${DB_NAME} user=${DB_USER} sslmode=disable" \
  -c "select inet_server_addr(), pg_is_in_recovery(), current_setting('transaction_read_only')"

The Writer Service should return pg_is_in_recovery = false and transaction_read_only = off. The Reader Service normally returns a read-only backend; when no Reader is available, it can temporarily return the Writer.

Configuration

An operator deployment is made up of five parts — the Custom Resource that defines what is managed, the Secret that holds DB/PgBouncer credentials, the Operator manager process, and the Role Binding / Service Account that grant permissions. The full manifests are under deploy/.

Custom Resource

One PgBouncerAurora CR manages one Aurora cluster. The minimal form is below; the full example is in deploy/cr.yaml.

apiVersion: pgbouncer-aurora.io/v1alpha1
kind: PgBouncerAurora
metadata:
  name: example-pg
spec:
  discovery:
    clusterName: example-pg
    domainName: xxxx.ap-northeast-2.rds.amazonaws.com
    port: 5432
    authSecretRef:
      name: pgbouncer-operator-db-auth
  pgbouncer:
    image: <your-registry>/pgbouncer:1.25.2
    authFileSecretRef:
      name: pgbouncer-userlist

CRD names

Item	Value
API group	pgbouncer-aurora.io
Version	v1alpha1
Kind	PgBouncerAurora
Plural	pgbouncerauroras
Short name	pgba
Scope	Namespaced

spec.discovery

Option	Required	Default	Unit	Description
spec.discovery.clusterName	Yes	none	string	Aurora DB cluster identifier prefix used to build cluster endpoints. e.g. example-pg.
spec.discovery.domainName	Yes	none	DNS suffix	Common Aurora endpoint suffix. e.g. xxxx.ap-northeast-2.rds.amazonaws.com.
spec.discovery.port	No	5432	TCP port	Port for the generated Writer/Reader/Instance endpoints.
spec.discovery.clusterEndpoints.writer.host	Advanced	generated	DNS name	Explicit Writer endpoint override.
spec.discovery.clusterEndpoints.writer.port	Advanced	spec.discovery.port	TCP port	Explicit Writer endpoint port override.
spec.discovery.clusterEndpoints.reader.host	Advanced	generated	DNS name	Explicit Reader endpoint override.
spec.discovery.clusterEndpoints.reader.port	Advanced	spec.discovery.port	TCP port	Explicit Reader endpoint port override.
spec.discovery.database	No	postgres	database name	Database used for the discovery connection.
spec.discovery.sslMode	No	require	PostgreSQL SSL mode	SSL mode used by Aurora discovery and direct DB monitor probe connections.
spec.discovery.authSecretRef.name	Yes	none	Secret name	Operator DB credential Secret. Used for Aurora discovery and the direct DB monitor probe.
spec.discovery.interval	No	3s	Go duration	Minimum interval for Aurora topology discovery checks. Internal floor 1s.
spec.discovery.timeout	No	3s	Go duration	Timeout for discovery DB queries.
spec.discovery.failureThreshold	No	3	count	Number of consecutive untrusted discoveries allowed before freezing instead of using cached topology.

Endpoint generation rules:

• Writer endpoint: {clusterName}.cluster-{domainName}
• Reader endpoint: {clusterName}.cluster-ro-{domainName}
• Instance endpoint: {dbInstanceIdentifier}.{domainName}

spec.discovery.interval defaults to 3s to follow Aurora failover quickly. This only raises the Aurora topology query frequency. AWS RDS metadata is refreshed by the operator process's shared metadata worker, which pulls the refresh forward on topology changes and feeds a completed refresh into the same decision pass. RDS API failures or timeouts do not affect normal Discovery.

The operator does not infer the AWS region from spec.discovery.domainName. RDS metadata lookups use the operator process flag --aws-region, so this region must match the region of the Aurora cluster that domainName points at. For multi-region operation, run a separate operator deployment per region.

spec.monitor

Option	Required	Default	Unit	Description
spec.monitor.interval	No	10s	Go duration	Minimum interval for monitor checks. Internal floor 1s.
spec.monitor.timeout	No	3s	Go duration	Timeout per backend monitor probe.
spec.monitor.failureThreshold	No	3	count	Consecutive failures before treating a healthy backend as unhealthy.
spec.monitor.recoveryThreshold	No	2	count	Consecutive successes before treating an unhealthy backend as healthy.

Monitor uses Pod readiness and an always-on direct DB probe.

• Direct DB probe — connects directly to the Aurora instance to check health/role. The connection DB is spec.discovery.database (default postgres), and the SSL mode follows spec.discovery.sslMode.
  • For sslmode=verify-full, the operator Pod must trust the Aurora CA. With the pgx driver this can be done by mounting the CA bundle into the operator Pod and setting PGSSLROOTCERT to that file path.

TLS from PgBouncer to Aurora can be configured through PgBouncer settings such as server_tls_sslmode and server_tls_ca_file, with the CA file mounted through spec.pgbouncer.volumes and volumeMounts. Client-to-PgBouncer TLS is not managed directly by the operator API; terminate it outside the operator or provide the required PgBouncer files through custom volumes if you choose to manage it yourself.

spec.pgbouncer

Option	Required	Default	Unit	Description
spec.pgbouncer.image	Yes	empty	image reference	Required; no default. This project does not ship a PgBouncer image. Build your own PgBouncer image and host it on a registry your cluster can pull from (e.g. Amazon ECR), or use a third-party hub image.
spec.pgbouncer.replicas	No	1	replicas	Default replica count for the per-instance PgBouncer Deployment.
spec.pgbouncer.terminationGracePeriodSeconds	No	30	seconds	Kubernetes Pod termination grace for PgBouncer Pods. Lower values shorten the upper bound for stale app→PgBouncer connections during Writer-change rollouts, but can cut off graceful drain earlier.
spec.pgbouncer.config	No	{}	PgBouncer config section	Structured PgBouncer config. See below.
spec.pgbouncer.instanceOverrides[]	No	[]	list	Per-Aurora-instance enable/disable, replicas, and config overrides.
spec.pgbouncer.authFileSecretRef.name	Yes	none	Secret name	Secret holding userlist.txt. Mounted at the default auth_file path /etc/pgbouncer/userlist.txt, or at the effective path when global [pgbouncer].auth_file is overridden.
spec.pgbouncer.resources	No	{}	Kubernetes ResourceRequirements	CPU/memory requests/limits for the main PgBouncer container.
spec.pgbouncer.labels	No	{}	map[string]string	Extra labels on the generated PgBouncer Pod template. Operator-managed labels win on conflict.
spec.pgbouncer.annotations	No	{}	map[string]string	Extra annotations on the generated PgBouncer Pod template. Operator-managed annotations win on conflict.
spec.pgbouncer.serviceAccountName	No	empty	ServiceAccount name	ServiceAccount used by the PgBouncer Pod when needed.
spec.pgbouncer.nodeSelector	No	{}	map[string]string	PgBouncer Pod nodeSelector.
spec.pgbouncer.affinity	No	{}	Kubernetes Affinity	node/pod affinity/anti-affinity.
spec.pgbouncer.tolerations	No	[]	list	Pod tolerations.
spec.pgbouncer.priorityClassName	No	empty	PriorityClass name	PgBouncer Pod priority class.
spec.pgbouncer.runtimeClassName	No	empty	RuntimeClass name	PgBouncer Pod runtime class.
spec.pgbouncer.podSecurityContext	No	{}	Kubernetes PodSecurityContext	Pod-level security context.
spec.pgbouncer.containerSecurityContext	No	{}	Kubernetes SecurityContext	Security context for the main pgbouncer container.
spec.pgbouncer.livenessProbe	No	config.pgbouncer.listen_port TCP probe	Kubernetes Probe	Main-container liveness probe override. Use with care.
spec.pgbouncer.readinessProbe	No	config.pgbouncer.listen_port TCP probe	Kubernetes Probe	Main-container readiness probe override. Affects Service membership and monitor behavior.
spec.pgbouncer.sidecars	No	[]	container list	Extra sidecar containers.
spec.pgbouncer.volumes	No	[]	volume list	Extra volumes. Operator-managed config/auth-file volumes win on conflict.
spec.pgbouncer.volumeMounts	No	[]	volume mount list	Extra mounts on the main pgbouncer container. Operator-managed mounts win on conflict.
spec.pgbouncer.imagePullSecrets	No	[]	Kubernetes LocalObjectReference[]	Image pull secrets added to the PgBouncer Pod.
spec.pgbouncer.topologySpreadConstraints	No	[]	Kubernetes TopologySpreadConstraint[]	Native Pod topology spread. Use this instead of an operator-specific policy for replica spreading.

The operator intentionally does not expose every Deployment field. Deployment strategy, init containers, lifecycle hooks, and arbitrary main-container command or probe wiring beyond the fields above are not part of the v1alpha1 API.

spec.pgbouncer.config maps directly to PgBouncer ini sections:

Option	Required	Default	Description
spec.pgbouncer.config.pgbouncer	No	{auth_type: md5, auth_file: /etc/pgbouncer/userlist.txt, listen_addr: 0.0.0.0, listen_port: 6432}	Rendered as [pgbouncer]. listen_port, listen_addr, auth_file, and pidfile are global-only settings and cannot be overridden per instance.
spec.pgbouncer.config.databases	No	{"*": {}}	Rendered as [databases]. Each key is a database entry name. host/port are operator-managed per Aurora instance and ignored in both global config and instance overrides.
spec.pgbouncer.config.users	No	{}	Rendered as [users].
spec.pgbouncer.config.peers	No	{}	Rendered as [peers].

Options handled by the operator:

Key	Value	Description
[pgbouncer].listen_addr	spec.pgbouncer.config.pgbouncer.listen_addr or 0.0.0.0	Global config only; not allowed in instance overrides. Set it to an address the Kubernetes Service can reach.
[pgbouncer].listen_port	spec.pgbouncer.config.pgbouncer.listen_port or 6432	Global config only; not allowed in instance overrides. Used by the container port, Service port, and default probes.
[pgbouncer].auth_file	spec.pgbouncer.config.pgbouncer.auth_file or /etc/pgbouncer/userlist.txt	Global config only; not allowed in instance overrides. The authFileSecretRef Secret is automatically mounted at the effective path.
[pgbouncer].pidfile	spec.pgbouncer.config.pgbouncer.pidfile	Global config only; not allowed in instance overrides. Optional PgBouncer key rendered only when set.
[databases].*.host	Aurora instance endpoint	Rendered per instance by the operator. Not overridable in global config or instance overrides.
[databases].*.port	Aurora instance port	Rendered per instance by the operator. Not overridable in global config or instance overrides.

Recommended PgBouncer settings:

Key	Recommended	Description
[pgbouncer].server_login_retry	1-2 seconds	The sample CR and Helm values set 2. Keeping this low reduces Aurora failover reconnect tail by avoiding PgBouncer's upstream 15-second backend login retry delay.
[pgbouncer].ignore_startup_parameters	extra_float_digits	The sample CR and Helm values set this because several PostgreSQL drivers send extra_float_digits during startup. PgBouncer should ignore it instead of rejecting the client connection.

The operator does not validate arbitrary PgBouncer config keys. Operator-handled keys are processed separately; for example, invalid listen_port values fall back to the default port. Ordinary PgBouncer key: value entries are rendered as-is into pgbouncer.ini, so invalid PgBouncer config is expected to fail at PgBouncer startup, not at CR admission time.

[!CAUTION] Changing [pgbouncer].listen_port on an existing CR updates the generated ConfigMaps, Deployments, Pods, probes, and Services. Treat it as an operationally sensitive change in production.

spec.pgbouncer.instanceOverrides[]

Option	Required	Default	Unit	Description
spec.pgbouncer.instanceOverrides[].name	Yes	none	Aurora DB instance identifier	Matches server_id from aurora_replica_status().
spec.pgbouncer.instanceOverrides[].enabled	No	true	boolean	Set to false to intentionally exclude that discovered instance from PgBouncer resources, monitor probes, and Writer/Reader Service membership. The instance remains visible in status as disabled.
spec.pgbouncer.instanceOverrides[].replicas	No	spec.pgbouncer.replicas	replicas	PgBouncer Deployment replica count for that instance.
spec.pgbouncer.instanceOverrides[].config	No	{}	PgBouncer config section	Deep-merged on top of spec.pgbouncer.config for that instance. [pgbouncer].listen_port, [pgbouncer].listen_addr, [pgbouncer].auth_file, [pgbouncer].pidfile, [databases].*.host, and [databases].*.port are operator-handled/global-only keys and cannot be overridden per instance.

spec.services

Option	Required	Default	Unit	Description
spec.services.writer.name	No	writer	DNS label suffix	Name suffix of the Writer traffic role Service. The final name is <cr-name>-<name>.
spec.services.writer.type	No	ClusterIP	Kubernetes Service type	Writer role Service type. One of ClusterIP/NodePort/LoadBalancer.
spec.services.writer.annotations	No	{}	map[string]string	Writer role Service annotations.
spec.services.reader.name	No	reader	DNS label suffix	Name suffix of the Reader traffic role Service. The final name is <cr-name>-<name>.
spec.services.reader.type	No	ClusterIP	Kubernetes Service type	Reader role Service type. One of ClusterIP/NodePort/LoadBalancer.
spec.services.reader.annotations	No	{}	map[string]string	Reader role Service annotations.
spec.services.perInstances.type	No	ClusterIP	Kubernetes Service type	Type of every per-instance PgBouncer Service. One of ClusterIP/NodePort/LoadBalancer.
spec.services.perInstances.annotations	No	{}	map[string]string	Annotations for each per-instance Service.

Because direct per-instance targeting depends on per-instance Services, a per-instance Service is always created for discovered instances.

spec.topologyPolicy

Option	Required	Default	Unit	Description
spec.topologyPolicy.removeAfterMissingCount	No	3	count	Minimum consecutive discovery misses before a missing instance can be removed from role Service traffic.
spec.topologyPolicy.removeAfterMissingSeconds	No	10	seconds	Minimum elapsed time since the first discovery miss before a missing instance can be removed from role Service traffic. Both count and seconds must be satisfied.
spec.topologyPolicy.removedInstanceRetention	No	1h	Go duration	How long to retain a removed instance's ConfigMap/Deployment/Service before deletion.

[!NOTE] When an Aurora cluster has only one Writer and one Reader, and the application uses driver-side role checks, prefer preferSecondary or any targets for Reader connections. During Aurora failover there can be a short period with no available Reader. If the driver strictly requires secondary/read-only, connections can keep failing until the demoted instance returns as a Reader.

Secret

The operator DB auth Secret referenced by spec.discovery.authSecretRef uses the keys below. The same credentials are used for Aurora discovery and the direct DB monitor probe.

Key	Required	Default	Unit	Description
username or user	Yes	none	string	PostgreSQL user used for discovery and monitor probe queries.
password	Yes	none	string	PostgreSQL password.

The PgBouncer auth file Secret referenced by spec.pgbouncer.authFileSecretRef:

Key	Required	Default	Unit	Description
userlist.txt	Yes	none	file content	Mounted at the effective global [pgbouncer].auth_file path (default /etc/pgbouncer/userlist.txt). Uses PgBouncer-compatible password entries.

Operator

The operator is a Deployment that watches a single namespace; the default manifest uses replicas: 2 and enables leader election because this process is part of the database availability path. The default manifest is deploy/operator.yaml, and the watch target is set by --watch-namespace (defaults to the operator Pod's namespace) and --watch-names (defaults to *, all CRs in that namespace).

[!IMPORTANT] Recommended operator deployment

Internal testing confirmed that one operator can manage up to 50 CRs (=Aurora clusters) and 100 backend instances smoothly. It uses up to 10 workers per CR (WORKERS_PER_CR), and core work such as Discovery, Monitor, and Reconcile was processed without issue.

Operator flags

These are manager process flags, not CR options.

Only the public operational flags are listed below. Discovery/Monitor execution is controlled by per-CR single-flight scheduling and does not expose a global worker pool for user tuning. DB probe throughput is controlled by monitor intervals, per-probe timeout, and --workers-per-cr. The AWS API limiter remains intentionally low as a last-resort guard against accidental hot-loop bugs in the shared metadata worker.

Flag	Default	Unit	Description
--metrics-bind-address	:8080	address	Metrics bind address.
--health-probe-bind-address	:8081	address	Health/readiness probe bind address.
--leader-elect	false	boolean	Enable controller-runtime leader election. The default manifest passes this flag.
--leader-election-id	auto-detected Deployment identity, then built-in fallback	string	Leader election Lease name. Replicas of the same operator Deployment must share this value. Different operator Deployments in the same namespace should use different values.
--operator-id	auto-detected Deployment identity, then leader election ID	string	Stable identity written to CR ownership claim Leases. Same-Deployment replicas must share this value.
--operator-claim-ttl	OPERATOR_CLAIM_TTL or 60s	Go duration	Lease duration for per-CR ownership claims. Another operator may take over after this expires.
--operator-claim-renew-interval	OPERATOR_CLAIM_RENEW_INTERVAL or 10s	Go duration	Renew interval for per-CR ownership claim Leases.
--aws-region	""	AWS region	Single AWS region for RDS metadata lookups. Must match the Aurora region the managed CRs use. The default manifest sets ap-northeast-2.
--rds-metadata-refresh-interval	RDS_METADATA_REFRESH_INTERVAL or 1m	Go duration	Shared RDS metadata refresh interval for AZ/DbiResourceId enrichment. Values below 10s are clamped to 10s.
--aws-api-qps	AWS_API_QPS or 1	requests/sec	Defensive AWS API limiter for the shared metadata worker. Normal load is already bounded by cluster de-duplication and the refresh interval.
--aws-api-burst	AWS_API_BURST or 1	requests	Defensive AWS API limiter burst.
--workers-per-cr	WORKERS_PER_CR or 10	count	Maximum concurrent backend monitor probes within one CR. The monitor creates only as many worker goroutines as the current backend count requires.
--max-concurrent-reconciles	64	count	Maximum concurrent reconciles across different CRs. controller-runtime still serializes the same workqueue key, so this raises cross-CR throughput without allowing concurrent reconciles for the same CR.
--watch-namespace	WATCH_NAMESPACE	namespace	Namespace the manager watches. Required. Cluster-wide watch is not supported. The default manifest uses the operator Pod namespace.
--watch-names	WATCH_NAMES	CR names	Optional PgBouncerAurora name filter. Empty/* watches all CRs in --watch-namespace; a,b,c or repeated --watch-names=a --watch-names=b watches only those CRs.
--reconcile-min-interval	RECONCILE_MIN_INTERVAL or 1s	Go duration	Minimum interval between heavy reconciles for the same CR. This is a per-CR guard, not a global reconcile throttle.
--k8s-api-timeout	K8S_API_TIMEOUT or 10s	Go duration	Timeout for each Kubernetes API request made by the operator.
--status-refresh-min-interval	STATUS_REFRESH_MIN_INTERVAL or 5s	Go duration	Minimum refresh interval for the cached snapshot the /status dashboard shows.
--status-recent-window	STATUS_RECENT_WINDOW or 1m	Go duration	Time window used to highlight recently changed /status items. Values are clamped to 1m–24h.
--log-format	LOG_FORMAT or pretty	pretty | json	Log encoder. pretty prints a human-readable single line per event (event-centric message, hierarchical logger name, local timestamp); json prints structured JSON with an ISO 8601 timestamp for log pipelines such as CloudWatch/Datadog. klog output (leader election, etc.) is routed through the same encoder.
--zap-devel	false	boolean	Development-mode logging. Lowers the level so V(1) trace lines are shown. Independent of --log-format; leave off in production.

The operator detects its Deployment identity from POD_NAME/POD_NAMESPACE and the owner chain Pod -> ReplicaSet -> Deployment. That identity is used for the default leader election ID and per-CR ownership claims. If detection is not possible, the built-in fallback remains available and a warning is logged.

Each managed PgBouncerAurora has a namespaced Lease named pgba-claim-<cr-name> that records which operator Deployment owns it. Discovery, monitoring, RDS metadata refresh, status updates, and child-resource writes all check this claim. To manually hand a CR to another operator, stop or exclude the old operator and delete the claim Lease:

kubectl -n <namespace> delete lease pgba-claim-<cr-name>

The status field .status.operatorClaim.operatorId is updated when ownership is acquired or taken over, but heartbeat renewals are stored only in the Lease to avoid noisy CR status writes.

Claim renewal currently piggybacks on reconcile, discovery, monitor, and shared RDS metadata worker activity. Keep the effective discovery/monitor intervals at or below half of --operator-claim-ttl, or increase the claim TTL. If a CR's configured intervals exceed half the TTL, the scheduler logs a warning because a mixed deployment can otherwise see ownership ping-pong. In a handoff, takeover latency is roughly claim TTL + the other operator's next job/reconcile interval; delete the claim Lease manually when immediate handoff is required.

Role Binding

The operator runs only within its own namespace via a namespaced Role (deploy/role.yaml) — not a ClusterRole, and it does not access resources in other namespaces. A RoleBinding (deploy/rolebinding.yaml) binds this Role to the operator ServiceAccount.

Permissions the Role grants:

Resource	Verbs	Purpose
pgbouncerauroras, .../status, .../finalizers	get, list, watch, update, patch	Watch the CR and update status/finalizers.
apps/deployments	get, list, watch, create, update, patch, delete	Manage per-instance PgBouncer Deployments.
apps/replicasets	get	Detect the operator Deployment identity from the running Pod owner chain.
services, configmaps	get, list, watch, create, update, patch, delete	Manage role/instance Services and PgBouncer config.
pods	get, list, watch, patch	Service membership labeling and operator Deployment identity detection.
secrets	get	Read DB auth/userlist Secrets.
coordination.k8s.io/leases	get, list, watch, create, update, patch, delete	Leader election and per-CR ownership claims.
events	create, patch, update	Record reconcile events.

Service Account

The operator Pod runs as the pgbouncer-aurora-operator ServiceAccount (deploy/serviceaccount.yaml) and gets its permissions only through the RoleBinding above. The PgBouncer Pod's ServiceAccount is separate and set, when needed, via spec.pgbouncer.serviceAccountName (unset by default).

Status dashboard

The operator provides a built-in web dashboard to view runtime state at a glance. It is served directly by the operator process, with no separate deployment.

Path	Content
/status	Human-facing HTML dashboard
/status.json	Machine-readable JSON of the same data

• Where it is served — it rides on the metrics server, so it is reachable on --metrics-bind-address (default :8080). For a quick check, run kubectl -n <namespace> port-forward deploy/pgbouncer-aurora-operator 8080:8080 and open http://localhost:8080/status in a browser.
• Refresh interval — the dashboard shows a cached snapshot, whose minimum refresh interval is controlled by --status-refresh-min-interval (default 5s).
• Recent-change highlight — summary cards, Managed CR rows, and Conditions are highlighted when lastAppliedTime or condition lastTransitionTime falls within --status-recent-window (default 1m, clamped to 1m–24h). Periodic discovery/monitor timestamps are not used for this highlight.

What it shows:

• Top summary — number of managed CRs, Writer/Reader Service member counts, and degraded/frozen CR counts.
• Managed CRs — all CRs the operator watches with each one's state (Ready/Degraded/Frozen), generation, and Writer/Reader counts.
• Selected CR detail
  • Lifecycle — last discovery/monitor/applied times, ownership claim owner, Lease name, claim time, consecutive discovery failures, and current Writer/Reader members.
  • Services — candidates/healthy/members/ready counts per Writer/Reader role Service.
  • Instances — role/health/ready/AZ/DbiResourceId/endpoint per Instance.
  • Conditions — DiscoveryTrusted, MonitorSucceeded, Reconciled, Frozen, BackendHealthy, WriterReady, ReaderReady, ReaderFallback, ReaderFallbackSuppressed, RoleMismatch, TrafficTransitioning, Degraded, and so on.

[!WARNING] /status and /status.json have no authentication and expose topology information such as Aurora endpoints and instance identifiers as-is. Do not expose the metrics/status port to outside the cluster or to untrusted networks; restrict access with a NetworkPolicy or port-forward.

Writer and Reader Service membership convergence performance

Failover recovery performance

This operator's performance metric is "how fast an Aurora role/topology change is reflected into Writer/Reader Service membership." Overall failover performance cannot be described by a single number — role/topology change, Service membership update, and application query recovery are different stages.

This operator focuses on converging the connection path quickly through SQL discovery and Kubernetes Service membership, without waiting for Aurora endpoint DNS propagation:

Aurora Writer/Reader role change
→ Operator discovery
→ Writer/Reader Service membership update
→ EndpointSlice update
→ PgBouncer backend reconnect
→ Application reconnect / pool recycle

When a failover is detected, a 30-second failover window opens; for its duration, discovery and monitor run on a 100ms fast cadence regardless of spec.discovery.interval/spec.monitor.interval, then return to their normal cadence when the window closes.

In repeated two-instance Aurora failover tests, the observed sequence was: new Writer restart starts, old Writer restart starts, the Writer endpoint changes around the Completed failover event, the old Writer finishes restart, and the Reader endpoint changes last. The Reader endpoint changed much later than the Writer endpoint and could point at the new Writer during that lag. This is why the operator treats SQL discovery (aurora_replica_status()) as the topology source of truth and does not use Aurora Writer/Reader endpoint DNS membership as topology truth.

The table below shows app-like driver client recovery times measured with Aurora PostgreSQL 17.4, a 2-member cluster (1 Writer, 1 Reader), two PgBouncer replicas per instance, and Reader fallback enabled by default. Most clients used the Writer/Reader Kubernetes Services as a host-list, with primary for Writer connections and preferSecondary for Reader connections. Connection timeout was 3 seconds, query timeout was 5 seconds, and failed connections were discarded and retried every 1 second.

Values are averages. Writer recovery is measured by role-match recovery; Reader recovery is measured by connection availability. Driver/library versions are from the test image.

[!NOTE] These values are reference numbers from the test clients and driver settings used here. Real application recovery time depends on connection pool behavior, timeout settings, validation queries, maximum connection lifetime, retry policy, and workload shape. They also measure how quickly new connections recover — they do not mean already-established connections or in-flight transactions survive without interruption; detecting disconnects, reconnecting, and discarding wrong-role connections remain the application responsibilities described in Limitations and operating assumptions.

Client	Driver/library version	Writer recovery time	Reader recovery time	Connection handling in test
c-libpq	libpq 15.18	15.4s	5.8s	host-list + libpq target_session_attrs
go-libpq	github.com/lib/pq v1.12.3	14.0s	5.0s	host-list + libpq-style target_session_attrs
go-pgx	github.com/jackc/pgx/v5 v5.5.5	11.0s	3.6s	host-list + pgx target_session_attrs + app-level validation
java-pgjdbc	pgjdbc 42.7.7	17.2s	2.1s	host-list + pgjdbc targetServerType
node-pg	pg 8.22.0	13.0s	2.0s	split datasources + app-level validation
node-pg-native	pg-native 3.8.0	14.8s	4.8s	host-list + libpq target_session_attrs
php-pgsql	PHP 8.2.31 + pdo_pgsql/pgsql	16.6s	4.8s	host-list + libpq target_session_attrs
python-asyncpg	asyncpg 0.31.0	14.2s	5.2s	host-list + target_session_attrs
python-psycopg3	psycopg 3.3.4	16.8s	4.8s	host-list + libpq target_session_attrs
ruby-pg	pg 1.6.3	16.2s	6.0s	host-list + libpq target_session_attrs
rust-tokio-postgres	tokio-postgres 0.7.18	12.0s	4.2s	split datasources + any + app-level validation

The Reader recovery times above use the default Reader fallback together with preferSecondary/any Reader targets. Strict secondary/read-only settings can keep failing until a true Reader is back.

With a 3-member Aurora cluster, internal tests showed Reader recovery time converging to 0 seconds because an extra Reader instance not involved in the failover remained available to accept connections.

External exposure reference measurements

When exposing the Services outside the cluster, you can create NLBs through AWS Load Balancer Controller. That path has target deregistration, health-check, and propagation latency, so Aurora topology changes can take 10-20+ seconds longer than the in-cluster Kubernetes Service path. To reduce that overhead, use a short deregistration delay, connection termination on deregistration, cross-zone balancing, and fast TCP health checks:

# spec.services.writer / spec.services.reader
type: LoadBalancer
annotations:
  service.beta.kubernetes.io/aws-load-balancer-type: external
  service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
  service.beta.kubernetes.io/aws-load-balancer-scheme: internal
  service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: TCP
  service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: traffic-port
  service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "2"
  service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "2"
  service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout: "3"
  service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "5"
  service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: deregistration_delay.timeout_seconds=10,deregistration_delay.connection_termination.enabled=true,load_balancing.cross_zone.enabled=true
  service.beta.kubernetes.io/aws-load-balancer-attributes: load_balancing.cross_zone.enabled=true

These are aggressive values intended to speed up NLB target replacement and health-check convergence. In internal repeated tests, they reduced failover recovery time on the external exposure path by more than 50% compared with the default NLB settings. Lower health-check intervals increase target and network probe traffic, so tune them from the actual traffic pattern and target count in production.

Topology change cases

Beyond failover, common operational topology-change cases were measured.

Case	Service reflection	Response time
Writer deletion + failover	Switch the Writer Service to the new Writer, and exclude the deleting instance from the Reader Service as well	about 3.1-3.2s
Reader addition	Include the new Reader in the Reader Service. Until confirmed, fallback treats it as an unconfirmed Reader	about 3.1-3.2s
Reader deletion	Exclude the Reader from the Reader Service according to the missing-count policy	about 3.1s
All Readers removed or unconfirmed	Temporarily include the Writer in the Reader Service	about 0.1-0.2s

The response times above include the default 3-second discovery interval. Fallback cases that can be decided immediately from Monitor results are applied in the same reconcile without waiting for the discovery interval.

In these measurements too, the topology decision source is aurora_replica_status(). RDS metadata is an assisting layer — a refresh failure does not make discovery untrusted, and ordinary add/remove proceeds via the discovery/monitor policy. The one exception: when fresh metadata confirms a Reader is deleting, the missing debounce is skipped and traffic is detached immediately, with the evidence logged (found rds instance status deleting).

Reader Service load balancing

This distribution is per connection, not per query. This project's scope is connection distribution across Reader instances; it does not provide per-query load balancing.

The Reader Service delegates connection distribution to the default balancing of the Kubernetes Service (kube-proxy). It is important to understand the characteristics of this distribution.

• Distribution is random, not round-robin. kube-proxy's default iptables mode picks a backend probabilistically at each connection-establishment time (statistic mode random). So the unit of distribution is the connection, and once established a connection is pinned to that Reader instance until it closes.
• Small samples can skew. Random distribution is independent trials, so with few connections (within a few multiples of the instance count) load can skew onto a particular Reader. In a long-lived connection pool, this initial distribution tends to persist.
• In real production, however, it usually distributes evenly. Applications typically run pools of dozens of connections per instance, with dozens to hundreds of such instances running at once. Since the total connection count comfortably exceeds the Reader instance count, random distribution statistically converges to uniform.
• Advanced networking data paths can be considered. The default kube-proxy setup is usually sufficient at operating scale. If you need stricter distribution or higher Service processing performance, consider IPVS schedulers such as rr (round-robin) or lc (least-connection), nftables-based kube-proxy, or a Cilium eBPF kube-proxy replacement.

In repeated tests with three Readers and a simulated connection pool, distribution became even quickly as the connection count increased. The large scenario models 100 application processes, each opening 10 connections.

Scenario	Application processes	Pool size	Total connections	Reader distribution	Balance skew
small	2	10	20	31.0% / 35.0% / 34.0%	2.18x
medium	10	10	100	35.4% / 34.6% / 30.0%	1.28x
large	100	10	1,000	33.4% / 32.9% / 33.7%	1.12x

Reader distribution and Balance skew are averages from repeated tests. Balance skew is the ratio between the most-connected Reader and the least-connected Reader in each run. Values closer to 1.00x indicate more even distribution.

Development

# full local checks
./hack/check.sh

# including the Docker build
DOCKER=true ./hack/check.sh

# race detector
go test -race -count=1 ./...

The smoke script is designed to be safe by default. ./hack/smoke-test.sh only validates the manifests locally unless you pass APPLY=true. For Kubernetes API-server validation without creating resources, use DRY_RUN=server VALIDATE=true ./hack/smoke-test.sh.

Build from source (custom image)

The public install uses the published Quay image, so you do not need to build anything to run the operator. To run your own image — for development or a private registry — build and push a multi-arch image, then point the Deployment at it instead of the Quay image:

docker buildx build --platform linux/amd64,linux/arm64 \
  -t <registry>/pgbouncer-aurora-operator:<tag> \
  --push .

kubectl -n pgbouncer-aurora set image deploy/pgbouncer-aurora-operator \
  manager=<registry>/pgbouncer-aurora-operator:<tag>

Contributing

See CONTRIBUTING.md for contribution, versioning, and release policies.

Contributions are welcome. The process is intentionally simple for now:

1. For non-trivial behavior changes, open an issue first or draft the intended change.
2. Keep changes small and focused.
3. Add/update tests for controller/discovery/monitor/planner/render behavior.
4. Run ./hack/check.sh before a PR.
5. Update the README when CRD options or user-visible behavior change.

License

This project is distributed under the ISC License. See LICENSE.