outboxer

README

Outboxer

Outboxer is a standalone worker application that reads events from a PostgreSQL database and sends them to AWS SQS or GCP Pub/Sub using the outbox pattern. This lets you create events from within your database transactions, making sure that SQS and Pub/Sub see your events in the same order in which they are serialized. Outboxer guarantees at-least-once delivery, ensuring that no event is ever lost.

Outboxer is high throughput, low latency, easy to operate, and integrates nicely with the GCP and AWS platforms. All event properties and options of SQS and Pub/Sub are supported.

Outboxer is best deployed as a Kubernetes component, a GCP Cloud Run service, or an AWS ECS service. It is configurable with environment variables. It is implemented in Go and published as a small Docker image.

ghcr.io/fvdsn/outboxer:latest

Basic Configuration and Usage

In PostgreSQL, you need to create a table with at least an id column and a payload column. The id determines event ordering and idempotency. The payload is passed as-is to the event backend. The table and column names are configurable in Outboxer. Other columns enable routing and backend-specific options for more advanced scenarios — see docs/events.md.

CREATE TABLE events (
    id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
    payload text NOT NULL
);

Then configure Outboxer to connect to that table and to the queue or topic of your choice.

resource "google_cloud_run_v2_service" "outboxer" {
  name     = "outboxer"
  location = "europe-west1"

  template {
    scaling {
      min_instance_count = 1
    }

    containers {
      image = "ghcr.io/fvdsn/outboxer:latest"

      args = [
        "--pubsub-enabled",
        "--default-pubsub-topic=projects/my-gcp-project/topics/events",
        "--pg-host=10.0.0.5",
        "--pg-user=outboxer",
        "--pg-password=change-me",
        "--pg-database=app",
      ]

      resources {
        cpu_idle = false
      }
    }
  }
}

Publishing an event is then a matter of inserting a row into the events table. Outboxer sends the event and deletes the row once the send is complete.

INSERT INTO events (payload)
VALUES ('{"type":"user.created","id":"123"}');

Note that the Outboxer must always run as at least one active instance. It continuously polls from a single PostgreSQL connection.

Supported Features

• GCP Pub/Sub
  • Ordering keys and ordered publishing
  • Custom attributes
  • Multiple topics
  • Default topic
  • Keyless publishing from GCP or AWS via Google external account credentials
• AWS SQS
  • Standard queues with batch sends
  • FIFO queues and message groups
  • Stable deduplication IDs from event IDs
  • Custom attributes
  • Multiple queues
  • Default queue
  • AWS role assumption
  • Keyless publishing from AWS or GCP via web identity federation
• Multi instances
  • Split per backend (GCP / AWS)
  • Split per queue / topic
• Outbox processing
  • At-least-once delivery
  • Ordered processing by event ID
  • Transactional delete after confirmed publish
  • Configurable table and column names
  • Dynamic routing with target and destination
  • Backend-specific event options (ordering keys, attributes, FIFO groups)
  • Optional destination ownership with PUBSUB_DESTINATIONS and SQS_DESTINATIONS
  • Safe multi-instance deployments when split by backend or destination
  • PostgreSQL dead letter queue for poison events
  • Age-based poison handling with MAX_EVENT_AGE_MS
  • Periodic operational statistics logging
  • Graceful shutdown, health endpoint, timeouts, and watchdog

Dead Letter Queue

Events that are permanently unsendable (oversized payloads, malformed options, expired events, and similar content poison) can be routed to a PostgreSQL dead letter table instead of being deleted. Set DLQ_TABLE to enable it. Routing failures and retryable provider errors are never dead-lettered — they stay in the outbox table for a later retry. See docs/dlq.md.

Documentation

• Events — table schema, routing, and backend-specific options
• Configuration — full environment variable and flag reference
• Logging — log catalog, levels, and the periodic statistics fields
• Dead Letter Queue — poison handling and the DLQ table
• Authentication — credentials and cross-cloud workload identity
• Deployment — Cloud Run, GKE, ECS Fargate, and EKS examples