outboxer

README

Outboxer

Outboxer is a standalone worker application that reads events from a PostgreSQL database and sends them to AWS SQS or GCP Pub/Sub using the outbox pattern. This lets you create events from within your database transactions, making sure that SQS and Pub/Sub see your events in the same order in which they are serialized. Outboxer guarantees at-least-once delivery, ensuring that no event is ever lost.

Outboxer is high throughput, low latency, easy to operate, and integrates nicely with the GCP and AWS platforms. All event properties and options of SQS and Pub/Sub are supported.

Outboxer is best deployed as a Kubernetes component, a GCP Cloud Run service, or an AWS ECS service. It is configurable with environment variables. It is implemented in Go and published as a small Docker image.

ghcr.io/fvdsn/outboxer:latest

Basic Configuration and Usage

In PostgreSQL, you need to create a table with at least an id column and a payload column. The id determines event ordering and idempotency. The payload is passed as-is to the event backend. The table and column names are configurable in Outboxer. Other columns can be configured for more advanced scenarios.

CREATE TABLE events (
    id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
    payload text NOT NULL
);

Then configure Outboxer to connect to that table and to the queue or topic of your choice.

resource "google_cloud_run_v2_service" "outboxer" {
  name     = "outboxer"
  location = "europe-west1"

  template {
    scaling {
      min_instance_count = 1
    }

    containers {
      image = "ghcr.io/fvdsn/outboxer:latest"

      args = [
        "--pubsub-enabled",
        "--default-pubsub-topic=projects/my-gcp-project/topics/events",
        "--pg-host=10.0.0.5",
        "--pg-user=outboxer",
        "--pg-password=change-me",
        "--pg-database=app",
      ]

      resources {
        cpu_idle = false
      }
    }
  }
}

Publishing an event is then a matter of inserting a row into the events table. Outboxer sends the event and deletes the row once the send is complete.

INSERT INTO events (payload)
VALUES ('{"type":"user.created","id":"123"}');

Note that the Outboxer must always run as at least one active instance. It continuously polls from a single PostgreSQL connection.

Supported Features

• GCP Pub/Sub
  • Ordering keys and ordered publishing
  • Custom attributes
  • Multiple topics
  • Default topic
  • Keyless publishing from GCP or AWS via Google external account credentials
• AWS SQS
  • Standard queues with batch sends
  • FIFO queues and message groups
  • Stable deduplication IDs from event IDs
  • Custom attributes
  • Multiple queues
  • Default queue
  • AWS role assumption
  • Keyless publishing from AWS or GCP via web identity federation
• Multi instances
  • Split per backend (GCP / AWS)
  • Split per queue / topic
• Outbox processing
  • At-least-once delivery
  • Ordered processing by event ID
  • Transactional delete after confirmed publish
  • Configurable table and column names
  • Dynamic routing with target and destination
  • Optional destination ownership with PUBSUB_DESTINATIONS and SQS_DESTINATIONS
  • Safe multi-instance deployments when split by backend or destination
  • Graceful shutdown, health endpoint, timeouts, and watchdog