obque

README

obque

Object Store Queue

obque is a Go queue implementation backed by a single queue.json object in object storage, using optimistic concurrency (CAS) and a stateless broker that group-commits operations. It supports pluggable object store adapters (S3, GCS, in-memory) and a Go client library.

Status: experimental (APIs and wire formats may change).

Components

• cmd/obque-broker: HTTP broker that owns writes to queue.json (CAS group commit).
• cmd/obque-producer: Small producer CLI (useful for demos/tests).
• cmd/obque-worker: Small worker CLI (useful for demos/tests).
• cmd/obque-inspect: Prints queue.json from the object store (useful for debugging).
• client: Go client that talks to the broker and resolves the active broker by reading queue.json.
• store/*: Object-store adapters (S3, GCS) and an in-memory adapter for tests.

Install

go get github.com/hasit/obque@latest

Semantics

• FIFO claim order (earliest claimable job first).
• At-least-once delivery (leases can expire; jobs may be re-claimed).
• Explicit Ack to complete a job.
• Heartbeats extend the lease.

Authentication (supported today)

This repo currently implements lightweight auth flows (no cloud SDK dependency).

S3

Supported credential sources:

• Environment variables:
  • AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY)
  • AWS_SECRET_ACCESS_KEY (or AWS_SECRET_KEY)
  • optional: AWS_SESSION_TOKEN
• Shared credentials file:
  • ~/.aws/credentials (or AWS_SHARED_CREDENTIALS_FILE)
  • profile via AWS_PROFILE (default: default)

Region:

• Provide --region or set AWS_REGION / AWS_DEFAULT_REGION.

Minimum permissions:

• s3:GetObject and s3:PutObject on the queue.json object key.

How to get credentials (local dev):

• Install/configure AWS CLI, then run aws configure (writes ~/.aws/credentials), or export the env vars above.
• Avoid committing long-lived credentials; prefer short-lived credentials (session tokens) where possible.

Not supported yet:

• EC2/ECS metadata credential chains, assume-role, web identity, etc.

GCS

Supported token sources:

• Service account JSON key via GOOGLE_APPLICATION_CREDENTIALS (recommended for local dev).
• User ADC via gcloud auth application-default login (writes an authorized-user refresh token to the ADC file).
• GCE metadata server token endpoint (works when running on GCE / runtimes with metadata enabled and a service account attached).

Minimum permissions:

• Read + write the queue.json object (e.g. storage.objects.get + storage.objects.create/storage.objects.update on that object).

How to get credentials (local dev):

• Create a service account with permissions to read/write the queue.json object, create a JSON key, download it, then:
  • export GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json
• Or (developer workstation): install gcloud, then run:
  • gcloud auth application-default login
  • This writes application_default_credentials.json (usually ~/.config/gcloud/application_default_credentials.json, or under CLOUDSDK_CONFIG if set).
• Avoid committing service account keys; treat them like passwords.

Not supported yet:

• Workload identity federation, impersonation, etc.

Quickstart (broker)

In-memory (no cloud)

go run ./cmd/obque-broker \
  --listen=:8080 \
  --advertise=http://localhost:8080

S3

go run ./cmd/obque-broker \
  --uri=s3://YOUR_BUCKET/queue.json \
  --region=us-east-1 \
  --listen=:8080 \
  --advertise=http://localhost:8080

GCS

go run ./cmd/obque-broker \
  --uri=gs://YOUR_BUCKET/queue.json \
  --listen=:8080 \
  --advertise=http://localhost:8080

Basic client usage

See examples/basic and examples/multi.

Run examples (in-process, no cloud)

go run ./examples/basic
go run ./examples/multi --jobs=100 --workers=4

Multi-process usage (running broker + separate workers)

Start a broker:

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json

go run ./cmd/obque-broker \
  --uri=gs://YOUR_BUCKET \
  --listen=:8080 \
  --advertise=http://localhost:8080

Start one or more workers in separate terminals:

go run ./cmd/obque-worker \
  --uri=gs://YOUR_BUCKET \
  --worker-id=worker-1 \
  --claim-wait=10s \
  --work=200ms \
  --heartbeat=50ms

Push jobs (from another terminal):

go run ./cmd/obque-producer \
  --uri=gs://YOUR_BUCKET \
  --count=10 \
  --payload="hello {i}"

Observe queue.json while work is in progress:

go run ./cmd/obque-producer --uri=gs://YOUR_BUCKET --count=3 --payload='{"work_ms":10000,"msg":"job {i}"}'

# In another terminal, watch the queue file (repeat a few times):
go run ./cmd/obque-inspect --uri=gs://YOUR_BUCKET