cells

package
v0.64.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 11, 2026 License: Apache-2.0 Imports: 22 Imported by: 0

Documentation

Overview

Package cells implements the synchronous routing plane for cell-based development: routing each tenant to an isolated cell with a fail-safe default, rejecting calls for a tenant that is moving, and enforcing the current route epoch cell-side. It is isolation, not load balancing — the router is a tenant→cell directory, and a tenant is pinned to exactly one cell at a time.

Safety derives from a monotonic per-tenant route epoch and per-instance admission tokens, never from clocks, and is enforced in depth (no single barrier is trusted):

  • RoutingTable is the single source of truth (Get/CompareAndSet/Watch). MemTable is the dev/test backend; a Raft/etcd adapter implements the same interface in a later module. Every mutation is an idempotent compare-and-swap and route epochs never decrease.

  • Router (L1) is a watch-fed cache that resolves a tenant to a Decision, falling back to the default cell for unknown tenants and when the table is unreachable (reads stay available; the Stale flag lets writes fail closed).

  • GateRegistry (L2) is the real correctness barrier: a per-instance, per-tenant admission gate (TryEnter/Leave/CloseForBarrier) that handlers and background workers alike must pass. GateController keeps gates aligned with the table via its watch.

  • UnaryServerInterceptor/StreamServerInterceptor/HTTPMiddleware wire L1+L2 into the request path: route, stamp metadata, reject mid-move (gRPC UNAVAILABLE/ABORTED, HTTP 503 + Retry-After), and admit through the gate.

  • MoveController drives the safe tenant-move protocol: a forward-only route epoch advances across quiesce → drain → fence → event-pause → commit, every transition an idempotent CAS, with rollback, crash recovery (the table is the recovery state), a drain deadline (liveness only), and a BudgetMeter. Fencer (L3) and EventBarrier (L4) are the storage- and event-plane barriers it drives; the persistence adapters supply the concrete backends, and MemFencer/MemEventBarrier are the in-memory references. Campaign schedules moves for rebalances; PlacementPolicy decides placement.

  • MemTable and FileTable are the in-memory and file-backed routing tables (the latter lets a CLI and a running service share routes); a Raft/etcd or CR/GitOps adapter implements the same interface for production.

Deferred and documented as the pluggable layer: data-owning cells' data catch-up (Phase 6 is a no-op for compute-only shared-DB cells), per-tenant PAUSE/DRAIN_QUEUE enforcement inside the event relay, and the etcd/CR routing-table backends. The model in spec/CellMove.tla checks the safety invariants of the move protocol.

Index

Constants

View Source 
const DefaultCellID = "default"

DefaultCellID is the fail-safe cell that serves any tenant lacking an explicit route (and any tenant whose route cannot be resolved). It matches middleware.DefaultCellID so a service that adopts cell routing with no routes populated behaves exactly as before: every tenant lands on "default".

View Source 
const DefaultMonthlyBudget = 130 * time.Second

DefaultMonthlyBudget is the default per-tenant unavailability allowance for one calendar month. 130s/month is roughly a 99.995% monthly availability target — the headroom a drain-and-cutover move spends on the brief reject window.

Variables

View Source 
var (
	// ErrTenantDraining means the tenant's gate is not accepting new work — it is
	// draining for a move barrier or closed because the tenant moved away.
	ErrTenantDraining = errors.New("cells: tenant gate not accepting new work")
	// ErrStaleRouteEpoch means the request's route epoch does not match the gate's
	// current epoch — a fencing conflict (the caller raced a move).
	ErrStaleRouteEpoch = errors.New("cells: stale route epoch")
)

Gate-admission errors.

View Source 
var (
	// ErrNoRouteToMove means a move was requested for a tenant with no existing
	// route. First placement goes through [MoveController.Assign], not Move.
	ErrNoRouteToMove = errors.New("cells: cannot move a tenant with no route (use Assign for first placement)")
	// ErrMoveDeadline means a move's liveness deadline elapsed before the source
	// drained or the event plane quiesced; the controller rolls the tenant back.
	ErrMoveDeadline = errors.New("cells: move deadline elapsed before the cut")
	// ErrSameCell means From and To name the same cell — nothing to move.
	ErrSameCell = errors.New("cells: move source and target are the same cell")
)

Move-controller errors.

View Source 
var (
	// ErrNoRoute means the tenant has no explicit route; the caller falls back to
	// the default cell.
	ErrNoRoute = errors.New("cells: no route for tenant")
	// ErrCASConflict means the CompareAndSet precondition (expected epoch+state)
	// did not match the stored route.
	ErrCASConflict = errors.New("cells: compare-and-set conflict")
	// ErrEpochRegression means a CompareAndSet attempted to lower a tenant's
	// RouteEpoch — forbidden (invariant 7: epochs never decrease).
	ErrEpochRegression = errors.New("cells: route epoch must not decrease")
)

Routing-table errors.

View Source 
var ErrBudgetExceeded = errors.New("cells: move would exceed tenant unavailability budget")

ErrBudgetExceeded is returned when a move would spend more of a tenant's monthly unavailability budget than remains — a non-forced move is refused rather than risk breaching the availability target.

View Source 
var ErrFenceRegression = errors.New("cells: fence epoch must not decrease")

ErrFenceRegression is returned when a Fencer or EventBarrier is asked to move a tenant's epoch backwards — forbidden (invariant 7: epochs never decrease).

View Source 
var ErrNoCells = errors.New("cells: no candidate cells for placement")

ErrNoCells is returned by a PlacementPolicy asked to place a tenant when no candidate cells were offered.

Functions

func DefaultIsMutating 

func DefaultIsMutating(fullMethod string) bool

DefaultIsMutating classifies a gRPC full method as mutating unless its local name begins with a known AIP read prefix. Used to fail closed for writes while keeping reads available when a tenant's route is uncertain.

func HTTPMiddleware 

func HTTPMiddleware(router *Router, opts ...Option) func(http.Handler) http.Handler

HTTPMiddleware returns L1 routing middleware for an HTTP handler (e.g. the grpc-gateway mux or an HTTP-native service). It rejects a moving tenant with 503 + Retry-After and fails closed for writes under route uncertainty. Full L2 admission for HTTP-native handlers is done by calling the gate directly.

func StreamServerInterceptor 

func StreamServerInterceptor(router *Router, gates *GateRegistry, opts ...Option) grpc.StreamServerInterceptor

StreamServerInterceptor returns the streaming counterpart of UnaryServerInterceptor. It applies the route/reject/admission decision once at stream start and holds the admission token for the stream's lifetime.

func UnaryServerInterceptor 

func UnaryServerInterceptor(router *Router, gates *GateRegistry, opts ...Option) grpc.UnaryServerInterceptor

UnaryServerInterceptor returns a gRPC unary interceptor that routes by tenant: it resolves the tenant's cell, rejects calls for a moving tenant, defends against a stale upstream router (wrong cell), fails closed for writes under route uncertainty, and — when gates is non-nil — acquires/releases an L2 admission token around the handler. A request with no tenant scope passes through unchanged. With no routes populated and the default cell, it is a no-op.

Types

type AdmissionToken 

type AdmissionToken struct {
	TenantID     string
	CellID       string
	InstanceID   string
	RouteEpoch   uint64
	AdmissionSeq uint64 // strictly increasing per tenant on this instance
	BarrierID    string
}

AdmissionToken is issued by GateRegistry.TryEnter for one admitted unit of tenant work. It carries the fencing identity a later storage/event layer (L3/L4) uses; in this routing slice it bounds the in-flight set and the barrier cutoff.

func AdmissionTokenFromContext 

func AdmissionTokenFromContext(ctx context.Context) (AdmissionToken, bool)

AdmissionTokenFromContext returns the L2 admission token for the in-flight request, if the routing interceptor admitted it.

type BudgetMeter added in v0.30.0 

type BudgetMeter struct {
	// contains filtered or unexported fields
}

BudgetMeter is a per-tenant monthly unavailability ledger. The move controller debits a tenant's budget by the unavailability a move costs (the reject window) and refuses a non-forced move whose estimate would breach the remainder. The ledger resets at each calendar-month boundary, observed through an injected clock so tests are deterministic.

It is concurrency-safe: a campaign may consult and record many tenants at once.

func NewBudgetMeter added in v0.30.0 

func NewBudgetMeter(opts ...BudgetOption) *BudgetMeter

NewBudgetMeter returns a meter with the default budget and clock unless overridden by options.

func (*BudgetMeter) Allowed added in v0.30.0 

func (m *BudgetMeter) Allowed(tenantID string, estimate time.Duration) bool

Allowed reports whether spending estimate would stay within tenantID's remaining budget this month.

func (*BudgetMeter) Record added in v0.30.0 

func (m *BudgetMeter) Record(tenantID string, spent time.Duration)

Record debits tenantID's budget by spent (the unavailability a completed move actually cost). Negative spends are ignored.

func (*BudgetMeter) Remaining added in v0.30.0 

func (m *BudgetMeter) Remaining(tenantID string) time.Duration

Remaining reports how much unavailability budget tenantID has left this month. It never returns negative.

type BudgetOption added in v0.30.0 

type BudgetOption func(*BudgetMeter)

BudgetOption configures a BudgetMeter.

func WithMeterClock added in v0.30.0 

func WithMeterClock(now func() time.Time) BudgetOption

WithMeterClock injects the clock the meter reads for month boundaries (default time.Now). Tests pass a controllable clock so resets are deterministic.

func WithTenantBudget added in v0.30.0 

func WithTenantBudget(d time.Duration) BudgetOption

WithTenantBudget overrides the per-tenant monthly budget (default DefaultMonthlyBudget).

type Campaign added in v0.30.0 

type Campaign struct {
	// contains filtered or unexported fields
}

Campaign realizes a CampaignPlan as a sequence (or bounded-concurrency set) of moves driven by a MoveController. It is abortable via ctx, budget-aware (a tenant whose move would breach budget is skipped unless Force is set), and idempotent on re-run.

func NewCampaign added in v0.30.0 

func NewCampaign(ctrl *MoveController) *Campaign

NewCampaign builds a campaign over ctrl. If ctrl has a budget meter the campaign uses it to pre-skip over-budget tenants (the controller still enforces it).

func (*Campaign) Run added in v0.30.0 

func (c *Campaign) Run(ctx context.Context, plan CampaignPlan) (CampaignResult, error)

Run realizes plan. For each tenant whose current cell differs from its target it drives a move; tenants already on target are skipped, budget-blocked tenants are skipped (unless plan.Force), and a cancelled ctx stops launching further moves. The result tallies moved/skipped/failed; Run returns a non-nil error only when at least one move failed (the per-tenant causes are in CampaignResult.Failed).

type CampaignPlan added in v0.30.0 

type CampaignPlan struct {
	Assignments   map[string]string // tenant → desired cell
	MaxConcurrent int               // bound on simultaneous moves (<=1 ⇒ sequential)
	Operator      string
	Force         bool // bypass the budget gate for every member
}

CampaignPlan is a desired tenant→cell placement to realize across many tenants (e.g. an even-distribution rebalance). Only tenants whose current cell differs from their target are moved; the rest are skipped, so a campaign is idempotent and resumable — re-running converges without re-moving placed tenants.

func PlanFromPolicy added in v0.30.0 

func PlanFromPolicy(ctx context.Context, tenants []string, cells []string, policy PlacementPolicy) (CampaignPlan, error)

PlanFromPolicy builds a CampaignPlan by asking policy to place each tenant across cells — the even-distribution rebalance helper. The plan is sequential (MaxConcurrent 1) by default; callers raise it. Placement is computed in sorted tenant order so a deterministic policy (e.g. round-robin) yields a stable plan.

type CampaignResult added in v0.30.0 

type CampaignResult struct {
	Moved   []string          // tenants successfully moved to their target
	Skipped map[string]string // tenant → reason (already placed, budget, ctx cancelled)
	Failed  map[string]error  // tenant → move error
}

CampaignResult records the per-tenant outcome of a Campaign.Run.

type CohortMoveError added in v0.30.0 

type CohortMoveError struct {
	CohortID   string
	FailedAt   string           // the tenant whose move failed
	Cause      error            // the underlying failure
	RolledBack []string         // members rolled back as a result (best-effort)
	Errors     map[string]error // per-tenant errors encountered during rollback
}

CohortMoveError aggregates the per-member outcome of a MoveController.MoveCohort.

func (*CohortMoveError) Error added in v0.30.0 

func (e *CohortMoveError) Error() string

Error implements error.

func (*CohortMoveError) Unwrap added in v0.30.0 

func (e *CohortMoveError) Unwrap() error

Unwrap exposes the underlying cause.

type ControllerOption added in v0.30.0 

type ControllerOption func(*MoveController)

ControllerOption configures a MoveController.

func WithBudgetMeter added in v0.30.0 

func WithBudgetMeter(m *BudgetMeter) ControllerOption

WithBudgetMeter attaches a BudgetMeter. When set, a non-forced move whose estimated cost would breach the tenant's remaining monthly budget is refused with ErrBudgetExceeded, and a completed move's cost is recorded.

func WithClock added in v0.30.0 

func WithClock(now func() time.Time) ControllerOption

WithClock injects the controller's clock (default time.Now). Used for move deadlines and budget accounting; tests pass a controllable clock.

func WithDrainDeadline added in v0.30.0 

func WithDrainDeadline(d time.Duration) ControllerOption

WithDrainDeadline sets how long a move may spend draining/quiescing before the controller forces a rollback (default 30s). It is liveness, never safety.

func WithDrainer added in v0.30.0 

func WithDrainer(d Drainer) ControllerOption

WithDrainer sets the L2 Drainer. nil ⇒ no local drain coordination.

func WithEventBarrier added in v0.30.0 

func WithEventBarrier(b EventBarrier) ControllerOption

WithEventBarrier sets the L4 EventBarrier. nil ⇒ event phases are no-ops.

func WithFencer added in v0.30.0 

func WithFencer(f Fencer) ControllerOption

WithFencer sets the L3 storage Fencer. nil ⇒ fencing is a no-op.

func WithIDGenerator added in v0.30.0 

func WithIDGenerator(gen func() string) ControllerOption

WithIDGenerator injects the barrier-ID generator (default: a per-controller monotonic counter). Tests pass a deterministic generator.

func WithPreflight added in v0.30.0 

func WithPreflight(fn func(ctx context.Context, plan MovePlan) error) ControllerOption

WithPreflight installs a hook run at Phase 0 of every move; a non-nil error aborts the move before any state changes (default: always allow).

type Cutoff 

type Cutoff struct {
	TenantID     string
	InstanceID   string
	BarrierEpoch uint64
	MaxSeq       uint64 // admissions with seq <= MaxSeq were let in before the barrier
	Drained      bool   // in-flight reached zero before the deadline
	Forced       bool   // the deadline fired with work still in flight (caller must fence stragglers)
}

Cutoff is the result of closing a gate for a barrier: the max admission_seq admitted before the barrier on this instance, and whether the gate fully drained before the deadline.

type Decision 

type Decision struct {
	TenantID   string
	Cell       string // the cell to route to (the default cell when Known is false)
	RouteEpoch uint64 // the epoch to admit at; 0 for default-cell tenants
	State      State
	AdmitNew   bool // the state admits new calls (false while the tenant is moving)
	IsDefault  bool // resolved to the fail-safe default cell
	Known      bool // an explicit route exists (false = default fallback)
	Stale      bool // served under uncertainty (table unreachable on a miss, or aged-out cache)
}

Decision is the outcome of resolving a tenant to a cell. It is consumed by the L1 middleware to route, stamp metadata, admit, or reject.

type Drainer added in v0.30.0 

type Drainer interface {
	// CloseAndDrain stops new admissions for tenantID and waits for in-flight work
	// to clear, returning the [Cutoff] (Forced when the deadline fired first). The
	// barrier epoch fences late admissions.
	CloseAndDrain(ctx context.Context, tenantID string, barrierEpoch uint64) (Cutoff, error)
}

Drainer closes a tenant's source gate for a move barrier and blocks until the in-flight set drains or the barrier deadline forces the cut. It is the move controller's view of the L2 admission gate (Phase 2). A nil Drainer means the controller does not coordinate a local drain (e.g. drains happen out of band).

type EventBarrier added in v0.30.0 

type EventBarrier interface {
	// SetPolicy applies the publisher mode for a tenant at eventEpoch:
	//   Phase 5 → PolicyPause or PolicyDrainQueue at the barrier epoch;
	//   Phase 7 → PolicyNormal at the new (post-move) epoch.
	// Idempotent; eventEpoch is forward-only.
	SetPolicy(ctx context.Context, tenantID string, policy EventPolicy, eventEpoch uint64) error
	// Drained reports whether the source publisher has flushed or paused every
	// event up to the barrier — the liveness check the controller waits on before
	// committing. A compute-only / event-free tenant is always drained.
	Drained(ctx context.Context, tenantID string, eventEpoch uint64) (bool, error)
}

EventBarrier is the L4 event/outbox plane as the move controller sees it. The controller pauses or drains a tenant's publisher at the barrier (Phase 5) and resumes NORMAL publishing at the new epoch after the cut (Phase 7). The concrete implementation is backed by the transactional outbox + relay.

A nil EventBarrier means the service emits no events; the controller treats the event phases as no-ops.

type EventPolicy added in v0.30.0 

type EventPolicy uint8

EventPolicy is the publisher mode for a tenant's outbox during the event plane (L4). At rest a tenant is NORMAL; a move pauses or drains its publisher so no stale-epoch event escapes after the cut. 

const (
	// PolicyNormal publishes to the normal destination at the current EventEpoch.
	PolicyNormal EventPolicy = iota
	// PolicyPause stops sending but keeps writing the outbox durably (never
	// drops): the relay halts; if the backlog grows past a threshold, new tenant
	// writes are rejected with a retryable error rather than lose events.
	PolicyPause
	// PolicyDrainQueue sends to a drain/quarantine destination during a move; the
	// envelope preserves RouteEpoch/EventEpoch/SourceCell/BarrierID so a replay
	// can merge by (tenant_id, event_seq).
	PolicyDrainQueue
)

func (EventPolicy) String added in v0.30.0 

func (p EventPolicy) String() string

String returns the canonical name of the event policy.

type Fencer added in v0.30.0 

type Fencer interface {
	// Seal blocks ALL tenant-scoped writes for tenantID for the duration of the
	// move barrier (Phase 3): no cell may mutate the tenant until SetOwner lifts
	// it. Idempotent; barrierEpoch is forward-only ([ErrFenceRegression] if it
	// regresses).
	Seal(ctx context.Context, tenantID string, barrierEpoch uint64) error
	// SetOwner installs (ownerCell, routeEpoch) as the only writer allowed and
	// lifts any seal (Phase 7 commit, or rollback). Idempotent; routeEpoch is
	// forward-only.
	SetOwner(ctx context.Context, tenantID, ownerCell string, routeEpoch uint64) error
}

Fencer is L3 storage fencing as the move controller sees it: it sets the authoritative (owner cell, route epoch) that the data layer's write-guard enforces on every tenant-scoped mutation. The write-guard itself lives in the persistence adapter (ent mixin / gorm callback) and rejects any write whose admission token (cell + route epoch, from AdmissionTokenFromContext) does not match the current fence — so a stale or zombie writer from the old cell is rejected at the row, even if it slipped past L1/L2.

A nil Fencer means compute-only with no adversarial fencing required (the route epoch + the L2 gate are the barrier); the controller treats fencing as a no-op.

type FileTable added in v0.30.0 

type FileTable struct {
	// contains filtered or unexported fields
}

FileTable is a JSON-file-backed RoutingTable: a persistent, dependency-light (stdlib-only) backend suitable for local development, where the `devedge cell` CLI and a running service share one routes file. Writes are guarded by a cross-process advisory lock and applied atomically (write-temp + rename), so a CompareAndSet is safe even when several processes share the file. Watch is poll-based (it re-reads the file on an interval) since the writer may be a different process.

Correctness never depends on the watch being instantaneous: the Router lazily re-Gets on a cache miss and every cell re-checks the epoch at L2/L3. A Raft/etcd-backed table implements the same interface for production.

func NewFileTable added in v0.30.0 

func NewFileTable(path string, opts ...FileTableOption) *FileTable

NewFileTable returns a routing table persisted at path. The file is created on the first write; a missing file reads as an empty table.

func (*FileTable) CompareAndSet added in v0.30.0 

func (t *FileTable) CompareAndSet(_ context.Context, expect, next TenantRoute) error

CompareAndSet implements RoutingTable with the same semantics as MemTable: the precondition is checked against the stored route's (RouteEpoch, State); creation requires expect to be the zero route and the tenant to be absent; a lower next.RouteEpoch is rejected with ErrEpochRegression.

func (*FileTable) Delete added in v0.30.0 

func (t *FileTable) Delete(_ context.Context, tenantID string) error

Delete removes a tenant's route (turning off a cell reverts its tenants to the default cell). Idempotent. Not part of RoutingTable — an admin affordance.

func (*FileTable) Get added in v0.30.0 

func (t *FileTable) Get(_ context.Context, tenantID string) (TenantRoute, error)

Get implements RoutingTable.

func (*FileTable) List added in v0.30.0 

func (t *FileTable) List(_ context.Context) ([]TenantRoute, error)

List returns a snapshot of every route (admin/status affordance; not part of RoutingTable). Used by `devedge cell status`.

func (*FileTable) Watch added in v0.30.0 

func (t *FileTable) Watch(ctx context.Context) (<-chan RouteEvent, error)

Watch implements RoutingTable by polling the file. It emits a RouteEvent for every tenant whose route changed since the previous poll, and a Deleted event for a tenant whose route was removed. Like MemTable, it does not replay the state that already exists when the watch starts (the Router lazily Gets on a miss). The channel is closed when ctx is cancelled.

type FileTableOption added in v0.30.0 

type FileTableOption func(*FileTable)

FileTableOption configures a FileTable.

func WithPollInterval added in v0.30.0 

func WithPollInterval(d time.Duration) FileTableOption

WithPollInterval sets how often watches re-read the file (default 500ms).

type GateController 

type GateController struct {
	// contains filtered or unexported fields
}

GateController keeps a GateRegistry's gates aligned with the routing table by consuming its watch stream and calling GateRegistry.Reconcile for each change. This is how a cell enforces the *current* tenant epoch: a tenant that moves away has its gate closed here, so a stale upstream router cannot get work admitted.

It reconciles forward from the watch (no bulk enumeration): tenants not yet seen are admitted via TryEnter's lazy open at their resolved epoch, and corrected the moment a route change for them arrives.

func NewGateController 

func NewGateController(reg *GateRegistry, table RoutingTable, defaultCell string) *GateController

NewGateController binds a registry to a table. defaultCell names the fail-safe cell so a deleted route reverts correctly: the default cell reopens the tenant, any other cell closes it.

func (*GateController) Start 

func (c *GateController) Start(ctx context.Context) error

Start begins watching and reconciling until ctx is cancelled. It returns once the watch is established; the background loop reconciles thereafter.

type GateDrainer added in v0.30.0 

type GateDrainer struct {
	// contains filtered or unexported fields
}

GateDrainer adapts a GateRegistry to the Drainer interface so the move controller can drive the local L2 barrier directly.

func NewGateDrainer added in v0.30.0 

func NewGateDrainer(reg *GateRegistry) *GateDrainer

NewGateDrainer wraps reg as a Drainer.

func (*GateDrainer) CloseAndDrain added in v0.30.0 

func (d *GateDrainer) CloseAndDrain(ctx context.Context, tenantID string, barrierEpoch uint64) (Cutoff, error)

CloseAndDrain implements Drainer by driving GateRegistry.CloseForBarrier. It returns ctx.Err() alongside the Cutoff so a forced (deadline) cut surfaces as an error the controller can act on, while a clean drain returns nil.

type GateRegistry 

type GateRegistry struct {
	// contains filtered or unexported fields
}

GateRegistry holds the per-tenant admission gates for one service instance in one cell. It is the L2 correctness barrier: every path into tenant work — edge handlers, internal calls, background workers, relays — must acquire admission through it. The L1 middleware does so automatically for gRPC; workers call GateRegistry.TryEnter/GateRegistry.Leave directly.

func NewGateRegistry 

func NewGateRegistry(cellID, instanceID string) *GateRegistry

NewGateRegistry returns a registry for the given cell and instance identity.

func (*GateRegistry) CellID 

func (gr *GateRegistry) CellID() string

CellID returns the cell this registry's instance belongs to.

func (*GateRegistry) CloseForBarrier 

func (gr *GateRegistry) CloseForBarrier(ctx context.Context, tenantID string, barrierEpoch uint64) Cutoff

CloseForBarrier flips tenantID's gate to draining (no new admissions), records the cutoff sequence, and blocks until in-flight reaches zero or ctx is done. On ctx deadline it returns a Cutoff marked Forced (it never waits forever — the caller fences stragglers at L3/L4 in a later phase). This is the barrier primitive the future move controller drives in Phase 2 of a move.

func (*GateRegistry) Inflight 

func (gr *GateRegistry) Inflight(tenantID string) int

Inflight returns the number of in-flight admissions for tenantID (0 if no gate).

func (*GateRegistry) Leave 

func (gr *GateRegistry) Leave(tok AdmissionToken)

Leave releases an admission. When the gate is draining and the last in-flight admission leaves, a pending GateRegistry.CloseForBarrier is signalled.

func (*GateRegistry) Open 

func (gr *GateRegistry) Open(tenantID string, routeEpoch uint64)

Open (re)opens tenantID's gate at routeEpoch, accepting new work. The epoch is monotonic: a lower epoch than the gate's current epoch is ignored. Called when a route is observed ACTIVE on this cell (see GateRegistry.Reconcile) and after a move commits/reopens at a higher epoch.

func (*GateRegistry) Reconcile 

func (gr *GateRegistry) Reconcile(route TenantRoute)

Reconcile aligns the local gates with the authoritative route for one tenant (driven by the routing-table watch — see GateController). It opens the gate when the tenant is ACTIVE on this cell, begins draining when a move with this cell as source starts, and closes the gate when the tenant is served elsewhere.

func (*GateRegistry) Reset 

func (gr *GateRegistry) Reset(tenantID string)

Reset forgets a tenant's gate so the next admission lazily re-opens it at the requested epoch. Used when a tenant's route is removed (a cell turned off) and the tenant reverts to the default cell: there is no longer a route epoch, so the default cell must admit at epoch 0 even though its gate may have advanced to a higher epoch (Open is monotonic and would otherwise ignore epoch 0).

Note: because a removed route carries no epoch, a later re-assignment must use a fresh, higher epoch to preserve forward-only ordering across the gap — the table tombstone that enforces this belongs to the move-controller phase.

func (*GateRegistry) TryEnter 

func (gr *GateRegistry) TryEnter(tenantID string, routeEpoch uint64) (AdmissionToken, error)

TryEnter admits one unit of work for tenantID at routeEpoch, returning an AdmissionToken to be released with GateRegistry.Leave. It fails with ErrTenantDraining when the gate is not accepting new work and ErrStaleRouteEpoch when routeEpoch does not match the gate's current epoch. An absent gate is lazily opened at routeEpoch, so a service with no routes populated admits at epoch 0 with no setup (backwards-compatible).

type MemEventBarrier added in v0.30.0 

type MemEventBarrier struct {
	// contains filtered or unexported fields
}

MemEventBarrier is an in-memory EventBarrier: it records the current policy and epoch per tenant and always reports drained (an in-memory bus has no in-flight broker sends to flush).

func NewMemEventBarrier added in v0.30.0 

func NewMemEventBarrier() *MemEventBarrier

NewMemEventBarrier returns an empty in-memory event barrier.

func (*MemEventBarrier) Drained added in v0.30.0 

func (b *MemEventBarrier) Drained(_ context.Context, _ string, _ uint64) (bool, error)

Drained implements EventBarrier; an in-memory barrier is always drained.

func (*MemEventBarrier) Policy added in v0.30.0 

func (b *MemEventBarrier) Policy(tenantID string) (EventPolicy, uint64)

Policy returns the current publisher policy and epoch recorded for tenantID.

func (*MemEventBarrier) SetPolicy added in v0.30.0 

func (b *MemEventBarrier) SetPolicy(_ context.Context, tenantID string, policy EventPolicy, eventEpoch uint64) error

SetPolicy implements EventBarrier.

type MemFencer added in v0.30.0 

type MemFencer struct {
	// contains filtered or unexported fields
}

MemFencer is an in-memory Fencer: it records the authoritative owner/epoch per tenant and can be consulted by an in-memory write-guard via MemFencer.Allow. The persistence-backed fencer enforces the identical contract against a DB row.

func NewMemFencer added in v0.30.0 

func NewMemFencer() *MemFencer

NewMemFencer returns an empty in-memory fencer.

func (*MemFencer) Allow added in v0.30.0 

func (f *MemFencer) Allow(tenantID, cell string, routeEpoch uint64) bool

Allow reports whether a write for tenantID from cell at routeEpoch is permitted under the current fence. A tenant with no fence row is allowed (fail-open only for never-fenced tenants — once a move seals a tenant, stale writers are rejected). This is what an in-memory write-guard calls; the DB write-guard runs the equivalent predicate in the writing transaction.

func (*MemFencer) Seal added in v0.30.0 

func (f *MemFencer) Seal(_ context.Context, tenantID string, barrierEpoch uint64) error

Seal implements Fencer.

func (*MemFencer) SetOwner added in v0.30.0 

func (f *MemFencer) SetOwner(_ context.Context, tenantID, ownerCell string, routeEpoch uint64) error

SetOwner implements Fencer.

type MemTable 

type MemTable struct {
	// contains filtered or unexported fields
}

MemTable is the in-memory RoutingTable — the dev/test backend. It is concurrency-safe. A Raft/etcd-backed table implements the same interface for production (later module); correctness never depends on this fan-out being lossless because every cell re-checks the epoch at L2/L3.

func NewMemTable 

func NewMemTable() *MemTable

NewMemTable returns an empty in-memory routing table.

func (*MemTable) CompareAndSet 

func (t *MemTable) CompareAndSet(_ context.Context, expect, next TenantRoute) error

CompareAndSet implements RoutingTable. The precondition is checked against the stored route's (RouteEpoch, State); creation requires expect to be the zero TenantRoute and the tenant to be absent. A lower next.RouteEpoch than the stored epoch is rejected with ErrEpochRegression.

func (*MemTable) Delete 

func (t *MemTable) Delete(_ context.Context, tenantID string) error

Delete removes a tenant's route (e.g. turning off a cell reverts its tenants to the default cell). It is idempotent and broadcasts a delete event. Not part of RoutingTable — admin/test affordance on the in-memory backend.

func (*MemTable) Get 

func (t *MemTable) Get(_ context.Context, tenantID string) (TenantRoute, error)

Get implements RoutingTable.

func (*MemTable) Watch 

func (t *MemTable) Watch(ctx context.Context) (<-chan RouteEvent, error)

Watch implements RoutingTable. The returned channel is closed when ctx is cancelled.

type MoveController added in v0.30.0 

type MoveController struct {
	// contains filtered or unexported fields
}

MoveController drives the move protocol that relocates a tenant from one cell to another in depth: a forward-only route epoch advances N→N+2 across a quiesce → drain → fence → event-pause → commit sequence, every routing-table mutation an idempotent compare-and-swap (never a blind set). It is crash-safe: the routing table is the recovery state, so MoveController.Resume re-derives the phase and finishes forward (or rolls back past the deadline).

A nil Fencer/EventBarrier/Drainer turns the corresponding phase into a no-op, so the same controller serves compute-only (shared-DB) cells and data-owning cells alike.

func NewMoveController added in v0.30.0 

func NewMoveController(table RoutingTable, opts ...ControllerOption) *MoveController

NewMoveController builds a controller over table. Without options it is compute-only (no fencer/events/drainer), uses time.Now, a monotonic barrier-ID counter, an allow-all preflight, and a 30s drain deadline.

func (*MoveController) Assign added in v0.30.0 

func (c *MoveController) Assign(ctx context.Context, tenantID, cell, operator string) error

Assign places tenantID on cell (sticky first placement). With no route it creates ACTIVE at epoch 1; if the tenant already rests on cell it is a no-op; otherwise it moves the tenant from its current cell to cell.

func (*MoveController) Move added in v0.30.0 

func (c *MoveController) Move(ctx context.Context, plan MovePlan) error

Move drives the full move protocol for one tenant from its current cell to plan.ToCell, advancing the route epoch N→N+2. It is idempotent: a Move whose effect already landed (e.g. a retry) converges without regressing the epoch or double-advancing. On any error before the Phase-7 commit CAS, the tenant is rolled back to its source cell at a fresh, higher epoch.

func (*MoveController) MoveCohort added in v0.30.0 

func (c *MoveController) MoveCohort(ctx context.Context, cohortID string, members []MovePlan) error

MoveCohort moves a set of tenants under one logical cohort barrier with all-or-nothing intent: members are moved in order, and if any member fails the already-committed members are reversed (moved back to their origin cell, best-effort) so the cohort never lands half-placed. v1 sequences the members.

Reversing a committed member is itself a forward move (back to its source cell), so the route epoch only ever advances — undoing a cohort never regresses an epoch.

func (*MoveController) Resume added in v0.30.0 

func (c *MoveController) Resume(ctx context.Context, tenantID string) error

Resume is crash recovery: the routing table is the recovery state, so Resume re-reads the route and drives it forward to a consistent ACTIVE — finishing a commit, continuing a quiesce/drain/copy, or rolling back if the move deadline has elapsed. A resting route (ACTIVE/ACTIVE_NEW/ABORTED) is a no-op. Idempotent and safe to call repeatedly.

func (*MoveController) Rollback added in v0.30.0 

func (c *MoveController) Rollback(ctx context.Context, tenantID string) error

Rollback abandons an in-progress move and returns the tenant to its SOURCE cell at a fresh, higher epoch (barrierEpoch+1) — forward-only: the epoch never decreases, so a fenced stale writer stays fenced. Fencing and the event plane are reset to the source at the new epoch before the route flips back to ACTIVE. Idempotent: a tenant already resting (ACTIVE/ACTIVE_NEW) is left untouched.

type MovePlan added in v0.30.0 

type MovePlan struct {
	TenantID string
	FromCell string
	ToCell   string
	Operator string
	CohortID string // optional: the logical cohort this move belongs to
	Force    bool   // bypass the budget gate
}

MovePlan describes one tenant's move from FromCell to ToCell. CohortID is set when the move is part of a cohort. Force bypasses the budget check.

type Option 

type Option func(*options)

Option configures the L1 routing middleware.

func WithHTTPTenantFunc 

func WithHTTPTenantFunc(f func(*http.Request) string) Option

WithHTTPTenantFunc overrides how the tenant is derived from an HTTP request (default: the "account-id" header).

func WithMutationPredicate 

func WithMutationPredicate(f func(fullMethod string) bool) Option

WithMutationPredicate overrides the read-vs-mutation classifier used for fail-closed-on-uncertainty (default: AIP method-name prefixes).

func WithRetryAfter 

func WithRetryAfter(d time.Duration) Option

WithRetryAfter sets the retry hint on mid-move rejections (default 2s).

func WithTenantFunc 

func WithTenantFunc(f func(context.Context) string) Option

WithTenantFunc overrides how the tenant is derived from a gRPC context (default: middleware.TenantIDFromContext, populated by middleware.TenantIDUnary).

type PlacementFunc added in v0.30.0 

type PlacementFunc func(ctx context.Context, tenantID string, cells []string) (string, error)

PlacementFunc adapts a function to a PlacementPolicy.

func (PlacementFunc) Place added in v0.30.0 

func (f PlacementFunc) Place(ctx context.Context, tenantID string, cells []string) (string, error)

Place implements PlacementPolicy.

type PlacementPolicy added in v0.30.0 

type PlacementPolicy interface {
	// Place returns the chosen cell for tenantID from cells, or [ErrNoCells] when
	// cells is empty. Implementations must be deterministic for a given input so a
	// rebalance plan is reproducible.
	Place(ctx context.Context, tenantID string, cells []string) (string, error)
}

PlacementPolicy decides which cell a tenant should be placed on, given the candidate cells. It is consulted for first placement (MoveController.Assign) and to compute rebalance targets (see Campaign). Correctness never depends on the policy — a route is pinned to exactly one cell regardless of how the cell was chosen.

func LeastLoadedPolicy added in v0.30.0 

func LeastLoadedPolicy(load func(cell string) int) PlacementPolicy

LeastLoadedPolicy places a tenant on the candidate cell with the smallest load, as reported by load (e.g. tenant count). Ties break toward the earliest cell in the candidate slice, keeping placement deterministic for a fixed load snapshot.

func RoundRobinPolicy added in v0.30.0 

func RoundRobinPolicy() PlacementPolicy

RoundRobinPolicy distributes tenants across the candidate cells in rotation, advancing deterministically per call. Thread-safe. With a fixed candidate set the assignment sequence is reproducible, so a rebalance plan built from it is stable.

func StickyDefaultPolicy added in v0.30.0 

func StickyDefaultPolicy(defaultCell string) PlacementPolicy

StickyDefaultPolicy always places a tenant on defaultCell, ignoring the candidate set — the v1 default: every tenant lands on one well-known cell and only moves on an explicit operator action.

type RouteEvent 

type RouteEvent struct {
	Route    TenantRoute // the new state of the tenant's route (zero TenantID if Deleted)
	TenantID string      // the affected tenant (always set, even when Deleted)
	Deleted  bool        // the route was removed; the tenant reverts to the default cell
	Revision uint64      // table-global monotonic revision at which this change applied
}

RouteEvent is a single change observed on a RoutingTable.Watch stream.

type Router 

type Router struct {
	// contains filtered or unexported fields
}

Router is a read-mostly, watch-fed cache over a RoutingTable. It resolves a tenant to a Decision, falling back to the fail-safe default cell for unknown tenants and when the table is unreachable (reads stay available; the Stale flag lets the write path fail closed). It implements [health.Check]: not ready until its watch has been established.

func NewRouter 

func NewRouter(table RoutingTable, opts ...RouterOption) *Router

NewRouter builds a Router over table. Call Router.Start to begin watching.

func (*Router) Check 

func (r *Router) Check(context.Context) error

Check implements [health.Check]: ready once the watch is established.

func (*Router) DefaultCell 

func (r *Router) DefaultCell() string

DefaultCell returns the cell served to tenants without an explicit route.

func (*Router) Name 

func (r *Router) Name() string

Name implements [health.Check].

func (*Router) Resolve 

func (r *Router) Resolve(ctx context.Context, tenantID string) Decision

Resolve returns the routing Decision for tenantID. A cache hit is served locally; a miss lazily reads the table (negative-caching unknown tenants). When the table is unreachable on a miss, it returns a Stale default-cell decision so reads stay available while writes can fail closed.

func (*Router) Start 

func (r *Router) Start(ctx context.Context) error

Start establishes the watch and keeps the cache converged until ctx is cancelled. It returns once the watch is established (the Router is then ready); the background loop applies events thereafter. Safe to call once.

type RouterOption 

type RouterOption func(*Router)

RouterOption configures a Router.

func WithDefaultCell 

func WithDefaultCell(cell string) RouterOption

WithDefaultCell overrides the fail-safe default cell (default: DefaultCellID).

func WithFreshness 

func WithFreshness(d time.Duration) RouterOption

WithFreshness marks a cached route older than d as Stale, so the write path fails closed rather than trust a route the watch may have failed to refresh. Zero (default) disables age-based staleness and relies on the watch stream.

type RoutingTable 

type RoutingTable interface {
	// Get returns the current route for tenantID, or ErrNoRoute if the tenant has
	// no explicit assignment.
	Get(ctx context.Context, tenantID string) (TenantRoute, error)

	// CompareAndSet atomically stores next for next.TenantID, but only if the
	// currently stored route matches expect on (RouteEpoch, State). To create the
	// first route for a tenant, pass the zero TenantRoute as expect (StateUnknown,
	// epoch 0) and the tenant must currently be absent. It returns ErrCASConflict
	// when the precondition fails and ErrEpochRegression when next.RouteEpoch is
	// below the stored epoch.
	CompareAndSet(ctx context.Context, expect, next TenantRoute) error

	// Watch returns a channel of route changes. The channel is closed when ctx is
	// cancelled. Implementations may coalesce rapid updates but must always
	// deliver the latest state for any changed tenant.
	Watch(ctx context.Context) (<-chan RouteEvent, error)
}

RoutingTable is the tenant→cell directory: the single source of truth for route state. It is consumed read-mostly by routers/cells via Watch; mutations go only through CompareAndSet (every transition an idempotent compare-and-swap, never a blind set). The in-memory MemTable is the dev/test backend; a Raft/etcd-backed adapter implements the same interface in a later module.

type State 

type State uint8

State is the lifecycle of a tenant's route. The resting states (ACTIVE, ACTIVE_NEW) admit new work; the transitional move states reject new work at L1 while the source cell drains. ABORTED is a move that was abandoned — the tenant stays on its current active cell. StateUnknown is the zero value and doubles as the "expect absent" precondition in RoutingTable.CompareAndSet. 

const (
	StateUnknown    State = iota // zero value; also "expect absent" in a CAS
	StateActive                  // serving normally on ActiveCell at RouteEpoch
	StateQuiescing               // move begun; gateways observing the new epoch reject new calls
	StateDraining                // source gates closed; waiting for in-flight to finish
	StateCopying                 // data/event catch-up (data-owning cells; P4)
	StateCommitting              // about to flip the route to the target cell
	StateActiveNew               // serving on the new cell at the post-move epoch
	StateAborted                 // move abandoned; tenant remains on ActiveCell
)

func (State) AdmitsNew 

func (s State) AdmitsNew() bool

AdmitsNew reports whether a tenant in this state accepts new calls. Only the resting states do; every transitional move state rejects new calls so the source cell can drain to a clean cut.

func (State) IsMoving 

func (s State) IsMoving() bool

IsMoving reports whether this state is a transitional move state (new calls rejected at L1).

func (State) String 

func (s State) String() string

String returns the canonical name of the state.

type TenantRoute 

type TenantRoute struct {
	TenantID   string
	RouteEpoch uint64 // monotonic per tenant; never decreases, even on rollback
	ActiveCell string // the cell currently serving the tenant
	SourceCell string // during a move: the cell being drained
	TargetCell string // during a move: the cell being moved to
	State      State

	BarrierID    string // identifies the in-progress move barrier
	BarrierEpoch uint64 // the epoch new work is rejected at while draining

	// Event plane (L4). EventEpoch fences old publishers: a publisher at an
	// EventEpoch at or below a superseded barrier can never publish NORMAL again.
	EventPolicy EventPolicy
	EventEpoch  uint64

	// Data-owning cells (P4). High-watermarks captured at the barrier so the
	// target can prove it has caught up before the route commits. Zero for
	// compute-only (shared-DB) cells, where data catch-up is a no-op.
	SourceDataHWM  uint64
	SourceEventHWM uint64

	// ControllerClass shards move orchestration like a Kubernetes IngressClass:
	// a route is driven only by the controller whose class matches. Empty binds
	// to the default class. Correctness is unaffected — class only decides who
	// orchestrates; an unclassed/ownerless route simply stays ACTIVE.
	ControllerClass string

	Deadline     time.Time // liveness deadline for the current move phase (never a safety mechanism)
	LastOperator string    // audit: who drove the last transition
}

TenantRoute is the per-tenant record in the routing table — the single source of truth for where a tenant is served and whether it is moving. Safety derives from RouteEpoch (monotonic, never decreases), not from clocks.

func (TenantRoute) IsZero 

func (r TenantRoute) IsZero() bool

IsZero reports whether r is the zero TenantRoute (no tenant set). Used to mean "absent" in CAS preconditions.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.