cluster

Documentation

Overview

Package cluster gives Coremetry pods a way to see each other in an HA / replicated deployment (v0.5.253). Coremetry's background workers (evaluator, anomaly detector, monitor runner, log templater, topology aggregator) have always been HA-safe — they take a per-tick Redis lock so only one replica runs each tick. The missing piece was operator visibility: "I scaled to 10 pods, are they all alive? Which one ran the last tick?"

Each pod writes a heartbeat to a Redis key `coremetry:pod:<id>` every 10s with a 30s TTL. The admin /api/admin/cluster endpoint scans those keys via the existing cache.Cache abstraction to render a live member list. Pods that disappear from Redis (crashed / rolled out) silently fall off the list within 30s.

Pure visibility — no leader election here. The existing per-tick TryAcquire pattern is the leader-election story; this package is purely "who's alive".

Index

• func PodIDFromHostname(id string) string
• type Member
• type Service
  • func New(c cache.Cache, version string) *Service
  • func (s *Service) Members(ctx context.Context) ([]Member, error)
  • func (s *Service) MyID() string
  • func (s *Service) SetLocalLocks(keys []string)
  • func (s *Service) Start(ctx context.Context)

Functions

func PodIDFromHostname

func PodIDFromHostname(id string) string

PodIDFromHostname extracts the short label most operators expect from a pod hostname — strips the random suffix so two replicas of the same Deployment cluster together visually. Used by the frontend's group-by-hostname rendering.

Types

type Member

type Member struct {
	ID          string   `json:"id"`
	Hostname    string   `json:"hostname"`
	Version     string   `json:"version"`
	StartedAt   int64    `json:"startedAt"` // unix ns
	LastSeen    int64    `json:"lastSeen"`  // unix ns
	IsThisPod   bool     `json:"isThisPod"`
	LeaderLocks []string `json:"leaderLocks,omitempty"`
}

Member is the public shape returned by the cluster admin endpoint. Mirrors the JSON the heartbeat key holds plus the derived `isThisPod` field (populated server-side per request).

type Service

type Service struct {
	// contains filtered or unexported fields
}

Service is the per-pod heartbeat loop + membership lookup. One instance lives in main(); the API handler reads from it.

func New

func New(c cache.Cache, version string) *Service

New constructs the service but does not start the heartbeat loop — call Start(ctx) once the process is ready to identify itself. Generates a stable pod ID: hostname + 4-byte random suffix so duplicate hostnames (rare but possible on bare-metal) don't collide.

The Noop cache short-circuits both heartbeat + Members(); the admin page renders a single "this pod" entry in dev mode.

func (*Service) Members

func (s *Service) Members(ctx context.Context) ([]Member, error)

Members returns the full live member list — every pod that's written a heartbeat in the last `ttl` window. Sorted by StartedAt ascending so the oldest replica leads the list.

When the cache is Noop (single-instance dev mode), returns a single-member list representing this process so the admin page renders sensibly in compose-up too.

func (*Service) MyID

func (s *Service) MyID() string

MyID returns this pod's stable identifier. Embedded in log lines + audit details when the operator wants to know "which replica did this".

func (*Service) SetLocalLocks

func (s *Service) SetLocalLocks(keys []string)

SetLocalLocks lets callers record which lock keys this pod currently holds (e.g. the evaluator's "evaluator:lock"). The admin page surfaces them as "this pod owns: …" so the operator can see which replica is the active worker for each tick. Last-write-wins; callers typically call this after a successful TryAcquire.

func (*Service) Start

func (s *Service) Start(ctx context.Context)

Start launches the heartbeat loop. Writes an immediate heartbeat so a freshly-rolled-out pod appears in the member list before the first interval elapses. Returns when ctx is cancelled; the last-written key expires naturally via its TTL (10-30s grace period before it falls off the list).