analytics

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 18, 2026 License: MIT Imports: 23 Imported by: 0

Documentation

Overview

Package analytics records and reads click data.

The privacy guarantee here is structural rather than procedural: no IP address is ever written to click_events, because the table has no column for one. Visitor identity is a keyed hash whose key is deleted on a schedule, and that deletion is what makes the day's hashes irreversible.

Index

Constants

View Source 
const (
	TotalsJob    = "analytics_rollup"
	DimensionJob = "analytics_dimension_rollup"
)

The two jobs' rows in job_state.

TotalsJob keeps the name the single job had, so an instance upgrading into M37 carries its watermark forward instead of reopening ninety days on the first tick. DimensionJob is new and therefore has no row, which is exactly right: its first run finds no watermark, covers the default two-day window, and is correct from then on.

View Source 
const DimensionInterval = 15 * time.Minute

DimensionInterval is how often the dimension breakdowns are recomputed.

Fifteen minutes against a job measured at 16-21 seconds is a duty cycle of about 2.3%, where sixty seconds was about 33%. That is the whole of the fix: the same work, on a clock it fits inside, with roughly forty times the headroom before it stops fitting again. The recorded fallback if cadence alone stops holding is to narrow the recomputed window — see Plan.md's dimension-rollup row and docs/slo.md.

The cost is paid by the reader, and it is named rather than hidden: a breakdown can be up to fifteen minutes behind the totals on the same page. The staleness gauge is what makes that observable instead of merely true.

View Source 
const PermRead = "analytics.read"

PermRead is the permission analytics reads require.

View Source 
const ReturningHashLength = 8

ReturningHashLength is how much of the visitor hash a set member keeps.

Eight bytes out of the sixteen VisitorHash already truncated to. This is a membership test for one link on one day, so the collision that matters is between two visitors of the same link on the same day: at a million distinct visitors that is a chance of roughly one in forty million, and the cost of losing is one visitor being routed as returning on their first visit. Against that, halving the key halves what Redis holds for a popular link and takes another eight bytes off a value that exists only to be compared with itself.

View Source 
const ReturningKeyPrefix = "lc:rv:"

ReturningKeyPrefix is the Redis namespace. Separate from the redirect cache's `lc:a:` so an operator reading the keyspace can tell the two apart, and so a domain sweep cannot reach these.

View Source 
const SaltLength = 32

SaltLength is the size of a daily salt.

View Source 
const SaltRetentionDays = 2

SaltRetentionDays is how long a salt is kept before deletion.

The salt must outlive the events it keyed for as long as unique-visitor counting needs to recompute, but no longer — its deletion is what makes those hashes permanently unlinkable to an address. Two days covers a same-day rollup plus the finalize pass over the previous day.

View Source 
const VisitorHashLength = 16

VisitorHashLength is how much of the HMAC is kept.

16 bytes is far beyond collision range for per-day, per-link counting, and truncating reduces what a database holds without weakening the property that matters: the hash cannot be reversed once the salt is gone.

Variables

View Source 
var ErrIngesterClosed = errors.New("analytics: ingester is closed")

Functions

func FormatHash 

func FormatHash(h []byte) string

FormatHash renders a hash for logs and debugging. Never used for storage.

func NewSalt 

func NewSalt() ([]byte, error)

NewSalt generates a day's salt.

func PrimaryLanguage 

func PrimaryLanguage(header string) string

PrimaryLanguage extracts the first tag from an Accept-Language header.

Only the language subtag is kept ("en" from "en-GB"). Region adds granularity nobody reports on and narrows the anonymity set.

func ReferrerHost 

func ReferrerHost(referrer string) string

ReferrerHost extracts the host from a referrer.

Only the host is kept. Full referrer URLs routinely carry query parameters with session tokens, search terms and personal data, so the rest is discarded at the edge rather than stored and cleaned up later.

func SaltDay 

func SaltDay(at time.Time) time.Time

SaltDay returns the UTC day a timestamp belongs to.

UTC always, never local time. A local-time boundary would rotate the salt at a different instant per deployment and, worse, make the same visitor hash differently on either side of a daylight-saving change.

func VisitorHash 

func VisitorHash(salt []byte, ip netip.Addr, userAgent string, workspaceID uuid.UUID) []byte

VisitorHash derives a per-day, per-workspace visitor identifier.

HMAC rather than a plain hash of salt||data: a plain concatenation is vulnerable to length extension, and HMAC is the construction actually designed for keying a hash.

workspaceID is part of the message, not the key. That is what stops the same person being correlated across two workspaces on the same instance — the salt is shared per day, but the derived hashes differ, so one workspace's analytics cannot be joined against another's.

The inputs are separated by a NUL byte so that ("ab", "c") and ("a", "bc") cannot produce the same hash. Without a separator, a crafted user agent could be made to collide with a different address.

Types

type Classification 

type Classification struct {
	Device  Device
	Browser string
	OS      string
	IsBot   bool
}

Classification is everything derived from a user agent.

func Classify 

func Classify(ua string) Classification

Classify buckets a user agent.

type CountryResolver 

type CountryResolver interface {
	Country(netip.Addr) string
}

CountryResolver turns an address into an ISO 3166-1 alpha-2 country code, or "" when it cannot.

An interface, not the geoip package, for two reasons: analytics should not depend on how geography is looked up, and a test needs to enrich events without a MaxMind database on disk.

type DayPoint 

type DayPoint struct {
	Day            string `json:"day"`
	Clicks         int64  `json:"clicks"`
	UniqueVisitors int64  `json:"unique_visitors"`
	BotClicks      int64  `json:"bot_clicks"`
}

DayPoint is one day in a time series.

type DestinationSplit added in v0.2.0 

type DestinationSplit struct {
	DestinationID uuid.UUID `json:"destination_id"`
	URL           string    `json:"url"`
	// Weight is the arm's configured weight, or 0 where it is not a weighted
	// arm. It is what makes the breakdown readable as a test: "40% configured,
	// 41% observed" is a working split and "40% configured, 3% observed" is a
	// broken destination.
	Weight int32 `json:"weight"`
	// IsPrimary marks the link's own destination.
	IsPrimary bool `json:"is_primary"`
	// Removed marks clicks attributed to a destination that has since been
	// deleted. They are reported rather than dropped: a running test's totals
	// must not change because somebody tidied up an arm.
	Removed bool `json:"removed,omitempty"`
	// Approximate marks a row whose click count includes traffic the
	// per-destination rollup has not attributed yet.
	//
	// It is only ever the link's own destination, and only because that row is
	// computed as a remainder rather than read. A click on the link's own
	// destination carries the zero uuid and the dimension rollup filters
	// `destination_id IS NOT NULL`, so the primary has no rollup row to read —
	// its clicks are whatever the 60-second totals hold that the 15-minute
	// destination rollup has not accounted for. Between those two cadences that
	// remainder is the primary's real clicks *plus* every split-arm click of the
	// last quarter-hour, and it was rendered as positive attribution to a named
	// destination (F107). Worst case is a split test viewed inside its first
	// fifteen minutes: 100% to the link's own destination and 0% to the arms.
	Approximate bool  `json:"approximate,omitempty"`
	Clicks      int64 `json:"clicks"`
	// UniqueVisitors is the count, and VisitorsKnown says whether it is one.
	//
	// Two fields rather than a pointer, because this crosses to a template and
	// `0` and *not measured* had been the same value on the primary row since
	// M36 — permanently, not just during the lag. The remainder carries no
	// visitor figure at all: unique visitors are counted per destination by the
	// rollup, and the row the rollup never writes has none to carry.
	UniqueVisitors int64   `json:"unique_visitors"`
	VisitorsKnown  bool    `json:"visitors_known"`
	Share          float64 `json:"share"`
}

DestinationSplit is one destination's share of a link's clicks (M36).

The row for the link's own destination is synthesized from what the other rows do not account for, because a click that went there carries a NULL destination_id — see migration 02200. So `Clicks` here sums to the link's non-bot total by construction rather than by the rollup happening to agree.

type Device 

type Device string
const (
	DeviceDesktop Device = "desktop"
	DeviceMobile  Device = "mobile"
	DeviceTablet  Device = "tablet"
	DeviceBot     Device = "bot"
	DeviceUnknown Device = "unknown"
)

type DimensionValue 

type DimensionValue struct {
	Value          string `json:"value"`
	Clicks         int64  `json:"clicks"`
	UniqueVisitors int64  `json:"unique_visitors"`
}

DimensionValue is one bucket of a breakdown.

type Event 

type Event struct {
	LinkID      uuid.UUID
	WorkspaceID uuid.UUID
	OccurredAt  time.Time
	IP          netip.Addr
	UserAgent   string
	Referrer    string
	// Source is a resolved attribution token (M41) — see domain.ClickSource. It
	// replaces the referrer host on the stored row when set, because the clicks
	// it describes carry no Referer at all: a QR scan comes from a camera.
	//
	// No new column and no new dimension: the value lands in `referrer_host` and
	// is rolled up as the `referrer` breakdown, beside the `direct` sentinel that
	// column already holds for a click with no referrer.
	Source    string
	Language  string
	LatencyUS int32

	// TrackReturning asks the ingester to remember this visitor in the
	// within-day returning-visitor set (M34).
	//
	// Set by the redirect handler, from the link's own rules, and false for
	// every link that has none — which is what keeps the set from being
	// maintained for the whole instance. The alternative is the ingester asking
	// which links have a returning-visitor rule, which is a query per batch
	// against data the handler already had in its hand.
	TrackReturning bool

	// DestinationID is the destinations row this click was sent to (M36).
	//
	// The zero uuid means the link's own destination, and is written to the
	// column as NULL — see destinationOrNil. Every click on every link without
	// rules carries it, which is why the column is nullable rather than
	// backfilled: a default would be a per-row copy of what the link already
	// says.
	DestinationID uuid.UUID
}

Event is a click as the redirect handler observes it.

The raw IP is present here and nowhere else. It is consumed to derive the visitor hash and then dropped; it is never written, never logged and never leaves this package.

type IngestConfig 

type IngestConfig struct {
	QueueSize     int
	BatchSize     int
	FlushInterval time.Duration
	Logger        *slog.Logger

	// Geo is optional. Nil leaves the country column null, which is the default
	// state: the MaxMind database cannot be shipped in the image.
	//
	// A CountryResolver, not the geoip package's whole Resolver, and that
	// narrowing is now load-bearing rather than tidy: M34 gave that type Region
	// and City, and the interface here is what says the click pipeline may not
	// call them. Region and city are resolvable on the redirect path and are
	// never stored, and this is where "never stored" is enforced by the type
	// system instead of by remembering.
	Geo CountryResolver

	// Returning maintains the within-day returning-visitor set (M34). Nil on an
	// instance with no Redis, and then nothing is written and every visitor
	// reads as new.
	Returning *ReturningSet
}

type Ingester 

type Ingester struct {
	Stats Stats
	// contains filtered or unexported fields
}

Ingester buffers click events and writes them in batches.

The contract that shapes everything else: recording a click must never delay a redirect and must never fail one. So the queue is bounded and Record drops rather than blocks — applying backpressure to the hot path would trade a complete analytics record for a slow site, which is the wrong way round. Drops are counted, and a non-zero counter is an alert rather than a silent gap.

func NewIngester 

func NewIngester(pool *pgxpool.Pool, salts *SaltCache, cfg IngestConfig) *Ingester

func (*Ingester) Close 

func (i *Ingester) Close(ctx context.Context) error

Close stops accepting events and flushes what is buffered.

Called during shutdown after the listener has closed, so no new events can arrive. Without this, every restart loses up to a full batch.

Signals through stop rather than closing ch. Record's closed check and its send cannot be one atomic step, so a redirect that passed the check just before Close ran would send on a closed channel and panic — killing the process during the one window where it is trying to save data. That is not hypothetical during a shutdown that times out with requests still in flight, which is exactly when Close is called. An event that races the drain is lost instead, which is what a full queue already does to it.

func (*Ingester) Counters 

func (i *Ingester) Counters() (enqueued, dropped, flushed, failed, batches int64)

Counters returns the lifetime totals, for the metrics collector.

One method returning all five rather than five accessors, so a scrape reads them in one call and the set cannot be sampled half a flush apart.

func (*Ingester) QueueDepth 

func (i *Ingester) QueueDepth() int

QueueDepth reports buffered events.

The leading indicator for the whole pipeline: depth climbing means the database is falling behind, minutes before drops start.

func (*Ingester) Record 

func (i *Ingester) Record(ev Event)

Record enqueues an event, or drops it.

Never blocks, never returns an error, never panics after Close. The default branch is the entire point: when the queue is full the event is discarded so the redirect returns on time.

func (*Ingester) Start 

func (i *Ingester) Start()

Start launches the batching goroutine.

type LinkStats 

type LinkStats struct {
	LinkID     uuid.UUID                   `json:"link_id"`
	From       string                      `json:"from"`
	To         string                      `json:"to"`
	Totals     Totals                      `json:"totals"`
	Series     []DayPoint                  `json:"series"`
	Dimensions map[string][]DimensionValue `json:"dimensions"`
	// Destinations is the per-destination breakdown a split test is read from
	// (M36). Empty for a link that has never sent a click anywhere other than
	// its own destination, which is every link on a default instance.
	Destinations []DestinationSplit `json:"destinations"`
	// QRCodes is the per-code breakdown (M50). Empty for a link nobody has
	// scanned, and one row — the default code — for a link with a single code,
	// which is every link until somebody adds a second.
	QRCodes []QRCodeSplit `json:"qr_codes"`
	// Caveat travels with the data rather than living only in documentation,
	// so a client rendering these numbers can surface it.
	Caveat string `json:"caveat"`
}

LinkStats is a link's analytics over a window.

type QRCodeSplit added in v0.3.0 

type QRCodeSplit struct {
	// Slug is the identity printed in the code's payload. Empty for the default
	// code — the one whose payload carries no code parameter, and the one every
	// picture drawn before M50 is.
	Slug string `json:"slug"`
	// Label is what the workspace calls this code. Empty for a code nobody has
	// named and for one that has since been deleted, which is why Removed exists
	// beside it: a client needs to know the difference between "unnamed" and
	// "gone".
	Label string `json:"label"`
	// Removed marks scans attributed to a code that no longer exists. They are
	// reported rather than dropped, exactly as a removed split arm's are: a
	// link's totals must not change because somebody retired a poster. New scans
	// of a deleted code's printed payload do not land here — an unrecognised
	// slug is recorded as the default code — so a removed row stops growing at
	// the moment the code is deleted.
	Removed        bool  `json:"removed,omitempty"`
	Clicks         int64 `json:"clicks"`
	UniqueVisitors int64 `json:"unique_visitors"`
}

QRCodeSplit is one QR code's share of a link's scans (M50).

**Read from the referrer dimension the scans were already rolled up into**, which is what keeps this a filter rather than a second rollup: `qr` is the default code and `qr:<slug>` is a named one, and both were written by the pass that writes every other breakdown.

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

Reader serves dashboard queries.

Every query here reads a rollup table, never click_events, except the deliberately bounded recent-activity feed. That is what holds analytics under the 2s target once the raw table reaches tens of millions of rows.

func NewReader 

func NewReader(pool *pgxpool.Pool) *Reader

func (*Reader) LinkStats 

func (r *Reader) LinkStats(ctx context.Context, actor *auth.Identity, linkID uuid.UUID, from, to time.Time) (*LinkStats, error)

func (*Reader) Overview 

func (r *Reader) Overview(ctx context.Context, actor *auth.Identity, from, to time.Time) (*WorkspaceOverview, error)

func (*Reader) RecentClicks 

func (r *Reader) RecentClicks(ctx context.Context, actor *auth.Identity, linkID uuid.UUID, limit int32) ([]RecentClick, error)

RecentClicks reads raw events, bounded and index-backed.

The one query that touches click_events directly. Safe because it is capped and served by the (link_id, occurred_at DESC) index, so cost does not grow with table size.

type RecentClick 

type RecentClick struct {
	OccurredAt time.Time `json:"occurred_at"`
	Device     string    `json:"device,omitempty"`
	Browser    string    `json:"browser,omitempty"`
	OS         string    `json:"os,omitempty"`
	Country    string    `json:"country,omitempty"`
	Referrer   string    `json:"referrer,omitempty"`
	IsBot      bool      `json:"is_bot"`
}

RecentClick is one entry in the live activity feed.

type ReturningSet added in v0.2.0 

type ReturningSet struct {
	// contains filtered or unexported fields
}

ReturningSet is the returning-visitor set, read on the redirect path and written by the ingester.

A nil *ReturningSet is valid and answers "not returning" to everything, which is what an instance with no Redis has.

func NewReturningSet added in v0.2.0 

func NewReturningSet(rdb *goredis.Client, salts *SaltCache, timeout time.Duration, log *slog.Logger) *ReturningSet

NewReturningSet builds one. A nil client returns nil, so "no Redis" is a nil pointer rather than a flag every call site has to check.

func (*ReturningSet) Enabled added in v0.2.0 

func (s *ReturningSet) Enabled() bool

Enabled reports whether there is anything to ask.

func (*ReturningSet) Member added in v0.2.0 

func (s *ReturningSet) Member(salt []byte, ip netip.Addr, userAgent string, workspaceID uuid.UUID) string

Member derives a set member from the same inputs the visitor hash uses.

Hex rather than raw bytes, because these end up in a Redis set an operator may well look at with redis-cli, and a binary member there is unreadable without being any more private.

func (*ReturningSet) Seen added in v0.2.0 

func (s *ReturningSet) Seen(
	ctx context.Context, linkID, workspaceID uuid.UUID,
	ip netip.Addr, userAgent string, at time.Time,
) bool

Seen reports whether this visitor was already seen on this link today.

**Nothing here can reach Postgres**, and that is the design rather than a happy accident. The salt is read from the cache's in-memory map with SaltCache.Cached, never with For — For would create or fetch the day's salt, which is a database query, and m34.md's claim is that rule evaluation adds no database query per request. A cache miss therefore answers "not returning".

That answer is not a degradation in the case it actually happens. The salt is absent from a process's memory only before that process has handled the day at all — at boot, which cmd/linkctrl warms past by loading today's salt before it listens, and immediately after midnight UTC, when the day's set is empty and "not returning" is true of everybody. What it is not is a silent fallback on a busy instance.

Every Redis failure — key absent, server down, timeout — is also "not returning", for the reason every other Redis failure on this path is a miss: the redirect must complete either way, and the honest degradation of a condition that cannot be evaluated is the condition not matching.

type Roller 

type Roller struct {
	// contains filtered or unexported fields
}

Roller recomputes the pre-aggregated tables the dashboard reads.

func NewRoller 

func NewRoller(pool *pgxpool.Pool, log *slog.Logger) *Roller

func (*Roller) Run 

func (r *Roller) Run(ctx context.Context, from, to time.Time) error

Run recomputes every rollup for [from, to).

Recomputation rather than incremental accumulation is deliberate. Each run derives whole days from the raw events and upserts, so a retry after a crash mid-run converges to the same numbers. An "add what arrived since the last watermark" design double-counts on every retry and drifts permanently once it does — and the drift is invisible until someone reconciles by hand.

The scheduler no longer calls this: since M37 the two halves run on different clocks. It stays as the whole-of-analytics pass for an explicit window, which is what a startup run and an operator recomputing a repaired day both want.

func (*Roller) RunDimensions added in v0.2.0 

func (r *Roller) RunDimensions(ctx context.Context, from, to time.Time) error

RunDimensions recomputes the per-dimension and per-destination breakdowns.

The expensive half, and the reason M37 exists. Measured on the SLO dataset (5.7M events, ~830k inside the recomputed window) this takes 16-21 seconds, and the cost is the ~553,053 conflicting tuples a whole-day recompute of (link, day, dimension, value) implies — not the scan, which was already rewritten to read click_events once. See docs/slo.md.

Recomputing whole days is not negotiable: it is what makes a retry converge instead of double-counting. So the fix is to run it less often than the totals, which is what DimensionInterval is, rather than to make it cheaper.

func (*Roller) RunRecent 

func (r *Roller) RunRecent(ctx context.Context, now time.Time) error

RunRecent recomputes everything not known to be final, and at minimum yesterday and today.

Both halves, under their own watermarks. The scheduler does not call this — it ticks the two halves separately — but a startup run and `lctl` do, because "make the numbers current" is one request even when the maintenance of them is two jobs.

func (*Roller) RunRecentDimensions added in v0.2.0 

func (r *Roller) RunRecentDimensions(ctx context.Context, now time.Time) error

RunRecentDimensions recomputes the breakdowns for everything not known to be final, on its own watermark.

Its own watermark and not a share of the totals' one, because the two jobs now advance at different rates: a dimension pass that read the totals' watermark would find it already past the day it had not covered yet and would leave that day permanently unaggregated — the exact failure the watermark was introduced to fix, reintroduced by splitting the cadence.

func (*Roller) RunRecentTotals added in v0.2.0 

func (r *Roller) RunRecentTotals(ctx context.Context, now time.Time) error

RunRecentTotals recomputes the per-link and per-workspace totals for everything not known to be final.

func (*Roller) RunTotals added in v0.2.0 

func (r *Roller) RunTotals(ctx context.Context, from, to time.Time) error

RunTotals recomputes the per-link and per-workspace daily totals.

The cheap half, and the half the dashboard's headline numbers come from. It writes one row per (link, day) and one per (workspace, day), so its upsert count is bounded by the number of links that were clicked rather than by the number of distinct dimension values they were clicked from.

func (*Roller) Staleness added in v0.2.0 

func (r *Roller) Staleness(ctx context.Context) ([]Staleness, error)

Staleness reports every job that has ever succeeded, and how long ago.

Jobs that have never succeeded are omitted by the query rather than reported as infinitely stale: a series invented for a job that has not run yet is indistinguishable from one for a job that stopped, and the first is what every fresh instance looks like for its first few seconds.

type SaltCache 

type SaltCache struct {
	// contains filtered or unexported fields
}

SaltCache resolves the salt for a UTC day, creating it on first use.

Cached in memory because every batch needs it and it changes once a day. The cache is small and bounded by retention, so it is a plain map rather than anything with eviction.

func NewSaltCache 

func NewSaltCache(pool *pgxpool.Pool) *SaltCache

func (*SaltCache) Cached added in v0.2.0 

func (c *SaltCache) Cached(day time.Time) ([]byte, bool)

Cached returns a day's salt only if it is already in memory.

The redirect path's caller, and the reason it exists: For creates or fetches the salt, which is a database query, and M34 claims that evaluating a routing rule adds none. A miss here is answered by the caller as "this visitor is new" rather than by going to Postgres — see ReturningSet.Seen for why that is true rather than merely cheap.

func (*SaltCache) For 

func (c *SaltCache) For(ctx context.Context, day time.Time) ([]byte, error)

For returns the salt for a day, creating it if absent.

func (*SaltCache) Purge 

func (c *SaltCache) Purge(ctx context.Context) (int64, error)

Purge deletes salts past their retention.

This is the de-identification step, not housekeeping: once a salt is gone, the hashes it produced cannot be linked back to an address even by someone holding the original addresses.

type Staleness added in v0.2.0 

type Staleness struct {
	Job     string
	Seconds float64
}

Staleness is how long ago a job last succeeded, read from job_state.

type Stats 

type Stats struct {
	Enqueued atomic.Int64
	Dropped  atomic.Int64
	Flushed  atomic.Int64
	Failed   atomic.Int64
	Batches  atomic.Int64
}

Stats are the ingester's counters.

type Totals 

type Totals struct {
	Clicks         int64 `json:"clicks"`
	UniqueVisitors int64 `json:"unique_visitors"`
	BotClicks      int64 `json:"bot_clicks"`
}

type WorkspaceOverview 

type WorkspaceOverview struct {
	From   string     `json:"from"`
	To     string     `json:"to"`
	Totals Totals     `json:"totals"`
	Series []DayPoint `json:"series"`
	Caveat string     `json:"caveat"`
}

WorkspaceOverview is the dashboard summary.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.