contentkit

package module
v0.12.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 20, 2026 License: MIT Imports: 19 Imported by: 0

README

contentkit

contentkit is the deterministic content library for the Doujins, Hentai0 and marketplace hosts: tenant-scoped interactions (posts, comments, reactions, favorites, polls), keyword search over host content, the ClickHouse signal plane (consumption, feedback, exposures, popularity, erasure) and the discovery reads over both. It needs no model provider, API key or vector extension. Semantic search belongs entirely to the separate, deferred User Intelligence library; ContentKit has no semantic search configuration or runtime hook.

Design: open-rails-tracker/contentkit/DESIGN.md. Host contract: HOST_INTEGRATION.md. Migrations: docs/migration.md, docs/restore.md.

Release policy

ContentKit is pre-stable and releases on v0.x; its API may change between minor versions. The historical v1.0.0-rc.1 through v1.1.1 tags were premature and are retracted. v1.1.2 is a self-retracted, withdrawal-only marker carrying Go module metadata, not a supported stable API release.

The marker and v0.12.2 identify the same source commit. Go reads retractions from the highest release before applying them, so the marker makes fresh @latest requests select the latest unretracted v0.x release, including future v0.x releases. Existing tags remain immutable; retraction preserves explicit pins and does not automatically downgrade existing consumers. See Go module retractions.

Vocabulary

Name Meaning
tenant_id one site: doujins, hentai0, the marketplace
content_id the host-owned work: a gallery, a video, a listing
content_version_id one selectable version of that work (optional)
taxonomy_id a generic ContentKit record: tag, artist, series, creator, character, voice actor
ContentRef {TenantID, ContentKind, ContentID, ContentVersionID *string} — the typed reference every API, key, index and cursor carries

A language is metadata on a document or version, never part of a reference. Every read is scoped to the tenant pinned at construction; a reference of another tenant is an error, never remapped.

Packages

Package Owns
contentref ContentRef, ContentKey, TaxonomyID
content posts, comments, reactions, favorites, polls (multiple-choice and free-text) and their counts over ContentRef, in the host schema's social_* tables; the Identity/Authorizer/ContentResolver/UserEnricher/MediaStore/ContentProcessor ports, the optional ContentModerator (held/review queue) and AnswerClassifier ports, and the HTTP routes
search PGroonga keyword search (exact/alias/prefix/typo, EN/ZH/JA/KO), documents and dirty queue, RRF, the DocumentSink port
worker one tenant's document maintenance: dirty queue, bounded backfill, sink delivery
taxonomy generic catalog: nodes (tags, artists, creators, characters, series, seasons, voice actors), localized names/aliases, edges, content assignments, effective tags, per-language counts, typeahead documents, admin routes
signal ClickHouse signal plane: canonical signals, compact subject state, daily rollups, windows, erasure fence, exposures/attribution, repair
popularity named ranking policy (PolicyV1) over the window metrics: ClickHouse RankExpr and Go Score in agreement, literal windows, session scorer, taxonomy popularity through the host Catalog port
eval lexical golden-case evaluation, reports, baselines
migrations the five migratekit lineages (social, keyword, legacy keyword, taxonomy, signal)
root Runtime (one constructor: hub + content + HTTP mount), Migrate (social, keyword, optional taxonomy, signal), Client (keyword search + typeahead), EmbeddedHub (signal + discovery)

Install

One call applies the social lineage into the host schema, the keyword profile into the search schema (PGroonga and pg_trgm required, no vector extension) and the signal lineage into a dedicated ClickHouse database:

_ = signal.CreateDatabase(ctx, adminCH, "hub", cluster)
_ = contentkit.Migrate(ctx, contentkit.MigrateConfig{
	DB: sqlDB, Schema: "doujins", SearchSchema: "doujins_searchkit",
	ClickHouse: &chmigrate.Config{ClientAddr: addr, Database: "hub", App: "contentkit_signal", Cluster: cluster},
})
_, _ = content.AssignTenant(ctx, pool, "doujins", "doujins") // once, after the first migrate on an existing install

Existing installations keep their lineages (socialkit, searchkit/ migrations.LegacyPostgres, searchkit_signal); see docs/migration.md.

Runtime

rt, _ := contentkit.NewRuntime(ctx, contentkit.RuntimeConfig{
	EmbeddedConfig: contentkit.EmbeddedConfig{PG: pool, PGSchema: "doujins_searchkit", Tenant: "doujins", CH: ch, CHDatabase: "hub"},
	Content: content.Options{Schema: "doujins", Identity: identity, Authz: authz, Resolver: resolver, ContentKinds: []string{"gallery", "post"}},
})
mux.Handle("/api/social/", http.StripPrefix("/api/social", rt.Handler()))
counts, _ := rt.Content.Counts(ctx, []contentkit.ContentRef{rt.Content.Ref("gallery", "42")})
_ = worker.SyncOnce(ctx, rt.WorkerOptions(hostWorkerOptions)) // host documents + posts

rt is the Hub (search, typeahead, signals, discovery) plus rt.Content (interactions). See HOST_INTEGRATION.md.

Documents are one per content reference and language: a gallery with an English original, an English colored edition and a Spanish original is three documents keyed by version. Hosts mark changes in the transaction that changes content and run one worker tick per schedule:

_ = search.MarkDirty(ctx, tx, schema, []search.DirtyMark{{DocumentKey: search.DocumentKey{ContentRef: ref, Language: "en"}}})

_ = worker.SyncOnce(ctx, worker.Options{
	Pool: pool, Schema: schema, Tenant: "doujins",
	SupportedLanguages: []string{"en", "es"}, ContentKinds: []string{"gallery"},
	ListContent:           listGalleries,          // bounded pages of refs, for backfill
	BuildKeywordDocuments: buildGalleryDocuments,  // refs -> KeywordDocument{Title, Aliases, Keywords}
	Sink:                  nil,                    // optional DocumentSink
})

Querying groups documents per work before paging; the host's eligibility join decides, per document, whether that one row is visible and which is preferred:

client, _ := contentkit.NewClient(contentkit.ClientConfig{Pool: pool, Schema: schema, Tenant: "doujins"})
page, _ := client.Search(ctx, query, contentkit.SearchOptions{
	Language: "es", ContentKinds: []string{"gallery"}, Limit: 20,
	Eligibility: &contentkit.Eligibility{SQL: eligibilitySQL, Args: args},
})
for _, hit := range page.Hits { /* hit.ContentID (work), hit.Version(), hit.Language, hit.Score */ }

SearchHit.Score is the keyword match tier (exact title 1, alias 0.9, token/prefix 0.75–0.77, typo 0.5–0.52). SearchWithTrace returns retrieval provenance for evaluation. Query limits: 256 characters, 16 tokens, a two-second ceiling per request.

Ports

Port Called when Contract
DocumentSink the worker publishes or deletes a document Upsert(PublishedDocument), Delete(DocumentKey, Version); at-least-once, atomic newer-version wins across both operations, with a retained deletion tombstone; a failing sink keeps the row queued and never blocks the keyword index

DocumentSink is a neutral document change feed for external indexes, caches or audit consumers. ContentKit ships no sink implementation or AI-specific configuration.

Errors

Both mounted handlers (content.Runtime.Handler, taxonomy.Handler) answer failures with one flat body. Branch on code; error is a human message and may change.

{"error":"not found","code":"not_found"}
Status Code Meaning
400 invalid_request malformed or semantically invalid input
401 unauthorized no identity
403 forbidden identity present, not permitted
404 not_found absent, unpublished or soft-deleted (existence is hidden)
409 conflict state or revision conflict
422 moderation_rejected a ContentModerator refused the write; error is the author-facing reason
501 not_configured the host never wired the port this route needs (MediaStore, AnswerClassifier)
500 tenant_mismatch a host port answered with another tenant's data
500 internal_error anything else

5xx bodies carry no cause: it goes to Options.Logger (slog.Default() when unset) with the request method, path, status and duration. Postgres constraint names, driver text and stack traces are logged, never served.

Taxonomy

Nodes, names, edges and assignments are tenant-scoped; effective tags are the work's assignments ∪ the selected version's; RequireAll makes a multi-node filter hold on one eligible version inside the same join as search. Apply migrations.Taxonomy after the keyword profile; see docs/taxonomy-migration.md.

store, _ := taxonomy.New(taxonomy.Options{Pool: pool, Schema: schema, Tenant: "doujins",
	Kinds: []string{"tag", "artist", "character", "series", "voice_actor"}, Languages: []string{"en", "es"},
	CountEligibility: &search.Eligibility{SQL: releasedVersionSQL}})
_ = store.WithTx(tx).Assign(ctx, []taxonomy.Assignment{{ContentRef: g1.WithVersion(v2), TaxonomyID: "colored"}}, taxonomy.AssignOptions{})
tags, _ := store.EffectiveTags(ctx, []contentkit.ContentRef{g1.WithVersion(v2)})
filter, args, _ := taxonomy.RequireAll(schema, []taxonomy.TaxonomyID{"colored"})
page, _ := client.Search(ctx, q, contentkit.SearchOptions{Language: "es", ContentKinds: []string{"gallery"}, FilterSQL: filter, FilterArgs: args, Eligibility: elig})
mux.Handle("/admin/taxonomy/", http.StripPrefix("/admin/taxonomy", taxonomy.Handler(store)))

ListNodes backs a catalog index page directly: the display name in the request language (falling back to the store's configured order, or pinned with LanguageMode), the per-language content count, an A-Z index, a name+alias search, hide-empty, five orders and offset paging with a total. The zero ListOptions keeps the keyset contract a full admin sync wants.

page, _ := store.ListNodes(ctx, taxonomy.ListOptions{Kind: "artist", Language: "es",
	ContentKind: "gallery", MinCount: 1, Sort: taxonomy.SortCount, Offset: 40, Limit: 20})
// page.Total, and per row: Name, NameLanguage, Count.
letter, _ := store.ListNodes(ctx, taxonomy.ListOptions{Kind: "artist", Language: "es", NamePrefix: "a"})
cast, _ := store.ListNodes(ctx, taxonomy.ListOptions{Kind: "character", Related: "s-fate", Relation: taxonomy.RelationMemberOf})

Worker: ContentKinds: append(hostKinds, store.Kinds()...), ListContent: store.Lister(listGalleries), BuildKeywordDocuments: store.Builder(buildGalleryDocuments).

Signal plane and discovery

hub, _ := contentkit.NewEmbedded(contentkit.EmbeddedConfig{
	PG: pool, PGSchema: schema, CH: ch, CHDatabase: "hub", Tenant: "doujins",
	Scorers:  map[string]signal.Scorer{"gallery": galleryScorer},
	Catalogs: map[string]contentkit.ContentCatalog{"gallery": galleryCatalog},
})
g1 := hub.Content("gallery", "g1")
_ = hub.RecordSignals(ctx, []signal.Signal{{ContentRef: g1, Subject: user, Type: signal.TypeView, EventID: sessionID, Revision: checkpoint, OccurredAt: start, Progress: 95, ProgressMax: 100}})
states, _ := hub.States(ctx, user, []contentkit.ContentRef{g1, g1.WithVersion("v2")})
top, _ := hub.Popular(ctx, "gallery", signal.PopularOptions{Window: signal.LastDays(30, time.Now())})
  • Identity is (tenant, content ref, subject, type, event id); retries, reordering and revisions converge, nothing is incremented. Record the work view and, separately, the selected version's view (g1.WithVersion(v)).
  • Work reads (History, Popular, SeenIDs, co-engagement) count the work once across versions; States and Metrics read exactly the references given, work or version.
  • Windows are whole UTC days, 7/30/90/365/all, no decay.
  • Rank by a named policy, not the default rank: popularity.ByName("v1"), popularity.New(popularity.Config{Source: hub, Policy: policy}), then ranker.Popular / ranker.Scores / ranker.Taxonomy (docs/popularity-policy.md).
  • EraseSubjects is account erasure with a quorum-written fence; see HOST_INTEGRATION.md.
  • Reactions and favorites reach the signal plane through ContentKit's own preference outbox (rt.DeliverPreferences, rt.ReplayPreferences): a compact snapshot per subject × reference × axis, revisions from a sequence under the source lock, revision-exact acknowledgment; see HOST_INTEGRATION.md.

Testing

CONTENTKIT_TEST_URL=postgres://...  CONTENTKIT_PROFILE_URL=postgres://... \
CONTENTKIT_TEST_CH_ADDR=localhost:9000 CONTENTKIT_TEST_CH_USER=... CONTENTKIT_TEST_CH_PASSWORD=... CONTENTKIT_TEST_CH_CLUSTER=... \
go test ./... -race -count=1 -p 2

Tests run against real PGroonga Postgres and ClickHouse+Keeper and skip without the variables; CONTENTKIT_PROFILE_URL needs CREATEDB (and pgvector for the legacy-lineage convergence test). Regenerate the eval baseline with CONTENTKIT_EVAL_UPDATE=1.

Documentation

Overview

Package contentkit is the deterministic content library: tenant-scoped keyword search over host content, the ClickHouse signal plane, and the discovery reads over both. The DocumentSink port publishes document changes to optional external consumers.

Index

Constants

View Source 
const PreferenceEventID = "current"

PreferenceEventID is the stable signal identity of a subject's current preference on one axis: (tenant, canonical reference, subject, axis, "current"). Every delivery of a newer snapshot supersedes the previous one by revision; a replay carries the same revision and converges.

Variables

View Source 
var ErrSignalPlaneDisabled = errors.New("contentkit: signal plane disabled (no ClickHouse configured)")

ErrSignalPlaneDisabled is returned by signal/discovery methods when the hub was constructed without a ClickHouse connection.

Functions

func Migrate 

func Migrate(ctx context.Context, cfg MigrateConfig) error

Migrate applies every ContentKit lineage: social in Schema, keyword in SearchSchema, optional taxonomy in SearchSchema, and the configured signal plane.

func NewEvalRunner 

func NewEvalRunner(client *Client, base SearchOptions) eval.CaseRunner

NewEvalRunner adapts a Client to eval.CaseRunner so a golden suite can be executed against real search. The base options carry cross-case settings (LanguageMode and filters); each case overrides Language, ContentKinds, and Limit from its own definition.

This adapter is the single seam where the client meets the dependency-free eval package.

Types

type CandidateTrace 

type CandidateTrace struct {
	Key   TraceKey `json:"key"`
	Rank  int      `json:"rank"`
	Score float32  `json:"score"`
}

CandidateTrace records one source candidate at its raw source rank.

type CatalogQuery 

type CatalogQuery struct {
	Limit int
}

CatalogQuery bounds a Universe read. Limit 0 = host-defined default.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client answers keyword search and typeahead for one tenant.

func NewClient 

func NewClient(cfg ClientConfig) (*Client, error)

func (*Client) Search 

func (c *Client) Search(ctx context.Context, userText string, opts SearchOptions) (SearchResult, error)

Search returns one page of content items. Documents from every searched language are grouped per work before Offset and Limit apply; each hit carries the matched document's reference and language.

func (*Client) SearchWithTrace 

func (c *Client) SearchWithTrace(ctx context.Context, userText string, opts SearchOptions) (SearchResult, SearchTrace, error)

SearchWithTrace executes Search and returns opt-in retrieval provenance. On failure, the returned trace contains all work completed before the error.

func (*Client) Tenant 

func (c *Client) Tenant() string

Tenant returns the tenant this client is scoped to.

func (*Client) Typeahead 

func (c *Client) Typeahead(ctx context.Context, userText string, opts TypeaheadOptions) ([]TypeaheadHit, error)

Typeahead returns suggestions while a user is typing (typos/substring matching), one per content item, grouped before Limit.

type ClientConfig 

type ClientConfig struct {
	Pool   *pgxpool.Pool
	Schema string
	// Tenant scopes every document, query and result. Required.
	Tenant string

	// Defaults.
	DefaultLanguage string
	DefaultLimit    int
}

ClientConfig configures the keyword search client of one tenant.

type ContentCatalog 

type ContentCatalog interface {
	Universe(ctx context.Context, tenant string, contentKind string, q CatalogQuery) ([]string, error)
}

ContentCatalog supplies the content "universe" for Unseen: live, non-deleted content ids of a kind, read from the host's own tables. The host owns visibility and gating (premium, region, ...) — ContentKit never interprets them. Order defines Unseen order (recommended: newest first).

type ContentCatalogFunc 

type ContentCatalogFunc func(ctx context.Context, tenant string, contentKind string, q CatalogQuery) ([]string, error)

ContentCatalogFunc adapts a function to the ContentCatalog interface.

func (ContentCatalogFunc) Universe 

func (f ContentCatalogFunc) Universe(ctx context.Context, tenant string, contentKind string, q CatalogQuery) ([]string, error)

type ContentKey 

type ContentKey = contentref.ContentKey

ContentKey is the comparable form of a ContentRef.

type ContentRef 

type ContentRef = contentref.ContentRef

ContentRef is the tenant-scoped reference to host-owned content (the work or one of its versions). See contentref.

type ContributionTrace 

type ContributionTrace struct {
	SourceIndex  int     `json:"source_index"`
	SourceRank   int     `json:"source_rank"`
	Weight       float32 `json:"weight"`
	Contribution float32 `json:"contribution"`
}

ContributionTrace records one exact source contribution to a result score.

type DocumentKey 

type DocumentKey = search.DocumentKey

DocumentKey identifies one keyword document: a ContentRef in one language.

type DocumentSink 

type DocumentSink = search.DocumentSink

DocumentSink is the optional document port (see search.DocumentSink): the worker delivers every published keyword document to it at least once.

type Eligibility 

type Eligibility = search.Eligibility

Eligibility is the host's per-document eligibility join; see search.Eligibility.

type EmbeddedConfig 

type EmbeddedConfig struct {
	// Content plane (Postgres). PG + PGSchema are required. PGSchema should
	// be a dedicated schema (not the host app's schema) to avoid table
	// collisions.
	PG       *pgxpool.Pool
	PGSchema string

	// Content-plane defaults (as in ClientConfig).
	DefaultLanguage string
	DefaultLimit    int
	// DefaultRRFK controls deterministic popularity/discovery fusion (default 60).
	DefaultRRFK int

	// Signal plane (ClickHouse). Optional: omit CH to run content-only
	// (signal/discovery methods return ErrSignalPlaneDisabled). CHDatabase
	// is the hub's dedicated ClickHouse database; apply
	// migrations.SignalClickHouse and gate startup on signal.CheckSchema.
	CH         signal.Conn
	CHDatabase string

	// Tenant is the single tenant of this embedded hub. Required: every
	// document, signal, cursor and result carries it.
	Tenant string

	// Scorers maps content kind → host Scorer. When a signal arrives for a
	// registered kind, the scorer's result overwrites Score / Progress /
	// ProgressMax / Completed before recording.
	Scorers map[string]signal.Scorer

	// Catalogs maps content kind → host ContentCatalog (the Unseen universe).
	Catalogs map[string]ContentCatalog
}

EmbeddedConfig configures an in-process hub against the shared DB.

type EmbeddedHub 

type EmbeddedHub struct {
	// contains filtered or unexported fields
}

EmbeddedHub implements Hub in-process. Construct with NewEmbedded.

func NewEmbedded 

func NewEmbedded(cfg EmbeddedConfig) (*EmbeddedHub, error)

NewEmbedded builds the embedded hub: in-process, shared DB, one tenant.

func (*EmbeddedHub) Attribution 

func (h *EmbeddedHub) Attribution(ctx context.Context, opts signal.AttributionOptions) (signal.AttributionPage, error)

Attribution exports renders at one stage with their clicks joined (see signal.Store.Attribution); the evaluation dataset source.

func (*EmbeddedHub) Client 

func (h *EmbeddedHub) Client() *Client

Client returns the underlying content-plane client (advanced use).

func (*EmbeddedHub) Content 

func (h *EmbeddedHub) Content(contentKind, contentID string) ContentRef

Content returns a reference to a work of this hub's tenant.

func (*EmbeddedHub) EnforceErasures 

func (h *EmbeddedHub) EnforceErasures(ctx context.Context) (signal.ErasureReport, error)

EnforceErasures physically removes residue of every recorded erasure of this tenant (see signal.Store.EnforceErasures): schedule it and run it after every restore.

func (*EmbeddedHub) EraseSubjects 

func (h *EmbeddedHub) EraseSubjects(ctx context.Context, subjects []signal.Subject) (signal.ErasureReport, error)

EraseSubjects permanently erases subjects from this tenant's signal plane: see signal.Store.EraseSubjects for the completion contract. Shared accounts exist in several tenants: each host erases its own tenant.

func (*EmbeddedHub) Forget 

func (h *EmbeddedHub) Forget(ctx context.Context, subject signal.Subject, contentKind, contentID string) error

Forget erases the subject's signals for one work and its versions (contentID set) or a whole content kind (contentID empty) — host "clear my history" support.

func (*EmbeddedHub) ForgetExposures 

func (h *EmbeddedHub) ForgetExposures(ctx context.Context, subject signal.Subject) error

ForgetExposures clears a subject's result-list exposures (search history).

func (*EmbeddedHub) History 

func (h *EmbeddedHub) History(ctx context.Context, subject signal.Subject, opts signal.HistoryOptions) ([]signal.StateRow, error)

func (*EmbeddedHub) HistoryCount 

func (h *EmbeddedHub) HistoryCount(ctx context.Context, subject signal.Subject, opts signal.HistoryOptions) (int64, error)

HistoryCount returns the total row count History would paginate over.

func (*EmbeddedHub) Inventory 

func (h *EmbeddedHub) Inventory(ctx context.Context) ([]signal.InventoryRow, error)

Inventory reports canonical event volume per content kind and signal type.

func (*EmbeddedHub) Metrics 

func (h *EmbeddedHub) Metrics(ctx context.Context, refs []ContentRef, window signal.Window) (map[ContentKey]signal.ContentMetrics, error)

Metrics returns named window metrics (viewers, views, completions, feedback, ...) for the references (works or versions).

func (*EmbeddedHub) Popular 

func (h *EmbeddedHub) Popular(ctx context.Context, contentKind string, opts signal.PopularOptions) ([]signal.PopularHit, error)

func (*EmbeddedHub) PopularityFor 

func (h *EmbeddedHub) PopularityFor(ctx context.Context, contentKind string, ids []string, window signal.Window) (map[string]float64, error)

PopularityFor scores a fixed candidate set (work ids of one kind) by the popularity ranking, returning content_id -> score. Use to rank a host- filtered universe (e.g. "galleries of artist X by popularity").

func (*EmbeddedHub) PurgeContentKinds 

func (h *EmbeddedHub) PurgeContentKinds(ctx context.Context, contentKinds []string) error

PurgeContentKinds irreversibly deletes whole content kinds from this tenant's signal plane (see signal.Store.PurgeContentKinds).

func (*EmbeddedHub) Recommend 

func (h *EmbeddedHub) Recommend(ctx context.Context, subject signal.Subject, opts RecommendOptions) ([]RecHit, error)

Recommend returns "for you" recommendations for a subject: co-engagement seeded from the subject's high-signal works ("subjects who engaged with your favorites also engaged with..."), fused across seeds (RRF), excluding already-seen, with a popularity fallback for cold start. Returns ranked references; the host hydrates.

func (*EmbeddedHub) RecordExposures 

func (h *EmbeddedHub) RecordExposures(ctx context.Context, exposures []signal.Exposure) error

RecordExposures logs one row per result list and stage (served, rendered, visible) so clicks can be attributed to what was actually exposed. Hosts call it once per list per stage, never per item.

func (*EmbeddedHub) RecordSignals 

func (h *EmbeddedHub) RecordSignals(ctx context.Context, signals []signal.Signal) error

RecordSignals applies each content kind's registered Scorer, then records the batch (see signal.Store.RecordSignals). A scorer error records nothing.

func (*EmbeddedHub) RefreshCoEngagement 

func (h *EmbeddedHub) RefreshCoEngagement(ctx context.Context, opts signal.RefreshCoEngagementOptions) error

RefreshCoEngagement (re)materializes the content_pairs co-engagement rollup for this tenant (see signal.Store.RefreshCoEngagement). Run periodically.

func (*EmbeddedHub) RepairProjections 

func (h *EmbeddedHub) RepairProjections(ctx context.Context, opts signal.RepairOptions) (signal.RepairResult, error)

RepairProjections is the bounded, host-scheduled projection repair (see signal.Store.RepairProjections): run it periodically with IngestedSince for crash repair, and with Rebuild over a window after projection changes.

func (*EmbeddedHub) Search 

func (h *EmbeddedHub) Search(ctx context.Context, userText string, opts HubSearchOptions) (SearchResult, error)

func (*EmbeddedHub) SeenIDs 

func (h *EmbeddedHub) SeenIDs(ctx context.Context, subject signal.Subject, contentKind string) (map[string]struct{}, error)

SeenIDs returns the subject's seen-set for one content kind (the signal-plane half of the unseen anti-join). Use when the host wants to run its own diff against a custom-filtered universe instead of Unseen's registered catalog.

func (*EmbeddedHub) SimilarTo 

func (h *EmbeddedHub) SimilarTo(ctx context.Context, ref ContentRef, opts SimilarOptions) ([]RecHit, error)

func (*EmbeddedHub) States 

func (h *EmbeddedHub) States(ctx context.Context, subject signal.Subject, refs []ContentRef) (map[ContentKey]signal.State, error)

func (*EmbeddedHub) Tenant 

func (h *EmbeddedHub) Tenant() string

Tenant returns the pinned tenant value.

func (*EmbeddedHub) Typeahead 

func (h *EmbeddedHub) Typeahead(ctx context.Context, userText string, opts TypeaheadOptions) ([]TypeaheadHit, error)

func (*EmbeddedHub) Unseen 

func (h *EmbeddedHub) Unseen(ctx context.Context, subject signal.Subject, opts UnseenOptions) ([]string, error)

Unseen returns catalog ids the subject has not seen (max_progress > 0 defines "seen"): host universe MINUS the subject's seen-set. The host catalog applies its own visibility/premium gating against its own tables.

type EmptyReason 

type EmptyReason string

EmptyReason explains a successful empty response. 

const (
	EmptyReasonNormalizedQuery EmptyReason = "normalized_query_empty"
	EmptyReasonNoCandidates    EmptyReason = "no_candidates"
)

type Hub 

type Hub interface {
	// Tenant returns the tenant this hub instance is scoped to.
	Tenant() string

	// Content plane.
	Search(ctx context.Context, userText string, opts HubSearchOptions) (SearchResult, error)
	Typeahead(ctx context.Context, userText string, opts TypeaheadOptions) ([]TypeaheadHit, error)
	SimilarTo(ctx context.Context, ref ContentRef, opts SimilarOptions) ([]RecHit, error)

	// Signal plane.
	RecordSignals(ctx context.Context, signals []signal.Signal) error
	RecordExposures(ctx context.Context, exposures []signal.Exposure) error
	ForgetExposures(ctx context.Context, subject signal.Subject) error
	Attribution(ctx context.Context, opts signal.AttributionOptions) (signal.AttributionPage, error)
	Forget(ctx context.Context, subject signal.Subject, contentKind, contentID string) error
	EraseSubjects(ctx context.Context, subjects []signal.Subject) (signal.ErasureReport, error)
	EnforceErasures(ctx context.Context) (signal.ErasureReport, error)

	// Discovery plane.
	History(ctx context.Context, subject signal.Subject, opts signal.HistoryOptions) ([]signal.StateRow, error)
	HistoryCount(ctx context.Context, subject signal.Subject, opts signal.HistoryOptions) (int64, error)
	SeenIDs(ctx context.Context, subject signal.Subject, contentKind string) (map[string]struct{}, error)
	Unseen(ctx context.Context, subject signal.Subject, opts UnseenOptions) ([]string, error)
	States(ctx context.Context, subject signal.Subject, refs []ContentRef) (map[ContentKey]signal.State, error)
	Metrics(ctx context.Context, refs []ContentRef, window signal.Window) (map[ContentKey]signal.ContentMetrics, error)
	Popular(ctx context.Context, contentKind string, opts signal.PopularOptions) ([]signal.PopularHit, error)
	PopularityFor(ctx context.Context, contentKind string, ids []string, window signal.Window) (map[string]float64, error)
	Recommend(ctx context.Context, subject signal.Subject, opts RecommendOptions) ([]RecHit, error)

	// Maintenance.
	RefreshCoEngagement(ctx context.Context, opts signal.RefreshCoEngagementOptions) error
	RepairProjections(ctx context.Context, opts signal.RepairOptions) (signal.RepairResult, error)
	Inventory(ctx context.Context) ([]signal.InventoryRow, error)
	PurgeContentKinds(ctx context.Context, contentKinds []string) error
}

Hub is the single surface host apps program against: content-plane queries (search/typeahead), the signal plane (RecordSignals), and the discovery plane (reads over content × signals). All methods return ranked content references (+ per-subject State); the host hydrates them into cards from its own DB. Every method is scoped to the tenant pinned at construction; a reference of another tenant is an error.

type HubSearchOptions 

type HubSearchOptions struct {
	SearchOptions
	Personalize *Personalization
}

HubSearchOptions extends content SearchOptions with optional signal-aware personalization.

type KeywordDocument 

type KeywordDocument = search.KeywordDocument

KeywordDocument is the host's canonical search input for one document.

type LanguageMode 

type LanguageMode string
const (
	// LanguageModeExact uses only the requested language.
	LanguageModeExact LanguageMode = "exact"
	// LanguageModeFallbackEnglish uses requested language first, then English.
	LanguageModeFallbackEnglish LanguageMode = "fallback_en"
)

type MigrateConfig 

type MigrateConfig struct {
	// DB holds DDL credentials for the host Postgres. Required.
	DB *sql.DB
	// Schema is the host schema: the social lineage lands here. Required.
	Schema string
	// SearchSchema receives the keyword lineage; "" = Schema.
	SearchSchema string
	// SearchApp is the keyword lineage's ledger label; "" = "contentkit".
	// Existing keyword installations keep the label they were created with.
	SearchApp string
	// LegacySearch applies migrations.LegacyPostgres (existing installations
	// created from the pre-ContentKit combined lineage; docs/migration.md).
	LegacySearch bool
	// Taxonomy applies the optional catalog lineage in SearchSchema after the
	// keyword lineage it depends on. Its ledger app is contentkit_taxonomy.
	Taxonomy bool
	// ClickHouse applies the signal lineage when set; PostgresDB defaults to DB.
	ClickHouse *chmigrate.Config
}

MigrateConfig names the stores contentkit.Migrate applies the lineages to.

type Personalization 

type Personalization struct {
	Subject signal.Subject

	// PopularityWeight is the RRF weight of the candidate-set popularity
	// list blended with the content ranking. Defaults to 0.25.
	PopularityWeight float32
	// PopularityWindow bounds candidate popularity (zero = all time).
	PopularityWindow signal.Window

	// AffinityWeight boosts works the subject already engaged with by
	// (1 + AffinityWeight·last_score/100). 0 = off.
	AffinityWeight float32

	// DemoteSeen demotes already-seen / completed works.
	DemoteSeen bool
	// SeenPenalty multiplies seen-but-not-completed scores (default 0.85).
	SeenPenalty float32
	// CompletedPenalty multiplies completed scores (default 0.6).
	CompletedPenalty float32
	// DislikePenalty multiplies works the subject has net-negative explicit
	// feedback for (default 0.3). Always applied when view context is loaded
	// (i.e. AffinityWeight > 0 or DemoteSeen).
	DislikePenalty float32
}

Personalization fuses signal aggregates into search ranking. Recall is unchanged — this is a ranking-only layer over the candidate set. A per-request toggle the host flips (e.g. only for logged-in users).

type PublishedDocument 

type PublishedDocument = search.PublishedDocument

PublishedDocument is one keyword document as delivered to a DocumentSink.

type RecHit 

type RecHit struct {
	ContentRef
	Score float32
}

RecHit is one ranked work from co-engagement or recommendations.

type RecommendOptions 

type RecommendOptions struct {
	// ContentKinds are the candidate kinds to recommend. Required.
	ContentKinds []string

	// Limit caps results (default: client default limit).
	Limit int

	// SeedLimit is how many of the subject's highest-signal works seed
	// co-engagement candidates (default 5).
	SeedLimit int

	// SeedContentKinds limits which kinds may seed (default: any).
	SeedContentKinds []string

	// IncludeSeen keeps already-seen works in results (default: excluded).
	IncludeSeen bool

	// PopularWindow is the popularity window used to fill out results on
	// cold start or thin candidate sets (zero = all time, like every Window).
	PopularWindow signal.Window
}

RecommendOptions controls Recommend ("for you": subject → works).

type ResultTrace 

type ResultTrace struct {
	Key           TraceKey            `json:"key"`
	Rank          int                 `json:"rank"`
	Score         float32             `json:"score"`
	ScoreKind     ScoreKind           `json:"score_kind"`
	Contributions []ContributionTrace `json:"contributions"`
}

ResultTrace records one returned item and its best document's contributions.

type RetrievalBackend 

type RetrievalBackend string

RetrievalBackend identifies one candidate source. 

const (
	BackendKeyword RetrievalBackend = "keyword"
)

type Runtime 

type Runtime struct {
	*EmbeddedHub
	Content *content.Runtime
}

Runtime is the one surface a host wires: the Hub (search, typeahead, signals, discovery) plus the content module and its HTTP routes.

func NewRuntime 

func NewRuntime(ctx context.Context, cfg RuntimeConfig) (*Runtime, error)

NewRuntime builds the hub and the content module over the host pool.

func (*Runtime) DeliverPreferences 

func (r *Runtime) DeliverPreferences(ctx context.Context, after content.PreferenceKey, pageSize, maxRows int) (content.PreferenceDelivery, error)

DeliverPreferences runs one bounded sweep of the tenant's pending preference snapshots into the signal plane (see content.Runtime.DeliverPreferences). Schedule it from the host's worker; with the signal plane disabled it returns ErrSignalPlaneDisabled and every row stays pending.

func (*Runtime) EraseSubjects 

func (r *Runtime) EraseSubjects(ctx context.Context, subjects []signal.Subject) (signal.ErasureReport, error)

EraseSubjects erases every configured runtime plane: signals, preference obligations, and C4 private-source/provider data. Current approved authored content remains under host retention policy (content.EraseSubjects). ContentKit-owned reactions, favorites, poll votes, preference obligations and private submissions are removed atomically behind the source fence. EmbeddedHub.EraseSubjects is the explicit analytics-only lower-level API.

AuthKit ACK means durable acceptance by the host's deletion ledger, not this downstream completion. Retry while error != nil or !report.Complete(). A disabled signal plane is intentionally absent. Remaining may include pending plane markers content_plane or signal_plane when a plane could not complete; these are not estimates of retained provider rows.

func (*Runtime) Handler 

func (r *Runtime) Handler() http.Handler

Handler returns the content routes (comments, reactions, favorites, polls, posts). Mount it under a prefix after the host's auth middleware.

func (*Runtime) ReplayPreferences 

func (r *Runtime) ReplayPreferences(ctx context.Context, after content.PreferenceKey, pageSize, maxRows int) (content.PreferenceDelivery, error)

ReplayPreferences replays every snapshot from after into the signal plane (acknowledged rows and zeros included): the repair for sink loss, resumable from the returned Next.

func (*Runtime) WorkerOptions 

func (r *Runtime) WorkerOptions(host worker.Options) worker.Options

WorkerOptions returns the host's keyword worker options extended with ContentKit's own documents: posts (content.KindPost) are listed and built by the content module, every other kind by the host callbacks.

type RuntimeConfig 

type RuntimeConfig struct {
	EmbeddedConfig
	Content content.Options
}

RuntimeConfig configures one tenant's full ContentKit: the search and signal planes (EmbeddedConfig) and the interaction module (Content). Pool, tenant and keyword schema are shared: Content.Pool, Content.Tenant and Content.SearchSchema are filled from the hub configuration when empty.

type ScoreKind 

type ScoreKind string

ScoreKind identifies the numeric domain of a candidate or result score. 

const (
	ScoreKeywordMatch ScoreKind = "keyword_match"
)

type SearchHit 

type SearchHit struct {
	// ContentRef names the matched document's content: the work, or the
	// version when the document was indexed per version.
	ContentRef
	// Language is the matched document's language.
	Language string
	// Score ranks the item by its best matching document in any searched
	// language, using the keyword match tier.
	Score float32
}

SearchHit is one content item, represented by its matched document.

type SearchOptions 

type SearchOptions struct {
	Language string
	// Defaults to LanguageModeExact when omitted.
	LanguageMode LanguageMode

	// ContentKinds selects the kinds searched. Required.
	ContentKinds []string

	// Limit is the page size in content items; Offset skips items. Documents
	// are grouped per item before either applies.
	Limit  int
	Offset int
	// CandidateLimit is the document window requested from each retrieval
	// source (per language) before grouping. It defaults to twice Offset+Limit
	// (at least 100) and is clamped to at least Offset+Limit. Pass the same
	// value on every page when a truncated window must stay identical.
	CandidateLimit int

	// Eligibility maps each document to the host's access, publication and
	// version-trait rules on that one document. Without it every document of
	// a work is eligible.
	Eligibility *Eligibility

	FilterSQL  string
	FilterArgs map[string]any
}

type SearchResult 

type SearchResult struct {
	Hits []SearchHit
	// HasMore is true when items follow this page in the grouped retrieval, or
	// when Truncated: documents beyond the window were never ranked, so the
	// next page may still be non-empty.
	HasMore bool
	// Truncated reports that a candidate window filled. Raise CandidateLimit
	// for complete deep pagination.
	Truncated bool
}

SearchResult is one page of items.

type SearchTrace 

type SearchTrace struct {
	NormalizedQuery         string        `json:"normalized_query"`
	RequestedLanguage       string        `json:"requested_language,omitempty"`
	RequestedLanguageMode   LanguageMode  `json:"requested_language_mode,omitempty"`
	Languages               []string      `json:"languages,omitempty"`
	RequestedResultLimit    int           `json:"requested_result_limit"`
	ResultLimit             int           `json:"result_limit"`
	RequestedCandidateLimit int           `json:"requested_candidate_limit"`
	CandidateLimit          int           `json:"candidate_limit"`
	Sources                 []SourceTrace `json:"sources,omitempty"`
	Results                 []ResultTrace `json:"results,omitempty"`
	EmptyReason             EmptyReason   `json:"empty_reason,omitempty"`
	ErrorCategory           string        `json:"error_category,omitempty"`
}

SearchTrace contains opt-in effective configuration and retrieval provenance.

type SimilarOptions 

type SimilarOptions struct {
	Limit int
	// ContentKinds limits result kinds (default: any).
	ContentKinds []string
	// Window bounds the co-engagement scan (default all time).
	Window signal.Window

	// ExcludeSeenFor drops works this subject has already seen (and always
	// drops works they negatively reacted to).
	ExcludeSeenFor *signal.Subject
}

SimilarOptions controls SimilarTo: "more like this" from co-engagement, subjects who engaged with the anchor also engaged with the result.

type SourceStatus 

type SourceStatus string

SourceStatus records whether an attempted retrieval source succeeded. 

const (
	SourceStatusSucceeded SourceStatus = "succeeded"
	SourceStatusFailed    SourceStatus = "failed"
)

type SourceTrace 

type SourceTrace struct {
	Backend       RetrievalBackend `json:"backend"`
	Language      string           `json:"language"`
	ScoreKind     ScoreKind        `json:"score_kind"`
	Limit         int              `json:"limit"`
	Status        SourceStatus     `json:"status"`
	ErrorCategory string           `json:"error_category,omitempty"`
	Candidates    []CandidateTrace `json:"candidates,omitempty"`
}

SourceTrace records one language-specific keyword retrieval and its candidates.

type TaxonomyID 

type TaxonomyID = contentref.TaxonomyID

TaxonomyID identifies a generic ContentKit catalog record (tag, artist, series, creator, character, voice actor).

type TraceKey 

type TraceKey struct {
	ContentRef
	Language string `json:"language"`
}

TraceKey identifies a document in retrieval provenance.

type TypeaheadHit 

type TypeaheadHit struct {
	ContentRef
	Language string
	Score    float32
}

TypeaheadHit is one suggested content item and its matched document.

type TypeaheadOptions 

type TypeaheadOptions struct {
	Language string
	// Defaults to LanguageModeExact when omitted.
	LanguageMode  LanguageMode
	ContentKinds  []string
	Limit         int
	MinSimilarity float32
	FilterSQL     string
	FilterArgs    map[string]any
	// Eligibility groups suggestions per content item; see SearchOptions.
	Eligibility *Eligibility
}

type UnseenOptions 

type UnseenOptions struct {
	// ContentKind selects which catalog universe to diff against. Required.
	ContentKind string
	// Limit caps the returned ids (default 50). Order follows the host
	// catalog's Universe order.
	Limit int
	// CatalogLimit is passed through to the host catalog's Universe call
	// (0 = host default).
	CatalogLimit int
}

UnseenOptions controls Unseen reads.

Source Files

View all Source files

Directories

Path Synopsis
Package content is ContentKit's interaction module: posts, comments, reactions, favorites and polls over tenant-scoped content references, stored in the host schema's social_* tables.
 Package content is ContentKit's interaction module: posts, comments, reactions, favorites and polls over tenant-scoped content references, stored in the host schema's social_* tables.
Package contentref defines the reference vocabulary every ContentKit package and port shares: a tenant-scoped reference to host-owned content and the identity of a generic taxonomy record.
 Package contentref defines the reference vocabulary every ContentKit package and port shares: a tenant-scoped reference to host-owned content and the identity of a generic taxonomy record.
Package eval provides dependency-free golden-query evaluation, aggregate search-quality metrics, versioned reports, baseline comparison, and score-domain-safe threshold sweeps.
 Package eval provides dependency-free golden-query evaluation, aggregate search-quality metrics, versioned reports, baseline comparison, and score-domain-safe threshold sweeps.
internal
normalize
Package normalize cleans user query text before retrieval.
 Package normalize cleans user query text before retrieval.
pgtest
Package pgtest provisions disposable keyword-profile schemas on the CONTENTKIT_TEST_URL Postgres for integration tests.
 Package pgtest provisions disposable keyword-profile schemas on the CONTENTKIT_TEST_URL Postgres for integration tests.
signaltest
Package signaltest provisions disposable signal-plane ClickHouse databases for integration tests by applying the real migration lineage.
 Package signaltest provisions disposable signal-plane ClickHouse databases for integration tests by applying the real migration lineage.
Package migrations embeds ContentKit's migratekit lineages: the keyword Postgres profile, the pre-ContentKit combined Postgres lineage that converges on it, the taxonomy and social Postgres lineages, and the ClickHouse signal plane.
 Package migrations embeds ContentKit's migratekit lineages: the keyword Postgres profile, the pre-ContentKit combined Postgres lineage that converges on it, the taxonomy and social Postgres lineages, and the ClickHouse signal plane.
Package popularity ranks host content by a named, bounded policy over the signal plane's canonical window metrics (docs/popularity-policy.md).
 Package popularity ranks host content by a named, bounded policy over the signal plane's canonical window metrics (docs/popularity-policy.md).
Package search is ContentKit's keyword retrieval over content_search_documents: exact names and aliases, native-script prefixes and bounded typos for every language, with the host's eligibility join applied inside every route.
 Package search is ContentKit's keyword retrieval over content_search_documents: exact names and aliases, native-script prefixes and bounded typos for every language, with the host's eligibility join applied inside every route.
Package signal implements ContentKit's signal plane: an append-only stream of host-defined interaction signals plus a durable per-(subject, content) current-state projection, both stored in ClickHouse.
 Package signal implements ContentKit's signal plane: an append-only stream of host-defined interaction signals plus a durable per-(subject, content) current-state projection, both stored in ClickHouse.
Package taxonomy is ContentKit's generic catalog: tenant-scoped nodes (tags, artists, creators, characters, series, seasons, voice actors, ...) with localized names and aliases, typed node relationships and typed assignments of host content (a work or one of its versions) to nodes.
 Package taxonomy is ContentKit's generic catalog: tenant-scoped nodes (tags, artists, creators, characters, series, seasons, voice actors, ...) with localized names and aliases, typed node relationships and typed assignments of host content (a work or one of its versions) to nodes.
Package worker maintains one tenant's keyword documents: it drains the dirty queue, runs a bounded cursor backfill and delivers every published document to the optional DocumentSink.
 Package worker maintains one tenant's keyword documents: it drains the dirty queue, runs a bounded cursor backfill and delivers every published document to the optional DocumentSink.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.