catalog

README

Nicos Catalog

Nicos Catalog is a typed, local-first software-catalog engine for repositories, services, products, documents, and the relationships between them. Hosts inject their own filesystem layout and providers; the engine supplies deterministic validation, indexing, BM25 full-text search, graph compilation, drift/reconcile gates, and a closed privacy-safe publication DTO.

The public core deliberately excludes personal telemetry, business valuation, private query text, runtime credentials, and host-specific portfolio policy. Those stay in host adapters.

Install

Requires Go 1.24 or newer.

go install github.com/nstranquist/nicos-catalog/cmd/nicos-catalog@v0.1.1
nicos-catalog version --expect v0.1.1

For a source checkout:

go test ./...
go install ./cmd/nicos-catalog

Five-minute smoke

The built-in demo contains synthetic entities only and writes to a temporary directory that is removed on exit.

nicos-catalog demo
nicos-catalog --json demo --query "developer platform"

To exercise an authored corpus from this repository:

nicos-catalog --root . --corpus demo/catalog validate
nicos-catalog --root . --corpus demo/catalog reindex
nicos-catalog --root . --corpus demo/catalog search --limit 3 "ownership graph"
nicos-catalog --root . --corpus demo/catalog graph
nicos-catalog --root . --corpus demo/catalog drift
nicos-catalog --root . --corpus demo/catalog --json project --visibility public --allow-hosts example.com

Host contract

layout, _ := (catalog.Layout{
    CorpusDir: "catalog",
    ConfigDir: ".catalog",
    CacheDir: ".catalog/cache",
    SidecarDataDir: ".catalog/sidecars",
}).Resolve(hostRoot)

engine, _ := catalog.New(layout, myProvider)
_, _ = engine.Reindex(context.Background())
results, _ := engine.Search("ownership graph", catalog.SearchOptions{Limit: 5})

A provider implements one small interface:

type Provider interface {
    Name() string
    Provide(context.Context, catalog.Layout) ([]catalog.Record, error)
}

FilesystemProvider handles YAML, JSON, and Markdown with YAML frontmatter. Its Strict mode rejects unknown fields, malformed frontmatter, trailing documents, and records without IDs; the CLI enables strict mode. It skips generated/cache directories plus _archive by default; hosts can add directory names through ExcludeDirs. StaticProvider supports embedded fixtures and API-backed hosts. Provider output is normalized and sorted; duplicate IDs fail closed across provider boundaries.

Privacy boundary

ProjectPublic produces a closed PublicEntity DTO. It cannot encode source paths, annotations, owner fields, telemetry, query text, valuation, or sidecar data. Hosts may further restrict visibility, kinds, tags, URL hosts, and summary length. Publication should consume this DTO rather than filtering the private index after serialization.

Why this exists

Large personal and organizational ecosystems become hard to reason about long before they become large enough to justify a heavyweight service catalog. Nicos Catalog keeps the data portable and reviewable while still providing the pieces that make a catalog operational: provider boundaries, deterministic derived state, search, typed relationships, and drift enforcement.

See docs/architecture.md for design boundaries and SECURITY.md for publication guidance.

License

Apache-2.0.

Documentation

Index

• Constants
• Variables
• type BuildInfo
  • func VersionInfo() BuildInfo
• type Document
• type DriftReport
• type Engine
  • func New(layout Layout, providers ...Provider) (*Engine, error)
  • func (e *Engine) Discover(ctx context.Context) ([]Record, error)
  • func (e *Engine) Drift(ctx context.Context) (DriftReport, error)
  • func (e *Engine) Layout() Layout
  • func (e *Engine) LoadIndex() (Index, error)
  • func (e *Engine) Reconcile(ctx context.Context, apply bool) (ReconcileReport, error)
  • func (e *Engine) Reindex(ctx context.Context) (ReindexReport, error)
  • func (e *Engine) Search(query string, options SearchOptions) ([]SearchResult, error)
  • func (e *Engine) Validate(ctx context.Context) (ValidationReport, error)
• type Entity
• type FilesystemProvider
  • func (p FilesystemProvider) Name() string
  • func (p FilesystemProvider) Provide(ctx context.Context, layout Layout) ([]Record, error)
• type Graph
  • func BuildGraph(index Index) Graph
  • func (g Graph) Mermaid() string
• type GraphEdge
• type GraphNode
• type Index
• type Layout
  • func DefaultLayout(root string) Layout
  • func (l Layout) Resolve(root string) (Layout, error)
  • func (l Layout) Validate() error
• type ProjectionPolicy
• type Provider
• type PublicConnection
• type PublicEntity
• type PublicProjection
  • func ProjectPublic(index Index, policy ProjectionPolicy) (PublicProjection, error)
• type ReconcileReport
• type Record
• type Ref
• type ReindexReport
• type SearchOptions
• type SearchResult
• type StaticProvider
  • func (p StaticProvider) Name() string
  • func (p StaticProvider) Provide(_ context.Context, _ Layout) ([]Record, error)
• type ValidationReport

Constants

const SchemaVersion = 1

Variables

var (
	Version = "v0.1.1"
	Commit  = "unknown"
)

Types

type BuildInfo

type BuildInfo struct {
	Version       string   `json:"version"`
	Commit        string   `json:"commit"`
	SchemaVersion int      `json:"schema_version"`
	Capabilities  []string `json:"capabilities"`
}

func VersionInfo

func VersionInfo() BuildInfo

type Document

type Document struct {
	EntityID      string         `json:"entity_id"`
	Length        int            `json:"length"`
	TermFrequency map[string]int `json:"term_frequency"`
}

type DriftReport

type DriftReport struct {
	OK             bool   `json:"ok"`
	Changed        bool   `json:"changed"`
	Reason         string `json:"reason,omitempty"`
	ExpectedDigest string `json:"expected_digest,omitempty"`
	ActualDigest   string `json:"actual_digest,omitempty"`
}

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine is a host-bound catalog compiler. It is safe to construct more than one engine with different Layouts in the same process.

func New

func New(layout Layout, providers ...Provider) (*Engine, error)

func (*Engine) Discover

func (e *Engine) Discover(ctx context.Context) ([]Record, error)

Discover collects, normalizes, validates, and deterministically orders all provider records. Duplicate IDs fail closed even across providers.

func (*Engine) Drift

func (e *Engine) Drift(ctx context.Context) (DriftReport, error)

func (*Engine) Layout

func (e *Engine) Layout() Layout

func (*Engine) LoadIndex

func (e *Engine) LoadIndex() (Index, error)

func (*Engine) Reconcile

func (e *Engine) Reconcile(ctx context.Context, apply bool) (ReconcileReport, error)

func (*Engine) Reindex

func (e *Engine) Reindex(ctx context.Context) (ReindexReport, error)

func (*Engine) Search

func (e *Engine) Search(query string, options SearchOptions) ([]SearchResult, error)

Search performs BM25 full-text retrieval over the deterministic local index.

func (*Engine) Validate

func (e *Engine) Validate(ctx context.Context) (ValidationReport, error)

type Entity

type Entity struct {
	ID          string            `json:"id" yaml:"id"`
	Name        string            `json:"name" yaml:"name"`
	Kind        string            `json:"kind" yaml:"kind"`
	Surface     string            `json:"surface,omitempty" yaml:"surface,omitempty"`
	Status      string            `json:"status,omitempty" yaml:"status,omitempty"`
	Description string            `json:"description,omitempty" yaml:"description,omitempty"`
	Entrypoint  string            `json:"entrypoint,omitempty" yaml:"entrypoint,omitempty"`
	Owner       string            `json:"owner,omitempty" yaml:"owner,omitempty"`
	Tags        []string          `json:"tags,omitempty" yaml:"tags,omitempty"`
	Refs        []Ref             `json:"refs,omitempty" yaml:"refs,omitempty"`
	PublicURL   string            `json:"public_url,omitempty" yaml:"public_url,omitempty"`
	Visibility  string            `json:"visibility,omitempty" yaml:"visibility,omitempty"`
	Annotations map[string]string `json:"annotations,omitempty" yaml:"annotations,omitempty"`
}

Entity is the portable catalog record. Host-only business, telemetry, and operator fields belong in host adapters rather than this public contract.

type FilesystemProvider

type FilesystemProvider struct {
	ProviderName string
	ExcludeDirs  []string
	// Strict rejects unknown fields, malformed frontmatter, trailing documents,
	// and entity files without IDs. Public and release paths should enable it.
	Strict bool
}

FilesystemProvider reads .md YAML frontmatter, .yaml/.yml, and .json entity records recursively from Layout.CorpusDir.

func (FilesystemProvider) Name

func (p FilesystemProvider) Name() string

func (FilesystemProvider) Provide

func (p FilesystemProvider) Provide(ctx context.Context, layout Layout) ([]Record, error)

type Graph

type Graph struct {
	Nodes []GraphNode `json:"nodes"`
	Edges []GraphEdge `json:"edges"`
}

func BuildGraph

func BuildGraph(index Index) Graph

func (Graph) Mermaid

func (g Graph) Mermaid() string

type GraphEdge

type GraphEdge struct {
	Source string `json:"source"`
	Kind   string `json:"kind"`
	Target string `json:"target"`
}

type GraphNode

type GraphNode struct {
	ID     string `json:"id"`
	Name   string `json:"name"`
	Kind   string `json:"kind"`
	Status string `json:"status,omitempty"`
}

type Index

type Index struct {
	SchemaVersion         int        `json:"schema_version"`
	SourceDigest          string     `json:"source_digest"`
	Entities              []Entity   `json:"entities"`
	Documents             []Document `json:"documents"`
	AverageDocumentLength float64    `json:"average_document_length"`
}

Index is the deterministic, portable catalog cache. It intentionally omits wall-clock timestamps so identical inputs produce byte-identical output.

type Layout

type Layout struct {
	CorpusDir      string `json:"corpus_dir" yaml:"corpus_dir"`
	ConfigDir      string `json:"config_dir" yaml:"config_dir"`
	CacheDir       string `json:"cache_dir" yaml:"cache_dir"`
	SidecarDataDir string `json:"sidecar_data_dir" yaml:"sidecar_data_dir"`
}

Layout injects every host-owned filesystem boundary used by the engine. The engine never assumes a repository name, home directory, or corpus shape.

func DefaultLayout

func DefaultLayout(root string) Layout

DefaultLayout returns a portable layout rooted at root.

func (Layout) Resolve

func (l Layout) Resolve(root string) (Layout, error)

Resolve converts relative paths to absolute paths under root and validates that all four host boundaries are explicit and distinct where required.

func (Layout) Validate

func (l Layout) Validate() error

Validate rejects ambiguous layouts and unsafe cache placement.

type ProjectionPolicy

type ProjectionPolicy struct {
	RequireVisibility string
	IncludeKinds      []string
	IncludeTags       []string
	AllowHosts        []string
	MaxSummaryBytes   int
}

type Provider

type Provider interface {
	Name() string
	Provide(context.Context, Layout) ([]Record, error)
}

Provider is the host extension boundary. Providers discover authored facts; the engine owns normalization, validation, indexing, graph, and drift.

type PublicConnection

type PublicConnection struct {
	Kind   string `json:"kind"`
	Target string `json:"target"`
}

type PublicEntity

type PublicEntity struct {
	ID          string             `json:"id"`
	Name        string             `json:"name"`
	Kind        string             `json:"kind"`
	Status      string             `json:"status,omitempty"`
	Summary     string             `json:"summary,omitempty"`
	Tags        []string           `json:"tags,omitempty"`
	URL         string             `json:"url,omitempty"`
	Connections []PublicConnection `json:"connections,omitempty"`
}

PublicEntity is a closed publication DTO. It cannot represent source paths, host annotations, owner telemetry, sidecars, valuation, or query text.

type PublicProjection

type PublicProjection struct {
	SchemaVersion int            `json:"schema_version"`
	Items         []PublicEntity `json:"items"`
}

func ProjectPublic

func ProjectPublic(index Index, policy ProjectionPolicy) (PublicProjection, error)

type ReconcileReport

type ReconcileReport struct {
	Drift   DriftReport    `json:"drift"`
	Applied bool           `json:"applied"`
	Reindex *ReindexReport `json:"reindex,omitempty"`
}

type Record

type Record struct {
	Entity   Entity `json:"entity"`
	Provider string `json:"provider"`
	Source   string `json:"source"`
	Digest   string `json:"digest"`
}

Record binds a portable entity to source provenance. Source paths never appear in public projections.

type Ref

type Ref struct {
	Kind   string `json:"kind" yaml:"kind"`
	Target string `json:"target" yaml:"target"`
}

type ReindexReport

type ReindexReport struct {
	OK            bool   `json:"ok"`
	EntityCount   int    `json:"entity_count"`
	DocumentCount int    `json:"document_count"`
	SourceDigest  string `json:"source_digest"`
	IndexPath     string `json:"index_path"`
}

type SearchOptions

type SearchOptions struct {
	Limit int
	Kinds []string
}

type SearchResult

type SearchResult struct {
	Entity       Entity   `json:"entity"`
	Score        float64  `json:"score"`
	MatchedTerms []string `json:"matched_terms"`
}

type StaticProvider

type StaticProvider struct {
	ProviderName string
	Entities     []Entity
}

StaticProvider is useful for tests, embedded demos, and API-backed hosts.

func (StaticProvider) Name

func (p StaticProvider) Name() string

func (StaticProvider) Provide

func (p StaticProvider) Provide(_ context.Context, _ Layout) ([]Record, error)

type ValidationReport

type ValidationReport struct {
	OK            bool     `json:"ok"`
	EntityCount   int      `json:"entity_count"`
	ProviderCount int      `json:"provider_count"`
	Warnings      []string `json:"warnings,omitempty"`
}