vfsindex

Documentation

Overview

Package vfsindex bridges a vfs.MountSession into brain knowledge objects.

Canonical architecture: docs/knowledge.md. This package is only the artifact ingest path (IndexPath / policy / schedulers).

VFS and brain work alone or together. This package is the optional composition layer: it imports both, while vfs and brain never import each other.

Model

Each text-like virtual file becomes a brain Document (parent) with Chunk parts:

Document.properties.vfs_path  = virtual path
Document.properties.content_hash, size, mtime, media_type
Chunk.properties.start_line, end_line, byte_start, byte_end
Chunk.properties.block_id, heading_path  = when chunked from Structured IR
Chunk.Content                 = chunk body (heading blocks for Markdown; line windows otherwise)

Live VFS bytes remain source of truth. The index is derived and may lag until re-index (IndexPath / IndexScheduler.Notify). Parent Documents keep metadata and content_hash — not a second agent-editable full-file body.

Index policy (MountSpec.IndexPolicy)

none       — no auto jobs; index_file errors
selective  — only index_file / host IndexPath (optional track set after index_file)
prefix     — IndexPrefix at bridge start + AfterPersist under the mount
watch      — same auto triggers as prefix (host-facing name)

Empty policy normalizes to selective (NormalizePolicy / AutoIndex helpers).

Single pipeline

All file→brain content updates go through IndexPath (or UnindexPath). Triggers fan in via IndexScheduler.Notify (AfterPersist), index_file, IndexPrefix, or host IndexPath API. content_hash skip returns PathSkipped without re-chunking.

Session-visible body

IndexPath uses MountSession.ReadText (markdown) and MountSession.Open (other text). Writes are write-through, so index_file / IndexPath see the last persist. AfterPersist still drives background reindex when policy allows.

Schedulers

Hosts wire Notify after writes via vfs.MountSession.SetAfterPersist, gated by policy:

br, err := vfsindex.Start(ms, eng, scope)
defer br.Close()
// Or wire by hand:
idx, err := vfsindex.NewMountIndexer(ms, eng, scope)
sched := vfsindex.NewAsyncScheduler(idx) // or NewSyncScheduler for inline
prev := ms.GetAfterPersist()
ms.SetAfterPersist(func(ctx context.Context, path string) error {
    if prev != nil {
        _ = prev(ctx, path)
    }
    // harness: only Notify when AutoIndex(spec) or selective track set
    return sched.Notify(ctx, path, vfsindex.ReasonSync)
})
defer sched.Close()
_ = idx.IndexPrefix(ctx, "/work", vfsindex.IndexOpts{})

SyncScheduler runs IndexPath inline (tests / hosts that want blocking reindex). AsyncScheduler enqueues with coalesce (last reason wins), bounded pending set, and a background worker; Notify never blocks on re-chunk.

The tacklr harness creates MountIndexer + AsyncScheduler and registers index_file / unindex when Brain + VFS + search namespace are set. It skips mounts with IndexPolicy=none (harness sets this on brain Engram mounts) and never remirrors those paths. Scratch /memory is attached only when a scratch profile exists and no brain Provider mount is present.

Kinds

Hosts that use a non-empty kind catalog should register MountIndexKinds() (or equivalent fields) before indexing. Open-catalog engines accept any props.

Content search over mounts is brain search/find_exact on Chunks with vfs_path. Live grep is run_command → rg on the FUSE tree. This package does not implement grep.

Index

• Constants
• Variables
• func AutoIndex(policy string) bool
• func MountIndexKinds() []brain.KindSpec
• func NormalizePolicy(raw string) string
• type AsyncScheduler
  • func NewAsyncScheduler(idx *MountIndexer) *AsyncScheduler
  • func (s *AsyncScheduler) Close() error
  • func (s *AsyncScheduler) Notify(ctx context.Context, virtualPath string, reason IndexReason) error
  • func (s *AsyncScheduler) SetObserver(observer func(SchedulerEvent))
• type Bridge
  • func Start(ms *vfs.MountSession, eng *brain.Engine, scope brain.Scope) (*Bridge, error)
  • func (b *Bridge) Close() error
  • func (b *Bridge) PolicyAt(virtualPath string) string
  • func (b *Bridge) SetObserver(observer func(SchedulerEvent))
  • func (b *Bridge) ShouldIndex(virtualPath string) bool
  • func (b *Bridge) Track(virtualPath string)
  • func (b *Bridge) Untrack(virtualPath string)
• type IndexOpts
• type IndexReason
• type IndexScheduler
• type MountIndexer
  • func NewMountIndexer(ms *vfs.MountSession, eng *brain.Engine, scope brain.Scope) (*MountIndexer, error)
  • func (x *MountIndexer) DocumentID(virtualPath string) uuid.UUID
  • func (x *MountIndexer) IndexFileResult(ctx context.Context, virtualPath string, st vfs.FileInfo) (PathIndexResult, error)
  • func (x *MountIndexer) IndexPath(ctx context.Context, virtualPath string) error
  • func (x *MountIndexer) IndexPathResult(ctx context.Context, virtualPath string) (PathIndexResult, error)
  • func (x *MountIndexer) IndexPrefix(ctx context.Context, prefix string, opts IndexOpts) (Stats, error)
  • func (x *MountIndexer) UnindexPath(ctx context.Context, virtualPath string) (bool, error)
• type PathIndexResult
• type SchedulerEvent
• type Stats
• type SyncScheduler
  • func NewSyncScheduler(idx *MountIndexer) *SyncScheduler
  • func (s *SyncScheduler) Notify(ctx context.Context, virtualPath string, reason IndexReason) error

Constants

const (
	DefaultLinesPerChunk = 40
	DefaultMaxIndexBytes = 64 << 20 // 64 MiB per file
)

Defaults for streaming index.

const (
	PropVFSPath     = "vfs_path"
	PropSize        = "size"
	PropMTime       = "mtime"
	PropContentHash = "content_hash"
	PropMediaType   = "media_type"
	PropStartLine   = "start_line"
	PropEndLine     = "end_line"
	PropByteStart   = "byte_start"
	PropByteEnd     = "byte_end"
	PropBlockID     = "block_id"
	PropHeadingPath = "heading_path"
)

Property keys written on mount-indexed objects.

const (
	DefaultDocumentKind = "Document"
	DefaultChunkKind    = "Chunk"
)

Default kind names (host may override on MountIndexer).

const (
	PolicyNone      = "none"
	PolicySelective = "selective"
	PolicyPrefix    = "prefix"
	PolicyWatch     = "watch"
)

Index policy values stored on vfs.MountSpec.IndexPolicy. Empty policy normalizes to PolicySelective.

const (
	DefaultAsyncQueueCap = 64
	DefaultAsyncTimeout  = 2 * time.Minute
)

Defaults for AsyncScheduler.

const MemoryPoint = "/workspace/memory"

MemoryPoint is the scratch knowledge alias when no brain Provider exists.

Variables

var (
	// ErrSchedulerClosed means Notify was called after shutdown.
	ErrSchedulerClosed = errors.New("vfsindex: scheduler closed")
	// ErrQueueFull means a distinct path could not be queued.
	ErrQueueFull = errors.New("vfsindex: scheduler queue full")
)

Functions

func AutoIndex

func AutoIndex(policy string) bool

AutoIndex reports whether AfterPersist / IndexPrefix should run for policy. prefix and watch both auto-index; selective and none do not (except track set).

func MountIndexKinds

func MountIndexKinds() []brain.KindSpec

MountIndexKinds returns KindSpecs for catalog mode when indexing mounts. Fields are optional so pure knowledge Documents/Chunks without vfs_path remain valid.

func NormalizePolicy

func NormalizePolicy(raw string) string

NormalizePolicy returns a canonical policy string. Unknown or empty values become PolicySelective.

Types

type AsyncScheduler

type AsyncScheduler struct {
	Indexer  *MountIndexer
	QueueCap int           // max distinct pending paths; default DefaultAsyncQueueCap
	Timeout  time.Duration // per-path IndexPath timeout; default DefaultAsyncTimeout
	// contains filtered or unexported fields
}

AsyncScheduler re-indexes paths on background worker(s). Notify is non-blocking: it enqueues the path (duplicates coalesce) and returns immediately. Under pressure (queue full and path not already pending), the notify is dropped. Close cancels in-flight work and stops the worker.

func NewAsyncScheduler

func NewAsyncScheduler(idx *MountIndexer) *AsyncScheduler

NewAsyncScheduler starts a single background worker. Call Close on teardown.

func (*AsyncScheduler) Close

func (s *AsyncScheduler) Close() error

Close stops the worker and cancels any in-flight IndexPath. Safe to call once or more; subsequent calls are no-ops.

func (*AsyncScheduler) Notify

func (s *AsyncScheduler) Notify(ctx context.Context, virtualPath string, reason IndexReason) error

Notify implements IndexScheduler. Never blocks on IndexPath. The caller's ctx is not used for enqueue: AfterPersist and similar short-lived contexts must not cancel pending work. IndexPath runs under the scheduler's own cancel + Timeout context.

func (*AsyncScheduler) SetObserver

func (s *AsyncScheduler) SetObserver(observer func(SchedulerEvent))

SetObserver installs a non-blocking host failure observer.

type Bridge

type Bridge struct {
	Indexer *MountIndexer
	// contains filtered or unexported fields
}

Bridge owns mount→brain index lifecycle: indexer, async reindex, selective track set, prefix/watch warm-up. Harness holds this; it is not the agent loop.

func Start

func Start(ms *vfs.MountSession, eng *brain.Engine, scope brain.Scope) (*Bridge, error)

Start builds an indexer, wires AfterPersist (composing any existing hook), and warms prefix/watch members under /workspace.

func (*Bridge) Close

func (b *Bridge) Close() error

Close stops warm-up and the async scheduler.

func (*Bridge) PolicyAt

func (b *Bridge) PolicyAt(virtualPath string) string

PolicyAt is the normalized IndexPolicy for a virtual path (selective if unknown).

func (*Bridge) SetObserver

func (b *Bridge) SetObserver(observer func(SchedulerEvent))

SetObserver replaces the asynchronous indexing failure observer.

func (*Bridge) ShouldIndex

func (b *Bridge) ShouldIndex(virtualPath string) bool

ShouldIndex reports whether AfterPersist should enqueue path.

func (*Bridge) Track

func (b *Bridge) Track(virtualPath string)

Track records a selective path so later persists reindex it.

func (*Bridge) Untrack

func (b *Bridge) Untrack(virtualPath string)

Untrack drops a path from the selective set.

type IndexOpts

type IndexOpts struct {
	// MaxFiles stops after this many files opened (0 = unlimited).
	MaxFiles int
}

IndexOpts configures a tree walk.

type IndexReason

type IndexReason int

IndexReason explains why a path was scheduled for re-index. Notify accepts a reason for host/API clarity; IndexPath does not branch on it.

const (
	// ReasonSync is a successful VFS WriteFile or Sync (AfterPersist bridge).
	ReasonSync IndexReason = iota
	// ReasonExplicit is a host or tool request (index_file / IndexPath).
	ReasonExplicit
)

type IndexScheduler

type IndexScheduler interface {
	Notify(ctx context.Context, virtualPath string, reason IndexReason) error
}

IndexScheduler receives path invalidations. SyncScheduler runs inline; AsyncScheduler enqueues work with the same interface.

type MountIndexer

type MountIndexer struct {
	VFS   *vfs.MountSession
	Brain *brain.Engine
	Scope brain.Scope

	DocumentKind  string // default Document
	ChunkKind     string // default Chunk
	LinesPerChunk int    // default DefaultLinesPerChunk
	MaxIndexBytes int64  // default DefaultMaxIndexBytes
	// contains filtered or unexported fields
}

MountIndexer streams text-like mount files into brain Document + Chunk objects.

func NewMountIndexer

func NewMountIndexer(ms *vfs.MountSession, eng *brain.Engine, scope brain.Scope) (*MountIndexer, error)

NewMountIndexer validates required fields. Scope.Namespace must be set (brain Put requires namespace attrs).

func (*MountIndexer) DocumentID

func (x *MountIndexer) DocumentID(virtualPath string) uuid.UUID

DocumentID returns the stable brain id for a virtual path under this scope.

func (*MountIndexer) IndexFileResult

func (x *MountIndexer) IndexFileResult(ctx context.Context, virtualPath string, st vfs.FileInfo) (PathIndexResult, error)

IndexFileResult indexes a path already known to be an existing file (caller Stat'd). Skips a second Stat round-trip — useful for remote mounts and batch tools that pre-validate paths before any write work.

func (*MountIndexer) IndexPath

func (x *MountIndexer) IndexPath(ctx context.Context, virtualPath string) error

IndexPath indexes one virtual file (or removes the brain mirror if missing).

func (*MountIndexer) IndexPathResult

func (x *MountIndexer) IndexPathResult(ctx context.Context, virtualPath string) (PathIndexResult, error)

IndexPathResult indexes one path and returns a compact outcome for tools/hosts.

func (*MountIndexer) IndexPrefix

func (x *MountIndexer) IndexPrefix(ctx context.Context, prefix string, opts IndexOpts) (Stats, error)

IndexPrefix walks a directory (or single file) and indexes text-like files.

func (*MountIndexer) UnindexPath

func (x *MountIndexer) UnindexPath(ctx context.Context, virtualPath string) (bool, error)

UnindexPath soft-deletes the brain Document/Chunks for virtualPath without touching the VFS file. Returns true when a mirror was present and removed, false when nothing was indexed (idempotent noop).

type PathIndexResult

type PathIndexResult string

PathIndexResult is a compact outcome of indexing one path (indexed|skipped|removed|directory). Used by agent tools and hosts.

const (
	// PathIndexed means Document/Chunks were written or re-chunked.
	PathIndexed PathIndexResult = "indexed"
	// PathSkipped means hash match, binary, non-text, or empty skip.
	PathSkipped PathIndexResult = "skipped"
	// PathRemoved means the path was missing and any brain mirror was soft-deleted.
	PathRemoved PathIndexResult = "removed"
	// PathDirectory means the path is a directory (IndexPath is a no-op for dirs).
	PathDirectory PathIndexResult = "directory"
)

type SchedulerEvent

type SchedulerEvent struct {
	Path   string
	Reason IndexReason
	Err    error
}

SchedulerEvent reports an asynchronous indexing outcome to the host.

type Stats

type Stats struct {
	Indexed int // files written or re-chunked
	Skipped int // hash match, binary, non-text, or empty skip
	Removed int // missing paths soft-deleted from brain
}

Stats summarizes IndexPrefix work.

type SyncScheduler

type SyncScheduler struct {
	Indexer *MountIndexer
}

SyncScheduler re-indexes immediately on Notify (v1).

func NewSyncScheduler

func NewSyncScheduler(idx *MountIndexer) *SyncScheduler

NewSyncScheduler returns an IndexScheduler that calls IndexPath inline.

func (*SyncScheduler) Notify

func (s *SyncScheduler) Notify(ctx context.Context, virtualPath string, reason IndexReason) error

Notify implements IndexScheduler.