focusedindex

package
v0.0.0-...-3d6ecf2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 25, 2026 License: Apache-2.0 Imports: 32 Imported by: 0

Documentation

Overview

Package focusedindex builds and validates zoekt shard sets whose Git input is restricted to one configured analysis unit.

Index

Constants

View Source 
const (
	RequestSchema  = "phebs-focused-index-request-v1"
	ResultSchema   = "phebs-focused-index-result-v1"
	ManifestSchema = "phebs-focused-shard-set-v1"
	BuilderPolicy  = "phebs-focused-builder-v2"

	MemberSuffix = ".phebs-member.json"
)
View Source 
const (
	SearchGenerationReceiptSchema = "phebs-search-generation-receipt-v1"
	SearchGenerationRootSchema    = "phebs-search-generation-root-v1"
	SearchGenerationMarkerSchema  = "phebs-search-generation-marker-v1"
	SearchBlobReaderGoGit         = "go_git"
	SearchBlobReaderLegacy        = "legacy_unrecorded"

	// The retained 1.61-million-owner diagnostic measured 28.8 GB for one
	// generation and 57.5 GB for current plus replacement. The frozen T40.1
	// target has 2.0 million owners. These prospective limits keep roughly
	// forty percent headroom at that scale without inheriting T35's unrelated
	// 50-GiB retention placeholder.
	MaxSearchGenerationLogicalBytes   int64 = 48 << 30
	MaxSearchGenerationAllocatedBytes int64 = 48 << 30
	MaxRetainedSearchLogicalBytes     int64 = 96 << 30
	MaxRetainedSearchAllocatedBytes   int64 = 96 << 30
	MaxSearchShardLogicalBytes        int64 = 512 << 20
	MaxSearchGenerationShards               = 256
	MaxSearchGenerationFiles                = repositoryindex.MaxSourceMembers + MaxSearchGenerationShards + 4
	MaxSearchLifecycleRepositories          = 4_096
	MaxSearchRepositoryGenerations          = 64
	SearchGenerationMaxAge                  = 14 * 24 * time.Hour
)
View Source 
const WholeManifestSchema = "phebs-whole-shard-set-v1"

Variables

View Source 
var (
	ErrMissingScope = errors.New("selected analysis-unit path is missing")
	ErrSpecialEntry = errors.New("selected analysis-unit path contains a non-regular Git entry")
	ErrShardSet     = errors.New("invalid focused shard set")
)
View Source 
var ErrSearchPublicationRevisionMismatch = errors.New(
	"search publication marker matches no committed generation",
)

Functions

func AcquireBackupLock 

func AcquireBackupLock(ctx context.Context, indexDir string) (func(), error)

AcquireBackupLock waits until every focused-index publication or artifact- reconciliation mutation admitted through this lock is complete, then excludes new ones while the caller exports the database and copies focused publication bytes. Candidate, resolver, and caller publication lifecycles use their own exact-state reconciliation rather than this lock.

func AcquireExclusiveMutationLock 

func AcquireExclusiveMutationLock(ctx context.Context, indexDir string) (func(), error)

AcquireExclusiveMutationLock excludes every publication/lifecycle mutation admitted through AcquireMutationLock. Callers acquire this before any per-repository lock, matching artifact reconciliation's lock order. Short cross-authority pointer transitions use it so a lifecycle rename cannot land between the final control check and pointer swap.

func AcquireMutationLock 

func AcquireMutationLock(ctx context.Context, indexDir string) (func(), error)

AcquireMutationLock admits one index/reconciliation mutation alongside other repository mutations while excluding the database+focused-index snapshot taken by online backup. Kernel lock release makes crashes safe.

func ArtifactBase 

func ArtifactBase(repository string) string

func CreateArchive 

func CreateArchive(indexDir, destination string) error

CreateArchive writes a deterministic inventory of every complete focused or repository-search v2 publication. Invalid or interrupted derived state is omitted so it cannot prevent backup of the precious database export.

func DigestRegularFile 

func DigestRegularFile(path string) (string, int64, error)

func DigestRegularFileContext 

func DigestRegularFileContext(
	ctx context.Context,
	path string,
) (string, int64, error)

DigestRegularFileContext hashes one stable regular-file identity and checks cancellation between bounded reads.

func FindBinary 

func FindBinary() (string, error)

func FinishPublication 

func FinishPublication(indexDir, repository string) error

func GenerationDigest 

func GenerationDigest(scope analysisunit.Scope, revisions []store.IndexedRevision) (string, error)

func IsManagedShardName 

func IsManagedShardName(name string) bool

IsManagedShardName reports whether name belongs to the cryptographic repository namespace used by focused or whole-repository publications. Decoded shard metadata must never override this ownership boundary.

func IsPublishing 

func IsPublishing(indexDir, repository string) bool

func IsRetentionArtifactName 

func IsRetentionArtifactName(name string) bool

IsRetentionArtifactName reports whether name belongs to the stable focused publication namespace. Whole-repository shards, publication markers, and build workspaces are intentionally outside this component.

func ManifestDigest 

func ManifestDigest(manifest Manifest) (string, error)

func ManifestName 

func ManifestName(repository string) string

func NewBuildWorkspace 

func NewBuildWorkspace(indexDir string) (workspace, outputDir string, err error)

func PublicationMarkerOwnedByCurrentProcess 

func PublicationMarkerOwnedByCurrentProcess(indexDir, repository string) bool

PublicationMarkerOwnedByCurrentProcess reports whether the marker belongs to an active publication created by this phebs process.

func PublishFocused 

func PublishFocused(
	ctx context.Context,
	indexDir, stageDir, repository string,
	state *analysisunit.State,
	revisions []store.IndexedRevision,
) error

PublishFocused validates the complete staged set, marks the repository unavailable, replaces every prior shard, and renames the manifest last. The caller must remove the marker with FinishPublication only after committing the matching database state.

func PublishWhole 

func PublishWhole(
	ctx context.Context,
	indexDir, stageDir, repository string,
	revisions []store.IndexedRevision,
) error

PublishWhole gives the whole-repository builder the same state-commit boundary and publishes an exact member receipt after every shard is durable.

func PublishWholeGeneration 

func PublishWholeGeneration(
	ctx context.Context,
	indexDir, shardStageDir, sourceStageDir, repository string,
	revisions []store.IndexedRevision,
	source repositoryindex.SourceManifest,
) error

PublishWholeGeneration publishes the T34.1 source/search authority beside the unchanged whole-shard v1 receipt. The stable publication marker hides both roots until the caller commits the matching repository row.

func PublishingName 

func PublishingName(repository string) string

func ReactivatePriorSearchGeneration 

func ReactivatePriorSearchGeneration(
	ctx context.Context,
	indexDir, repository string,
	revisions []store.IndexedRevision,
) (bool, error)

ReactivatePriorSearchGeneration selects the one retained prior generation when its immutable revision identity exactly matches the requested source. Whole-search shard bytes include builder timestamps, so rebuilding the same Git authority would create a different physical generation and defeat exact A-to-B-to-A recovery even though the retained bytes are already complete.

func ReadRepositorySearchGeneration 

func ReadRepositorySearchGeneration(
	indexDir, repository string,
	revisions []store.IndexedRevision,
) (repositoryindex.SearchManifest, repositoryindex.SourceManifest, error)

ReadRepositorySearchGeneration opens only the closed v2 control roots. It deliberately does not hash shard or source-member content; hot readers pair it with a whole-cache lease whose fill already performed the complete generation validation and retained identity fences for every named file.

func RecoverSearchPublication 

func RecoverSearchPublication(
	ctx context.Context, indexDir, repository string, revisions []store.IndexedRevision,
) (bool, error)

RecoverSearchPublication selects only the generation matching the durable repository row. It is the startup repair for every publication crash point.

func RemoveRepository 

func RemoveRepository(ctx context.Context, indexDir, repository string) error

func RepositoryPrefix 

func RepositoryPrefix(repository string) string

func RestoreArchive 

func RestoreArchive(archivePath, indexDir string) error

RestoreArchive performs a complete structural pass before creating the target or staging output. It then verifies all extracted bytes in a private process-owned directory and installs shards and sidecars before manifests.

func RetireSearchGenerationRoot 

func RetireSearchGenerationRoot(indexDir, repository string) error

RetireSearchGenerationRoot releases whole-search lifecycle authority after the durable repository row has committed a focused posture. Immutable bytes remain lease-protected and are reclaimed by the bounded lifecycle owner.

func RollbackSearchPublication 

func RollbackSearchPublication(ctx context.Context, indexDir, repository string) error

func SearchGenerationReaderCounts 

func SearchGenerationReaderCounts(source repositoryindex.SourceManifest) (batch, fallback int, err error)

func SearchGenerationReservation 

func SearchGenerationReservation(source repositoryindex.SourceManifest) (int64, error)

func SearchGenerationRootDirectory 

func SearchGenerationRootDirectory(indexDir string) string

func SearchGenerationRootName 

func SearchGenerationRootName(repository string) string

func ShardPrefix 

func ShardPrefix(repository, generationDigest string) string

func ValidateCommittedRepositorySearchGeneration 

func ValidateCommittedRepositorySearchGeneration(
	ctx context.Context,
	indexDir, repository string,
	revisions []store.IndexedRevision,
) (repositoryindex.SearchManifest, error)

ValidateCommittedRepositorySearchGeneration permits only the prior-process publication marker used by startup recovery. Every v2 source/search/shard byte must otherwise validate before the marker can be retired.

func ValidateRepositorySearchGeneration 

func ValidateRepositorySearchGeneration(
	ctx context.Context,
	indexDir, repository string,
	revisions []store.IndexedRevision,
) (repositoryindex.SearchManifest, error)

ValidateRepositorySearchGeneration proves the T34.1 source/search root against the already strict whole-shard publication it names.

func ValidateRevisions 

func ValidateRevisions(revisions []store.IndexedRevision) error

func VerifyArchive 

func VerifyArchive(archivePath string) error

func WholeManifestDigest 

func WholeManifestDigest(manifest WholeManifest) (string, error)

func WholeManifestName 

func WholeManifestName(repository string) string

func WholeShardName 

func WholeShardName(
	repository string,
	formatVersion, ordinal int,
) string

func WholeShardPrefix 

func WholeShardPrefix(repository string) string

func WriteControlFile 

func WriteControlFile(path string, value any) error

Types

type ArchiveReport 

type ArchiveReport struct {
	Publications        int
	OmittedPublications int
	OmittedArtifacts    int
	StaleMarkers        int
}

ArchiveReport records derived search state that was safe to preserve and residue that was deliberately omitted. Omission never weakens restore: archives still contain only complete self-contained publications.

func CreateArchiveWithReport 

func CreateArchiveWithReport(
	indexDir, destination string,
) (ArchiveReport, error)

CreateArchiveWithReport is CreateArchive with omission accounting for the operator-facing backup diagnostic.

func CreateArchiveWithReportContext 

func CreateArchiveWithReportContext(
	ctx context.Context,
	indexDir, destination string,
) (ArchiveReport, error)

CreateArchiveWithReportContext is the cancellable backup boundary used by recovery while it holds the shared index backup lock.

func VerifyArchiveWithReport 

func VerifyArchiveWithReport(archivePath string) (ArchiveReport, error)

VerifyArchiveWithReport is VerifyArchive with independently recovered publication accounting. Explicit verification deliberately retains the complete restore round trip; backup creation uses a cheaper construction proof instead.

type BuildOptions 

type BuildOptions struct {
	// ShardMax is test-only tuning when non-zero. Production callers and the
	// child use the fixed default.
	ShardMax int
}

type LifecycleCleanupReport 

type LifecycleCleanupReport struct {
	Workspaces       int
	TemporaryMarkers int
}

LifecycleCleanupReport describes private staging residue reclaimed from a prior phebs process. Names carry the current process token from creation, so runtime reconciliation cannot mistake an active same-process workspace for abandoned state.

func CleanupAbandonedLifecycle 

func CleanupAbandonedLifecycle(indexDir string) (LifecycleCleanupReport, error)

CleanupAbandonedLifecycle removes only staging paths whose names prove that they were created by a different phebs process. Callers hold the index mutation lock, but builds can run before taking that lock; the name token is therefore the primary active-work guard.

type Manifest 

type Manifest struct {
	Schema           string                  `json:"schema"`
	Repository       string                  `json:"repository"`
	Scope            analysisunit.Scope      `json:"scope"`
	UnitDigest       string                  `json:"unit_digest"`
	GenerationDigest string                  `json:"generation_digest"`
	BuilderPolicy    string                  `json:"builder_policy"`
	Revisions        []store.IndexedRevision `json:"revisions"`
	Members          []ShardMember           `json:"members"`
	Digest           string                  `json:"digest"`
}

Manifest is the search visibility authority for one configured repository. Digest is a publication identity: builder timestamps may change shard bytes across semantically equivalent rebuilds.

func ValidateCommittedPublication 

func ValidateCommittedPublication(
	indexDir, repository string,
	state *analysisunit.State,
	revisions []store.IndexedRevision,
) (Manifest, error)

ValidateCommittedPublication proves a focused publication against committed database state while deliberately ignoring its publication marker. Startup reconciliation uses this only to decide whether a prior-process marker can be safely removed; search callers must continue to use ValidatePublished.

func ValidatePublished 

func ValidatePublished(
	indexDir, repository string,
	state *analysisunit.State,
	revisions []store.IndexedRevision,
) (Manifest, error)

func ValidatePublishedContext 

func ValidatePublishedContext(
	ctx context.Context,
	indexDir, repository string,
	state *analysisunit.State,
	revisions []store.IndexedRevision,
) (Manifest, error)

ValidatePublishedContext is ValidatePublished with cancellation propagated through repository-local shard hashing. Query admission uses this form so one cold or invalid publication cannot outlive the query-wide work budget.

func ValidateSelfContained 

func ValidateSelfContained(indexDir, repository string) (Manifest, error)

ValidateSelfContained is used by backup/restore before the database is available. The canonical scope embedded in the manifest re-derives both the unit and generation identities; startup reconciliation subsequently binds the same bytes to committed repository state.

func ValidateStage 

func ValidateStage(
	indexDir, repository string,
	state *analysisunit.State,
	revisions []store.IndexedRevision,
) (Manifest, error)

type Request 

type Request struct {
	Schema    string                  `json:"schema"`
	RepoDir   string                  `json:"repo_dir"`
	OutputDir string                  `json:"output_dir"`
	Scope     analysisunit.Scope      `json:"scope"`
	Revisions []store.IndexedRevision `json:"revisions"`
}

Request is the strict parent-to-child build contract. OutputDir must be an unpublished staging directory.

func ReadRequest 

func ReadRequest(path string) (Request, error)

type Result 

type Result struct {
	Schema              string `json:"schema"`
	Repository          string `json:"repository"`
	UnitDigest          string `json:"unit_digest"`
	GenerationDigest    string `json:"generation_digest"`
	ManifestDigest      string `json:"manifest_digest"`
	OpenedBlobCount     int    `json:"opened_blob_count"`
	OpenedBlobBytes     int64  `json:"opened_blob_bytes"`
	OutOfUnitBlobReads  int    `json:"out_of_unit_blob_reads"`
	AdmittedDocuments   int    `json:"admitted_documents"`
	AdmittedSourceBytes int64  `json:"admitted_source_bytes"`
	ShardCount          int    `json:"shard_count"`
	ShardBytes          int64  `json:"shard_bytes"`
	BuildWallNanos      int64  `json:"build_wall_nanos"`
}

Result is written only after the complete staged shard set and its manifest are durable.

func Build 

func Build(ctx context.Context, request Request, options BuildOptions) (Result, error)

func ReadResult 

func ReadResult(path string) (Result, error)

type SearchGenerationLease 

type SearchGenerationLease struct {
	// contains filtered or unexported fields
}

func (*SearchGenerationLease) Release 

func (lease *SearchGenerationLease) Release()

type SearchGenerationLifecycleResult 

type SearchGenerationLifecycleResult struct {
	Cursor         string
	Scanned        int
	Deleted        int
	More           bool
	LogicalBytes   int64
	AllocatedBytes int64
	AllocatedState string
}

func SweepSearchGenerationLifecycle 

func SweepSearchGenerationLifecycle(
	ctx context.Context,
	indexDir string,
	now time.Time,
	cursor string,
	pins SearchGenerationPinChecker,
	deleteLimit int,
) (SearchGenerationLifecycleResult, error)

SweepSearchGenerationLifecycle examines one bounded repository namespace. Current, rollback, a transition marker, and active reader pins are exact roots. One stale generation is renamed before at most deleteLimit regular files are removed; a later fair turn resumes the collecting directory.

type SearchGenerationPinChecker 

type SearchGenerationPinChecker interface {
	Pinned(repository, generation string) bool
}

type SearchGenerationPins 

type SearchGenerationPins struct {
	// contains filtered or unexported fields
}

SearchGenerationPins is the process-local reader fence shared by search and lifecycle. Publication roots remain the durable authority; pins only delay retirement after a reader has already bound an exact immutable generation.

func (*SearchGenerationPins) Acquire 

func (pins *SearchGenerationPins) Acquire(repository, generation string) (*SearchGenerationLease, error)

func (*SearchGenerationPins) Pinned 

func (pins *SearchGenerationPins) Pinned(repository, generation string) bool

type SearchGenerationReceipt 

type SearchGenerationReceipt struct {
	Schema            string                  `json:"schema"`
	Repository        string                  `json:"repository"`
	SearchDigest      string                  `json:"search_digest"`
	SourceDigest      string                  `json:"source_digest"`
	Revisions         []store.IndexedRevision `json:"revisions"`
	BlobReaderMode    string                  `json:"blob_reader_mode"`
	FilesOffered      int                     `json:"files_offered"`
	BatchReadCount    int                     `json:"batch_read_count"`
	FallbackReadCount int                     `json:"fallback_read_count"`
	LogicalBytes      int64                   `json:"logical_bytes"`
	AllocatedBytes    int64                   `json:"allocated_bytes,omitempty"`
	AllocatedState    string                  `json:"allocated_state"`
	ShardCount        int                     `json:"shard_count"`
	FileCount         int                     `json:"file_count"`
}

type SearchGenerationRef 

type SearchGenerationRef struct {
	GenerationDigest string                  `json:"generation_digest"`
	Directory        string                  `json:"directory"`
	Revisions        []store.IndexedRevision `json:"revisions"`
	LogicalBytes     int64                   `json:"logical_bytes"`
	AllocatedBytes   int64                   `json:"allocated_bytes,omitempty"`
	AllocatedState   string                  `json:"allocated_state"`
	ShardCount       int                     `json:"shard_count"`
	FileCount        int                     `json:"file_count"`
}

SearchGenerationRef is the small immutable identity used by publication, recovery, readers, and lifecycle. Logical and allocated bytes are distinct; allocated bytes are unavailable rather than guessed on unsupported hosts.

type SearchGenerationRoot 

type SearchGenerationRoot struct {
	Schema     string               `json:"schema"`
	Repository string               `json:"repository"`
	Current    SearchGenerationRef  `json:"current"`
	Prior      *SearchGenerationRef `json:"prior,omitempty"`
}

func ReadSearchGenerationRoot 

func ReadSearchGenerationRoot(indexDir, repository string) (SearchGenerationRoot, error)

type ShardMember 

type ShardMember struct {
	Ordinal          int    `json:"ordinal"`
	Count            int    `json:"count"`
	Name             string `json:"name"`
	ContentDigest    string `json:"content_digest"`
	MetadataDigest   string `json:"metadata_digest"`
	UnitDigest       string `json:"unit_digest"`
	GenerationDigest string `json:"generation_digest"`
}

type WholeManifest 

type WholeManifest struct {
	Schema     string                  `json:"schema"`
	Repository string                  `json:"repository"`
	Revisions  []store.IndexedRevision `json:"revisions"`
	Members    []WholeShardMember      `json:"members"`
	Digest     string                  `json:"digest"`
}

WholeManifest gives the asynchronous whole-repository reader an exact, durable generation receipt. Digest identifies publication content, not semantic Git identity: builder timestamps can change bytes across rebuilds.

func ReadWholeManifest 

func ReadWholeManifest(
	indexDir, repository string,
	revisions []store.IndexedRevision,
) (WholeManifest, error)

ReadWholeManifest validates the canonical control-plane identity without re-hashing member bytes. Search uses it for its warm identity check; exact transition binding calls ValidateWholePublishedContext before mmap.

func ReadWholeManifestSelfContained 

func ReadWholeManifestSelfContained(
	indexDir, repository string,
) (WholeManifest, error)

func ValidateCommittedWholePublication 

func ValidateCommittedWholePublication(
	ctx context.Context,
	indexDir, repository string,
	revisions []store.IndexedRevision,
) (WholeManifest, error)

ValidateCommittedWholePublication proves a whole publication while deliberately allowing a prior-process marker. Reconciliation uses it only to decide whether that marker can be removed.

func ValidateWholePublishedContext 

func ValidateWholePublishedContext(
	ctx context.Context,
	indexDir, repository string,
	revisions []store.IndexedRevision,
) (WholeManifest, error)

func ValidateWholeReceipt 

func ValidateWholeReceipt(
	indexDir, repository string,
	revisions []store.IndexedRevision,
) (WholeManifest, error)

ValidateWholeReceipt checks the control identity, declared file identities, and decoded metadata without reading complete shard contents. Reconciliation uses this repository-local check on every pass; the search transition binder performs the full content-digest validation before opening an exact reader.

type WholeShardMember 

type WholeShardMember struct {
	Ordinal        int    `json:"ordinal"`
	Count          int    `json:"count"`
	Name           string `json:"name"`
	ContentDigest  string `json:"content_digest"`
	ContentBytes   int64  `json:"content_bytes"`
	MetadataDigest string `json:"metadata_digest"`
}

WholeShardMember is the immutable member identity for a whole-repository publication. Whole shards are renamed into a repository-keyed namespace before publication so cleanup can identify ownership even if bytes become unreadable.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.