benchmark

package
v1.13.15 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 28, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// CIStableV1FixtureID identifies the fixed Phase 11 calibration candidate.
	// Changing any value in this profile requires a new fixture identifier.
	CIStableV1FixtureID = "ci-stable-v1"
	// Paired fixture IDs identify immutable profile-specific inputs used by the
	// paired Phase 11 benchmark contract. Historical IDs remain supported.
	CIPairedW1V1FixtureID = "ci-paired-w1-v1"
	CIPairedW4V1FixtureID = "ci-paired-w4-v1"
	CIPairedW1V2FixtureID = "ci-paired-w1-v2"
	CIPairedW4V2FixtureID = "ci-paired-w4-v2"
	FixtureSeed           = int64(1701)
)

Variables

This section is empty.

Functions

func EqualRestoredTreeHashes 

func EqualRestoredTreeHashes(left, right map[string]string) (bool, string)

EqualRestoredTreeHashes returns (true, "") when left and right contain identical relative-path→SHA-256 mappings. On mismatch it returns false and a human-readable description of the first difference found (sorted by path for deterministic output).

func GenerateDataset 

func GenerateDataset(baseDir string, cfg DatasetConfig) error

GenerateDataset creates a deterministic benchmark dataset under baseDir.

Every generated file is named deterministically as: file_0001.bin, file_0002.bin, ...

func HashRestoredTree 

func HashRestoredTree(root string) (map[string]string, error)

HashRestoredTree walks the directory tree rooted at root deterministically (sorted by relative path) and returns a map of relative path → SHA-256 hex digest of file contents. Empty directories are ignored. This is used to verify that store→restore produces identical user-visible output across independent benchmark runs.

func RecordCPU added in v1.9.0 

func RecordCPU(operationType string, cpuTime time.Duration)

RecordCPU records CPU time spent in each operation phase. Duration should be the CPU time (not wall-clock time) for accurate overhead calculation. Calls made outside Measure are ignored.

func RecordMemory added in v1.9.0 

func RecordMemory(peakMemoryBytes int64, allocations int64)

RecordMemory records peak heap usage and allocation count. Calls made outside Measure are ignored.

func RecordProcessed 

func RecordProcessed(files int, bytes int64)

RecordProcessed increments files and bytes counters for the active Measure call. Calls made outside Measure are ignored.

func RecordStorage added in v1.9.0 

func RecordStorage(logicalBytes, compressedBytes, storedBytes int64)

RecordStorage records storage metrics: logical, compressed, and final stored sizes. Ratios are computed internally for stability across runs. Calls made outside Measure are ignored.

func RecordStructural added in v1.9.0 

func RecordStructural(compressedBlocks, uncompressedBlocks, storeIfSmallerFallbacks int64)

RecordStructural records compression effectiveness: blocks compressed, not compressed, and fallbacks. Calls made outside Measure are ignored.

func RecordThroughput added in v1.9.0 

func RecordThroughput(operationType string, mbps float64)

RecordThroughput records operation-specific throughput in MB/s. Calls made outside Measure are ignored.

func RequiresCaseDatabaseIsolation added in v1.13.11 

func RequiresCaseDatabaseIsolation(preset DatasetPreset) bool

RequiresCaseDatabaseIsolation reports whether a preset owns a fresh database and filesystem environment for every benchmark case.

func WriteDeterministicFile 

func WriteDeterministicFile(path string, size int64, seed int64) error

WriteDeterministicFile writes size bytes of deterministic pseudo-random content derived from seed to path. It is exported so that external validation helpers (e.g. restore-hash determinism checks) can produce the same byte sequence.

Types

type BenchmarkCase 

type BenchmarkCase struct {
	Name      string
	Run       func(ctx BenchmarkContext) error
	Execution execution.Options
}

BenchmarkCase defines one benchmark scenario execution unit.

func CoreScenarios 

func CoreScenarios(cfg ScenarioConfig) []BenchmarkCase

CoreScenarios returns the v1.7 Step 4 real-world benchmark cases.

type BenchmarkContext 

type BenchmarkContext struct {
	RepoPath string
	DataPath string
	ExtraEnv map[string]string
}

BenchmarkContext contains per-case isolated paths.

type CaseEnvironmentFactory added in v1.13.11 

type CaseEnvironmentFactory func(caseName string) (map[string]string, func() error, error)

CaseEnvironmentFactory creates per-case environment overrides and cleanup. It runs before the case timer starts.

type CommandRunner 

type CommandRunner func(spec CommandSpec) error

CommandRunner executes one benchmark command.

type CommandSpec 

type CommandSpec struct {
	Executable string
	Args       []string
	WorkingDir string
	Env        []string
}

CommandSpec describes a coldkeep command invocation used by benchmark scenarios.

type CorpusBuilder added in v1.9.0 

type CorpusBuilder struct {
	// contains filtered or unexported fields
}

CorpusBuilder orchestrates reproducible corpus generation with versioning.

func NewCorpusBuilder added in v1.9.0 

func NewCorpusBuilder(baseDir string) *CorpusBuilder

NewCorpusBuilder creates a new builder for a corpus location.

func (*CorpusBuilder) GenerateCorpus added in v1.9.0 

func (cb *CorpusBuilder) GenerateCorpus(def CorpusDefinition) error

GenerateCorpus creates all files for a corpus definition with SHA256 validation.

func (*CorpusBuilder) ValidateCorpus added in v1.9.0 

func (cb *CorpusBuilder) ValidateCorpus(def CorpusDefinition) (bool, error)

ValidateCorpus checks all files in a generated corpus match expected hashes and sizes.

type CorpusContent added in v1.9.0 

type CorpusContent struct {
	Type string // "json", "logs", "source", "binary", "jpeg_sim", "zip_sim", "random", "encrypted"
	Seed int64
	// CompressionRatio is estimated compressed_size / original_size for benchmark corpus shaping.
	// Typical interpretation: < 0.5 highly compressible, 0.5-0.8 mixed, > 0.95 already compressed.
	CompressionRatio float64

	// JSON-specific
	JSONObjects int64

	// Log-specific
	LogLines int64
	LogSize  int64

	// Binary/Office-specific
	TextRatio float64 // % of file that is plain text (0.0 - 1.0)

	// Random/Encrypted
	EntropyLevel float64 // 0.0 = zero, 1.0 = maximum (random)
}

CorpusContent describes how to generate file content.

type CorpusDefinition added in v1.9.0 

type CorpusDefinition struct {
	Type        CorpusType
	Version     string // e.g., "v1.0"
	Name        string // descriptive name
	Files       []CorpusFile
	Seed        int64 // deterministic generation seed
	SHA256      string
	Description string
}

CorpusDefinition describes a specific benchmark corpus with stable content.

func StandardCorpora added in v1.9.0 

func StandardCorpora() []CorpusDefinition

StandardCorpora defines all stable benchmark corpora.

type CorpusFile added in v1.9.0 

type CorpusFile struct {
	Name    string // unique filename within corpus
	Size    int64  // size in bytes
	Content CorpusContent
	SHA256  string // perfile validation hash
}

CorpusFile describes a single file within a corpus.

type CorpusRegistry added in v1.9.0 

type CorpusRegistry struct {
	// contains filtered or unexported fields
}

CorpusRegistry manages benchmark corpora for reproducible performance testing.

func NewCorpusRegistry added in v1.9.0 

func NewCorpusRegistry(baseDir string) *CorpusRegistry

NewCorpusRegistry creates a new corpus registry at the specified base directory.

func (*CorpusRegistry) CleanupCorpora added in v1.9.0 

func (cr *CorpusRegistry) CleanupCorpora() error

CleanupCorpora removes all generated corpora to force regeneration.

func (*CorpusRegistry) EnsureCorpora added in v1.9.0 

func (cr *CorpusRegistry) EnsureCorpora() error

EnsureCorpora ensures all standard corpora are generated and validated.

func (*CorpusRegistry) GetCorpusFiles added in v1.9.0 

func (cr *CorpusRegistry) GetCorpusFiles(corpusType CorpusType, version string) ([]string, error)

GetCorpusFiles returns the list of files in a generated corpus.

func (*CorpusRegistry) GetCorpusManifest added in v1.9.0 

func (cr *CorpusRegistry) GetCorpusManifest(corpusType CorpusType, version string) (string, error)

GetCorpusManifest returns the manifest content for a corpus.

func (*CorpusRegistry) GetCorpusPath added in v1.9.0 

func (cr *CorpusRegistry) GetCorpusPath(corpusType CorpusType, version string) string

GetCorpusPath returns the directory path for a specific corpus.

func (*CorpusRegistry) GetCorpusStats added in v1.9.0 

func (cr *CorpusRegistry) GetCorpusStats(corpusType CorpusType) CorpusStats

GetCorpusStats returns statistics for a specific corpus.

func (*CorpusRegistry) PrintCorpusStats added in v1.9.0 

func (cr *CorpusRegistry) PrintCorpusStats()

PrintCorpusStats prints human-readable corpus statistics.

func (*CorpusRegistry) ValidateAllCorpora added in v1.9.0 

func (cr *CorpusRegistry) ValidateAllCorpora() []error

ValidateAllCorpora checks integrity of all generated corpora.

type CorpusStats added in v1.9.0 

type CorpusStats struct {
	Type      CorpusType
	Version   string
	Name      string
	FileCount int
	TotalSize int64
	AvgRatio  float64
	Generated bool
}

CorpusStats provides summary statistics about a corpus.

type CorpusType added in v1.9.0 

type CorpusType string

CorpusType represents different benchmark corpus classifications. 

const (
	// CorpusTypeHighlyCompressible: source code, JSON, logs, plaintext with high redundancy.
	CorpusTypeHighlyCompressible CorpusType = "highly_compressible"
	// CorpusMixedRealistic: office files, sqlite databases, binaries, mixed real-world files.
	CorpusTypeMixedRealistic CorpusType = "mixed_realistic"
	// CorpusTypeAlreadyCompressed: JPEG, MP4, ZIP, PDF - already optimized formats.
	CorpusTypeAlreadyCompressed CorpusType = "already_compressed"
	// CorpusTypeAdversarial: random bytes, encrypted blobs - should skip compression.
	CorpusTypeAdversarial CorpusType = "adversarial_random"
)

type DatasetConfig 

type DatasetConfig struct {
	NumFiles      int
	FileSizeBytes int
	Pattern       string // "random" | "repeated" | "mixed"
	Seed          int64
}

DatasetConfig controls synthetic dataset generation for benchmark runs.

type DatasetPreset 

type DatasetPreset string
const (
	DatasetPresetSmall        DatasetPreset = "small"
	DatasetPresetMedium       DatasetPreset = "medium"
	DatasetPresetLarge        DatasetPreset = "large"
	DatasetPresetCIStableV1   DatasetPreset = CIStableV1FixtureID
	DatasetPresetCIPairedW1V1 DatasetPreset = CIPairedW1V1FixtureID
	DatasetPresetCIPairedW4V1 DatasetPreset = CIPairedW4V1FixtureID
	DatasetPresetCIPairedW1V2 DatasetPreset = CIPairedW1V2FixtureID
	DatasetPresetCIPairedW4V2 DatasetPreset = CIPairedW4V2FixtureID
)

func ParseDatasetPreset 

func ParseDatasetPreset(raw string) (DatasetPreset, error)

type FinalStateObserver added in v1.13.11 

type FinalStateObserver func(caseName string, ctx BenchmarkContext) (json.RawMessage, error)

FinalStateObserver captures benchmark-only evidence after a case finishes and before its external resources and temporary paths are cleaned up. The raw message must encode one sanitized JSON object.

type FixtureCase added in v1.13.11 

type FixtureCase struct {
	Name string `json:"name"`
	Seed int64  `json:"seed"`
}

FixtureCase identifies one ordered benchmark case and its deterministic seed.

type FixtureDescriptor added in v1.13.11 

type FixtureDescriptor struct {
	ID                     string        `json:"id"`
	Seed                   int64         `json:"seed"`
	LargeFileSizeBytes     int64         `json:"large_file_size_bytes"`
	ManySmallFileCount     int           `json:"many_small_file_count"`
	ManySmallFileSizeBytes int           `json:"many_small_file_size_bytes"`
	MixedFileCount         int           `json:"mixed_file_count"`
	MixedMinFileSizeBytes  int           `json:"mixed_min_file_size_bytes"`
	MixedMaxFileSizeBytes  int           `json:"mixed_max_file_size_bytes"`
	RemoveEvery            int           `json:"remove_every"`
	CaseDatabaseIsolation  bool          `json:"case_database_isolation"`
	OrderedCases           []FixtureCase `json:"ordered_cases"`
}

FixtureDescriptor records the inputs that define a benchmark fixture.

func FixtureDescriptorFor added in v1.13.11 

func FixtureDescriptorFor(preset DatasetPreset, cfg ScenarioConfig) FixtureDescriptor

FixtureDescriptorFor returns the deterministic descriptor for a preset.

type IterationReport 

type IterationReport struct {
	Iteration int      `json:"iteration"`
	Results   []Result `json:"results"`
}

type Metrics 

type Metrics struct {
	// Core operation metrics
	Duration       time.Duration
	FilesProcessed int
	BytesProcessed int64
	ThroughputMBps float64

	// Storage Metrics: distinguish logical vs physical data sizes
	LogicalBytes    int64 // input data size (uncompressed)
	CompressedBytes int64 // size after compression (before storage overhead)
	StoredBytes     int64 // final data on disk (storage + encryption + overhead)
	// CompressionRatio is the size fraction CompressedBytes / LogicalBytes.
	// This benchmark metric is intentionally <= 1.0 for compressible data.
	// Storage/read-path docs use the inverse term CompressionFactor.
	CompressionRatio float64
	PhysicalRatio    float64 // StoredBytes / LogicalBytes, includes all overhead

	// Throughput Metrics: operation-specific throughput
	StoreMBps   float64 // MB/s during store operation
	RestoreMBps float64 // MB/s during restore operation
	VerifyMBps  float64 // MB/s during verify operation

	// CPU Metrics: time spent in each operation phase
	CompressionCPUTime time.Duration // CPU time for compression phase
	RestoreCPUTime     time.Duration // CPU time for restore/decompression
	VerifyCPUTime      time.Duration // CPU time for verify operation

	// Memory Metrics: heap and allocation pressure
	PeakMemoryBytes int64 // peak heap usage during operation
	AllocationCount int64 // number of allocations (alloc churn indicator)

	// Structural Metrics: compression effectiveness tracking
	CompressedBlocks       int64 // blocks where compression was applied
	UncompressedBlocks     int64 // blocks stored as-is (compression ineffective or skipped)
	StoreIfSmallerFallback int64 // instances where uncompressed fallback was used
}

Metrics captures comprehensive benchmark run measurements across five categories: Storage, Throughput, CPU, Memory, and Structural metrics.

Storage Metrics distinguish logical input size from compressed and stored sizes, enabling analysis of compression effectiveness and storage transform overhead.

Throughput Metrics track MB/s for store, restore, and verify operations separately, allowing identification of CPU-intensive transformations.

CPU Metrics segment CPU time by operation phase to identify compression overhead.

Memory Metrics monitor peak heap usage and allocation churn from transforms.

Structural Metrics track compression decisions (blocks compressed vs not, fallbacks) for repositories with mixed stored/uncompressed content.

func Measure 

func Measure(fn func() error) (Metrics, error)

Measure executes fn, captures elapsed time, and computes throughput.

Throughput is calculated as: MB/s = bytes / duration

The fn can report processed counters via RecordProcessed, RecordStorage, RecordCPU, RecordMemory, and RecordStructural.

type Result 

type Result struct {
	Name      string
	Duration  time.Duration
	Metrics   Metrics
	Execution execution.Options
	ExecStats execution.ExecutionStats
	// DiagnosticFinalState is separately versioned diagnostic evidence. It does
	// not change the surrounding benchmark report schema.
	DiagnosticFinalState json.RawMessage
	Success              bool
	Error                string
}

Result captures one benchmark case execution outcome.

func RunBenchmark 

func RunBenchmark(cases []BenchmarkCase) ([]Result, error)

RunBenchmark executes benchmark cases sequentially with isolated temp paths.

func RunBenchmarkWithEnvironmentFactory added in v1.13.11 

func RunBenchmarkWithEnvironmentFactory(cases []BenchmarkCase, factory CaseEnvironmentFactory) ([]Result, error)

RunBenchmarkWithEnvironmentFactory executes cases with per-case external resources, such as isolated benchmark databases.

func RunBenchmarkWithEnvironmentFactoryAndObserver added in v1.13.11 

func RunBenchmarkWithEnvironmentFactoryAndObserver(
	cases []BenchmarkCase,
	factory CaseEnvironmentFactory,
	observer FinalStateObserver,
) ([]Result, error)

RunBenchmarkWithEnvironmentFactoryAndObserver executes cases with per-case external resources and captures final state before any cleanup runs.

type RunReport 

type RunReport struct {
	GeneratedAtUTC string            `json:"generated_at_utc"`
	Dataset        DatasetPreset     `json:"dataset"`
	Repeat         int               `json:"repeat"`
	Fixture        FixtureDescriptor `json:"fixture"`
	Iterations     []IterationReport `json:"iterations"`
}

func RunPreset 

func RunPreset(preset DatasetPreset, repeat int, base ScenarioConfig) (RunReport, error)

type ScenarioConfig 

type ScenarioConfig struct {
	ColdkeepExecutable     string
	Codec                  string
	Compression            string
	Execution              execution.Options
	Seed                   int64
	LargeFileSizeBytes     int64
	ManySmallFileCount     int
	ManySmallFileSizeBytes int
	MixedFileCount         int
	MixedMinFileSizeBytes  int
	MixedMaxFileSizeBytes  int
	RemoveEvery            int
	RunTag                 string
	ExtraEnv               map[string]string
	Runner                 CommandRunner
	CaseEnvironmentFactory CaseEnvironmentFactory
	CaseDatabaseIsolation  bool
}

ScenarioConfig controls Step 4 core benchmark scenarios.

func CIPairedW1V1ScenarioConfig added in v1.13.11 

func CIPairedW1V1ScenarioConfig() ScenarioConfig

CIPairedW1V1ScenarioConfig is the fixed workers=1 paired-gate fixture.

func CIPairedW1V2ScenarioConfig added in v1.13.11 

func CIPairedW1V2ScenarioConfig() ScenarioConfig

CIPairedW1V2ScenarioConfig is the bounded workers=1 paired diagnostic fixture.

func CIPairedW4V1ScenarioConfig added in v1.13.11 

func CIPairedW4V1ScenarioConfig() ScenarioConfig

CIPairedW4V1ScenarioConfig is the fixed workers=4 paired-gate fixture.

func CIPairedW4V2ScenarioConfig added in v1.13.11 

func CIPairedW4V2ScenarioConfig() ScenarioConfig

CIPairedW4V2ScenarioConfig is the bounded workers=4 paired diagnostic fixture.

func CIStableV1ScenarioConfig added in v1.13.11 

func CIStableV1ScenarioConfig() ScenarioConfig

CIStableV1ScenarioConfig is the fixed release-gate calibration candidate.

func PresetScenarioConfig 

func PresetScenarioConfig(preset DatasetPreset) (ScenarioConfig, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.