embedbench

README

Generic GopherLua embedding benchmark

embedbench compares Badger Lua with an immutable GopherLua reference through the exported Go API. This is a separate lane because native Lua cannot serve as the control for Go callbacks, userdata, LState lifecycle, or public LTable methods.

The benchmark worker is a nested Go module. Build the identical worker source once against the candidate source and once against the frozen reference source. Both workers must come from the same clean candidate worker-harness revision:

The frozen reference checkout must be at commit 1fbb92b959ead3685092f24c4173570806e5f9e2.

cd benchmarks/embedworkload

GOWORK=off go build -trimpath -o /tmp/badger-lua-embed-candidate .

cp stock.mod /tmp/badger-lua-embed-reference.mod
cp stock.sum /tmp/badger-lua-embed-reference.sum
GOWORK=off go mod edit -modfile=/tmp/badger-lua-embed-reference.mod \
  -replace=github.com/yuin/gopher-lua=/absolute/path/to/vm1-baseline-1fbb92b
GOWORK=off go build -trimpath -tags gopherlua_reference \
  -modfile=/tmp/badger-lua-embed-reference.mod \
  -o /tmp/badger-lua-embed-reference .

cd ../..

The runner never downloads or builds either implementation. Qualification is fail-closed: both exact SHA-256 digests, both source directories, and both full 40-character source revisions are required. It reads each binary's Go build info, binds each worker's recorded engine replacement to its declared source directory. The candidate must link github.com/mmcdole/badger-lua with no selection tag; the reference must link github.com/yuin/gopher-lua with exactly the gopherlua_reference tag. Both workers must report the same clean worker-harness revision. VM2 is the candidate repository's only linked execution engine. The reference is a separately built frozen binary; it is never linked into or selected by the candidate process.

Workloads

Every workload has an independently recomputed checksum in both the worker and the runner.

Workload	Public behavior measured
state_full	create and close an LState with standard libraries
state_minimal	create and close a bounded batch of SkipOpenLibs states with automatic GC disabled
state_minimal_gc_churn	create and close minimal states in 64-state batches with one forced collection per batch
compile_execute	compile and execute a fixed nested Lua 5.1 program through LState.Load
lua_to_go	a Lua numeric/string loop calling an LGFunction
go_to_lua	protected Go calls into one compiled Lua function with two results
lua_over_go_numeric_table	a hot Lua sum over a wide numeric table populated through the public Go API
userdata	metatable method lookup, userdata assertion, and Go method call
api_dense	public-API construction and readback of runtime-grown dense tables
api_wide_numeric_scan	public-API construction and one-pass readback of 1,024 unique non-preloaded numbers
api_repeated_record	hot repeated four-field record construction and reads
api_unique_record	cold four-field layouts beyond the shape-cache budget
api_shared_prefix_unique_record	four-field records sharing kind = "id" before three per-record unique fields
api_generic_map	512-entry generic string maps through public methods
api_mixed_walk_churn	mixed dense/string tables traversed before and after update, delete, and reinsert churn
retained_program_graph	retained heap per loaded prototype/module/closure/data graph
retained_states_full	retained heap per live full-library state
retained_states_minimal	retained heap per live minimal state
retained_dense_tables	retained heap per live 64-element dense table
retained_wide_numeric_tables	retained heap per live 1,024-element non-preloaded numeric table
retained_repeated_records	retained heap per live repeated record
retained_unique_records	retained heap per live cold-layout record
retained_generic_maps	retained heap per live 512-entry string map

Record keys, generic-map keys, and mixed-table string keys are prepared before the timed/allocation interval. For retained table cases they are also present in the baseline heap. Thus the lane measures table representation and any additional key ownership introduced by the runtime, rather than charging both implementations for benchmark key generation.

The program workloads use one fixed in-memory Lua 5.1 chunk with nested function prototypes, closed upvalues, string and numeric constants, handler closures, and module/data tables. The state and source exist before measurement, but every compile_execute unit calls LState.Load and executes the newly compiled chunk inside the timed/allocation interval. retained_program_graph also compiles and executes each unit, then keeps both its entry function and returned module graph strongly live through the retained-heap collection. Both implementations must return the independently checked result 505 per unit. The runner and worker require suite protocol v5 for this workload set.

The two minimal-state workloads deliberately separate constructor cost from reclamation cost. state_minimal disables automatic GC for its one-shot worker and uses 1,024 units at scale 1, bounding the reference allocation interval near 110 MiB. state_minimal_gc_churn also disables automatic GC, but forces a collection after every 64 closed states, making collection cadence identical across implementations. Every worker record includes gc_mode; the runner rejects a policy mismatch. This replaces the former 20,000-state cell, whose roughly 2.2 GiB allocation interval mixed constructor timing with nondeterministic GC pacing.

Qualification

Run 15 randomized fresh-process samples after two discarded warmups:

go run ./benchmarks/cmd/embedbench \
  -candidate /tmp/badger-lua-embed-candidate \
  -reference /tmp/badger-lua-embed-reference \
  -candidate-source /absolute/path/to/badger-lua \
  -reference-source /absolute/path/to/vm1-baseline-1fbb92b \
  -expect-candidate-sha256 CANDIDATE_WORKER_SHA256 \
  -expect-reference-sha256 REFERENCE_WORKER_SHA256 \
  -expect-candidate-revision CANDIDATE_FULL_REVISION \
  -expect-reference-revision 1fbb92b959ead3685092f24c4173570806e5f9e2 \
  -runs 15 -warmups 2 -seed 1 -scale 1 \
  -output /tmp/gopher-lua-embedding.jsonl

For each workload, the JSONL report contains distributions and candidate/reference ratios for:

• process CPU nanoseconds per operation;
• operation-wall nanoseconds per operation;
• allocated bytes per operation;
• allocations per operation; and
• retained bytes per live unit for retained workloads.

The process CPU metric includes worker setup and shutdown, while operation wall isolates the operation interval. The isolated operation-wall metric is also a hard gate for timing workloads by default; retained-workload wall time remains descriptive because those cases deliberately force garbage collection. Allocation metrics use runtime.MemStats around only the operation interval. Retained cases collect immediately before construction and again while every measured object is strongly live.

The five-percent timing CV guard applies unchanged to both minimal-state timing workloads and every other timing workload. Retained workloads intentionally force a collection while holding a large live graph, so their CPU, allocation, and retained-byte medians remain gated but GC scheduling does not make their CV an evidence-validity failure. Every retained family has a corresponding timing workload that remains subject to the CV guard. The controlled churn workload is not exempted merely because it invokes GC.

The default CPU, operation-wall, B/op, allocs/op, and retained-byte ceilings are all 1.10 relative to the frozen reference and apply independently. Operation wall applies only to timing workloads and retained bytes only to retained workloads. The api_unique_record CPU ceiling is always at most 1.10 whenever qualification gates are active, even if a larger general CPU ceiling is requested. The api_shared_prefix_unique_record CPU, operation-wall, B/op, and allocs/op ceilings are likewise capped at 1.10. Together these guards prevent hot repeated shapes from buying back cold-layout or shared-prefix cache-pollution regressions. compile_execute has the same locked CPU, operation-wall, B/op, and allocs/op caps, and retained_program_graph has a locked 1.10 retained-bytes cap, so compiler or loaded-program memory regressions cannot be hidden by unrelated embedding wins. The wide numeric public scan and Lua-over-Go-table cells also lock all four timing and allocation ceilings at 1.10; the corresponding retained-table cell locks retained bytes at 1.10. These controls prevent a boundary-view optimization from winning on repeated small preloads while regressing unique non-preloaded numbers.

Allocation ratios have defined zero-baseline behavior: zero versus zero records a ratio of 1, candidate zero versus positive reference records 0, and candidate positive versus zero reference records a null ratio plus an *_introduced_from_zero flag. Introducing allocations from zero fails an enabled gate and is marked new! in the text report.

Qualification requires at least 15 samples. A one-sample protocol smoke must disable every gate explicitly:

go run ./benchmarks/cmd/embedbench \
  -candidate /tmp/gopher-lua-embed-candidate \
  -reference /tmp/gopher-lua-embed-reference \
  -runs 1 -warmups 0 -scale 0.001 \
  -max-cv 0 -max-cpu-ratio 0 -max-operation-wall-ratio 0 \
  -max-bytes-ratio 0 \
  -max-allocs-ratio 0 -max-retained-ratio 0 \
  -output /tmp/gopher-lua-embedding-smoke.jsonl

Evidence schema v2 writes a policy record first and a completion record last. The policy fixes the runtime provenance, gate thresholds, seed, scale, and exact expected record counts. The completion record repeats the runtime digests, proves every runtime/workload cell has the exact sample count, and records the noise and regression gate outcomes. A failed gate still gets a complete, inspectable archive; a collection or staged-binary integrity failure has no completion record. Workers are copied into a private staging directory as non-writable executables before sampling and rehashed afterward. -overwrite is required to replace a prior file.

Tests

The runner belongs to the parent module:

go test ./benchmarks/cmd/embedbench

The replaceable worker is tested from its nested module:

cd benchmarks/embedworkload
GOWORK=off go test ./...
GOWORK=off go test -tags gopherlua_reference -modfile=stock.mod ./...

Documentation

Overview

Command embedbench compares GopherLua embedding workers built against a candidate fork and an immutable reference checkout.