Documentation
¶
Overview ¶
Package extractor turns external state-diff sources (RPC, files, or synthetic generators) into rows in the zksp analysis DB.
All extractors share the Extractor interface so the analysis pipeline can be driven by a mock generator today and a real RPC client tomorrow without touching downstream code.
Index ¶
- Constants
- func ClearRPCState(ctx context.Context, db *storage.DB) error
- func WriteCapability(ctx context.Context, db *storage.DB, cap Capability) error
- type Capability
- type Diagnostics
- type ExtractLimits
- type Extractor
- type MockConfig
- type MockExtractor
- type RPCConfig
- type RPCDiagnostics
- type RPCExtractor
- type StateDiffDiagnostics
- type StateDiffExtractor
Constants ¶
const CapabilityKey = "last_extractor_capability"
CapabilityKey is the schema_meta row that stores the most-recent Extractor.Capability() as a JSON blob. Updated after every successful Extract; read by report / simulate so their output header can self-document which data source the numbers came from.
const ExtractLimitsKey = "extract_limits"
ExtractLimitsKey is the schema_meta key holding the JSON-serialised ExtractLimits used at extract time. On --resume we read this back and refuse to continue if the new run's limits differ — otherwise the resulting DB would be a hybrid (some blocks filtered at one threshold, the rest at another), which the analysis layer can't reason about.
const RPCHighWaterKey = "rpc_high_water_block"
RPCHighWaterKey is the schema_meta key the RPC extractor uses to track the last fully-flushed block. Exposed so callers (e.g. the CLI's `--force` path) can clear it explicitly — storage.Reset() intentionally does not touch schema_meta so it remains a pure data-table truncation, not an extractor-state wipe.
const ScrollPublicRPC = "https://rpc.scroll.io"
ScrollPublicRPC is the default Scroll mainnet endpoint. No API key, rate-limited, fine for pulling a few thousand blocks for a bench.
Variables ¶
This section is empty.
Functions ¶
func ClearRPCState ¶
ClearRPCState removes the RPC extractor's schema_meta keys so a forced re-extraction starts from the user-supplied --start rather than resuming past the old high-water mark. Wipes the stamped extract limits too so a follow-up run is free to choose new ones.
func WriteCapability ¶
WriteCapability stamps cap into schema_meta under CapabilityKey as a JSON blob. Callers typically invoke it from the CLI right after a successful Extract, so report / simulate invocations on the same DB can later read what data source produced the rows.
Types ¶
type Capability ¶
type Capability struct {
// Source is the --source flag value this extractor answers to
// ("mock" / "rpc" / "statediff" / …). Carried here so JSON
// reports don't need to cross-reference the CLI.
Source string `json:"source"`
// ObservesReads is true iff the extractor captures SLOAD-level
// state reads (not just writes). Transfer-log surrogates set this
// to false because Transfer events are emitted on writes only.
ObservesReads bool `json:"observes_reads"`
// ObservesNonTransferWrite is true iff the extractor captures
// every SSTORE, not just writes that emit an ERC-20 / ERC-721
// Transfer event. Mock and real state-diff extractors set this
// to true; the Transfer-log surrogate sets it to false.
ObservesNonTransferWrite bool `json:"observes_non_transfer_write"`
// SlotIDForm is a short human-readable description of how the
// extractor mints slot_id (e.g. "synthetic / deterministic",
// "contract:holder (surrogate)", "contract:slotkey (real)").
// Informational only; downstream code never parses it.
SlotIDForm string `json:"slot_id_form"`
}
Capability is a self-description of what an Extractor sees and how it mints slot_id. It is persisted into schema_meta on every successful Extract call so downstream consumers can self-document their output without re-inspecting the extractor.
Two extractors with different capabilities should produce comparable rows (slot_id format differs but the survival pipeline treats slot_id as opaque), but downstream comparisons of Hill α, censoring rate, or cost regime between sources with different capabilities should be flagged — a Transfer-log surrogate will systematically under-report writes, for example.
func ReadCapability ¶
ReadCapability returns the most recently persisted Capability for this DB. The second return distinguishes "no extractor has run yet" from a SQL error — a fresh DB returns (zero, false, nil), which the CLI can surface as "unknown data source" rather than falsely stamping the output.
type Diagnostics ¶
type Diagnostics struct {
Contracts int
Slots int
Events int
PreWindowContracts int
PreWindowSlots int
PeriodicContracts int
EventsInWindow int
}
Diagnostics is a small report produced by the mock extractor describing how many contracts / slots / events landed in emergent buckets like "pre-window" or "periodic". The simulator config declares the inputs; these are the observed outputs we can later compare against plan assumptions.
type ExtractLimits ¶
type ExtractLimits struct {
Source string `json:"source"`
MaxEventsPerBlock uint64 `json:"max_events_per_block"`
MaxContractsPerBlock uint64 `json:"max_contracts_per_block"`
MaxSlotsPerBlock uint64 `json:"max_slots_per_block"`
}
ExtractLimits is the persisted form of MaxEventsPerBlock / MaxContractsPerBlock / MaxSlotsPerBlock. Stamped into schema_meta at the end of every successful Extract; the next --resume reads it and refuses to continue if the new limits don't match (different filtering would silently produce a hybrid-DB the analysis layer can't reason about).
type Extractor ¶
type Extractor interface {
// Extract is expected to be idempotent: calling it twice on the same DB
// must not duplicate access events. Mock implementations achieve this by
// truncating; real RPC implementations should use a high-water mark.
Extract(ctx context.Context, db *storage.DB) error
// Capability describes what slot touches the extractor observes
// and how it mints slot_id. Downstream report / simulate output
// stamp this so a reader of a Brier score or a cost table knows
// whether the data was produced by a full state-diff source, a
// Transfer-log surrogate, or a synthetic generator. Must be
// stable — i.e. a constant the implementation can return from
// any state.
Capability() Capability
}
Extractor is anything that can populate a zksp DB with contracts, slots and access events.
type MockConfig ¶
type MockConfig struct {
Seed uint64
NumContracts int
// SlotsPerContract is sampled as a Pareto with these parameters; with
// alpha=2 and xmin=100 the population mean is 200, matching the spec.
SlotsPerContractXmin float64
SlotsPerContractAlpha float64
SlotsPerContractMax int
// AccessRate is per-slot per-block intensity, drawn from a Pareto.
// Heavy tail (small alpha) means most slots are nearly dead while a
// handful are extremely hot, mirroring real on-chain access patterns.
AccessRateXmin float64
AccessRateAlpha float64
MaxEventsPerSlot int
IntraContractCorrelation float64
PeriodicContractsRatio float64
// PeriodBlocks is the cycle length used by periodic contracts.
PeriodBlocks uint64
TotalBlocks uint64
// Window is the analysis observation window. The mock itself generates
// the full trace [0, TotalBlocks), but downstream EDA / survival only
// sees events inside Window, which is where censoring and truncation
// come from. Keeping it on MockConfig lets the generator hit a target
// pre-window slot fraction deterministically.
Window domain.ObservationWindow
// PreWindowSlotFraction controls the share of contracts whose deploy
// block lands before Window.Start. A value of 0.3 reproduces the plan's
// "30% slots pre-exist the observation window" assumption; those slots
// will be flagged as left-truncated by the interval builder.
PreWindowSlotFraction float64
ContractTypeDistribution map[domain.ContractCategory]float64
}
MockConfig parameterizes the synthetic state-diff generator. All fields have sensible defaults via DefaultMockConfig; tests typically shrink NumContracts and TotalBlocks to keep runs fast.
func DefaultMockConfig ¶
func DefaultMockConfig() MockConfig
DefaultMockConfig returns the canonical Phase-1 generator settings. They match configs/default.yaml.
type MockExtractor ¶
type MockExtractor struct {
// contains filtered or unexported fields
}
MockExtractor is the deterministic synthetic extractor used by Phase-1 development and tests.
func NewMockExtractor ¶
func NewMockExtractor(cfg MockConfig) *MockExtractor
func (*MockExtractor) Capability ¶
func (*MockExtractor) Capability() Capability
Capability describes what the mock extractor sees. The generator is fully synthetic so by construction it captures every SLOAD and every SSTORE — there is no Transfer-log surrogate gap.
func (*MockExtractor) Extract ¶
Extract generates contracts/slots/events according to cfg and writes them to db. Reusing the same Seed produces byte-identical output.
func (*MockExtractor) LastDiagnostics ¶
func (m *MockExtractor) LastDiagnostics() Diagnostics
LastDiagnostics returns the diagnostics from the most recent Extract call. Zero value if Extract has not been called yet.
type RPCConfig ¶
type RPCConfig struct {
Endpoint string
Start uint64
End uint64
HTTPClient *http.Client
BatchSize int
// StrictCategories, when true, makes a statediff Extract return
// an error (instead of a slog.Warn) if more than
// otherCategoryWarnRatio of contracts the run touched failed
// classification and landed in ContractOther. The Transfer-log
// surrogate (RPCExtractor) ignores this flag — its category
// signal comes from log.Topics, not from the function-selector
// + bytecode pipeline that this guardrail watches.
StrictCategories bool
// MaxRetries is the number of *retries* (not total attempts)
// rpcCall will perform on transient HTTP-level failures
// (timeout / connection reset / 5xx). Default (0 in struct, 3
// after DefaultRPCConfig) is what production runs use; tests
// often set it to 0 to fail fast on the first error.
// Protocol-level RPC errors (e.g. -32601 method not found) are
// never retried — they don't recover by being asked again.
MaxRetries int
// RetryBaseDelay is the first-retry backoff. Subsequent
// retries double the delay (exponential), capped at 10s per
// attempt. Zero falls back to 200ms in rpcCall.
RetryBaseDelay time.Duration
// Limits are per-block guardrails. Zero means "no limit"
// (preserves existing behaviour for callers that don't opt in).
// When > 0, processBlock fail-closes with a structured error
// the moment any tally exceeds the threshold.
//
// Calibration on scroll_100k (Transfer-log surrogate, 100k
// blocks): observed max events/block = 218, max distinct
// contracts/block = 12, max distinct slots/block = 218.
// Recommended thresholds are 10× headroom over those — see
// internal/extractor/EXTRACT_LIMITS.md for the full data,
// rationale, and the SQL used to derive them.
MaxEventsPerBlock uint64
MaxContractsPerBlock uint64
MaxSlotsPerBlock uint64
}
RPCConfig configures the RPC extractor. Required fields: Endpoint, Start, End. HTTPClient defaults to http.DefaultClient with a conservative timeout. BatchSize controls how many events accumulate before a SQLite flush.
func DefaultRPCConfig ¶
func DefaultRPCConfig() RPCConfig
DefaultRPCConfig returns a Scroll-mainnet-friendly default. Callers typically override Start/End to point at the block range they want.
Retry defaults (3 retries, 200ms base) mean a single transient blip eats ~3.4s before recovering, and a sustained outage takes 200+400+800+1600 ≈ 3s of waits before the run fails. Tuned to public-RPC reliability — `rpc.scroll.io` drops connections often enough that without retry, multi-hour extracts almost never finish.
type RPCDiagnostics ¶
type RPCDiagnostics struct {
BlocksRequested int
BlocksFetched int
ReceiptsFetched int
LogsSeen int
TransferLogs int
SlotsCreated int
// EventsAttempted is the number of rows passed into
// InsertAccessEvents across all flushes for this run — i.e.
// everything the extractor wanted to write. EventsPersisted is
// the number actually inserted after the unique index dropped
// duplicates on a resume re-fetch. When Attempted > Persisted,
// the delta is the number of rows a previous incarnation of
// this run had already committed.
EventsAttempted int
EventsPersisted int
ContractsCreated int
StartBlock uint64
EndBlock uint64
}
RPCDiagnostics mirrors the mock extractor's diagnostics shape so CLI / tests can print "here's what the run produced" without peeking inside the extractor.
type RPCExtractor ¶
type RPCExtractor struct {
// contains filtered or unexported fields
}
RPCExtractor is a TRANSFER-LOG SURROGATE extractor, not a true state-diff / storage-access extractor. It walks blocks over a JSON-RPC endpoint and synthesizes pseudo "balance slot" rows from ERC-20 / ERC-721 Transfer events — the one log signature every EVM-compatible chain shares by convention. The downstream analysis pipeline treats slot_id as an opaque key, so these rows feed the same EDA / survival / tiering passes the mock extractor does, and let us run against real on-chain activity without needing trace APIs most public endpoints don't expose.
IT DOES NOT CAPTURE:
- slot writes that don't emit a Transfer (storage rebalances, admin settings, DEX pool state, governance bookkeeping…),
- slot reads at all,
- the real EVM storage-slot identifiers (we hash contract||holder as a deterministic surrogate id).
For full state-access traces, the right data source is debug_traceBlockByNumber with a prestate/stateDiff tracer on an archive node, or a chain's specialized state-diff endpoint. Those are a drop-in replacement: the Extractor interface is the only contract downstream code depends on.
func NewRPCExtractor ¶
func NewRPCExtractor(cfg RPCConfig) (*RPCExtractor, error)
NewRPCExtractor validates cfg and returns a ready-to-run extractor.
func (*RPCExtractor) Capability ¶
func (*RPCExtractor) Capability() Capability
Capability declares this extractor as a Transfer-log surrogate: it only sees slot touches emitted as ERC-20 / ERC-721 Transfer events. Non-Transfer writes (arbitrary SSTORE, DEX pool updates, governance state) and all reads are invisible. A full state-diff replacement (debug_traceBlockByNumber + prestateTracer) is planned — see the statediff extractor when it lands — and will differ from this one by both ObservesReads and ObservesNonTransferWrite being true.
func (*RPCExtractor) Extract ¶
Extract honours the Extractor interface. Walks blocks in [Start, End] fetching each block and its receipts, synthesizing (slot, event) rows from Transfer logs. Persists contracts/slots/events to db.
Idempotency is driven by the schema_meta high-water mark "rpc_high_water_block": Extract skips blocks up to and including the stored value, and updates it as it makes progress. Callers that want a full refresh should call db.Reset first (via `extract --force`).
func (*RPCExtractor) LastDiagnostics ¶
func (e *RPCExtractor) LastDiagnostics() RPCDiagnostics
LastDiagnostics returns the diagnostics from the most recent Extract. Zero value if Extract has not been called yet.
type StateDiffDiagnostics ¶
type StateDiffDiagnostics struct {
BlocksRequested int
BlocksFetched int
// StorageTouches is the union over all tx traces in all blocks
// of distinct (contract, slot_key, block) slot touches. Equals
// StorageReads + StorageWrites by construction.
StorageTouches int
StorageReads int
StorageWrites int
ContractsCreated int
SlotsCreated int
EventsAttempted int
EventsPersisted int
// OtherCategoryContracts is the number of contracts the
// 4-byte-signature classifier could not place; the run aborts
// (or warns loudly, depending on policy) if this exceeds
// otherCategoryWarnRatio of ContractsCreated.
OtherCategoryContracts int
StartBlock uint64
EndBlock uint64
}
StateDiffDiagnostics mirrors RPCDiagnostics' shape so CLI / tests have a uniform "what did this run produce" surface, but tracks the state-diff-specific counters that don't exist on the surrogate (read-vs-write split, Other-category guardrail).
type StateDiffExtractor ¶
type StateDiffExtractor struct {
// contains filtered or unexported fields
}
StateDiffExtractor walks blocks via debug_traceBlockByNumber + prestateTracer (diffMode) and lands every storage touch — read AND write, Transfer-emitting or not — as an access_event. This is the "real" extractor the Transfer-log surrogate (RPCExtractor) was always a stand-in for.
Coverage:
- prestateTracer's `pre` block lists every slot a tx READ or wrote (it's the state required to replay the tx).
- The `post` block lists slots whose value CHANGED.
- A slot in `pre` ∩ `post` is a write; a slot in `pre` only is a read; a slot in `post` only doesn't happen (writes always have a pre-state).
The result: ObservesReads and ObservesNonTransferWrite both hold, and slot_id is "contract:<32-byte-slotKey>" — the actual EVM storage key, not a holder-derived surrogate.
Event granularity: events land per (slot, block), not per (slot, tx). Per-block aggregation matches what the tiering policy cares about (was this slot touched in this block?) and avoids duplicating rows for slots multiple txs in the same block touch. Promotion rule: any write in the block supersedes a read; the first tx_hash that touched the slot is kept on the persisted row for traceability.
Idempotency / resume: same schema_meta high-water mark as RPCExtractor (RPCHighWaterKey). They are mutually exclusive on a given DB — switching `--source` between rpc and statediff requires `--force` (which calls ClearRPCState).
Contract classification: prestateTracer doesn't carry tx input data, so the 4-byte function-selector heuristic that drives stratification needs a separate signal. We pull it from eth_getBlockByNumber(blockHash, true), which returns every tx's input in one call — substantially cheaper than the alternative (callTracer fan-out, +1 expensive trace per tx). When the selector pipeline can't classify a contract, the run falls back to a bytecode fingerprint via eth_getCode (cached per address). A contract that fails both pipelines lands in ContractOther; the otherCategoryWarnRatio guardrail surfaces a high Other-rate as a loud warning by default, and when StrictCategories is enabled (CLI: --strict-categories), it errors out the run instead — use strict in CI / scheduled jobs to catch classifier regressions.
Cost: debug_trace* is the most expensive RPC method by 100×–500× vs eth_getBlockByNumber. Per-block, this extractor sends:
- 1× debug_traceBlockByNumber (the trace)
- 1× eth_getBlockByNumber(true) (txs+inputs for classification)
- 0..N× eth_getCode (bytecode fallback, only first time per addr)
Public chain endpoints rarely expose debug_trace*; production runs need an archive-capable node (Alchemy Growth / QuickNode archive / self-hosted Erigon). The extractor surfaces a clear error when the endpoint refuses the method instead of silently falling back to the surrogate.
func NewStateDiffExtractor ¶
func NewStateDiffExtractor(cfg RPCConfig) (*StateDiffExtractor, error)
NewStateDiffExtractor validates cfg and returns a ready-to-run extractor. Mirrors NewRPCExtractor: same defaults, same validation surface, so the CLI doesn't need to special-case which extractor kind it built.
func (*StateDiffExtractor) Capability ¶
func (*StateDiffExtractor) Capability() Capability
Capability declares full state-diff coverage: every SLOAD and every SSTORE, regardless of whether the writing tx emitted a Transfer event. slot_id is "contract:<32-byte-slotKey>", the real EVM storage key — collision-free across contracts because we concatenate the contract address.
func (*StateDiffExtractor) Extract ¶
Extract walks blocks in [Start, End], fetching each block's per-tx prestateTracer trace, parsing the (pre, post) state diff, and emitting one AccessEvent per (slot, block) touch. Resume is driven by RPCHighWaterKey — same as RPCExtractor, so a forced re-run goes through `--force` + ClearRPCState.
func (*StateDiffExtractor) LastDiagnostics ¶
func (e *StateDiffExtractor) LastDiagnostics() StateDiffDiagnostics
LastDiagnostics returns the diagnostics from the most recent Extract call. Zero value if Extract has not been called yet.
Source Files