Documentation
¶
Overview ¶
Package seed is the meguri binary seed format, .seed, the raw block-framed corpus input that replaces gzipped JSONL for the scale passes (Spec 2074 doc 08). It is splittable at block boundaries, seekable by a per-block first-key index, and parse-free: a reader pulls a URL string with a uvarint length read and no JSON, and a worker handed a disjoint block range shares nothing with the others. It holds only URL bytes; the URLKey is derived at ingest as before.
Index ¶
Constants ¶
const ( // HeaderSize is the fixed header length. It sits at offset zero so a reader // learns the geometry in one read. HeaderSize = 64 // DefaultBlockSize is the block granularity, 1 MiB. A block holds thousands of // adjacent sorted URLs, which is a good split unit and, under the zstd codec, a // good compression window. DefaultBlockSize = 1 << 20 )
The on-disk shape is a 64-byte header, a run of blocks, and a footer block index, with an 8-byte trailer (footer length and CRC) at the very end.
const ManifestName = "manifest.json"
ManifestName is the fixed filename of the shard map written beside the shards.
Variables ¶
var ( ErrShortFile = errors.New("seed: file shorter than header") ErrBadMagic = errors.New("seed: bad magic") ErrVersion = errors.New("seed: unknown format version") ErrCorrupt = errors.New("seed: corrupt file") ErrChecksum = errors.New("seed: footer checksum mismatch") ErrRecordTooBig = errors.New("seed: record does not fit in a block") )
var Magic = [4]byte{'S', 'E', 'E', 'D'}
Magic marks a .seed file.
Functions ¶
func ShardFileName ¶
ShardFileName is the .seed filename for shard i, the name the store build and the manifest both reference.
func WriteManifest ¶
WriteManifest writes m to ManifestName under dir.
Types ¶
type BlockReader ¶
type BlockReader struct {
// contains filtered or unexported fields
}
BlockReader is a cursor over one block's records.
func (*BlockReader) Next ¶
func (br *BlockReader) Next() ([]byte, bool)
Next returns the next URL bytes and true, or nil and false at the block end. The bytes alias the block body and are valid only until the next Next call.
type Codec ¶
type Codec uint8
Codec selects how a block body is stored. Raw is the default: zero decode CPU, fixed block offsets, the simplest splittable form. Zstd compresses each block independently so the split survives while the file stays small.
type Manifest ¶
type Manifest struct {
Version int `json:"version"`
BlockSize int `json:"block_size"`
Codec Codec `json:"codec"`
Records uint64 `json:"records"`
Shards []ShardMeta `json:"shards"`
}
Manifest is the shard map: the ordered list of hostkey-range shards that make up one sharded seed (and, later, the store built from it). It is small and read on open, so a driver learns the whole shard set and each shard's range without touching a shard body. The ranges tile the uint64 hostkey space with no gap and no overlap: shard k covers [HostLo, HostHi), the last shard's HostHi is the max.
func ReadManifest ¶
ReadManifest reads the shard map from dir.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader reads a .seed file. It mmaps the file so a block is a subslice of the mapping (zero copy for the raw codec) and its resident pages are reclaimable page cache, not heap, matching the .meguri read path. It is safe for concurrent Block calls only across distinct block indices sharing no decoder; a caller that fans blocks to workers gives each worker its own Reader or its own BlockReader.
func (*Reader) BlockReader ¶
func (r *Reader) BlockReader(i int) (*BlockReader, error)
BlockReader returns a cursor over block i. For the raw codec the cursor reads straight from the mapping; for zstd it inflates the block into scratch once and reads from there. The returned URL bytes alias the mapping or the scratch and stay valid until the next Next call, so a caller that keeps a URL copies it.
func (*Reader) FirstKey ¶
FirstKey returns block i's first URL bytes, the seek key. A caller binary-searches these to find where a lexical range begins.
func (*Reader) HostRange ¶
HostRange is the hostkey span this seed covers, from the header, the shard's range as seedpack recorded it.
func (*Reader) RecordCount ¶
RecordCount is the total number of URL records in the file.
type ShardMeta ¶
type ShardMeta struct {
Index int `json:"index"`
Path string `json:"path"` // filename relative to the manifest dir
HostLo uint64 `json:"host_lo"`
HostHi uint64 `json:"host_hi"`
Records uint64 `json:"records"`
URLBytes uint64 `json:"url_bytes"`
}
ShardMeta is one shard's entry in the manifest.
type ShardSet ¶
type ShardSet struct {
// contains filtered or unexported fields
}
ShardSet is the write side of a sharded seed: N hostkey-range .seed writers plus the manifest that ties them together. It is the one place the shard geometry lives, so seedpack (a single-threaded corpus scan) and an external bulk producer (many parallel readers, e.g. a Common Crawl parquet fan-out) build byte-compatible seeds through the same routing.
The shard count rounds up to a power of two so the top bits of a hostkey select the shard, and the ranges tile the whole uint64 hostkey space with no gap or overlap. Because the hostkey is a uniform hash of the host, equal-width ranges hold near-equal URL counts, and a host maps to exactly one hostkey so its URLs never split across shards. The caller supplies the hostkey (meguri.HostKeyOf of the host grouping), which keeps this package free of the canon and hashing dependencies and guarantees the key a producer routes on is the same key BulkLoad derives at ingest.
Add is safe for concurrent use: each shard has its own writer and its own lock, so P producers routing to N shards contend only when two happen to hit the same shard, which at N much larger than P is rare. The resident cost is one block buffer per shard plus each shard's growing block index, independent of the corpus size.
func NewShardSet ¶
NewShardSet opens n hostkey-range writers under dir (n rounded up to a power of two) and returns the set ready for Add. dir must already exist. A failure part-way closes the writers opened so far.
func (*ShardSet) Add ¶
Add routes url to the shard its hostKey selects and appends it there. hostKey is the meguri.HostKeyOf of the URL's host grouping, computed by the caller. It is safe for concurrent use across goroutines.
func (*ShardSet) Close ¶
Close flushes and closes every shard writer, writes the manifest under dir, and returns it. After Close the set must not be used again.
func (*ShardSet) HostHi ¶
HostHi is shard i's exclusive high hostkey bound; the last shard's HostHi is the max so the ranges tile the whole space.
func (*ShardSet) N ¶
N is the actual shard count (the requested count rounded up to a power of two).
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer builds one .seed file, appending URL records into fixed-size blocks and writing the header, block bodies, and footer index. It is single-goroutine, the one writer per shard seed. It streams: the resident cost is one block buffer plus the growing block index, not the corpus.
func NewWriter ¶
func NewWriter(path string, opts WriterOptions) (*Writer, error)
NewWriter opens path for writing and reserves the header. The final header is written on Close once the counts and footer offset are known, which is why the file must be seekable.
func (*Writer) Add ¶
Add appends one URL record. A record is a uvarint length followed by the URL bytes; when the next record would not fit the current block, the block is flushed and a fresh one begins, so a record never straddles a block boundary.
func (*Writer) Close ¶
Close flushes the last block, writes the footer index and trailer, then rewrites the header with the final counts and footer offset.
func (*Writer) Records ¶
Records is the number of records added so far, so a rolling producer can decide when a shard has hit its target count.
func (*Writer) SetHostRange ¶
SetHostRange records the hostkey span this shard covers, written into the header on Close. A producer that rolls shards at a host boundary knows a shard's HostHi only when it sees the next shard's first key, so it sets the range before closing.
func (*Writer) URLByteCount ¶
URLByteCount is the total URL bytes added so far.
Source Files