Documentation
¶
Index ¶
- Constants
- Variables
- func SerializeDV(bitmap *RoaringPositionBitmap) ([]byte, error)
- type DVWriter
- type RoaringPositionBitmap
- func DeserializeDV(data []byte, expectedCardinality int64) (*RoaringPositionBitmap, error)
- func DeserializeRoaringPositionBitmap(data []byte) (*RoaringPositionBitmap, error)
- func NewRoaringPositionBitmap() *RoaringPositionBitmap
- func ReadDV(fs iceio.IO, dvFile iceberg.DataFile) (*RoaringPositionBitmap, error)
- func (b *RoaringPositionBitmap) Cardinality() int64
- func (b *RoaringPositionBitmap) Contains(pos uint64) bool
- func (b *RoaringPositionBitmap) IsEmpty() bool
- func (b *RoaringPositionBitmap) KeepMaskBytes(length int64) []byte
- func (b *RoaringPositionBitmap) Or(other *RoaringPositionBitmap)
- func (b *RoaringPositionBitmap) Positions() iter.Seq[uint64]
- func (b *RoaringPositionBitmap) RunLengthEncode()
- func (b *RoaringPositionBitmap) Serialize(w io.Writer) error
- func (b *RoaringPositionBitmap) Set(pos uint64)
- func (b *RoaringPositionBitmap) SetRange(startInclusive, endExclusive uint64)
- type SpecResolver
Constants ¶
const ( // DVMagicNumber is the magic number for deletion vectors. // Spec bytes: D1 D3 39 64 (big-endian) = 0x6439D3D1 (little-endian uint32) DVMagicNumber uint32 = 0x6439D3D1 )
Variables ¶
var ErrInvalidDeletionVector = errors.New("invalid deletion vector")
Functions ¶
func SerializeDV ¶
func SerializeDV(bitmap *RoaringPositionBitmap) ([]byte, error)
SerializeDV produces the spec-format DV binary envelope from a bitmap:
- Length (4 bytes, big-endian): size of magic + bitmap data, excluding CRC-32
- Magic (4 bytes, little-endian): DVMagicNumber
- Bitmap (variable): roaring bitmap in Iceberg portable format
- CRC-32 (4 bytes, big-endian): checksum over magic + bitmap
The bitmap is run-length encoded in place before it is written, matching Java's BitmapPositionDeleteIndex.serialize.
Types ¶
type DVWriter ¶
type DVWriter struct {
// contains filtered or unexported fields
}
DVWriter accumulates deletion positions per data file and flushes them as a single Puffin file containing one deletion-vector-v1 blob per data file. The returned DataFile entries are ready for RowDelta.AddDeletes().
func NewDVWriter ¶
func NewDVWriter(fs iceio.WriteFileIO, specByID SpecResolver) *DVWriter
NewDVWriter creates a DVWriter backed by the given writable filesystem. specByID resolves PartitionSpec values at Flush time; typically the caller passes Metadata.PartitionSpecByID directly (wrapped to convert int → int32). The unpartitioned path can pass a resolver that returns iceberg.UnpartitionedSpec for id 0.
func (*DVWriter) Add ¶
func (w *DVWriter) Add(dataFilePath string, positions []int64, specID int32, partitionData map[int]any) error
Add accumulates positions to delete for a given data file. specID and partitionData come from the data file's own manifest entry (typically partitionContext.specID + partitionContext.partitionData on the caller side) and are propagated to the output DV manifest entry, so partitioned tables produce DV entries with the correct spec id and partition record. Positions are deduplicated via the underlying roaring bitmap.
partitionData keys must be partition field IDs (PartitionField.FieldID), not source column IDs. NewDataFileBuilder iterates spec.Fields() and re-keys by field name using these IDs; wrong keys silently produce an empty partition record on the output DataFile.
First Add for a given dataFilePath captures specID and partitionData on the entry; later Adds for the same path append positions only and ignore the new specID/partitionData args. This mirrors Java's BaseDVFileWriter, which stores partition metadata via computeIfAbsent on the same key. Callers must not pass conflicting partition values across Adds for the same data file — the writer trusts the first-Add values for the rest of the writer's life. Add validates that every position is non-negative; negative positions are rejected and returned as errors to avoid silently writing malformed deletion-vector metadata.
func (*DVWriter) Flush ¶
Flush writes one Puffin file containing one blob per data file, and returns manifest entries ready for RowDelta.AddDeletes(). Each output DataFile carries the partition spec and partition record of the data file it references, with the spec resolved via the writer's SpecResolver at Flush time. A specID with no corresponding spec is a programming error and is surfaced as an error here rather than producing a malformed DataFile.
The location parameter is the full path (including filename) for the Puffin file to create. The caller is responsible for generating a unique path within the table's metadata directory.
func (*DVWriter) Load ¶
func (w *DVWriter) Load(dataFilePath string, bitmap *RoaringPositionBitmap, specID int32, partitionData map[int]any)
Load seeds the writer with an already-written deletion vector for a data file so subsequent Add calls merge new positions into it. The spec permits at most one DV per data file per snapshot, so when a data file that already has a DV receives more deletes the old positions must be carried into the replacement DV (and the old DV superseded) rather than dropped.
Load is intended to be called before any Add for the same path: on the first call for a path it captures specID and partitionData exactly as Add would and unions the supplied bitmap into a fresh entry. If an entry already exists (because Add or Load ran first for this path), Load only unions the bitmap in and the earlier specID/partitionData are kept — the later values are ignored, mirroring Add's first-write-wins behavior. Callers must not pass conflicting partition values across calls for the same path.
bitmap must be non-nil; its set positions are copied and the caller retains ownership. (A nil bitmap would register a cardinality-0 entry, which is never useful — collectExistingDVs always supplies a non-nil bitmap read from the existing DV.)
type RoaringPositionBitmap ¶
type RoaringPositionBitmap struct {
// contains filtered or unexported fields
}
RoaringPositionBitmap supports 64-bit positions using a sparse map of 32-bit Roaring bitmaps. Positions are split into a 32-bit key (high bits) and 32-bit value (low bits).
Compatible with the Java Iceberg RoaringPositionBitmap serialization format.
func DeserializeDV ¶
func DeserializeDV(data []byte, expectedCardinality int64) (*RoaringPositionBitmap, error)
DeserializeDV parses a deletion vector blob and returns a bitmap of deleted positions.
The DV binary format is:
- Length (4 bytes, big-endian): size of magic + bitmap data, excluding CRC-32
- Magic (4 bytes, little-endian): must be 0x6439D3D1
- Bitmap (variable): roaring bitmap in Iceberg portable format
- CRC-32 (4 bytes, big-endian): checksum over magic + bitmap
If expectedCardinality >= 0, the bitmap's cardinality is validated against it.
func DeserializeRoaringPositionBitmap ¶
func DeserializeRoaringPositionBitmap(data []byte) (*RoaringPositionBitmap, error)
DeserializeRoaringPositionBitmap reads a bitmap from the Iceberg portable format. Format: [count] { [key][bitmap] } .....{[key_n][bitmap_n]}
func NewRoaringPositionBitmap ¶
func NewRoaringPositionBitmap() *RoaringPositionBitmap
NewRoaringPositionBitmap creates an empty bitmap.
func ReadDV ¶
ReadDV reads a deletion vector from a puffin file using the manifest entry metadata. ContentOffset and ContentSizeInBytes must be set on the DataFile (required by v3 spec).
The decoded bitmap's cardinality is cross-validated against two independent sources, so a truncated or partially-overwritten blob whose CRC still validates over the bytes that are present is rejected:
- The manifest entry's record_count (dvFile.Count()). This is field 103, a required non-nullable long, so it is always available — zero means an empty deletion vector, not "unknown". Java's BitmapPositionDeleteIndex validates against this value, so it is our primary expected cardinality.
- The puffin blob's spec-mandated `cardinality` property, when present.
When both sources are available they must agree; a disagreement (e.g. a stale manifest record_count against a freshly written blob) is a writer bug and fails fast. The bitmap is then validated against the manifest count.
ReadDV also requires the selected blob to be a deletion-vector blob whose referenced-data-file property matches the manifest. This is stricter than Java's DVUtil.readDV and PyIceberg, which read by offset and size without validating blob type or referenced-data-file, but prevents a valid Puffin blob for another data file from being applied here. A missing or empty referenced-data-file property is fatal because it cannot establish blob identity; a missing cardinality property is only warned about because the manifest record_count still bounds the decoded bitmap. The compression codec is also rejected because deletion-vector-v1 stores raw bytes here.
Blobs missing the spec-required cardinality property are still validated against the manifest record_count and accepted with a slog warning rather than rejected — the Go writer always emits the property, but third-party writers may not, and the per-byte CRC check in DeserializeDV still applies.
func (*RoaringPositionBitmap) Cardinality ¶
func (b *RoaringPositionBitmap) Cardinality() int64
Cardinality returns the total number of set positions.
func (*RoaringPositionBitmap) Contains ¶
func (b *RoaringPositionBitmap) Contains(pos uint64) bool
Contains checks if a position is set.
func (*RoaringPositionBitmap) IsEmpty ¶
func (b *RoaringPositionBitmap) IsEmpty() bool
IsEmpty returns true if no positions are set. Returns true both for the no-bucket case and for the (currently impossible-via-public-API) case where a bucket exists but its inner roaring bitmap has zero cardinality — the latter only matters if a future Remove-style method ever lets a bucket drop to empty without being deleted from the map.
func (*RoaringPositionBitmap) KeepMaskBytes ¶
func (b *RoaringPositionBitmap) KeepMaskBytes(length int64) []byte
KeepMaskBytes returns a bit-packed []byte of length ⌈length/8⌉ where bit i (LSB-first within a byte) is 1 iff position i is NOT in the bitmap. The layout matches Arrow Boolean buffer convention so callers can wrap the result via memory.NewBufferBytes / array.NewBoolean without re-packing.
length bounds the range of positions the caller cares about — typically the data file's row count. Bits past length-1 in the final byte are cleared. Positions in the bitmap that fall outside [0, length) are ignored, so a caller can safely pass a length smaller than the bitmap's max position (e.g. when the file row count is below a stale upper bound).
Bucket-key arithmetic is exact: each 32-bit bucket covers exactly 2^32 positions, so per-bucket bit offsets are 8-byte-aligned and the writer can pack inverted dense words straight in. bitutil.BitmapWordWriter handles host-endianness internally (PutNextWord LE-packs regardless of platform), so the helper is portable on any GOARCH.
func (*RoaringPositionBitmap) Or ¶
func (b *RoaringPositionBitmap) Or(other *RoaringPositionBitmap)
Or merges every position set in other into b. Buckets present only in other are cloned so the two bitmaps stay independent after the merge; shared buckets are unioned in place. A nil other is a no-op. Used to fold a previously written deletion vector into a new one when a data file that already has a DV receives more deletes.
func (*RoaringPositionBitmap) Positions ¶
func (b *RoaringPositionBitmap) Positions() iter.Seq[uint64]
Positions returns an iterator over every set position in ascending order. The positions yielded are the same 64-bit values passed to Set, mirroring Java's RoaringPositionBitmap#forEach.
The bitmap must not be modified while iterating: the key set is captured up front while each bucket is consulted lazily, so a concurrent or re-entrant Set/Or produces an inconsistent view (as with the rest of the type, which is not safe for concurrent use).
func (*RoaringPositionBitmap) RunLengthEncode ¶
func (b *RoaringPositionBitmap) RunLengthEncode()
RunLengthEncode re-encodes each bucket's containers as runs wherever runs are more compact, like Java's runLengthEncode. Membership and cardinality are unchanged.
func (*RoaringPositionBitmap) Serialize ¶
func (b *RoaringPositionBitmap) Serialize(w io.Writer) error
Serialize writes in the Iceberg portable format (little-endian):
- bitmap count (8 bytes, LE): number of non-empty bitmaps
- for each bitmap in ascending key order: key (4 bytes, LE) + roaring portable data
Only non-empty bitmaps are written, matching Java Iceberg behavior.
func (*RoaringPositionBitmap) Set ¶
func (b *RoaringPositionBitmap) Set(pos uint64)
Set marks a position in the bitmap.
func (*RoaringPositionBitmap) SetRange ¶
func (b *RoaringPositionBitmap) SetRange(startInclusive, endExclusive uint64)
SetRange marks every position in [startInclusive, endExclusive), like Java's setRange. An empty or inverted range is a no-op (no mutator on this type panics or reports errors).
type SpecResolver ¶
type SpecResolver func(specID int32) *iceberg.PartitionSpec
SpecResolver looks up a PartitionSpec by its id, mirroring Metadata.PartitionSpecByID. Returns nil if the id is not known.
Source Files