segment

Documentation

Overview

Package segment implements a custom binary segment file format (.spdx) for persisting inverted-index data to disk. Each segment has a fixed-size header, a postings region, a JSON dictionary, and a CRC32 footer. The Writer creates new segments atomically, and the Reader provides random-access search over them.

Index

• Constants
• type DictEntry
• type Reader
  • func OpenReader(path string) (*Reader, error)
  • func (r *Reader) Close() error
  • func (r *Reader) DocCount() uint32
  • func (r *Reader) Name() string
  • func (r *Reader) Search(term string) (index.PostingList, error)
  • func (r *Reader) Terms() int
• type SegmentHeader
• type Writer
  • func NewWriter(dataDir string) *Writer
  • func (w *Writer) Write(entries []index.TermEntry) (string, error)

Constants

const (
	MagicBytes    uint32 = 0x53504458
	FormatVersion uint32 = 1
	HeaderSize    int    = 64
	FooterSize    int    = 32
)

MagicBytes identifies a valid .spdx segment file.

Types

type DictEntry

type DictEntry struct {
	Term       string `json:"t"`
	PostOffset int64  `json:"o"`
	PostLen    int    `json:"l"`
	DocFreq    int    `json:"d"`
}

DictEntry maps a term to its postings offset, length, and document frequency in the segment file.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader provides read-only access to a single .spdx segment file. It memory-maps the dictionary on open and performs random-access reads for posting lists.

func OpenReader

func OpenReader(path string) (*Reader, error)

OpenReader opens an existing segment file, validates the magic bytes, and loads the term dictionary into memory.

func (*Reader) Close

func (r *Reader) Close() error

Close releases the underlying file handle.

func (*Reader) DocCount

func (r *Reader) DocCount() uint32

DocCount returns the number of unique documents stored in this segment.

func (*Reader) Name

func (r *Reader) Name() string

Name returns the base file name (e.g. "seg_123456.spdx") of this segment.

func (*Reader) Search

func (r *Reader) Search(term string) (index.PostingList, error)

Search performs a binary search over the term dictionary and reads the matching PostingList from disk.

func (*Reader) Terms

func (r *Reader) Terms() int

Terms returns the number of unique terms stored in this segment.

type SegmentHeader

type SegmentHeader struct {
	Magic      uint32
	Version    uint32
	TermCount  uint32
	DocCount   uint32
	CreatedAt  int64
	DictOffset int64
	DictSize   int64
	PostOffset int64
	PostSize   int64
}

SegmentHeader is the 64-byte header written at the start of every segment.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer serialises TermEntry slices into new .spdx segment files.

func NewWriter

func NewWriter(dataDir string) *Writer

NewWriter creates a Writer that writes segments into the given directory.

func (*Writer) Write

func (w *Writer) Write(entries []index.TermEntry) (string, error)

Write atomically creates a new segment file containing the given term entries. It writes to a .tmp file first and renames on success.