segment

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 19, 2025 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package segment provides functionality for reading and writing single segment files.

Index

Constants

View Source 
const DefaultPreAllocationSize = 64 * 1024 * 1024

DefaultPreAllocationSize is a segment size which should work well for most use cases.

Variables

View Source 
var (
	ReadEntryTotal = prometheus.NewCounter(
		prometheus.CounterOpts{
			Name: "wal_read_entry_total",
			Help: "Total number of entries read.",
		},
	)
	ReadEntryBytes = prometheus.NewCounter(
		prometheus.CounterOpts{
			Name: "wal_read_entry_bytes_total",
			Help: "Total number of bytes read (excluding metadata).",
		},
	)

	AppendEntryTotal = prometheus.NewCounter(
		prometheus.CounterOpts{
			Name: "wal_append_entry_total",
			Help: "Total number of entries appended.",
		},
	)
	AppendEntryBytes = prometheus.NewCounter(
		prometheus.CounterOpts{
			Name: "wal_append_entry_bytes_total",
			Help: "Total number of bytes appended (excluding metadata).",
		},
	)

	SyncTotal = prometheus.NewCounter(
		prometheus.CounterOpts{
			Name: "wal_sync_total",
			Help: "Total number of syncs executed.",
		},
	)

	SyncDuration = prometheus.NewHistogram(
		prometheus.HistogramOpts{
			Name:    "wal_sync_duration_seconds",
			Help:    "Duration of syncs in seconds.",
			Buckets: prometheus.ExponentialBuckets(0.0001, 2, 16),
		},
	)
)
View Source 
var ErrEntryNone = errors.New("this is no WAL entry")

Functions

func GetSegments 

func GetSegments(directory string) ([]uint64, error)

GetSegments returns a list of sequence numbers representing the start of the corresponding segment. The sequence numbers are sorted in ascending order.

func RegisterMetrics 

func RegisterMetrics(registerer prometheus.Registerer) error

RegisterMetrics registers all metrics collectors with the given prometheus registerer.

func SegmentFileName 

func SegmentFileName(sequenceNumber uint64) string

func SegmentFromSequenceNumber 

func SegmentFromSequenceNumber(directory string, sequenceNumber uint64) (uint64, error)

Types

type CreateSegmentConfig 

type CreateSegmentConfig struct {
	// PreAllocationSize is the number of bytes the new segment should be in size. Pre-allocation helps to avoid
	// fragmentation on disk and reduces the overhead for growing the file on each individual write.
	PreAllocationSize int64

	// EntryLengthEncoding is the encoding of entry lengths.
	EntryLengthEncoding encoding.EntryLengthEncoding

	// EntryChecksumType is the type of entry checksum to use.
	EntryChecksumType encoding.EntryChecksumType
}

CreateSegmentConfig is the configuration required for a call to CreateSegment.

type NewSegmentReaderConfig 

type NewSegmentReaderConfig struct {
	// Header is the segment file header.
	Header encoding.Header

	// Offset is the current position in bytes from the start of the file.
	Offset int64

	// NextSequenceNumber is the sequence number the next entry will receive.
	NextSequenceNumber uint64

	// FileSize is the total size in bytes of the segment file.
	FileSize int64
}

NewSegmentReaderConfig is the configuration required for a call to NewSegmentReader.

type NewSegmentWriterConfig 

type NewSegmentWriterConfig struct {
	// Header is the segment file header.
	Header encoding.Header

	// Offset is the current position in bytes from the start of the file.
	Offset int64

	// NextSequenceNumber is the sequence number the next entry will receive.
	NextSequenceNumber uint64
}

NewSegmentWriterConfig is the configuration required for a call to NewSegmentWriter.

type SegmentReader 

type SegmentReader struct {
	// contains filtered or unexported fields
}

SegmentReader provides functionality for reading from a single segment file.

Instances of SegmentReader are NOT safe to use concurrently. You need to provide external synchronization.

func NewSegmentReader 

func NewSegmentReader(file SegmentReaderFile, newSegmentReaderConfig NewSegmentReaderConfig) (*SegmentReader, error)

NewSegmentReader creates a SegmentReader from a file which is already open.

func OpenSegment 

func OpenSegment(directory string, firstSequenceNumber uint64) (*SegmentReader, error)

OpenSegment creates a new segment reader for the file path given as parameter.

To avoid resources leaking, the returned SegmentReader needs to be closed by calling Shutdown(). Returns an error if the file cannot be opened, read from or the header is malformed.

func (*SegmentReader) Close 

func (r *SegmentReader) Close() error

Close closes the file the SegmentReader is reading from.

func (*SegmentReader) Err 

func (r *SegmentReader) Err() error

Err returns the error for the last call to Next(). Returns ErrEntryNone when no entry could be read. This indicates either a corrupt entry or the end of the written entries in the pre-allocated segment file. Returns io.EOF when the end of the segment file was reached and no more data could be read. This error is still wrapped in ErrEntryNone but can be checked for separately.

func (*SegmentReader) FilePath 

func (r *SegmentReader) FilePath() string

FilePath returns the file path of the file this reader is reading from.

func (*SegmentReader) Header 

func (r *SegmentReader) Header() encoding.Header

Header returns the segment file header.

func (*SegmentReader) Next 

func (r *SegmentReader) Next() bool

Next reports if an entry has been successfully read. When it returns true, Err() returns nil and Value() contains valid data. When it returns false, Err() contains the error and Value() contains invalid data.

func (*SegmentReader) NextSequenceNumber 

func (r *SegmentReader) NextSequenceNumber() uint64

NextSequenceNumber returns the sequence number the next entry will receive.

func (*SegmentReader) Offset 

func (r *SegmentReader) Offset() int64

Offset returns the offset in bytes from the start of the file.

func (*SegmentReader) ToWriter 

func (r *SegmentReader) ToWriter() (*SegmentWriter, error)

ToWriter returns a SegmentWriter to append to the open segment file. You must have read all entries of the segment before you call this method. Otherwise, it will fail. After a call to ToWriter(), you cannot use the SegmentReader anymore.

func (*SegmentReader) Value 

func (r *SegmentReader) Value() SegmentReaderValue

Value returns the last entry read from the segment file. The values are only valid after the first call to Next() and while Err() is nil.

type SegmentReaderFile 

type SegmentReaderFile interface {
	io.ReadCloser
	io.Seeker
	Name() string
}

SegmentReaderFile is an interface which needs to be implemented by the file to read from.

type SegmentReaderValue 

type SegmentReaderValue struct {
	// The sequence number of the entry.
	SequenceNumber uint64

	// The data of the entry.
	Data []byte
}

SegmentReaderValue is the value returned by the SegmentReader.

type SegmentWriter 

type SegmentWriter struct {
	// contains filtered or unexported fields
}

SegmentWriter provides functionality for writing to a single segment file.

Instances of SegmentWriter are NOT safe to use concurrently. You need to provide external synchronization.

func CreateSegment 

func CreateSegment(directory string, firstSequenceNumber uint64, createSegmentConfig CreateSegmentConfig) (*SegmentWriter, error)

CreateSegment creates a new segment file in the given directory. It will create the new file with the file extension ".new" appended to the file name and rename it after the header has been written to. This ensures that the new segment file is only visible in the directory when the header was correctly written and flushed to stable storage.

directory is the directory all segment files are located in. firstSequenceNumber is used for deriving the file name and for storing it in the segment header. createSegmentConfig provides more configuration for the new segment.

func NewSegmentWriter 

func NewSegmentWriter(file SegmentWriterFile, newSegmentWriterConfig NewSegmentWriterConfig) (*SegmentWriter, error)

NewSegmentWriter creates a SegmentWriter from a file which is already open.

func (*SegmentWriter) AppendEntry 

func (w *SegmentWriter) AppendEntry(data []byte) (uint64, error)

AppendEntry adds the given entry to the segment.

func (*SegmentWriter) Close 

func (w *SegmentWriter) Close() error

Close flushes all pending changes to disk and closes the file.

func (*SegmentWriter) FilePath 

func (w *SegmentWriter) FilePath() string

FilePath returns the file path of the file this writer is writing to.

func (*SegmentWriter) Header 

func (w *SegmentWriter) Header() encoding.Header

Header returns the segment file header.

func (*SegmentWriter) NextSequenceNumber 

func (w *SegmentWriter) NextSequenceNumber() uint64

NextSequenceNumber returns the sequence number the next entry will receive.

func (*SegmentWriter) Offset 

func (w *SegmentWriter) Offset() int64

Offset returns the offset in bytes from the start of the file.

func (*SegmentWriter) Sync 

func (w *SegmentWriter) Sync() error

Sync flushes the content of the segment to stable storage.

func (*SegmentWriter) Truncate 

func (w *SegmentWriter) Truncate() error

Truncate sets the size of the segment to the current position. This is needed when the segment has been pre-allocated but is rolled over before reaching the pre-allocated size. Without truncation the segment reader would not see the end of the file.

type SegmentWriterFile 

type SegmentWriterFile interface {
	io.WriteCloser
	Name() string
	Sync() error
	Truncate(size int64) error
}

SegmentWriterFile is an interface which needs to be implemented by the file to write to.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.