commitlog

package
v1.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 7, 2022 License: Apache-2.0 Imports: 35 Imported by: 8

README

Overview

The commitlog package contains all the code for reading and writing commitlog files.

Implementation

Writer

The commitlog writer consists of two main components:

  1. The CommitLog itself which contains all of the public APIs as well as the logic for managing concurrency, rotating files, and inspecting the status of the commitlog (via the ActiveLogs and QueueLength APIs).
  2. The CommitLogWriter which contains all of the logic for opening/closing files and writing bytes to disk.
Commitlog

At a high-level, the CommitLog handles writes by inserting them into a queue (buffered channel) in batches and then dequeing the batches in a single-threaded manner and writing all of the individual writes for a batch into the commitlog file one at a time.

Synchronization

The primary synchronization that happens in the commitlog is via the queue (buffered channel). Many goroutines will insert pending writes into the queue concurrently via the Write() and WriteBatch() APIs, however, only a single goroutine will pull items off the queue and write them to disk.

┌──────────────────────────┐
│Goroutine 1: WriteBatch() ├─────┐                                                         ┌───────────────────────────────────┐          ┌───────────────────────────────────┐
└──────────────────────────┘     │                                                         │                                   │          │                                   │
                                 │                                                         │                                   │          │                                   │
                                 │                                                         │                                   │          │                                   │
┌──────────────────────────┐     │       ┌─────────────────────────────────────┐           │             Commitlog             │          │              Writer               │
│Goroutine 2: WriteBatch() │─────┼──────▶│           Commitlog Queue           ├──────────▶│                                   │─────────▶│                                   │
└──────────────────────────┘     │       └─────────────────────────────────────┘           │ Single-threaded Writer Goroutine  │          │           Write to disk           │
                                 │                                                         │                                   │          │                                   │
                                 │                                                         │                                   │          │                                   │
┌──────────────────────────┐     │                                                         │                                   │          │                                   │
│Goroutine 3: WriteBatch() ├─────┘                                                         └───────────────────────────────────┘          └───────────────────────────────────┘
└──────────────────────────┘

The monopic file that was used to generate the diagram above is located at docs/code_assets/commitlog/queue.monopic.

Since there is only one goroutine pulling items off of the queue, any state that it alone manages can remain unsynchronized since no other goroutines will interact with it.

In addition to the queue, the commitlog has two other forms of synchronization:

  1. The closedState lock which is an RWLock. An RLock is held for the duration of any operation during which the commitlog must remain open.
  2. The flushState lock. The scope of this lock is very narrow as its only used to protect access to the lastFlushAt field.
Rotating Files

Rotating commitlog files is initiated by the RotateLogs() API so that callers can control when this occurs. The CommitLog itself will never rotate files on its own without the RotateLogs() API being called.

The commitlog files are not rotated immediately when the RotateLogs() method is called because that would require a lot of complicated and expensive synchronization with the CommitLogWriter goroutine. Instead, a rotateLogsEventType is pushed into the queue and when the single-threaded writer goroutine pulls this event off of the channel it will rotate the commitlog files (since it has exclusive access to them) and then invoke a callback function which notifies the RotateLogs() method call (which has been blocked this whole time) to complete and return success to the caller.

While the CommitLog only writes to a single file at once, it maintains two open writers at all times so that they can be "hot-swapped" when the commitlog files need to be rotated. This allows the single-threaded writer goroutine to continue uninterrupted by syscalls and I/O during rotation events which in turn prevents the queue from backing up. Otherwise, rotation events could block the writer for so long (while it waited for a new file to be created) that it caused the queue to back up significantly.

When a rotation event occurs, instead of waiting for a new file to be opened, the CommitLog writer goroutine will swap the primary and secondary CommitLogWriter such that the secondary CommitLogWriter (which has an empty file) becomes the primary and vice versa. This allows the CommitLog writer goroutine to continue writing uninterrupted.

In the meantime, a goroutine is started in the background that is responsible for resetting the now secondary (formerly primary) CommitLogWriter by closing it (which will flush any pending / buffered writes to disk) and re-opening it (which will create a new empty commitlog file in anticipation of the next rotation event).

Finally, the next time the CommitLog attempts to rotate its commitlogs it will need to use the associated sync.WaitGroup to ensure that the previously spawned background goroutine has completed resetting the secondary CommitLogWriter before it attempts a new hot-swap.

Handling Errors

The current implementation will panic if any I/O errors are ever encountered while writing bytes to disk or opening/closing files. In the future a "commitlog failure policy" similar to Cassandra's "stop" may be introduced.

Testing

The commitlog package is tested via:

  1. Standard unit tests
  2. Property tests
  3. Concurrency tests

File Format

See /docs/architecture/m3db/commitlogs.md.

Documentation

Overview

Package commitlog is a generated GoMock package.

Index

Constants

View Source 
const (

	// MaximumQueueSizeQueueChannelSizeRatio is the maximum ratio between the
	// backlog queue size and backlog queue channel size.
	MaximumQueueSizeQueueChannelSizeRatio = 8.0
)

Variables

View Source 
var (
	// ErrCommitLogQueueFull is raised when trying to write to the commit log
	// when the queue is full
	ErrCommitLogQueueFull = errors.New("commit log queue is full")
)

Functions

func NewIterator 

func NewIterator(iterOpts IteratorOpts) (iter Iterator, corruptFiles []ErrorWithPath, err error)

NewIterator creates a new commit log iterator

func NextFile added in v0.7.0 

func NextFile(opts Options) (string, int, error)

NextFile returns the next commitlog file.

func ReadLogInfo 

func ReadLogInfo(filePath string, opts Options) (int64, error)

ReadLogInfo reads the commit log info out of a commitlog file

Types

type CommitLog 

type CommitLog interface {
	// Open the commit log
	Open() error

	// Write will write an entry in the commit log for a given series
	Write(
		ctx context.Context,
		series ts.Series,
		datapoint ts.Datapoint,
		unit xtime.Unit,
		annotation ts.Annotation,
	) error

	// WriteBatch is the same as Write, but in batch.
	WriteBatch(
		ctx context.Context,
		writes writes.WriteBatch,
	) error

	// Close the commit log
	Close() error

	// ActiveLogs returns a slice of the active commitlogs.
	ActiveLogs() (persist.CommitLogFiles, error)

	// RotateLogs rotates the commitlog and returns the File that represents
	// the new commitlog file.
	RotateLogs() (persist.CommitLogFile, error)

	// QueueLength returns the number of writes that are currently in the commitlog
	// queue.
	QueueLength() int64
}

CommitLog provides a synchronized commit log

func NewCommitLog 

func NewCommitLog(opts Options) (CommitLog, error)

NewCommitLog creates a new commit log

type ErrorWithPath added in v0.4.8 

type ErrorWithPath struct {
	// contains filtered or unexported fields
}

ErrorWithPath is an error that includes the path of the file that had the error.

func Files added in v0.4.0 

func Files(opts Options) (persist.CommitLogFiles, []ErrorWithPath, error)

Files returns a slice of all available commit log files on disk along with their associated metadata.

func NewErrorWithPath added in v0.4.8 

func NewErrorWithPath(err error, path string) ErrorWithPath

NewErrorWithPath creates a new ErrorWithPath.

func (ErrorWithPath) Error added in v0.4.8 

func (e ErrorWithPath) Error() string

Error returns the error.

func (ErrorWithPath) Path added in v0.4.8 

func (e ErrorWithPath) Path() string

Path returns the path of the file that the error is associated with.

type FailureCallback added in v1.4.2 

type FailureCallback func(err error) bool

FailureCallback is used in the FailureStrategyCallback failure mode. If this function returns false, the error will be treated as fatal. This function MUST exit quickly

type FailureStrategy added in v1.4.2 

type FailureStrategy int

FailureStrategy describes the commit log failure strategy 

const (
	// FailureStrategyPanic describes the failure strategy that causes
	// the commit log to panic on I/O errors. This provides the only correct
	// behavior for M3DB nodes.
	FailureStrategyPanic FailureStrategy = iota

	// FailureStrategyCallback describes the failure strategy that calls the configured
	// FailureCallback. The return value of that callback determines whether to panic
	// or ignore the passed error.
	FailureStrategyCallback

	// FailureStrategyIgnore describes the failure strategy that causes the
	// commit log to swallow I/O errors.
	FailureStrategyIgnore
)

type FileFilterInfo added in v0.9.0 

type FileFilterInfo struct {
	// If isCorrupt is true then File will contain a valid CommitLogFile, otherwise
	// ErrorWithPath will contain an error and the path of the corrupt file.
	File      persist.CommitLogFile
	Err       ErrorWithPath
	IsCorrupt bool
}

FileFilterInfo contains information about a commitog file that can be used to determine whether the iterator should filter it out or not.

type FileFilterPredicate 

type FileFilterPredicate func(f FileFilterInfo) bool

FileFilterPredicate is a predicate that allows the caller to determine which commitlogs the iterator should read from.

func ReadAllPredicate 

func ReadAllPredicate() FileFilterPredicate

ReadAllPredicate can be passed as the ReadCommitLogPredicate for callers that want a convenient way to read all the commitlogs

type Iterator 

type Iterator interface {
	// Next returns whether the iterator has the next value.
	Next() bool

	// Current returns the current commit log entry.
	Current() LogEntry

	// Err returns an error if an error occurred.
	Err() error

	// Close the iterator.
	Close()
}

Iterator provides an iterator for commit logs.

type IteratorOpts 

type IteratorOpts struct {
	CommitLogOptions    Options
	FileFilterPredicate FileFilterPredicate
	// ReturnMetadataAsRef will return all series metadata such as ID,
	// tags and namespace as a reference instead of returning pooled
	// or allocated byte/string/ID references.
	// Useful if caller does not hold onto the result between calls to
	// the next read log entry and wants to avoid allocations and pool
	// contention.
	// Note: Series metadata will only be set on the result of a log
	// entry read if the series is read for the first time for the
	// combined tuple of FileReadID and SeriesUniqueIndex returned by
	// the LogEntryMetadata. EncodedTags will also be returned
	// instead of Tags on the series metadata.
	ReturnMetadataAsRef bool
}

IteratorOpts is a struct that contains coptions for the Iterator.

type LogEntry added in v0.15.0 

type LogEntry struct {
	Series    ts.Series
	Datapoint ts.Datapoint
	Unit      xtime.Unit
	// Annotation gets invalidates on every read.
	Annotation ts.Annotation
	Metadata   LogEntryMetadata
}

LogEntry is a commit log entry being read.

type LogEntryMetadata added in v0.15.0 

type LogEntryMetadata struct {
	// FileReadID is a unique index for the current commit log
	// file that is being read (only unique per-process).
	FileReadID uint64
	// SeriesUniqueIndex is the series unique index relative to the
	// current commit log file being read.
	SeriesUniqueIndex uint64
}

LogEntryMetadata is a set of metadata about a commit log entry being read.

type MockCommitLog 

type MockCommitLog struct {
	// contains filtered or unexported fields
}

MockCommitLog is a mock of CommitLog interface.

func NewMockCommitLog 

func NewMockCommitLog(ctrl *gomock.Controller) *MockCommitLog

NewMockCommitLog creates a new mock instance.

func (*MockCommitLog) ActiveLogs added in v0.4.8 

func (m *MockCommitLog) ActiveLogs() (persist.CommitLogFiles, error)

ActiveLogs mocks base method.

func (*MockCommitLog) Close 

func (m *MockCommitLog) Close() error

Close mocks base method.

func (*MockCommitLog) EXPECT 

func (m *MockCommitLog) EXPECT() *MockCommitLogMockRecorder

EXPECT returns an object that allows the caller to indicate expected use.

func (*MockCommitLog) Open 

func (m *MockCommitLog) Open() error

Open mocks base method.

func (*MockCommitLog) QueueLength added in v0.7.3 

func (m *MockCommitLog) QueueLength() int64

QueueLength mocks base method.

func (*MockCommitLog) RotateLogs added in v0.4.8 

func (m *MockCommitLog) RotateLogs() (persist.CommitLogFile, error)

RotateLogs mocks base method.

func (*MockCommitLog) Write 

func (m *MockCommitLog) Write(ctx context.Context, series ts.Series, datapoint ts.Datapoint, unit time0.Unit, annotation ts.Annotation) error

Write mocks base method.

func (*MockCommitLog) WriteBatch added in v0.4.8 

func (m *MockCommitLog) WriteBatch(ctx context.Context, writes writes.WriteBatch) error

WriteBatch mocks base method.

type MockCommitLogMockRecorder 

type MockCommitLogMockRecorder struct {
	// contains filtered or unexported fields
}

MockCommitLogMockRecorder is the mock recorder for MockCommitLog.

func (*MockCommitLogMockRecorder) ActiveLogs added in v0.4.8 

func (mr *MockCommitLogMockRecorder) ActiveLogs() *gomock.Call

ActiveLogs indicates an expected call of ActiveLogs.

func (*MockCommitLogMockRecorder) Close 

func (mr *MockCommitLogMockRecorder) Close() *gomock.Call

Close indicates an expected call of Close.

func (*MockCommitLogMockRecorder) Open 

func (mr *MockCommitLogMockRecorder) Open() *gomock.Call

Open indicates an expected call of Open.

func (*MockCommitLogMockRecorder) QueueLength added in v0.7.3 

func (mr *MockCommitLogMockRecorder) QueueLength() *gomock.Call

QueueLength indicates an expected call of QueueLength.

func (*MockCommitLogMockRecorder) RotateLogs added in v0.4.8 

func (mr *MockCommitLogMockRecorder) RotateLogs() *gomock.Call

RotateLogs indicates an expected call of RotateLogs.

func (*MockCommitLogMockRecorder) Write 

func (mr *MockCommitLogMockRecorder) Write(ctx, series, datapoint, unit, annotation interface{}) *gomock.Call

Write indicates an expected call of Write.

func (*MockCommitLogMockRecorder) WriteBatch added in v0.4.8 

func (mr *MockCommitLogMockRecorder) WriteBatch(ctx, writes interface{}) *gomock.Call

WriteBatch indicates an expected call of WriteBatch.

type MockIterator 

type MockIterator struct {
	// contains filtered or unexported fields
}

MockIterator is a mock of Iterator interface.

func NewMockIterator 

func NewMockIterator(ctrl *gomock.Controller) *MockIterator

NewMockIterator creates a new mock instance.

func (*MockIterator) Close 

func (m *MockIterator) Close()

Close mocks base method.

func (*MockIterator) Current 

func (m *MockIterator) Current() LogEntry

Current mocks base method.

func (*MockIterator) EXPECT 

func (m *MockIterator) EXPECT() *MockIteratorMockRecorder

EXPECT returns an object that allows the caller to indicate expected use.

func (*MockIterator) Err 

func (m *MockIterator) Err() error

Err mocks base method.

func (*MockIterator) Next 

func (m *MockIterator) Next() bool

Next mocks base method.

type MockIteratorMockRecorder 

type MockIteratorMockRecorder struct {
	// contains filtered or unexported fields
}

MockIteratorMockRecorder is the mock recorder for MockIterator.

func (*MockIteratorMockRecorder) Close 

func (mr *MockIteratorMockRecorder) Close() *gomock.Call

Close indicates an expected call of Close.

func (*MockIteratorMockRecorder) Current 

func (mr *MockIteratorMockRecorder) Current() *gomock.Call

Current indicates an expected call of Current.

func (*MockIteratorMockRecorder) Err 

func (mr *MockIteratorMockRecorder) Err() *gomock.Call

Err indicates an expected call of Err.

func (*MockIteratorMockRecorder) Next 

func (mr *MockIteratorMockRecorder) Next() *gomock.Call

Next indicates an expected call of Next.

type MockOptions 

type MockOptions struct {
	// contains filtered or unexported fields
}

MockOptions is a mock of Options interface.

func NewMockOptions 

func NewMockOptions(ctrl *gomock.Controller) *MockOptions

NewMockOptions creates a new mock instance.

func (*MockOptions) BacklogQueueChannelSize added in v0.4.8 

func (m *MockOptions) BacklogQueueChannelSize() int

BacklogQueueChannelSize mocks base method.

func (*MockOptions) BacklogQueueSize 

func (m *MockOptions) BacklogQueueSize() int

BacklogQueueSize mocks base method.

func (*MockOptions) BlockSize 

func (m *MockOptions) BlockSize() time.Duration

BlockSize mocks base method.

func (*MockOptions) BytesPool 

func (m *MockOptions) BytesPool() pool.CheckedBytesPool

BytesPool mocks base method.

func (*MockOptions) ClockOptions 

func (m *MockOptions) ClockOptions() clock.Options

ClockOptions mocks base method.

func (*MockOptions) EXPECT 

func (m *MockOptions) EXPECT() *MockOptionsMockRecorder

EXPECT returns an object that allows the caller to indicate expected use.

func (*MockOptions) FailureCallback added in v1.4.2 

func (m *MockOptions) FailureCallback() FailureCallback

FailureCallback mocks base method.

func (*MockOptions) FailureStrategy added in v1.4.2 

func (m *MockOptions) FailureStrategy() FailureStrategy

FailureStrategy mocks base method.

func (*MockOptions) FilesystemOptions 

func (m *MockOptions) FilesystemOptions() fs.Options

FilesystemOptions mocks base method.

func (*MockOptions) FlushInterval 

func (m *MockOptions) FlushInterval() time.Duration

FlushInterval mocks base method.

func (*MockOptions) FlushSize 

func (m *MockOptions) FlushSize() int

FlushSize mocks base method.

func (*MockOptions) IdentifierPool 

func (m *MockOptions) IdentifierPool() ident.Pool

IdentifierPool mocks base method.

func (*MockOptions) InstrumentOptions 

func (m *MockOptions) InstrumentOptions() instrument.Options

InstrumentOptions mocks base method.

func (*MockOptions) ReadConcurrency 

func (m *MockOptions) ReadConcurrency() int

ReadConcurrency mocks base method.

func (*MockOptions) SetBacklogQueueChannelSize added in v0.4.8 

func (m *MockOptions) SetBacklogQueueChannelSize(value int) Options

SetBacklogQueueChannelSize mocks base method.

func (*MockOptions) SetBacklogQueueSize 

func (m *MockOptions) SetBacklogQueueSize(value int) Options

SetBacklogQueueSize mocks base method.

func (*MockOptions) SetBlockSize 

func (m *MockOptions) SetBlockSize(value time.Duration) Options

SetBlockSize mocks base method.

func (*MockOptions) SetBytesPool 

func (m *MockOptions) SetBytesPool(value pool.CheckedBytesPool) Options

SetBytesPool mocks base method.

func (*MockOptions) SetClockOptions 

func (m *MockOptions) SetClockOptions(value clock.Options) Options

SetClockOptions mocks base method.

func (*MockOptions) SetFailureCallback added in v1.4.2 

func (m *MockOptions) SetFailureCallback(value FailureCallback) Options

SetFailureCallback mocks base method.

func (*MockOptions) SetFailureStrategy added in v1.4.2 

func (m *MockOptions) SetFailureStrategy(value FailureStrategy) Options

SetFailureStrategy mocks base method.

func (*MockOptions) SetFilesystemOptions 

func (m *MockOptions) SetFilesystemOptions(value fs.Options) Options

SetFilesystemOptions mocks base method.

func (*MockOptions) SetFlushInterval 

func (m *MockOptions) SetFlushInterval(value time.Duration) Options

SetFlushInterval mocks base method.

func (*MockOptions) SetFlushSize 

func (m *MockOptions) SetFlushSize(value int) Options

SetFlushSize mocks base method.

func (*MockOptions) SetIdentifierPool 

func (m *MockOptions) SetIdentifierPool(value ident.Pool) Options

SetIdentifierPool mocks base method.

func (*MockOptions) SetInstrumentOptions 

func (m *MockOptions) SetInstrumentOptions(value instrument.Options) Options

SetInstrumentOptions mocks base method.

func (*MockOptions) SetReadConcurrency 

func (m *MockOptions) SetReadConcurrency(concurrency int) Options

SetReadConcurrency mocks base method.

func (*MockOptions) SetStrategy 

func (m *MockOptions) SetStrategy(value Strategy) Options

SetStrategy mocks base method.

func (*MockOptions) Strategy 

func (m *MockOptions) Strategy() Strategy

Strategy mocks base method.

func (*MockOptions) Validate 

func (m *MockOptions) Validate() error

Validate mocks base method.

type MockOptionsMockRecorder 

type MockOptionsMockRecorder struct {
	// contains filtered or unexported fields
}

MockOptionsMockRecorder is the mock recorder for MockOptions.

func (*MockOptionsMockRecorder) BacklogQueueChannelSize added in v0.4.8 

func (mr *MockOptionsMockRecorder) BacklogQueueChannelSize() *gomock.Call

BacklogQueueChannelSize indicates an expected call of BacklogQueueChannelSize.

func (*MockOptionsMockRecorder) BacklogQueueSize 

func (mr *MockOptionsMockRecorder) BacklogQueueSize() *gomock.Call

BacklogQueueSize indicates an expected call of BacklogQueueSize.

func (*MockOptionsMockRecorder) BlockSize 

func (mr *MockOptionsMockRecorder) BlockSize() *gomock.Call

BlockSize indicates an expected call of BlockSize.

func (*MockOptionsMockRecorder) BytesPool 

func (mr *MockOptionsMockRecorder) BytesPool() *gomock.Call

BytesPool indicates an expected call of BytesPool.

func (*MockOptionsMockRecorder) ClockOptions 

func (mr *MockOptionsMockRecorder) ClockOptions() *gomock.Call

ClockOptions indicates an expected call of ClockOptions.

func (*MockOptionsMockRecorder) FailureCallback added in v1.4.2 

func (mr *MockOptionsMockRecorder) FailureCallback() *gomock.Call

FailureCallback indicates an expected call of FailureCallback.

func (*MockOptionsMockRecorder) FailureStrategy added in v1.4.2 

func (mr *MockOptionsMockRecorder) FailureStrategy() *gomock.Call

FailureStrategy indicates an expected call of FailureStrategy.

func (*MockOptionsMockRecorder) FilesystemOptions 

func (mr *MockOptionsMockRecorder) FilesystemOptions() *gomock.Call

FilesystemOptions indicates an expected call of FilesystemOptions.

func (*MockOptionsMockRecorder) FlushInterval 

func (mr *MockOptionsMockRecorder) FlushInterval() *gomock.Call

FlushInterval indicates an expected call of FlushInterval.

func (*MockOptionsMockRecorder) FlushSize 

func (mr *MockOptionsMockRecorder) FlushSize() *gomock.Call

FlushSize indicates an expected call of FlushSize.

func (*MockOptionsMockRecorder) IdentifierPool 

func (mr *MockOptionsMockRecorder) IdentifierPool() *gomock.Call

IdentifierPool indicates an expected call of IdentifierPool.

func (*MockOptionsMockRecorder) InstrumentOptions 

func (mr *MockOptionsMockRecorder) InstrumentOptions() *gomock.Call

InstrumentOptions indicates an expected call of InstrumentOptions.

func (*MockOptionsMockRecorder) ReadConcurrency 

func (mr *MockOptionsMockRecorder) ReadConcurrency() *gomock.Call

ReadConcurrency indicates an expected call of ReadConcurrency.

func (*MockOptionsMockRecorder) SetBacklogQueueChannelSize added in v0.4.8 

func (mr *MockOptionsMockRecorder) SetBacklogQueueChannelSize(value interface{}) *gomock.Call

SetBacklogQueueChannelSize indicates an expected call of SetBacklogQueueChannelSize.

func (*MockOptionsMockRecorder) SetBacklogQueueSize 

func (mr *MockOptionsMockRecorder) SetBacklogQueueSize(value interface{}) *gomock.Call

SetBacklogQueueSize indicates an expected call of SetBacklogQueueSize.

func (*MockOptionsMockRecorder) SetBlockSize 

func (mr *MockOptionsMockRecorder) SetBlockSize(value interface{}) *gomock.Call

SetBlockSize indicates an expected call of SetBlockSize.

func (*MockOptionsMockRecorder) SetBytesPool 

func (mr *MockOptionsMockRecorder) SetBytesPool(value interface{}) *gomock.Call

SetBytesPool indicates an expected call of SetBytesPool.

func (*MockOptionsMockRecorder) SetClockOptions 

func (mr *MockOptionsMockRecorder) SetClockOptions(value interface{}) *gomock.Call

SetClockOptions indicates an expected call of SetClockOptions.

func (*MockOptionsMockRecorder) SetFailureCallback added in v1.4.2 

func (mr *MockOptionsMockRecorder) SetFailureCallback(value interface{}) *gomock.Call

SetFailureCallback indicates an expected call of SetFailureCallback.

func (*MockOptionsMockRecorder) SetFailureStrategy added in v1.4.2 

func (mr *MockOptionsMockRecorder) SetFailureStrategy(value interface{}) *gomock.Call

SetFailureStrategy indicates an expected call of SetFailureStrategy.

func (*MockOptionsMockRecorder) SetFilesystemOptions 

func (mr *MockOptionsMockRecorder) SetFilesystemOptions(value interface{}) *gomock.Call

SetFilesystemOptions indicates an expected call of SetFilesystemOptions.

func (*MockOptionsMockRecorder) SetFlushInterval 

func (mr *MockOptionsMockRecorder) SetFlushInterval(value interface{}) *gomock.Call

SetFlushInterval indicates an expected call of SetFlushInterval.

func (*MockOptionsMockRecorder) SetFlushSize 

func (mr *MockOptionsMockRecorder) SetFlushSize(value interface{}) *gomock.Call

SetFlushSize indicates an expected call of SetFlushSize.

func (*MockOptionsMockRecorder) SetIdentifierPool 

func (mr *MockOptionsMockRecorder) SetIdentifierPool(value interface{}) *gomock.Call

SetIdentifierPool indicates an expected call of SetIdentifierPool.

func (*MockOptionsMockRecorder) SetInstrumentOptions 

func (mr *MockOptionsMockRecorder) SetInstrumentOptions(value interface{}) *gomock.Call

SetInstrumentOptions indicates an expected call of SetInstrumentOptions.

func (*MockOptionsMockRecorder) SetReadConcurrency 

func (mr *MockOptionsMockRecorder) SetReadConcurrency(concurrency interface{}) *gomock.Call

SetReadConcurrency indicates an expected call of SetReadConcurrency.

func (*MockOptionsMockRecorder) SetStrategy 

func (mr *MockOptionsMockRecorder) SetStrategy(value interface{}) *gomock.Call

SetStrategy indicates an expected call of SetStrategy.

func (*MockOptionsMockRecorder) Strategy 

func (mr *MockOptionsMockRecorder) Strategy() *gomock.Call

Strategy indicates an expected call of Strategy.

func (*MockOptionsMockRecorder) Validate 

func (mr *MockOptionsMockRecorder) Validate() *gomock.Call

Validate indicates an expected call of Validate.

type OptionSetter added in v1.5.0 

type OptionSetter func(o *optionsInput)

OptionSetter is a function that modifies the behavior of NewOptions

func WithBytesPoolOptions added in v1.5.0 

func WithBytesPoolOptions(o pool.ObjectPoolOptions) OptionSetter

WithBytesPoolOptions is an OptionsSetter that provides options to BytesPool

func WithFileSystemOptions added in v1.5.0 

func WithFileSystemOptions(o fs.Options) OptionSetter

WithFileSystemOptions is an OptionsSetter that provides custom fs.Options Passing nil will be equivalent to calling Options.SetFilesystemOptions(nil)

func WithIdentPoolOptions added in v1.5.0 

func WithIdentPoolOptions(o ident.PoolOptions) OptionSetter

WithIdentPoolOptions is an OptionsSetter that provides options to the IdentifierPool

type Options 

type Options interface {
	// Validate validates the Options.
	Validate() error

	// SetClockOptions sets the clock options.
	SetClockOptions(value clock.Options) Options

	// ClockOptions returns the clock options.
	ClockOptions() clock.Options

	// SetInstrumentOptions sets the instrumentation options.
	SetInstrumentOptions(value instrument.Options) Options

	// InstrumentOptions returns the instrumentation options,
	InstrumentOptions() instrument.Options

	// SetBlockSize sets the block size.
	SetBlockSize(value time.Duration) Options

	// BlockSize returns the block size.
	BlockSize() time.Duration

	// SetFilesystemOptions sets the filesystem options.
	SetFilesystemOptions(value fs.Options) Options

	// FilesystemOptions returns the filesystem options.
	FilesystemOptions() fs.Options

	// SetFlushSize sets the flush size.
	SetFlushSize(value int) Options

	// FlushSize returns the flush size.
	FlushSize() int

	// SetStrategy sets the strategy.
	SetStrategy(value Strategy) Options

	// Strategy returns the strategy.
	Strategy() Strategy

	// SetFlushInterval sets the flush interval.
	SetFlushInterval(value time.Duration) Options

	// FlushInterval returns the flush interval.
	FlushInterval() time.Duration

	// SetBacklogQueueSize sets the backlog queue size.
	SetBacklogQueueSize(value int) Options

	// BacklogQueueSize returns the backlog queue size.
	BacklogQueueSize() int

	// SetBacklogQueueChannelSize sets the size of the Golang channel
	// that backs the queue.
	SetBacklogQueueChannelSize(value int) Options

	// BacklogQueueChannelSize returns the size of the Golang channel
	// that backs the queue.
	BacklogQueueChannelSize() int

	// SetBytesPool sets the checked bytes pool.
	SetBytesPool(value pool.CheckedBytesPool) Options

	// BytesPool returns the checked bytes pool.
	BytesPool() pool.CheckedBytesPool

	// SetReadConcurrency sets the concurrency of the reader.
	SetReadConcurrency(concurrency int) Options

	// ReadConcurrency returns the concurrency of the reader.
	ReadConcurrency() int

	// SetIdentifierPool sets the IdentifierPool to use for pooling identifiers.
	SetIdentifierPool(value ident.Pool) Options

	// IdentifierPool returns the IdentifierPool to use for pooling identifiers.
	IdentifierPool() ident.Pool

	// SetFailureStrategy sets the strategy.
	SetFailureStrategy(value FailureStrategy) Options

	// FailureStrategy returns the strategy.
	FailureStrategy() FailureStrategy

	// SetFailureCallback sets the strategy.
	SetFailureCallback(value FailureCallback) Options

	// FailureCallback returns the strategy.
	FailureCallback() FailureCallback
}

Options represents the options for the commit log.

func NewOptions 

func NewOptions(setters ...OptionSetter) Options

NewOptions creates new commit log options

type Reader added in v1.2.0 

type Reader interface {
	// Open opens the commit log for reading
	Open(filePath string) (int64, error)

	// Read returns the next id and data pair or error, will return io.EOF at end of volume
	Read() (LogEntry, error)

	// Close the reader
	Close() error
}

Reader reads a commit log file.

func NewReader added in v1.2.0 

func NewReader(opts ReaderOptions) Reader

NewReader returns a new Reader.

type ReaderOptions added in v1.2.0 

type ReaderOptions struct {
	// contains filtered or unexported fields
}

ReaderOptions are the options for Reader.

func NewReaderOptions added in v1.2.0 

func NewReaderOptions(opts Options, returnMetadataAsRef bool) ReaderOptions

NewReaderOptions returns new ReaderOptions.

type Strategy 

type Strategy int

Strategy describes the commit log writing strategy 

const (
	// StrategyWriteWait describes the strategy that waits
	// for the buffered commit log chunk that contains a write to flush
	// before acknowledging a write
	StrategyWriteWait Strategy = iota

	// StrategyWriteBehind describes the strategy that does not wait
	// for the buffered commit log chunk that contains a write to flush
	// before acknowledging a write
	StrategyWriteBehind
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.