buffer

package
v1.6.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 6, 2022 License: Apache-2.0 Imports: 17 Imported by: 4

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Buffer 

type Buffer interface {
	Add(context.Context, *entry.Entry) error
	Read([]*entry.Entry) (Clearer, int, error)
	ReadWait(context.Context, []*entry.Entry) (Clearer, int, error)
	ReadChunk(context.Context) ([]*entry.Entry, Clearer, error)
	Close() error
	MaxChunkDelay() time.Duration
	MaxChunkSize() uint
	SetMaxChunkDelay(time.Duration)
	SetMaxChunkSize(uint)
}

Buffer is an interface for an entry buffer

type Builder added in v0.12.2 

type Builder interface {
	Build(context operator.BuildContext, pluginID string) (Buffer, error)
}

Builder builds a Buffer given build context

type Clearer added in v0.13.2 

type Clearer interface {
	MarkAllAsFlushed() error
	MarkRangeAsFlushed(uint, uint) error
}

Clearer is an interface that is responsible for clearing entries from a buffer

type Config 

type Config struct {
	Builder
}

Config is a struct that wraps a Builder

func NewConfig 

func NewConfig() Config

NewConfig returns a default Config

func (Config) MarshalJSON added in v0.12.3 

func (bc Config) MarshalJSON() ([]byte, error)

MarshalJSON marshals JSON

func (Config) MarshalYAML added in v0.12.3 

func (bc Config) MarshalYAML() (interface{}, error)

MarshalYAML marshals YAML

func (*Config) UnmarshalJSON added in v0.10.0 

func (bc *Config) UnmarshalJSON(data []byte) error

UnmarshalJSON unmarshals JSON

func (*Config) UnmarshalYAML added in v0.10.0 

func (bc *Config) UnmarshalYAML(f func(interface{}) error) error

UnmarshalYAML unmarshals YAML

type DiskBuffer added in v0.10.0 

type DiskBuffer struct {
	sync.Mutex
	// contains filtered or unexported fields
}

DiskBuffer is a buffer for storing entries on disk until they are flushed to their final destination.

func NewDiskBuffer added in v0.10.0 

func NewDiskBuffer(maxDiskSize int64) *DiskBuffer

NewDiskBuffer creates a new DiskBuffer

func (*DiskBuffer) Add added in v0.10.0 

func (d *DiskBuffer) Add(ctx context.Context, newEntry *entry.Entry) error

Add adds an entry to the buffer, blocking until it is either added or the context is cancelled.

func (*DiskBuffer) Close added in v0.10.0 

func (d *DiskBuffer) Close() error

Close flushes the current metadata to disk, then closes the underlying files

func (*DiskBuffer) Compact added in v0.10.0 

func (d *DiskBuffer) Compact() error

Compact removes all flushed entries from disk

func (*DiskBuffer) MaxChunkDelay added in v0.14.2 

func (d *DiskBuffer) MaxChunkDelay() time.Duration

MaxChunkDelay returns the max chunk delay

func (*DiskBuffer) MaxChunkSize added in v0.14.2 

func (d *DiskBuffer) MaxChunkSize() uint

MaxChunkSize returns the max chunk size

func (*DiskBuffer) Open added in v0.10.0 

func (d *DiskBuffer) Open(path string, sync bool) error

Open opens the disk buffer files from a database directory

func (*DiskBuffer) Read added in v0.10.0 

func (d *DiskBuffer) Read(dst []*entry.Entry) (f Clearer, i int, err error)

Read copies entries from the disk into the destination buffer. It returns a function that, when called, marks the entries as flushed, the number of entries read, and an error.

func (*DiskBuffer) ReadChunk added in v0.13.2 

func (d *DiskBuffer) ReadChunk(ctx context.Context) ([]*entry.Entry, Clearer, error)

ReadChunk is a thin wrapper around ReadWait that simplifies the call at the expense of an extra allocation

func (*DiskBuffer) ReadWait added in v0.10.0 

func (d *DiskBuffer) ReadWait(ctx context.Context, dst []*entry.Entry) (Clearer, int, error)

ReadWait reads entries from the buffer, waiting until either there are enough entries in the buffer to fill dst or the context is cancelled. This amortizes the cost of reading from the disk. It returns a function that, when called, marks the read entries as flushed, the number of entries read, and an error.

func (*DiskBuffer) SetMaxChunkDelay added in v0.14.2 

func (d *DiskBuffer) SetMaxChunkDelay(delay time.Duration)

SetMaxChunkDelay sets the max chunk delay

func (*DiskBuffer) SetMaxChunkSize added in v0.14.2 

func (d *DiskBuffer) SetMaxChunkSize(size uint)

SetMaxChunkSize sets the max chunk size

type DiskBufferConfig added in v0.10.0 

type DiskBufferConfig struct {
	Type string `json:"type" yaml:"type"`

	// MaxSize is the maximum size in bytes of the data file on disk
	MaxSize helper.ByteSize `json:"max_size" yaml:"max_size"`

	// Path is a path to a directory which contains the data and metadata files
	Path string `json:"path" yaml:"path"`

	// Sync indicates whether to open the files with O_SYNC. If this is set to false,
	// in cases like power failures or unclean shutdowns, logs may be lost or the
	// database may become corrupted.
	Sync bool `json:"sync" yaml:"sync"`

	MaxChunkDelay helper.Duration `json:"max_delay"   yaml:"max_delay"`
	MaxChunkSize  uint            `json:"max_chunk_size" yaml:"max_chunk_size"`
}

DiskBufferConfig is a configuration struct for a DiskBuffer

func NewDiskBufferConfig added in v0.10.0 

func NewDiskBufferConfig() *DiskBufferConfig

NewDiskBufferConfig creates a new default disk buffer config

func (DiskBufferConfig) Build added in v0.10.0 

func (c DiskBufferConfig) Build(_ operator.BuildContext, _ string) (Buffer, error)

Build creates a new Buffer from a DiskBufferConfig

type MemoryBuffer 

type MemoryBuffer struct {
	// contains filtered or unexported fields
}

MemoryBuffer is a buffer that holds all entries in memory until Close() is called, at which point it saves the entries into a database. It provides no guarantees about lost entries if shut down uncleanly.

func (*MemoryBuffer) Add added in v0.10.0 

func (m *MemoryBuffer) Add(ctx context.Context, e *entry.Entry) error

Add inserts an entry into the memory database, blocking until there is space

func (*MemoryBuffer) Close added in v0.10.0 

func (m *MemoryBuffer) Close() error

Close closes the memory buffer, saving all entries currently in the memory buffer to the agent's database.

func (*MemoryBuffer) MaxChunkDelay added in v0.14.2 

func (m *MemoryBuffer) MaxChunkDelay() time.Duration

MaxChunkDelay returns the max chunk delay

func (*MemoryBuffer) MaxChunkSize added in v0.14.2 

func (m *MemoryBuffer) MaxChunkSize() uint

MaxChunkSize returns the max chunk size

func (*MemoryBuffer) Read added in v0.10.0 

func (m *MemoryBuffer) Read(dst []*entry.Entry) (Clearer, int, error)

Read reads entries until either there are no entries left in the buffer or the destination slice is full. The returned function must be called once the entries are flushed to remove them from the memory buffer.

func (*MemoryBuffer) ReadChunk added in v0.13.2 

func (m *MemoryBuffer) ReadChunk(ctx context.Context) ([]*entry.Entry, Clearer, error)

ReadChunk is a thin wrapper around ReadWait that simplifies the call at the expense of an extra allocation

func (*MemoryBuffer) ReadWait added in v0.10.0 

func (m *MemoryBuffer) ReadWait(ctx context.Context, dst []*entry.Entry) (Clearer, int, error)

ReadWait reads entries until either the destination slice is full, or the context passed to it is cancelled. The returned function must be called once the entries are flushed to remove them from the memory buffer

func (*MemoryBuffer) SetMaxChunkDelay added in v0.14.2 

func (m *MemoryBuffer) SetMaxChunkDelay(delay time.Duration)

SetMaxChunkDelay sets the max chunk delay

func (*MemoryBuffer) SetMaxChunkSize added in v0.14.2 

func (m *MemoryBuffer) SetMaxChunkSize(size uint)

SetMaxChunkSize sets the max chunk size

type MemoryBufferConfig added in v0.10.0 

type MemoryBufferConfig struct {
	Type          string          `json:"type"        yaml:"type"`
	MaxEntries    int             `json:"max_entries" yaml:"max_entries"`
	MaxChunkDelay helper.Duration `json:"max_delay"   yaml:"max_delay"`
	MaxChunkSize  uint            `json:"max_chunk_size" yaml:"max_chunk_size"`
}

MemoryBufferConfig holds the configuration for a memory buffer

func NewMemoryBufferConfig added in v0.10.0 

func NewMemoryBufferConfig() *MemoryBufferConfig

NewMemoryBufferConfig creates a new default MemoryBufferConfig

func (MemoryBufferConfig) Build added in v0.10.0 

func (c MemoryBufferConfig) Build(context operator.BuildContext, pluginID string) (Buffer, error)

Build builds a MemoryBufferConfig into a Buffer, loading any entries that were previously unflushed back into memory

type Metadata added in v0.10.0 

type Metadata struct {
	// contains filtered or unexported fields
}

Metadata is a representation of the on-disk metadata file. It contains information about the layout, location, and flushed status of entries stored in the data file

func OpenMetadata added in v0.10.0 

func OpenMetadata(path string, sync bool) (*Metadata, error)

OpenMetadata opens and parses the metadata

func (*Metadata) Close added in v0.10.0 

func (m *Metadata) Close() error

Close syncs metadata to disk and closes the underlying file descriptor

func (*Metadata) MarshalBinary added in v0.10.0 

func (m *Metadata) MarshalBinary(wr io.Writer) (err error)

MarshalBinary marshals a metadata struct to a binary stream

func (*Metadata) Sync added in v0.10.0 

func (m *Metadata) Sync() error

Sync persists the metadata to disk

func (*Metadata) UnmarshalBinary added in v0.10.0 

func (m *Metadata) UnmarshalBinary(r io.Reader) error

UnmarshalBinary unmarshals metadata from a binary stream (usually a file)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.