buffers

Documentation

Index

• Constants
• type Buffer
  • func NewBuffer(leftOffset uint64, data []byte, capacity uint64) *Buffer
  • func (b *Buffer) AddrBelongsToKey(addr uint64, key []byte) (bool, error)
  • func (b *Buffer) Append(data []byte) uint64
  • func (b *Buffer) CanAppend(addr uint64) bool
  • func (b *Buffer) Contains(addr uint64) bool
  • func (b *Buffer) Eq(other *Buffer) bool
  • func (b *Buffer) GetValue(addr uint64, key []byte) (*entries.KeyValueEntry, error)
  • func (b *Buffer) ReadAt(addr uint64, size uint64) ([]byte, error)
  • func (b *Buffer) Replace(addr uint64, data []byte) error
  • func (b *Buffer) TryDeleteKvEntry(addr uint64, key []byte) (bool, error)
• type BufferPool
  • func NewBufferPool(capacity *uint64, filePath string, maxKeys *uint64, redundantBlocks *uint16, ...) (*BufferPool, error)
  • func (bp *BufferPool) AddrBelongsToKey(kvAddress uint64, key []byte) (bool, error)
  • func (bp *BufferPool) Append(data []byte) (uint64, error)
  • func (bp *BufferPool) ClearFile() error
  • func (bp *BufferPool) Close() error
  • func (bp *BufferPool) CompactFile() error
  • func (bp *BufferPool) Eq(other *BufferPool) bool
  • func (bp *BufferPool) GetValue(kvAddress uint64, key []byte) (*entries.KeyValueEntry, error)
  • func (bp *BufferPool) ReadIndex(addr uint64) ([]byte, error)
  • func (bp *BufferPool) TryDeleteKvEntry(kvAddress uint64, key []byte) (bool, error)
  • func (bp *BufferPool) UpdateIndex(addr uint64, data []byte) error

Constants

const DefaultPoolCapacity uint64 = 5

Types

type Buffer

type Buffer struct {
	Data        []byte
	LeftOffset  uint64
	RightOffset uint64
	// contains filtered or unexported fields
}

Buffer is the in-memory cache for byte arrays read from file Its `LeftOffset` is the file_offset where the byte array `Data` is read from while its `RightOffset` is the *exclusive* upper bound file offset of the same. the `RightOffset` is not an offset within this buffer but is the left_offset of the buffer that would be got from the file to the immediate right of this buffer's data array

func NewBuffer

func NewBuffer(leftOffset uint64, data []byte, capacity uint64) *Buffer

NewBuffer creates a new Buffer with the given left_offset FIXME: Add test for this

func (*Buffer) AddrBelongsToKey

func (b *Buffer) AddrBelongsToKey(addr uint64, key []byte) (bool, error)

AddrBelongsToKey checks to see if the given address is for the given key

func (*Buffer) Append

func (b *Buffer) Append(data []byte) uint64

Append appends the data to the end of the array It returns the address (or offset) where the data was appended

It is possible for data appended to this buffer to make it exceed its capacity. However, after that Buffer.CanAppend will always return false So make sure you call `can_append()` always. This is a trade-off that allows us to limit the number of re-allocations for buffers

func (*Buffer) CanAppend

func (b *Buffer) CanAppend(addr uint64) bool

CanAppend checks if the given address can be appended to this buffer The buffer should be contiguous thus this is true if `address` is equal to the exclusive `RightOffset` and the capacity has not been reached yet.

func (*Buffer) Contains

func (b *Buffer) Contains(addr uint64) bool

Contains checks if the given address is in this buffer

func (*Buffer) Eq

func (b *Buffer) Eq(other *Buffer) bool

Eq checks to see if two buffers are equal

func (*Buffer) GetValue

func (b *Buffer) GetValue(addr uint64, key []byte) (*entries.KeyValueEntry, error)

GetValue returns the *entries.KeyValueEntry at the given address if the key there corresponds to the given key Otherwise, it returns nil. This is to handle hash collisions.

func (*Buffer) ReadAt

func (b *Buffer) ReadAt(addr uint64, size uint64) ([]byte, error)

ReadAt reads an arbitrary array at the given address and of given size and returns it

func (*Buffer) Replace

func (b *Buffer) Replace(addr uint64, data []byte) error

Replace replaces the data at the given address with the new data

func (*Buffer) TryDeleteKvEntry

func (b *Buffer) TryDeleteKvEntry(addr uint64, key []byte) (bool, error)

TryDeleteKvEntry tries to delete the kv entry at the given address It returns false if the kv entry at the given address is not for the given key

type BufferPool

type BufferPool struct {
	File     *os.File
	FilePath string
	FileSize uint64
	// contains filtered or unexported fields
}

BufferPool is a pool of key-value and index Buffer's.

It is possible to have more than one buffer with the same address in a kind of overlap In order to avoid corruption, we always update the last kv buffer that has a given address since buffers are in FIFO queue. When retrieving a value, we also use the last buffer that has a given address

func NewBufferPool

func NewBufferPool(capacity *uint64, filePath string, maxKeys *uint64, redundantBlocks *uint16, bufferSize *uint32) (*BufferPool, error)

NewBufferPool creates a new BufferPool with the given `capacity` number of Buffers and for the file at the given path (creating it if necessary)

func (*BufferPool) AddrBelongsToKey

func (bp *BufferPool) AddrBelongsToKey(kvAddress uint64, key []byte) (bool, error)

AddrBelongsToKey checks to see if the given kv address is for the given key. Note that this returns true for expired keys as long as compaction has not yet been done. This avoids duplicate entries for the same key being tracked in separate index entries

It also returns false if the address goes beyond the size of the file

func (*BufferPool) Append

func (bp *BufferPool) Append(data []byte) (uint64, error)

Append appends a given data array to the file attached to this buffer pool It returns the address where the data was appended

func (*BufferPool) ClearFile

func (bp *BufferPool) ClearFile() error

ClearFile clears all data on disk and memory making it like a new store

func (*BufferPool) Close

func (bp *BufferPool) Close() error

Close closes the buffer pool, freeing up any resources

func (*BufferPool) CompactFile

func (bp *BufferPool) CompactFile() error

CompactFile removes any deleted or expired entries from the file. It must first lock the buffer and the file. In order to be more efficient, it creates a new file, copying only that data which is not deleted or expired

func (*BufferPool) Eq

func (bp *BufferPool) Eq(other *BufferPool) bool

Eq checks that other is equal to bp

func (*BufferPool) GetValue

func (bp *BufferPool) GetValue(kvAddress uint64, key []byte) (*entries.KeyValueEntry, error)

GetValue returns the *entries.KeyValueEntry at the given address if the key there corresponds to the given key Otherwise, it returns nil. This is to handle hash collisions.

func (*BufferPool) ReadIndex

func (bp *BufferPool) ReadIndex(addr uint64) ([]byte, error)

ReadIndex reads the index at the given address and returns it

If the address is less than [HEADER_SIZE_IN_BYTES] or [BufferPool.key_values_start_point], an ErrOutOfBounds error is returned

func (*BufferPool) TryDeleteKvEntry

func (bp *BufferPool) TryDeleteKvEntry(kvAddress uint64, key []byte) (bool, error)

TryDeleteKvEntry attempts to delete the key-value entry for the given kv_address as long as the key it holds is the same as the key provided. It returns true if successful

func (*BufferPool) UpdateIndex

func (bp *BufferPool) UpdateIndex(addr uint64, data []byte) error

UpdateIndex updates the index at the given address with the new data.

- This will fail if the data could spill into the key-value entry section or in the header section e.g. if the address is less than entries.HEADER_SIZE_IN_BYTES or (addr + data length) is greater than or equal BufferPool.keyValuesStartPoint