storage

package
v1.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 14, 2022 License: MPL-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package storage contains the low-level API for data storage. Data is stored in slots. The interface defines methods to store, retrieve, update and delete a given object to and from the disk. There are 3 main implementations:

DiskStorageManager

A disk storage manager handles the data storage on disk. It controls the actual PhysicalSlotManager and LogicalSlotManager objects. It holds references to all involved files and ensures exclusive access to them through a generated lock file. The lockfile is checked and attempting to open another instance of the DiskStorageManager on the same files will result in an error. The DiskStorageManager is also responsible for marshalling given abstract objects into a binary form which can be written to physical slots.

CachedDiskStorageManager

The CachedDiskStorageManager is a cache wrapper for the DiskStorageManager. Its purpose is to intercept calls and to maintain a cache of stored objects. The cache is limited in size by the number of total objects it references. Once the cache is full it will forget the objects which have been requested the least.

MemoryStorageManager

A storage manager which keeps all its data in memory and provides several error simulation facilities.

Index

Constants

View Source 
const (
	AccessNotInCache                = 1 // The address will not be accessible via FetchCached
	AccessFetchError                = 2 // The address will not be accessible via Fetch
	AccessUpdateError               = 3 // The address will not be accessible via Update
	AccessFreeError                 = 4 // The address will not be accessible via Free
	AccessInsertError               = 5 // The address will not be accessible via Insert
	AccessCacheAndFetchError        = 6 // The address will not be accessible via FetchCached nor Fetch
	AccessCacheAndFetchSeriousError = 7 // The address will not be accessible via FetchCached nor Fetch
)

Special flags which cause the manager to return errors on specific function calls

View Source 
const BlockSizeFreeSlots = 1024

BlockSizeFreeSlots is the block for a free slot files. Files containing only free slot pointers will always be small. They only need tiny blocks.

View Source 
const BlockSizeLogicalSlots = 1024 * 2

BlockSizeLogicalSlots is the block for a logical slot file. Logical slots contain only pointers they only need small blocks.

View Source 
const BlockSizePhysicalSlots = 1024 * 8

BlockSizePhysicalSlots is the block for a physical slot file. Physical slots will contain actual data they need to have fairly large block sizes.

View Source 
const FileSiffixLockfile = "lck"

FileSiffixLockfile is the file ending for lockfiles

View Source 
const FileSuffixLogicalFreeSlots = "ixf"

FileSuffixLogicalFreeSlots is the file ending for a free logical slot storage

View Source 
const FileSuffixLogicalSlots = "ix"

FileSuffixLogicalSlots is the file ending for a logical slot storage

View Source 
const FileSuffixPhysicalFreeSlots = "dbf"

FileSuffixPhysicalFreeSlots is the file ending for a free physical slot storage

View Source 
const FileSuffixPhysicalSlots = "db"

FileSuffixPhysicalSlots is the file ending for a physical slot storage

View Source 
const RootIDVersion = 1

RootIDVersion is the root id holding the version.

View Source 
const VERSION = 1

VERSION constains the version of the storage API

Variables

View Source 
var (
	ErrSlotNotFound = errors.New("Slot not found")
	ErrNotInCache   = errors.New("No entry in cache")
)

Common storage manager related errors.

View Source 
var BufferPool = pools.NewByteBufferPool()

BufferPool is a pool of byte buffers.

View Source 
var ErrReadonly = errors.New("Storage is readonly")

ErrReadonly is returned when attempting a write operation on a readonly datastore.

View Source 
var MsmCallNumClose int

MsmCallNumClose counter how often Close is called

View Source 
var MsmCallNumFlush int

MsmCallNumFlush counter how often Flush is called

View Source 
var MsmCallNumRollback int

MsmCallNumRollback counter how often Rollback is called

View Source 
var MsmRetClose error

MsmRetClose nil or the error which should be returned by a Close call

View Source 
var MsmRetFlush error

MsmRetFlush nil or the error which should be returned by a Flush call

View Source 
var MsmRetRollback error

MsmRetRollback nil or the error which should be returned by a Rollback call

Functions

func DataFileExist 

func DataFileExist(filename string) bool

DataFileExist checks if the main datastore file exists.

Types

type ByteDiskStorageManager 

type ByteDiskStorageManager struct {
	// contains filtered or unexported fields
}

ByteDiskStorageManager is a disk storage manager which can only store byte slices.

func NewByteDiskStorageManager 

func NewByteDiskStorageManager(filename string, readonly bool, onlyAppend bool,
	transDisabled bool, lockfileDisabled bool) *ByteDiskStorageManager

NewByteDiskStorageManager creates a new disk storage manager with optional transaction management which can only store byte slices. If the onlyAppend flag is set then the manager will not attempt to reuse space once it was released after use. If the transDisabled flag is set then the storage manager will not support transactions.

func (*ByteDiskStorageManager) Close 

func (bdsm *ByteDiskStorageManager) Close() error

Close closes the StorageManager and write all pending changes to disk.

func (*ByteDiskStorageManager) Fetch 

func (bdsm *ByteDiskStorageManager) Fetch(loc uint64, o interface{}) error

Fetch fetches an object from a given storage location and writes it to a given data container.

func (*ByteDiskStorageManager) FetchCached 

func (bdsm *ByteDiskStorageManager) FetchCached(loc uint64) (interface{}, error)

FetchCached is not implemented for a ByteDiskStorageManager. Only defined to satisfy the StorageManager interface.

func (*ByteDiskStorageManager) Flush 

func (bdsm *ByteDiskStorageManager) Flush() error

Flush writes all pending changes to disk.

func (*ByteDiskStorageManager) Free 

func (bdsm *ByteDiskStorageManager) Free(loc uint64) error

Free frees a storage location.

func (*ByteDiskStorageManager) Insert 

func (bdsm *ByteDiskStorageManager) Insert(o interface{}) (uint64, error)

Insert inserts an object and return its storage location.

func (*ByteDiskStorageManager) Name 

func (bdsm *ByteDiskStorageManager) Name() string

Name returns the name of the StorageManager instance.

func (*ByteDiskStorageManager) Rollback 

func (bdsm *ByteDiskStorageManager) Rollback() error

Rollback cancels all pending changes which have not yet been written to disk.

func (*ByteDiskStorageManager) Root 

func (bdsm *ByteDiskStorageManager) Root(root int) uint64

Root returns a root value.

func (*ByteDiskStorageManager) SetRoot 

func (bdsm *ByteDiskStorageManager) SetRoot(root int, val uint64)

SetRoot writes a root value.

func (*ByteDiskStorageManager) Update 

func (bdsm *ByteDiskStorageManager) Update(loc uint64, o interface{}) error

Update updates a storage location.

type CachedDiskStorageManager 

type CachedDiskStorageManager struct {
	// contains filtered or unexported fields
}

CachedDiskStorageManager data structure

func NewCachedDiskStorageManager 

func NewCachedDiskStorageManager(diskstoragemanager *DiskStorageManager, maxObjects int) *CachedDiskStorageManager

NewCachedDiskStorageManager creates a new cache wrapper for a DiskStorageManger.

func (*CachedDiskStorageManager) Close 

func (cdsm *CachedDiskStorageManager) Close() error

Close the StorageManager and write all pending changes to disk.

func (*CachedDiskStorageManager) Fetch 

func (cdsm *CachedDiskStorageManager) Fetch(loc uint64, o interface{}) error

Fetch fetches an object from a given storage location and writes it to a given data container.

func (*CachedDiskStorageManager) FetchCached 

func (cdsm *CachedDiskStorageManager) FetchCached(loc uint64) (interface{}, error)

FetchCached fetches an object from a cache and returns its reference. Returns a storage.ErrNotInCache error if the entry is not in the cache.

func (*CachedDiskStorageManager) Flush 

func (cdsm *CachedDiskStorageManager) Flush() error

Flush writes all pending changes to disk.

func (*CachedDiskStorageManager) Free 

func (cdsm *CachedDiskStorageManager) Free(loc uint64) error

Free frees a storage location.

func (*CachedDiskStorageManager) Insert 

func (cdsm *CachedDiskStorageManager) Insert(o interface{}) (uint64, error)

Insert inserts an object and return its storage location.

func (*CachedDiskStorageManager) Name 

func (cdsm *CachedDiskStorageManager) Name() string

Name returns the name of the StorageManager instance.

func (*CachedDiskStorageManager) Rollback 

func (cdsm *CachedDiskStorageManager) Rollback() error

Rollback cancels all pending changes which have not yet been written to disk.

func (*CachedDiskStorageManager) Root 

func (cdsm *CachedDiskStorageManager) Root(root int) uint64

Root returns a root value.

func (*CachedDiskStorageManager) SetRoot 

func (cdsm *CachedDiskStorageManager) SetRoot(root int, val uint64)

SetRoot writes a root value.

func (*CachedDiskStorageManager) Update 

func (cdsm *CachedDiskStorageManager) Update(loc uint64, o interface{}) error

Update updates a storage location.

type DiskStorageManager 

type DiskStorageManager struct {
	*ByteDiskStorageManager
}

DiskStorageManager is a storage manager which can store any gob serializable datastructure.

func NewDiskStorageManager 

func NewDiskStorageManager(filename string, readonly bool, onlyAppend bool,
	transDisabled bool, lockfileDisabled bool) *DiskStorageManager

NewDiskStorageManager creates a new disk storage manager with optional transaction management. If the onlyAppend flag is set then the manager will not attempt to reuse space once it was released after use. If the transDisabled flag is set then the storage manager will not support transactions.

func (*DiskStorageManager) Fetch 

func (dsm *DiskStorageManager) Fetch(loc uint64, o interface{}) error

Fetch fetches an object from a given storage location and writes it to a given data container.

func (*DiskStorageManager) Insert 

func (dsm *DiskStorageManager) Insert(o interface{}) (uint64, error)

Insert inserts an object and return its storage location.

func (*DiskStorageManager) Name 

func (dsm *DiskStorageManager) Name() string

Name returns the name of the StorageManager instance.

func (*DiskStorageManager) Serialize 

func (dsm *DiskStorageManager) Serialize(o interface{}) ([]byte, error)

Serialize serializes an object into a byte slice.

func (*DiskStorageManager) Update 

func (dsm *DiskStorageManager) Update(loc uint64, o interface{}) error

Update updates a storage location.

type Manager 

type Manager interface {

	/*
	   Name returns the name of the StorageManager instance.
	*/
	Name() string

	/*
		Root returns a root value.
	*/
	Root(root int) uint64

	/*
		SetRoot writes a root value.
	*/
	SetRoot(root int, val uint64)

	/*
	   Insert inserts an object and return its storage location.
	*/
	Insert(o interface{}) (uint64, error)

	/*
	   Update updates a storage location.
	*/
	Update(loc uint64, o interface{}) error

	/*
		Free frees a storage location.
	*/
	Free(loc uint64) error

	/*
		Fetch fetches an object from a given storage location and writes it to
		a given data container.
	*/
	Fetch(loc uint64, o interface{}) error

	/*
		FetchCached fetches an object from a cache and returns its reference.
		Returns a storage.ErrNotInCache error if the entry is not in the cache.
	*/
	FetchCached(loc uint64) (interface{}, error)

	/*
	   Flush writes all pending changes to disk.
	*/
	Flush() error

	/*
		Rollback cancels all pending changes which have not yet been written to disk.
	*/
	Rollback() error

	/*
		Close the StorageManager and write all pending changes to disk.
	*/
	Close() error
}

Manager describes an abstract storage manager.

type ManagerError added in v1.2.0 

type ManagerError struct {
	Type        error
	Detail      string
	Managername string
}

ManagerError is a storage manager related error.

func NewStorageManagerError added in v1.2.0 

func NewStorageManagerError(smeType error, smeDetail string, smeManagername string) *ManagerError

NewStorageManagerError returns a new StorageManager specific error.

func (*ManagerError) Error added in v1.2.0 

func (e *ManagerError) Error() string

Error returns a string representation of the error.

type MemoryStorageManager 

type MemoryStorageManager struct {
	Roots map[int]uint64         // Map of roots
	Data  map[uint64]interface{} // Map of data

	LocCount  uint64         // Counter for locations - Must start > 0
	AccessMap map[uint64]int // Special map to simulate access issues
	// contains filtered or unexported fields
}

MemoryStorageManager data structure

func NewMemoryStorageManager 

func NewMemoryStorageManager(name string) *MemoryStorageManager

NewMemoryStorageManager creates a new MemoryStorageManager

func (*MemoryStorageManager) Close 

func (msm *MemoryStorageManager) Close() error

Close the StorageManager and write all pending changes to disk.

func (*MemoryStorageManager) Fetch 

func (msm *MemoryStorageManager) Fetch(loc uint64, o interface{}) error

Fetch fetches an object from a given storage location and writes it to a given data container.

func (*MemoryStorageManager) FetchCached 

func (msm *MemoryStorageManager) FetchCached(loc uint64) (interface{}, error)

FetchCached fetches an object from a cache and returns its reference. Returns a storage.ErrNotInCache error if the entry is not in the cache.

func (*MemoryStorageManager) Flush 

func (msm *MemoryStorageManager) Flush() error

Flush writes all pending changes to disk.

func (*MemoryStorageManager) Free 

func (msm *MemoryStorageManager) Free(loc uint64) error

Free frees a storage location.

func (*MemoryStorageManager) Insert 

func (msm *MemoryStorageManager) Insert(o interface{}) (uint64, error)

Insert inserts an object and return its storage location.

func (*MemoryStorageManager) Name 

func (msm *MemoryStorageManager) Name() string

Name returns the name of the StorageManager instance.

func (*MemoryStorageManager) Rollback 

func (msm *MemoryStorageManager) Rollback() error

Rollback cancels all pending changes which have not yet been written to disk.

func (*MemoryStorageManager) Root 

func (msm *MemoryStorageManager) Root(root int) uint64

Root returns a root value. Default (empty) value is 0.

func (*MemoryStorageManager) SetRoot 

func (msm *MemoryStorageManager) SetRoot(root int, val uint64)

SetRoot writes a root value.

func (*MemoryStorageManager) String 

func (msm *MemoryStorageManager) String() string

Show a string representation of the storage manager.

func (*MemoryStorageManager) Update 

func (msm *MemoryStorageManager) Update(loc uint64, o interface{}) error

Update updates a storage location.

Source Files

View all Source files

Directories

Path Synopsis
Package file deals with low level file storage and transaction management.
 Package file deals with low level file storage and transaction management.
Package paging contains functions and constants necessary for paging of records.
 Package paging contains functions and constants necessary for paging of records.
view
Package view contains general page view constants and functions.
 Package view contains general page view constants and functions.
Package slotting contains managers which deal with slots on pages.
 Package slotting contains managers which deal with slots on pages.
pageview
Package pageview contains object wrappers for different page types.
 Package pageview contains object wrappers for different page types.
Package util contains utility functions for slot headers.
 Package util contains utility functions for slot headers.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.