fs1

package
v1.8.2-0...-f7776fc Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 2, 2024 License: GPL-2.0, GPL-3.0 Imports: 25 Imported by: 0

Documentation

Overview

Package fs1 provides so-called FileStorage version 1 ZODB storage.

FileStorage is a single file organized as a simple append-only log of transactions with data changes. Every transaction record consists of:

  • transaction record header represented by TxnHeader,
  • several data records corresponding to modified objects,
  • redundant transaction length at the end of transaction record.

Every data record consists of:

  • data record header represented by DataHeader,
  • actual data following the header.

The "actual data" in addition to raw content, can be a back-pointer indicating that the actual content should be retrieved from a past revision.

In addition to append-only transaction/data log, an index is automatically maintained mapping oid -> latest data record which modified this oid. The index is used to implement zodb.IStorage.Load for latest data without linear scan.

The data format is bit-to-bit identical to FileStorage format implemented in ZODB/py. Please see the following links for original FileStorage format definition: 

https://github.com/zopefoundation/ZODB/blob/a89485c1/src/ZODB/FileStorage/format.py
https://github.com/zopefoundation/ZODB/blob/a89485c1/src/ZODB/fstools.py

The index format is interoperable with ZODB/py (index uses pickles which allow various valid encodings of a given object). Please see the following links for original FileStorage/py index definition: 

https://github.com/zopefoundation/ZODB/blob/a89485c1/src/ZODB/fsIndex.py
https://github.com/zopefoundation/ZODB/commit/1bb14faf

Unless one is doing something FileStorage-specific, it is advised not to use fs1 package directly, and instead link-in lab.nexedi.com/kirr/neo/go/zodb/wks, open storage by zodb.Open and use it by way of zodb.IStorage interface.

The fs1 package exposes all FileStorage data format details and most of internal workings so that it is possible to implement FileStorage-specific tools.

See also package lab.nexedi.com/kirr/neo/go/zodb/storage/fs1/fs1tools and associated fs1 command for basic tools related to FileStorage maintenance.

Index

Constants

View Source 
const (
	Magic = "FS21" // every FileStorage file starts with this

	// on-disk sizes
	FileHeaderSize   = 4
	TxnHeaderFixSize = 8 + 8 + 1 + 2 + 2 + 2 // without user/desc/ext strings

	DataHeaderSize = 8 + 8 + 8 + 8 + 2 + 8
)

Variables

This section is empty.

Functions

This section is empty.

Types

type DataHeader 

type DataHeader struct {
	Pos        int64 // position of data record start
	Oid        zodb.Oid
	Tid        zodb.Tid
	PrevRevPos int64 // position of this oid's previous-revision data record
	TxnPos     int64 // position of transaction record this data record belongs to
	//_        uint16   // 2-bytes with zero values. (Was version length.)
	DataLen int64 // length of following data. if 0 -> following = 8 bytes backpointer
	// contains filtered or unexported fields
}

DataHeader represents header of a data record.

func DataHeaderAlloc 

func DataHeaderAlloc() *DataHeader

DataHeaderAlloc allocates DataHeader from freelist.

func (*DataHeader) Free 

func (dh *DataHeader) Free()

Free puts dh back into DataHeader freelist.

Caller must not use dh after call to Free.

func (*DataHeader) Len 

func (dh *DataHeader) Len() int64

Len returns whole data record length.

func (*DataHeader) Load 

func (dh *DataHeader) Load(r io.ReaderAt, pos int64) error

Load reads and decodes data record header @ pos.

Only the data record header is loaded, not data itself. See LoadData for actually loading record's data.

No prerequisite requirements are made to previous dh state.

func (*DataHeader) LoadBack 

func (dh *DataHeader) LoadBack(r io.ReaderAt) error

LoadBack reads and decodes data header for revision linked via back-pointer.

Prerequisite: dh is loaded and .DataLen == 0.

If link is to zero (means deleted record) io.EOF is returned.

func (*DataHeader) LoadBackRef 

func (dh *DataHeader) LoadBackRef(r io.ReaderAt) (backPos int64, err error)

LoadBackRef reads data for the data record and decodes it as backpointer reference.

Prerequisite: dh loaded and .LenData == 0 (data record with back-pointer).

func (*DataHeader) LoadData 

func (dh *DataHeader) LoadData(r io.ReaderAt) (*mem.Buf, error)

LoadData loads data for the data record taking backpointers into account.

On success dh state is changed to data header of original data transaction.

NOTE "deleted" records are indicated via returning buf with .Data=nil without error.

func (*DataHeader) LoadNext 

func (dh *DataHeader) LoadNext(r io.ReaderAt, txnh *TxnHeader) error

LoadNext reads and decodes data header for next data record in the same transaction.

Prerequisite: dh .Pos .DataLen are initialized.

When there is no more data records: io.EOF is returned.

func (*DataHeader) LoadPrevRev 

func (dh *DataHeader) LoadPrevRev(r io.ReaderAt) error

LoadPrevRev reads and decodes previous revision data record header.

Prerequisite: dh .Oid .Tid .PrevRevPos are initialized.

When there is no previous revision: io.EOF is returned.

type FileHeader 

type FileHeader struct {
	Magic [4]byte
}

FileHeader represents header of whole data file.

func (*FileHeader) Load 

func (fh *FileHeader) Load(r io.ReaderAt) error

Load reads and decodes file header.

type FileStorage 

type FileStorage struct {
	// contains filtered or unexported fields
}

FileStorage is a ZODB storage which stores data in simple append-only file organized as transactional log.

It is on-disk compatible with FileStorage from ZODB/py.

func Open 

func Open(ctx context.Context, path string, opt *zodb.DriverOptions) (_ *FileStorage, at0 zodb.Tid, err error)

Open opens FileStorage @path.

func (*FileStorage) Close 

func (fs *FileStorage) Close() error

func (*FileStorage) Iterate 

func (fs *FileStorage) Iterate(_ context.Context, tidMin, tidMax zodb.Tid) zodb.ITxnIterator

Iterate creates zodb-level iterator for tidMin..tidMax range.

func (*FileStorage) LastOid 

func (fs *FileStorage) LastOid(_ context.Context) (zodb.Oid, error)

func (*FileStorage) Load 

func (fs *FileStorage) Load(_ context.Context, xid zodb.Xid) (buf *mem.Buf, serial zodb.Tid, err error)

func (*FileStorage) Sync 

func (fs *FileStorage) Sync(ctx context.Context) (head zodb.Tid, err error)

Sync implements zodb.IStorageDriver.

func (*FileStorage) URL 

func (fs *FileStorage) URL() string

type Index 

type Index struct {
	// this index covers data file up to < .TopPos
	// usually for whole-file index TopPos is position pointing just past
	// the last committed transaction.
	TopPos int64

	*fsb.Tree
}

Index is in-RAM Oid -> Data record position mapping used to associate Oid with Data record in latest transaction which changed it.

func BuildIndex 

func BuildIndex(ctx context.Context, r io.ReaderAt, progress func(*IndexUpdateProgress)) (*Index, error)

BuildIndex builds new in-memory index for data in r.

non-nil valid and consistent index is always returned - even in case of error the index will describe data till top-position of highest transaction that could be read without error.

In such cases the index building could be retried to be finished with index.Update().

func BuildIndexForFile 

func BuildIndexForFile(ctx context.Context, path string, progress func(*IndexUpdateProgress)) (index *Index, err error)

BuildIndexForFile builds new in-memory index for data in file @ path.

See BuildIndex for semantic description.

func IndexNew 

func IndexNew() *Index

IndexNew creates new empty index.

func LoadIndex 

func LoadIndex(r io.Reader) (fsi *Index, err error)

LoadIndex loads index from a reader.

func LoadIndexFile 

func LoadIndexFile(path string) (fsi *Index, err error)

LoadIndexFile loads index from a file @ path.

func (*Index) Equal 

func (a *Index) Equal(b *Index) bool

Equal returns whether two indices are the same.

func (*Index) Save 

func (fsi *Index) Save(w io.Writer) (err error)

Save saves index to a writer.

func (*Index) SaveFile 

func (fsi *Index) SaveFile(path string) error

SaveFile saves index to a file @ path.

Index data is first saved to a temporary file and when complete the temporary is renamed to be at requested path. This way file @ path will be updated only with complete index data.

func (*Index) Update 

func (index *Index) Update(ctx context.Context, r io.ReaderAt, topPos int64, progress func(*IndexUpdateProgress)) (err error)

Update updates in-memory index from r's FileStorage data in byte-range index.TopPos..topPos

The use case is: we have index computed till some position; we open FileStorage and see there is more data; we update index from data range not-yet covered by the index.

topPos=-1 means data range to update from is index.TopPos..EOF

The index stays valid even in case of error - then index is updated but only partially. The index always stays consistent as updates to it are applied as a whole once per data transaction. On return index.TopPos indicates till which position in data the index could be updated.

On success returned error is nil and index.TopPos is set to either:

  • topPos (if it is != -1), or
  • r's position at which read got EOF (if topPos=-1).

func (*Index) Verify 

func (index *Index) Verify(ctx context.Context, r io.ReaderAt, ntxn int, progress func(*IndexVerifyProgress)) (oidChecked map[zodb.Oid]struct{}, err error)

Verify checks index correctness against FileStorage data in r.

For ntxn transactions starting from index.TopPos backwards, it verifies whether oid there have correct entries in the index.

ntxn=-1 means data range to verify is till start of the file.

If whole data file was covered (either ntxn is big enough or was set = -1) additional checks are performed to make sure there is no extra entries in the index. For whole-data file cases Verify thus checks whether index is exactly the same as if it was build anew for data in range ..index.TopPos .

Returned error is either:

  • of type *IndexCorruptError, when data in index was found not to match original data, or
  • any other error type representing e.g. IO error when reading original data or something else.

func (*Index) VerifyForFile 

func (index *Index) VerifyForFile(ctx context.Context, path string, ntxn int, progress func(*IndexVerifyProgress)) (oidChecked map[zodb.Oid]struct{}, err error)

VerifyForFile checks index correctness against FileStorage data in file @ path.

See Verify for semantic description.

type IndexCorruptError 

type IndexCorruptError struct {
	DataFileName string // present if data IO object was with .Name()
	Detail       string
}

IndexCorruptError is the error type returned by index verification routines when index was found to not match original FileStorage data.

func (*IndexCorruptError) Error 

func (e *IndexCorruptError) Error() string

type IndexLoadError 

type IndexLoadError struct {
	Filename string // present if used IO object was with .Name()
	Pos      int64
	Err      error
}

IndexLoadError is the error type returned by index load routines.

func (*IndexLoadError) Error 

func (e *IndexLoadError) Error() string

type IndexSaveError 

type IndexSaveError struct {
	Err error // error that occurred during the operation
}

IndexSaveError is the error type returned by index save routines.

func (*IndexSaveError) Error 

func (e *IndexSaveError) Error() string

type IndexUpdateProgress 

type IndexUpdateProgress struct {
	TopPos     int64  // data range to update to; if = -1 -- till EOF
	TxnIndexed int    // # transactions read/indexed so far
	Index      *Index // index built so far
	Iter       *Iter  // iterator through data	XXX needed?
}

IndexUpdateProgress is data sent by Index.Update to notify about progress.

type IndexVerifyProgress 

type IndexVerifyProgress struct {
	TxnTotal   int // total # of transactions to verify; if = -1 -- whole data
	TxnChecked int
	Index      *Index                // index verification runs for
	Iter       *Iter                 // iterator through data
	OidChecked map[zodb.Oid]struct{} // oid checked so far
}

IndexVerifyProgress is data sent by Index.Verify to notify about progress.

type Iter 

type Iter struct {
	R   io.ReaderAt
	Dir IterDir

	Txnh  TxnHeader  // current transaction record information
	Datah DataHeader // current data record information
}

Iter is combined 2-level iterator over transaction and data records.

func Iterate 

func Iterate(r io.ReaderAt, posStart int64, dir IterDir) *Iter

Iterate creates Iter to iterate over r starting from posStart in direction dir.

func IterateFile 

func IterateFile(path string, dir IterDir) (iter *Iter, file *os.File, err error)

IterateFile opens file @ path read-only and creates Iter to iterate over it.

The iteration will use buffering over os.File optimized for sequential access. You are responsible to eventually close the file after the iteration is done.

func (*Iter) NextData 

func (it *Iter) NextData() error

NextData iterates to next data record header inside current transaction.

func (*Iter) NextTxn 

func (it *Iter) NextTxn(flags TxnLoadFlags) error

NextTxn iterates to next/previous transaction record according to iteration direction.

The data header is reset to iterate inside transaction record that becomes current.

type IterDir 

type IterDir int
const (
	IterForward IterDir = iota
	IterBackward
)

type RecordError 

type RecordError struct {
	Path   string // path of the data file; present if used IO object was with .Name()
	Record string // record kind - "file header", "transaction record", "data record", ...
	Pos    int64  // position of record
	Subj   string // subject context for the error - e.g. "read" or "check"
	Err    error  // actual error
}

RecordError represents error associated with operation on a record in FileStorage data file.

func (*RecordError) Cause 

func (e *RecordError) Cause() error

func (*RecordError) Error 

func (e *RecordError) Error() string

func (*RecordError) Unwrap 

func (e *RecordError) Unwrap() error

type TxnHeader 

type TxnHeader struct {
	Pos     int64 // position of transaction start
	LenPrev int64 // whole previous transaction record length (see EOF/error rules in Load)
	Len     int64 // whole transaction record length          (see EOF/error rules in Load)

	// transaction metadata itself
	zodb.TxnInfo
	// contains filtered or unexported fields
}

TxnHeader represents header of a transaction record.

func (*TxnHeader) CloneFrom 

func (txnh *TxnHeader) CloneFrom(txnh2 *TxnHeader)

CloneFrom copies txnh2 to txnh making sure underlying slices (.workMem .User .Desc ...) are not shared.

func (*TxnHeader) DataLen 

func (txnh *TxnHeader) DataLen() int64

DataLen returns length of all data inside transaction record container.

func (*TxnHeader) DataPos 

func (txnh *TxnHeader) DataPos() int64

DataPos returns start position of data inside transaction record.

func (*TxnHeader) HeaderLen 

func (txnh *TxnHeader) HeaderLen() int64

HeaderLen returns whole transaction header length including its variable part.

NOTE: data records start right after transaction header.

func (*TxnHeader) Load 

func (txnh *TxnHeader) Load(r io.ReaderAt, pos int64, flags TxnLoadFlags) error

Load reads and decodes transaction record header @ pos.

Both transaction header starting at pos, and redundant length of previous transaction are loaded. The data read is verified for consistency lightly.

No prerequisite requirements are made to previous txnh state.

Rules for .Len/.LenPrev returns: 

.Len = -1	transaction header could not be read
.Len = 0	EOF forward
.Len > 0 	transaction was read normally

.LenPrev = -1	prev record length could not be read
.LenPrev = 0	EOF backward
.LenPrev > 0	LenPrev was read/checked normally

For example when pos points to the end of file .Len will be returned = -1, but .LenPrev will be usually valid if file has at least 1 transaction.

func (*TxnHeader) LoadNext 

func (txnh *TxnHeader) LoadNext(r io.ReaderAt, flags TxnLoadFlags) error

LoadNext reads and decodes next transaction record header.

prerequisite: txnh .Pos, .Len and .Tid should be already initialized.

func (*TxnHeader) LoadPrev 

func (txnh *TxnHeader) LoadPrev(r io.ReaderAt, flags TxnLoadFlags) error

LoadPrev reads and decodes previous transaction record header.

prerequisites:

  • txnh .Pos, .LenPrev are initialized
  • optionally if .Len is initialized and txnh was loaded - tid↓ will be also checked

type TxnLoadFlags 

type TxnLoadFlags int

flags for TxnHeader.Load 

const (
	LoadAll       TxnLoadFlags = 0x00 // load whole transaction header
	LoadNoStrings TxnLoadFlags = 0x01 // do not load user/desc/ext strings
)

Source Files

View all Source files

Directories

Path Synopsis
cmd
fs1
fs1 is a driver program for running and invoking fs1 subcommands.
 fs1 is a driver program for running and invoking fs1 subcommands.
Package fs1tools provides tools for managing and maintaining ZODB FileStorage v1 databases.
 Package fs1tools provides tools for managing and maintaining ZODB FileStorage v1 databases.
Package fsb specializes cznic/b.Tree for FileStorage index needs.
 Package fsb specializes cznic/b.Tree for FileStorage index needs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.