Documentation

Overview

    Package fs1 provides so-called FileStorage version 1 ZODB storage.

    FileStorage is a single file organized as a simple append-only log of transactions with data changes. Every transaction record consists of:

    - transaction record header represented by TxnHeader,
    - several data records corresponding to modified objects,
    - redundant transaction length at the end of transaction record.
    

    Every data record consists of:

    - data record header represented by DataHeader,
    - actual data following the header.
    

    The "actual data" in addition to raw content, can be a back-pointer indicating that the actual content should be retrieved from a past revision.

    In addition to append-only transaction/data log, an index is automatically maintained mapping oid -> latest data record which modified this oid. The index is used to implement zodb.IStorage.Load for latest data without linear scan.

    The data format is bit-to-bit identical to FileStorage format implemented in ZODB/py. Please see the following links for original FileStorage format definition:

    https://github.com/zopefoundation/ZODB/blob/a89485c1/src/ZODB/FileStorage/format.py
    https://github.com/zopefoundation/ZODB/blob/a89485c1/src/ZODB/fstools.py
    

    The index format is interoperable with ZODB/py (index uses pickles which allow various valid encodings of a given object). Please see the following links for original FileStorage/py index definition:

    https://github.com/zopefoundation/ZODB/blob/a89485c1/src/ZODB/fsIndex.py
    https://github.com/zopefoundation/ZODB/commit/1bb14faf
    

    Unless one is doing something FileStorage-specific, it is advised not to use fs1 package directly, and instead link-in lab.nexedi.com/kirr/neo/go/zodb/wks, open storage by zodb.Open and use it by way of zodb.IStorage interface.

    The fs1 package exposes all FileStorage data format details and most of internal workings so that it is possible to implement FileStorage-specific tools.

    See also package lab.nexedi.com/kirr/neo/go/zodb/storage/fs1/fs1tools and associated fs1 command for basic tools related to FileStorage maintenance.

    Index

    Constants

    View Source
    const (
    	Magic21 = "FS21" // FileStorage file produced by Python2 starts with this
    	Magic30 = "FS30" // ----//---- by Python3	XXX +test
    
    	// on-disk sizes
    	FileHeaderSize   = 4
    	TxnHeaderFixSize = 8 + 8 + 1 + 2 + 2 + 2 // without user/desc/ext strings
    
    	DataHeaderSize = 8 + 8 + 8 + 8 + 2 + 8
    )

    Variables

    This section is empty.

    Functions

    This section is empty.

    Types

    type DataHeader

    type DataHeader struct {
    	Pos        int64 // position of data record start
    	Oid        zodb.Oid
    	Tid        zodb.Tid
    	PrevRevPos int64 // position of this oid's previous-revision data record
    	TxnPos     int64 // position of transaction record this data record belongs to
    	//_        uint16   // 2-bytes with zero values. (Was version length.)
    	DataLen int64 // length of following data. if 0 -> following = 8 bytes backpointer
    	// contains filtered or unexported fields
    }

      DataHeader represents header of a data record.

      func DataHeaderAlloc

      func DataHeaderAlloc() *DataHeader

        DataHeaderAlloc allocates DataHeader from freelist.

        func (*DataHeader) Free

        func (dh *DataHeader) Free()

          Free puts dh back into DataHeader freelist.

          Caller must not use dh after call to Free.

          func (*DataHeader) Len

          func (dh *DataHeader) Len() int64

            Len returns whole data record length.

            func (*DataHeader) Load

            func (dh *DataHeader) Load(r io.ReaderAt, pos int64) error

              Load reads and decodes data record header @ pos.

              Only the data record header is loaded, not data itself. See LoadData for actually loading record's data.

              No prerequisite requirements are made to previous dh state.

              func (*DataHeader) LoadBack

              func (dh *DataHeader) LoadBack(r io.ReaderAt) error

                LoadBack reads and decodes data header for revision linked via back-pointer.

                Prerequisite: dh is loaded and .DataLen == 0.

                If link is to zero (means deleted record) io.EOF is returned.

                func (*DataHeader) LoadBackRef

                func (dh *DataHeader) LoadBackRef(r io.ReaderAt) (backPos int64, err error)

                  LoadBackRef reads data for the data record and decodes it as backpointer reference.

                  Prerequisite: dh loaded and .LenData == 0 (data record with back-pointer).

                  func (*DataHeader) LoadData

                  func (dh *DataHeader) LoadData(r io.ReaderAt) (*mem.Buf, error)

                    LoadData loads data for the data record taking backpointers into account.

                    On success dh state is changed to data header of original data transaction.

                    NOTE "deleted" records are indicated via returning buf with .Data=nil without error.

                    func (*DataHeader) LoadNext

                    func (dh *DataHeader) LoadNext(r io.ReaderAt, txnh *TxnHeader) error

                      LoadNext reads and decodes data header for next data record in the same transaction.

                      Prerequisite: dh .Pos .DataLen are initialized.

                      When there is no more data records: io.EOF is returned.

                      func (*DataHeader) LoadPrevRev

                      func (dh *DataHeader) LoadPrevRev(r io.ReaderAt) error

                        LoadPrevRev reads and decodes previous revision data record header.

                        Prerequisite: dh .Oid .Tid .PrevRevPos are initialized.

                        When there is no previous revision: io.EOF is returned.

                        type FileHeader

                        type FileHeader struct {
                        	Magic [4]byte
                        }

                          FileHeader represents header of whole data file.

                          func (*FileHeader) Load

                          func (fh *FileHeader) Load(r io.ReaderAt) error

                            Load reads and decodes file header.

                            type FileStorage

                            type FileStorage struct {
                            	// contains filtered or unexported fields
                            }

                              FileStorage is a ZODB storage which stores data in simple append-only file organized as transactional log.

                              It is on-disk compatible with FileStorage from ZODB/py.

                              func Open

                              func Open(ctx context.Context, path string, opt *zodb.DriverOptions) (_ *FileStorage, at0 zodb.Tid, err error)

                                Open opens FileStorage @path.

                                func (*FileStorage) Close

                                func (fs *FileStorage) Close() error

                                func (*FileStorage) Iterate

                                func (fs *FileStorage) Iterate(_ context.Context, tidMin, tidMax zodb.Tid) zodb.ITxnIterator

                                  Iterate creates zodb-level iterator for tidMin..tidMax range.

                                  func (*FileStorage) LastOid

                                  func (fs *FileStorage) LastOid(_ context.Context) (zodb.Oid, error)

                                  func (*FileStorage) Load

                                  func (fs *FileStorage) Load(_ context.Context, xid zodb.Xid) (buf *mem.Buf, serial zodb.Tid, err error)

                                  func (*FileStorage) Load_XXXWithNextSerialXXX

                                  func (fs *FileStorage) Load_XXXWithNextSerialXXX(_ context.Context, xid zodb.Xid) (buf *mem.Buf, serial, nextSerial zodb.Tid, err error)

                                    XXX temporary function - will go away:

                                    FIXME kill Load_XXXWithNextSerialXXX after neo/py cache does not depend on next_serial https://github.com/zopefoundation/ZODB/pull/323

                                    func (*FileStorage) Sync

                                    func (fs *FileStorage) Sync(ctx context.Context) (head zodb.Tid, err error)

                                      Sync implements zodb.IStorageDriver.

                                      func (*FileStorage) URL

                                      func (fs *FileStorage) URL() string

                                      type Index

                                      type Index struct {
                                      	// this index covers data file up to < .TopPos
                                      	// usually for whole-file index TopPos is position pointing just past
                                      	// the last committed transaction.
                                      	TopPos int64
                                      
                                      	*fsb.Tree
                                      }

                                        Index is in-RAM Oid -> Data record position mapping used to associate Oid with Data record in latest transaction which changed it.

                                        func BuildIndex

                                        func BuildIndex(ctx context.Context, r io.ReaderAt, progress func(*IndexUpdateProgress)) (*Index, error)

                                          BuildIndex builds new in-memory index for data in r.

                                          non-nil valid and consistent index is always returned - even in case of error the index will describe data till top-position of highest transaction that could be read without error.

                                          In such cases the index building could be retried to be finished with index.Update().

                                          func BuildIndexForFile

                                          func BuildIndexForFile(ctx context.Context, path string, progress func(*IndexUpdateProgress)) (index *Index, err error)

                                            BuildIndexForFile builds new in-memory index for data in file @ path.

                                            See BuildIndex for semantic description.

                                            func IndexNew

                                            func IndexNew() *Index

                                              IndexNew creates new empty index.

                                              func LoadIndex

                                              func LoadIndex(r io.Reader) (fsi *Index, err error)

                                                LoadIndex loads index from a reader.

                                                func LoadIndexFile

                                                func LoadIndexFile(path string) (fsi *Index, err error)

                                                  LoadIndexFile loads index from a file @ path.

                                                  func (*Index) Equal

                                                  func (a *Index) Equal(b *Index) bool

                                                    Equal returns whether two indices are the same.

                                                    func (*Index) Save

                                                    func (fsi *Index) Save(w io.Writer) (err error)

                                                      Save saves index to a writer.

                                                      func (*Index) SaveFile

                                                      func (fsi *Index) SaveFile(path string) error

                                                        SaveFile saves index to a file @ path.

                                                        Index data is first saved to a temporary file and when complete the temporary is renamed to be at requested path. This way file @ path will be updated only with complete index data.

                                                        func (*Index) Update

                                                        func (index *Index) Update(ctx context.Context, r io.ReaderAt, topPos int64, progress func(*IndexUpdateProgress)) (err error)

                                                          Update updates in-memory index from r's FileStorage data in byte-range index.TopPos..topPos

                                                          The use case is: we have index computed till some position; we open FileStorage and see there is more data; we update index from data range not-yet covered by the index.

                                                          topPos=-1 means data range to update from is index.TopPos..EOF

                                                          The index stays valid even in case of error - then index is updated but only partially. The index always stays consistent as updates to it are applied as a whole once per data transaction. On return index.TopPos indicates till which position in data the index could be updated.

                                                          On success returned error is nil and index.TopPos is set to either:

                                                          - topPos (if it is != -1), or
                                                          - r's position at which read got EOF (if topPos=-1).
                                                          

                                                          func (*Index) Verify

                                                          func (index *Index) Verify(ctx context.Context, r io.ReaderAt, ntxn int, progress func(*IndexVerifyProgress)) (oidChecked map[zodb.Oid]struct{}, err error)

                                                            Verify checks index correctness against FileStorage data in r.

                                                            For ntxn transactions starting from index.TopPos backwards, it verifies whether oid there have correct entries in the index.

                                                            ntxn=-1 means data range to verify is till start of the file.

                                                            If whole data file was covered (either ntxn is big enough or was set = -1) additional checks are performed to make sure there is no extra entries in the index. For whole-data file cases Verify thus checks whether index is exactly the same as if it was build anew for data in range ..index.TopPos .

                                                            Returned error is either:

                                                            - of type *IndexCorruptError, when data in index was found not to match original data, or
                                                            - any other error type representing e.g. IO error when reading original data or something else.
                                                            

                                                            func (*Index) VerifyForFile

                                                            func (index *Index) VerifyForFile(ctx context.Context, path string, ntxn int, progress func(*IndexVerifyProgress)) (oidChecked map[zodb.Oid]struct{}, err error)

                                                              VerifyForFile checks index correctness against FileStorage data in file @ path.

                                                              See Verify for semantic description.

                                                              type IndexCorruptError

                                                              type IndexCorruptError struct {
                                                              	DataFileName string // present if data IO object was with .Name()
                                                              	Detail       string
                                                              }

                                                                IndexCorruptError is the error type returned by index verification routines when index was found to not match original FileStorage data.

                                                                func (*IndexCorruptError) Error

                                                                func (e *IndexCorruptError) Error() string

                                                                type IndexLoadError

                                                                type IndexLoadError struct {
                                                                	Filename string // present if used IO object was with .Name()
                                                                	Pos      int64
                                                                	Err      error
                                                                }

                                                                  IndexLoadError is the error type returned by index load routines.

                                                                  func (*IndexLoadError) Error

                                                                  func (e *IndexLoadError) Error() string

                                                                  type IndexSaveError

                                                                  type IndexSaveError struct {
                                                                  	Err error // error that occurred during the operation
                                                                  }

                                                                    IndexSaveError is the error type returned by index save routines.

                                                                    func (*IndexSaveError) Error

                                                                    func (e *IndexSaveError) Error() string

                                                                    type IndexUpdateProgress

                                                                    type IndexUpdateProgress struct {
                                                                    	TopPos     int64  // data range to update to; if = -1 -- till EOF
                                                                    	TxnIndexed int    // # transactions read/indexed so far
                                                                    	Index      *Index // index built so far
                                                                    	Iter       *Iter  // iterator through data	XXX needed?
                                                                    }

                                                                      IndexUpdateProgress is data sent by Index.Update to notify about progress.

                                                                      type IndexVerifyProgress

                                                                      type IndexVerifyProgress struct {
                                                                      	TxnTotal   int // total # of transactions to verify; if = -1 -- whole data
                                                                      	TxnChecked int
                                                                      	Index      *Index                // index verification runs for
                                                                      	Iter       *Iter                 // iterator through data
                                                                      	OidChecked map[zodb.Oid]struct{} // oid checked so far
                                                                      }

                                                                        IndexVerifyProgress is data sent by Index.Verify to notify about progress.

                                                                        type Iter

                                                                        type Iter struct {
                                                                        	R   io.ReaderAt
                                                                        	Dir IterDir
                                                                        
                                                                        	Txnh  TxnHeader  // current transaction record information
                                                                        	Datah DataHeader // current data record information
                                                                        }

                                                                          Iter is combined 2-level iterator over transaction and data records.

                                                                          func Iterate

                                                                          func Iterate(r io.ReaderAt, posStart int64, dir IterDir) *Iter

                                                                            Iterate creates Iter to iterate over r starting from posStart in direction dir.

                                                                            func IterateFile

                                                                            func IterateFile(path string, dir IterDir) (iter *Iter, file *os.File, err error)

                                                                              IterateFile opens file @ path read-only and creates Iter to iterate over it.

                                                                              The iteration will use buffering over os.File optimized for sequential access. You are responsible to eventually close the file after the iteration is done.

                                                                              func (*Iter) NextData

                                                                              func (it *Iter) NextData() error

                                                                                NextData iterates to next data record header inside current transaction.

                                                                                func (*Iter) NextTxn

                                                                                func (it *Iter) NextTxn(flags TxnLoadFlags) error

                                                                                  NextTxn iterates to next/previous transaction record according to iteration direction.

                                                                                  The data header is reset to iterate inside transaction record that becomes current.

                                                                                  type IterDir

                                                                                  type IterDir int
                                                                                  const (
                                                                                  	IterForward IterDir = iota
                                                                                  	IterBackward
                                                                                  )

                                                                                  type RecordError

                                                                                  type RecordError struct {
                                                                                  	Path   string // path of the data file; present if used IO object was with .Name()
                                                                                  	Record string // record kind - "file header", "transaction record", "data record", ...
                                                                                  	Pos    int64  // position of record
                                                                                  	Subj   string // subject context for the error - e.g. "read" or "check"
                                                                                  	Err    error  // actual error
                                                                                  }

                                                                                    RecordError represents error associated with operation on a record in FileStorage data file.

                                                                                    func (*RecordError) Cause

                                                                                    func (e *RecordError) Cause() error

                                                                                    func (*RecordError) Error

                                                                                    func (e *RecordError) Error() string

                                                                                    func (*RecordError) Unwrap

                                                                                    func (e *RecordError) Unwrap() error

                                                                                    type TxnHeader

                                                                                    type TxnHeader struct {
                                                                                    	Pos     int64 // position of transaction start
                                                                                    	LenPrev int64 // whole previous transaction record length (see EOF/error rules in Load)
                                                                                    	Len     int64 // whole transaction record length          (see EOF/error rules in Load)
                                                                                    
                                                                                    	// transaction metadata itself
                                                                                    	zodb.TxnInfo
                                                                                    	// contains filtered or unexported fields
                                                                                    }

                                                                                      TxnHeader represents header of a transaction record.

                                                                                      func (*TxnHeader) CloneFrom

                                                                                      func (txnh *TxnHeader) CloneFrom(txnh2 *TxnHeader)

                                                                                        CloneFrom copies txnh2 to txnh making sure underlying slices (.workMem .User .Desc ...) are not shared.

                                                                                        func (*TxnHeader) DataLen

                                                                                        func (txnh *TxnHeader) DataLen() int64

                                                                                          DataLen returns length of all data inside transaction record container.

                                                                                          func (*TxnHeader) DataPos

                                                                                          func (txnh *TxnHeader) DataPos() int64

                                                                                            DataPos returns start position of data inside transaction record.

                                                                                            func (*TxnHeader) HeaderLen

                                                                                            func (txnh *TxnHeader) HeaderLen() int64

                                                                                              HeaderLen returns whole transaction header length including its variable part.

                                                                                              NOTE: data records start right after transaction header.

                                                                                              func (*TxnHeader) Load

                                                                                              func (txnh *TxnHeader) Load(r io.ReaderAt, pos int64, flags TxnLoadFlags) error

                                                                                                Load reads and decodes transaction record header @ pos.

                                                                                                Both transaction header starting at pos, and redundant length of previous transaction are loaded. The data read is verified for consistency lightly.

                                                                                                No prerequisite requirements are made to previous txnh state.

                                                                                                Rules for .Len/.LenPrev returns:

                                                                                                .Len = -1	transaction header could not be read
                                                                                                .Len = 0	EOF forward
                                                                                                .Len > 0 	transaction was read normally
                                                                                                
                                                                                                .LenPrev = -1	prev record length could not be read
                                                                                                .LenPrev = 0	EOF backward
                                                                                                .LenPrev > 0	LenPrev was read/checked normally
                                                                                                

                                                                                                For example when pos points to the end of file .Len will be returned = -1, but .LenPrev will be usually valid if file has at least 1 transaction.

                                                                                                func (*TxnHeader) LoadNext

                                                                                                func (txnh *TxnHeader) LoadNext(r io.ReaderAt, flags TxnLoadFlags) error

                                                                                                  LoadNext reads and decodes next transaction record header.

                                                                                                  prerequisite: txnh .Pos, .Len and .Tid should be already initialized.

                                                                                                  func (*TxnHeader) LoadPrev

                                                                                                  func (txnh *TxnHeader) LoadPrev(r io.ReaderAt, flags TxnLoadFlags) error

                                                                                                    LoadPrev reads and decodes previous transaction record header.

                                                                                                    prerequisites:

                                                                                                    - txnh .Pos, .LenPrev are initialized
                                                                                                    - optionally if .Len is initialized and txnh was loaded - tid↓ will be also checked
                                                                                                    

                                                                                                    type TxnLoadFlags

                                                                                                    type TxnLoadFlags int

                                                                                                      flags for TxnHeader.Load

                                                                                                      const (
                                                                                                      	LoadAll       TxnLoadFlags = 0x00 // load whole transaction header
                                                                                                      	LoadNoStrings              = 0x01 // do not load user/desc/ext strings
                                                                                                      )

                                                                                                      Directories

                                                                                                      Path Synopsis
                                                                                                      cmd
                                                                                                      fs1
                                                                                                      fs1 is a driver program for running and invoking fs1 subcommands.
                                                                                                      fs1 is a driver program for running and invoking fs1 subcommands.
                                                                                                      Package fs1tools provides tools for managing and maintaining ZODB FileStorage v1 databases.
                                                                                                      Package fs1tools provides tools for managing and maintaining ZODB FileStorage v1 databases.
                                                                                                      Package fsb specializes cznic/b.Tree for FileStorage index needs.
                                                                                                      Package fsb specializes cznic/b.Tree for FileStorage index needs.