fsmgr

Documentation

Overview

Package fsmgr manages fs objects

Index

• Constants
• Variables
• func Exists(repoID string, objID string) (bool, error)
• func GetObjIDByPath(repoID, rootID, path string) (string, uint32, error)
• func GetOneZlibReader() io.ReadCloser
• func GetSeafdirIDByPath(repoID, rootID, path string) (string, error)
• func Init(seafileConfPath string, seafileDataDir string, fsCacheLimit int64)
• func IsDir(m uint32) bool
• func IsRegular(m uint32) bool
• func ReadRaw(repoID string, objID string, w io.Writer) error
• func ReturnOneZlibReader(reader io.ReadCloser)
• func SaveSeafdir(repoID string, seafdir *SeafDir) error
• func SaveSeafile(repoID string, seafile *Seafile) error
• func VerifyObjectID(objID string, compressed []byte, reader io.ReadCloser) error
• func WriteRaw(repoID string, objID string, r io.Reader) error
• func WriteRawIngested(repoID string, objID string, data []byte, reader io.ReadCloser) error
• type FileCountInfo
  • func GetFileCountInfoByPath(repoID, rootID, path string) (*FileCountInfo, error)
• type SeafDir
  • func GetSeafdir(repoID string, dirID string) (*SeafDir, error)
  • func GetSeafdirByPath(repoID string, rootID string, path string) (*SeafDir, error)
  • func GetSeafdirWithZlibReader(repoID string, dirID string, reader io.ReadCloser) (*SeafDir, error)
  • func NewSeafdir(version int, entries []*SeafDirent) (*SeafDir, error)
  • func (seafdir *SeafDir) FromData(p []byte, reader io.ReadCloser) error
  • func (seafdir *SeafDir) ToData(w io.Writer) error
• type SeafDirent
  • func GetDirentByPath(repoID, rootID, rpath string) (*SeafDirent, error)
  • func NewDirent(id string, name string, mode uint32, mtime int64, modifier string, size int64) *SeafDirent
• type Seafile
  • func GetSeafile(repoID string, fileID string) (*Seafile, error)
  • func GetSeafileWithZlibReader(repoID string, fileID string, reader io.ReadCloser) (*Seafile, error)
  • func NewSeafile(version int, fileSize int64, blkIDs []string) (*Seafile, error)
  • func (seafile *Seafile) FromData(p []byte, reader io.ReadCloser) error
  • func (seafile *Seafile) ToData(w io.Writer) error

Constants

const (
	SeafMetadataTypeInvalid = iota
	SeafMetadataTypeFile
	SeafMetadataTypeLink
	SeafMetadataTypeDir
)

Meta data type of dir or file

const (
	EmptySha1 = "0000000000000000000000000000000000000000"
)

Empty value of sha1

const MaxObjectSize = 64 << 20

MaxObjectSize bounds how much an fs object may expand to when decompressed.

Without a bound, a few kilobytes of zlib expand to gigabytes and the copy into memory OOM-kills the fileserver. The compressed bytes arrive from a client — recvFSCB stores them without decompressing or verifying that the object ID matches their hash — and are decompressed later, when the commit that references them is processed.

64MB is far above anything real. An fs object is a JSON block list, so the largest legitimate one belongs to the largest single file: at the 8MB fixed block size and ~43 bytes per block ID, 64MB of block list describes a file of roughly 12TB.

Variables

var ErrPathNoExist = fmt.Errorf("path does not exist")

ErrPathNoExist is an error indicating that the file does not exist

var ErrVerification = errors.New("fs object failed verification")

ErrVerification marks an object that failed the check against its own id. It says the data is wrong, not that the server is: callers translating this to a status code should answer 400 rather than 500.

Functions

func Exists

func Exists(repoID string, objID string) (bool, error)

Exists check if fs object is exists.

func GetObjIDByPath

func GetObjIDByPath(repoID, rootID, path string) (string, uint32, error)

GetObjIDByPath gets the obj id by path

func GetOneZlibReader

func GetOneZlibReader() io.ReadCloser

GetOneZlibReader gets a zlib reader from zlibReaders.

func GetSeafdirIDByPath

func GetSeafdirIDByPath(repoID, rootID, path string) (string, error)

GetSeafdirIDByPath gets the dirID of SeafDir by path.

func Init

func Init(seafileConfPath string, seafileDataDir string, fsCacheLimit int64)

Init initializes fs manager and creates underlying object store.

func IsDir

func IsDir(m uint32) bool

IsDir check if the mode is dir.

func IsRegular

func IsRegular(m uint32) bool

IsRegular Check if the mode is regular.

func ReadRaw

func ReadRaw(repoID string, objID string, w io.Writer) error

ReadRaw reads data in binary format from storage backend.

func ReturnOneZlibReader

func ReturnOneZlibReader(reader io.ReadCloser)

func SaveSeafdir

func SaveSeafdir(repoID string, seafdir *SeafDir) error

SaveSeafdir saves seafdir to storage backend.

func SaveSeafile

func SaveSeafile(repoID string, seafile *Seafile) error

SaveSeafile saves seafile to storage backend.

func VerifyObjectID

func VerifyObjectID(objID string, compressed []byte, reader io.ReadCloser) error

VerifyObjectID checks that a compressed fs object is the one its id names.

Unlike a block, whose id is the SHA-1 of the bytes stored, an fs object's id is the SHA-1 of its *uncompressed* JSON while the wire and the store both carry the compressed form — so verifying one costs an inflate. That is why this is behind option.VerifyFSObjectHashes rather than unconditional.

It goes through the same bounded decompression as every other read, so a zlib bomb cannot ride in through the check that exists to reject bad data.

reader may be nil; a caller verifying a run of objects should pass one from GetOneZlibReader so the whole run shares a single inflate window instead of allocating one per object.

func WriteRaw

func WriteRaw(repoID string, objID string, r io.Reader) error

WriteRaw writes data in binary format to storage backend.

func WriteRawIngested

func WriteRawIngested(repoID string, objID string, data []byte, reader io.ReadCloser) error

WriteRawIngested stores a compressed fs object that arrived from a client, checking it against its id first when option.VerifyFSObjectHashes is on.

The option is honoured here rather than at the handler for the same reason SyncObjectWrites is: both describe how an object reaches the store, so a second ingest path should not be able to skip either by forgetting to ask. It is separate from WriteRaw because the server's own writers derive the id from the very bytes they are about to write — re-inflating those to confirm what is true by construction would be the one case where the check buys nothing.

reader may be nil; see VerifyObjectID.

Types

type FileCountInfo

type FileCountInfo struct {
	FileCount int64
	Size      int64
	DirCount  int64
}

FileCountInfo contains information of files

func GetFileCountInfoByPath

func GetFileCountInfoByPath(repoID, rootID, path string) (*FileCountInfo, error)

GetFileCountInfoByPath gets the count info of file by path.

type SeafDir

type SeafDir struct {
	Version int           `json:"version"`
	DirType int           `json:"type"`
	DirID   string        `json:"-"`
	Entries []*SeafDirent `json:"dirents"`
	// contains filtered or unexported fields
}

SeafDir is a dir object

func GetSeafdir

func GetSeafdir(repoID string, dirID string) (*SeafDir, error)

GetSeafdir gets seafdir from storage backend.

func GetSeafdirByPath

func GetSeafdirByPath(repoID string, rootID string, path string) (*SeafDir, error)

GetSeafdirByPath gets the object of seafdir by path.

func GetSeafdirWithZlibReader

func GetSeafdirWithZlibReader(repoID string, dirID string, reader io.ReadCloser) (*SeafDir, error)

GetSeafdir gets seafdir from storage backend with a zlib reader.

func NewSeafdir

func NewSeafdir(version int, entries []*SeafDirent) (*SeafDir, error)

NewSeafdir initializes a SeafDir object

func (*SeafDir) FromData

func (seafdir *SeafDir) FromData(p []byte, reader io.ReadCloser) error

FromData reads from p and converts JSON-encoded data to SeafDir.

func (*SeafDir) ToData

func (seafdir *SeafDir) ToData(w io.Writer) error

ToData converts seafdir to JSON-encoded data and writes to w.

type SeafDirent

type SeafDirent struct {
	Mode     uint32 `json:"mode"`
	ID       string `json:"id"`
	Name     string `json:"name"`
	Mtime    int64  `json:"mtime"`
	Modifier string `json:"modifier"`
	Size     int64  `json:"size"`
}

SeafDirent is a dir entry object

func GetDirentByPath

func GetDirentByPath(repoID, rootID, rpath string) (*SeafDirent, error)

func NewDirent

func NewDirent(id string, name string, mode uint32, mtime int64, modifier string, size int64) *SeafDirent

NewDirent initializes a SeafDirent object

type Seafile

type Seafile struct {
	Version  int      `json:"version"`
	FileType int      `json:"type"`
	FileID   string   `json:"-"`
	FileSize uint64   `json:"size"`
	BlkIDs   []string `json:"block_ids"`
	// contains filtered or unexported fields
}

Seafile is a file object

func GetSeafile

func GetSeafile(repoID string, fileID string) (*Seafile, error)

GetSeafile gets seafile from storage backend.

func GetSeafileWithZlibReader

func GetSeafileWithZlibReader(repoID string, fileID string, reader io.ReadCloser) (*Seafile, error)

GetSeafileWithZlibReader gets seafile from storage backend with a zlib reader.

func NewSeafile

func NewSeafile(version int, fileSize int64, blkIDs []string) (*Seafile, error)

NewSeafile initializes a Seafile object

func (*Seafile) FromData

func (seafile *Seafile) FromData(p []byte, reader io.ReadCloser) error

FromData reads from p and converts JSON-encoded data to Seafile.

func (*Seafile) ToData

func (seafile *Seafile) ToData(w io.Writer) error

ToData converts seafile to JSON-encoded data and writes to w.