gotfs

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 24, 2026 License: GPL-3.0 Imports: 25 Imported by: 0

README

GotFS

GotFS is a filesystem implemented on top of a key-value store. The key value store is GotKV, which is an immutable key-value store, sitting on top of a content-addressed data store. Every operation on a GotKV store returns a new reference to a new store with the change. GotFS inherits this property. All operations return a reference to a new filesystem. A GotFS filesystem is just a GotKV store with a specific key structure used to represent files and directories.

Objects

Ref

A reference to data in the content-addressed store. This type is provided by GotKV.

Extent

A part of a file. It includes a Ref, an offset, and a length. It is basically an instruction to:

  1. Get the referenced data from content-addressed storage
  2. Only include the data from offset to offset+length.

This enables packing small files into the same blob.

Info

Information about a file or directory. Most importantly the permissions and type of file.

Key Layout

All objects are represented by an Info entry at a specific key, and content stored under keys prefixed with the metadata key.

File data is stored in a content-addressed store, and references to the data are stored in GotKV.

In the following examples <0> will be used to represent a single 0 (NULL) byte.

Example: 1 File

For example: The file "test.txt" with 10B of data in it would produce the following key value pairs.

<0>< 64 bit: 0  >            -> Info (dir)
<0>test.txt<0>< 64 bit: 0  >   -> Info (file)
<0>test.txt<0>< 64 bit: 10 >   -> Extent
Example: 2 File + 1 Directory

A directory is stored as a metadata object.

<0>< 64 bit: 0  >                     -> Info (dir)
<0>mydir<0>< 64 bit: 0 >                -> Info (dir)
<0>mydir<0>myfile.txt<0>< 64 bit: 0     >  -> Info (file)
<0>mydir<0>myfile.txt<0>< 64 bit offset >  -> Part
Example 3: File at the Root

It is possible for a file to be at the root

<0>< 64 bit: 0      >       -> Info (file)
<0>< 64 bits extent >       -> Extent
<0>< next offset    >       -> Extent

Reading A File

To read from a file in GotFS you first lookup the Info entry for the path of the file. If there is not an entry at the path or the entry is not for a regular file, then return an error.

Then convert the offset to read from to a key. Seek to the first entry after that key. The extent referenced by that entry will end at the offset in the key, and will contain data overlapping the target range.

Documentation

Index

Constants

View Source 
const (
	DefaultMaxBlobSize          = 1 << 21
	DefaultMinBlobSizeData      = 1 << 12
	DefaultMeanBlobSizeData     = 1 << 20
	DefaultMeanBlobSizeMetadata = 1 << 13
)
View Source 
const MaxPathLen = gotkv.MaxKeySize - 9
View Source 
const RootSize = gdat.RefSize + 1
View Source 
const Sep = '/'

Variables

This section is empty.

Functions

func Dump 

func Dump(ctx context.Context, s stores.Reading, root Root, w io.Writer) error

func Equal 

func Equal(a, b Root) bool

Equal returns true if a and b contain equivalent data.

func SpanForPath 

func SpanForPath(p string) gotkv.Span

func SplitPath 

func SplitPath(p string) []string

Types

type Builder 

type Builder struct {
	// contains filtered or unexported fields
}

Builder manages building a filesystem.

func (*Builder) BeginFile 

func (b *Builder) BeginFile(p string, mode os.FileMode) error

BeginFile creates a metadata entry for a regular file at p and directs Write calls to the content of that file.

func (*Builder) Finish 

func (b *Builder) Finish() (*Root, error)

Finish closes the builder and returns the Root to the filesystem. Finish is idempotent, and is safe to call multiple times. Not calling finish is not an error, the builder does not allocate resources other than memory.

func (*Builder) IsFinished 

func (b *Builder) IsFinished() bool

func (*Builder) Mkdir 

func (b *Builder) Mkdir(p string, mode os.FileMode) error

Mkdir creates a directory for p.

func (*Builder) Write 

func (b *Builder) Write(data []byte) (int, error)

type Delta 

type Delta gotkv.Root

Delta is the different between two Roots

type DeltaBuilder 

type DeltaBuilder struct {
	// contains filtered or unexported fields
}

func (*DeltaBuilder) Delete 

func (db *DeltaBuilder) Delete(ctx context.Context, p string) error

func (*DeltaBuilder) Finish 

func (db *DeltaBuilder) Finish(ctx context.Context) (*Delta, error)

func (*DeltaBuilder) PutInfo 

func (db *DeltaBuilder) PutInfo(ctx context.Context, p string, info *Info) error

type DeltaEntry 

type DeltaEntry struct {
	Path string

	PutInfo    *Info
	Delete     *struct{}
	PutContent *PutContent
}

type DeltaIterator 

type DeltaIterator struct {
	// contains filtered or unexported fields
}

func (*DeltaIterator) Next 

func (di *DeltaIterator) Next(ctx context.Context, dsts []DeltaEntry) (int, error)

type Differ 

type Differ struct {
	// contains filtered or unexported fields
}

func (*Differ) Next 

func (d *Differ) Next(ctx context.Context, dsts []DeltaEntry) (int, error)

type DirEnt 

type DirEnt struct {
	Name string
	Mode os.FileMode
}

type Expr 

type Expr struct {
	// Root is the filesystem to copy from
	Root Root
	// AddPrefix is applied to Root before copying
	AddPrefix string
}

type Extent 

type Extent = gotlob.Extent

type GetPostExister 

type GetPostExister interface {
	cadata.Getter
	cadata.PostExister
}

type Info 

type Info struct {
	Mode  fs.FileMode
	Attrs map[string][]byte
}

type Key 

type Key struct {
	// contains filtered or unexported fields
}

Key is a key that GotFS stores in GotKV. It contains a path and either an Extent end offset or an Info sentinel value.

func (Key) ChildrenSpan 

func (k Key) ChildrenSpan() gotkv.Span

ChildrenSpan returns a span that contains all children or the path if it was a directory

func (*Key) EndAt 

func (k *Key) EndAt() uint64

EndAt returns the ending offset for an Extent

func (*Key) IsInfo 

func (k *Key) IsInfo() bool

IsInfo returns true if the key is for an Info object

func (Key) Marshal 

func (k Key) Marshal(out []byte) []byte

func (*Key) Path 

func (k *Key) Path() string

Path returns the path associated with the key.

func (Key) Prefix 

func (k Key) Prefix(out []byte) []byte

Prefix returns a prefix which all keys for this path, including Infos and Extents will have. The prefix will also include any children of the object.

func (*Key) Unmarshal 

func (k *Key) Unmarshal(data []byte) error

type Machine 

type Machine struct {
	// contains filtered or unexported fields
}

func NewMachine 

func NewMachine(opts ...Option) *Machine

func (*Machine) Check 

func (mach *Machine) Check(ctx context.Context, s stores.Reading, root Root, checkData func(ref gdat.Ref) error) error

func (*Machine) CreateFile 

func (mach *Machine) CreateFile(ctx context.Context, ss [2]stores.RW, x Root, p string, r io.Reader) (*Root, error)

CreateFile creates a file at p with data from r If there is an entry at p CreateFile returns an error ms is the store used for metadata ds is the store used for data.

func (*Machine) Exists 

func (mach *Machine) Exists(ctx context.Context, ms stores.Reading, root Root, p string) (bool, error)

func (*Machine) FileFromReader 

func (mach *Machine) FileFromReader(ctx context.Context, ss [2]stores.RW, mode posixfs.FileMode, r io.Reader) (*Root, error)

func (*Machine) FileFromReaders 

func (mach *Machine) FileFromReaders(ctx context.Context, ss [2]stores.RW, mode posixfs.FileMode, rs []io.Reader) (*Root, error)

ImportReaders creates a single file at the root from concatenating the data in rs. Each reader will be imported from in parallel.

func (*Machine) ForEach 

func (mach *Machine) ForEach(ctx context.Context, s stores.Reading, root Root, p string, fn func(p string, md *Info) error) error

func (*Machine) ForEachLeaf 

func (mach *Machine) ForEachLeaf(ctx context.Context, s stores.Reading, root Root, p string, fn func(p string, md *Info) error) error

ForEachLeaf calls fn with each regular file in root, beneath p.

func (*Machine) GetDirInfo 

func (mach *Machine) GetDirInfo(ctx context.Context, s stores.Reading, x Root, p string) (*Info, error)

GetDirInfo returns directory metadata at p if it exists, and errors otherwise

func (*Machine) GetFileInfo 

func (mach *Machine) GetFileInfo(ctx context.Context, s stores.Reading, x Root, p string) (*Info, error)

GetFileInfo returns the file metadata at p if it exists, and errors otherwise

func (*Machine) GetInfo 

func (mach *Machine) GetInfo(ctx context.Context, s stores.Reading, x Root, p string) (*Info, error)

GetInfo retrieves the metadata at p if it exists and errors otherwise

func (*Machine) Graft 

func (mach *Machine) Graft(ctx context.Context, ss [2]stores.RW, root Root, p string, branch Root) (*Root, error)

Graft places branch at p in root. If p == "" then branch is returned unaltered.

func (*Machine) MaxInfo 

func (mach *Machine) MaxInfo(ctx context.Context, ms stores.Reading, root Root, span Span) (string, *Info, error)

MaxInfo returns the maximum path and the corresponding Info for the path. If no Info entry can be found MaxInfo returns ("", nil, nil)

func (*Machine) MeanBlobSizeData 

func (mach *Machine) MeanBlobSizeData() int

func (*Machine) MeanBlobSizeMetadata 

func (mach *Machine) MeanBlobSizeMetadata() int

func (*Machine) Mkdir 

func (mach *Machine) Mkdir(ctx context.Context, s stores.RW, x Root, p string) (*Root, error)

Mkdir creates a directory at path p

func (*Machine) MkdirAll 

func (mach *Machine) MkdirAll(ctx context.Context, s stores.RW, x Root, p string) (*Root, error)

MkdirAll creates the directory p and any of p's ancestors if necessary.

func (*Machine) NewBuilder 

func (mach *Machine) NewBuilder(ctx context.Context, ms, ds stores.RW) *Builder

func (*Machine) NewDeltaBuilder 

func (mach *Machine) NewDeltaBuilder(ms, ds stores.RW) *DeltaBuilder

func (*Machine) NewDeltaIterator 

func (mach *Machine) NewDeltaIterator(ms, ds stores.Reading, delta Delta) *DeltaIterator

func (*Machine) NewDiffer 

func (mach *Machine) NewDiffer(ms stores.Reading, left, right Root) *Differ

func (*Machine) NewEmpty 

func (mach *Machine) NewEmpty(ctx context.Context, s stores.RW) (*Root, error)

NewEmpty creates a new filesystem with an empty root directory

func (*Machine) NewReader 

func (mach *Machine) NewReader(ctx context.Context, ss [2]stores.Reading, x Root, p string) (*Reader, error)

NewReader returns an io.Reader | io.Seeker | io.ReaderAt

func (*Machine) Populate 

func (mach *Machine) Populate(ctx context.Context, s stores.Reading, root Root, mdSet, dataSet cadata.Set) error

Populate adds the ID for all the metadata blobs to mdSet and all the data blobs to dataSet

func (*Machine) PutFile 

func (mach *Machine) PutFile(ctx context.Context, ss [2]stores.RW, x Root, p string, r io.Reader) (*Root, error)

PutFile creates or replaces the file at path using data from r

func (*Machine) PutInfo 

func (mach *Machine) PutInfo(ctx context.Context, s stores.RW, x Root, p string, md *Info) (*Root, error)

PutInfo assigns metadata to p

func (*Machine) ReadDir 

func (mach *Machine) ReadDir(ctx context.Context, s stores.Reading, x Root, p string, fn func(e DirEnt) error) error

ReadDir calls fn for every child of the directory at p.

func (*Machine) ReadFile 

func (mach *Machine) ReadFile(ctx context.Context, ss [2]stores.Reading, x Root, p string, max int) ([]byte, error)

func (*Machine) ReadFileAt 

func (mach *Machine) ReadFileAt(ctx context.Context, ss [2]stores.Reading, x Root, p string, start int64, buf []byte) (int, error)

ReadFileAt fills `buf` with data in the file at `p` starting at offset `start`

func (*Machine) RemoveAll 

func (mach *Machine) RemoveAll(ctx context.Context, s Store, x Root, p string) (*Root, error)

func (*Machine) Select 

func (mach *Machine) Select(ctx context.Context, s stores.RW, root Root, p string) (*Root, error)

Select returns a new root containing everything under p, shifted to the root.

func (*Machine) SizeOfFile 

func (mach *Machine) SizeOfFile(ctx context.Context, s stores.Reading, x Root, p string) (uint64, error)

SizeOfFile returns the size of the file at p in bytes.

func (*Machine) Splice 

func (mach *Machine) Splice(ctx context.Context, ss [2]stores.RW, segs []Segment) (*Root, error)

func (*Machine) Sync 

func (mach *Machine) Sync(ctx context.Context, src [2]stores.Reading, dst [2]stores.Writing, root Root) error

Sync ensures dst has all the data reachable from root dst and src should both be metadata stores. copyData will be called to sync metadata

type Option 

type Option func(a *Machine)

func WithContentCacheSize 

func WithContentCacheSize(n int) Option

WithContentCacheSize sets the size of the cache for raw data

func WithMetaCacheSize 

func WithMetaCacheSize(n int) Option

WithMetaCacheSize sets the size of the cache for metadata

func WithSalt 

func WithSalt(salt *[32]byte) Option

type PutContent 

type PutContent struct {
	Begin, End uint64
	Extents    []Extent
}

type Reader 

type Reader = gotlob.Reader

type Ref 

type Ref = gotkv.Ref

type Root 

type Root struct {
	Ref   Ref   `json:"ref"`
	Depth uint8 `json:"depth"`
}

func ParseRoot 

func ParseRoot(data []byte) (*Root, error)

func (Root) Marshal 

func (r Root) Marshal(out []byte) []byte

Marshal appends the root data to out and returns the new slice.

func (Root) ToGotKV 

func (r Root) ToGotKV() gotkv.Root

func (*Root) Unmarshal 

func (r *Root) Unmarshal(data []byte) error

Unmarshal parses the root data from data and returns an error if the data is invalid.

type Segment 

type Segment struct {
	// Span is the span in the final Splice operation
	Span gotkv.Span
	// Contents is what will go in the Span.
	Contents Expr
}

Segment is a span of a GotFS instance.

func ChangesOnBase 

func ChangesOnBase(base Root, changes []Segment) []Segment

ChangesOnBase inserts segments from base between each Segment in changes.

func (Segment) String 

func (s Segment) String() string

type Span 

type Span = gotkv.Span

type Store 

type Store = gotkv.Store

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.