glfs

package module
v0.0.0-...-7fe8f5e Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 6, 2025 License: MPL-2.0 Imports: 22 Imported by: 3

README

GLFS

GLFS is a Git-Like FileSystem.

If you are looking for a simple way to store objects larger than a single blob in blobcache, you have come to the right place.

Take a look at ARCHITECTURE.md to learn more about the design. If you have a good understanding of how Git works, that's about 9/10 of it. There are a few tricks not in Git.

Examples live in the examples/ directory.

Documentation

Index

Constants

View Source 
const (
	TypeBlob = Type("blob")
	TypeTree = Type("tree")
)
View Source 
const DefaultBlockSize = 1 << 21

Variables

This section is empty.

Functions

func CleanPath 

func CleanPath(x string) string

CleanPath removes leading and trailing slashes, and changes "." to ""

func GetBlobBytes 

func GetBlobBytes(ctx context.Context, s cadata.Getter, x Ref, maxSize int) ([]byte, error)

GetBlobBytes reads the entire contents of the blob at x into memory and returns the slice of bytes.

func IsErrNoEnt 

func IsErrNoEnt(err error) bool

func IsValidName 

func IsValidName(x string) bool

IsValidName returns true if x can be used as a TreeEntry name

func Replace 

func Replace(ents []TreeEntry, ent TreeEntry)

Replace replaces the entry in tree with name == ent.Name with ent.

func SizeOf 

func SizeOf(x Ref) uint64

SizeOf returns the size of the data at x

func SortTreeEntries 

func SortTreeEntries(ents []TreeEntry)

func Sync 

func Sync(ctx context.Context, dst cadata.PostExister, src cadata.Getter, x Ref) error

Sync ensures that all data referenced by x exists in dst, copying from src if necessary. Sync assumes there are no dangling references, and skips copying data when its existence is implied.

func ValidateTreeEntries 

func ValidateTreeEntries(ents []TreeEntry) error

func WalkRefs 

func WalkRefs(ctx context.Context, s cadata.Getter, ref Ref, fn RefWalker) error

WalkRefs calls fn with every Ref reacheable from ref, including Ref. The only guarentee about order is bottom up. if a tree is encoutered the child refs will be visited first.

func WalkTree 

func WalkTree(ctx context.Context, store cadata.Getter, ref Ref, f WalkTreeFunc) error

WalkTree walks the tree and calls f with tree entries in lexigraphical order file1.txt comes before file2.txt dir1/ comes before dir1/file1.txt

Types

type AddExister 

type AddExister interface {
	cadata.Adder
	cadata.Exister
}

type BlobMapper 

type BlobMapper func(p string, in io.Reader, out io.Writer) error

type Diff 

type Diff struct {
	Left  *Ref
	Right *Ref
	Both  *Ref
}

Diff contains the result of a Compare Left contains only elements in left Right contains only elements in right Both contains elements common to left and right

type ErrNoEnt 

type ErrNoEnt struct {
	Name string
}

func (ErrNoEnt) Error 

func (e ErrNoEnt) Error() string

type ErrRefType 

type ErrRefType struct {
	Have, Want Type
}

func (ErrRefType) Error 

func (e ErrRefType) Error() string

type GCOption 

type GCOption func(*gcConfig)

type GCResult 

type GCResult struct {
	Reachable uint64
	Scanned   uint64
	Deleted   uint64
}

func GC 

func GC(ctx context.Context, store GetListDeleter, keep []Ref, opts ...GCOption) (*GCResult, error)

GC will remove objects from store which are not referenced by any of the refs in keep. If GC does not successfully complete, referential integrity may be violated, and GC will need to be run to completion before it is safe to call Sync on the store again.

type GetListDeleter 

type GetListDeleter interface {
	cadata.Getter
	cadata.Lister
	cadata.Deleter
}

type Machine 

type Machine struct {
	// contains filtered or unexported fields
}

Machine holds a configuration, and caches. Machine configuration is immutable once it is created. Any cache state should be transparent to the user, so the Machine should appear stateless. Machines are thread-safe.

Creating a new Machine will perform better than using the default Machine when there are many concurrent operations being performed on unrelated filesystems, since the unreleated tasks won't be affecting the same cache.

Repeated lookups within a given filesystem will be much faster when rerun on the same machine. This is because each hop must be read from the store and decrypted, the decrypted plaintext will be cached by the machine.

func NewMachine 

func NewMachine(opts ...Option) *Machine

func (*Machine) Compare 

func (ag *Machine) Compare(ctx context.Context, dst cadata.PostExister, src cadata.Getter, left, right Ref) (*Diff, error)

Compare compares left and right and returns a diff. Left and right must both point only to data in s.

func (*Machine) Concat 

func (ag *Machine) Concat(ctx context.Context, store cadata.Store, layers ...Ref) (*Ref, error)

func (*Machine) FilterPaths 

func (ag *Machine) FilterPaths(ctx context.Context, dst cadata.PostExister, src cadata.Getter, root Ref, f func(string) bool) (*Ref, error)

FilterPaths returns a version of root with paths filtered using f as a predicate. If f returns true for a path it will be included in the output, otherwise it will not.

func (*Machine) GC 

func (ag *Machine) GC(ctx context.Context, store GetListDeleter, keep []Ref, opts ...GCOption) (*GCResult, error)

GC will remove objects from store which are not referenced by any of the refs in keep. If GC does not successfully complete, referential integrity may be violated, and GC will need to be run to completion before it is safe to call Sync on the store again.

func (*Machine) GetAtPath 

func (ag *Machine) GetAtPath(ctx context.Context, store cadata.Getter, ref Ref, subpath string) (*Ref, error)

GetAtPath returns a ref to the object under ref at subpath. ErrNoEnt is returned if there is no entry at that path.

func (*Machine) GetBlob 

func (ag *Machine) GetBlob(ctx context.Context, s cadata.Getter, x Ref) (*Reader, error)

GetBlob returns an io.ReadSeeker for accessing data from the blob at x

func (*Machine) GetBlobBytes 

func (ag *Machine) GetBlobBytes(ctx context.Context, s cadata.Getter, x Ref, maxSize int) ([]byte, error)

GetBlobBytes reads the entire contents of the blob at x into memory and returns the slice of bytes.

func (*Machine) GetTreeSlice 

func (ag *Machine) GetTreeSlice(ctx context.Context, store cadata.Getter, ref Ref, maxEnts int) ([]TreeEntry, error)

GetTree retreives the tree in store at Ref if it exists. If ref.Type != TypeTree ErrRefType is returned.

func (*Machine) GetTyped 

func (ag *Machine) GetTyped(ctx context.Context, s cadata.Getter, ty Type, x Ref) (*Reader, error)

GetTyped retrieves the object in s at x. If x.Type != ty, ErrRefType is returned.

func (*Machine) Lookup 

func (ag *Machine) Lookup(ctx context.Context, store cadata.Getter, ent TreeEntry, subpath string) (*TreeEntry, error)

func (*Machine) MapBlobs 

func (ag *Machine) MapBlobs(ctx context.Context, dst cadata.PostExister, src cadata.Getter, root Ref, f BlobMapper) (*Ref, error)

func (*Machine) MapEntries 

func (ag *Machine) MapEntries(ctx context.Context, dst cadata.PostExister, src cadata.Getter, root Ref, f TreeEntryMapper) (*Ref, error)

func (*Machine) MapEntryAt 

func (ag *Machine) MapEntryAt(ctx context.Context, dst cadata.PostExister, src cadata.Getter, root Ref, p string, f TreeEntryMapper) (*Ref, error)

func (*Machine) MapLeaves 

func (ag *Machine) MapLeaves(ctx context.Context, dst cadata.PostExister, src cadata.Getter, root Ref, f RefMapper) (*Ref, error)

func (*Machine) Merge 

func (ag *Machine) Merge(ctx context.Context, dst cadata.PostExister, src cadata.Getter, layers ...Ref) (*Ref, error)

Merge merges the refs in layers with increasing prescedence. layer[i+1] is higher prescendence than layer[i]

Merging is associative, but not commutative Merge(tree, blob) -> blob Merge(blob, tree) -> tree Merge(tree1, tree2) -> set of entry names given by tree1 + tree2. value at entry x given by Merge(tree1[x], tree2[x]) Although not written as such for performance reasons: Merging(1, 2, 3, 4, 5) == Merge(Merge(Merge(Merge(1, 2), 3), 4), 5)

Merge will call Sync to protect referential integrity in dst.

func (*Machine) NewBlobReader 

func (ag *Machine) NewBlobReader(ctx context.Context, s cadata.Getter, x Ref) (*Reader, error)

func (*Machine) NewBlobWriter 

func (ag *Machine) NewBlobWriter(s cadata.Poster) *TypedWriter

func (*Machine) NewTreeReader 

func (ag *Machine) NewTreeReader(s cadata.Getter, x Ref) (*TreeReader, error)

func (*Machine) NewTreeWriter 

func (ag *Machine) NewTreeWriter(s cadata.PostExister) *TreeWriter

func (*Machine) NewTypedWriter 

func (ag *Machine) NewTypedWriter(s cadata.Poster, ty Type) *TypedWriter

NewTypedWriter returns a new writer for ty.

func (*Machine) Populate 

func (ag *Machine) Populate(ctx context.Context, store GetListDeleter, x Ref, dst AddExister) error

Populate adds everything reachable form x to dst

func (*Machine) PostBlob 

func (ag *Machine) PostBlob(ctx context.Context, s cadata.Poster, r io.Reader) (*Ref, error)

PostBlob creates a new blob with data from r, and returns a Ref to it.

func (*Machine) PostTree 

func (ag *Machine) PostTree(ctx context.Context, s cadata.PostExister, ents iter.Seq[TreeEntry]) (*Ref, error)

func (*Machine) PostTreeMap 

func (ag *Machine) PostTreeMap(ctx context.Context, s cadata.PostExister, m map[string]Ref) (*Ref, error)

func (*Machine) PostTreeSlice 

func (ag *Machine) PostTreeSlice(ctx context.Context, dst cadata.PostExister, ents []TreeEntry) (*Ref, error)

func (*Machine) PostTyped 

func (ag *Machine) PostTyped(ctx context.Context, s cadata.Poster, ty Type, r io.Reader) (*Ref, error)

PostTyped posts data with an arbitrary type. This can be used to extend the types provided by glfs, without interfering with syncing.

func (*Machine) ReadTreeFrom 

func (ag *Machine) ReadTreeFrom(r io.Reader) *TreeReader

func (*Machine) ShardLeaves 

func (ag *Machine) ShardLeaves(ctx context.Context, dst cadata.PostExister, src cadata.Getter, root Ref, n int) ([]Ref, error)

func (*Machine) Sync 

func (ag *Machine) Sync(ctx context.Context, dst cadata.PostExister, src cadata.Getter, x Ref) error

Sync ensures that all data referenced by x exists in dst, copying from src if necessary. Sync assumes there are no dangling references, and skips copying data when its existence is implied.

func (*Machine) Traverse 

func (ag *Machine) Traverse(ctx context.Context, s cadata.Getter, sem *semaphore.Weighted, x Ref, tr Traverser) error

func (*Machine) WalkRefs 

func (ag *Machine) WalkRefs(ctx context.Context, s cadata.Getter, ref Ref, fn RefWalker) error

WalkRefs calls fn with every Ref reacheable from ref, including Ref. The only guarentee about order is bottom up. if a tree is encoutered the child refs will be visited first.

func (*Machine) WalkTree 

func (ag *Machine) WalkTree(ctx context.Context, store cadata.Getter, ref Ref, f WalkTreeFunc) error

WalkTree walks the tree and calls f with tree entries in lexigraphical order file1.txt comes before file2.txt dir1/ comes before dir1/file1.txt

type Option 

type Option func(*Machine)

func WithSalt 

func WithSalt(salt [32]byte) Option

WithSalt sets the salt used for deriving encryption keys.

type PostLister 

type PostLister interface {
	cadata.Poster
	cadata.Lister
}

type Reader 

type Reader = bigblob.Reader

func GetBlob 

func GetBlob(ctx context.Context, s cadata.Getter, x Ref) (*Reader, error)

GetBlob returns an io.ReadSeeker for accessing data from the blob at x

type Ref 

type Ref struct {
	Type Type `json:"type"`
	bigblob.Root
}

Ref is a reference to a glfs Object, which could be: - Tree - Blob

func FilterPaths 

func FilterPaths(ctx context.Context, dst cadata.PostExister, src cadata.Getter, root Ref, f func(string) bool) (*Ref, error)

FilterPaths returns a version of root with paths filtered using f as a predicate. If f returns true for a path it will be included in the output, otherwise it will not.

func GetAtPath 

func GetAtPath(ctx context.Context, store cadata.Getter, ref Ref, subpath string) (*Ref, error)

GetAtPath returns a ref to the object under ref at subpath. ErrNoEnt is returned if there is no entry at that path.

func MapBlobs 

func MapBlobs(ctx context.Context, dst cadata.PostExister, src cadata.Getter, root Ref, f BlobMapper) (*Ref, error)

MapBlobs calls MapBlobs on the default Machine

func MapEntryAt 

func MapEntryAt(ctx context.Context, dst cadata.PostExister, src cadata.Getter, root Ref, p string, f TreeEntryMapper) (*Ref, error)

MapEntryAt calls MapEntryAt on the default Machine

func MapLeaves 

func MapLeaves(ctx context.Context, dst cadata.PostExister, src cadata.Getter, root Ref, f RefMapper) (*Ref, error)

MapLeaves calls MapLeaves on the default Machine

func Merge 

func Merge(ctx context.Context, dst cadata.PostExister, src cadata.Getter, layers ...Ref) (*Ref, error)

Merge calls Merge on the default Machine

func MustPostBlob 

func MustPostBlob(s cadata.Poster, x []byte) Ref

func MustPostTreeMap 

func MustPostTreeMap(s cadata.PostExister, m map[string]Ref) Ref

func MustPostTreeSlice 

func MustPostTreeSlice(s cadata.PostExister, ents []TreeEntry) Ref

func PostBlob 

func PostBlob(ctx context.Context, s cadata.Poster, r io.Reader) (*Ref, error)

PostBlob creates a new blob with data from r, and returns a Ref to it.

func PostTree 

func PostTree(ctx context.Context, s cadata.PostExister, ents iter.Seq[TreeEntry]) (*Ref, error)

func PostTreeMap 

func PostTreeMap(ctx context.Context, s cadata.PostExister, m map[string]Ref) (*Ref, error)

func PostTreeSlice 

func PostTreeSlice(ctx context.Context, s cadata.PostExister, ents []TreeEntry) (*Ref, error)

func PostTyped 

func PostTyped(ctx context.Context, s cadata.Poster, ty Type, r io.Reader) (*Ref, error)

PostRaw calls PostRaw on the default Machine

func ShardLeaves 

func ShardLeaves(ctx context.Context, dst cadata.PostExister, s cadata.Getter, root Ref, n int) ([]Ref, error)

ShardLeaves calls ShardLeaves on the default Machine

func (Ref) Equals 

func (a Ref) Equals(b Ref) bool

func (Ref) String 

func (r Ref) String() string

type RefMapper 

type RefMapper func(p string, ref Ref) (*Ref, error)

type RefWalker 

type RefWalker func(ref Ref) error

type Traverser 

type Traverser struct {
	// Enter is called before visiting a node, if false is returned the node is skipped.
	Enter func(ctx context.Context, id cadata.ID) (bool, error)
	// Exit is called before leaving a node.  After all it's children have been visited.
	Exit func(ctx context.Context, ty Type, level int, ref bigblob.Ref) error
}

type TreeEntry 

type TreeEntry struct {
	Name     string      `json:"name"`
	FileMode os.FileMode `json:"mode"`
	Ref      Ref         `json:"ref"`
}

TreeEntry is a single entry in a tree, uniquely identified by Name

func GetTreeSlice 

func GetTreeSlice(ctx context.Context, store cadata.Getter, ref Ref, maxEnts int) ([]TreeEntry, error)

GetTreeSlice retreives the tree in store at Ref if it exists. If ref.Type != TypeTree ErrRefType is returned.

func Lookup 

func Lookup(ents []TreeEntry, name string) *TreeEntry

Lookup returns the entry in the tree with name if it exists, or nil if it does not.

func (*TreeEntry) Validate 

func (te *TreeEntry) Validate() error

type TreeEntryMapper 

type TreeEntryMapper func(ent TreeEntry) (*TreeEntry, error)

type TreeReader 

type TreeReader struct {
	// contains filtered or unexported fields
}

func (*TreeReader) Next 

func (tr *TreeReader) Next(ctx context.Context, dst *TreeEntry) error

type TreeWriter 

type TreeWriter struct {
	// contains filtered or unexported fields
}

func (*TreeWriter) Finish 

func (tw *TreeWriter) Finish(ctx context.Context) (*Ref, error)

func (*TreeWriter) Put 

func (tw *TreeWriter) Put(ctx context.Context, te TreeEntry) error

type Type 

type Type string

Type is the type of data pointed to by a Ref

func ParseType 

func ParseType(x []byte) (Type, error)

type TypedWriter 

type TypedWriter struct {
	// contains filtered or unexported fields
}

func (*TypedWriter) Finish 

func (tw *TypedWriter) Finish(ctx context.Context) (*Ref, error)

func (*TypedWriter) SetWriteContext 

func (tw *TypedWriter) SetWriteContext(ctx context.Context)

func (*TypedWriter) Write 

func (tw *TypedWriter) Write(data []byte) (int, error)

type WalkTreeFunc 

type WalkTreeFunc = func(prefix string, tree TreeEntry) error

WalkTreeFunc is the type of functions passed to WalkTree

Source Files

View all Source files

Directories

Path Synopsis
examples
write-read-blob
package glfstar converts GLFS to/from TAR
 package glfstar converts GLFS to/from TAR
internal
slices2

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.