Documentation ¶
Overview ¶
Package siva implements a go-borges library that uses siva files as its storage backend.
More information about siva files: https://github.com/src-d/go-siva
Basics ¶
In this storage each location contains a single bare git repository that can contain the objects from several other logical repositories. These logical repositories are stored as remotes in the configuration file. Its ID is the remote name but it can also search its URLs. All repositories returned from a location have the same objects and references. The function AddLocation creates an empty location that is initialized when using Location.Init. It's initialization consists of initializing the repository if it's not already created and adding a remote with its name and URLs to the provided ID.
For example:
r1, _ := library.Get("github.com/src-d/go-borges") println(r1.Name()) # "0168e2c7-eedc-7358-0a09-39ba833bdd54" r2, _ := library.Get("0168e2c7-eedc-7358-0a09-39ba833bdd54") println(r1.Name()) # "0168e2c7-eedc-7358-0a09-39ba833bdd54" loc, _ := library.AddLocation("test") r1, _ := loc.Init("repo1") # the first repo initializes the git repository r1.Commit() r2, _ := loc.Init("repos2") # the second just adds a new remote r2.Commit() loc.Has("repo1") # true loc.Has("repo2") # true
After use of repositories they should be closed. When the library is transactional it can be closed with Commit (only for read write mode) or Close (save changes or rollback). When the library is non transactional it must be closed with Close. In both cases the repository should not be used again after closing it. A double Close returns error.
Transactions ¶
The storage supports transactions and has location lock on write when using the same library. When the transaction starts a checkpoint file is created with the size of the file. This file is used to recover broken siva files to the last known good state. Writes are done to a temporary siva that holds only the added files. Locations can be accessed in read only mode while the repository is performing a transaction and its content remains stable.
Committing a transaction finishes the writes to the temporary siva file, appends the data to the original siva and deletes the checkpoint file. Rollback truncates the siva file to the last good size and deletes the checkpoint file.
In case files need to be deleted, like for example, when packing loose references, the deletion indexes are written directly to the original siva before appending the temporary one.
Only one repository can be opened in read write mode in the same location when the library is transactional. When a second repository wants to be opened in RW mode in the same location the library will wait a grace period for the previous repository to close. By default is 1 minute but it can be configured when creating the library.
For example:
loc, _ := library.Location("foo") r1, _ := loc.Get("github.com/src-d/go-borges", borges.ReadOnlyMode) r2, _ := loc.Get("github.com/src-d/go-borges", borges.RWMode) r2.R().CreateTag("tag", plumbing.ZeroHash, nil) r1.R().Tag("tag") # not found r3, _ := loc.Get("github.com/src-d/go-borges", borges.ReadOnlyMode) r3.R().Tag("tag") # not found # errors after configured timeout as r2 transaction is not completed r4, _ := loc.Get("github.com/src-d/go-borges", borges.RWMode) r2.Commit() r1.R().Tag("tag") # not found r5, _ := loc.Get("github.com/src-d/go-borges", borges.ReadOnlyMode) r5.R().Tag("tag") # found
Note: When using repositories in non transactional mode you should call Close after finishing, otherwise the siva file will be corrupted.
Index ¶
- Variables
- func NewRefStorage(iter storer.ReferenceIter) (memory.ReferenceStorage, error)
- type Committer
- type Library
- func (l *Library) AddLocation(id borges.LocationID) (borges.Location, error)
- func (l *Library) Get(repoID borges.RepositoryID, mode borges.Mode) (borges.Repository, error)
- func (l *Library) GetOrInit(borges.RepositoryID) (borges.Repository, error)
- func (l *Library) Has(name borges.RepositoryID) (bool, borges.LibraryID, borges.LocationID, error)
- func (l *Library) ID() borges.LibraryID
- func (l *Library) Init(borges.RepositoryID) (borges.Repository, error)
- func (l *Library) Location(id borges.LocationID) (borges.Location, error)
- func (l *Library) Locations() (borges.LocationIterator, error)
- func (l *Library) Repositories(mode borges.Mode) (borges.RepositoryIterator, error)
- func (l *Library) SetVersion(n int) error
- func (l *Library) Version() (int, error)
- type LibraryOptions
- type Location
- func (l *Location) Commit(mode borges.Mode) error
- func (l *Location) DeleteVersion(n int)
- func (l *Location) FS(mode borges.Mode) (sivafs.SivaFS, error)
- func (l *Location) Get(id borges.RepositoryID, mode borges.Mode) (borges.Repository, error)
- func (l *Location) GetOrInit(id borges.RepositoryID) (borges.Repository, error)
- func (l *Location) Has(repoID borges.RepositoryID) (bool, error)
- func (l *Location) ID() borges.LocationID
- func (l *Location) Init(id borges.RepositoryID) (borges.Repository, error)
- func (l *Location) LastVersion() int
- func (l *Location) Library() borges.Library
- func (l *Location) Repositories(mode borges.Mode) (borges.RepositoryIterator, error)
- func (l *Location) Rollback(mode borges.Mode) error
- func (l *Location) SaveMetadata() error
- func (l *Location) SetVersion(n int, v *Version)
- func (l *Location) Version(v int) (*Version, error)
- type ReadOnlyStorer
- func (s *ReadOnlyStorer) Close() error
- func (s *ReadOnlyStorer) Config() (*config.Config, error)
- func (s *ReadOnlyStorer) CountLooseRefs() (int, error)
- func (s *ReadOnlyStorer) IterReferences() (storer.ReferenceIter, error)
- func (s *ReadOnlyStorer) Reference(name plumbing.ReferenceName) (*plumbing.Reference, error)
- type Repository
- func (r *Repository) Close() error
- func (r *Repository) Commit() error
- func (r *Repository) FS() billy.Filesystem
- func (r *Repository) ID() borges.RepositoryID
- func (r *Repository) Location() borges.Location
- func (r *Repository) Mode() borges.Mode
- func (r *Repository) R() *git.Repository
- func (r *Repository) VersionOnCommit(n int)
- type RootedStorage
- func (r *RootedStorage) CheckAndSetReference(new *plumbing.Reference, old *plumbing.Reference) error
- func (r *RootedStorage) Close() error
- func (r *RootedStorage) Commit() error
- func (r *RootedStorage) IterEncodedObjects(t plumbing.ObjectType) (storer.EncodedObjectIter, error)
- func (r *RootedStorage) IterReferences() (storer.ReferenceIter, error)
- func (r *RootedStorage) Reference(name plumbing.ReferenceName) (*plumbing.Reference, error)
- func (r *RootedStorage) RemoveReference(ref plumbing.ReferenceName) error
- func (r *RootedStorage) SetReference(ref *plumbing.Reference) error
- type Storage
- func (s *Storage) CheckAndSetReference(new, old *plumbing.Reference) error
- func (s *Storage) Close() (err error)
- func (s *Storage) Commit() error
- func (s *Storage) CountLooseRefs() (int, error)
- func (s *Storage) IterReferences() (storer.ReferenceIter, error)
- func (s *Storage) PackRefs() (err error)
- func (s *Storage) PackfileWriter() (io.WriteCloser, error)
- func (s *Storage) Reference(name plumbing.ReferenceName) (*plumbing.Reference, error)
- func (s *Storage) RemoveReference(name plumbing.ReferenceName) error
- func (s *Storage) SetReference(ref *plumbing.Reference) error
- type Version
Constants ¶
This section is empty.
Variables ¶
var ( // ErrCannotUseCheckpointFile is returned on checkpoint problems. ErrCannotUseCheckpointFile = errors.NewKind("cannot use checkpoint file: %s") // ErrCannotUseSivaFile is returned on siva problems. ErrCannotUseSivaFile = errors.NewKind("cannot use siva file: %s") )
var ( // ErrMalformedData when checkpoint data is invalid. ErrMalformedData = errors.NewKind("malformed data") // ErrInvalidSize means that the siva size could not be correctly // retrieved. ErrInvalidSize = errors.NewKind("invalid siva size") )
var ErrEmptyCommit = errors.NewKind("there weren't changes to commit")
ErrEmptyCommit is returned when a repository opened in RW mode tries to commit no changes.
var ErrLocationExists = errors.NewKind("location %s already exists")
ErrLocationExists when the location to be created already exists.
var ErrRepoAlreadyClosed = errors.NewKind("repository %s already closed")
ErrRepoAlreadyClosed is returned when a repository opened in RW mode was already closed.
var ErrTransactionTimeout = errors.NewKind("timeout exceeded: unable to " +
"retrieve repository from location %s in transactional mode.")
ErrTransactionTimeout is returned when a repository can't be retrieved in transactional mode because of a timeout.
Functions ¶
func NewRefStorage ¶
func NewRefStorage(iter storer.ReferenceIter) (memory.ReferenceStorage, error)
NewRefStorage creates a new memory.ReferenceStorage with references from a reference iterator.
Types ¶
type Committer ¶
type Committer interface { // Commit applies the changes to a storer if it's in transactional mode. Commit() error // Close signals the end of usage of a storer. If it's in transactional // mode this means rollback. Close() error }
Committer interface has transactional Commit and Close methods for a storer.
type Library ¶
type Library struct {
// contains filtered or unexported fields
}
Library represents a borges.Library implementation based on siva files.
func NewLibrary ¶
func NewLibrary( id string, fs billy.Filesystem, options *LibraryOptions, ) (*Library, error)
NewLibrary creates a new siva.Library. When is not in MetadataReadOnly it will generate an id if not provided the first time the metadata is created.
func (*Library) AddLocation ¶
AddLocation creates a new borges.Location if it does not exist.
func (*Library) Get ¶
func (l *Library) Get(repoID borges.RepositoryID, mode borges.Mode) (borges.Repository, error)
Get implements borges.Library interface.
func (*Library) GetOrInit ¶
func (l *Library) GetOrInit(borges.RepositoryID) (borges.Repository, error)
GetOrInit implements borges.Library interface.
func (*Library) Has ¶
func (l *Library) Has(name borges.RepositoryID) (bool, borges.LibraryID, borges.LocationID, error)
Has implements borges.Library interface.
func (*Library) Init ¶
func (l *Library) Init(borges.RepositoryID) (borges.Repository, error)
Init implements borges.Library interface.
func (*Library) Locations ¶
func (l *Library) Locations() (borges.LocationIterator, error)
Locations implements borges.Library interface.
func (*Library) Repositories ¶
Repositories implements borges.Library interface.
func (*Library) SetVersion ¶
SetVersion sets the current version to the given number.
type LibraryOptions ¶
type LibraryOptions struct { // Transactional enables transactions for repository writes. Transactional bool // TransactionTimeout is the time it will wait while another transaction // is being done before error. 0 means default. TransactionTimeout time.Duration // Timeout set a timeout for library operations. Some operations could // potentially take long so timing out them will make an error be // returned. A 0 value sets a default value of 20 seconds. Timeout time.Duration // RegistryCache is the maximum number of locations in the cache. A value // of 0 will be set a default value of 10000. RegistryCache int // TempFS is the temporary filesystem to do transactions and write files. TempFS billy.Filesystem // Bucket level to use to search and create siva files. Bucket int // RootedRepo makes the repository show only the references for the remote // named with the repository ID. RootedRepo bool // Cache specifies the shared cache used in repositories. If not defined // a new default cache will be created for each repository. Cache cache.Object // Performance enables performance options in read only git repositories // (ExclusiveAccess and KeepDescriptors). Performance bool // MetadataReadOnly doesn't create or modify metadata for the library. MetadataReadOnly bool }
LibraryOptions hold configuration options for the library.
type Location ¶
type Location struct {
// contains filtered or unexported fields
}
Location represents a siva file archiving several git repositories.
func (*Location) Commit ¶
Commit persists transactional or write operations performed on the repositories.
func (*Location) DeleteVersion ¶
DeleteVersion removes the given version number.
func (*Location) Get ¶
func (l *Location) Get(id borges.RepositoryID, mode borges.Mode) (borges.Repository, error)
Get implements the borges.Location interface.
func (*Location) GetOrInit ¶
func (l *Location) GetOrInit(id borges.RepositoryID) (borges.Repository, error)
GetOrInit implements the borges.Location interface.
func (*Location) Has ¶
func (l *Location) Has(repoID borges.RepositoryID) (bool, error)
Has implements the borges.Location interface.
func (*Location) ID ¶
func (l *Location) ID() borges.LocationID
ID implements the borges.Location interface.
func (*Location) Init ¶
func (l *Location) Init(id borges.RepositoryID) (borges.Repository, error)
Init implements the borges.Location interface.
func (*Location) LastVersion ¶
LastVersion returns the last defined version number in metadata or -1 if there are not versions.
func (*Location) Repositories ¶
Repositories implements the borges.Location interface.
func (*Location) Rollback ¶
Rollback discard transactional or write operations performed on the repositories.
func (*Location) SaveMetadata ¶
SaveMetadata writes the location metadata to disk.
func (*Location) SetVersion ¶
SetVersion adds or changes a version to the location.
type ReadOnlyStorer ¶
type ReadOnlyStorer struct { // This wrapper is necessary becuse when a git.Open is performed there must be a // HEAD reference. Since we have rooted repositories packed in siva files without // a HEAD file we need to overwrite the Reference method to be able to return a // reference even if there's no any HEAD. util.ReadOnlyStorer // contains filtered or unexported fields }
ReadOnlyStorer is a wrapper for util.ReadOnlyStorer
func NewReadOnlyStorer ¶
NewReadOnlyStorer returns a new *ReadOnlyStorer initialized with the given storage.Storer.
func NewReadOnlyStorerInitialized ¶
func NewReadOnlyStorerInitialized( sto storage.Storer, sync sivafs.SivaSync, refs memory.ReferenceStorage, config *config.Config, ) (*ReadOnlyStorer, error)
NewReadOnlyStorerInitialized creates a new *ReadOnlyStorer with preloaded references and git config.
func (*ReadOnlyStorer) Close ¶
func (s *ReadOnlyStorer) Close() error
Close implements io.Closer interface.
func (*ReadOnlyStorer) Config ¶
func (s *ReadOnlyStorer) Config() (*config.Config, error)
Config implements config.ConfigStorer interface.
func (*ReadOnlyStorer) CountLooseRefs ¶
func (s *ReadOnlyStorer) CountLooseRefs() (int, error)
CountLooseRefs implements storer.ReferenceStorer.
func (*ReadOnlyStorer) IterReferences ¶
func (s *ReadOnlyStorer) IterReferences() (storer.ReferenceIter, error)
IterReferences implements storer.ReferenceStorer.
func (*ReadOnlyStorer) Reference ¶
func (s *ReadOnlyStorer) Reference(name plumbing.ReferenceName) (*plumbing.Reference, error)
Reference implements the storer.ReferenceStorer interface.
type Repository ¶
type Repository struct {
// contains filtered or unexported fields
}
Repository is an implementation for siva files of borges.Repository interface.
func (*Repository) Close ¶
func (r *Repository) Close() error
Close implements borges.Repository interface.
func (*Repository) Commit ¶
func (r *Repository) Commit() error
Commit implements borges.Repository interface.
func (*Repository) FS ¶
func (r *Repository) FS() billy.Filesystem
FS returns the filesystem to read or write directly to the repository or nil if not available.
func (*Repository) ID ¶
func (r *Repository) ID() borges.RepositoryID
ID implements borges.Repository interface.
func (*Repository) Location ¶
func (r *Repository) Location() borges.Location
Location implements borges.Repository interface.
func (*Repository) Mode ¶
func (r *Repository) Mode() borges.Mode
Mode implements borges.Repository interface.
func (*Repository) R ¶
func (r *Repository) R() *git.Repository
R implements borges.Repository interface.
func (*Repository) VersionOnCommit ¶
func (r *Repository) VersionOnCommit(n int)
VersionOnCommit specifies the version that will be set when the changes are committed. Only works for transactional repositories.
type RootedStorage ¶
RootedStorage returns only the references and objects from a repository stored as a remote in a rooted repository.
func NewRootedStorage ¶
func NewRootedStorage(s storage.Storer, id string) *RootedStorage
NewRootedStorage creates a new storer that only shows references from an specific remote ID.
func (*RootedStorage) CheckAndSetReference ¶
func (r *RootedStorage) CheckAndSetReference( new *plumbing.Reference, old *plumbing.Reference, ) error
CheckAndSetReference implements ReferenceStorer interface.
func (*RootedStorage) Close ¶
func (r *RootedStorage) Close() error
Close implements Committer interface.
func (*RootedStorage) Commit ¶
func (r *RootedStorage) Commit() error
Commit implements Committer interface.
func (*RootedStorage) IterEncodedObjects ¶
func (r *RootedStorage) IterEncodedObjects( t plumbing.ObjectType, ) (storer.EncodedObjectIter, error)
IterEncodedObjects implements EncodedObjectStorer interface.
func (*RootedStorage) IterReferences ¶
func (r *RootedStorage) IterReferences() (storer.ReferenceIter, error)
IterReferences implements ReferenceStorer interface.
func (*RootedStorage) Reference ¶
func (r *RootedStorage) Reference( name plumbing.ReferenceName, ) (*plumbing.Reference, error)
Reference implements ReferenceStorer interface.
func (*RootedStorage) RemoveReference ¶
func (r *RootedStorage) RemoveReference(ref plumbing.ReferenceName) error
RemoveReference implements ReferenceStorer interface.
func (*RootedStorage) SetReference ¶
func (r *RootedStorage) SetReference(ref *plumbing.Reference) error
SetReference implements ReferenceStorer interface.
type Storage ¶
type Storage struct { storage.Storer memory.ReferenceStorage // contains filtered or unexported fields }
Storage holds a ReadWrite siva storage. It can be transactional in which case it will write to a temporary siva file and will append it to the original siva on Commit.
func NewStorage ¶
func NewStorage( base billy.Filesystem, path string, tmp billy.Filesystem, transaction bool, cache cache.Object, ) (*Storage, error)
NewStorage creates a new Storage struct. A new temporary directory is created for the siva filesystem that can be later deleted with Cleanup.
func (*Storage) CheckAndSetReference ¶
CheckAndSetReference implements storer.ReferenceStorer.
func (*Storage) Commit ¶
Commit finishes the writes on a Storage. If transactional mode is enabled the backing transaction siva finishes writing and it is appended to the original siva file. If it's not transactional the original siva file is closed.
func (*Storage) CountLooseRefs ¶
CountLooseRefs implements storer.ReferenceStorer.
func (*Storage) IterReferences ¶
func (s *Storage) IterReferences() (storer.ReferenceIter, error)
IterReferences implements storer.ReferenceStorer.
func (*Storage) PackRefs ¶
PackRefs packs the references kept in memory and write them to the siva storage.
func (*Storage) PackfileWriter ¶
func (s *Storage) PackfileWriter() (io.WriteCloser, error)
PackfileWriter implements storer.PackfileWriter interface.
func (*Storage) RemoveReference ¶
func (s *Storage) RemoveReference(name plumbing.ReferenceName) error
RemoveReference implements storer.ReferenceStorer.