storage

package
v1.66.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 13, 2026 License: Apache-2.0 Imports: 46 Imported by: 0

Documentation

Overview

Package storage serves a store to agents over gRPC and reaches one: Store holds a store's operations over its index, Server serves them, and Client is the backup.Index an agent uses to talk to a store server. BucketStore keeps a store's archives in a gocloud bucket.

Index

Constants

This section is empty.

Variables

View Source 
var ErrInvalidRequest = errors.New("invalid request")

ErrInvalidRequest is a call the store cannot act on as given: a malformed ref, an object that fails validation, a ref of the wrong object type.

Functions

func ArchiveKey 

func ArchiveKey(name string) string

ArchiveKey is where a BucketStore keeps the archive file name in its bucket.

func ClientTLS 

func ClientTLS(caFile string) (*tls.Config, error)

ClientTLS returns a client TLS configuration that trusts only the PEM certificates in caFile, or the system roots when caFile is empty.

func NewBucketObjectStore 

func NewBucketObjectStore(bucket *blob.Bucket, index pack.ArchiveIndex, cacheDir string, cacheSize int64, extra ...pack.PackOption) (*pack.PackStorage, error)

NewBucketObjectStore returns a pack store over a remote bucket that keeps its archives in index and, if cacheDir is not empty, a local metadata cache of at most cacheSize bytes (0 is unbounded); extra pack options follow.

func ServerTLS 

func ServerTLS(certFile, keyFile string, plaintextBehindProxy bool) (credentials.TransportCredentials, *tls.Config, error)

ServerTLS returns what a store server listens with: TLS from the certificate and key files, or plaintext when the operator declared a TLS-terminating proxy in front and gave neither. The TLS configuration is nil for plaintext.

func SignArchiveRange 

func SignArchiveRange(ctx context.Context, bucket *blob.Bucket, r SignedRange) (*proto.Location, error)

SignArchiveRange signs r against bucket, which holds archives as a BucketStore keeps them. A bucket whose driver cannot bind the range and headers answers pack.ErrNoSignedURL; only Cloud Storage binds headers beyond the range.

func ToStatus 

func ToStatus(err error) error

ToStatus maps the store's sentinel errors to gRPC codes; an error that already is a status passes through.

Types

type BucketOption 

type BucketOption func(*BucketStore)

A BucketOption adjusts a BucketStore.

func WithStoreName 

func WithStoreName(name string) BucketOption

WithStoreName labels what this store sends to the object store, so the cost of one store can be told from another's in a process holding several of them.

type BucketStore 

type BucketStore struct {
	// contains filtered or unexported fields
}

BucketStore is a pack.ArchiveStorage over a gocloud bucket, keyed pack/<extension>/<name>.

func NewBucketStore 

func NewBucketStore(bucket *blob.Bucket, options ...BucketOption) *BucketStore

NewBucketStore returns an archive storage over bucket.

func (*BucketStore) Checksum 

func (c *BucketStore) Checksum(name string) ([]byte, error)

Checksum implements pack.Checksummer with the MD5 the bucket reports, empty when it has none.

func (*BucketStore) Create 

func (c *BucketStore) Create(name string) (pack.File, error)

Create implements pack.ArchiveStorage.

func (*BucketStore) CreateNew 

func (c *BucketStore) CreateNew(name string, data []byte) error

CreateNew implements pack.ArchiveStorage with the bucket's own precondition, so it holds across processes on GCS and S3; a file:// bucket only checks within one process.

func (*BucketStore) Delete 

func (c *BucketStore) Delete(name string) error

Delete implements pack.ArchiveStorage.

func (*BucketStore) DeleteAll 

func (c *BucketStore) DeleteAll() error

DeleteAll implements pack.ArchiveStorage.

func (*BucketStore) List 

func (c *BucketStore) List(extension string) ([]string, error)

List implements pack.ArchiveStorage.

func (*BucketStore) ListInfo 

func (c *BucketStore) ListInfo(extension string) ([]pack.ListedFile, error)

ListInfo implements pack.InfoLister.

func (*BucketStore) Open 

func (c *BucketStore) Open(name string) (pack.File, error)

Open implements pack.ArchiveStorage.

func (*BucketStore) OpenListed 

func (c *BucketStore) OpenListed(file pack.ListedFile) (pack.File, error)

OpenListed implements pack.ListedOpener.

func (*BucketStore) SignRange 

func (c *BucketStore) SignRange(ctx context.Context, name string, offset, length int64, ttl time.Duration) (*proto.Location, error)

SignRange implements pack.RangeSigner. The URL is bound to the range, so the recipient can read those bytes and nothing else of the archive. A bucket whose driver cannot bind a header answers ErrNoSignedURL.

type Client 

type Client struct {

	// HTTP fetches the locations a server answers with; nil uses the
	// default client.
	HTTP *http.Client
	// contains filtered or unexported fields
}

Client is a backup.Index over a store server's gRPC API.

func NewClient 

func NewClient(addr string, creds auth.Credentials, tlsConfig *tls.Config) (*Client, error)

NewClient connects to a store server over TLS, presenting creds on every call. A nil tlsConfig trusts the system roots; see ClientTLS for a pinned certificate authority.

func NewPlaintextClient 

func NewPlaintextClient(addr string, creds auth.Credentials) (*Client, error)

NewPlaintextClient connects to a store server without TLS, putting the credentials and every object on the wire in the clear. It is for a store on the same machine as the agent; anywhere else, use NewClient. The credentials must allow it.

func (*Client) BeginCommit 

func (r *Client) BeginCommit(ctx context.Context, set string) (*backup.CommitGrant, error)

BeginCommit implements backup.CommitGate; a refusal is ErrCommitDenied with the server's reason.

func (*Client) BeginSession 

func (r *Client) BeginSession(ctx context.Context, s *backup.Session) (context.Context, error)

BeginSession opens a session on the server and returns a context whose calls belong to it.

func (*Client) Close 

func (r *Client) Close() error

Close is a no-op; close the connection instead.

func (*Client) CommitInfo 

func (r *Client) CommitInfo(ctx context.Context, set string, notAfter time.Time, count int) ([]*proto.Commit, error)

CommitInfo implements backup.Index.

func (*Client) CommitSizes 

func (r *Client) CommitSizes(ctx context.Context, set string, notAfter time.Time, count int) ([]*proto.Commit, []*proto.CommitSize, error)

CommitSizes implements backup.CommitSizer. A store that measures nothing answers each commit with an empty size.

func (*Client) CountCommits 

func (r *Client) CountCommits(ctx context.Context, set string, period proto.Period, from, to time.Time, zone string, deleted bool) ([]*proto.CommitCount, error)

CountCommits implements backup.Retention.

func (*Client) Delete 

func (r *Client) Delete(context.Context, *proto.Ref) error

Delete is not offered by the server; retention deletes commits and sets.

func (*Client) DeleteCommit 

func (r *Client) DeleteCommit(ctx context.Context, ref *proto.Ref) error

DeleteCommit implements backup.Retention.

func (*Client) DeleteSet 

func (r *Client) DeleteSet(ctx context.Context, set string, erase bool) error

DeleteSet implements backup.Retention.

func (*Client) Downloaded 

func (r *Client) Downloaded() int64

Downloaded returns how many bytes the client has received so far, from the server and from the locations it followed, as they came over the wire.

func (*Client) EndSession 

func (r *Client) EndSession(ctx context.Context) error

EndSession drops what the context's session has not committed; a session the store no longer has, such as one its commit ended, is backup.ErrNoSession.

func (*Client) EscrowedKeys 

func (r *Client) EscrowedKeys(ctx context.Context) ([]backup.EscrowedKey, error)

EscrowedKeys returns every escrowed copy of the store key the server keeps.

func (*Client) FileInfo 

func (r *Client) FileInfo(ctx context.Context, set string, name string, notAfter time.Time, count int) ([]*proto.TreeNode, error)

FileInfo implements backup.Index.

func (*Client) Get 

func (r *Client) Get(ctx context.Context, ref *proto.Ref) (*proto.Object, error)

Get implements backup.ObjectStore. A server that answers with a location is followed, so where the bytes came from stays its business.

func (*Client) GetTree 

func (r *Client) GetTree(ctx context.Context, ref *proto.Ref, maxDepth uint32) ([]*proto.Object, error)

GetTree implements backup.TreeFetcher over the streaming RPC, fetching the runs it is pointed at runFetchers at a time. It returns only trees reached from ref through the refs of trees it returns, and fails with proto.ErrRefMismatch on any other object the server sends.

func (*Client) Has 

func (r *Client) Has(context.Context, *proto.Ref) (bool, error)

Has is not offered by the server; presence filters answer it.

func (*Client) LatestCommit 

func (r *Client) LatestCommit(ctx context.Context, set string) (*proto.Ref, error)

LatestCommit implements backup.Index.

func (*Client) Open 

func (r *Client) Open() error

Open is a no-op; the connection is dialed by NewClient.

func (*Client) Pins 

func (r *Client) Pins(ctx context.Context) ([]*proto.PinInfo, error)

Pins implements backup.Retention.

func (*Client) Presence 

func (r *Client) Presence(ctx context.Context, set string) (presence.Set, error)

Presence implements backup.PresenceSource.

func (*Client) Put 

func (r *Client) Put(ctx context.Context, object *proto.Object) error

Put implements backup.ObjectStore.

func (*Client) PutEscrowedKey 

func (r *Client) PutEscrowedKey(ctx context.Context, key backup.EscrowedKey) error

PutEscrowedKey implements backup.KeyEscrow through the server's Admin service, which takes the same credentials as the store.

func (*Client) PutFile 

func (r *Client) PutFile(ctx context.Context, object *proto.Object, assumed []*proto.Ref) ([]*proto.Ref, error)

PutFile implements backup.Confirmer.

func (*Client) ReIndex 

func (r *Client) ReIndex(context.Context) (backup.ReIndexReport, error)

ReIndex is not offered by the server.

func (*Client) ReadDir 

func (r *Client) ReadDir(ctx context.Context, set string, dir string, notAfter time.Time) ([]*proto.TreeNode, error)

ReadDir implements backup.DirLister.

func (*Client) ReadFiles 

func (r *Client) ReadFiles(ctx context.Context, files []backup.FileRead, objectsOnly bool, object func(int, *proto.Object) error, part func(int, int, *proto.Object) error) error

ReadFiles implements backup.FilesReader through the ReadFiles stream, fetching the runs it is pointed at runFetchers at a time.

func (*Client) ReadParts 

func (r *Client) ReadParts(ctx context.Context, file *proto.Ref, skip []int, fn func(int, *proto.Object) error) error

ReadParts implements backup.PartReader through the ReadFile stream, fetching the runs it is pointed at runFetchers at a time.

func (*Client) TrashedCommits 

func (r *Client) TrashedCommits(ctx context.Context, set string, before time.Time, limit int) ([]*proto.TrashedCommit, error)

TrashedCommits implements backup.Retention.

func (*Client) UndeleteCommit 

func (r *Client) UndeleteCommit(ctx context.Context, ref *proto.Ref) error

UndeleteCommit implements backup.Retention.

func (*Client) UndeleteSet 

func (r *Client) UndeleteSet(ctx context.Context, set string) error

UndeleteSet implements backup.Retention.

func (*Client) Unpin 

func (r *Client) Unpin(ctx context.Context, pin *proto.Ref) error

Unpin implements backup.Retention.

func (*Client) Walk 

func (r *Client) Walk(context.Context, bool, proto.ObjectType, backup.ObjectReceiver) error

Walk is not offered by the server; an agent reads through Get and ReadParts.

type Operation 

type Operation string

Operation is a request a store makes to the object store holding it. Object stores bill by request, and each charges these differently, so they are counted apart and priced by whoever pays the bill. 

const (
	// OpGet reads an object or a range of one.
	OpGet Operation = "get"
	// OpHead reads an object's attributes without its body.
	OpHead Operation = "head"
	// OpPut writes an object.
	OpPut Operation = "put"
	// OpList reads one page of a listing.
	OpList Operation = "list"
	// OpDelete removes an object.
	OpDelete Operation = "delete"
)

type Receipt 

type Receipt struct {
	Ref     *proto.Ref
	Object  *proto.Object
	Missing []*proto.Ref
}

Receipt answers an Upload: the stored ref, the stamped object for a commit or pin, or the assumed refs that are missing, in which case nothing was stored.

type Server 

type Server struct {
	proto.UnsafeStoreServer
	// contains filtered or unexported fields
}

Server is the gRPC adapter of a Store: every RPC is one Store operation and a translation of its errors to status codes.

func NewServer 

func NewServer(store *Store) *Server

NewServer serves store over gRPC.

func (*Server) BeginCommit 

func (r *Server) BeginCommit(ctx context.Context, request *proto.BeginCommitRequest) (*proto.BeginCommitResponse, error)

BeginCommit implements proto.StoreServer.

func (*Server) BeginSession 

func (r *Server) BeginSession(ctx context.Context, request *proto.BeginSessionRequest) (*proto.BeginSessionResponse, error)

BeginSession implements proto.StoreServer.

func (*Server) CommitInfo 

func (r *Server) CommitInfo(ctx context.Context, request *proto.CommitInfoRequest) (*proto.CommitInfoResponse, error)

CommitInfo implements proto.StoreServer.

func (*Server) CountCommits 

func (r *Server) CountCommits(ctx context.Context, request *proto.CountCommitsRequest) (*proto.CountCommitsResponse, error)

CountCommits implements proto.StoreServer.

func (*Server) DeleteCommit 

func (r *Server) DeleteCommit(ctx context.Context, request *proto.DeleteCommitRequest) (*proto.DeleteCommitResponse, error)

DeleteCommit implements proto.StoreServer.

func (*Server) DeleteSet 

func (r *Server) DeleteSet(ctx context.Context, request *proto.DeleteSetRequest) (*proto.DeleteSetResponse, error)

DeleteSet implements proto.StoreServer.

func (*Server) EndSession 

func (r *Server) EndSession(ctx context.Context, request *proto.EndSessionRequest) (*proto.EndSessionResponse, error)

EndSession implements proto.StoreServer.

func (*Server) EscrowedKeys 

func (r *Server) EscrowedKeys(ctx context.Context, _ *proto.EscrowedKeysRequest) (*proto.EscrowedKeysResponse, error)

EscrowedKeys implements proto.StoreServer.

func (*Server) FileInfo 

func (r *Server) FileInfo(ctx context.Context, request *proto.FileInfoRequest) (*proto.FileInfoResponse, error)

FileInfo implements proto.StoreServer.

func (*Server) Get 

func (r *Server) Get(ctx context.Context, request *proto.GetRequest) (*proto.GetResponse, error)

Get implements proto.StoreServer.

func (*Server) GetPresence 

func (r *Server) GetPresence(request *proto.GetPresenceRequest, stream proto.Store_GetPresenceServer) error

GetPresence streams the head filters of the caller's scope, each split into pieces that share an index.

func (*Server) GetTree 

func (r *Server) GetTree(request *proto.GetTreeRequest, stream proto.Store_GetTreeServer) error

GetTree implements proto.StoreServer.

func (*Server) LatestCommit 

func (r *Server) LatestCommit(ctx context.Context, request *proto.LatestCommitRequest) (*proto.LatestCommitResponse, error)

LatestCommit answers with an empty ref when the set has no commit.

func (*Server) ListPins 

func (r *Server) ListPins(ctx context.Context, _ *proto.ListPinsRequest) (*proto.ListPinsResponse, error)

ListPins implements proto.StoreServer.

func (*Server) ListTrash 

func (r *Server) ListTrash(ctx context.Context, request *proto.ListTrashRequest) (*proto.ListTrashResponse, error)

ListTrash implements proto.StoreServer.

func (*Server) Put 

func (r *Server) Put(ctx context.Context, request *proto.PutRequest) (*proto.PutResponse, error)

Put implements proto.StoreServer.

func (*Server) ReadDir 

func (r *Server) ReadDir(ctx context.Context, request *proto.ReadDirRequest) (*proto.ReadDirResponse, error)

ReadDir implements proto.StoreServer.

func (*Server) ReadFile 

func (r *Server) ReadFile(request *proto.ReadFileRequest, stream proto.Store_ReadFileServer) error

ReadFile streams a file's parts, as runs to fetch or as stored objects, running the loop server-side so the client authorises once per file.

func (*Server) ReadFiles 

func (r *Server) ReadFiles(request *proto.ReadFilesRequest, stream proto.Store_ReadFilesServer) error

ReadFiles streams many files' objects and then their parts, as runs to fetch or as stored objects.

func (*Server) StreamInterceptor 

func (r *Server) StreamInterceptor() grpc.StreamServerInterceptor

StreamInterceptor is UnaryInterceptor for streaming calls.

func (*Server) UnaryInterceptor 

func (r *Server) UnaryInterceptor() grpc.UnaryServerInterceptor

UnaryInterceptor resolves the session header; it runs after authentication.

func (*Server) UndeleteCommit 

func (r *Server) UndeleteCommit(ctx context.Context, request *proto.UndeleteCommitRequest) (*proto.UndeleteCommitResponse, error)

UndeleteCommit implements proto.StoreServer.

func (*Server) UndeleteSet 

func (r *Server) UndeleteSet(ctx context.Context, request *proto.UndeleteSetRequest) (*proto.UndeleteSetResponse, error)

UndeleteSet implements proto.StoreServer.

func (*Server) Unpin 

func (r *Server) Unpin(ctx context.Context, request *proto.UnpinRequest) (*proto.UnpinResponse, error)

Unpin implements proto.StoreServer.

type SessionLookup 

type SessionLookup interface {
	LookupSession(ctx context.Context, id string) (*backup.Session, error)
}

SessionLookup resolves a session id presented by a caller.

type SignedRange 

type SignedRange struct {
	Name           string
	Offset, Length int64
	TTL            time.Duration
	Now            time.Time
	Header         map[string]string
}

SignedRange asks for a GET of bytes [Offset, Offset+Length) of the archive Name, working for TTL from Now (zero means time.Now). Header is bound into the signature beside the range, and the request must send it as given.

type Store 

type Store struct {
	Index    backup.Index
	Sessions backup.SessionStore

	// Lease is what BeginSession promises the client: the session
	// store's own when it has one.
	Lease time.Duration
}

Store is one store's operations over an index and its sessions: the work behind every RPC of proto.Store, with no transport in it. The caller's principal and session travel in the context.

func NewStore 

func NewStore(index backup.Index, sessions backup.SessionStore) *Store

NewStore returns a Store over index. sessions is nil for a store without sessions.

func (*Store) BeginCommit 

func (s *Store) BeginCommit(ctx context.Context, set string) (*backup.CommitGrant, error)

BeginCommit asks the index whether the caller may commit to set. An index without a gate allows every commit and assigns the set id later; a refusal is backup.ErrCommitDenied with the reason.

func (*Store) BeginSession 

func (s *Store) BeginSession(ctx context.Context, set string, restore *proto.Ref, initial *retention.Policy) (*backup.Session, error)

BeginSession opens a session on set for the caller; restore names the commit a restore session reads, nil for a backup. initial, when set, is the retention policy set starts with if the session creates it; an invalid one is ErrInvalidRequest.

func (*Store) EndSession 

func (s *Store) EndSession(ctx context.Context, id string) error

EndSession drops what the caller's session id has not committed.

func (*Store) Escrow 

func (s *Store) Escrow() (backup.KeyEscrow, error)

Escrow returns where the store keeps its escrowed key.

func (*Store) Get 

func (s *Store) Get(ctx context.Context, ref *proto.Ref) (*proto.Object, error)

Get fetches a commit, tree or file object the caller references. Anything else, and anything absent, is backup.ErrNotFound.

func (*Store) Presence 

func (s *Store) Presence(ctx context.Context, set string) ([]*proto.PresenceFilter, error)

Presence returns the head filters of the caller's scope for set: nil when the index keeps none or the scope is off.

func (*Store) Put 

func (s *Store) Put(ctx context.Context, up Upload) (*Receipt, error)

Put stores up.Object after recomputing its ref from the received bytes; a client ref that does not match is proto.ErrRefMismatch. Commits and pins are stamped with the receipt time, and a commit with its set id, so their ref is computed here and returned with the object.

func (*Store) ReadDir 

func (s *Store) ReadDir(ctx context.Context, set string, dir string, notAfter time.Time) ([]*proto.TreeNode, error)

ReadDir lists what a set held directly under dir at notAfter, sorted by name; backup.ErrNotImplemented for an index that cannot answer.

func (*Store) ReadFile 

func (s *Store) ReadFile(ctx context.Context, ref *proto.Ref, skip []uint32, fn func(*proto.ReadFileResponse) error) error

ReadFile hands fn the file's parts except the indexes in skip: runs of them the index can locate, and the stored objects of the rest in order. An inline part is never handed over.

func (*Store) ReadFiles 

func (s *Store) ReadFiles(ctx context.Context, files []*proto.ReadFilesEntry, objectsOnly bool, fn func(*proto.ReadFilesResponse) error) error

ReadFiles hands fn the object of each of files in order and then, unless objectsOnly, the parts of each except the indexes in its skip, a part several of them hold once: runs of those the index can locate, all located together, and the stored objects of the rest in order. An inline part is never handed over.

func (*Store) Retention 

func (s *Store) Retention() (backup.Retention, error)

Retention is the index's retention state, backup.ErrNotImplemented for an index that keeps none.

func (*Store) Session 

func (s *Store) Session(ctx context.Context, id string) (*backup.Session, error)

Session returns the live session behind id if it belongs to the caller; an unknown id is backup.ErrNotFound, another caller's auth.ErrForbidden.

func (*Store) Tree 

func (s *Store) Tree(ctx context.Context, ref *proto.Ref, maxDepth uint32, fn func(*proto.GetTreeResponse) error) error

Tree walks the tree at ref breadth-first, its splits, and the trees of directories below it down to maxDepth levels, handing fn each of them in walk order.

type Upload 

type Upload struct {
	Object  *proto.Object
	Ref     *proto.Ref
	Assumed []*proto.Ref
}

Upload is one Put: the object, the ref the client computed for it (nil for a commit or pin, which the store stamps itself) and the refs a file assumes are stored already.

Source Files

View all Source files

Directories

Path Synopsis
Package maintenance runs a store's housekeeping on a schedule.
 Package maintenance runs a store's housekeeping on a schedule.
Package mapping turns what the store service carries into goback's own types and back.
 Package mapping turns what the store service carries into goback's own types and back.
gen
Package pack stores objects in archives, each with an index of what it holds, in an ArchiveStorage such as a directory or a bucket.
 Package pack stores objects in archives, each with an index of what it holds, in an ArchiveStorage such as a directory or a bucket.
packtest
Package wrapped finds the store beneath stores layered over it, such as a cache: Innermost(store) is the store that keeps the objects, to test for what it implements.
 Package wrapped finds the store beneath stores layered over it, such as a cache: Innermost(store) is the store that keeps the objects, to test for what it implements.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.