Documentation

Overview

    Package fragment is concerned with the mapping of journal offsets to protocol.Fragments, to corresponding local or remote journal content. It provides implementation for:

    * Interacting with remote fragment stores.
    * Indexing local and remote Fragments (see Index).
    * The construction of new Fragments from a replication stream (see Spool).
    * The persisting of constructed Fragments to remote stores (see Persister).
    

    Index

    Constants

    This section is empty.

    Variables

    View Source
    var FileSystemStoreRoot = "/dev/null/invalid/example/path/to/store"

      FileSystemStoreRoot is the filesystem path which roots fragment ContentPaths of a file:// fragment store. It must be set at program startup prior to use.

      Functions

      func List

      func List(ctx context.Context, store pb.FragmentStore, name pb.Journal, callback func(pb.Fragment)) error

        List Fragments of the FragmentStore for a given journal. |callback| is invoked with each listed Fragment, and any returned error aborts the listing.

        func Open

        func Open(ctx context.Context, fragment pb.Fragment) (io.ReadCloser, error)

          Open a Reader of the Fragment on the store. The returned ReadCloser does not perform any applicable client-side decompression, but does request server decompression in the case of GZIP_OFFLOAD_DECOMPRESSION.

          func Persist

          func Persist(ctx context.Context, spool Spool) error

            Persist a Spool to its store. If the Spool Fragment is already present, this is a no-op. If the Spool has not been compressed incrementally, it will be compressed before being persisted.

            func Remove

            func Remove(ctx context.Context, fragment pb.Fragment) error

              Remove |fragment| from its BackingStore.

              func SignGetURL

              func SignGetURL(fragment pb.Fragment, d time.Duration) (string, error)

                SignGetURL returns a URL authenticating the bearer to perform a GET operation of the Fragment for the provided Duration from the current time.

                Types

                type CoverSet

                type CoverSet []Fragment

                  CoverSet maintains Fragments ordered on |Begin| and |End|, with the invariant that no Fragment is fully overlapped by another Fragment in the set (though it may be overlapped by a combination of other Fragments). Intuitively, CoverSet represents the set of offsets which are "covered" by a collection of Fragments, and is able to map, for each byte offset, a "best" covering Fragment. It employs a heuristic of preferring larger fragments (and will replace spans of overlapped smaller fragments). An implication of its invariant is that no two Fragments have the same |Begin| or |End| (as that would imply an overlap). Both are monotonically increasing in the set: set[0].Begin represents the minimum offset, and set[len(set)-1].End represents the maximum offset.

                  func CoverSetDifference

                  func CoverSetDifference(a, b CoverSet) CoverSet

                    CoverSetDifference returns the subset of Fragments in |a| which cover byte offsets not also covered by Fragments in |b|.

                    func WalkAllStores

                    func WalkAllStores(ctx context.Context, name pb.Journal, stores []pb.FragmentStore) (CoverSet, error)

                      WalkAllStores enumerates Fragments from each of |stores| into the returned CoverSet, or returns an encountered error.

                      func (CoverSet) Add

                      func (s CoverSet) Add(fragment Fragment) (CoverSet, bool)

                        Add the Fragment to the CoverSet. The CoverSet is returned, along with an indication of whether an offset span was updated to reflect Fragment. All updates occur in-place.

                        func (CoverSet) BeginOffset

                        func (s CoverSet) BeginOffset() int64

                          BeginOffset returns the first (lowest) Begin offset of any Fragment in the CoverSet.

                          func (CoverSet) EndOffset

                          func (s CoverSet) EndOffset() int64

                            EndOffset returns the last (largest) End offset of any Fragment in the set.

                            func (CoverSet) LongestOverlappingFragment

                            func (s CoverSet) LongestOverlappingFragment(offset int64) (ind int, found bool)

                              LongestOverlappingFragment finds and returns the index |ind| of the Fragment covering |offset| which also has the most content following |offset|. If no fragment covers |offset|, the index of the next Fragment beginning after |offset| is returned (which may be beyond the current CoverSet range). |found| indicates whether an overlapping Fragment was found.

                              type File

                              type File interface {
                              	io.ReaderAt
                              	io.Seeker
                              	io.WriterAt
                              	io.Writer
                              	io.Closer
                              }

                                File is the subset of os.File used in backing Fragments with local files.

                                type Fragment

                                type Fragment struct {
                                	protocol.Fragment
                                	// Local uncompressed file of the Fragment, or nil iff the Fragment is remote.
                                	File File
                                }

                                  Fragment wraps the protocol.Fragment type with a nil-able backing local File.

                                  type Index

                                  type Index struct {
                                  	// contains filtered or unexported fields
                                  }

                                    Index maintains a queryable index of local and remote journal Fragments.

                                    func NewIndex

                                    func NewIndex(ctx context.Context) *Index

                                      NewIndex returns a new, empty Index.

                                      func (*Index) EndOffset

                                      func (fi *Index) EndOffset() int64

                                        EndOffset returns the last (largest) End offset in the index.

                                        func (*Index) Inspect

                                        func (fi *Index) Inspect(callback func(CoverSet) error) error

                                          Inspect will call |callback| with a CoverSet represeting a snapshot of all the fragments in the index. While |callback| is executing there will be no changes to the fragment set of the index.

                                          func (*Index) Query

                                          func (fi *Index) Query(ctx context.Context, req *pb.ReadRequest) (*pb.ReadResponse, File, error)

                                            Query the Index for a Fragment matching the ReadRequest.

                                            func (*Index) ReplaceRemote

                                            func (fi *Index) ReplaceRemote(set CoverSet)

                                              ReplaceRemote replaces all remote Fragments in the index with |set|.

                                              func (*Index) SpoolCommit

                                              func (fi *Index) SpoolCommit(frag Fragment)

                                                SpoolCommit adds local Spool Fragment |frag| to the index.

                                                func (*Index) WaitForFirstRemoteRefresh

                                                func (fi *Index) WaitForFirstRemoteRefresh(ctx context.Context) error

                                                  WaitForFirstRemoteRefresh blocks until ReplaceRemote has been called at least one time, or until the context is cancelled.

                                                  type Persister

                                                  type Persister struct {
                                                  	// contains filtered or unexported fields
                                                  }

                                                  func NewPersister

                                                  func NewPersister(ks *keyspace.KeySpace) *Persister

                                                    NewPersister returns an empty, initialized Persister.

                                                    func (*Persister) Finish

                                                    func (p *Persister) Finish()

                                                    func (*Persister) Serve

                                                    func (p *Persister) Serve()

                                                    func (*Persister) SpoolComplete

                                                    func (p *Persister) SpoolComplete(spool Spool, primary bool)

                                                    type Spool

                                                    type Spool struct {
                                                    	// Fragment at time of last commit.
                                                    	Fragment
                                                    	// FirstAppendTime is the timestamp of the first append of the current fragment.
                                                    	FirstAppendTime time.Time
                                                    	// contains filtered or unexported fields
                                                    }

                                                      Spool is a Fragment which is in the process of being created, backed by a local *os.File. As commits occur and the file extent is updated, the Spool Fragment is also updated to reflect the new committed extent. At all times, the Spool Fragment is a consistent, valid Fragment.

                                                      func NewSpool

                                                      func NewSpool(journal pb.Journal, observer SpoolObserver) Spool

                                                        NewSpool returns an empty Spool of |journal|.

                                                        func (*Spool) Apply

                                                        func (s *Spool) Apply(r *pb.ReplicateRequest, primary bool) (pb.ReplicateResponse, error)

                                                          Apply the ReplicateRequest to the Spool, returning any encountered error.

                                                          func (*Spool) MustApply

                                                          func (s *Spool) MustApply(r *pb.ReplicateRequest)

                                                            MustApply applies the ReplicateRequest, and panics if a !OK status is returned or error occurs. MustApply is a convenience for cases such as rollbacks, where the request is derived from the Spool itself and cannot reasonably fail.

                                                            func (*Spool) Next

                                                            func (s *Spool) Next() pb.Fragment

                                                              Next returns the next Fragment which can be committed by the Spool.

                                                              func (Spool) String

                                                              func (s Spool) String() string

                                                                String returns a debugging representation of the Spool.

                                                                type SpoolObserver

                                                                type SpoolObserver interface {
                                                                	// SpoolCommit is called when the Spool Fragment is extended.
                                                                	SpoolCommit(Fragment)
                                                                	// SpoolComplete is called when the Spool has been completed.
                                                                	SpoolComplete(_ Spool, primary bool)
                                                                }

                                                                  SpoolObserver is notified of important events in the Spool lifecycle.