fileset

Documentation

Index

• Constants
• type FileSet
  • func (f *FileSet) Close() error
  • func (f *FileSet) Delete(name string)
  • func (f *FileSet) StartTag(tag string)
  • func (f *FileSet) Write(data []byte) (int, error)
  • func (f *FileSet) WriteHeader(hdr *tar.Header) error
• type Option
  • func WithRoot(root string) Option
• type Reader
  • func (r *Reader) Close() error
  • func (r *Reader) Next() (*index.Header, error)
  • func (r *Reader) Peek() (*index.Header, error)
  • func (r *Reader) PeekTag() (*index.Tag, error)
  • func (r *Reader) Read(data []byte) (int, error)
• type ShardFunc
• type Storage
  • func NewStorage(objC obj.Client, chunks *chunk.Storage, opts ...StorageOption) *Storage
  • func (s *Storage) Merge(ctx context.Context, outputFileSet string, inputFileSets []string, ...) error
  • func (s *Storage) New(ctx context.Context, fileSet string, opts ...Option) *FileSet
  • func (s *Storage) NewReader(ctx context.Context, fileSet string, opts ...index.Option) *Reader
  • func (s *Storage) NewWriter(ctx context.Context, fileSet string) *Writer
  • func (s *Storage) Shard(ctx context.Context, fileSets []string, shardFunc ShardFunc) error
• type StorageOption
  • func ServiceEnvToOptions(env *serviceenv.ServiceEnv) []StorageOption
  • func WithMemoryThreshold(threshold int64) StorageOption
  • func WithShardThreshold(threshold int64) StorageOption
• type Writer
  • func (w *Writer) Close() error
  • func (w *Writer) CopyFiles(r *Reader, pathBound ...string) error
  • func (w *Writer) CopyTags(r *Reader, tagBound ...string) error
  • func (w *Writer) StartTag(id string)
  • func (w *Writer) Write(data []byte) (int, error)
  • func (w *Writer) WriteHeader(hdr *index.Header) error

Constants

const (

	// DefaultMemoryThreshold is the default for the memory threshold that must
	// be met before a file set part is serialized (excluding close).
	DefaultMemoryThreshold = 1024 * chunk.MB
	// DefaultShardThreshold is the default for the size threshold that must
	// be met before a shard is created by the shard function.
	DefaultShardThreshold = 1024 * chunk.MB
	// Compacted is the suffix of a path that points to the compaction of the prefix.
	Compacted = "compacted"
)

Types

type FileSet

type FileSet struct {
	// contains filtered or unexported fields
}

FileSet is a set of files. This may be a full filesystem or a subfilesystem (e.g. datum / datum set / shard).

func (*FileSet) Close

func (f *FileSet) Close() error

Close closes the file set.

func (*FileSet) Delete

func (f *FileSet) Delete(name string)

Delete deletes a file from the file set. (bryce) might need to delete ancestor directories in certain cases.

func (*FileSet) StartTag

func (f *FileSet) StartTag(tag string)

StartTag starts a tag for the next set of files.

func (*FileSet) Write

func (f *FileSet) Write(data []byte) (int, error)

Write writes to the current file in the tar stream.

func (*FileSet) WriteHeader

func (f *FileSet) WriteHeader(hdr *tar.Header) error

WriteHeader writes a tar header and prepares to accept the file's contents. (bryce) should we prevent directories from being written here?

type Option

type Option func(f *FileSet)

Option configures a file set.

func WithRoot

func WithRoot(root string) Option

WithRoot sets the root path of the file set.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader reads the serialized format of a fileset.

func (*Reader) Close

func (r *Reader) Close() error

Close closes the reader.

func (*Reader) Next

func (r *Reader) Next() (*index.Header, error)

Next returns the next header, and prepares the file's content for reading.

func (*Reader) Peek

func (r *Reader) Peek() (*index.Header, error)

Peek returns the next header without progressing the reader.

func (*Reader) PeekTag

func (r *Reader) PeekTag() (*index.Tag, error)

PeekTag returns the next tag without progressing the reader.

func (*Reader) Read

func (r *Reader) Read(data []byte) (int, error)

Read reads from the current file in the tar stream.

type ShardFunc

type ShardFunc func(*index.PathRange) error

ShardFunc is a callback that returns a PathRange for each shard.

type Storage

type Storage struct {
	// contains filtered or unexported fields
}

Storage is the abstraction that manages fileset storage.

func NewStorage

func NewStorage(objC obj.Client, chunks *chunk.Storage, opts ...StorageOption) *Storage

NewStorage creates a new Storage.

func (*Storage) Merge

func (s *Storage) Merge(ctx context.Context, outputFileSet string, inputFileSets []string, opts ...index.Option) error

Merge merges the file sets with the passed in prefix.

func (*Storage) New

func (s *Storage) New(ctx context.Context, fileSet string, opts ...Option) *FileSet

New creates a new in-memory fileset.

func (*Storage) NewReader

func (s *Storage) NewReader(ctx context.Context, fileSet string, opts ...index.Option) *Reader

NewReader creates a new Reader for a file set. (bryce) expose some notion of read ahead (read a certain number of chunks in parallel). this will be necessary to speed up reading large files.

func (*Storage) NewWriter

func (s *Storage) NewWriter(ctx context.Context, fileSet string) *Writer

NewWriter creates a new Writer.

func (*Storage) Shard

func (s *Storage) Shard(ctx context.Context, fileSets []string, shardFunc ShardFunc) error

Shard shards the merge of the file sets with the passed in prefix into file ranges. (bryce) this should be extended to be more configurable (different criteria for creating shards).

type StorageOption

type StorageOption func(s *Storage)

StorageOption configures a storage.

func ServiceEnvToOptions

func ServiceEnvToOptions(env *serviceenv.ServiceEnv) []StorageOption

ServiceEnvToOptions converts a service environment configuration (specifically the storage configuration) to a set of storage options.

func WithMemoryThreshold

func WithMemoryThreshold(threshold int64) StorageOption

WithMemoryThreshold sets the memory threshold that must be met before a file set part is serialized (excluding close).

func WithShardThreshold

func WithShardThreshold(threshold int64) StorageOption

WithShardThreshold sets the size threshold that must be met before a shard is created by the shard function.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer writes the serialized format of a fileset. The serialized format of a fileset consists of indexes and content which are both realized as compressed tar stream chunks.

func (*Writer) Close

func (w *Writer) Close() error

Close closes the writer. (bryce) is not closing the tar writer the right choice here? We cannot include it in the last file's range because it will effect the hash. Our indexing is what will exit the reading of the tar stream, not the end of tar entry (two empty 512 bytes).

func (*Writer) CopyFiles

func (w *Writer) CopyFiles(r *Reader, pathBound ...string) error

CopyFiles does a cheap copy of files from a reader to a writer. (bryce) need to handle delete operations.

func (*Writer) CopyTags

func (w *Writer) CopyTags(r *Reader, tagBound ...string) error

CopyTags does a cheap copy of tagged file data from a reader to a writer.

func (*Writer) StartTag

func (w *Writer) StartTag(id string)

StartTag starts a tag for the next set of bytes (used for the reverse index, mapping file output to datums).

func (*Writer) Write

func (w *Writer) Write(data []byte) (int, error)

Write writes to the current file in the tar stream.

func (*Writer) WriteHeader

func (w *Writer) WriteHeader(hdr *index.Header) error

WriteHeader writes a tar header and prepares to accept the file's contents.