fileset

package
v1.10.0-rc4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 6, 2020 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Index

Constants

View Source 
const (

	// DefaultMemoryThreshold is the default for the memory threshold that must
	// be met before a file set part is serialized (excluding close).
	DefaultMemoryThreshold = 1024 * chunk.MB
	// DefaultShardThreshold is the default for the size threshold that must
	// be met before a shard is created by the shard function.
	DefaultShardThreshold = 1024 * chunk.MB
	// DefaultLevelZeroSize is the default size for level zero in the compacted
	// representation of a file set.
	DefaultLevelZeroSize = 1 * chunk.MB
	// DefaultLevelSizeBase is the default base of the exponential growth function
	// for level sizes in the compacted representation of a file set.
	DefaultLevelSizeBase = 10
	// Diff is the suffix of a path that points to the diff of the prefix.
	Diff = "diff"
	// Compacted is the suffix of a path that points to the compaction of the prefix.
	Compacted = "compacted"
)

Variables

This section is empty.

Functions

func SubFileSetStr 

func SubFileSetStr(subFileSet int64) string

SubFileSetStr returns the string representation of a subfileset.

Types

type CompactSpec 

type CompactSpec struct {
	Output string
	Input  []string
}

CompactSpec specifies the input and output for a compaction operation.

type FileMergeReader 

type FileMergeReader struct {
	// contains filtered or unexported fields
}

FileMergeReader is an abstraction for reading a merged file.

func (*FileMergeReader) Get 

func (fmr *FileMergeReader) Get(w io.Writer) error

Get writes the merged file. (bryce) it might be cleaner to check if w is of type *Writer then use WriteTo rather than Get.

func (*FileMergeReader) Header 

func (fmr *FileMergeReader) Header() (*tar.Header, error)

Header returns the tar header for the merged file.

func (*FileMergeReader) Index 

func (fmr *FileMergeReader) Index() *index.Index

Index returns the index for the merged file.

func (*FileMergeReader) TagSetMergeReader 

func (fmr *FileMergeReader) TagSetMergeReader() (*TagSetMergeReader, error)

TagSetMergeReader returns the tagset merge reader for the file. This is how you would get just the data in the file (excludes the tar header and padding).

func (*FileMergeReader) WriteTo 

func (fmr *FileMergeReader) WriteTo(w *Writer) error

WriteTo writes the merged file to the passed in fileset writer.

type FileReader 

type FileReader struct {
	// contains filtered or unexported fields
}

FileReader is an abstraction for reading a file.

func (*FileReader) Get 

func (fr *FileReader) Get(w io.Writer) error

Get writes the file.

func (*FileReader) Header 

func (fr *FileReader) Header() (*tar.Header, error)

Header returns the tar header for the file.

func (*FileReader) Index 

func (fr *FileReader) Index() *index.Index

Index returns the index for the file.

func (*FileReader) Iterate 

func (fr *FileReader) Iterate(f func(*chunk.DataReader) error, tagUpperBound ...string) error

Iterate iterates over the data readers for the data in the file. tagUpperBound is an optional parameter for specifiying the upper bound (exclusive) of the iteration.

func (*FileReader) NextTagReader 

func (fr *FileReader) NextTagReader() *chunk.TagReader

NextTagReader returns a tag reader for the next tagged data in the file.

func (*FileReader) PeekTag 

func (fr *FileReader) PeekTag() (*chunk.Tag, error)

PeekTag returns the next tag in the file without progressing the reader.

type FileSet added in v1.9.6 

type FileSet struct {
	// contains filtered or unexported fields
}

FileSet is a set of files. This may be a full filesystem or a subfilesystem (e.g. datum / datum set / shard).

func (*FileSet) Close added in v1.9.6 

func (f *FileSet) Close() error

Close closes the file set.

func (*FileSet) Delete added in v1.9.6 

func (f *FileSet) Delete(name string)

Delete deletes a file from the file set. (bryce) might need to delete ancestor directories in certain cases.

func (*FileSet) Put 

func (f *FileSet) Put(r io.Reader) error

Put reads files from a tar stream and adds them to the fileset.

type MergeReader 

type MergeReader struct {
	// contains filtered or unexported fields
}

MergeReader merges a file's content that shows up across multiple fileset streams. A file's content is ordered based on the lexicographical order of the tagged content, so the output file content is produced by performing a merge of the tagged content.

func (*MergeReader) Get 

func (mr *MergeReader) Get(w io.Writer) error

Get writes the merged fileset.

func (*MergeReader) Iterate 

func (mr *MergeReader) Iterate(f func(*FileMergeReader) error) error

Iterate iterates over the file merge readers in the merged fileset.

func (*MergeReader) WriteTo 

func (mr *MergeReader) WriteTo(w *Writer) error

WriteTo writes the merged fileset to the passed in fileset writer.

type Option added in v1.9.6 

type Option func(f *FileSet)

Option configures a file set.

func WithRoot added in v1.9.6 

func WithRoot(root string) Option

WithRoot sets the root path of the file set.

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

Reader reads the serialized format of a fileset.

func (*Reader) Get 

func (r *Reader) Get(w io.Writer) error

Get writes the fileset.

func (*Reader) Iterate 

func (r *Reader) Iterate(f func(*FileReader) error, pathBound ...string) error

Iterate iterates over the file readers in the fileset. pathBound is an optional parameter for specifiying the upper bound (exclusive) of the iteration.

func (*Reader) Next 

func (r *Reader) Next() (*FileReader, error)

Next returns the next file reader and progresses the reader.

func (*Reader) Peek 

func (r *Reader) Peek() (*index.Index, error)

Peek returns the next file index without progressing the reader.

type ShardFunc added in v1.9.8 

type ShardFunc func(*index.PathRange) error

ShardFunc is a callback that returns a PathRange for each shard.

type Storage 

type Storage struct {
	// contains filtered or unexported fields
}

Storage is the abstraction that manages fileset storage.

func NewStorage 

func NewStorage(objC obj.Client, chunks *chunk.Storage, opts ...StorageOption) *Storage

NewStorage creates a new Storage.

func (*Storage) Compact 

func (s *Storage) Compact(ctx context.Context, outputFileSet string, inputFileSets []string, opts ...index.Option) error

Compact compacts a set of filesets into an output fileset.

func (*Storage) CompactSpec 

func (s *Storage) CompactSpec(ctx context.Context, fileSet string, compactedFileSet ...string) (*CompactSpec, error)

CompactSpec returns a compaction specification that determines the input filesets (the diff file set and potentially compacted filesets) and output fileset.

func (*Storage) Delete 

func (s *Storage) Delete(ctx context.Context, fileSet string) error

Delete deletes a fileset.

func (*Storage) New added in v1.9.6 

func (s *Storage) New(ctx context.Context, fileSet, tag string, opts ...Option) *FileSet

New creates a new in-memory fileset.

func (*Storage) NewMergeReader 

func (s *Storage) NewMergeReader(ctx context.Context, fileSets []string, opts ...index.Option) (*MergeReader, error)

NewMergeReader returns a merge reader for a set for filesets.

func (*Storage) Shard added in v1.9.8 

func (s *Storage) Shard(ctx context.Context, fileSets []string, shardFunc ShardFunc) error

Shard shards the merge of the file sets with the passed in prefix into file ranges. (bryce) this should be extended to be more configurable (different criteria for creating shards).

type StorageOption added in v1.9.8 

type StorageOption func(s *Storage)

StorageOption configures a storage.

func ServiceEnvToOptions added in v1.9.8 

func ServiceEnvToOptions(env *serviceenv.ServiceEnv) []StorageOption

ServiceEnvToOptions converts a service environment configuration (specifically the storage configuration) to a set of storage options.

func WithLevelSizeBase 

func WithLevelSizeBase(base int) StorageOption

WithLevelSizeBase sets the base of the exponential growth function for level sizes in the compacted representation of a file set.

func WithLevelZeroSize 

func WithLevelZeroSize(size int64) StorageOption

WithLevelZeroSize sets the size for level zero in the compacted representation of a file set.

func WithMemoryThreshold added in v1.9.8 

func WithMemoryThreshold(threshold int64) StorageOption

WithMemoryThreshold sets the memory threshold that must be met before a file set part is serialized (excluding close).

func WithShardThreshold added in v1.9.8 

func WithShardThreshold(threshold int64) StorageOption

WithShardThreshold sets the size threshold that must be met before a shard is created by the shard function.

type TagMergeReader 

type TagMergeReader struct {
	// contains filtered or unexported fields
}

TagMergeReader is an abstraction for reading a merged tag. This abstraction is necessary because a tag in a file can appear across multiple filesets.

func (*TagMergeReader) Get 

func (tmr *TagMergeReader) Get(w io.Writer) error

Get writes the merged tagged data.

func (*TagMergeReader) Iterate 

func (tmr *TagMergeReader) Iterate(f func(*chunk.DataReader) error) error

Iterate iterates over the data readers for the tagged data being merged.

func (*TagMergeReader) WriteTo 

func (tmr *TagMergeReader) WriteTo(w *Writer) error

WriteTo writes the merged tagged data to the passed in fileset writer.

type TagSetMergeReader 

type TagSetMergeReader struct {
	// contains filtered or unexported fields
}

TagSetMergeReader is an abstraction for reading the merged tagged data in a merged file.

func (*TagSetMergeReader) Get 

func (tsmr *TagSetMergeReader) Get(w io.Writer) error

Get writes the merged tagset.

func (*TagSetMergeReader) Iterate 

func (tsmr *TagSetMergeReader) Iterate(f func(*TagMergeReader) error) error

Iterate iterates over the tag merge readers in the merged tagset.

func (*TagSetMergeReader) WriteTo 

func (tsmr *TagSetMergeReader) WriteTo(w *Writer) error

WriteTo writes the merged tagset to the passed in fileset writer.

type Writer 

type Writer struct {
	// contains filtered or unexported fields
}

Writer writes the serialized format of a fileset. The serialized format of a fileset consists of indexes and content.

func (*Writer) Close 

func (w *Writer) Close() error

Close closes the writer.

func (*Writer) CopyFile 

func (w *Writer) CopyFile(fr *FileReader) error

CopyFile copies a file (header and tags included).

func (*Writer) CopyTags added in v1.9.5 

func (w *Writer) CopyTags(dr *chunk.DataReader) error

CopyTags copies the tagged data from the passed in data reader.

func (*Writer) Tag 

func (w *Writer) Tag(id string)

Tag starts a tag for the next set of bytes (used for the reverse index, mapping file output to datums).

func (*Writer) Write 

func (w *Writer) Write(data []byte) (int, error)

Write writes to the current file in the tar stream.

func (*Writer) WriteHeader 

func (w *Writer) WriteHeader(hdr *tar.Header) error

WriteHeader writes a tar header and prepares to accept the file's contents.

Source Files

View all Source files

Directories

Path Synopsis
Package tar implements access to tar archives.
 Package tar implements access to tar archives.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.