storage

Documentation

Overview

Package storage is for metadata of a tar archive.

Packing and unpacking the Entries of the stream. The types of streams are either segments of raw bytes (for the raw headers and various padding) and for an entry marking a file payload.

The raw bytes are stored precisely in the packed (marshalled) Entry, whereas the file payload marker include the name of the file, size, and crc64 checksum (for basic file integrity).

Index

• Variables
• type Entries
  • func (e Entries) Len() int
  • func (e Entries) Less(i, j int) bool
  • func (e Entries) Swap(i, j int)
• type Entry
  • func (e *Entry) GetName() string
  • func (e *Entry) GetNameBytes() []byte
  • func (e *Entry) SetName(name string)
  • func (e *Entry) SetNameBytes(name []byte)
• type FileGetPutter
  • func NewBufferFileGetPutter() FileGetPutter
• type FileGetter
  • func NewPathFileGetter(relpath string) FileGetter
• type FilePutter
  • func NewDiscardFilePutter() FilePutter
• type Packer
  • func NewJSONPacker(w io.Writer) Packer
• type Type
• type Unpacker
  • func NewJSONUnpacker(r io.Reader) Unpacker

Variables

var CRCTable = crc64.MakeTable(crc64.ISO)

CRCTable is the default table used for crc64 sum calculations

var ErrDuplicatePath = errors.New("duplicates of file paths not supported")

ErrDuplicatePath occurs when a tar archive has more than one entry for the same file path

Types

type Entries

type Entries []Entry

Entries is for sorting by Position

func (Entries) Len

func (e Entries) Len() int

func (Entries) Less

func (e Entries) Less(i, j int) bool

func (Entries) Swap

func (e Entries) Swap(i, j int)

type Entry

type Entry struct {
	Type     Type   `json:"type"`
	Name     string `json:"name,omitempty"`
	NameRaw  []byte `json:"name_raw,omitempty"`
	Size     int64  `json:"size,omitempty"`
	Payload  []byte `json:"payload"` // SegmentType stores payload here; FileType stores crc64 checksum here;
	Position int    `json:"position"`
}

Entry is the structure for packing and unpacking the information read from the Tar archive.

FileType Payload checksum is using `hash/crc64` for basic file integrity, _not_ for cryptography. From http://www.backplane.com/matt/crc64.html, CRC32 has almost 40,000 collisions in a sample of 18.2 million, CRC64 had none.

func (*Entry) GetName

func (e *Entry) GetName() string

GetName returns the string for the entry's name, regardless of the field stored in

func (*Entry) GetNameBytes

func (e *Entry) GetNameBytes() []byte

GetNameBytes returns the bytes for the entry's name, regardless of the field stored in

func (*Entry) SetName

func (e *Entry) SetName(name string)

SetName will check name for valid UTF-8 string, and set the appropriate field. See https://github.com/vbatts/tar-split/issues/17

func (*Entry) SetNameBytes

func (e *Entry) SetNameBytes(name []byte)

SetNameBytes will check name for valid UTF-8 string, and set the appropriate field

type FileGetPutter

type FileGetPutter interface {
	FileGetter
	FilePutter
}

FileGetPutter is the interface that groups both Getting and Putting file payloads.

func NewBufferFileGetPutter

func NewBufferFileGetPutter() FileGetPutter

NewBufferFileGetPutter is a simple in-memory FileGetPutter

Implication is this is memory intensive... Probably best for testing or light weight cases.

type FileGetter

type FileGetter interface {
	// Get returns a stream for the provided file path
	Get(filename string) (output io.ReadCloser, err error)
}

FileGetter is the interface for getting a stream of a file payload, addressed by name/filename. Presumably, the names will be scoped to relative file paths.

func NewPathFileGetter

func NewPathFileGetter(relpath string) FileGetter

NewPathFileGetter returns a FileGetter that is for files relative to path relpath.

type FilePutter

type FilePutter interface {
	// Put returns the size of the stream received, and the crc64 checksum for
	// the provided stream
	Put(filename string, input io.Reader) (size int64, checksum []byte, err error)
}

FilePutter is the interface for storing a stream of a file payload, addressed by name/filename.

func NewDiscardFilePutter

func NewDiscardFilePutter() FilePutter

NewDiscardFilePutter is a bit bucket FilePutter

type Packer

type Packer interface {
	// AddEntry packs the Entry and returns its position
	AddEntry(e Entry) (int, error)
}

Packer describes the methods to pack Entries to a storage destination

func NewJSONPacker

func NewJSONPacker(w io.Writer) Packer

NewJSONPacker provides a Packer that writes each Entry (SegmentType and FileType) as a json document.

The Entries are delimited by new line.

type Type

type Type int

Type of Entry

const (
	// FileType represents a file payload from the tar stream.
	//
	// This will be used to map to relative paths on disk. Only Size > 0 will get
	// read into a resulting output stream (due to hardlinks).
	FileType Type = 1 + iota
	// SegmentType represents a raw bytes segment from the archive stream. These raw
	// byte segments consist of the raw headers and various padding.
	//
	// Its payload is to be marshalled base64 encoded.
	SegmentType
)

type Unpacker

type Unpacker interface {
	// Next returns the next Entry being unpacked, or error, until io.EOF
	Next() (*Entry, error)
}

Unpacker describes the methods to read Entries from a source

func NewJSONUnpacker

func NewJSONUnpacker(r io.Reader) Unpacker

NewJSONUnpacker provides an Unpacker that reads Entries (SegmentType and FileType) as a json document.

Each Entry read are expected to be delimited by new line.