compression

Documentation

Index

• Constants
• type GzipZinfo
  • func (i *GzipZinfo) Bytes() ([]byte, error)
  • func (i *GzipZinfo) Close()
  • func (i *GzipZinfo) EndCompressedOffset(spanID SpanID, fileSize Offset) Offset
  • func (i *GzipZinfo) EndUncompressedOffset(spanID SpanID, fileSize Offset) Offset
  • func (i *GzipZinfo) ExtractDataFromBuffer(compressedBuf []byte, uncompressedSize, uncompressedOffset Offset, ...) ([]byte, error)
  • func (i *GzipZinfo) ExtractDataFromFile(fileName string, uncompressedSize, uncompressedOffset Offset) ([]byte, error)
  • func (i *GzipZinfo) MaxSpanID() SpanID
  • func (i *GzipZinfo) SpanSize() Offset
  • func (i *GzipZinfo) StartCompressedOffset(spanID SpanID) Offset
  • func (i *GzipZinfo) StartUncompressedOffset(spanID SpanID) Offset
  • func (i *GzipZinfo) UncompressedOffsetToSpanID(offset Offset) SpanID
  • func (i *GzipZinfo) VerifyHeader(r io.Reader) error
• type Offset
• type SpanID
• type TarZinfo
  • func (i *TarZinfo) Bytes() (fb []byte, err error)
  • func (i *TarZinfo) Close()
  • func (i *TarZinfo) EndCompressedOffset(spanID SpanID, fileSize Offset) Offset
  • func (i *TarZinfo) EndUncompressedOffset(spanID SpanID, fileSize Offset) Offset
  • func (i *TarZinfo) ExtractDataFromBuffer(compressedBuf []byte, uncompressedSize, uncompressedOffset Offset, ...) ([]byte, error)
  • func (i *TarZinfo) ExtractDataFromFile(fileName string, uncompressedSize, uncompressedOffset Offset) ([]byte, error)
  • func (i *TarZinfo) MaxSpanID() SpanID
  • func (i *TarZinfo) SpanSize() Offset
  • func (i *TarZinfo) StartCompressedOffset(spanID SpanID) Offset
  • func (i *TarZinfo) StartUncompressedOffset(spanID SpanID) Offset
  • func (i *TarZinfo) UncompressedOffsetToSpanID(offset Offset) SpanID
  • func (i *TarZinfo) VerifyHeader(r io.Reader) error
• type Zinfo
  • func NewZinfo(compressionAlgo string, zinfoBytes []byte) (Zinfo, error)
  • func NewZinfoFromFile(compressionAlgo string, filename string, spanSize int64) (Zinfo, error)

Constants

const (
	Gzip         = "gzip"
	Zstd         = "zstd"
	Uncompressed = "uncompressed"
	Unknown      = "unknown"
)

Compression algorithms used by an image layer. They should be kept consistent with the return of `DiffCompression` from containerd. https://github.com/containerd/containerd/blob/v1.7.0-beta.3/images/mediatypes.go#L66

Types

type GzipZinfo

type GzipZinfo struct {
	// contains filtered or unexported fields
}

GzipZinfo is a go struct wrapper of the gzip zinfo's C implementation.

func (*GzipZinfo) Bytes

func (i *GzipZinfo) Bytes() ([]byte, error)

Bytes returns the byte slice containing the zinfo.

func (*GzipZinfo) Close

func (i *GzipZinfo) Close()

Close calls `C.free` on the pointer to `C.struct_gzip_zinfo`.

func (*GzipZinfo) EndCompressedOffset

func (i *GzipZinfo) EndCompressedOffset(spanID SpanID, fileSize Offset) Offset

EndCompressedOffset returns the end offset of the span in the compressed stream. If it's the last span, returns the size of the compressed stream.

func (*GzipZinfo) EndUncompressedOffset

func (i *GzipZinfo) EndUncompressedOffset(spanID SpanID, fileSize Offset) Offset

EndUncompressedOffset returns the end offset of the span in the uncompressed stream. If it's the last span, returns the size of the uncompressed stream.

func (*GzipZinfo) ExtractDataFromBuffer

func (i *GzipZinfo) ExtractDataFromBuffer(compressedBuf []byte, uncompressedSize, uncompressedOffset Offset, spanID SpanID) ([]byte, error)

ExtractDataFromBuffer wraps the call to `C.extract_data_from_buffer`, which takes in the compressed bytes and returns the decompressed bytes.

func (*GzipZinfo) ExtractDataFromFile

func (i *GzipZinfo) ExtractDataFromFile(fileName string, uncompressedSize, uncompressedOffset Offset) ([]byte, error)

ExtractDataFromFile wraps `C.extract_data_from_file` and returns the decompressed bytes given the name of the .tar.gz file, offset and the size in uncompressed stream.

func (*GzipZinfo) MaxSpanID

func (i *GzipZinfo) MaxSpanID() SpanID

MaxSpanID returns the max span ID.

func (*GzipZinfo) SpanSize

func (i *GzipZinfo) SpanSize() Offset

SpanSize returns the span size of the constructed ztoc.

func (*GzipZinfo) StartCompressedOffset

func (i *GzipZinfo) StartCompressedOffset(spanID SpanID) Offset

StartCompressedOffset returns the start offset of the span in the compressed stream.

func (*GzipZinfo) StartUncompressedOffset

func (i *GzipZinfo) StartUncompressedOffset(spanID SpanID) Offset

StartUncompressedOffset returns the start offset of the span in the uncompressed stream.

func (*GzipZinfo) UncompressedOffsetToSpanID

func (i *GzipZinfo) UncompressedOffsetToSpanID(offset Offset) SpanID

UncompressedOffsetToSpanID returns the ID of the span containing the data pointed by uncompressed offset.

func (*GzipZinfo) VerifyHeader

func (i *GzipZinfo) VerifyHeader(r io.Reader) error

VerifyHeader checks if the given zinfo has a proper header

type Offset

type Offset int64

Offset will hold any file size and offset values

type SpanID

type SpanID int32

SpanID will hold any span related values (SpanID, MaxSpanID, etc)

type TarZinfo

type TarZinfo struct {
	// contains filtered or unexported fields
}

TarZinfo implements the `Zinfo` interface for uncompressed tar files. It only needs a span size and tar file size, since a tar file is already uncompressed. For tar file, `compressed`-related concepts (e.g., `CompressedArchiveSize`) are only to santisfy the `Zinfo` interface and equal to their `uncompressed`-equivalent.

func (*TarZinfo) Bytes

func (i *TarZinfo) Bytes() (fb []byte, err error)

Bytes returns the byte slice containing the `TarZinfo`. Integers are serialized to `LittleEndian` binaries.

func (*TarZinfo) Close

func (i *TarZinfo) Close()

Close doesn't do anything since there is nothing to close/release.

func (*TarZinfo) EndCompressedOffset

func (i *TarZinfo) EndCompressedOffset(spanID SpanID, fileSize Offset) Offset

EndCompressedOffset returns the end offset of the span in the compressed stream. If it's the last span, returns the size of the compressed stream.

func (*TarZinfo) EndUncompressedOffset

func (i *TarZinfo) EndUncompressedOffset(spanID SpanID, fileSize Offset) Offset

EndUncompressedOffset returns the end offset of the span in the uncompressed stream. If it's the last span, returns the size of the uncompressed stream.

func (*TarZinfo) ExtractDataFromBuffer

func (i *TarZinfo) ExtractDataFromBuffer(compressedBuf []byte, uncompressedSize, uncompressedOffset Offset, spanID SpanID) ([]byte, error)

ExtractDataFromBuffer does sanity checks and returns the bytes specified by offset and size from the buffer, since for tar file the buffer is already uncompressed.

func (*TarZinfo) ExtractDataFromFile

func (i *TarZinfo) ExtractDataFromFile(fileName string, uncompressedSize, uncompressedOffset Offset) ([]byte, error)

ExtractDataFromFile does sanity checks and returns the bytes specified by offset and size by reading from the tar file, since for tar file the buffer is already uncompressed.

func (*TarZinfo) MaxSpanID

func (i *TarZinfo) MaxSpanID() SpanID

MaxSpanID returns the max span ID.

func (*TarZinfo) SpanSize

func (i *TarZinfo) SpanSize() Offset

SpanSize returns the span size of the constructed zinfo.

func (*TarZinfo) StartCompressedOffset

func (i *TarZinfo) StartCompressedOffset(spanID SpanID) Offset

StartCompressedOffset returns the start offset of the span in the compressed stream.

func (*TarZinfo) StartUncompressedOffset

func (i *TarZinfo) StartUncompressedOffset(spanID SpanID) Offset

StartUncompressedOffset returns the start offset of the span in the uncompressed stream.

func (*TarZinfo) UncompressedOffsetToSpanID

func (i *TarZinfo) UncompressedOffsetToSpanID(offset Offset) SpanID

UncompressedOffsetToSpanID returns the ID of the span containing the data pointed by uncompressed offset.

func (*TarZinfo) VerifyHeader

func (i *TarZinfo) VerifyHeader(r io.Reader) error

VerifyHeader checks if the given zinfo has a proper header

type Zinfo

type Zinfo interface {
	// ExtractDataFromBuffer extracts the uncompressed data from `compressedBuf` and returns
	// as a byte slice.
	ExtractDataFromBuffer(compressedBuf []byte, uncompressedSize, uncompressedOffset Offset, spanID SpanID) ([]byte, error)
	// ExtractDataFromFile extracts the uncompressed data directly from a compressed file
	// (e.g. a gzip file) and returns as a byte slice.
	ExtractDataFromFile(fileName string, uncompressedSize, uncompressedOffset Offset) ([]byte, error)
	// Close releases any resources held by the interface implementation.
	Close()

	// Bytes serilizes the underlying zinfo data (depending on implementation) into bytes for storage.
	Bytes() ([]byte, error)
	// MaxSpanID returns the maximum span ID after chunking the compress stream into spans.
	MaxSpanID() SpanID
	// SpanSize returns the span size used to chunk compress stream into spans.
	SpanSize() Offset

	// UncompressedOffsetToSpanID returns the ID of the span containing given `offset`.
	UncompressedOffsetToSpanID(offset Offset) SpanID
	// StartCompressedOffset returns the offset (in compressed stream)
	// of the 1st byte belonging to `spanID`.
	StartCompressedOffset(spanID SpanID) Offset
	// EndCompressedOffset returns the offset (in compressed stream)
	// of the last byte belonging to `spanID`. If it's the last span, `fileSize` is returned.
	EndCompressedOffset(spanID SpanID, fileSize Offset) Offset
	// StartUncompressedOffset returns the offset (in uncompressed stream)
	// of the 1st byte belonging to `spanID`.
	StartUncompressedOffset(spanID SpanID) Offset
	// EndUncompressedOffset returns the offset (in uncompressed stream)
	// of the last byte belonging to `spanID`. If it's the last span, `fileSize` is returned.
	EndUncompressedOffset(spanID SpanID, fileSize Offset) Offset
	// VerifyHeader checks if the given zinfo has a proper header
	VerifyHeader(r io.Reader) error
}

Zinfo is the interface for dealing with compressed data efficiently. It chunks a compressed stream (e.g. a gzip file) into spans and records the chunk offset, so that you can interact with the compressed stream per span individually (or in parallel). For example, you can extract uncompressed data/file from the relevant compressed spans only (i.e., without uncompressing the whole compress file).

The interface contains methods that are used to:

1. build a zinfo (e.g., `SpanSize`);
2. extract a chunk of uncompressed data (e.g., from a compressed buffer or file);
3. conversion between span and its start and end offset in the (un)compressed stream so you can work on the individual span data only.

func NewZinfo

func NewZinfo(compressionAlgo string, zinfoBytes []byte) (Zinfo, error)

NewZinfo deseralizes given zinfo bytes into a zinfo struct. This is often used when you have a serialized zinfo bytes and want to get the zinfo struct.

func NewZinfoFromFile

func NewZinfoFromFile(compressionAlgo string, filename string, spanSize int64) (Zinfo, error)

NewZinfoFromFile creates a zinfo struct given a compressed file and a span size. This is often used when you have a compressed file (e.g. gzip) and want to create a new zinfo for it.