decmpfs

Documentation

Overview

Package decmpfs decodes Apple's transparent file compression.

decmpfs is a property of the file, not of the file system: a com.apple.decmpfs extended attribute holds a 16-byte header naming the compression type and the uncompressed size, and the compressed bytes live either immediately after that header ("inline") or in the file's resource fork, chunked into 64 KiB blocks. HFS+ introduced it and APFS carries it unchanged, so both readers need the same decoder and neither should own it.

The caller supplies a Source — the byte range the compressed data lives in — and this package handles the header, the block table and the codecs. What it deliberately does not know is how to find either of those things, since that is where the two file systems differ.

Index

• Constants
• Variables
• func CheckInline(attrValue []byte) error
• func Decompress(compressedData []byte, compressionMethod int, uncompressedData []byte, ...) error
• func MethodFor(decmpfsType uint32) (int, error)
• func StoresDataInResourceFork(decmpfsType uint32) bool
• func Validate(attrValue []byte, dataForkSize int, resourceFork []byte) error
• type Handle
  • func NewHandle(compressedDataStream Source, uncompressedDataSize uint64, ...) (*Handle, error)
  • func (cdh *Handle) Close() error
  • func (cdh *Handle) ReadSegmentData(segmentIndex int, segmentData []byte) (int, error)
  • func (cdh *Handle) SeekSegmentOffset(segmentIndex int, segmentOffset int64) (int64, error)
• type Header
  • func ParseHeader(data []byte) (*Header, error)
  • func (h *Header) ReadData(data []byte) (bool, error)
• type Source

Constants

const (
	AttributeName    = "com.apple.decmpfs"
	ResourceForkName = "com.apple.ResourceFork"
)

The two extended attributes a compressed file is made of: the header (and, for the odd types, the payload) in AttributeName, and the chunked payload for the even types in ResourceForkName. They are named here so the readers and writers of both file systems agree on the spelling.

const (
	MethodNone    = 0
	MethodDeflate = 1
	MethodLZFSE   = 2
	MethodLZVN    = 3

	// MethodUnknown5 is retained only because it is reachable through a
	// deprecated alias in pkg/apfs. Nothing maps to it: see MethodFor.
	MethodUnknown5 = 5
)

Internal compression method codes. These are this package's own vocabulary, not the decmpfs type numbers stored on disk; MethodFor translates. Several on-disk types share one method, because whether the data sits inline or in a resource fork is decided by where it is read from rather than by how it is decoded.

const BlockSize = 65536

BlockSize is the uncompressed size of one decmpfs chunk. Every block except the last decompresses to exactly this much.

It is unrelated to any file system's block size.

const HeaderSize = 16

HeaderSize is the size of the com.apple.decmpfs header in bytes.

Variables

var HeaderSignature = [4]byte{'f', 'p', 'm', 'c'}

HeaderSignature is the magic every com.apple.decmpfs attribute begins with.

Functions

func CheckInline

func CheckInline(attrValue []byte) error

CheckInline verifies that attrValue is a whole com.apple.decmpfs attribute with its header still attached, as inline decoding requires.

The block-offset loader identifies inline storage by finding the header magic at offset zero. Handing it the payload with the header stripped hides that magic and sends the stream down the resource-fork branch, which misparses the first bytes of the compressed data as a block-offset table and produces plausible nonsense. Checking here makes that mistake fail loudly instead.

func Decompress

func Decompress(
	compressedData []byte,
	compressionMethod int,
	uncompressedData []byte,
	uncompressedDataSize *int,
) error

Decompress decompresses one decmpfs chunk with the given method, writing into uncompressedData and updating uncompressedDataSize to the number of bytes produced.

func MethodFor

func MethodFor(decmpfsType uint32) (int, error)

MethodFor maps a raw com.apple.decmpfs type to the internal method code.

The types come in attribute/resource-fork pairs, the odd member of each pair storing the data inline in the com.apple.decmpfs attribute and the even member storing it in the com.apple.ResourceFork attribute, chunked. The numbering is Apple's, from copyfile.c:

 1       uncompressed, stored inline
 3 /  4  zlib
 5       de-duplication within the generation store
 6       unused
 7 /  8  LZVN
 9 / 10  uncompressed
11 / 12  LZFSE
13 / 14  LZBITMAP

The attribute-versus-resource-fork distinction is handled by where the data is read from, not by the method code, so each supported pair maps to one method.

func StoresDataInResourceFork

func StoresDataInResourceFork(decmpfsType uint32) bool

StoresDataInResourceFork reports whether an on-disk decmpfs type keeps its payload in com.apple.ResourceFork rather than in the attribute itself.

The types come in pairs, the odd member storing inline and the even member storing in the fork; see MethodFor.

func Validate

func Validate(attrValue []byte, dataForkSize int, resourceFork []byte) error

Validate checks that a com.apple.decmpfs attribute and the file carrying it agree, so a writer refuses an inconsistent file rather than producing one a checker will reject or a reader will silently misread.

dataForkSize is the length of the file's data fork and resourceFork the com.apple.ResourceFork attribute value, nil when the file has none.

Types

type Handle

type Handle struct {
	// The current segment offset
	CurrentSegmentOffset int64

	// CompressedDataStream is the compressed byte range. The field keeps its
	// original name because it is reachable as a field of pkg/apfs's
	// CompressedDataHandle, which is an alias for this type.
	CompressedDataStream Source

	// The uncompressed data size
	UncompressedDataSize uint64

	// The compression method
	CompressionMethod int

	// The current compressed block index
	CurrentCompressedBlockIndex uint32

	// The compressed segment data buffer
	CompressedSegmentData []byte

	// The (uncompressed) segment data buffer
	SegmentData []byte

	// The (uncompressed) segment data size
	SegmentDataSize int

	// The number of compressed blocks
	NumberOfCompressedBlocks uint32

	// The compressed block offsets
	CompressedBlockOffsets []uint32
}

Handle decodes one compressed stream: it locates the chunk boundaries in the Source, then decompresses on demand, caching the current chunk.

A Handle is stateful and not safe for concurrent use. It also holds two BlockSize buffers, so opening many compressed files at once is not free — build one per open file and let it go when the file closes.

func NewHandle

func NewHandle(
	compressedDataStream Source,
	uncompressedDataSize uint64,
	compressionMethod int,
) (*Handle, error)

NewHandle creates a decoder for a compressed stream. compressionMethod is one of this package's method codes, as returned by MethodFor.

func (*Handle) Close

func (cdh *Handle) Close() error

Close releases the decoder's buffers.

func (*Handle) ReadSegmentData

func (cdh *Handle) ReadSegmentData(
	segmentIndex int,
	segmentData []byte,
) (int, error)

ReadSegmentData reads decompressed data from the current offset into segmentData. It is the read callback a data stream drives.

func (*Handle) SeekSegmentOffset

func (cdh *Handle) SeekSegmentOffset(
	segmentIndex int,
	segmentOffset int64,
) (int64, error)

SeekSegmentOffset moves the decoder to an offset in the uncompressed data.

type Header

type Header struct {
	// CompressionMethod is the raw on-disk decmpfs type, not one of this
	// package's method codes. Pass it to MethodFor.
	CompressionMethod uint32

	// UncompressedDataSize is the size the file presents once decoded. It is
	// the authoritative length: the data fork of a compressed file is empty.
	UncompressedDataSize uint64
}

Header is the com.apple.decmpfs attribute header.

Its fields are little-endian regardless of the file system underneath. That is unremarkable on APFS, whose structures are little-endian throughout, but it is a trap on HFS+, where everything else on disk is big-endian: the attribute is an opaque payload that HFS+ never interprets, so it keeps the byte order decmpfs itself uses.

func ParseHeader

func ParseHeader(data []byte) (*Header, error)

ParseHeader parses a com.apple.decmpfs header. It returns nil with no error when the signature does not match.

func (*Header) ReadData

func (h *Header) ReadData(data []byte) (bool, error)

ReadData reads the header from data. It reports whether the signature matched; a non-matching signature is not an error, because the caller may be probing a value that turns out not to be a decmpfs attribute at all.

type Source

type Source interface {
	ReadAt(p []byte, off int64) (int, error)

	// Size is the length of the compressed range in bytes.
	Size() uint64
}

Source is the compressed byte range a decmpfs stream lives in: an APFS data stream, or an HFS+ resource fork, or the attribute value itself when the data is stored inline.