xar

README

xar

A pure-Go, read-only reader for the XAR (eXtensible ARchive) format — the container a macOS .pkg installer is, and the format /usr/bin/xar writes.

CGO_ENABLED=0, no dependency outside the standard library except go-filesystems/interface. compress/zlib, compress/bzip2 and encoding/xml do all the decoding.

Use

import (
	"os"

	"github.com/go-filesystems/xar"
)

f, err := os.Open("Installer.pkg")
if err != nil {
	return err
}
defer f.Close()
st, err := f.Stat()
if err != nil {
	return err
}

fsys, err := xar.OpenReader(f, st.Size())
if err != nil {
	return err
}
defer fsys.Close()

entries, err := fsys.ListDir("/")     // Distribution, PackageInfo, Bom, Payload…
data, err := fsys.ReadFile("/PackageInfo")

OpenReader returns a filesystem.Filesystem. Only the header and the table of contents are read; each member is decoded when it is read.

The optional filesystem.Opener capability is implemented, so a member can be streamed rather than materialised:

if o, ok := fsys.(filesystem.Opener); ok {
	h, err := o.OpenFile("/Payload")
	if err != nil {
		return err
	}
	defer h.Close()
	_, err = io.Copy(dst, io.NewSectionReader(h, 0, h.Size()))
}

A stored member is served straight from the backing io.ReaderAt: real random access, nothing buffered, no lock. A compressed member has no random access to give, because a zlib or bzip2 stream can only be read forwards from its start, so the handle keeps a decoder and its position — a read ahead of it discards forward, a read behind it restarts the stream. Memory is O(1) either way, and a sequential reader pays one pass in total.

An archive is read-only. WriteFile, MkDir, DeleteFile, DeleteDir and Rename return ErrReadOnly, which wraps fs.ErrPermission. A path that is not there wraps fs.ErrNotExist; a corrupt archive deliberately does not, so a damaged file cannot reach a user as a routine 404.

The layout

offset	size	field
0	4	magic, xar! (0x78617221)
4	2	HeaderSize — the size of this header
6	2	Version
8	8	TOCLengthCompressed
16	8	TOCLengthUncompressed
24	4	ChecksumAlgorithm

Then TOCLengthCompressed bytes of zlib-compressed XML, then the heap. The heap begins at HeaderSize + TOCLengthCompressed, and every member's <offset> is relative to that — never to the file.

HeaderSize, not the constant 28, is the base. The field exists so the header can grow, and both halves of the sum matter: using TOCLengthUncompressed instead lands in the middle of the heap, because the compressed table of contents is what physically occupies the file.

The table of contents is <xar><toc>, with <file> elements nested inside one another to express directories. There is no path string anywhere in the format, only a <name> per level, so a path exists only once the nesting has been walked.

Four things the format does that a reader will get wrong

Each was observed in an archive written by xar 1.8dev, not inferred from a specification.

1. encoding style="application/x-gzip" is not gzip. It is a raw zlib stream (RFC 1950) — 78 da, no gzip member header, no gzip trailer. compress/gzip refuses it outright, so at least it fails loudly; the naming is the trap. Stored is application/octet-stream, and bzip2 is application/x-bzip2.

2. An empty file has no <data> element at all. Not <data> with <size>0</size> — the element is simply absent. A reader that requires <data> loses every empty member.

3. <mode> is octal and its leading zero is optional. xar writes <mode>0644</mode> for an ordinary file but <mode>4755</mode> for a setuid one. Both are octal, so strconv.ParseUint with base 0 is wrong in a way no leading-zero mode can reveal: it reads 0644 as octal, because of the zero, and 4755 as decimal. Only an explicit base 8 is right.

4. <ea> looks exactly like <data>. An extended attribute lives in the heap with its own <offset>, <length>, <size>, <encoding> — and its own <name>. A reader that searches a <file> subtree for any descendant offset serves an xattr's bytes as the file's contents, most visibly on an empty file, which has an <ea> and no <data>. One that searches for any descendant name calls the file com.apple.provenance.

What this package does not do

The header's ChecksumAlgorithm, the table of contents' own <checksum>, and each member's <extracted-checksum>/<archived-checksum> are parsed past, not verified. Nothing here authenticates an archive; a .pkg signature in particular is not checked, and a caller that needs provenance must establish it by other means.

LZMA members (application/x-lzma) are refused with ErrUnsupportedEncoding. The refusal is per member and is raised when the data is read, so an archive with one LZMA member still opens and lists. It is untested against a real archive on purpose: macOS's xar 1.8dev reports "lzma support not compiled in" and cannot write one, and a fixture hand-built to agree with this reader would prove nothing about it.

Extended attributes, uid/gid, the timestamps and the Finder metadata are read past. Symlinks are reported, with their target, but not followed — a link target is a string the archive supplied and may name anything at all.

Fixtures

testdata/*.xar are written by the reference implementation, and testdata/gen.sh regenerates them. The script asserts its own premise: it makes the reference list and extract each archive and compares every member byte for byte against the source tree before leaving the fixture behind. Two traps it records:

• xar -cf out.xar -C srcdir files… prints Error adding file X for every member and exits 0, leaving a 208-byte archive with an empty table of contents. xar has to be run with its working directory already inside the source tree.
• xar --compression=lzma reports "lzma support not compiled in", so no LZMA fixture can be produced on macOS.

The fixtures are embedded with //go:embed, not read from testdata at run time: the CI matrix builds the test binary with go test -c and runs it in a container for four emulated architectures, where testdata/ is not mounted.

Tests

100% statement coverage, gated in CI on four native lanes (linux/amd64, linux/arm64, darwin/arm64, windows/amd64) plus four emulated ones (riscv64, loong64, ppc64le, s390x).

Every positive claim is made against an archive the reference wrote. The hand-built archives in craft_test.go exist only to be refused — the reference cannot produce a truncated table of contents or a mode that is not octal — and they are kept honest by TestCraftBaselineOpens: the builder's unmutated output must open and read correctly, so every refusal differs from a working archive by exactly one thing.

Each decision was ablated one at a time and the ablation re-run to see which test caught it. Three ablations initially passed, and each was a missing assertion rather than a harmless one:

ablation	caught by, before	fix
heap base uses the constant 28 instead of HeaderSize	nothing — every archive xar writes declares 28, so the field and the constant are the same number in every real fixture	TestDeclaredHeaderSizeLocatesEverything, a crafted archive with a 32-byte header
the declared TOCLengthUncompressed is not enforced	nothing — the existing case declared one byte fewer, which cut the XML mid-document, so the XML parser caught it and the length check was never reached	TestTOCLongerThanDeclaredWithValidPrefix, whose table of contents ends in a newline so the short prefix is still valid XML
ReadFile sizes its buffer from the untrusted <size>	nothing that mattered — the test declared 2 GiB and asserted an error, and the allocating version allocated 2 GiB and returned the same error	the declared size is now 256 TiB, which a reader that trusts it cannot survive

Licence

BSD-3-Clause.

Documentation

Overview

Package xar is a pure-Go, read-only driver for the XAR (eXtensible ARchive) format — the container a macOS `.pkg` installer is, and the format /usr/bin/xar writes.

OpenReader returns a filesystem.Filesystem over an archive held in any io.ReaderAt, so a `.pkg` can be walked and read without being unpacked. The optional filesystem.Opener capability is implemented, so a caller can stream one member instead of materialising it.

The layout

A 28-byte big-endian header, a zlib-compressed XML table of contents, then a heap of member data:

offset  size  field
0       4     magic, 'xar!' (0x78617221)
4       2     HeaderSize — the size of THIS header
6       2     Version
8       8     TOCLengthCompressed
16      8     TOCLengthUncompressed
24      4     ChecksumAlgorithm

The heap begins at HeaderSize + TOCLengthCompressed. HeaderSize, not the constant 28, is the base: the field exists so the header can grow, and a reader that assumes 28 is reading a different file than the one it was given. Every member's <offset> is relative to that heap start, never to the file.

The table of contents is <xar><toc>, with <file> elements nested inside one another to express directories. A member's path is therefore built by walking the nesting — there is no path string anywhere in the format, only a <name> per level.

Four things the format does that a reader will get wrong

Each of these was observed in an archive written by /usr/bin/xar 1.8dev, not inferred from a specification.

1. encoding style="application/x-gzip" IS NOT GZIP. It is a raw zlib stream (RFC 1950): 0x78 0xda, no gzip member header, no gzip trailer. compress/gzip refuses it outright, which at least fails loudly; the naming is the trap. The stored encoding is "application/octet-stream" and bzip2 is "application/x-bzip2".

2. AN EMPTY FILE HAS NO <data> ELEMENT AT ALL. It is not <data> with <size>0</size>; the element is simply absent. A reader that requires <data> loses every empty member.

3. <mode> IS OCTAL, AND SOMETIMES HAS NO LEADING ZERO. xar writes <mode>0644</mode> for an ordinary file but <mode>4755</mode> for a setuid one. Both are octal. strconv.ParseUint with base 0 is therefore wrong in a way no leading-zero mode can reveal: it reads "0644" as octal, because of the zero, and "4755" as decimal. Only an explicit base 8 is right.

4. <ea> ELEMENTS LOOK EXACTLY LIKE <data>. An extended attribute is stored in the heap with its own <offset>, <length>, <size> and <encoding>, and its own <name>. A reader that searches a <file> for any descendant offset/length hands back an xattr's bytes as the file's contents — most visibly on an empty file, which has an <ea> and no <data>. A reader that takes any descendant <name> names the file after the xattr.

Members are compressed individually, each with its own encoding, so there is no single archive-wide stream and no flat offset index into decoded bytes.

What this package does not do

The header's ChecksumAlgorithm and the TOC's own <checksum> element are parsed past, not verified, and neither are the per-member <extracted-checksum>/<archived-checksum> digests. Nothing here authenticates an archive; a `.pkg` signature in particular is not checked, and a caller that needs provenance must establish it by other means.

LZMA-encoded members ("application/x-lzma") are recognised and refused with ErrUnsupportedEncoding rather than decoded. The refusal is per member and happens when the data is read, so an archive with one LZMA member still opens and lists. It is untested against a real archive on purpose: macOS's xar 1.8dev reports "lzma support not compiled in" and cannot write one, and a fixture hand-built to agree with this reader would prove nothing about it.

Extended attributes, uid/gid, the timestamps and the Finder metadata are read past. An archive is read-only: every mutating method of filesystem.Filesystem returns ErrReadOnly.

Index

• Constants
• Variables
• func OpenReader(r io.ReaderAt, size int64) (filesystem.Filesystem, error)
• type FS
  • func (f *FS) Close() error
  • func (f *FS) DeleteDir(string) error
  • func (f *FS) DeleteFile(string) error
  • func (f *FS) ListDir(p string) ([]filesystem.DirEntry, error)
  • func (f *FS) MkDir(string, os.FileMode) error
  • func (f *FS) OpenFile(p string) (filesystem.File, error)
  • func (f *FS) ReadFile(p string) ([]byte, error)
  • func (f *FS) ReadLink(p string) (string, error)
  • func (f *FS) Rename(string, string) error
  • func (f *FS) Stat(p string) (filesystem.Stat, error)
  • func (f *FS) WriteFile(string, []byte, os.FileMode) error

Constants

const Magic uint32 = 0x78617221

Magic is the four-byte signature at offset 0 of every XAR archive: 'xar!'.

Variables

var (
	// ErrReadOnly is returned by every mutating method (WriteFile, MkDir,
	// DeleteFile, DeleteDir, Rename). XAR is an archive format: an existing
	// archive cannot be edited in place.
	//
	// It wraps iofs.ErrPermission so a generic caller — a WebDAV, NFS or SFTP
	// server over this filesystem — can classify it without matching on the
	// message, which is what the interface's error contract asks for.
	ErrReadOnly = fmt.Errorf("xar: archive is read-only: %w", iofs.ErrPermission)

	// ErrBadMagic is returned when the first four bytes are not 'xar!'.
	ErrBadMagic = errors.New("xar: not a xar archive")

	// ErrBadHeader is returned when the 28-byte header is present but says
	// something impossible: a header shorter than itself, a table of contents
	// that extends past the end of the archive, and so on.
	ErrBadHeader = errors.New("xar: malformed header")

	// ErrUnsupportedVersion is returned for a header version this package does
	// not know. XAR has only ever had version 1.
	ErrUnsupportedVersion = errors.New("xar: unsupported archive version")

	// ErrCorrupt is returned when the table of contents parses but describes
	// something the archive cannot contain — an offset past the end of the
	// heap, a negative length, two entries with one name in one directory, a
	// member whose data stops before its declared size.
	ErrCorrupt = errors.New("xar: corrupt archive")

	// ErrUnsupportedEncoding is returned when a member's encoding style is one
	// this package cannot decode. It is raised when the member's data is read,
	// not when the archive is opened, so one undecodable member does not make
	// the rest of the archive unreachable.
	ErrUnsupportedEncoding = errors.New("xar: unsupported entry encoding")

	// ErrNotFound is returned when a path is not in the archive. It wraps
	// iofs.ErrNotExist, which the interface module requires of every driver.
	ErrNotFound = fmt.Errorf("xar: path not found: %w", iofs.ErrNotExist)

	// ErrNotDirectory is returned when ListDir names something that is not a
	// directory.
	ErrNotDirectory = errors.New("xar: not a directory")

	// ErrNotRegular is returned when ReadFile or OpenFile names a directory, a
	// symlink, or one of the special types XAR can record (fifo, socket,
	// device node). None of them has readable contents.
	ErrNotRegular = errors.New("xar: not a regular file")

	// ErrNotSymlink is returned by ReadLink for an entry XAR did not record a
	// link target for.
	ErrNotSymlink = errors.New("xar: not a symbolic link")
)

Sentinel errors. Compare with errors.Is so wrapped errors keep matching.

Functions

func OpenReader

func OpenReader(r io.ReaderAt, size int64) (filesystem.Filesystem, error)

OpenReader reads the header and the whole table of contents of the archive in r, which must be size bytes long, and returns a read-only filesystem over it.

Member data is NOT read: only the header and the table of contents are, and each member is decoded when it is read. r must stay usable for as long as the returned filesystem is.

Types

type FS

type FS struct {
	// contains filtered or unexported fields
}

FS is an opened, read-only XAR archive.

It satisfies filesystem.Filesystem and the optional filesystem.Opener. A filesystem.File obtained from OpenFile is valid only while the FS is open.

func (*FS) Close

func (f *FS) Close() error

Close releases nothing: FS never opened the io.ReaderAt it was handed, so it does not close it either. It is here because filesystem.Filesystem has it.

func (*FS) DeleteDir

func (f *FS) DeleteDir(string) error

func (*FS) DeleteFile

func (f *FS) DeleteFile(string) error

func (*FS) ListDir

func (f *FS) ListDir(p string) ([]filesystem.DirEntry, error)

ListDir returns the entries of the directory at p, sorted by name.

func (*FS) MkDir

func (f *FS) MkDir(string, os.FileMode) error

func (*FS) OpenFile

func (f *FS) OpenFile(p string) (filesystem.File, error)

OpenFile opens the member at p for random access, implementing filesystem.Opener.

Nothing is read here: the entry's heap extent and encoding were already decoded from the table of contents, and a decoder is started on the first ReadAt. See handle for what "random access" costs on a compressed member.

func (*FS) ReadFile

func (f *FS) ReadFile(p string) ([]byte, error)

ReadFile returns the whole decoded contents of the member at p.

The buffer is GROWN from what is actually read rather than allocated from the declared <size>. A table of contents is untrusted input and its size field is not bounded by the archive's own length, so trusting it to size an allocation lets a few hundred bytes of XML ask for as much memory as it likes. Use OpenFile for anything whose size was not chosen here.

func (*FS) ReadLink

func (f *FS) ReadLink(p string) (string, error)

ReadLink returns the target of the symlink at p.

func (*FS) Rename

func (f *FS) Rename(string, string) error

func (*FS) Stat

func (f *FS) Stat(p string) (filesystem.Stat, error)

Stat reports what the table of contents recorded for p.

func (*FS) WriteFile

func (f *FS) WriteFile(string, []byte, os.FileMode) error