cpio

README

cpio

Reads the records of a cpio archive, in all four variants, and stops there — pure Go, CGO_ENABLED=0, builds for every target Go builds for.

recs, err := cpio.Records(ra, size)

variant	magic	fields	padding
newc (SVR4, the initramfs one)	070701	8 hex digits	name and data to four
crc	070702	8 hex digits	as newc; the checksum is not verified
odc (POSIX.1)	070707	6 octal digits	none
old binary	0o070707 as a 16-bit word	16-bit words	name and data to an even length

Why records and not a filesystem

Two packages needed this parser and they need different filesystems over it: go-filesystems/unarchive indexes into an io.ReaderAt it keeps open, and go-filesystems/rpm has the whole payload decompressed in memory. Sharing the filesystem would have meant one of them accepting the other's shape. Sharing the parser costs neither anything.

And the parser is where the defects live. Record carries the name, the mode, the size, the offset of the data in the reader, a symbolic link's target and the mtime — everything the header said, and nothing invented.

The traps, each of them measured

• newc pads the name AND the data to four bytes, from the start of the archive. Forgetting either puts every later header one to three bytes out.
• odc pads nothing at all. A reader that applies newc's rule to it reads garbage from the second record onward.
• The old binary variant's magic is 0o070707 as a 16-bit word, so the same two bytes mean it on the machine that wrote them and 0o143561 on one of the other endianness. Both exist, and the order is detected, never assumed.
• That variant's 32-bit fields are two words, high word first, whatever the byte order of each word — and it pads to an even length, not to four.
• A symbolic link's target is its DATA, not a header field, so it is read here: a Record carries the target rather than an offset to it.
• A reader that computes the next offset rather than being handed it returned zero once, and the walk re-read record one for ever. The suite reported a ten-minute timeout, not a failure.

Record.Name is the name as recorded

Not a clean path. GNU cpio writes ./real.txt where other writers write real.txt, and an archive made on Windows can carry backslashes. Normalising is left to the caller, because both callers already do it — for lookups as well as for listing — and a parser that silently rewrote names would make their two spellings disagree with this one.

FileMode is a translation, not a cast

os.ModeDir is 1 << 31; POSIX's S_IFDIR is 0o040000. Narrowing one to the other drops every type bit. FileMode maps the seven S_IF* kinds, and setuid, setgid and sticky — an archive records those, and a setuid binary read back as an ordinary one is a mode meaning something other than what was packed.

Errors

ErrNotCpio	the bytes begin with no cpio magic — try another reader
ErrTruncated	a header, a name or a record's data runs past the end — this archive is damaged

They say different things and a caller acts differently on them, so six bytes of newc magic and nothing else is truncated, not unrecognised.

A read failure from underneath is returned as itself, wrapped in neither: a failing disk must not read as a damaged file.

Fixtures

testdata/gen.sh rebuilds them, and asserts its own premise — cpio must read each one back before it is kept. bin-swapped.cpio is crafted by testdata/gen_swapped.py, because no tool here writes the other byte order, and the test asserts the two binary fixtures really carry different magic words.

The four are read by TestEveryVariantReadsTheSameArchive, which is the assertion no single-variant test can make: four header formats over one tree must agree about every record, compared entry by entry.

Licence

BSD-3-Clause.

Documentation

Overview

Package cpio reads the records of a cpio archive, in all four variants, and stops there.

It returns records rather than a filesystem, and that is the whole design. Two packages needed this parser -- go-filesystems/unarchive for an initramfs and a .deb's data member, go-filesystems/rpm for a payload -- and they need DIFFERENT filesystems over it: unarchive indexes into an io.ReaderAt it keeps open, rpm has the whole payload decompressed in memory. Sharing the filesystem would have meant one of them accepting the other's shape; sharing the parser costs neither anything.

And the parser is where the defects live. Every trap below was measured, and each one had been got wrong at least once:

• newc pads the name AND the data to four bytes, from the start of the archive. Forgetting either puts every later header one to three bytes out.
• odc pads nothing at all.
• the old binary variant's magic is 0o070707 as a 16-bit WORD, so the same two bytes mean it on the machine that wrote them and 0o143561 on one of the other endianness. Both orders exist and the order is DETECTED, not assumed.
• that variant's 32-bit fields are two words, HIGH WORD FIRST, whatever the byte order of each word -- and it pads to an even length, not to four.
• a reader that returns the next offset as a computed value rather than a parameter returned zero once, and the walk re-read record one for ever. The suite reported a ten-minute timeout, not a failure.

CGO is not used, so this builds for every target Go builds for.

Index

• Constants
• Variables
• func FileMode(m int64) iofs.FileMode
• type Record
  • func Records(ra io.ReaderAt, size int64) ([]Record, error)

Constants

const (
	MagicNewc = magicNewc
	MagicCRC  = magicCRC
	MagicODC  = magicODC
	// TrailerName is the name of the record that ends an archive. It is the NAME,
	// not a magic: the trailer is an ordinary header whose name says stop.
	TrailerName = trailerName
)

binaryMagic is 0o070707 as a 16-bit WORD, which is what makes the old binary format byte-order dependent: the same two bytes are 0o070707 on the machine that wrote them and 0o143561 on one of the other endianness.

So the order is DETECTED from the magic rather than assumed, and both are read. Guessing silently misreads every field of an archive written on the other kind of machine. The magics and the trailer name, exported because a WRITER needs them and they are facts about the format rather than about this reader.

go-filesystems/unarchive writes newc, and had its own copies of these three until the parser moved here -- which is the duplication this package exists to end, one string constant included.

Variables

var ErrNotCpio = errors.New("cpio: not a cpio archive")

ErrNotCpio is returned when the bytes do not begin with any cpio magic.

var ErrTruncated = errors.New("cpio: the archive ends inside a record")

ErrTruncated is returned when a header or a name runs past the end.

Functions

func FileMode

func FileMode(m int64) iofs.FileMode

FileMode turns a POSIX st_mode into an iofs.FileMode.

⛔ The type is in the HIGH bits of the octal value, not in iofs.FileMode's own places: S_IFDIR is 0o040000 while iofs.ModeDir is 1<<31. Narrowing one into the other without this switch leaves every directory looking like a file with peculiar permissions.

Types

type Record

type Record struct {
	// Name is the name AS RECORDED, which is not the same as a clean path: GNU
	// cpio writes "./real.txt" where other writers write "real.txt", and an
	// archive made on Windows can carry backslashes. Normalising is left to the
	// caller because both callers already do it, for lookups as well as for
	// listing, and a parser that silently rewrote names would make their two
	// spellings disagree with this one.
	Name string
	// Mode carries the permissions AND the type bits, already translated from
	// POSIX st_mode -- see FileMode for why that translation is not a cast.
	Mode iofs.FileMode
	Size int64
	// Offset is where this record's data begins in the reader.
	Offset int64
	// Link is the target of a symbolic link, read from the record's data. A cpio
	// stores it there rather than in the header, which is why it is read here and
	// not left to the caller.
	Link  string
	MTime time.Time
}

Record is one entry, with its data left where it lies.

Offset points into the reader Records was given, so a caller can hand out a section of it rather than copying anything.

func Records

func Records(ra io.ReaderAt, size int64) ([]Record, error)

Records reads every record in the archive.

The variant is decided per record from its own magic, because an archive concatenated from two others is a thing that exists. Trailing padding after the trailer is normal -- an initramfs is padded to a block -- so what follows the trailer is ignored rather than refused.