cpio

README

cpio

Reads the records of a cpio archive, in all four variants, and stops there — pure Go, CGO_ENABLED=0, builds for every target Go builds for.

recs, err := cpio.Records(ra, size)

variant	magic	fields	padding
newc (SVR4, the initramfs one)	070701	8 hex digits	name and data to four
crc	070702	8 hex digits	as newc; the checksum is not verified
odc (POSIX.1)	070707	6 octal digits	none
old binary	0o070707 as a 16-bit word	16-bit words	name and data to an even length

Why records and not a filesystem

Two packages needed this parser and they need different filesystems over it: go-filesystems/unarchive indexes into an io.ReaderAt it keeps open, and go-filesystems/rpm has the whole payload decompressed in memory. Sharing the filesystem would have meant one of them accepting the other's shape. Sharing the parser costs neither anything.

And the parser is where the defects live. Record carries the name, the mode, the size, the offset of the data in the reader, a symbolic link's target and the mtime — everything the header said, and nothing invented.

The traps, each of them measured

• newc pads the name AND the data to four bytes, from the start of the archive. Forgetting either puts every later header one to three bytes out.
• odc pads nothing at all. A reader that applies newc's rule to it reads garbage from the second record onward.
• The old binary variant's magic is 0o070707 as a 16-bit word, so the same two bytes mean it on the machine that wrote them and 0o143561 on one of the other endianness. Both exist, and the order is detected, never assumed.
• That variant's 32-bit fields are two words, high word first, whatever the byte order of each word — and it pads to an even length, not to four.
• A symbolic link's target is its DATA, not a header field, so it is read here: a Record carries the target rather than an offset to it.
• A reader that computes the next offset rather than being handed it returned zero once, and the walk re-read record one for ever. The suite reported a ten-minute timeout, not a failure.

The inode is read, and is not portable across the variants

Record.Inode is the number the header records, because a consumer reads it — go-filesystems/rpm reports it in Stat — and a parser that dropped it would force that consumer to invent one.

⛔ It does not mean the same thing in every variant. odc's field is six octal digits and the old binary variant's is a 16-bit word, so neither can hold a real inode: cpio(1) renumbers them 1..n when it writes those. Measured on one tree — 257757054 in newc, 1 in both of the others. A caller using it as an identity across two archives of the same files will be wrong.

It is also not unique within one archive: a cpio records hard links by repeating an inode.

Record.Name is the name as recorded

Not a clean path. GNU cpio writes ./real.txt where other writers write real.txt, and an archive made on Windows can carry backslashes. Normalising is left to the caller, because both callers already do it — for lookups as well as for listing — and a parser that silently rewrote names would make their two spellings disagree with this one.

FileMode is a translation, not a cast

os.ModeDir is 1 << 31; POSIX's S_IFDIR is 0o040000. Narrowing one to the other drops every type bit. FileMode maps the seven S_IF* kinds, and setuid, setgid and sticky — an archive records those, and a setuid binary read back as an ordinary one is a mode meaning something other than what was packed.

Two functions, because the two consumers are each right

recs, err := cpio.Records(ra, size)       // an initramfs: block-padded, tolerant
recs, err := cpio.RecordsExact(ra, size)  // an RPM payload: exact length, strict

	Records	RecordsExact
zero padding after the trailer	accepted	accepted
non-zero bytes after the trailer	accepted, stops there	refused
junk where a header should be	accepted, stops there	refused
no trailer at all	accepted	refused

An initramfs is padded to a block, so refusing the zeros after its trailer would refuse every one there is. An RPM payload has a length known exactly — from the end of the header to the end of the file — so anything unparsed in it is damage, and a reader that stops early and reports success hands back a package with files missing.

⛔ RecordsExact first required the archive to end exactly at the trailer, which refuses everything cpio(1) writes: one clean fixture here has its trailer ending at 756 inside a 1024-byte file. The rule is that the tail must be zeros, which keeps the padding and refuses the junk.

Every numeric field is checked

Not the three the parser happens to use. A header field that is not a number means the header is not a header, and reading it as zero gives a record that looks ordinary and is not — an mtime of 1970 and an inode of 0 are both perfectly representable.

⛔ The two readers this package replaced disagreed about that: rpm checked all of them, unarchive ignored the errors on inode and mtime. The strict one is right, and it was the one tested against real packages. Consolidating on the lenient behaviour would have been a regression that no test in either repository could see.

Errors

ErrNotCpio	the bytes begin with no cpio magic — try another reader
ErrTruncated	a header, a name or a record's data runs past the end — this archive is damaged

They say different things and a caller acts differently on them, so six bytes of newc magic and nothing else is truncated, not unrecognised.

A read failure from underneath is returned as itself, wrapped in neither: a failing disk must not read as a damaged file.

Fixtures

testdata/gen.sh rebuilds them, and asserts its own premise — cpio must read each one back before it is kept. bin-swapped.cpio is crafted by testdata/gen_swapped.py, because no tool here writes the other byte order, and the test asserts the two binary fixtures really carry different magic words.

The four are read by TestEveryVariantReadsTheSameArchive, which is the assertion no single-variant test can make: four header formats over one tree must agree about every record, compared entry by entry.

Licence

BSD-3-Clause.

Documentation

Overview

Package cpio reads the records of a cpio archive, in all four variants, and stops there.

It returns records rather than a filesystem, and that is the whole design. Two packages needed this parser -- go-filesystems/unarchive for an initramfs and a .deb's data member, go-filesystems/rpm for a payload -- and they need DIFFERENT filesystems over it: unarchive indexes into an io.ReaderAt it keeps open, rpm has the whole payload decompressed in memory. Sharing the filesystem would have meant one of them accepting the other's shape; sharing the parser costs neither anything.

And the parser is where the defects live. Every trap below was measured, and each one had been got wrong at least once:

• newc pads the name AND the data to four bytes, from the start of the archive. Forgetting either puts every later header one to three bytes out.
• odc pads nothing at all.
• the old binary variant's magic is 0o070707 as a 16-bit WORD, so the same two bytes mean it on the machine that wrote them and 0o143561 on one of the other endianness. Both orders exist and the order is DETECTED, not assumed.
• that variant's 32-bit fields are two words, HIGH WORD FIRST, whatever the byte order of each word -- and it pads to an even length, not to four.
• a reader that returns the next offset as a computed value rather than a parameter returned zero once, and the walk re-read record one for ever. The suite reported a ten-minute timeout, not a failure.

CGO is not used, so this builds for every target Go builds for.

Index

• Constants
• Variables
• func FileMode(m int64) iofs.FileMode
• type Record
  • func Records(ra io.ReaderAt, size int64) ([]Record, error)
  • func RecordsExact(ra io.ReaderAt, size int64) ([]Record, error)

Constants

const (
	MagicNewc = magicNewc
	MagicCRC  = magicCRC
	MagicODC  = magicODC
	// TrailerName is the name of the record that ends an archive. It is the NAME,
	// not a magic: the trailer is an ordinary header whose name says stop.
	TrailerName = trailerName
)

binaryMagic is 0o070707 as a 16-bit WORD, which is what makes the old binary format byte-order dependent: the same two bytes are 0o070707 on the machine that wrote them and 0o143561 on one of the other endianness.

So the order is DETECTED from the magic rather than assumed, and both are read. Guessing silently misreads every field of an archive written on the other kind of machine. The magics and the trailer name, exported because a WRITER needs them and they are facts about the format rather than about this reader.

go-filesystems/unarchive writes newc, and had its own copies of these three until the parser moved here -- which is the duplication this package exists to end, one string constant included.

Variables

var ErrNotCpio = errors.New("cpio: not a cpio archive")

⛔ EVERY numeric field is checked, not the three the parser happens to use. A header field that is not a number means the header is not a header, and reading it as zero gives a record that looks ordinary and is not -- an mtime of 1970 and an inode of 0 are both perfectly representable.

Two readers disagreed about this before they were merged: go-filesystems/rpm checked all of them, go-filesystems/unarchive ignored the errors on inode and mtime. The strict one is right, and it was the one tested against real packages.

ErrNotCpio is returned when the bytes do not begin with any cpio magic.

var ErrTruncated = errors.New("cpio: the archive ends inside a record")

ErrTruncated is returned when a header or a name runs past the end.

Functions

func FileMode

func FileMode(m int64) iofs.FileMode

FileMode turns a POSIX st_mode into an iofs.FileMode.

⛔ The type is in the HIGH bits of the octal value, not in iofs.FileMode's own places: S_IFDIR is 0o040000 while iofs.ModeDir is 1<<31. Narrowing one into the other without this switch leaves every directory looking like a file with peculiar permissions.

Types

type Record

type Record struct {
	// Name is the name AS RECORDED, which is not the same as a clean path: GNU
	// cpio writes "./real.txt" where other writers write "real.txt", and an
	// archive made on Windows can carry backslashes. Normalising is left to the
	// caller because both callers already do it, for lookups as well as for
	// listing, and a parser that silently rewrote names would make their two
	// spellings disagree with this one.
	Name string
	// Mode carries the permissions AND the type bits, already translated from
	// POSIX st_mode -- see FileMode for why that translation is not a cast.
	Mode iofs.FileMode
	Size int64
	// Offset is where this record's data begins in the reader.
	Offset int64
	// Link is the target of a symbolic link, read from the record's data. A cpio
	// stores it there rather than in the header, which is why it is read here and
	// not left to the caller.
	Link  string
	MTime time.Time
	// Inode is the number the header records. It is kept because a consumer reads
	// it -- go-filesystems/rpm reports it in Stat -- and because a parser that
	// dropped it would force that consumer to invent one, which is a different
	// number wearing the same name.
	//
	// It is NOT unique in general: a cpio records hard links by repeating an inode,
	// and an archive concatenated from two others can repeat one by accident.
	//
	// ⛔ And it is NOT portable across the variants. odc's field is six octal digits
	// and the old binary variant's is a 16-bit word, so neither can hold a real
	// inode -- cpio(1) RENUMBERS them 1..n when it writes those. Measured: one tree
	// came out as 257757054 in newc and as 1 in both of the others. A caller using
	// it as an identity across two archives of the same files will be wrong.
	Inode uint64
}

Record is one entry, with its data left where it lies.

Offset points into the reader Records was given, so a caller can hand out a section of it rather than copying anything.

func Records

func Records(ra io.ReaderAt, size int64) ([]Record, error)

Records reads every record in the archive.

The variant is decided per record from its own magic, because an archive concatenated from two others is a thing that exists. Trailing padding after the trailer is normal -- an initramfs is padded to a block -- so what follows the trailer is ignored rather than refused.

func RecordsExact

func RecordsExact(ra io.ReaderAt, size int64) ([]Record, error)

RecordsExact is Records for an archive whose length is known to be exact, and refuses what Records tolerates.

⛔ The difference is not strictness for its own sake: the two consumers need different answers and each is right about its own case.

• An RPM payload is a cpio of a known length, from the end of the header to the end of the file. Anything in it that does not parse is damage, and a reader that stops early and reports success hands back a package with files missing.
• An initramfs is padded to a block. The bytes after the trailer are zeros nobody wrote, and refusing them would refuse every initramfs there is.

So this requires a TRAILER record, and requires the archive to end there. Records requires neither.