unarchive

README

unarchive

Open an archive without being told what it is, and extract it while checking that what came out is what the archive promised.

Pure Go, CGO_ENABLED=0, and it builds for every target Go builds for.

go install github.com/go-filesystems/unarchive/cmd/unarchive@latest

unarchive film.part1.rar          # into ./film/
unarchive -C /tmp/out backup.tar.zst
unarchive -f already-there.zip    # overwrite
unarchive -n suspicious.7z        # list, extract nothing

unarchive -o film.7z film.rar     # convert, extracting nothing
unarchive -o backup.tar.zst backup.zip
unarchive -o disc.7z disc.iso     # yes, that works

Formats

	read	notes
RAR 1.5–4.x, RAR5	✅ read, ✅ write stored	multi-volume sets followed by volume number; a .rar written here carries no compression at all — see below
ZIP	✅	including the jar/epub/odf family, and entries compressed with bzip2, LZMA, xz or zstd — not just deflate
7z	✅	follows its own .001 chain, by name
tar	✅	v7, USTAR, PAX and GNU alike, with real random access, symbolic links included
ar	✅	static libraries and .deb — both long-name spellings, SysV and BSD
cpio	✅	newc, crc, odc and the old binary variant, both byte orders — parsed by go-filesystems/cpio
gzip, bzip2, xz, zstd, lz4	✅	stream wrappers: .tar.gz, .tgz, .tar.zst, a lone notes.txt.gz …
lzop .lzo	✅	likewise, through go-compressions/lzo — LZO1X, no compressor
lzip .lz	✅	likewise, through go-compressions/lzip — LZMA in lzip's framing, not xz's
compress .Z	read only	the LZW compress/lzw cannot read — see below
xar	✅	the macOS .pkg container, through go-filesystems/xar
CAB	✅	Microsoft Cabinet, None and MSZIP, through go-filesystems/cab
RPM	✅	headers and payload, gzip/xz/zstd/bzip2, through go-filesystems/rpm
WARC	✅	a web crawl, one record per entry, through go-filesystems/warc
ISO 9660	✅	a disc image is an archive too — read through go-filesystems/iso9660
SquashFS	✅	likewise, through go-filesystems/squashfs
HFS+	✅	what a .dmg held before APFS — through go-filesystems/hfsplus
DMG	✅	a UDIF container around one of the above: peeled, then sniffed
plakar .ptar	recognised, not unpacked	and the reason is below

Writing a .rar gives you an uncompressed one

RAR's compression is proprietary and unpublished, and the UnRAR licence says the source "cannot be used to develop RAR (WinRAR) compatible archiver and to re-create RAR compression algorithm". So there is nothing here to finish later: go-filesystems/rar writes the stored method, which the published RAR 5.0 container note is enough for, and nothing else.

That is said out loud rather than left to be noticed — unarchive -o out.rar prints it before writing a byte. Entries still carry a CRC-32 of their data, so a damaged one is detectable; a stored entry with no checksum would be an archive with nothing in it to verify.

Four formats read from the handle, and closed unlike an image

xar, CAB, RPM and WARC are opened through an io.ReaderAt exactly the way an image is, and they differ from one in the one respect that matters: each is

func (f *FS) Close() error { return nil }

while iso9660 and squashfs forward Close to the reader they were given. Wrapping those closes the file twice and says so, late; not wrapping these leaks it, once per open, and says nothing ever. Six drivers, one signature, two ownership rules — so each one's Close was read rather than assumed, and the wrapping is watched by a test with an injected closer rather than believed.

A numbered split — film.zip.001, .002, … — is read as one file, whatever the format inside it, so unarchive film.zip.001 works and extracts into film/. RAR is the exception: its own volume sets are followed by number already, and a plain numbered split of a RAR is a different thing this has no entry point for.

Written

.tar, .tar.gz, .tgz, .tar.xz, .txz, .tar.zst, .tzst, .tar.lz4, .tar.bz2, .tbz2, .tbz, .zip, .jar, .7z, .a, .deb, .cpio, .iso.

ar is written in the BSD convention, not SysV — measured, not chosen: written SysV, ar t on macOS listed every member with its slash attached (notes.txt/) and ar p archive notes.txt then found nothing. BSD also needs no string table, which is why it streams: SysV puts long names in a // member that must precede every entry.

cpio is written as newc, whose eight hex digits reach 4 GiB, rather than the octal odc whose six do not reach 8 MiB.

⚠ An .iso holds every entry in memory while it is built. That is upstream's shape rather than a choice here: iso9660.Builder.AddFile takes a []byte. And ISO 9660 without Rock Ridge or Joliet upper-cases names and cannot carry a long one — the builder writes neither extension.

Not written: RAR, because no free writer exists, and .Z, because a new .Z is a file nobody should be making — gzip, xz and zstd all compress better and are read everywhere. Both say which and why:

$ unarchive -o out.tar.Z film.tar.Z
unarchive: out.tar.Z: unarchive: this format is read but not written: the LZW of
compress(1). A new .Z is a file nobody should be making: gzip, xz and zstd all
compress better and are read everywhere

.Z is read through go-compressions/compress, written for this: compress/lzw stops at 12 bits, has no clear code, and knows nothing of the group alignment, so it cannot read a .Z at all.

bzip2 used to be on it, for a different kind of reason — the standard library decompresses bzip2 and nothing compressed it. A missing library is not a property of a format, so the library was written (go-compressions/bzip2) and the sentence stopped being true. The two reasons look identical from the outside and only one of them is permanent.

brotli is absent on purpose. It has no signature: a brotli stream begins with the first bits of its own data, so there is nothing to recognise it by. The only way to open one is to be told, by an extension or a flag, and this package decides from the bytes.

plakar .ptar: recognised, and deliberately not unpacked

A .ptar is not an archive. It is a Kloset repository in a file — a config blob, a packfile region and a state region, addressed by MAC — holding snapshots and deduplicated chunks, and encrypted unless it was made with -plaintext.

Reading one means implementing their storage container, their state index, their packfile format, their snapshot model, and decryption with a key the caller must supply. That is a backup tool, not an unarchiver, and "which snapshot?" is a question this command has nowhere to ask. So it is recognised and says what it is:

$ unarchive backup.ptar
unarchive: backup.ptar: unarchive: recognised, but this format is not read yet:
ptar is a plakar Kloset archive: a content-addressed repository of snapshots and
deduplicated chunks, encrypted unless it was made with -plaintext. Open it with
plakar, which has the key

Recognition earns its keep on its own: without it, a .ptar answered "the bytes match no format this knows", which sends somebody hunting for a corrupt file.

The magic is _PLATAR_, read out of PlakarKorp's own storage driver. Guessing it from the extension — the thing this package exists not to do — would have got it wrong.

An image is an archive too

A filesystem image is one file holding a tree, handed around to be unpacked — which is what an archive is, from the outside. So .iso and .squashfs open, list, extract and convert like any other input, and nothing here decodes them: go-filesystems already owns those drivers, and the detection is go-filesystems/detect's hardened prober rather than a second magic table written here.

The line is drawn at formats people distribute. You download an .iso, you ship a .squashfs, somebody sends you a .dmg; you do not hand somebody an ext4 image expecting them to unpack it. The org has drivers for a dozen more filesystems and this opens three, on purpose.

A .dmg is not one of them — it is a container around one

Its sectors are compressed and may be stored out of order, so there is nothing for a driver to read in place. It is peeled like a stream wrapper: the sectors are decoded to a spool through go-diskimages/dmg, and the spool is sniffed, because a .dmg holds HFS+, or APFS, or an ISO 9660, and the container does not record which.

Two consequences worth stating rather than discovering:

• An unrecognised spool is an error, not a one-entry directory. A .gz legitimately holds one ordinary file and openCompressed hands that back as such; a UDIF image never does — its sectors are a filesystem or they are nothing. APFS is the case this matters for: detect names it and no driver here opens it, and that sentence is more useful than a megabyte of raw sectors presented as an archive of one file.
• A partition map is not handled. An image carrying one holds a table where a filesystem's magic would be, so the sniff finds nothing and says so, naming the .dmg it came out of. Splitting that apart is a volume manager's job, and go-volumes has one.

The trailer is what names a UDIF, and it has to be: a .dmg whose sectors are stored raw begins with the filesystem inside it, so a prober reading only the head would recognise that filesystem and read straight past the container. That is the wrong answer even when it works — the same image written sparse or compressed would then be unreadable.

Symbolic links

A link comes out a link, with the target the archive recorded. Every one of them came out as an empty regular file until v0.10.0 — Extract asked only whether an entry was a directory, so a link fell through to the file path and wrote the zero bytes a link's body contains. No error, and it was counted in Files. ReadLink had the target the whole time.

Result counts them separately now, and that separation is deliberate: a count that lumped them in with files would have passed just as well against the old behaviour.

The target is written as recorded, including an absolute one

Creating a link writes nothing outside the destination — a target is a string in an inode. What would write outside is a later entry going through the link, and that is closed separately and unconditionally:

a directory where a link already sits	refused, ErrEscapes. MkdirAll walks through a link to a directory
a file where a link already sits	the link is removed first, or refused. O_TRUNC follows a link, so overwriting one writes to whatever it points at
an entry listed beneath a link	refused at Open, ErrNestedUnderNonDirectory

Those three are the guarantee. Refusing the target would add nothing to them, and it would make an RPM or a .deb unextractable — absolute links are ordinary in both. Rewriting the target instead, as bsdtar does by stripping a leading separator, produces a tree that means something other than what was packed.

So a caller walking the extracted result is in the same position as one walking what tar wrote. That is said here rather than left to be found out.

Why the nesting case is refused when the archive is opened

Nothing was ever written outside for it: Extract descends only into directories, so an entry under a link was never reached. What it was is silent — Files 0, no error, and a file the archive held simply gone. By extraction time there is nothing left to report it with, so the archive is refused up front, naming both the link and the entry under it. go-filesystems/xar refuses the same shape in its own reader.

Why the bytes and not the name

A format is recognised by what is in the file. An extension is a claim by whoever last renamed it: a .rar that is really a zip, a video saved as .mp4 that is an MPEG transport stream, a volume renamed by a download manager — all ordinary, and all wrong if the name decides.

The one thing a name is used for is what to call the single file inside a wrapper: somebody who gunzips notes.txt.gz expects notes.txt, and nothing in the bytes says so.

Why the check

An extractor that reports success is not the same as an extractor that got everything.

libarchive — which is bsdtar, and therefore macOS — does not follow a multi-volume RAR chain. It writes the first volume, pads the remainder with zeros to the declared length, and exits 0. The result is exactly the right size, which is what almost every check afterwards looks at. A 1.26 GiB file came out of it here, verified by size, believed, and the source archives were deleted on the strength of it.

So this package compares what was written against what the header declared, per entry, and says so when they differ (ErrIncomplete).

Three kinds of no

	means
ErrUnknownFormat	I do not know what this is
ErrNotImplemented	I know exactly what this is and do not read it yet
anything else	I read this format and this file is broken

They send a person to three different places, which is why they are three errors. Nothing is in the middle bucket today; a count in the tests fails if a format is ever added to the sniffer alone, so that stays a decision rather than a default.

Changing an archive without rewriting it

An archive can be opened as a filesystem you may write to, where the changes cost nothing until you say so:

o, format, err := unarchive.Writable("backup.tar.zst")
if err != nil { return err }
defer o.Close()                  // discards the changes; writes nothing back

o.WriteFile("etc/hosts", newHosts, 0o644)
o.MkDir("etc/extra", 0o755)
o.DeleteFile("var/log/old.log")

err = unarchive.SealTo(o, "backup.7z")   // ONE rewrite, here

Renaming one entry in a 4 GiB archive costs a rename, not 4 GiB of rewriting. The archive underneath is untouched until SealTo, so abandoning the changes is free and an interrupted session leaves the original exactly as it was. SealTo builds the new archive beside the target and renames over it, so an interruption there leaves whatever the target held before, rather than half of something new.

The target's format comes from its name. That is the one place in this package where a name decides anything, and it has to: an output file has no bytes yet to read.

unarchive -o is this, with no changes: a conversion.

As a filesystem

Open returns a filesystem.Filesystem from go-filesystems/interface, so an archive can be read like a mounted volume — ReadFile, ListDir, Stat, and OpenFile for a seekable handle, without extracting anything.

fsys, format, err := unarchive.Open("backup.tar.xz")
if err != nil { return err }
defer fsys.Close()          // removes the spool, for a compressed wrapper

b, err := fsys.ReadFile("etc/hosts")

A stream wrapper is spooled to a temporary file first, and that is the format's doing rather than a shortcut: every archive here needs random access — a tar index, a zip central directory, a 7z header at the end — and a decompressor gives a one-way stream. Reading it twice means decompressing it twice; holding it in memory means holding a whole archive in memory. Close removes the spool.

Nesting is capped: a wrapper holds one stream and that stream may be another wrapper, so a few hundred bytes of .gz nested on itself would otherwise ask for however much spool the machine has.

Untrusted input

An archive is somebody else's bytes, and every size in it is a claim. These limits apply by default; none of them is on the hot path (they are checked per header or per megabyte, never per byte).

what the archive asks for	limit	default	on excess
bytes Extract writes	Options.MaxBytes	64 GiB	ErrTooLarge; an entry declaring more than is left is refused before its file is created
entries Extract creates	Options.MaxEntries	4,194,304	ErrTooLarge
depth Extract recurses to	Options.MaxDepth	1024	ErrTooLarge (a Go stack that runs out is fatal, not recoverable)
paths an index holds (tar, ar, cpio)	DefaultLimits.MaxEntries	1,048,576, directories included	ErrTooLarge
one path	fixed	4096 bytes	ErrTooLarge
all paths of one index	fixed	256 MiB	ErrTooLarge
what ReadFile returns	DefaultLimits.MaxReadFile	1 GiB	ErrTooLarge, before reserving; Extract and OpenFile stream
what a .gz, .xz, .zst ... wrapper may spool to disk	DefaultLimits.MaxSpool	64 GiB	ErrTooLarge; the spool is removed
an LZMA2 / LZMA dictionary (.xz wrappers and zip methods 14 and 95)	DefaultLimits.MaxDictionary	256 MiB	ErrTooLarge (wrapping the decoder's), before the window is allocated; the window grows with the data decoded, so a header that merely declares more is not refused, only data that fills more
a zstd window	DefaultLimits.MaxDictionary	256 MiB	the decoder's own error, before the memory is reserved

For Options, zero is the default and a negative value removes the limit. DefaultLimits is process-wide: set it once at start-up, before opening anything. Open has no per-call limits yet.

Measured before, each with a hand-built archive: a 218-byte zip declaring 4 GiB made ReadFile allocate 4,294,975,728 bytes (io/fs.ReadFile reserves what Stat reports); a 9-byte LZMA header in a zip entry reserved a 1 GiB dictionary (ulikunitz/xz's DictCap bounded the .lzma header, but not the xz path, which allocated whatever dictionary a Block Header declared: a 44-byte .xz declaring 1.5 GiB allocated 1,610,663,976 bytes; .tar.xz and zip method 95 took no limit at all). The xz, LZMA and zip LZMA decoders are go-compressions/xz now, and the same planted bombs (1.5 GiB and 4 GiB - 1 declared over a few bytes) allocate about 36 KB and 68 KB; ulikunitz/xz remains for the writers only; a 1.4 MB ar archive with 20,000 long-named members allocated 2.5 GB (the string table was copied once per member); a 400 KB ar holding one member with a 400,001 byte BSD name ran past 4 min 40 s synthesising 200,000 parent directories; 1000 tar entries 1000 directories deep (3 MB) allocated 435 MB. Indexing is linear now: a path is linked to its parents once, and the walk up stops at the first ancestor already known.

Not covered here: the cpio, rpm, xar, rar, cab, iso9660, squashfs, hfsplus and dmg parsers live in their own modules and are audited there; this module raises its rar and xar requirements to the audited releases.

Licence

BSD-3-Clause.

Documentation

Overview

Package unarchive opens an archive without being told what it is, and extracts it while checking that what came out is what the archive promised.

Why the bytes and not the name

A format is recognised by what is IN the file. An extension is a claim by whoever last renamed it: a .rar that is really a zip, a video saved as .mp4 that is an MPEG transport stream, a volume renamed by a download manager -- all ordinary, and all wrong if the name decides.

Why the check

An extractor that reports success is not the same as an extractor that got everything. libarchive, which is bsdtar and therefore macOS, does not follow a multi-volume RAR chain: it writes the first volume, pads the remainder with ZEROS to the declared length and exits 0. The result is exactly the right SIZE, which is what every check afterwards looks at. This package compares what was written against what the header declared, per entry, and says so when they differ.

Index

• Constants
• Variables
• func FromFS(fsys iofs.FS) filesystem.Filesystem
• func FromFSWithModes(fsys iofs.FS, modes map[string]os.FileMode) filesystem.Filesystem
• func InnerName(name string) string
• func SealTo(o *overlay.Overlay, target string) error
• func WithoutPartNumber(name string) string
• type Entry
  • func List(fsys filesystem.Filesystem) ([]Entry, error)
• type Format
  • func Open(path string) (filesystem.Filesystem, Format, error)
  • func Sniff(r io.ReaderAt, size int64) (Format, error)
  • func Writable(path string) (*overlay.Overlay, Format, error)
  • func (f Format) Compressed() bool
  • func (f Format) Image() bool
  • func (f Format) Note() string
  • func (f Format) ReaderArchive() bool
  • func (f Format) String() string
• type Limits
• type Options
• type Result
  • func Extract(fsys filesystem.Filesystem, dest string, opt Options) (*Result, error)
• type WriteTarget
  • func TargetFor(name string) (WriteTarget, error)
  • func (t WriteTarget) String() string

Constants

const (
	// DefaultMaxBytes is 64 GiB.
	DefaultMaxBytes = 64 << 30
	// DefaultMaxEntries is 4,194,304.
	DefaultMaxEntries = 1 << 22
	// DefaultMaxDepth is 1024 levels, a quarter of Linux's PATH_MAX at the
	// shortest name an entry can have.
	DefaultMaxDepth = 1024
)

Defaults for Options.

Variables

var DefaultLimits = Limits{
	MaxEntries:    1 << 20,
	MaxReadFile:   1 << 30,
	MaxSpool:      64 << 30,
	MaxDictionary: 256 << 20,
}

DefaultLimits is what Open and the readers behind it apply. It is process wide: set its fields once at start-up, before opening anything; it is read, never written, by this package.

The defaults sit above anything an honest archive needs and below what a few dozen hostile bytes can ask for: a 9-byte LZMA header asked for a 1 GiB dictionary and a 218-byte zip for a 4 GiB ReadFile.

var ErrCannotWrite = errors.New("unarchive: this format is read but not written")

ErrCannotWrite is returned for a format this package reads and does not write.

A third sentence alongside the other two: "I know what this is, I can read it, and I cannot produce one". Only RAR is left in it, and for a reason that is not going to change here: no free writer exists.

bzip2 used to be in it too, because the standard library decompresses bzip2 and nothing compressed it. That was a missing LIBRARY rather than a property of the format, so the library was written -- go-compressions/bzip2 -- and the sentence stopped being true. Worth saying, because the two reasons look identical from the outside and only one of them is permanent.

var ErrEscapes = errors.New("unarchive: entry would be written outside the destination")

ErrEscapes is returned for an entry whose path would land outside the destination.

An archive is somebody else's list of names, and "../../.ssh/authorized_keys" is a name. Every extractor that trusted it has had the same bug, so this one does not trust it: the resolved path is checked against the destination, and an entry that leaves it is refused rather than sanitised, because silently rewriting a path puts the file somewhere the person did not ask for either.

var ErrExists = errors.New("unarchive: file exists")

ErrExists is returned when a file is already there and Overwrite is not set.

var ErrIncomplete = errors.New("unarchive: entry is shorter than its header declares")

ErrIncomplete is returned when an entry yielded fewer bytes than its header declared.

It is the point of this package. An extractor that writes what it got and reports success leaves a file of exactly the right SIZE when the source was zero-padded, and every check afterwards agrees with it.

var ErrMissingPart = errors.New("unarchive: a part of this split is missing")

ErrMissingPart is returned when a numbered set has a hole in it.

A hole is not the same as the end. Concatenating parts 1, 2 and 4 produces a file of a plausible SIZE whose middle is wrong, and every reader then reports corruption -- of the archive, which is intact. Refusing here names the part that is absent.

var ErrNestedUnderNonDirectory = errors.New("unarchive: an entry is nested under something that is not a directory")

ErrNestedUnderNonDirectory is returned when an archive lists an entry beneath a path that is not a directory -- a file under a symbolic link, most often.

var ErrNotImplemented = errors.New("unarchive: recognised, but this format is not read yet")

ErrNotImplemented is returned for a format that is recognised and not yet read. Telling the two apart matters to somebody holding a file: "I do not know what this is" and "I know exactly what this is and cannot open it yet" send them to different places.

var ErrReadOnly = errors.New("unarchive: archive is read-only")

ErrReadOnly is returned by every mutating method of a filesystem this package builds. An archive is read-only.

var ErrTooDeeplyNested = errors.New("unarchive: more stream wrappers than this will peel")

ErrTooDeeplyNested is returned when a file is wrapped more times than maxNesting.

var ErrTooLarge = errors.New("unarchive: exceeds the size limit")

ErrTooLarge is returned when an archive asks for more than a limit allows: more entries, longer paths, a larger dictionary, a bigger spool, a larger ReadFile than Limits (or, for Extract, Options) permit. It is raised before the memory or the disk is spent. The archive is not necessarily corrupt: raise the limit if the size is expected.

var ErrUnknownFormat = errors.New("unarchive: the bytes match no format this knows")

ErrUnknownFormat is returned when nothing recognises the bytes.

Functions

func FromFS

func FromFS(fsys iofs.FS) filesystem.Filesystem

FromFS presents any io/fs.FS as a filesystem.Filesystem.

It exists because the standard library already answers fs.FS for some of these formats -- a *zip.Reader is one -- so the driver for them is an adaptation and not a decoder. Writing a second zip reader to satisfy a different interface would be a duplicate of something already correct.

⛔ Random access costs what the underlying fs.FS charges. An fs.File is a stream: unless it also answers io.ReaderAt, reaching an offset means reading what comes before it, and reading BACKWARDS means opening the entry again. A zip entry is stored on its own, so a fresh open is cheap; an entry inside a solidly-compressed archive is not, and that is the format's shape rather than this adapter's.

func FromFSWithModes

func FromFSWithModes(fsys iofs.FS, modes map[string]os.FileMode) filesystem.Filesystem

FromFSWithModes is FromFS for an archive whose own metadata records modes the io/fs view does not report.

⛔ It exists because both io/fs adapters here LOSE a directory's permissions. archive/zip answers every directory with `fs.ModeDir | 0555` from its synthesised fileListEntry, and bodgit/sevenzip does the same, whatever the archive stored -- so a 0755 directory converted to tar came out dr-xr-xr-x, and the extracted tree could not be deleted without a chmod. Measured, not guessed: tar to tar kept 0755, zip to tar and 7z to tar both lost the owner's write bit, which is what pointed at the adapters rather than at the writers.

The map is keyed by path with no trailing slash. An entry whose recorded permissions are ZERO is skipped rather than believed: some archives carry no Unix mode at all, and a mode of 0 is the absence of a statement rather than a statement that nobody may read the file.

func InnerName

func InnerName(name string) string

InnerName is what the stream inside a compressed wrapper is called: the base name with the wrapper's suffix taken off, so notes.txt.gz gives notes.txt and archive.tgz gives archive.tar.

It is exported because the command needs the same answer for a different question -- what to call the directory it extracts into -- and computing it from filepath.Ext there gives "fixture.tar" for fixture.tar.gz, which is a directory named after half a suffix.

func SealTo

func SealTo(o *overlay.Overlay, target string) error

SealTo writes everything the overlay holds into target, once.

The format comes from target's name -- see TargetFor -- and the write is atomic: the archive is built beside target and RENAMED over it, so an interrupted seal leaves whatever was there before untouched rather than half-replaced.

func WithoutPartNumber

func WithoutPartNumber(name string) string

WithoutPartNumber strips a trailing part number, so film.zip.001 gives film.zip.

It is exported for the same reason InnerName is: the command needs the answer for a different question -- what to call the directory it extracts into -- and computing it from filepath.Ext there leaves "film.zip", a directory named after the archive's own extension.

Types

type Entry

type Entry struct {
	Path string
	Size int64
	Dir  bool
}

Entry is one thing an archive holds, as List reports it.

func List

func List(fsys filesystem.Filesystem) ([]Entry, error)

List names everything in fsys without writing anything.

It walks the same way Extract does -- and refuses the same names, so a person who lists before extracting is told about an entry that would leave the destination BEFORE several gigabytes land somewhere.

type Format

type Format string

Format is an archive format this package can recognise.

const (
	FormatUnknown Format = ""
	FormatRAR     Format = "rar"   // RAR 1.5-4.x and RAR5, single or multi-volume
	FormatZIP     Format = "zip"   // PKZIP, including the jar/epub/odf family
	Format7z      Format = "7z"    // 7-Zip
	FormatTar     Format = "tar"   // POSIX tar: v7, USTAR, PAX and GNU alike
	FormatAr      Format = "ar"    // the static-library and .deb container
	FormatCpio    Format = "cpio"  // an initramfs, and an RPM's payload
	FormatGzip    Format = "gzip"  // a stream wrapper, usually around a tar
	FormatBzip2   Format = "bzip2" // likewise
	FormatXZ      Format = "xz"    // likewise
	FormatZstd    Format = "zstd"  // likewise
	FormatLZ4     Format = "lz4"   // likewise
	// compress(1)'s .Z, read and not written -- see Format.Note.
	FormatZ Format = "compress"
	// Two more stream wrappers, same shape as the ones above.
	FormatLZO  Format = "lzo"  // lzop's container around LZO1X
	FormatLzip Format = "lzip" // LZMA in lzip's framing, not xz's
	// Archives whose driver reads them through an io.ReaderAt, like the images
	// below -- but they are archives, not filesystems, and they do NOT adopt the
	// handle they are given. See openReaderArchive.
	FormatXAR  Format = "xar"  // the macOS .pkg container
	FormatCAB  Format = "cab"  // Microsoft Cabinet
	FormatRPM  Format = "rpm"  // lead + headers + one compressed cpio payload
	FormatWARC Format = "warc" // a web crawl, one record per entry
	// Filesystem IMAGES, which are archives in every way that matters here: one
	// file holding a tree, handed around to be unpacked. They are read through
	// the org's own drivers rather than anything written here.
	//
	// The line is drawn at formats people DISTRIBUTE. You download an .iso and
	// you ship a .squashfs; you do not pass somebody an ext4 image expecting them
	// to unpack it. go-filesystems has drivers for a dozen more and this opens
	// two, on purpose.
	FormatISO9660  Format = "iso9660"
	FormatSquashFS Format = "squashfs"
	// HFS+ is here for the same reason: it is what a .dmg made before APFS
	// holds, so it is a format people receive rather than one they administer.
	FormatHFSPlus Format = "hfsplus"
	// A UDIF disk image -- a .dmg -- which is a CONTAINER around one of the
	// images above rather than an image itself. Its sectors are compressed and
	// may be stored out of order, so there is nothing for a driver to read in
	// place; it is peeled like a stream wrapper and what comes out is sniffed.
	FormatDMG Format = "dmg"
	// Recognised and deliberately not unpacked. See Format.Note.
	FormatPtar Format = "ptar"
)

func Open

func Open(path string) (filesystem.Filesystem, Format, error)

Open reads the archive at path, deciding what it is from its bytes, and returns it as a filesystem together with the format it turned out to be.

A multi-volume set is followed from here: the rar driver resolves the rest of the set by volume NUMBER within path's directory, so files carrying an id a download inserted are still reached and nothing is renamed.

func Sniff

func Sniff(r io.ReaderAt, size int64) (Format, error)

Sniff names the format of the archive or filesystem image in r.

The size is needed for the image half and not for the archive half: an archive's identity is in its first 262 bytes, while ISO 9660's is at offset 32769 and a probe that deep has to know where the file ends, or a truncated image turns into a read past the end.

The image probing is go-filesystems/detect's, not a second table written here. It is bounded through safeio, it handles both endiannesses of squashfs, and it knows a dozen filesystems -- so duplicating the two entries this package opens would mean re-deriving something deliberately hardened, and would drift from it.

func Writable

func Writable(path string) (*overlay.Overlay, Format, error)

Writable opens an archive and returns it as a filesystem that can be CHANGED, without rewriting the archive.

Changes go into a temporary cache; the archive underneath is untouched until SealTo. So renaming one entry in a 4 GiB archive costs a rename, not 4 GiB of rewriting, and an interrupted session leaves the original exactly as it was.

The caller closes the returned Overlay, which discards the cache. Closing it does NOT write anything back: that is SealTo's job, and keeping them apart is what makes "I changed my mind" free.

func (Format) Compressed

func (f Format) Compressed() bool

Compressed says whether a format is a stream wrapper rather than an archive of its own: gzip holds one stream, which is usually a tar and might be anything.

func (Format) Image

func (f Format) Image() bool

Image says whether a format is a filesystem image rather than an archive.

The distinction is not cosmetic: an image is already a filesystem, so there is nothing to decode and nothing to spool -- Open hands back a driver reading the file in place.

func (Format) Note

func (f Format) Note() string

Note returns a sentence about a format this package recognises and does not unpack, or "" for the rest.

It exists because "recognised, not read yet" is true of ptar and MISLEADING on its own: it suggests somebody will get round to it. What a .ptar is makes that the wrong expectation, and the person holding one is better served by being told which tool opens it than by waiting.

func (Format) ReaderArchive

func (f Format) ReaderArchive() bool

ReaderArchive says whether a format's driver reads it in place through an io.ReaderAt, the way an image is read, while being an ARCHIVE and not a filesystem.

The distinction is not academic. Both kinds are opened from the handle Open already has and neither is spooled, so they look alike from here -- but they differ about who CLOSES that handle, and getting it the wrong way round fails in opposite directions.

func (Format) String

func (f Format) String() string

String is the format's name, or "unknown".

type Limits

type Limits struct {
	// MaxEntries bounds the paths an index holds (tar, ar, cpio), counting the
	// directories an archive implies.
	MaxEntries int

	// MaxReadFile bounds what ReadFile returns, in bytes. Extract and OpenFile
	// stream and are not affected.
	MaxReadFile int64

	// MaxSpool bounds what a stream wrapper (.gz, .xz, .zst, ...) may decompress
	// to the temporary file it is spooled to.
	MaxSpool int64

	// MaxDictionary bounds the dictionary or window a decoder may reserve: the
	// LZMA2 or LZMA dictionary of an .xz stream or a zip entry (the window grows
	// with the data decoded and ErrTooLarge is returned before it would pass this),
	// a zstd window (refused from the frame header).
	MaxDictionary int64
}

Limits bounds what an untrusted archive may make Open, ReadFile and the stream wrappers reserve. A field that is zero or negative keeps its default.

type Options

type Options struct {
	// Overwrite lets an existing file be replaced. Off by default: an
	// extraction that overwrites by default destroys data on a mistyped
	// destination, and the mistake is not recoverable.
	Overwrite bool

	// Perm is the mode for created files and directories, before umask.
	// Zero means 0o644 for files and 0o755 for directories.
	Perm os.FileMode

	// Progress, when set, is told each entry as it completes.
	Progress func(name string, bytes int64)

	// MaxBytes bounds the bytes written in all. Zero means DefaultMaxBytes;
	// negative removes the limit. A compressed archive of a few kilobytes can
	// hold terabytes (42.zip), and an extractor that trusts it fills the disk.
	MaxBytes int64

	// MaxEntries bounds the files, directories and links created. Zero means
	// DefaultMaxEntries; negative removes the limit.
	MaxEntries int

	// MaxDepth bounds how deep the tree goes. Zero means DefaultMaxDepth;
	// negative removes the limit. The walk recurses once per level, and a Go
	// stack that runs out is a fatal error, not a panic that can be recovered.
	MaxDepth int
}

Options tune an extraction.

type Result

type Result struct {
	Dirs  int
	Files int
	// Links counts symbolic links written. They were counted as Files until
	// today, and written as empty regular ones -- see walk.
	Links int
	Bytes int64
}

Result is what an extraction did.

func Extract

func Extract(fsys filesystem.Filesystem, dest string, opt Options) (*Result, error)

Extract writes everything in fsys under dest.

Each file is compared against the size its header declared, and a shortfall is an error rather than a smaller number -- see ErrIncomplete.

type WriteTarget

type WriteTarget struct {
	Archive Format // FormatTar, FormatZIP or Format7z
	Wrapper Format // FormatUnknown, or one of the stream formats
}

WriteTarget is what a target name resolves to: an archive format, and an optional stream wrapper around it.

func TargetFor

func TargetFor(name string) (WriteTarget, error)

TargetFor says what to write into a file with this name.

⛔ This is the one place in the package where the NAME decides, and it has to: an output file has no bytes yet to read. Everywhere else a format comes from the content, because an extension is a claim by whoever last renamed the file; here the extension is the only statement of intent there is.

func (WriteTarget) String

func (t WriteTarget) String() string

String names the pair the way a file name spells it.