unarchive

package module
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 25, 2026 License: BSD-3-Clause Imports: 30 Imported by: 0

README

unarchive

Open an archive without being told what it is, and extract it while checking that what came out is what the archive promised.

Pure Go, CGO_ENABLED=0, and it builds for every target Go builds for.

go install github.com/go-filesystems/unarchive/cmd/unarchive@latest

unarchive film.part1.rar          # into ./film/
unarchive -C /tmp/out backup.tar.zst
unarchive -f already-there.zip    # overwrite
unarchive -n suspicious.7z        # list, extract nothing

unarchive -o film.7z film.rar     # convert, extracting nothing
unarchive -o backup.tar.zst backup.zip
unarchive -o disc.7z disc.iso     # yes, that works

Formats

read notes
RAR 1.5–4.x, RAR5 ✅ multi-volume sets followed by volume number
ZIP ✅ including the jar/epub/odf family
7z ✅ follows its own .001 chain, by name
tar ✅ v7, USTAR, PAX and GNU alike, with real random access
ar ✅ static libraries and .deb — both long-name spellings, SysV and BSD
cpio ✅ newc, crc and odc — an initramfs, an RPM's payload
gzip, bzip2, xz, zstd, lz4 ✅ stream wrappers: .tar.gz, .tgz, .tar.zst, a lone notes.txt.gz …
ISO 9660 ✅ a disc image is an archive too — read through go-filesystems/iso9660
SquashFS ✅ likewise, through go-filesystems/squashfs
plakar .ptar recognised, not unpacked and the reason is below

A numbered split — film.zip.001, .002, … — is read as one file, whatever the format inside it, so unarchive film.zip.001 works and extracts into film/. RAR is the exception: its own volume sets are followed by number already, and a plain numbered split of a RAR is a different thing this has no entry point for.

Written

.tar, .tar.gz, .tgz, .tar.xz, .txz, .tar.zst, .tzst, .tar.lz4, .tar.bz2, .tbz2, .tbz, .zip, .jar, .7z.

Not written: RAR, because no free writer exists. That is the whole list.

bzip2 used to be on it, for a different kind of reason — the standard library decompresses bzip2 and nothing compressed it. A missing library is not a property of a format, so the library was written (go-compressions/bzip2) and the sentence stopped being true. The two reasons look identical from the outside and only one of them is permanent.

brotli is absent on purpose. It has no signature: a brotli stream begins with the first bits of its own data, so there is nothing to recognise it by. The only way to open one is to be told, by an extension or a flag, and this package decides from the bytes.

plakar .ptar: recognised, and deliberately not unpacked

A .ptar is not an archive. It is a Kloset repository in a file — a config blob, a packfile region and a state region, addressed by MAC — holding snapshots and deduplicated chunks, and encrypted unless it was made with -plaintext.

Reading one means implementing their storage container, their state index, their packfile format, their snapshot model, and decryption with a key the caller must supply. That is a backup tool, not an unarchiver, and "which snapshot?" is a question this command has nowhere to ask. So it is recognised and says what it is:

$ unarchive backup.ptar
unarchive: backup.ptar: unarchive: recognised, but this format is not read yet:
ptar is a plakar Kloset archive: a content-addressed repository of snapshots and
deduplicated chunks, encrypted unless it was made with -plaintext. Open it with
plakar, which has the key

Recognition earns its keep on its own: without it, a .ptar answered "the bytes match no format this knows", which sends somebody hunting for a corrupt file.

The magic is _PLATAR_, read out of PlakarKorp's own storage driver. Guessing it from the extension — the thing this package exists not to do — would have got it wrong.

An image is an archive too

A filesystem image is one file holding a tree, handed around to be unpacked — which is what an archive is, from the outside. So .iso and .squashfs open, list, extract and convert like any other input, and nothing here decodes them: go-filesystems already owns those drivers, and the detection is go-filesystems/detect's hardened prober rather than a second magic table written here.

The line is drawn at formats people distribute. You download an .iso and you ship a .squashfs; you do not hand somebody an ext4 image expecting them to unpack it. The org has drivers for a dozen more filesystems and this opens two, on purpose.

Why the bytes and not the name

A format is recognised by what is in the file. An extension is a claim by whoever last renamed it: a .rar that is really a zip, a video saved as .mp4 that is an MPEG transport stream, a volume renamed by a download manager — all ordinary, and all wrong if the name decides.

The one thing a name is used for is what to call the single file inside a wrapper: somebody who gunzips notes.txt.gz expects notes.txt, and nothing in the bytes says so.

Why the check

An extractor that reports success is not the same as an extractor that got everything.

libarchive — which is bsdtar, and therefore macOS — does not follow a multi-volume RAR chain. It writes the first volume, pads the remainder with zeros to the declared length, and exits 0. The result is exactly the right size, which is what almost every check afterwards looks at. A 1.26 GiB file came out of it here, verified by size, believed, and the source archives were deleted on the strength of it.

So this package compares what was written against what the header declared, per entry, and says so when they differ (ErrIncomplete).

Three kinds of no

means
ErrUnknownFormat I do not know what this is
ErrNotImplemented I know exactly what this is and do not read it yet
anything else I read this format and this file is broken

They send a person to three different places, which is why they are three errors. Nothing is in the middle bucket today; a count in the tests fails if a format is ever added to the sniffer alone, so that stays a decision rather than a default.

Changing an archive without rewriting it

An archive can be opened as a filesystem you may write to, where the changes cost nothing until you say so:

o, format, err := unarchive.Writable("backup.tar.zst")
if err != nil { return err }
defer o.Close()                  // discards the changes; writes nothing back

o.WriteFile("etc/hosts", newHosts, 0o644)
o.MkDir("etc/extra", 0o755)
o.DeleteFile("var/log/old.log")

err = unarchive.SealTo(o, "backup.7z")   // ONE rewrite, here

Renaming one entry in a 4 GiB archive costs a rename, not 4 GiB of rewriting. The archive underneath is untouched until SealTo, so abandoning the changes is free and an interrupted session leaves the original exactly as it was. SealTo builds the new archive beside the target and renames over it, so an interruption there leaves whatever the target held before, rather than half of something new.

The target's format comes from its name. That is the one place in this package where a name decides anything, and it has to: an output file has no bytes yet to read.

unarchive -o is this, with no changes: a conversion.

As a filesystem

Open returns a filesystem.Filesystem from go-filesystems/interface, so an archive can be read like a mounted volume — ReadFile, ListDir, Stat, and OpenFile for a seekable handle, without extracting anything.

fsys, format, err := unarchive.Open("backup.tar.xz")
if err != nil { return err }
defer fsys.Close()          // removes the spool, for a compressed wrapper

b, err := fsys.ReadFile("etc/hosts")

A stream wrapper is spooled to a temporary file first, and that is the format's doing rather than a shortcut: every archive here needs random access — a tar index, a zip central directory, a 7z header at the end — and a decompressor gives a one-way stream. Reading it twice means decompressing it twice; holding it in memory means holding a whole archive in memory. Close removes the spool.

Nesting is capped: a wrapper holds one stream and that stream may be another wrapper, so a few hundred bytes of .gz nested on itself would otherwise ask for however much spool the machine has.

Licence

BSD-3-Clause.

Documentation

Overview

Package unarchive opens an archive without being told what it is, and extracts it while checking that what came out is what the archive promised.

Why the bytes and not the name

A format is recognised by what is IN the file. An extension is a claim by whoever last renamed it: a .rar that is really a zip, a video saved as .mp4 that is an MPEG transport stream, a volume renamed by a download manager -- all ordinary, and all wrong if the name decides.

Why the check

An extractor that reports success is not the same as an extractor that got everything. libarchive, which is bsdtar and therefore macOS, does not follow a multi-volume RAR chain: it writes the first volume, pads the remainder with ZEROS to the declared length and exits 0. The result is exactly the right SIZE, which is what every check afterwards looks at. This package compares what was written against what the header declared, per entry, and says so when they differ.

Index

Constants

This section is empty.

Variables

View Source
var ErrCannotWrite = errors.New("unarchive: this format is read but not written")

ErrCannotWrite is returned for a format this package reads and does not write.

A third sentence alongside the other two: "I know what this is, I can read it, and I cannot produce one". Only RAR is left in it, and for a reason that is not going to change here: no free writer exists.

bzip2 used to be in it too, because the standard library decompresses bzip2 and nothing compressed it. That was a missing LIBRARY rather than a property of the format, so the library was written -- go-compressions/bzip2 -- and the sentence stopped being true. Worth saying, because the two reasons look identical from the outside and only one of them is permanent.

View Source
var ErrEscapes = errors.New("unarchive: entry would be written outside the destination")

ErrEscapes is returned for an entry whose path would land outside the destination.

An archive is somebody else's list of names, and "../../.ssh/authorized_keys" is a name. Every extractor that trusted it has had the same bug, so this one does not trust it: the resolved path is checked against the destination, and an entry that leaves it is refused rather than sanitised, because silently rewriting a path puts the file somewhere the person did not ask for either.

View Source
var ErrExists = errors.New("unarchive: file exists")

ErrExists is returned when a file is already there and Overwrite is not set.

View Source
var ErrIncomplete = errors.New("unarchive: entry is shorter than its header declares")

ErrIncomplete is returned when an entry yielded fewer bytes than its header declared.

It is the point of this package. An extractor that writes what it got and reports success leaves a file of exactly the right SIZE when the source was zero-padded, and every check afterwards agrees with it.

View Source
var ErrMissingPart = errors.New("unarchive: a part of this split is missing")

ErrMissingPart is returned when a numbered set has a hole in it.

A hole is not the same as the end. Concatenating parts 1, 2 and 4 produces a file of a plausible SIZE whose middle is wrong, and every reader then reports corruption -- of the archive, which is intact. Refusing here names the part that is absent.

View Source
var ErrNotImplemented = errors.New("unarchive: recognised, but this format is not read yet")

ErrNotImplemented is returned for a format that is recognised and not yet read. Telling the two apart matters to somebody holding a file: "I do not know what this is" and "I know exactly what this is and cannot open it yet" send them to different places.

View Source
var ErrReadOnly = errors.New("unarchive: archive is read-only")

ErrReadOnly is returned by every mutating method of a filesystem this package builds. An archive is read-only.

View Source
var ErrTooDeeplyNested = errors.New("unarchive: more stream wrappers than this will peel")

ErrTooDeeplyNested is returned when a file is wrapped more times than maxNesting.

View Source
var ErrUnknownFormat = errors.New("unarchive: the bytes match no format this knows")

ErrUnknownFormat is returned when nothing recognises the bytes.

Functions

func FromFS

func FromFS(fsys iofs.FS) filesystem.Filesystem

FromFS presents any io/fs.FS as a filesystem.Filesystem.

It exists because the standard library already answers fs.FS for some of these formats -- a *zip.Reader is one -- so the driver for them is an adaptation and not a decoder. Writing a second zip reader to satisfy a different interface would be a duplicate of something already correct.

⛔ Random access costs what the underlying fs.FS charges. An fs.File is a stream: unless it also answers io.ReaderAt, reaching an offset means reading what comes before it, and reading BACKWARDS means opening the entry again. A zip entry is stored on its own, so a fresh open is cheap; an entry inside a solidly-compressed archive is not, and that is the format's shape rather than this adapter's.

func FromFSWithModes

func FromFSWithModes(fsys iofs.FS, modes map[string]os.FileMode) filesystem.Filesystem

FromFSWithModes is FromFS for an archive whose own metadata records modes the io/fs view does not report.

⛔ It exists because both io/fs adapters here LOSE a directory's permissions. archive/zip answers every directory with `fs.ModeDir | 0555` from its synthesised fileListEntry, and bodgit/sevenzip does the same, whatever the archive stored -- so a 0755 directory converted to tar came out dr-xr-xr-x, and the extracted tree could not be deleted without a chmod. Measured, not guessed: tar to tar kept 0755, zip to tar and 7z to tar both lost the owner's write bit, which is what pointed at the adapters rather than at the writers.

The map is keyed by path with no trailing slash. An entry whose recorded permissions are ZERO is skipped rather than believed: some archives carry no Unix mode at all, and a mode of 0 is the absence of a statement rather than a statement that nobody may read the file.

func InnerName

func InnerName(name string) string

InnerName is what the stream inside a compressed wrapper is called: the base name with the wrapper's suffix taken off, so notes.txt.gz gives notes.txt and archive.tgz gives archive.tar.

It is exported because the command needs the same answer for a different question -- what to call the directory it extracts into -- and computing it from filepath.Ext there gives "fixture.tar" for fixture.tar.gz, which is a directory named after half a suffix.

func SealTo

func SealTo(o *overlay.Overlay, target string) error

SealTo writes everything the overlay holds into target, once.

The format comes from target's name -- see TargetFor -- and the write is atomic: the archive is built beside target and RENAMED over it, so an interrupted seal leaves whatever was there before untouched rather than half-replaced.

func WithoutPartNumber added in v0.4.0

func WithoutPartNumber(name string) string

WithoutPartNumber strips a trailing part number, so film.zip.001 gives film.zip.

It is exported for the same reason InnerName is: the command needs the answer for a different question -- what to call the directory it extracts into -- and computing it from filepath.Ext there leaves "film.zip", a directory named after the archive's own extension.

Types

type Entry

type Entry struct {
	Path string
	Size int64
	Dir  bool
}

Entry is one thing an archive holds, as List reports it.

func List

func List(fsys filesystem.Filesystem) ([]Entry, error)

List names everything in fsys without writing anything.

It walks the same way Extract does -- and refuses the same names, so a person who lists before extracting is told about an entry that would leave the destination BEFORE several gigabytes land somewhere.

type Format

type Format string

Format is an archive format this package can recognise.

const (
	FormatUnknown Format = ""
	FormatRAR     Format = "rar"   // RAR 1.5-4.x and RAR5, single or multi-volume
	FormatZIP     Format = "zip"   // PKZIP, including the jar/epub/odf family
	Format7z      Format = "7z"    // 7-Zip
	FormatTar     Format = "tar"   // POSIX tar: v7, USTAR, PAX and GNU alike
	FormatAr      Format = "ar"    // the static-library and .deb container
	FormatCpio    Format = "cpio"  // an initramfs, and an RPM's payload
	FormatGzip    Format = "gzip"  // a stream wrapper, usually around a tar
	FormatBzip2   Format = "bzip2" // likewise
	FormatXZ      Format = "xz"    // likewise
	FormatZstd    Format = "zstd"  // likewise
	FormatLZ4     Format = "lz4"   // likewise
	// Filesystem IMAGES, which are archives in every way that matters here: one
	// file holding a tree, handed around to be unpacked. They are read through
	// the org's own drivers rather than anything written here.
	//
	// The line is drawn at formats people DISTRIBUTE. You download an .iso and
	// you ship a .squashfs; you do not pass somebody an ext4 image expecting them
	// to unpack it. go-filesystems has drivers for a dozen more and this opens
	// two, on purpose.
	FormatISO9660  Format = "iso9660"
	FormatSquashFS Format = "squashfs"
	// Recognised and deliberately not unpacked. See Format.Note.
	FormatPtar Format = "ptar"
)

func Open

func Open(path string) (filesystem.Filesystem, Format, error)

Open reads the archive at path, deciding what it is from its bytes, and returns it as a filesystem together with the format it turned out to be.

A multi-volume set is followed from here: the rar driver resolves the rest of the set by volume NUMBER within path's directory, so files carrying an id a download inserted are still reached and nothing is renamed.

func Sniff

func Sniff(r io.ReaderAt, size int64) (Format, error)

Sniff names the format of the archive or filesystem image in r.

The size is needed for the image half and not for the archive half: an archive's identity is in its first 262 bytes, while ISO 9660's is at offset 32769 and a probe that deep has to know where the file ends, or a truncated image turns into a read past the end.

The image probing is go-filesystems/detect's, not a second table written here. It is bounded through safeio, it handles both endiannesses of squashfs, and it knows a dozen filesystems -- so duplicating the two entries this package opens would mean re-deriving something deliberately hardened, and would drift from it.

func Writable

func Writable(path string) (*overlay.Overlay, Format, error)

Writable opens an archive and returns it as a filesystem that can be CHANGED, without rewriting the archive.

Changes go into a temporary cache; the archive underneath is untouched until SealTo. So renaming one entry in a 4 GiB archive costs a rename, not 4 GiB of rewriting, and an interrupted session leaves the original exactly as it was.

The caller closes the returned Overlay, which discards the cache. Closing it does NOT write anything back: that is SealTo's job, and keeping them apart is what makes "I changed my mind" free.

func (Format) Compressed

func (f Format) Compressed() bool

Compressed says whether a format is a stream wrapper rather than an archive of its own: gzip holds one stream, which is usually a tar and might be anything.

func (Format) Image added in v0.3.0

func (f Format) Image() bool

Image says whether a format is a filesystem image rather than an archive.

The distinction is not cosmetic: an image is already a filesystem, so there is nothing to decode and nothing to spool -- Open hands back a driver reading the file in place.

func (Format) Note added in v0.4.0

func (f Format) Note() string

Note returns a sentence about a format this package recognises and does not unpack, or "" for the rest.

It exists because "recognised, not read yet" is true of ptar and MISLEADING on its own: it suggests somebody will get round to it. What a .ptar is makes that the wrong expectation, and the person holding one is better served by being told which tool opens it than by waiting.

func (Format) String

func (f Format) String() string

String is the format's name, or "unknown".

type Options

type Options struct {
	// Overwrite lets an existing file be replaced. Off by default: an
	// extraction that overwrites by default destroys data on a mistyped
	// destination, and the mistake is not recoverable.
	Overwrite bool

	// Perm is the mode for created files and directories, before umask.
	// Zero means 0o644 for files and 0o755 for directories.
	Perm os.FileMode

	// Progress, when set, is told each entry as it completes.
	Progress func(name string, bytes int64)
}

Options tune an extraction.

type Result

type Result struct {
	Dirs  int
	Files int
	Bytes int64
}

Result is what an extraction did.

func Extract

func Extract(fsys filesystem.Filesystem, dest string, opt Options) (*Result, error)

Extract writes everything in fsys under dest.

Each file is compared against the size its header declared, and a shortfall is an error rather than a smaller number -- see ErrIncomplete.

type WriteTarget

type WriteTarget struct {
	Archive Format // FormatTar, FormatZIP or Format7z
	Wrapper Format // FormatUnknown, or one of the stream formats
}

WriteTarget is what a target name resolves to: an archive format, and an optional stream wrapper around it.

func TargetFor

func TargetFor(name string) (WriteTarget, error)

TargetFor says what to write into a file with this name.

⛔ This is the one place in the package where the NAME decides, and it has to: an output file has no bytes yet to read. Everywhere else a format comes from the content, because an extension is a claim by whoever last renamed the file; here the extension is the only statement of intent there is.

func (WriteTarget) String

func (t WriteTarget) String() string

String names the pair the way a file name spells it.

Directories

Path Synopsis
cmd
unarchive command
Command unarchive extracts an archive without being told what it is.
Command unarchive extracts an archive without being told what it is.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL