archives

README

archives

Recursively walk archive and compressed-container files in Go and get a stream for every file inside, no matter how deeply nested. Point it at an APK, a .deb, or a firmware.tar.gz and it hands you each contained file in turn — descending through ar → tar.xz → ELF or zip → dex automatically — with built-in decompression-bomb guards.

Supported containers: zip (and zip-based .apk/.jar/.ipa/.aar), tar, gzip, bzip2, xz, zstd, and ar (.deb). Nested combinations are recursed automatically.

Extracted from txtr, where it powers --recurse.

Install

go get github.com/richardwooding/archives

Usage

package main

import (
	"fmt"
	"io"

	"github.com/richardwooding/archives"
)

func main() {
	err := archives.Walk("pkg.deb", archives.Options{}, func(path string, r io.Reader) error {
		n, _ := io.Copy(io.Discard, r)
		fmt.Printf("%-60s %d bytes\n", path, n)
		// pkg.deb!/data.tar!/usr/bin/htop   123456 bytes
		return nil
	})
	if err != nil {
		panic(err)
	}
}

Walk opens the file, detects its container format by magic bytes, and invokes the callback once per leaf file with a virtual path (using !/ to separate nesting levels) and a reader over its decompressed contents. A non-archive file is emitted as a single leaf at its own path, so callers can use Walk uniformly without checking the type first.

Safety

Untrusted archives can be hostile (decompression bombs, deeply nested containers). Walk is bounded by Options:

archives.Walk(path, archives.Options{
	MaxDepth: 8,                 // max nesting; deeper members are emitted as opaque leaves
	MaxBytes: 2 << 30,           // max total decompressed bytes per input (<0 = unlimited)
}, fn)

Zero values use the defaults (DefaultMaxDepth = 8, DefaultMaxBytes = 2 GiB). When the byte budget is exhausted, the walk stops gracefully rather than erroring.

Design

• Streaming — files are handed to the callback as io.Readers; the whole archive is never loaded into memory (a top-level zip is read via random access on the file).
• Magic-byte detection — format is detected from content, not file extension (IsArchiveName is provided separately for extension-based pre-filtering).
• Pure Go — standard library plus the pure-Go klauspost/compress (zstd) and ulikunitz/xz decoders. No CGO.

License

MIT — see LICENSE.

Documentation

Overview

Package archives provides recursive walking of archive and compressed-container files (zip, tar, gzip, bzip2, xz, zstd, ar/deb), yielding each contained file as a stream. It is built for tools that need to read inside nested containers — firmware images, Android APKs, Debian packages, JARs — such as scanners, indexers, and forensics utilities.

The entry point is Walk. Walking is bounded against decompression bombs and pathological nesting via Options (maximum depth and maximum total decompressed bytes). The package depends only on the standard library plus pure-Go xz and zstd decoders, so it adds no CGO.

Index

• Constants
• func IsArchiveName(name string) bool
• func Walk(path string, opts Options, fn WalkFunc) error
• type Format
• type Options
• type WalkFunc

Constants

const (
	DefaultMaxDepth = 8
	// DefaultMaxBytes caps total decompressed bytes processed per top-level
	// input (2 GiB), a backstop against decompression bombs.
	DefaultMaxBytes int64 = 2 << 30
)

Default limits guarding against decompression bombs and pathological nesting.

const PathSeparator = "!/"

PathSeparator joins an archive path to a member path in virtual paths, e.g. "firmware.tar.gz!/bin/busybox". It mirrors the convention used by jar URLs.

Functions

func IsArchiveName

func IsArchiveName(name string) bool

IsArchiveName reports whether a filename looks like a supported archive by its extension. Used to decide, before opening, whether recursion is applicable.

func Walk

func Walk(path string, opts Options, fn WalkFunc) error

Walk opens the file at path, detects its container format, and invokes fn for every leaf file in the (possibly nested) archive tree. A non-archive file is emitted as a single leaf at its own path. Decompression-bomb and depth limits are enforced per Options.

Types

type Format

type Format int

Format identifies a container/compression format.

const (
	FormatNone  Format = iota // not a recognized archive (leaf file)
	FormatZip                 // zip and zip-based (apk/jar/ipa/...)
	FormatTar                 // uncompressed tar
	FormatGzip                // gzip stream
	FormatBzip2               // bzip2 stream
	FormatXz                  // xz stream
	FormatZstd                // zstandard stream
	FormatAr                  // Unix ar archive (Debian .deb)
)

Supported formats.

type Options

type Options struct {
	// MaxDepth bounds archive nesting; members deeper than this are emitted as
	// opaque leaves rather than descended into. 0 uses DefaultMaxDepth.
	MaxDepth int
	// MaxBytes bounds total decompressed bytes per top-level input. 0 uses
	// DefaultMaxBytes; negative means unlimited.
	MaxBytes int64
}

Options configures a Walk.

type WalkFunc

type WalkFunc func(path string, r io.Reader) error

WalkFunc is invoked once per leaf file discovered in the archive tree. path is the virtual path (e.g. "pkg.deb!/data.tar!/usr/bin/htop") and r streams the member's decompressed contents. Returning an error aborts the walk.