archives

README

archives

Recursively walk archive and compressed-container files in Go and get a stream for every file inside, no matter how deeply nested. Point it at an APK, a .deb, or a firmware.tar.gz and it hands you each contained file in turn — descending through ar → tar.xz → ELF or zip → dex automatically — with built-in decompression-bomb guards.

Supported containers: zip (and zip-based .apk/.jar/.ipa/.aar), tar, gzip, bzip2, xz, zstd, and ar (.deb). Nested combinations are recursed automatically.

Extracted from txtr, where it powers --recurse.

Install

go get github.com/richardwooding/archives

Usage

package main

import (
	"context"
	"fmt"
	"io"

	"github.com/richardwooding/archives"
)

func main() {
	ctx := context.Background()
	err := archives.Walk(ctx, "pkg.deb", archives.Options{}, func(ctx context.Context, path string, r io.Reader) error {
		n, _ := io.Copy(io.Discard, r)
		fmt.Printf("%-60s %d bytes\n", path, n)
		// pkg.deb!/data.tar!/usr/bin/htop   123456 bytes
		return nil
	})
	if err != nil {
		panic(err)
	}
}

Walk opens the file, detects its container format by magic bytes, and invokes the callback once per leaf file with a virtual path (using !/ to separate nesting levels) and a reader over its decompressed contents. A non-archive file is emitted as a single leaf at its own path, so callers can use Walk uniformly without checking the type first.

Cancellation

Walk takes a context.Context and observes cancellation both between members and mid-stream during long reads. When the context is cancelled (or its deadline expires), Walk stops and returns ctx.Err() (context.Canceled / context.DeadlineExceeded). The same context is passed to the callback so it can honor cancellation in its own work.

ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()

err := archives.Walk(ctx, path, archives.Options{}, func(ctx context.Context, path string, r io.Reader) error {
	return store(ctx, path, r) // cancellation propagates into downstream work
})

Safety

Untrusted archives can be hostile (decompression bombs, deeply nested containers). Walk is bounded by Options:

archives.Walk(ctx, path, archives.Options{
	MaxDepth: 8,                 // max nesting; deeper members are emitted as opaque leaves
	MaxBytes: 2 << 30,           // max total decompressed bytes per input (<0 = unlimited)
}, fn)

Zero values use the defaults (DefaultMaxDepth = 8, DefaultMaxBytes = 2 GiB). When the byte budget is exhausted, the walk stops gracefully rather than erroring.

Diagnostics

Non-fatal anomalies — an unreadable member, a malformed sub-archive, the byte budget being hit — are recovered (the walk skips the member, scans the raw bytes, or stops gracefully) and reported through an optional *slog.Logger. The library is silent by default: a nil Logger discards these warnings, and they are never written to stderr.

archives.Walk(ctx, path, archives.Options{
	Logger: slog.Default(), // opt in to non-fatal walk warnings
}, fn)

Design

• Streaming — files are handed to the callback as io.Readers; the whole archive is never loaded into memory (a top-level zip is read via random access on the file).
• Magic-byte detection — format is detected from content, not file extension (IsArchiveName is provided separately for extension-based pre-filtering).
• Pure Go — standard library plus the pure-Go klauspost/compress (zstd) and ulikunitz/xz decoders. No CGO.

License

MIT — see LICENSE.

Documentation

Overview

Package archives provides recursive walking of archive and compressed-container files (zip, tar, gzip, bzip2, xz, zstd, ar/deb), yielding each contained file as a stream. It is built for tools that need to read inside nested containers — firmware images, Android APKs, Debian packages, JARs — such as scanners, indexers, and forensics utilities.

The entry point is Walk. Walking is bounded against decompression bombs and pathological nesting via Options (maximum depth and maximum total decompressed bytes). The package depends only on the standard library plus pure-Go xz and zstd decoders, so it adds no CGO.

Index

• Constants
• func IsArchiveName(name string) bool
• func Walk(ctx context.Context, path string, opts Options, fn WalkFunc) error
• type Format
• type Options
• type WalkFunc

Constants

const (
	DefaultMaxDepth = 8
	// DefaultMaxBytes caps total decompressed bytes processed per top-level
	// input (2 GiB), a backstop against decompression bombs.
	DefaultMaxBytes int64 = 2 << 30
)

Default limits guarding against decompression bombs and pathological nesting.

const PathSeparator = "!/"

PathSeparator joins an archive path to a member path in virtual paths, e.g. "firmware.tar.gz!/bin/busybox". It mirrors the convention used by jar URLs.

Functions

func IsArchiveName

func IsArchiveName(name string) bool

IsArchiveName reports whether a filename looks like a supported archive by its extension. Used to decide, before opening, whether recursion is applicable.

func Walk

func Walk(ctx context.Context, path string, opts Options, fn WalkFunc) error

Walk opens the file at path, detects its container format, and invokes fn for every leaf file in the (possibly nested) archive tree. A non-archive file is emitted as a single leaf at its own path. Decompression-bomb and depth limits are enforced per Options.

The walk is cancellable via ctx: cancellation is observed between members and mid-stream during long reads, and Walk returns ctx.Err() (context.Canceled or context.DeadlineExceeded) when the walk is aborted. A graceful budget-exceeded stop is reported as a nil error.

Types

type Format

type Format int

Format identifies a container/compression format.

const (
	FormatNone  Format = iota // not a recognized archive (leaf file)
	FormatZip                 // zip and zip-based (apk/jar/ipa/...)
	FormatTar                 // uncompressed tar
	FormatGzip                // gzip stream
	FormatBzip2               // bzip2 stream
	FormatXz                  // xz stream
	FormatZstd                // zstandard stream
	FormatAr                  // Unix ar archive (Debian .deb)
)

Supported formats.

type Options

type Options struct {
	// MaxDepth bounds archive nesting; members deeper than this are emitted as
	// opaque leaves rather than descended into. 0 uses DefaultMaxDepth.
	MaxDepth int
	// MaxBytes bounds total decompressed bytes per top-level input. 0 uses
	// DefaultMaxBytes; negative means unlimited.
	MaxBytes int64
	// Logger receives non-fatal warnings encountered during a walk — an
	// unreadable member, a malformed sub-archive, the byte budget being hit.
	// These conditions are recovered (the walk skips the member, scans the raw
	// bytes, or stops gracefully) rather than returned as errors, so they are
	// reported here. A nil Logger (the default) discards them, keeping the
	// library silent unless the caller opts in.
	Logger *slog.Logger
}

Options configures a Walk.

type WalkFunc

type WalkFunc func(ctx context.Context, path string, r io.Reader) error

WalkFunc is invoked once per leaf file discovered in the archive tree. ctx is the context passed to Walk, allowing the callback to honor cancellation in its own work. path is the virtual path (e.g. "pkg.deb!/data.tar!/usr/bin/htop") and r streams the member's decompressed contents. Returning an error aborts the walk.