deepscan

README

Tools for Deep Scanning Archives and Repositories

src.agwa.name/deepscan is a Go library that recursively descends into archives and repositories, executing a given function for every file that it finds. It's extensible, allowing you to add support for your own URL schemes and archive formats.

govulncheck-deep

govulncheck-deep is a command line program that runs govulncheck on every Go binary it can find at a specified URL, descending into archives as needed. There are other programs that do deep vulnerability scanning, but they don't use govulncheck so they return many false positives.

For example, if you have an S3 bucket of tarballs containing Go binaries, you can run the following command to check them for vulnerabilities:

govulncheck-deep s3://mybucket

The argument must be a URL with one of the supported schemes listed below.

If govulncheck finds a vulnerability, the path to the binary is written to stdout followed by the output of govulncheck. If no vulnerabilities are found, nothing is written to stdout.

To install govulncheck-deep, run:

go install src.agwa.name/deepscan/cmd/govulncheck-deep@latest

Supported Archive Formats

• .deb
• .tar
• .tar.bz2
• .tar.gz
• .tgz
• .zip

Supported URL Schemes

• apt+http://, apt+https:// -- followed by a hostname and path to an APT repository; see the documentation for more information
• file:// -- followed by a path to a file or directory
• s3:// -- followed by the name of an S3 bucket

Documentation

Overview

Package deepscan contains functions for recursively scanning archives

Index

• Variables
• func Scan(ctx context.Context, path []string, f io.Reader, fn ScanFunc) error
• func ScanFS(ctx context.Context, path []string, fsys fs.FS, fn ScanFunc) error
• func ScanURL(ctx context.Context, url string, fn ScanFunc) error
• type ScanFunc

Variables

var Formats = make(map[string]func(ctx context.Context, path []string, f io.Reader, fn ScanFunc) error)

Formats is a map from filename suffix (e.g. ".tar.gz") to a function that deep scans files with that suffix. You can add support for new formats by adding to this map from a package init function.

var Protocols = make(map[string]func(ctx context.Context, url string, fn ScanFunc) error)

Protocols is a map from URL scheme to a function that deep scans URLs of that scheme You can add support for new protocols by adding to this map from a package init function.

var UserAgent = "deepscan"

UserAgent is the User-Agent string sent with HTTP requests made by this library

Functions

func Scan

func Scan(ctx context.Context, path []string, f io.Reader, fn ScanFunc) error

Scan scans a file found at the given path (which must be non-empty). If the filename suffix has a handler in Formats, then the handler is invoked to deep scan the file. Otherwise, the ScanFunc is called.

func ScanFS

func ScanFS(ctx context.Context, path []string, fsys fs.FS, fn ScanFunc) error

ScanFS walks the given filesystem, invoking Scan on every file. If an error occurs, the ScanFunc is called with the error.

func ScanURL

func ScanURL(ctx context.Context, url string, fn ScanFunc) error

ScanURL deep scans a URL, descending into archives as necessary, invoking the ScanFunc for every file found at the URL. The URL's scheme must have a handler registered in Protocols.

Types

type ScanFunc

type ScanFunc func(ctx context.Context, path []string, r io.Reader, err error) error

ScanFunc is a function which is called for every terminal (non-archive) file. path contains the URLs and filenames which were followed to reach the file. If the file was successfully opened, the io.Reader is non-nil; otherwise, the error is non-nil. If the ScanFunc returns a non-nil error, the deep scan is immediately terminated and the error is returned from ScanURL, Scan, or ScanFS.