hashset

README

go-hashset

Hash allowlist / denylist lookup for forensic and security workflows — filter known-good files against an NSRL reference set, or flag known-bad hashes from a threat-intel feed. MD5, SHA-1, SHA-256.

Two backing stores behind one Set interface:

Store	Build with	For
in-memory	LoadText / LoadTextFile	lists up to ~1M entries; O(1) lookup
bbolt-backed	Build → OpenBolt	NSRL-scale (~50M entries); O(log N), stays on disk

Only dependency: go.etcd.io/bbolt.

Install

go get github.com/richardwooding/go-hashset

Usage

Query an in-memory list:

import hashset "github.com/richardwooding/go-hashset"

set, err := hashset.LoadTextFile("known-bad.txt") // one hex hash per line, # comments, mixed algos
if err != nil { /* ... */ }
defer set.Close()

if set.Contains("sha256", "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824") {
    // flag it
}
fmt.Println(set.Counts()) // map[md5:.. sha1:.. sha256:..]

Build a bbolt set from a big NSRL drop, then query it:

in, _ := os.Open("NSRLFile.txt")
defer in.Close()
err := hashset.Build(in, "nsrl.hashset", hashset.BuildOpts{
    Format:   "auto", // "text" | "nsrl" | "auto" (sniffs the header)
    Progress: func(n int64) { log.Printf("%d hashes", n) },
})

set, _ := hashset.OpenBolt("nsrl.hashset") // read-only
defer set.Close()
known := set.Contains("sha1", sha1hex)

Open(path) auto-detects: a bbolt file loads via OpenBolt, anything else falls back to LoadTextFile.

Formats

• Text — one hex-encoded hash per line; algorithm auto-detected by length (32 = md5, 40 = sha1, 64 = sha256); blank lines and # comments ignored; mixed algorithms allowed.
• NSRL — the NSRLFile.txt quoted-CSV with a "SHA-1","MD5",… header; the SHA-1 and MD5 columns are extracted (header-driven, tolerant of reordering / extra columns).

The bbolt file is a four-bucket database (md5/sha1/sha256 + meta with a schema_version); raw hash bytes are the keys.

Extracted from file-search-on, where it backs the is_known_good / is_known_bad search predicates.

License

MIT — see LICENSE.

Documentation

Overview

Package hashset provides hash-allowlist / hash-denylist lookup for forensic and security workflows — e.g. filtering known-good files against an NSRL reference set, or flagging known-bad hashes from a threat-intel feed.

Two backing stores share one Set interface:

• in-memory (NewMemory + LoadText): three maps keyed by raw hash bytes. Fast for lists up to ~1M entries; held entirely in RAM.
• bbolt-backed (OpenBolt, built via Build): three buckets keyed by raw hash bytes. Used for NSRL-scale lists (~50M entries) where holding everything in RAM is impractical.

Lookup is O(1) for in-memory and O(log N) for bbolt — either is fast enough that allowlist filtering doesn't dominate a walk where the per-file hashing is already paid for. Supported algorithms: MD5, SHA-1, SHA-256.

Index

• Constants
• Variables
• func Build(r io.Reader, outPath string, opts BuildOpts) error
• type BuildOpts
• type Set
  • func LoadText(r io.Reader) (Set, error)
  • func LoadTextFile(path string) (Set, error)
  • func NewMemory() Set
  • func Open(path string) (Set, error)
  • func OpenBolt(path string) (Set, error)

Constants

const DefaultBuildBatchSize = 50_000

DefaultBuildBatchSize is the bbolt transaction batch size. 50k rows balances commit overhead against memory usage during the build pass.

Variables

var Algorithms = []string{"md5", "sha1", "sha256"}

Algorithms is the canonical set of algorithm names supported by every backing store. Other values passed to Contains return false.

var ErrInvalidHex = errors.New("invalid hex hash")

ErrInvalidHex is returned when a hex string contains non-hex characters.

var ErrSchemaMismatch = errors.New("hashset: schema version mismatch (rebuild via Build)")

ErrSchemaMismatch is returned by OpenBolt when the file's `meta/schema_version` key isn't the current "1".

var ErrUnknownAlgo = errors.New("unknown hash algorithm")

ErrUnknownAlgo is returned when a hex string's length doesn't match any supported algorithm (md5=32, sha1=40, sha256=64).

var LineLengthToAlgo = map[int]string{
	32: "md5",
	40: "sha1",
	64: "sha256",
}

LineLengthToAlgo maps hex-string lengths back to the algorithm name. Used by the text loader to auto-detect each line's algo without explicit markers.

Functions

func Build

func Build(r io.Reader, outPath string, opts BuildOpts) error

Build reads from r and writes a bbolt hashset file at outPath. Existing files are overwritten.

The output is a bbolt database with four buckets:

• md5 / sha1 / sha256 — raw 16/20/32-byte hash bytes as keys, empty values (membership is the signal).
• meta — schema_version=1.

Read via OpenBolt / Open. Single-threaded; the bottleneck on NSRL-scale input is fsync, not parse — running multiple builders in parallel doesn't help.

Types

type BuildOpts

type BuildOpts struct {
	Format string // "text" / "nsrl" / "auto" (default "auto")
	// Progress, when non-nil, is invoked roughly every BatchSize
	// rows with the cumulative count read so far. Useful for
	// CLI progress reporting against multi-GB NSRL drops.
	Progress func(total int64)
	// BatchSize controls how many rows are buffered before each
	// bbolt commit. Bigger = fewer commits but bigger transactions.
	// 0 → DefaultBuildBatchSize.
	BatchSize int
}

BuildOpts tunes Build. Format selects how to interpret the input file: "text" for newline-separated hashes (mixed algorithms, auto-detected by length), "nsrl" for the NSRLFile.txt CSV format. "auto" (the default for the CLI) detects by inspecting the first non-blank line.

type Set

type Set interface {
	Contains(algo, hexHash string) bool
	Counts() map[string]int
	Close() error
}

Set is the read interface every hashset backing store exposes.

Contains reports whether the given hex-encoded hash (lowercase, canonical length: 32 for md5, 40 for sha1, 64 for sha256) appears in the set. Hashes of unrecognised length always return false. algo is "md5" / "sha1" / "sha256"; other values return false.

Counts returns the per-algorithm entry count snapshot, so a caller can sanity-check that a set loaded as expected.

Close releases resources (the bbolt-backed impl closes the file; in-memory is a no-op). Safe to call more than once; subsequent calls are no-ops.

func LoadText

func LoadText(r io.Reader) (Set, error)

LoadText reads a newline-separated hash list from r and inserts every hash into an in-memory Set.

File format:

• One hex-encoded hash per line, mixed algorithms allowed (each line's algorithm is auto-detected by its length).
• Blank lines are ignored.
• Lines beginning with `#` are comments and ignored.
• Inline whitespace is trimmed.
• Lines that aren't blank, comments, or valid hex of a known length are rejected with an error naming the line number.

func LoadTextFile

func LoadTextFile(path string) (Set, error)

LoadTextFile opens path and calls LoadText against its contents.

func NewMemory

func NewMemory() Set

NewMemory returns an empty in-memory Set. Callers populate it via LoadText (the typical entry point) or by inserting hashes through AddHex.

func Open

func Open(path string) (Set, error)

Open auto-detects the file format: a bbolt file (with the right schema metadata) loads via OpenBolt; everything else falls through to LoadTextFile. Useful for code paths that accept either format transparently.

Detection strategy: try OpenBolt first; on ErrSchemaMismatch OR any bbolt open error, fall back to text. This catches both real text files AND bbolt files written by an incompatible version (the latter retries through text, fails clearly, and prompts a rebuild).

func OpenBolt

func OpenBolt(path string) (Set, error)

OpenBolt opens an existing bbolt-format hashset file for read-only queries. Build the file with Build.

The file is opened in bbolt's read-only mode so queries can't accidentally mutate the set. ErrSchemaMismatch surfaces when the file's schema_version meta key doesn't match this package — protecting against future format changes.