walkman

README

walkman

A concurrent filesystem walker for Go, powered by work stealing.

Status: experimental / actively evolving. Core walker, tests, and benchmarks are in place; API not yet stable.

walkman turns each directory into an independent task and runs those tasks on a configurable work-stealing pool. Idle workers steal pending directories from busier ones, so on wide trees with lots of independent directories, more CPU stays busy than a sequential walk allows.

Contents

• Features
• Installation
• Basic usage
• API
• Traversal semantics
• How it works
• CLI tools
• Benchmarks
• Testing
• Limitations / roadmap
• Project layout

Features

• Concurrent traversal via a work-stealing pool, worker count configurable (defaults to GOMAXPROCS)
• Streaming results through a channel, one WalkResult per directory
• Per-directory error reporting that doesn't abort unrelated work
• Skip entries by name (prunes matching directories), optional max depth, optional symlink following
• Optional atomic traversal statistics
• Benchmark suite vs filepath.WalkDir and Rust's walkdir crate, plus a synthetic tree generator (wide/deep/mixed)

Installation

go get github.com/PAKIWASI/walkman

Basic usage

package main

import (
    "fmt"
    "log"

    "github.com/PAKIWASI/walkman"
)

func main() {
    w := walkman.NewWalkman(
        false, // follow symlinks
        0,     // max depth: 0 = unlimited
        nil,   // names to skip
    )

    for result := range w.Walk(".") {
        if result.Err != nil {
            log.Printf("%s: %v", result.Dir, result.Err)
            continue
        }
        for _, entry := range result.Ret {
            fmt.Printf("%s/%s\n", result.Dir, entry.Name())
        }
    }

    if err := w.Wait(); err != nil {
        log.Fatal(err)
    }
}

Drain before you wait. Walk's channel stays open until all queued work completes — consume it fully, then call Wait() for the terminal pool error. A per-directory error lives inside its WalkResult and doesn't abort the rest of the traversal.

API

NewWalkman

func NewWalkman(followLinks bool, maxDepth uint32, skipList []string) *Walkman

Defaults: PoolSize = GOMAXPROCS, InitialWorkerCap = 32, ResultBuffSize = 64, stats disabled.

NewWalkmanWithConfig

func NewWalkmanWithConfig(followLinks bool, maxDepth uint32, skipList []string, trackStats bool, pc PoolConfig) *Walkman

type PoolConfig struct {
    PoolSize         int // number of workers
    InitialWorkerCap int // initial local-queue capacity per worker
    ResultBuffSize   int // result-channel buffer size
}

Walk

func (w *Walkman) Walk(root string) <-chan WalkResult

type WalkResult struct {
    Dir string
    Ret []fs.DirEntry // this directory's direct entries
    Err error
}

Child directories are scheduled as separate tasks, not included in Ret.

Wait

func (w *Walkman) Wait() error

Blocks until the pool finishes; returns the first fatal pool-level error.

Stats

func (w *Walkman) Stats() (files, dirs, links, skipped, maxDepthReached uint32)

Point-in-time snapshot (atomic, since tasks run concurrently). Zero unless trackStats is enabled.

Traversal semantics

Aspect	Behavior
Ordering	Completion-ordered, not path-sorted. Layer sorting/BFS on top of the stream if you need it.
Skip list	Matches entry names at every depth (e.g. .git, node_modules); a match prunes the whole subtree.
Max depth	0 = unlimited; otherwise the walker won't descend past that depth.
Symlinks	Not followed by default. When enabled, resolves via os.Stat and descends if the target is a directory. No cycle detection yet.
Errors	A directory that fails to open reports its error in its own WalkResult; other workers keep going.

How it works

Each task is intentionally small:

type walkItem struct {
    path  string
    depth uint32
}

For every directory: open it, read entries with File.ReadDir(-1) (skips the sort os.ReadDir does), drop skipped names, classify entries, optionally resolve symlinks, emit a WalkResult, and push child directories back onto the pool. Stats stay off by default to avoid atomic-counter overhead on the common path.

The pool itself lives in github.com/PAKIWASI/workstealpool.

CLI tools

A Go CLI (main/) and a Rust reference CLI (rust_walkdir/, wrapping the walkdir crate) share matching options for comparison.

go build -o build/main ./main
./build/main --workers 8 --skip .git,node_modules /home/me/project

--max-depth N      0 = unlimited
--follow-links     follow symlinks
--skip a,b,c       comma-separated names to prune
--print            print every entry
--bench N          repeat N times and report timing
--quiet            suppress the summary line
--workers N        worker-pool size; 0 = GOMAXPROCS

cd rust_walkdir && cargo build --release   # produces build/walkdir-cli

Benchmarks

Same deterministic wide tree (1,884 dirs, 11,304 files), Intel i5-1135G7, measured two ways.

CLI (hyperfine, full process per run):

Walker	Workers	Mean
Rust walkdir	seq	17.1 ms
walkman	1	12.4 ms
walkman	2	7.4 ms
walkman	4	5.4 ms
walkman	8	8.3 ms

In-process (go test -bench=WorkerSweep -count=10):

Workers	Mean
1	11.9 ms
2	6.9 ms
4	4.1 ms
8	3.2 ms

CLI timing plateaus/regresses past 4 workers (process overhead dominates); the pool itself keeps scaling to 8. Reproduce with:

./test/build_tree.sh --shape wide --root /tmp/tree_wide --seed 42
./test/bench_harness.sh --walkman ./build/main --walkdir ./build/walkdir-cli --tree /tmp/tree_wide --workers "1,2,4,8" --out test/results_wide.csv
WALKMAN_BENCH_ROOT=/tmp/tree_wide go test -bench=WorkerSweep -benchmem -count=10 ./...

Raw logs (including million-file runs) are in test/*.log.

Testing

go test ./...              # full suite
go test -race ./...        # race detector
go test -bench=. -benchmem ./...

Covers empty/flat directories, equivalence with filepath.WalkDir, skip-list pruning, max-depth, symlinks (following, broken links), permission errors, stats accuracy, consistency across worker counts, and worker parking/wake/termination behavior.

Limitations / roadmap

Deliberately deferred so far:

• Deterministic/sorted or breadth-first output
• iter.Seq-based iteration, FilterEntry-style filtering, ContentsFirst ordering
• A cap on simultaneously open directories
• Global symlink-cycle detection
• fs.FS-based traversal

Ordering/scheduling policy is kept out of the core walker on purpose: sorted or BFS output needs buffering, which belongs above the streaming core, not inside it.

Project layout

walkman/
├── walkman.go                      # core concurrent walker
├── walkman_test.go                 # correctness + concurrency tests
├── walkman_bench_test.go           # benchmark suite
├── walkman_worker_sweep_test.go    # WorkerSweep benchmark
├── main/main.go                    # Go CLI / benchmark wrapper
├── rust_walkdir/src/main.rs        # comparable Rust walkdir wrapper
├── build/build_tree.sh             # simple synthetic-tree generator
├── test/                           # benchmark harness + recorded logs
└── help.md                         # project notes / implementation plan

License

MIT

Documentation

Overview

Package walkman

Index

• type PoolConfig
  • func DefaultPoolConfig() PoolConfig
• type WalkResult
• type Walkman
  • func NewWalkman(followLinks bool, maxDepth uint32, skipList []string) *Walkman
  • func NewWalkmanWithConfig(followLinks bool, maxDepth uint32, skipList []string, trackStats bool, ...) *Walkman
  • func (w *Walkman) Stats() (files, dirs, links, skipped, maxDepthReached uint32)
  • func (w *Walkman) Wait() error
  • func (w *Walkman) Walk(root string) <-chan WalkResult

Types

type PoolConfig

type PoolConfig struct {
	PoolSize         int
	InitialWorkerCap int
	ResultBuffSize   int
}

PoolConfig exposes the underlying worker-pool knobs

func DefaultPoolConfig

func DefaultPoolConfig() PoolConfig

DefaultPoolConfig matches PoolSize to GOMAXPROCS. Per the workstealpool README's own benchmarks: speedup on a CPU-bound divide-and-conquer workload tracks physical/logical core count and then plateaus right at GOMAXPROCS, with a slight regression going meaningfully past it (oversubscription). InitialWorkerCap/ResultBuffSize matter far less (a few hundred µs across their whole sweep) — 32/64 here just matches the fixed baseline used across their other sweeps, not a walking-specific finding

type WalkResult

type WalkResult struct {
	Dir string
	Ret []fs.DirEntry
	Err error
}

type Walkman

type Walkman struct {
	// contains filtered or unexported fields
}

func NewWalkman

func NewWalkman(followLinks bool, maxDepth uint32, skipList []string) *Walkman

NewWalkman builds a Walkman with GOMAXPROCS-based pool sizing and stats tracking off. Use NewWalkmanWithConfig for explicit pool sizing or to turn stats on.

func NewWalkmanWithConfig

func NewWalkmanWithConfig(followLinks bool, maxDepth uint32, skipList []string, trackStats bool, pc PoolConfig) *Walkman

NewWalkmanWithConfig is NewWalkman but with every knob explicit: whether to track walkStats (costs an atomic.Add per entry when on), and pool sizing, for callers who've measured what suits their workload rather than accepting the defaults.

func (*Walkman) Stats

func (w *Walkman) Stats() (files, dirs, links, skipped, maxDepthReached uint32)

Stats returns a point-in-time snapshot of walk statistics. Always zero unless trackStats was enabled via NewWalkmanWithConfig. Safe to call while a walk is in progress, though the numbers will still be moving.

func (*Walkman) Wait

func (w *Walkman) Wait() error

Wait blocks until the walk has fully finished and returns the first fatal error encountered (nil on normal completion). Call it after draining the channel returned by Walk

func (*Walkman) Walk

func (w *Walkman) Walk(root string) <-chan WalkResult

Walk starts walking root and returns a channel of per-directory results. The channel closes once every worker has finished (no work left, or a fatal error occurred). Call Wait after draining the channel to get the terminal error, if any.