powerwalk

README

Powerwalk

Go package for walking files and concurrently calling user code to handle each file. This package walks the file system in the same way filepath.Walk does, except instead of calling the walkFn inline, it uses goroutines to allow the files to be handled concurrently.

Powerwalk functions by walking concurrently over many files. In order to realize any benefits from this approach, you must tell the runtime to use multiple CPUs. For example:

runtime.GOMAXPROCS(runtime.NumCPU())

Usage

Powerwalk is a drop-in replacement for the filepath.Walk method (read about that for more details), and so has the same signature, even using the filepath.WalkFunc too.

powerwalk.Walk(root string, walkFn filepath.WalkFunc) error

By default, Powerwalk will call the walkFn for powerwalk.DefaultConcurrentWalks (currently 100) files at a time. To be specific about the number of concurrent files to walk, use the WalkLimit alternative.

powerwalk.WalkLimit(root string, walkFn filepath.WalkFunc, limit int) error

The WalkLimit function does the same as Walk, except allows you to specify the number of files to concurrently walk using the limit argument. The limit argument must be one or higher (i.e. >0). Specificying a limit that's too high, causes unnecessary overhead so sensible numbers are encouraged but not enforced.

See the godoc documentation for more information.

Documentation

Overview

Package powerwalk concurrently walks file trees. Aside from SkipDir functionality not working and the fact that the WalkFunc gets run concurrently, this is a drop-in replacement for filepath.Walk.

Index

• Constants
• func Walk(root string, walkFn filepath.WalkFunc) error
• func WalkLimit(root string, walkFn filepath.WalkFunc, limit int) error

Constants

const DefaultConcurrentWalks int = 100

DefaultConcurrentWalks is the default number of files that will be walked at the same time when the Walk function is called. To use a value other than this one, use the WalkLimit function.

Functions

func Walk

func Walk(root string, walkFn filepath.WalkFunc) error

Walk walks the file tree rooted at root, calling walkFn for each file or directory in the tree, including root. All errors that arise visiting files and directories are filtered by walkFn. The output is non-deterministic. WalkLimit does not follow symbolic links.

For each file and directory encountered, Walk will trigger a new Go routine allowing you to handle each item concurrently. A maximum of DefaultConcurrentWalks walkFns will be called at any one time.

func WalkLimit

func WalkLimit(root string, walkFn filepath.WalkFunc, limit int) error

WalkLimit walks the file tree rooted at root, calling walkFn for each file or directory in the tree, including root. All errors that arise visiting files and directories are filtered by walkFn. The output is non-deterministic. WalkLimit does not follow symbolic links.

For each file and directory encountered, Walk will trigger a new Go routine allowing you to handle each item concurrently. A maximum of limit walkFns will be called at any one time.