godirwalk

README

godirwalk

godirwalk is a library for walking a directory tree on a file system. It runs on unix like operating systems and Windows. When compared against filepath.Walk in benchmarks, it runs up to ten times the speed on unix, and about four times the speed on Windows.

More importantly, unlike when using filepath.Walk on Windows, it doesn't result in an infinite loop when walking a directory tree hierarchy that contains a symbolic link loop.

Description

The Go standard library provides a flexible function for traversing a file system directory tree. However it invokes os.Stat on every file system node encountered in order to provide the file info data structure, i.e. os.FileInfo, to the upstream client. For many uses the os.Stat is needless and adversely impacts performance. Many clients need to branch based on file system node type, and, that information is already provided by the system call used to read a directory's children nodes.

There are two major differences and one minor difference to the operation of filepath.Walk and the directory traversal algorithm in this library.

First, filepath.Walk invokes the callback function with a slashed version of the pathname regardless of the os-specific path separator, while godirwalk invokes callback function with the os-specific pathname separator.

Second, while filepath.Walk invokes callback function with the os.FileInfo for every node, this library invokes the callback function with the os.FileMode set to the type of file system node it is, namely, by masking the file system mode with os.ModeType. It does this because this eliminates the need to invoke os.Stat on every file system node. On the occassion that the callback function needs the full stat information, it can call os.Stat when required.

Third, since this library does not invoke os.Stat on every node, there is no possible error event for the callback function to filter on. The third argument in the callback function signature for the stat error is no longer necessary.

Usage

Documentation is available via .

package main

import (
	"fmt"
	"os"
	"path/filepath"

	"github.com/karrick/godirwalk"
)

func main() {
	dirname := "."
	if len(os.Args) > 1 {
		dirname = os.Args[1]
	}
	err := godirwalk.Walk(dirname, callback)
	if err != nil {
		fmt.Fprintf(os.Stderr, "%s", err)
		os.Exit(1)
	}
}

func callback(osPathname string, mode os.FileMode) error {
	fmt.Printf("%s %s\n", mode, osPathname)
	return nil
}

Documentation

Index

• func ReadDirnames(osDirname string, max int) ([]string, error)
• func Walk(pathname string, walkFn WalkFunc) error
• func WalkFollowSymlinks(pathname string, walkFn WalkFunc) error
• type Dirent
• type Dirents
  • func ReadDirents(osDirname string, max int) (Dirents, error)
  • func (l Dirents) Len() int
  • func (l Dirents) Less(i, j int) bool
  • func (l Dirents) Swap(i, j int)
• type WalkFunc

Functions

func ReadDirnames

func ReadDirnames(osDirname string, max int) ([]string, error)

ReadDirnames returns a slice of strings, representing the file system children of the specified directory. If the specified directory is a symbolic link, it will be resolved.

children, err := godirwalk.ReadDirnames(osPathname, 0)
if err != nil {
	return nil, errors.Wrap(err, "cannot get list of directory children")
}
sort.Strings(children)
for _, child := range children {
    fmt.Printf("%s\n", child)
}

func Walk

func Walk(pathname string, walkFn WalkFunc) error

Walk walks the file tree rooted at the specified directory, calling the specified callback function for each file system node in the tree, including root, symbolic links, and other node types. The nodes are walked in lexical order, which makes the output deterministic but means that for very large directories this function can be inefficient.

This function is often much faster than filepath.Walk because it does not invoke os.Stat for every node it encounters, but rather obtains the file system node type when it reads the parent directory.

func main() {
	dirname := "."
	if len(os.Args) > 1 {
		dirname = os.Args[1]
	}
	if err := godirwalk.Walk(dirname, callback); err != nil {
		fmt.Fprintf(os.Stderr, "%s\n", err)
		os.Exit(1)
	}
}

func callback(osPathname string, mode os.FileMode) error {
	fmt.Printf("%s %s\n", mode, osPathname)
	return nil
}

func WalkFollowSymlinks

func WalkFollowSymlinks(pathname string, walkFn WalkFunc) error

WalkFollowSymlinks walks the file tree rooted at the specified directory, calling the specified callback function for each file system node in the tree, including root, symbolic links, and other node types. The nodes are walked in lexical order, which makes the output deterministic but means that for very large directories this function can be inefficient.

This function is often much faster than filepath.Walk because it does not invoke os.Stat every node it encounters, but rather obtains the file system node type when it reads the parent directory.

This function also follows symbolic links that point to directories, and therefore ought to be used with caution, as calling it may cause an infinite loop in cases where the file system includes a logical loop of symbolic links.

Types

type Dirent

type Dirent struct {
	// Name is the filename of the file system entry, relative to its parent.
	Name string

	// ModeType is the mode bits that specify the file system entry type. We
	// could make our own enum-like data type for encoding the file type, but
	// Go's runtime already gives us architecture independent file modes, as
	// discussed in `os/types.go`:
	//
	//    Go's runtime FileMode type has same definition on all systems, so that
	//    information about files can be moved from one system to another
	//    portably.
	ModeType os.FileMode
}

Dirent stores the name and file system mode type of discovered file system entries.

type Dirents

type Dirents []*Dirent

Dirents represents a slice of Dirent pointers, which are sortable by name. This type satisfies the `sort.Interface` interface.

func ReadDirents

func ReadDirents(osDirname string, max int) (Dirents, error)

ReadDirents returns a slice of pointers to Dirent structures, representing the file system children of the specified directory. If the specified directory is a symbolic link, it will be resolved.

children, err := godirwalk.ReadDirents(osPathname, 0)
if err != nil {
	return nil, errors.Wrap(err, "cannot get list of directory children")
}
sort.Sort(children)
for _, child := range children {
    fmt.Printf("%s %s\n", child.ModeType, child.Name)
}

func (Dirents) Len

func (l Dirents) Len() int

Len returns the count of Dirent structures in the slice.

func (Dirents) Less

func (l Dirents) Less(i, j int) bool

Less returns true if and only if the Name of the element specified by the first index is lexicographically less than that of the second index.

func (Dirents) Swap

func (l Dirents) Swap(i, j int)

Swap exchanges the two Dirent entries specified by the two provided indexes.

type WalkFunc

type WalkFunc func(osPathname string, mode os.FileMode) error

WalkFunc is the type of the function called for each file system node visited by Walk. The path argument contains the argument to Walk as a prefix; that is, if Walk is called with "dir", which is a directory containing the file "a", the provided WalkFunc will be invoked with the argument "dir/a", using the correct os.PathSeparator for the Go Operating System architecture, GOOS. The mode argument is the os.FileMode for the named path, masked to the bits that identify the file system node type, i.e., os.ModeType.

If an error is returned by the walk function, processing stops. The sole exception is when the function returns the special value filepath.SkipDir. If the function returns filepath.SkipDir when invoked on a directory, Walk skips the directory's contents entirely. If the function returns filepath.SkipDir when invoked on a non-directory file system node, Walk skips the remaining files in the containing directory.