Back to godoc.org
github.com/dixonwille/skywalker

package skywalker

v0.0.0 (0769845)
Latest Go to latest
Published: Aug 4, 2017 | License: BSD-3-Clause | Module: github.com/dixonwille/skywalker

Overview

Package skywalker walks through a filesystem concurrently.

Index

Examples

type ListType

type ListType int

ListType is used to specify how to handle the contents of a list

const (
	//LTBlacklist is used to specify that a list is to exclude the contents.
	LTBlacklist ListType = iota
	//LTWhitelist is used to specify that a list is to include the contents.
	LTWhitelist
)

type Skywalker

type Skywalker struct {
	//Root is where the file walker starts. It is converted to an absolute path before start.
	Root string

	//List and ListType should only be used for fine filtering of paths.
	//It uses https://github.com/gobwas/glob for glob checking on each patch check.
	ListType ListType
	List     []string

	//ExtList and ExtListType are used to narrow down the files by their extensions.
	//Make sure to include the preceding ".".
	ExtListType ListType
	ExtList     []string

	//DirList and DirListType are used to narrow down by directories.
	//Will skip the appropriate directories and their files/subfolders.
	DirListType ListType
	DirList     []string

	//NumWorkers are how many workers are listening to the queue to do the work.
	NumWorkers int

	//QueueSize is how many paths to queue up at a time.
	//Useful for fine control over memory usage if needed.
	QueueSize int

	//Worker is the function that is called on each file/directory.
	Worker Worker

	//FilesOnly should be set to true if you only want to queue up files.
	FilesOnly bool
	// contains filtered or unexported fields
}

Skywalker can concurrently go through files in Root and call Worker on everything. It is recommended to use DirList and ExtList as much as possible as it is more perfomant than List.

Example

Code:

package skywalker_test

import (
	"fmt"
	"sort"
	"strings"
	"sync"

	"github.com/dixonwille/skywalker"
)

type ExampleWorker struct {
	*sync.Mutex
	found []string
}

func (ew *ExampleWorker) Work(path string) {
	//This is where the necessary work should be done.
	//This will get concurrently so make sure it is thread safe if you need info across threads.
	ew.Lock()
	defer ew.Unlock()
	ew.found = append(ew.found, path)
}

func ExampleSkywalker() {
	//Following two functions are only to create and destroy data for the example
	defer teardownData()
	standupData()

	ew := new(ExampleWorker)
	ew.Mutex = new(sync.Mutex)

	//root is the root directory of the data that was stood up above
	sw := skywalker.New(root, ew)
	sw.DirListType = skywalker.LTBlacklist
	sw.DirList = []string{"sub"}
	sw.ExtListType = skywalker.LTWhitelist
	sw.ExtList = []string{".pdf"}
	err := sw.Walk()
	if err != nil {
		fmt.Println(err)
		return
	}
	sort.Sort(sort.StringSlice(ew.found))
	for _, f := range ew.found {
		show := strings.Replace(f, sw.Root, "", 1)
		show = strings.Replace(show, "\\", "/", -1)
		fmt.Println(show)
	}
	// Output:
	// /subfolder/few.pdf
	// /the/few.pdf
}

Output:

/subfolder/few.pdf
/the/few.pdf

func New

func New(root string, worker Worker) *Skywalker

New creates a new Skywalker that can walk through the specified root and calls the Worker on each file and/or directory. Defaults Skywalker to have 20 workers, a QueueSize of 100 and only queue files.

func (*Skywalker) Walk

func (sw *Skywalker) Walk() error

Walk goes through the files and folders in Root and calls the worker on each. Checks the lists specified to check whether it should ignore files or directories. It also handles the creation of workers and queues needed for walking.

type Worker

type Worker interface {
	Work(path string)
}

Worker is anything that knows what to do with a path.

Documentation was rendered with GOOS=linux and GOARCH=amd64.

Jump to identifier

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to identifier