strit

package module
Version: v0.0.0-...-93a3cac Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 22, 2021 License: BSD-3-Clause Imports: 10 Imported by: 1

README

strit

GoDoc Go Report Card

Package strit (STRing ITerator) assists in development of string processing pipelines by providing a simple iteration model that allows for easy composition of processing stages.

Motivation

Suppose we want to develop a function that reads a file line by line, removes leading and trailing whitespace from each line, selects only non-empty lines that also do not start with the # symbol, and stores those lines in a slice of strings. Using the Go standard library one possible implementation of the function may look like this:

func ReadConfig(fileName string) ([]string, error) {
	file, err := os.Open(fileName)

	if err != nil {
		return nil, err
	}

	defer file.Close()

	var res []string
	src := bufio.NewScanner(file)

	for src.Scan() {
		line := bytes.TrimSpace(src.Bytes())

		if len(line) > 0 && line[0] != '#' {
			res = append(res, string(line))
		}
	}

	if err = src.Err(); err != nil {
		return nil, err
	}

	return res, nil
}

Using strit package the implementation can be simplified down to:

func ReadConfig(fileName string) ([]string, error) {
	return strit.FromFile(fileName).
           Map(bytes.TrimSpace).
           Filter(strit.Not(strit.Empty).AndNot(strit.StartsWith("#"))).
           Strings()
Features
More examples:
  • Naïve grep:
func main() {
	_, err := strit.FromReader(os.Stdin).
			Filter(regexp.MustCompile(os.Args[1]).Match).
			WriteSepTo(os.Stdout, "\n")

	if err != nil {
		os.Stderr.WriteString(err.Error() + "\n")
		os.Exit(1)
	}
}
  • Recursively find all the filesystem entries matching the given regular expression:
func selectEntries(root string, re *regexp.Regexp) ([]string, error) {
	return FromDirWalk(root, nil).Filter(re.Match).Strings()
}
  • Build a list of .flac files in the given directory, annotating each name with its corresponding track number from FLAC metadata:
func namesWithTrackNumbers(dir string) ([]string, error) {
	return strit.FromDir(dir, func(info os.FileInfo) bool { return info.Mode().IsRegular() }).
		Filter(strit.EndsWith(".flac")).
		GenMap(prependTrackNo).
		Strings()
}

func prependTrackNo(file []byte) ([]byte, error) {
	name := string(file)

	no, err := strit.FromCommand(exec.Command("metaflac", "--list", "--block-type=VORBIS_COMMENT", name)).
		FirstNonEmpty(func(s []byte) []byte {
			if m := match(s); len(m) == 2 {
				return m[1]
			}

			return nil
		}).
		String()

	if err != nil {
		return nil, err
	}

	if len(no) == 0 {
		return []byte("???: " + filepath.Base(name)), nil
	}

	return []byte(no + ": " + filepath.Base(name)), nil
}

var match = regexp.MustCompile(`tracknumber=([[:digit:]]+)$`).FindSubmatch
Project status

The project is in a beta state. Tested on Linux Mint 19.1, with Go version 1.12. Should also work on other platforms supported by Go runtime, but currently this is not very well tested.

License: BSD

Documentation

Overview

Package strit introduces a string iterator of "push" type, as well as a number of iterator constructors and wrappers.

Typically, iteration over some source of data involves creating an object of some reader type, then repeatedly calling a method on the object to get the data elements, and finally calling some method to release the resources allocated inside the reader object. And in the languages without exceptions (like Go) there must be an error check on each step. In other words, there is an interface and some protocol all the clients must follow, which usually results in certain amount of boilerplate code to write.

The above example describes the so called "pull" model of iteration where the client has to call a method on the reader to "pull" the next data element out of the reader object. Alternative to that is the "push" model where client gives the iterator a callback function to be invoked once per each element of input data.

Another aspect of the iteration model is about dealing with errors. In the absence of exceptions, like in Go language, the "pull" model implies that the result of each step of the iteration is either a data item, or an error. The same is true for pure "push" model where each invocation of the callback has to receive two parameters: a data element and an error. Or there maybe two callbacks, one for data and another for errors, but this is conceptually the same.

This library is an experiment in combining both "push" and "pull" models: it utilises "push" model for data elements, and "pull" model for error propagation. The data elements here are strings only, as currently the Go language does not offer any generic programming mechanism. The iterator type itself is a function that invokes the supplied callback once per each input string, and it returns either an error or nil. This arrangement reduces the amount of boilerplate code, at least in the most common scenarios, plus it allows for easy iterator combination, especially useful where dynamic creation of string processing pipelines is required.

The string iterator type used throughout this library is simply a function of the type func(Func) error, where Func is the type of the callback, func([]byte) error. Strings are represented as byte slices for performance reasons, to allow for in-place modifications where possible, like in bytes.TrimSpace() function. Iteration is simply an invocation of the iterator function with a callback to receive strings. The iterator function returns whatever error may have occurred during the iteration. The callback function itself may also return an error which will stop the iteration and will be returned from the iterator function. The special error value of io.EOF is treated as a request to stop the iteration and will not be propagated up to the caller.

The library offers a number of iterator constructors making iterators from different sources like strings, byte slices, byte readers, files and shell command outputs. There is also a number of mapping and filtering functions, as well as writers for strings, byte slices and files. All the above is composable using fluent API. For example, reading a file line by line, removing leading and trailing whitespace from each line, selecting only non-empty lines that also do not start from the character '#', and storing the result in a slice of strings, may look like this:

lines, err := strit.FromFile("some-file").
              Map(bytes.TrimSpace).
              Filter(strit.Not(strit.Empty).OrNot(strit.StartsWith("#"))).
              Strings()

Here strit.FromFile is an iterator constructor, Map and Filter operations are iterator wrappers each creating a new iterator from the existing one, and Strings() is the iterator invocation. Also notice the use of the provided predicate combinators in the parameter to the Filter() function.

Given that each iterator is itself a function, other iterators can be derived from the existing ones. For example, the following function takes an iterator and creates a new iterator that prepends the line number to each line from the original iterator:

func Numbered(iter Iter) Iter {
    return func(fn Func) error { // this is the new iterator function
        i := 0

        // this is the invocation of the original iterator function
        return iter(func(line []byte) error {
            i++
            line = []byte(fmt.Sprintf("%d %s", i, string(line)))
            return fn(line)	// invocation of the callback
        })
    }
}

Iterators in this library are lazy, meaning that the actual iteration happens only when the iterator function is called, not when it is created. Some of the provided iterators are reusable, meaning that such iterator may be called more than once, but in general this property cannot be guaranteed for those sources that cannot be read multiple times (like io.Reader), so in general it is advised that in the absence of any specific knowledge every iterator should be treated as not reusable. All the iterator implementations have O(n) complexity in time and O(1) in space.

One disadvantage of the suggested model is the complex implementation of parallel composition of iterators, see the comment to the Merge() function for more details. Also, there is a certain cost of abstraction incurred, in other words, a straight-forward implementation of the above example could be somewhat more efficient in terms of CPU cycles and memory consumption, though the difference would mostly come from the extra function calls and as such would probably be not so substantial, especially for i/o-bound sources. The actual benchmaking of some simple processing scenarios shows that the performance difference is truly minor.

Index

Constants

This section is empty.

Variables

View Source
var ErrSkip = errors.New("Item skipped")

ErrSkip is a special error type to skip an item in GenMap() function.

Functions

func Empty

func Empty(line []byte) bool

Empty is a predicate that returns 'true' only if the input string is empty.

func ScanNullTerminatedLines

func ScanNullTerminatedLines(data []byte, atEOF bool) (advance int, token []byte, err error)

ScanNullTerminatedLines is a split function that splits input on null bytes. Useful mostly with FromCommandSF function, in cases where the invoked command generates null-terminated strings, like 'find ... -print0'.

Types

type ExitError

type ExitError struct {
	ExitCode int
	Stderr   string
}

ExitError is the error type used for delivering command exit code and 'stderr' output.

func (*ExitError) Error

func (e *ExitError) Error() string

Error formats error message from ExitError type.

type Func

type Func func([]byte) error

Func is the type of callback function used by Iter.

type Iter

type Iter func(Func) error

Iter is the iterator type.

func Chain

func Chain(its ...Iter) Iter

Chain implements sequential composition of its input iterators. The returned iterator invokes all the input iterators one after another, from left to the right.

func FromBytes

func FromBytes(src []byte) Iter

FromBytes constructs a new iterator that reads the specified byte slice and breaks it into lines with line termination stripped.

func FromBytesSF

func FromBytesSF(sf bufio.SplitFunc, src []byte) Iter

FromBytesSF constructs a new iterator that reads the specified byte slice and breaks it into tokens using the supplied split function. If the function is set to nil then the iterator breaks the input into lines with line termination stripped. Internally the iterator is implemented using bufio.Scanner, please refer to its documentation for more details on split functions.

func FromCommand

func FromCommand(cmd *exec.Cmd) Iter

FromCommand constructs a new iterator that invokes the specified command, reads the command output (stdout) and breaks it into lines with line termination stripped.

func FromCommandSF

func FromCommandSF(cmd *exec.Cmd, sf bufio.SplitFunc) Iter

FromCommandSF constructs a new iterator that invokes the specified command, reads the command output (stdout) and breaks it into tokens using the supplied split function. If the split function is set to nil then the iterator breaks the command output into lines with line termination stripped. Internally the iterator is implemented using bufio.Scanner, please refer to its documentation for more details on split functions.

func FromDir

func FromDir(name string, pred func(os.FileInfo) bool) Iter

FromDir creates a new iterator that reads all entries from the specified directory and produces the names of those entries for which the supplied predicate returns 'true'. If the predicate is set to nil then the iterator produces only regular files, directories and symlinks to files.

func FromDirWalk

func FromDirWalk(root string, wf filepath.WalkFunc) Iter

FromDirWalk creates a new iterator that wraps filepath.Walk recursive directory traversal function. The iterator produces names of filesystem entries in accordance with the supplied WalkFunc, please refer to the filepath.Walk documentation for more details. If the WalkFunc is set to nil then the default function will be used which accepts all the filesystem entries.

func FromFile

func FromFile(name string) Iter

FromFile constructs a new iterator that reads its input byte stream from the specified file and breaks the stream into lines with line termination stripped.

func FromFileSF

func FromFileSF(sf bufio.SplitFunc, name string) Iter

FromFileSF constructs a new iterator that reads its input byte stream from the specified file and breaks the stream into tokens using the supplied split function. If the function is set to nil then the iterator breaks the input into lines with line termination stripped. Internally the iterator is implemented using bufio.Scanner, please refer to its documentation for more details on split functions.

func FromReadCloser

func FromReadCloser(input io.ReadCloser) Iter

FromReadCloser constructs a new iterator that reads its input byte stream from the specified Reader and breaks the input into lines with line termination stripped. The input Reader gets closed at the end of the iteration.

func FromReadCloserSF

func FromReadCloserSF(sf bufio.SplitFunc, input io.ReadCloser) Iter

FromReadCloserSF is a wrapper around FromReaderSF with exactly the same functionality that also closes the input Reader at the end of the iteration.

func FromReader

func FromReader(input io.Reader) Iter

FromReader constructs a new iterator that reads its input byte stream from the specified Reader and breaks the input into lines with line termination stripped.

func FromReaderSF

func FromReaderSF(sf bufio.SplitFunc, input io.Reader) Iter

FromReaderSF constructs a new iterator that reads its input byte stream from the specified Reader and breaks the stream into tokens using the supplied split function. If the function is set to nil then the iterator breaks the input into lines with line terminators stripped. Internally the iterator is implemented using bufio.Scanner, please refer to its documentation for more details on split functions.

func FromString

func FromString(src string) Iter

FromString constructs a new iterator that reads the specified string and breaks it into lines with line termination stripped.

func FromStringSF

func FromStringSF(sf bufio.SplitFunc, src string) Iter

FromStringSF constructs a new iterator that reads the specified string and breaks it into tokens using the supplied split function. If the function is set to nil then the iterator breaks the input into lines with line termination stripped. Internally the iterator is implemented using bufio.Scanner, please refer to its documentation for more details on split functions.

func FromStrings

func FromStrings(src []string) Iter

FromStrings constructs a new iterator that reads the specified slice of strings, one string at a time.

func Merge

func Merge(sep string, its ...Iter) Iter

Merge implements parallel composition of its input iterators. The returned iterator on each step invokes all its inputs in parallel, taking one string from each input iterator, and then joins those strings around the specified separator to produce one output string. The iteration stops when any input iterator gets exhausted. It should be noted that the iterator type used in this library provides for easy sequential composition of iterators, but the parallel composition is substantially more complex. In the current implementation each input iterator except the first one runs in a dedicated goroutine which pipes its output back through a channel. That complexity is usually not a problem if at least one of the input iterators is i/o-bound, but it is likely to become a performance bottleneck if all the iterators read from the memory. Please use this function responsibly.

func (Iter) Bytes

func (iter Iter) Bytes() (res []byte, err error)

Bytes invokes the iterator and concatenates its output into one byte slice.

func (Iter) Filter

func (iter Iter) Filter(pred Pred) Iter

Filter makes a new iterator that produces only the strings for which the supplied predicate returns 'true'.

func (Iter) FirstNonEmpty

func (iter Iter) FirstNonEmpty(mapper func([]byte) []byte) Iter

FirstNonEmpty makes a new iterator that skips everything until the supplied mapper function returns a non-empty slice, which gets passed down the pipeline, and then the iteration stops. This is essentially a search for the first non-empty string returned from the mapper.

func (Iter) GenMap

func (iter Iter) GenMap(mapper func([]byte) ([]byte, error)) Iter

GenMap makes a new iterator that applies the supplied function to every input string. The function may return a non-nil error, in which case the iteration will stop and the error will be propagated back to the source. A special error value of ErrSkip instructs the iterator to simply skip the current string.

func (Iter) Join

func (iter Iter) Join(sep string) (res string, err error)

Join invokes the iterator and collects its output into one string, delimited by the specified separator.

func (Iter) JoinBytes

func (iter Iter) JoinBytes(sep string) (res []byte, err error)

JoinBytes invokes the iterator and collects its output into one byte slice, delimited by the specified separator.

func (Iter) Map

func (iter Iter) Map(mapper func([]byte) []byte) Iter

Map makes a new iterator that applies the specified function to every input string.

func (Iter) Parse

func (iter Iter) Parse(p Parser) error

Parse feeds the supplied parser from the given iterator.

func (Iter) Pipe

func (iter Iter) Pipe(cmd *exec.Cmd) Iter

Pipe makes an iterator that pumps the data from its parent through the specified command and iterates over the command's stdout.

func (Iter) PipeSF

func (iter Iter) PipeSF(cmd *exec.Cmd, sf bufio.SplitFunc) Iter

PipeSF makes an iterator that pumps the data from its parent through the specified command and iterates over the command's stdout, using the given splitter to separate strings.

func (Iter) Skip

func (iter Iter) Skip(numLines uint64) Iter

Skip makes a new iterator that skips the specified number of input strings and passes all the remaining strings to the callback function.

func (Iter) SkipWhile

func (iter Iter) SkipWhile(pred Pred) Iter

SkipWhile makes a new iterator that skips input strings until the supplied predicate returns 'false' for the first time and passes the remaining strings to the callback function.

func (Iter) String

func (iter Iter) String() (res string, err error)

String invokes the iterator and concatenates its output into one string.

func (Iter) Strings

func (iter Iter) Strings() (res []string, err error)

Strings invokes the iterator and collects its output into a slice of strings.

func (Iter) Take

func (iter Iter) Take(numLines uint64) Iter

Take makes a new iterator that produces no more than the specified number of output strings.

func (Iter) TakeWhile

func (iter Iter) TakeWhile(pred Pred) Iter

TakeWhile makes a new iterator that produces its output while the specified predicate returns 'true'.

func (Iter) WriteSepTo

func (iter Iter) WriteSepTo(dest io.Writer, sep string) (n int64, err error)

WriteSepTo invokes the iterator and writes its output strings, delimited by the specified separator, into the specified Writer. The method returns the total number of bytes written, or an error.

func (Iter) WriteSepToFile

func (iter Iter) WriteSepToFile(name, sep string) (n int64, err error)

WriteSepToFile creates a file with the specified name, and then invokes the iterator and writes its output strings, delimited by the specified separator, into the file. The method returns the total number of bytes written, or an error, in which case the resulting file gets deleted.

func (Iter) WriteTo

func (iter Iter) WriteTo(dest io.Writer) (int64, error)

WriteTo invokes the iterator and writes its output strings, delimited by new line character, into the specified Writer. The method returns the total number of bytes written, or an error.

func (Iter) WriteToFile

func (iter Iter) WriteToFile(name string) (int64, error)

WriteToFile creates a file with the specified name, and then invokes the iterator and writes its output strings, delimited by new line character, into the file. The method returns the total number of bytes written, or an error, in which case the resulting file gets deleted.

type Parser

type Parser interface {
	// Enter is the entry point of the parser and gets called on the first iteration.
	// The returned function usually refers to another method of the same class, thus implementing
	// the parser state machine. Any non-nil error stops the iteration.
	Enter([]byte) (ParserFunc, error)

	// Done is called after the iteration is complete or an error has occurred. The error
	// it returns (if any) becomes the return value of the iterator.
	Done(error) error
}

Parser is the interface of a line parser.

type ParserFunc

type ParserFunc func([]byte) (ParserFunc, error)

ParserFunc represents class of functions implementing parser state machines.

type Pred

type Pred func([]byte) bool

Pred is the type of string predicate. The type has a number of combining methods allowing for convenient composition of predicate functions, for example:

strit.Not(strit.Empty).AndNot(strit.StartsWith("#"))

or

strit.StartsWith("xyz").Or(strit.StartsWith("abc"))

func EndsWith

func EndsWith(prefix string) Pred

EndsWith is a predicate that returns 'true' if the input string has the specified suffix.

func Not

func Not(pred Pred) Pred

Not is a predicate wrapper that returns a new predicate negating the result of the supplied predicate.

func StartsWith

func StartsWith(prefix string) Pred

StartsWith is a predicate that returns 'true' if the input string has the specified prefix.

func (Pred) And

func (orig Pred) And(other Pred) Pred

And is a predicate combinator. It creates a new predicate that applies logical 'and' to the original and the supplied predicates.

func (Pred) AndNot

func (orig Pred) AndNot(next Pred) Pred

AndNot is a predicate combinator. It creates a new predicate that returns 'true' only if the original returns 'true' and the other predicate returns 'false'.

func (Pred) Or

func (orig Pred) Or(other Pred) Pred

Or is a predicate combinator. It creates a new predicate that applies logical 'or' to the original and the supplied predicates.

func (Pred) OrNot

func (orig Pred) OrNot(other Pred) Pred

OrNot is a predicate combinator. It creates a new predicate that returns 'true' if either the original returns 'true' or the other predicate returns 'false'.

Source Files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL