util

Documentation

Index

• func ContentHash(files []string) ([]byte, error)
• func MaybeClose(writer io.Writer)
• func Pformat(value interface{}) string
• func TransformFiles(files []string, inputPattern string, outputPattern string) (map[string]string, error)
• type ArrayLister
  • func (l *ArrayLister) List() ([]string, error)
• type FileLister
• type ReMatch
  • func FilterByRe(l StringLister, p *regexp.Regexp) ([]ReMatch, error)
• type StringLister

Functions

func ContentHash

func ContentHash(files []string) ([]byte, error)

FilesHash generates a hash based on the contents of a list of files. This is intended to be used to detect when one or more files has changed.

func MaybeClose

func MaybeClose(writer io.Writer)

MaybeClose will close the writer if its a Closer. Intended to be used with calls to defer.

func Pformat

func Pformat(value interface{}) string

Pformat returns a pretty format output of any value.

func TransformFiles

func TransformFiles(files []string, inputPattern string, outputPattern string) (map[string]string, error)

TransformFiles matches the input pattern against the input files. For any matching file the corresponding output is generated according to outputPattern.

inputPattern is a regex that should contain named groups using the syntax (?P<name>regex)

For example: inputPattern: "gs://someBucket/(?p<name>.*)\.pdf"

The outputPattern can reference the named patterns using the syntax {{.name}} e.g.

outputPattern: "gs://outputBucket/{{.name}}.csv" regex groups to capture groups. The outputPattern is a go template that uses {{.g1}, {{.g2}}, ..., {{.gn}} to refer to the captured groups

Types

type ArrayLister

type ArrayLister struct {
	Items []string
}

ArrayLister implements the Lister interface for an array.

func (*ArrayLister) List

func (l *ArrayLister) List() ([]string, error)

List lists the items in the array

type FileLister

type FileLister interface {
	ListByRe(pattern string) ([]ReMatch, error)
}

FileLister is an interface intended to transparently handle working with GCS and local files.

type ReMatch

type ReMatch struct {
	// Value is the matched value
	Value string

	// Groups is a map of the named groups if any captured from the regex
	Groups map[string]string
}

ReMatch represents a regex match

func FilterByRe

func FilterByRe(l StringLister, p *regexp.Regexp) ([]ReMatch, error)

type StringLister

type StringLister interface {
	List() ([]string, error)
}

StringLister is an interface to return a list of strings