spool

Documentation

Index

• Variables
• type File
• type FileSystem
• type Spool
  • func New(reader, writer FileSystem, opt ...opts.Option[Spool]) *Spool
  • func (spool *Spool) ForEach(ctx context.Context, walker Walker, f Spooler) error
  • func (spool *Spool) Shard(ctx context.Context, dir string, ...) error
  • func (spool *Spool) Write(path string, r io.Reader) error
  • func (spool *Spool) WriteFile(path string, b []byte) error
• type Spooler
• type Walker
  • func WalkDir(dir string) Walker
  • func WalkFiles(files []string) Walker

Variables

var (

	// spool is immutable, consumed tasks remains in spool.
	IsImmutable = withMutability(immutable)
	// spool is mutable, consumed tasks are removed
	IsMutable = withMutability(mutable)

	// spool fails on error
	WithStrict = withStrict(strict)
	// spool skips error, continue as warning
	WithSkipError = withStrict(skiperror)

	// spool files matching the pattern
	WithPattern = opts.ForName[Spool, string]("pattern")

	// output files with new extension
	WithFileExt = opts.ForName[Spool, string]("ext")
)

Types

type File

type File = interface {
	io.Writer
	io.Closer
	Stat() (fs.FileInfo, error)
	Cancel() error
}

File provides I/O access to individual object on the file system.

type FileSystem

type FileSystem interface {
	fs.FS
	Create(path string, attr *struct{}) (File, error)
	Remove(path string) error
}

A FileSystem abstraction provides access to a hierarchical file system or sequence of files

Use https://github.com/fogfish/stream to implement FileSystem for local file system or AWS S3.

type Spool

type Spool struct {
	// contains filtered or unexported fields
}

func New

func New(reader, writer FileSystem, opt ...opts.Option[Spool]) *Spool

func (*Spool) ForEach

func (spool *Spool) ForEach(ctx context.Context, walker Walker, f Spooler) error

ForEach iterates over the files in the spool's reader using the provided walker. It applies the Spooler function f to each file that matches the spool's pattern (if set). Directories are skipped, and files not matching the pattern are ignored. The function returns an error if the regex compilation fails, walking encounters an error, or if applying the Spooler function fails for any file.

func (*Spool) Shard

func (spool *Spool) Shard(
	ctx context.Context,
	dir string,
	f func(context.Context, string, io.Reader) (string, error),
) error

Apply the shard function over each file in the reader filesystem. The shard function is given the file path and a reader for the input file, and returns the shard name for the output file. The output file is written to the writer filesystem with the same path, but prefixed with the shard name.

If the shard function returns an empty string, the file is skipped. After processing, if the spool is mutable, the original file is removed from the reader filesystem.

func (*Spool) Write

func (spool *Spool) Write(path string, r io.Reader) error

Write file to spool, consuming all data from the reader.

func (*Spool) WriteFile

func (spool *Spool) WriteFile(path string, b []byte) error

Write file to spool

type Spooler

type Spooler = func(context.Context, string, io.Reader, io.Writer) error

Spooler is a function applied to each file. It is given the file path, a reader for the input file and a writer for the output file.

type Walker

type Walker interface {
	Walk(fs.FS, fs.WalkDirFunc) error
}

Walker provides Walk method to traverse the files.

func WalkDir

func WalkDir(dir string) Walker

WalkDir returns a Walker that traverses the file system rooted at the given directory.

func WalkFiles

func WalkFiles(files []string) Walker

WalkFiles returns a Walker that traverses the given list of files. Each file is processed as if it were a separate root directory.