components

package
v0.11.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 1, 2021 License: MIT Imports: 12 Imported by: 2

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type CommandToParams added in v0.9.0 

type CommandToParams struct {
	scipipe.BaseProcess
	// contains filtered or unexported fields
}

CommandToParams takes a shell command, runs it, and sens each of its files as parameters on its OutParam parameter port

func NewCommandToParams added in v0.9.0 

func NewCommandToParams(wf *scipipe.Workflow, name string, command string) *CommandToParams

NewCommandToParams returns an initialized new CommandToParams

func (*CommandToParams) Fail added in v0.11.0 

func (p *CommandToParams) Fail(msg interface{})

func (*CommandToParams) Failf added in v0.11.0 

func (p *CommandToParams) Failf(msg string, parts ...interface{})

func (*CommandToParams) OutParam added in v0.9.0 

func (p *CommandToParams) OutParam() *scipipe.OutParamPort

OutParam returns an parameter out-port with lines of the files being read

func (*CommandToParams) Run added in v0.9.0 

func (p *CommandToParams) Run()

Run the CommandToParams

type Concatenator 

type Concatenator struct {
	scipipe.BaseProcess
	OutPath    string
	GroupByTag string
}

Concatenator is a process that concatenates the content of multiple files received in the in-port In, into one file returned on its out-port, Out. You can optionally specify a tag name to GroupByTag, which will make files go into separate output files if they have different values for that tag. These output files will have the tag name appended to the base file name.

func NewConcatenator 

func NewConcatenator(wf *scipipe.Workflow, name string, outPath string) *Concatenator

NewConcatenator returns a new, initialized Concatenator process

func (*Concatenator) Fail added in v0.11.0 

func (p *Concatenator) Fail(msg interface{})

func (*Concatenator) Failf added in v0.11.0 

func (p *Concatenator) Failf(msg string, parts ...interface{})

func (*Concatenator) In 

func (p *Concatenator) In() *scipipe.InPort

In returns the (only) in-port for this process

func (*Concatenator) Out 

func (p *Concatenator) Out() *scipipe.OutPort

Out returns the (only) out-port for this process

func (*Concatenator) Run 

func (p *Concatenator) Run()

Run runs the Concatenator process

type FileCombinator added in v0.9.1 

type FileCombinator struct {
	scipipe.BaseProcess
	// contains filtered or unexported fields
}

FileCombinator takes a set of input streams of FileIPs, and returns the same number of output streams, where the FileIPs are multiplied so as to guarantee that all combinations of the ips in the input streams are created. Input ports and corresponding out-ports (with the same port names) are created on demand, by accessing them with the p.In(PORTNAME) method. The corresponding out-porta can then be accessed with the same port name with p.Out(PORTNAME)

func NewFileCombinator added in v0.9.1 

func NewFileCombinator(wf *scipipe.Workflow, name string) *FileCombinator

NewFileCombinator returns a new initialized FileCombinator process

func (*FileCombinator) Fail added in v0.11.0 

func (p *FileCombinator) Fail(msg interface{})

func (*FileCombinator) Failf added in v0.11.0 

func (p *FileCombinator) Failf(msg string, parts ...interface{})

func (*FileCombinator) In added in v0.9.1 

func (p *FileCombinator) In(pName string) *scipipe.InPort

In returns the in-port with name pName. If it does not exist, it will create that in-port, and a corresponding out-port with the same port name.

func (*FileCombinator) Out added in v0.9.1 

func (p *FileCombinator) Out(pName string) *scipipe.OutPort

Out returns the outport

func (*FileCombinator) Run added in v0.9.1 

func (p *FileCombinator) Run()

Run runs the FileCombinator process

type FileGlobber 

type FileGlobber struct {
	scipipe.BaseProcess
	// contains filtered or unexported fields
}

FileGlobber is initiated with a set of glob patterns paths, which it will use to find concrete file paths, for which it will return a stream of corresponding File IPs on its outport Out()

func NewFileGlobber 

func NewFileGlobber(wf *scipipe.Workflow, name string, globPatterns ...string) *FileGlobber

NewFileGlobber returns a new initialized FileGlobber process

func NewFileGlobberDependent added in v0.8.2 

func NewFileGlobberDependent(wf *scipipe.Workflow, name string, globPatterns ...string) *FileGlobber

NewFileGlobberDependent returns a new FileGlobber that depends on upstream files to be received on the InPort InDependency() before it starts globbing files.

func (*FileGlobber) Fail added in v0.11.0 

func (p *FileGlobber) Fail(msg interface{})

func (*FileGlobber) Failf added in v0.11.0 

func (p *FileGlobber) Failf(msg string, parts ...interface{})

func (*FileGlobber) InDependency added in v0.8.2 

func (p *FileGlobber) InDependency() *scipipe.InPort

InDependency takes files which it will wait for before it starts to execute.

func (*FileGlobber) Out 

func (p *FileGlobber) Out() *scipipe.OutPort

Out returns the out-port, on which file IPs based on the file paths the process was initialized with, will be retrieved.

func (*FileGlobber) Run 

func (p *FileGlobber) Run()

Run runs the FileGlobber process

type FileSource added in v0.6.3 

type FileSource struct {
	scipipe.BaseProcess
	// contains filtered or unexported fields
}

FileSource is initiated with a set of file paths, which it will send as a stream of File IPs on its outport Out()

func NewFileSource added in v0.6.3 

func NewFileSource(wf *scipipe.Workflow, name string, filePaths ...string) *FileSource

NewFileSource returns a new initialized FileSource process

func (*FileSource) Fail added in v0.11.0 

func (p *FileSource) Fail(msg interface{})

func (*FileSource) Failf added in v0.11.0 

func (p *FileSource) Failf(msg string, parts ...interface{})

func (*FileSource) Out added in v0.6.3 

func (p *FileSource) Out() *scipipe.OutPort

Out returns the out-port, on which file IPs based on the file paths the process was initialized with, will be retrieved.

func (*FileSource) Run added in v0.6.3 

func (p *FileSource) Run()

Run runs the FileSource process

type FileSplitter 

type FileSplitter struct {
	scipipe.BaseProcess
	LinesPerSplit int
}

FileSplitter is a process that will split a file into multiple files, each with LinesPerSplit number of lines per file

func NewFileSplitter 

func NewFileSplitter(wf *scipipe.Workflow, name string, linesPerSplit int) *FileSplitter

NewFileSplitter returns an initialized FileSplitter process that will split a file into multiple files, each with linesPerSplit number of lines per file

func (*FileSplitter) Audit added in v0.11.0 

func (p *FileSplitter) Audit(msg interface{})

func (*FileSplitter) Auditf added in v0.11.0 

func (p *FileSplitter) Auditf(msg string, parts ...interface{})

func (*FileSplitter) Fail added in v0.11.0 

func (p *FileSplitter) Fail(msg interface{})

func (*FileSplitter) Failf added in v0.11.0 

func (p *FileSplitter) Failf(msg string, parts ...interface{})

func (*FileSplitter) InFile 

func (p *FileSplitter) InFile() *scipipe.InPort

InFile returns the port for the input file

func (*FileSplitter) OutSplitFile 

func (p *FileSplitter) OutSplitFile() *scipipe.OutPort

OutSplitFile returns the resulting split (part) files generated0

func (*FileSplitter) Run 

func (p *FileSplitter) Run()

Run runs the FileSplitter process

type FileToParamsReader added in v0.9.0 

type FileToParamsReader struct {
	scipipe.BaseProcess
	// contains filtered or unexported fields
}

FileToParamsReader takes a file path on its FilePath in-port, and returns the file content as []byte on its out-port Out

func NewFileToParamsReader added in v0.9.0 

func NewFileToParamsReader(wf *scipipe.Workflow, name string, filePath string) *FileToParamsReader

NewFileToParamsReader returns an initialized new FileToParamsReader

func (*FileToParamsReader) Fail added in v0.11.0 

func (p *FileToParamsReader) Fail(msg interface{})

func (*FileToParamsReader) Failf added in v0.11.0 

func (p *FileToParamsReader) Failf(msg string, parts ...interface{})

func (*FileToParamsReader) OutLine added in v0.9.0 

func (p *FileToParamsReader) OutLine() *scipipe.OutParamPort

OutLine returns an parameter out-port with lines of the files being read

func (*FileToParamsReader) Run added in v0.9.0 

func (p *FileToParamsReader) Run()

Run the FileToParamsReader

type IPSelectorSync added in v0.9.13 

type IPSelectorSync struct {
	sp.BaseProcess
	// contains filtered or unexported fields
}

IPSelectorSync enables filtering IPs (FileIPs to be specific) by applying the supplied function includeFunc, which, if it returns true for an IP, will include it. The IPSelectorSync requires that the same number and names of ports are used and connected both for in-ports and out-ports. So, if you have an in-port 'data1', and 'data2', you need to create and connect also out-ports 'data1', and 'data2'. It will read all in-ports in a synchronous manner, and drop all IPs in the current iteration, if the condition in the includeFunc is not met.

func NewIPSelectorSync added in v0.9.13 

func NewIPSelectorSync(wf *sp.Workflow, name string, includeFunc func(ip *sp.FileIP) bool) *IPSelectorSync

NewIPSelectorSync returns a new IPSelectorSync component. See the docs for IPSelectorSync for more information about how to configure and use it.

func (*IPSelectorSync) Fail added in v0.11.0 

func (p *IPSelectorSync) Fail(msg interface{})

func (*IPSelectorSync) Failf added in v0.11.0 

func (p *IPSelectorSync) Failf(msg string, parts ...interface{})

func (*IPSelectorSync) In added in v0.9.13 

func (p *IPSelectorSync) In(name string) *sp.InPort

In returns an in-port if it exists, or creates it before, if it does not exist

func (*IPSelectorSync) Out added in v0.9.13 

func (p *IPSelectorSync) Out(name string) *sp.OutPort

Out returns an out-port if it exists, or creates it before, if it does not exist

func (*IPSelectorSync) Run added in v0.9.13 

func (p *IPSelectorSync) Run()

Run runs the component

type MapToTags added in v0.8.0 

type MapToTags struct {
	scipipe.BaseProcess
	// contains filtered or unexported fields
}

MapToTags is a process that runs a function provided by the user, upon initialization, that will provide a map of tag:value pairs, based in IPs read on the In-port. The tag:value pairs (maps) are added to the IPs on the out-port, which are identical to the incoming IPs, except for the new tag:value map

func NewMapToTags added in v0.8.0 

func NewMapToTags(wf *scipipe.Workflow, name string, mapFunc func(ip *scipipe.FileIP) map[string]string) *MapToTags

NewMapToTags returns an initialized MapToTags process

func (*MapToTags) Fail added in v0.11.0 

func (p *MapToTags) Fail(msg interface{})

func (*MapToTags) Failf added in v0.11.0 

func (p *MapToTags) Failf(msg string, parts ...interface{})

func (*MapToTags) In added in v0.8.0 

func (p *MapToTags) In() *scipipe.InPort

In takes input files the content of which the map function will be run, to generate tags

func (*MapToTags) Out added in v0.8.0 

func (p *MapToTags) Out() *scipipe.OutPort

Out outputs files that are supplemented with tags by the map function.

func (*MapToTags) Run added in v0.8.0 

func (p *MapToTags) Run()

Run runs the MapToTags process

type ParamCombinator added in v0.9.4 

type ParamCombinator struct {
	scipipe.BaseProcess
}

ParamCombinator takes a set of input params, and returns the same number of param streams, where the params are multiplied so as to guarantee that all combinations of the params in the streams are created. Input ports and corresponding out-ports (with the same port names) are created on demand, by accessing them with the p.InParam(PORTNAME) method. The corresponding out-porta can then be accessed with the same port name with p.OutParam(PORTNAME)

func NewParamCombinator added in v0.9.4 

func NewParamCombinator(wf *scipipe.Workflow, name string) *ParamCombinator

NewParamCombinator returns a new initialized ParamCombinator process

func (*ParamCombinator) Fail added in v0.11.0 

func (p *ParamCombinator) Fail(msg interface{})

func (*ParamCombinator) Failf added in v0.11.0 

func (p *ParamCombinator) Failf(msg string, parts ...interface{})

func (*ParamCombinator) InParam added in v0.9.4 

func (p *ParamCombinator) InParam(pName string) *scipipe.InParamPort

InParam returns the in-port with name pName. If it does not exist, it will create that in-port, and a corresponding out-port with the same port name.

func (*ParamCombinator) OutParam added in v0.9.4 

func (p *ParamCombinator) OutParam(pName string) *scipipe.OutParamPort

OutParam returns the outport

func (*ParamCombinator) Run added in v0.9.4 

func (p *ParamCombinator) Run()

Run runs the ParamCombinator process

type ParamSource added in v0.6.3 

type ParamSource struct {
	scipipe.BaseProcess
	// contains filtered or unexported fields
}

ParamSource will feed parameters on an out-port

func NewParamSource added in v0.6.3 

func NewParamSource(wf *scipipe.Workflow, name string, params ...string) *ParamSource

NewParamSource returns a new ParamSource

func (*ParamSource) Fail added in v0.11.0 

func (p *ParamSource) Fail(msg interface{})

func (*ParamSource) Failf added in v0.11.0 

func (p *ParamSource) Failf(msg string, parts ...interface{})

func (*ParamSource) Out added in v0.6.3 

func (p *ParamSource) Out() *scipipe.OutParamPort

Out returns the out-port, on which parameters the process was initialized with, will be retrieved.

func (*ParamSource) Run added in v0.6.3 

func (p *ParamSource) Run()

Run runs the process

type StreamToSubStream 

type StreamToSubStream struct {
	scipipe.BaseProcess
}

StreamToSubStream takes a normal stream of IP's representing individual files, and returns one IP where the incoming IPs are sent on its substream.

func NewStreamToSubStream 

func NewStreamToSubStream(wf *scipipe.Workflow, name string) *StreamToSubStream

NewStreamToSubStream instantiates a new StreamToSubStream process

func (*StreamToSubStream) Fail added in v0.11.0 

func (p *StreamToSubStream) Fail(msg interface{})

func (*StreamToSubStream) Failf added in v0.11.0 

func (p *StreamToSubStream) Failf(msg string, parts ...interface{})

func (*StreamToSubStream) In 

func (p *StreamToSubStream) In() *scipipe.InPort

In returns the in-port

func (*StreamToSubStream) OutSubStream 

func (p *StreamToSubStream) OutSubStream() *scipipe.OutPort

OutSubStream returns the out-port

func (*StreamToSubStream) Run 

func (p *StreamToSubStream) Run()

Run runs the StreamToSubStream

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.