Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type CommandToParams ¶ added in v0.9.0
type CommandToParams struct { scipipe.BaseProcess // contains filtered or unexported fields }
CommandToParams takes a shell command, runs it, and sens each of its files as parameters on its OutParam parameter port
func NewCommandToParams ¶ added in v0.9.0
func NewCommandToParams(wf *scipipe.Workflow, name string, command string) *CommandToParams
NewCommandToParams returns an initialized new CommandToParams
func (*CommandToParams) OutParam ¶ added in v0.9.0
func (p *CommandToParams) OutParam() *scipipe.OutParamPort
OutParam returns an parameter out-port with lines of the files being read
func (*CommandToParams) Run ¶ added in v0.9.0
func (p *CommandToParams) Run()
Run the CommandToParams
type Concatenator ¶
type Concatenator struct { scipipe.BaseProcess OutPath string GroupByTag string }
Concatenator is a process that concatenates the content of multiple files received in the in-port In, into one file returned on its out-port, Out. You can optionally specify a tag name to GroupByTag, which will make files go into separate output files if they have different values for that tag. These output files will have the tag name appended to the base file name.
func NewConcatenator ¶
func NewConcatenator(wf *scipipe.Workflow, name string, outPath string) *Concatenator
NewConcatenator returns a new, initialized Concatenator process
func (*Concatenator) In ¶
func (p *Concatenator) In() *scipipe.InPort
In returns the (only) in-port for this process
func (*Concatenator) Out ¶
func (p *Concatenator) Out() *scipipe.OutPort
Out returns the (only) out-port for this process
type FileCombinator ¶ added in v0.9.1
type FileCombinator struct { scipipe.BaseProcess // contains filtered or unexported fields }
FileCombinator takes a set of input streams of FileIPs, and returns the same number of output streams, where the FileIPs are multiplied so as to guarantee that all combinations of the ips in the input streams are created. Input ports and corresponding out-ports (with the same port names) are created on demand, by accessing them with the p.In(PORTNAME) method. The corresponding out-porta can then be accessed with the same port name with p.Out(PORTNAME)
func NewFileCombinator ¶ added in v0.9.1
func NewFileCombinator(wf *scipipe.Workflow, name string) *FileCombinator
NewFileCombinator returns a new initialized FileCombinator process
func (*FileCombinator) In ¶ added in v0.9.1
func (p *FileCombinator) In(pName string) *scipipe.InPort
In returns the in-port with name pName. If it does not exist, it will create that in-port, and a corresponding out-port with the same port name.
func (*FileCombinator) Out ¶ added in v0.9.1
func (p *FileCombinator) Out(pName string) *scipipe.OutPort
Out returns the outport
func (*FileCombinator) Run ¶ added in v0.9.1
func (p *FileCombinator) Run()
Run runs the FileCombinator process
type FileGlobber ¶
type FileGlobber struct { scipipe.BaseProcess // contains filtered or unexported fields }
FileGlobber is initiated with a set of glob patterns paths, which it will use to find concrete file paths, for which it will return a stream of corresponding File IPs on its outport Out()
func NewFileGlobber ¶
func NewFileGlobber(wf *scipipe.Workflow, name string, globPatterns ...string) *FileGlobber
NewFileGlobber returns a new initialized FileGlobber process
func NewFileGlobberDependent ¶ added in v0.8.2
func NewFileGlobberDependent(wf *scipipe.Workflow, name string, globPatterns ...string) *FileGlobber
NewFileGlobberDependent returns a new FileGlobber that depends on upstream files to be received on the InPort InDependency() before it starts globbing files.
func (*FileGlobber) InDependency ¶ added in v0.8.2
func (p *FileGlobber) InDependency() *scipipe.InPort
InDependency takes files which it will wait for before it starts to execute.
func (*FileGlobber) Out ¶
func (p *FileGlobber) Out() *scipipe.OutPort
Out returns the out-port, on which file IPs based on the file paths the process was initialized with, will be retrieved.
type FileSource ¶ added in v0.6.3
type FileSource struct { scipipe.BaseProcess // contains filtered or unexported fields }
FileSource is initiated with a set of file paths, which it will send as a stream of File IPs on its outport Out()
func NewFileSource ¶ added in v0.6.3
func NewFileSource(wf *scipipe.Workflow, name string, filePaths ...string) *FileSource
NewFileSource returns a new initialized FileSource process
func (*FileSource) Out ¶ added in v0.6.3
func (p *FileSource) Out() *scipipe.OutPort
Out returns the out-port, on which file IPs based on the file paths the process was initialized with, will be retrieved.
type FileSplitter ¶
type FileSplitter struct { scipipe.BaseProcess LinesPerSplit int }
FileSplitter is a process that will split a file into multiple files, each with LinesPerSplit number of lines per file
func NewFileSplitter ¶
func NewFileSplitter(wf *scipipe.Workflow, name string, linesPerSplit int) *FileSplitter
NewFileSplitter returns an initialized FileSplitter process that will split a file into multiple files, each with linesPerSplit number of lines per file
func (*FileSplitter) InFile ¶
func (p *FileSplitter) InFile() *scipipe.InPort
InFile returns the port for the input file
func (*FileSplitter) OutSplitFile ¶
func (p *FileSplitter) OutSplitFile() *scipipe.OutPort
OutSplitFile returns the resulting split (part) files generated0
type FileToParamsReader ¶ added in v0.9.0
type FileToParamsReader struct { scipipe.BaseProcess // contains filtered or unexported fields }
FileToParamsReader takes a file path on its FilePath in-port, and returns the file content as []byte on its out-port Out
func NewFileToParamsReader ¶ added in v0.9.0
func NewFileToParamsReader(wf *scipipe.Workflow, name string, filePath string) *FileToParamsReader
NewFileToParamsReader returns an initialized new FileToParamsReader
func (*FileToParamsReader) OutLine ¶ added in v0.9.0
func (p *FileToParamsReader) OutLine() *scipipe.OutParamPort
OutLine returns an parameter out-port with lines of the files being read
func (*FileToParamsReader) Run ¶ added in v0.9.0
func (p *FileToParamsReader) Run()
Run the FileToParamsReader
type IPSelectorSync ¶ added in v0.9.13
type IPSelectorSync struct { sp.BaseProcess // contains filtered or unexported fields }
IPSelectorSync enables filtering IPs (FileIPs to be specific) by applying the supplied function includeFunc, which, if it returns true for an IP, will include it. The IPSelectorSync requires that the same number and names of ports are used and connected both for in-ports and out-ports. So, if you have an in-port 'data1', and 'data2', you need to create and connect also out-ports 'data1', and 'data2'. It will read all in-ports in a synchronous manner, and drop all IPs in the current iteration, if the condition in the includeFunc is not met.
func NewIPSelectorSync ¶ added in v0.9.13
func NewIPSelectorSync(wf *sp.Workflow, name string, includeFunc func(ip *sp.FileIP) bool) *IPSelectorSync
NewIPSelectorSync returns a new IPSelectorSync component. See the docs for IPSelectorSync for more information about how to configure and use it.
func (*IPSelectorSync) In ¶ added in v0.9.13
func (p *IPSelectorSync) In(name string) *sp.InPort
In returns an in-port if it exists, or creates it before, if it does not exist
type MapToTags ¶ added in v0.8.0
type MapToTags struct { scipipe.BaseProcess // contains filtered or unexported fields }
MapToTags is a process that runs a function provided by the user, upon initialization, that will provide a map of tag:value pairs, based in IPs read on the In-port. The tag:value pairs (maps) are added to the IPs on the out-port, which are identical to the incoming IPs, except for the new tag:value map
func NewMapToTags ¶ added in v0.8.0
func NewMapToTags(wf *scipipe.Workflow, name string, mapFunc func(ip *scipipe.FileIP) map[string]string) *MapToTags
NewMapToTags returns an initialized MapToTags process
func (*MapToTags) In ¶ added in v0.8.0
In takes input files the content of which the map function will be run, to generate tags
type ParamCombinator ¶ added in v0.9.4
type ParamCombinator struct {
scipipe.BaseProcess
}
ParamCombinator takes a set of input params, and returns the same number of param streams, where the params are multiplied so as to guarantee that all combinations of the params in the streams are created. Input ports and corresponding out-ports (with the same port names) are created on demand, by accessing them with the p.InParam(PORTNAME) method. The corresponding out-porta can then be accessed with the same port name with p.OutParam(PORTNAME)
func NewParamCombinator ¶ added in v0.9.4
func NewParamCombinator(wf *scipipe.Workflow, name string) *ParamCombinator
NewParamCombinator returns a new initialized ParamCombinator process
func (*ParamCombinator) InParam ¶ added in v0.9.4
func (p *ParamCombinator) InParam(pName string) *scipipe.InParamPort
InParam returns the in-port with name pName. If it does not exist, it will create that in-port, and a corresponding out-port with the same port name.
func (*ParamCombinator) OutParam ¶ added in v0.9.4
func (p *ParamCombinator) OutParam(pName string) *scipipe.OutParamPort
OutParam returns the outport
func (*ParamCombinator) Run ¶ added in v0.9.4
func (p *ParamCombinator) Run()
Run runs the ParamCombinator process
type ParamSource ¶ added in v0.6.3
type ParamSource struct { scipipe.BaseProcess // contains filtered or unexported fields }
ParamSource will feed parameters on an out-port
func NewParamSource ¶ added in v0.6.3
func NewParamSource(wf *scipipe.Workflow, name string, params ...string) *ParamSource
NewParamSource returns a new ParamSource
func (*ParamSource) Out ¶ added in v0.6.3
func (p *ParamSource) Out() *scipipe.OutParamPort
Out returns the out-port, on which parameters the process was initialized with, will be retrieved.
type StreamToSubStream ¶
type StreamToSubStream struct {
scipipe.BaseProcess
}
StreamToSubStream takes a normal stream of IP's representing individual files, and returns one IP where the incoming IPs are sent on its substream.
func NewStreamToSubStream ¶
func NewStreamToSubStream(wf *scipipe.Workflow, name string) *StreamToSubStream
NewStreamToSubStream instantiates a new StreamToSubStream process
func (*StreamToSubStream) In ¶
func (p *StreamToSubStream) In() *scipipe.InPort
In returns the in-port
func (*StreamToSubStream) OutSubStream ¶
func (p *StreamToSubStream) OutSubStream() *scipipe.OutPort
OutSubStream returns the out-port