Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type CommandToParams

type CommandToParams struct {
	scipipe.BaseProcess
	// contains filtered or unexported fields
}

    CommandToParams takes a shell command, runs it, and sens each of its files as parameters on its OutParam parameter port

    func NewCommandToParams

    func NewCommandToParams(wf *scipipe.Workflow, name string, command string) *CommandToParams

      NewCommandToParams returns an initialized new CommandToParams

      func (*CommandToParams) OutParam

      func (p *CommandToParams) OutParam() *scipipe.OutParamPort

        OutParam returns an parameter out-port with lines of the files being read

        func (*CommandToParams) Run

        func (p *CommandToParams) Run()

          Run the CommandToParams

          type Concatenator

          type Concatenator struct {
          	scipipe.BaseProcess
          	OutPath    string
          	GroupByTag string
          }

            Concatenator is a process that concatenates the content of multiple files received in the in-port In, into one file returned on its out-port, Out. You can optionally specify a tag name to GroupByTag, which will make files go into separate output files if they have different values for that tag. These output files will have the tag name appended to the base file name.

            func NewConcatenator

            func NewConcatenator(wf *scipipe.Workflow, name string, outPath string) *Concatenator

              NewConcatenator returns a new, initialized Concatenator process

              func (*Concatenator) In

              func (p *Concatenator) In() *scipipe.InPort

                In returns the (only) in-port for this process

                func (*Concatenator) Out

                func (p *Concatenator) Out() *scipipe.OutPort

                  Out returns the (only) out-port for this process

                  func (*Concatenator) Run

                  func (p *Concatenator) Run()

                    Run runs the Concatenator process

                    type FileCombinator

                    type FileCombinator struct {
                    	scipipe.BaseProcess
                    	// contains filtered or unexported fields
                    }

                      FileCombinator takes a set of input streams of FileIPs, and returns the same number of output streams, where the FileIPs are multiplied so as to guarantee that all combinations of the ips in the input streams are created. Input ports and corresponding out-ports (with the same port names) are created on demand, by accessing them with the p.In(PORTNAME) method. The corresponding out-porta can then be accessed with the same port name with p.Out(PORTNAME)

                      func NewFileCombinator

                      func NewFileCombinator(wf *scipipe.Workflow, name string) *FileCombinator

                        NewFileCombinator returns a new initialized FileCombinator process

                        func (*FileCombinator) In

                        func (p *FileCombinator) In(pName string) *scipipe.InPort

                          In returns the in-port with name pName. If it does not exist, it will create that in-port, and a corresponding out-port with the same port name.

                          func (*FileCombinator) Out

                          func (p *FileCombinator) Out(pName string) *scipipe.OutPort

                            Out returns the outport

                            func (*FileCombinator) Run

                            func (p *FileCombinator) Run()

                              Run runs the FileCombinator process

                              type FileGlobber

                              type FileGlobber struct {
                              	scipipe.BaseProcess
                              	// contains filtered or unexported fields
                              }

                                FileGlobber is initiated with a set of glob patterns paths, which it will use to find concrete file paths, for which it will return a stream of corresponding File IPs on its outport Out()

                                func NewFileGlobber

                                func NewFileGlobber(wf *scipipe.Workflow, name string, globPatterns ...string) *FileGlobber

                                  NewFileGlobber returns a new initialized FileGlobber process

                                  func NewFileGlobberDependent

                                  func NewFileGlobberDependent(wf *scipipe.Workflow, name string, globPatterns ...string) *FileGlobber

                                    NewFileGlobberDependent returns a new FileGlobber that depends on upstream files to be received on the InPort InDependency() before it starts globbing files.

                                    func (*FileGlobber) InDependency

                                    func (p *FileGlobber) InDependency() *scipipe.InPort

                                      InDependency takes files which it will wait for before it starts to execute.

                                      func (*FileGlobber) Out

                                      func (p *FileGlobber) Out() *scipipe.OutPort

                                        Out returns the out-port, on which file IPs based on the file paths the process was initialized with, will be retrieved.

                                        func (*FileGlobber) Run

                                        func (p *FileGlobber) Run()

                                          Run runs the FileGlobber process

                                          type FileSource

                                          type FileSource struct {
                                          	scipipe.BaseProcess
                                          	// contains filtered or unexported fields
                                          }

                                            FileSource is initiated with a set of file paths, which it will send as a stream of File IPs on its outport Out()

                                            func NewFileSource

                                            func NewFileSource(wf *scipipe.Workflow, name string, filePaths ...string) *FileSource

                                              NewFileSource returns a new initialized FileSource process

                                              func (*FileSource) Out

                                              func (p *FileSource) Out() *scipipe.OutPort

                                                Out returns the out-port, on which file IPs based on the file paths the process was initialized with, will be retrieved.

                                                func (*FileSource) Run

                                                func (p *FileSource) Run()

                                                  Run runs the FileSource process

                                                  type FileSplitter

                                                  type FileSplitter struct {
                                                  	scipipe.BaseProcess
                                                  	LinesPerSplit int
                                                  }

                                                    FileSplitter is a process that will split a file into multiple files, each with LinesPerSplit number of lines per file

                                                    func NewFileSplitter

                                                    func NewFileSplitter(wf *scipipe.Workflow, name string, linesPerSplit int) *FileSplitter

                                                      NewFileSplitter returns an initialized FileSplitter process that will split a file into multiple files, each with linesPerSplit number of lines per file

                                                      func (*FileSplitter) InFile

                                                      func (p *FileSplitter) InFile() *scipipe.InPort

                                                        InFile returns the port for the input file

                                                        func (*FileSplitter) OutSplitFile

                                                        func (p *FileSplitter) OutSplitFile() *scipipe.OutPort

                                                          OutSplitFile returns the resulting split (part) files generated0

                                                          func (*FileSplitter) Run

                                                          func (p *FileSplitter) Run()

                                                            Run runs the FileSplitter process

                                                            type FileToParamsReader

                                                            type FileToParamsReader struct {
                                                            	scipipe.BaseProcess
                                                            	// contains filtered or unexported fields
                                                            }

                                                              FileToParamsReader takes a file path on its FilePath in-port, and returns the file content as []byte on its out-port Out

                                                              func NewFileToParamsReader

                                                              func NewFileToParamsReader(wf *scipipe.Workflow, name string, filePath string) *FileToParamsReader

                                                                NewFileToParamsReader returns an initialized new FileToParamsReader

                                                                func (*FileToParamsReader) OutLine

                                                                func (p *FileToParamsReader) OutLine() *scipipe.OutParamPort

                                                                  OutLine returns an parameter out-port with lines of the files being read

                                                                  func (*FileToParamsReader) Run

                                                                  func (p *FileToParamsReader) Run()

                                                                    Run the FileToParamsReader

                                                                    type IPSelectorSync

                                                                    type IPSelectorSync struct {
                                                                    	sp.BaseProcess
                                                                    	// contains filtered or unexported fields
                                                                    }

                                                                      IPSelectorSync enables filtering IPs (FileIPs to be specific) by applying the supplied function includeFunc, which, if it returns true for an IP, will include it. The IPSelectorSync requires that the same number and names of ports are used and connected both for in-ports and out-ports. So, if you have an in-port 'data1', and 'data2', you need to create and connect also out-ports 'data1', and 'data2'. It will read all in-ports in a synchronous manner, and drop all IPs in the current iteration, if the condition in the includeFunc is not met.

                                                                      func NewIPSelectorSync

                                                                      func NewIPSelectorSync(wf *sp.Workflow, name string, includeFunc func(ip *sp.FileIP) bool) *IPSelectorSync

                                                                        NewIPSelectorSync returns a new IPSelectorSync component. See the docs for IPSelectorSync for more information about how to configure and use it.

                                                                        func (*IPSelectorSync) In

                                                                        func (p *IPSelectorSync) In(name string) *sp.InPort

                                                                          In returns an in-port if it exists, or creates it before, if it does not exist

                                                                          func (*IPSelectorSync) Out

                                                                          func (p *IPSelectorSync) Out(name string) *sp.OutPort

                                                                            Out returns an out-port if it exists, or creates it before, if it does not exist

                                                                            func (*IPSelectorSync) Run

                                                                            func (p *IPSelectorSync) Run()

                                                                              Run runs the component

                                                                              type MapToTags

                                                                              type MapToTags struct {
                                                                              	scipipe.BaseProcess
                                                                              	// contains filtered or unexported fields
                                                                              }

                                                                                MapToTags is a process that runs a function provided by the user, upon initialization, that will provide a map of tag:value pairs, based in IPs read on the In-port. The tag:value pairs (maps) are added to the IPs on the out-port, which are identical to the incoming IPs, except for the new tag:value map

                                                                                func NewMapToTags

                                                                                func NewMapToTags(wf *scipipe.Workflow, name string, mapFunc func(ip *scipipe.FileIP) map[string]string) *MapToTags

                                                                                  NewMapToTags returns an initialized MapToTags process

                                                                                  func (*MapToTags) In

                                                                                  func (p *MapToTags) In() *scipipe.InPort

                                                                                    In takes input files the content of which the map function will be run, to generate tags

                                                                                    func (*MapToTags) Out

                                                                                    func (p *MapToTags) Out() *scipipe.OutPort

                                                                                      Out outputs files that are supplemented with tags by the map function.

                                                                                      func (*MapToTags) Run

                                                                                      func (p *MapToTags) Run()

                                                                                        Run runs the MapToTags process

                                                                                        type ParamCombinator

                                                                                        type ParamCombinator struct {
                                                                                        	scipipe.BaseProcess
                                                                                        }

                                                                                          ParamCombinator takes a set of input params, and returns the same number of param streams, where the params are multiplied so as to guarantee that all combinations of the params in the streams are created. Input ports and corresponding out-ports (with the same port names) are created on demand, by accessing them with the p.InParam(PORTNAME) method. The corresponding out-porta can then be accessed with the same port name with p.OutParam(PORTNAME)

                                                                                          func NewParamCombinator

                                                                                          func NewParamCombinator(wf *scipipe.Workflow, name string) *ParamCombinator

                                                                                            NewParamCombinator returns a new initialized ParamCombinator process

                                                                                            func (*ParamCombinator) InParam

                                                                                            func (p *ParamCombinator) InParam(pName string) *scipipe.InParamPort

                                                                                              InParam returns the in-port with name pName. If it does not exist, it will create that in-port, and a corresponding out-port with the same port name.

                                                                                              func (*ParamCombinator) OutParam

                                                                                              func (p *ParamCombinator) OutParam(pName string) *scipipe.OutParamPort

                                                                                                OutParam returns the outport

                                                                                                func (*ParamCombinator) Run

                                                                                                func (p *ParamCombinator) Run()

                                                                                                  Run runs the ParamCombinator process

                                                                                                  type ParamSource

                                                                                                  type ParamSource struct {
                                                                                                  	scipipe.BaseProcess
                                                                                                  	// contains filtered or unexported fields
                                                                                                  }

                                                                                                    ParamSource will feed parameters on an out-port

                                                                                                    func NewParamSource

                                                                                                    func NewParamSource(wf *scipipe.Workflow, name string, params ...string) *ParamSource

                                                                                                      NewParamSource returns a new ParamSource

                                                                                                      func (*ParamSource) Out

                                                                                                      func (p *ParamSource) Out() *scipipe.OutParamPort

                                                                                                        Out returns the out-port, on which parameters the process was initialized with, will be retrieved.

                                                                                                        func (*ParamSource) Run

                                                                                                        func (p *ParamSource) Run()

                                                                                                          Run runs the process

                                                                                                          type StreamToSubStream

                                                                                                          type StreamToSubStream struct {
                                                                                                          	scipipe.BaseProcess
                                                                                                          }

                                                                                                            StreamToSubStream takes a normal stream of IP's representing individual files, and returns one IP where the incoming IPs are sent on its substream.

                                                                                                            func NewStreamToSubStream

                                                                                                            func NewStreamToSubStream(wf *scipipe.Workflow, name string) *StreamToSubStream

                                                                                                              NewStreamToSubStream instantiates a new StreamToSubStream process

                                                                                                              func (*StreamToSubStream) In

                                                                                                              func (p *StreamToSubStream) In() *scipipe.InPort

                                                                                                                In returns the in-port

                                                                                                                func (*StreamToSubStream) OutSubStream

                                                                                                                func (p *StreamToSubStream) OutSubStream() *scipipe.OutPort

                                                                                                                  OutSubStream returns the out-port

                                                                                                                  func (*StreamToSubStream) Run

                                                                                                                  func (p *StreamToSubStream) Run()

                                                                                                                    Run runs the StreamToSubStream