Documentation

Overview

    Package util has various helper functions used by components of ratchet.

    Index

    Constants

    This section is empty.

    Variables

    This section is empty.

    Functions

    func CSVProcess

    func CSVProcess(params *CSVParameters, d data.JSON, outputChan chan data.JSON, killChan chan error)

      CSVProcess writes the contents to the file and optionally sends the written bytes upstream on outputChan

      func CSVString

      func CSVString(v interface{}) string

        CSVString returns an empty string for nil values to make sure that the text "null" is not written to a file

        func DeleteS3Objects

        func DeleteS3Objects(client *s3.S3, bucket string, objKeys []string) (*s3.DeleteObjectsOutput, error)

          DeleteS3Objects deletes the objects specified by the given object keys

          func ExecuteSQLQuery

          func ExecuteSQLQuery(db *sql.DB, query string) error

            ExecuteSQLQuery allows you to execute arbitrary SQL statements

            func GetDataFromSQLQuery

            func GetDataFromSQLQuery(db *sql.DB, query string, batchSize int, structDest interface{}) (chan data.JSON, error)

              GetDataFromSQLQuery is a util function that, given a properly intialized sql.DB and a valid SQL query, will handle executing the query and getting back data.JSON objects. This function is asynch, and data.JSON should be received on the return data channel. If there was a problem setting up the query, then an error will also be returned immediately. It is also possible for errors to occur during execution as data is retrieved from the query. If this happens, the object returned will be a JSON object in the form of {"Error": "description"}.

              func GetS3Object

              func GetS3Object(client *s3.S3, bucket, objKey string) (*s3.GetObjectOutput, error)

                GetS3Object returns the object output for the given object key

                func KillPipelineIfErr

                func KillPipelineIfErr(err error, killChan chan error)

                  KillPipelineIfErr is an error-checking helper.

                  func ListS3Objects

                  func ListS3Objects(client *s3.S3, bucket, keyPrefix string) ([]string, error)

                    ListS3Objects returns all object keys matching the given prefix. Note that delimiter is set to "/". See http://docs.aws.amazon.com/AmazonS3/latest/dev/ListingKeysHierarchy.html

                    func SQLInsertData

                    func SQLInsertData(db *sql.DB, d data.JSON, tableName string, onDupKeyUpdate bool, onDupKeyFields []string, batchSize int) error

                      SQLInsertData abstracts building and executing a SQL INSERT statement for the given Data object.

                      Note that the Data must be a valid JSON object (or an array of valid objects all with the same keys), where the keys are column names and the the values are SQL values to be inserted into those columns.

                      func SftpClient

                      func SftpClient(server string, username string, authMethod []ssh.AuthMethod, opts ...func(*sftp.Client) error) (*sftp.Client, error)

                        SftpClient sets up and return the client

                        func SftpKeyAuth

                        func SftpKeyAuth(privateKeyPath string) (auth ssh.AuthMethod, err error)

                          SftpKeyAuth generates an ssh.AuthMethod given the path of a private key

                          func WriteS3Object

                          func WriteS3Object(data []string, config *aws.Config, bucket string, key string, lineSeparator string, compress bool) (string, error)

                            WriteS3Object writes the data to the given key, optionally compressing it first

                            Types

                            type CSVParameters

                            type CSVParameters struct {
                            	Writer        *CSVWriter
                            	WriteHeader   bool
                            	HeaderWritten bool
                            	Header        []string
                            	SendUpstream  bool
                            	QuoteEscape   string
                            }

                              CSVParameters allows you to define all of your csv writing preferences in a single struct for reuse in multiple processors

                              type CSVWriter

                              type CSVWriter struct {
                              	Comma   rune
                              	UseCRLF bool
                              
                              	AlwaysEncapsulate bool   // If the content should be encapsulated independent of its type
                              	QuoteEscape       string // String to use to escape a quote character
                              	// contains filtered or unexported fields
                              }

                                CSVWriter reimplements the standard library csv.Writer adding AlwaysEncapsulate and QuoteEscape

                                func NewCSVWriter

                                func NewCSVWriter() *CSVWriter

                                  NewCSVWriter instantiates a new instance of CSVWriter

                                  func (*CSVWriter) Error

                                  func (w *CSVWriter) Error() error

                                    Error reports any error that has occurred during a previous Write or Flush.

                                    func (*CSVWriter) Flush

                                    func (w *CSVWriter) Flush()

                                      Flush writes any buffered data to the underlying io.Writer. To check if an error occurred during the Flush, call Error.

                                      func (*CSVWriter) SetWriter

                                      func (w *CSVWriter) SetWriter(writer io.Writer)

                                        SetWriter allows you to change the writer (which is not directly exposed)

                                        func (*CSVWriter) Write

                                        func (w *CSVWriter) Write(record []string) (err error)

                                          Write writes a single CSV record to w along with any necessary quoting. A record is a slice of strings with each string being one field.

                                          func (*CSVWriter) WriteAll

                                          func (w *CSVWriter) WriteAll(records [][]string) (err error)

                                            WriteAll writes multiple CSV records to w using Write and then calls Flush.

                                            type SftpParameters

                                            type SftpParameters struct {
                                            	Server      string
                                            	Username    string
                                            	Path        string
                                            	AuthMethods []ssh.AuthMethod
                                            }

                                              SftpParameters is used for storing connection parameters for later executing sftp commands

                                              type SftpPath

                                              type SftpPath struct {
                                              	Path string `json:"path,omitempty"`
                                              }

                                                SftpPath is a simple struct for storing the full path of an object

                                                func (SftpPath) FileName

                                                func (t SftpPath) FileName() string

                                                  FileName defers to filepath.Base

                                                  type Timer

                                                  type Timer struct {
                                                  	// contains filtered or unexported fields
                                                  }

                                                    Timer is a basic mechanism for measuring execution time.

                                                    func StartTimer

                                                    func StartTimer() (t *Timer)

                                                      StartTimer returns a new Timer that's already "started".

                                                      func (*Timer) Duration

                                                      func (t *Timer) Duration() time.Duration

                                                        Duration returns either the total executino duration (if Timer stopped) or the duration until time.Now() if timer is still running.

                                                        func (*Timer) Stop

                                                        func (t *Timer) Stop() *Timer

                                                          Stop sets the end time for the Timer and returns itself.

                                                          func (*Timer) Stopped

                                                          func (t *Timer) Stopped() bool

                                                            Stopped returns true if Stop() has been called on the timer.

                                                            func (*Timer) String

                                                            func (t *Timer) String() string