transforms

Documentation

Overview

package transforms contains PPS Pipeline Transform implementations

Index

• func IdentityPM(x string) string
• func RunSQLRaw(ctx context.Context, params SQLRunParams) error
• func SQLIngest(ctx context.Context, params SQLIngestParams) error
• func SQLQueryGeneration(ctx context.Context, params SQLQueryGenerationParams) error
• type DataMapper
• type PathMapper
• type SQLIngestParams
• type SQLQueryGenerationParams
• type SQLRunParams

Functions

func IdentityPM

func IdentityPM(x string) string

IdentityPM is the PathMapper which maps a path to itself

func RunSQLRaw

func RunSQLRaw(ctx context.Context, params SQLRunParams) error

func SQLIngest

func SQLIngest(ctx context.Context, params SQLIngestParams) error

SQLIngest connects to a SQL database at params.URL and runs queries read from files in the input. The resulting rows are written to files in params.OutputDir. The format of the output file is controlled by params.Format. Valid options are "json" and "csv"

It makes outgoing connections using pachsql.OpenURL It accesses the filesystem only within params.InputDir, and params.OutputDir

func SQLQueryGeneration

func SQLQueryGeneration(ctx context.Context, params SQLQueryGenerationParams) error

SQLQueryGeneration generates queries with a timestamp in the comments

Types

type DataMapper

type DataMapper = func(r io.Reader, w io.Writer) error

DataMapper maps one stream to another or errors

type PathMapper

type PathMapper = func(string) string

PathMapper is a function that maps one path to another

type SQLIngestParams

type SQLIngestParams struct {
	// PFS In/Out
	InputDir, OutputDir string

	URL       pachsql.URL
	Password  secrets.Secret
	Format    string
	HasHeader bool
}

SQLIngestParams are the parameters passed to SQLIngest

type SQLQueryGenerationParams

type SQLQueryGenerationParams struct {
	InputDir, OutputDir string

	URL      string
	Query    string
	Password secrets.Secret
	Format   string
}

type SQLRunParams

type SQLRunParams struct {
	OutputDir, OutputFile string
	Query                 string
	Password              secrets.Secret
	URL                   pachsql.URL
	HasHeader             bool
	Format                string
}