s3

Documentation

Index

• type Main
  • func NewMain() *Main
  • func (m *Main) Run() error
• type RawSource
  • func NewRawSource(region, bucket, prefix string) (*RawSource, error)
  • func (rs *RawSource) NextReader() (pdk.NamedReadCloser, error)
• type Source
  • func NewSource(opts ...SrcOption) (s *Source, err error)
  • func (s *Source) Record() (rec interface{}, err error)
• type SrcOption
  • func OptSrcBucket(bucket string) SrcOption
  • func OptSrcBufSize(bufsize int) SrcOption
  • func OptSrcPrefix(prefix string) SrcOption
  • func OptSrcRegion(region string) SrcOption
  • func OptSrcSubjectAt(key string) SrcOption

Types

type Main

type Main struct {
	Bucket      string   `help:"S3 bucket name from which to read objects."`
	Prefix      string   `help:"Only objects in the bucket matching this prefix will be used."`
	Region      string   `help:"AWS region to use."`
	PilosaHosts []string `help:"Comma separated list of Pilosa hosts and ports."`
	Index       string   `help:"Pilosa index."`
	BatchSize   uint     `help:"Batch size for Pilosa imports (latency/throughput tradeoff)."`
	Framer      pdk.DashField
	SubjectAt   string   `help:"Tells the S3 source to add a unique 'subject' key to each record which is the s3 object key + record number."`
	SubjectPath []string `help:"Path to value in each record that should be mapped to column ID. Blank gets a sequential ID."`
	Proxy       string   `help:"Bind to this address to proxy and translate requests to Pilosa"`
}

Main contains the configuration for an ingester with an S3 Source.

func NewMain

func NewMain() *Main

NewMain gets a new Main with the default configuration.

func (*Main) Run

func (m *Main) Run() error

Run runs the ingester.

type RawSource

type RawSource struct {
	// contains filtered or unexported fields
}

func NewRawSource

func NewRawSource(region, bucket, prefix string) (*RawSource, error)

func (*RawSource) NextReader

func (rs *RawSource) NextReader() (pdk.NamedReadCloser, error)

type Source

type Source struct {
	// contains filtered or unexported fields
}

Source is a pdk.Source which reads data from S3.

func NewSource

func NewSource(opts ...SrcOption) (s *Source, err error)

NewSource returns a new Source with the options applied. It is hardcoded to read line separated json objects. This is deprecated... consider using RawSource and feeding that to (e.g) json.NewSourceFromRawSource.

func (*Source) Record

func (s *Source) Record() (rec interface{}, err error)

Record parses the next JSON object from the current file in the bucket, or moves to the next file and parses and returns the first json object. A map[string]interface{} will be returned unless there is an error.

type SrcOption

type SrcOption func(s *Source)

SrcOption is a functional option type for s3.Source.

func OptSrcBucket

func OptSrcBucket(bucket string) SrcOption

OptSrcBucket is a SrcOption which sets the S3 bucket for a Source.

func OptSrcBufSize

func OptSrcBufSize(bufsize int) SrcOption

OptSrcBufSize sets the number of records to buffer while waiting for Record to be called.

func OptSrcPrefix

func OptSrcPrefix(prefix string) SrcOption

OptSrcPrefix tells the source to list only the objects in the bucket that match the specified prefix.

func OptSrcRegion

func OptSrcRegion(region string) SrcOption

OptSrcRegion is a SrcOption which sets the AWS region for a Source.

func OptSrcSubjectAt

func OptSrcSubjectAt(key string) SrcOption

OptSrcSubjectAt tells the source to add a new key to each record whose value will be <S3 bucket>.<S3 object key>#<record number>.