config

Documentation

Index

• type Config
  • func (conf *Config) FromString(stringified string) (err error)
  • func (conf Config) MapQueue(queue string) (target string, err error)
  • func (conf Config) ReturnMatchingFiles(files []string) (matches []string)
  • func (conf Config) Validate() error
• type Downloader
  • func NewDownloader(config *Config, minio minio.Config) Downloader
  • func (cd *Downloader) Start(wg *sync.WaitGroup)
  • func (cd *Downloader) StartBlocking()

Types

type Config

type Config struct {
	QueueMapping map[string]string `json:"queue_mapping"` // nillable, transformation-step output -> message queue
	ConfigFiles  map[string]string `json:"config_files"`
}

Config is the struct holding configurable information This can be set via the environment variable ITERUM_CONFIG

func (*Config) FromString

func (conf *Config) FromString(stringified string) (err error)

FromString converts a string value into an instance of Config and also does validation

func (Config) MapQueue

func (conf Config) MapQueue(queue string) (target string, err error)

MapQueue tries to map a transformation step output to a target MQ. If either the map is nil or the key does not exist it returns an error

func (Config) ReturnMatchingFiles

func (conf Config) ReturnMatchingFiles(files []string) (matches []string)

ReturnMatchingFiles returns the list of files matching the ConfigSelectors of conf

func (Config) Validate

func (conf Config) Validate() error

Validate does validation of a config struct, ensuring that it's members contain valid values

type Downloader

type Downloader struct {
	Config *Config
	Minio  minio.Config
	// contains filtered or unexported fields
}

Downloader is the structure responsible for downloading the config files used by the fragmenter to properly fragment data files

func NewDownloader

func NewDownloader(config *Config, minio minio.Config) Downloader

NewDownloader instantiates a new config downloader without starting it

func (*Downloader) Start

func (cd *Downloader) Start(wg *sync.WaitGroup)

Start is an asyncrhonous alternative to StartBlocking by spawning a goroutine

func (*Downloader) StartBlocking

func (cd *Downloader) StartBlocking()

StartBlocking starts the process of downloading the config files