config

package
v0.0.0-...-20989ac Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 15, 2022 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewConfigValidator 

func NewConfigValidator() (*validator.Validate, error)

func StringToQuantityHookFunc 

func StringToQuantityHookFunc() mapstructure.DecodeHookFunc

StringToQuantityHookFunc converts string to quantity type. This function is required since viper uses mapstructure to unmarshal values. https://github.com/spf13/viper#unmarshaling https://pkg.go.dev/github.com/mitchellh/mapstructure#DecodeHookFunc

Types

type AlertConfig 

type AlertConfig struct {
	Enabled bool
	GitLab  *GitlabConfig
	// PlaybookURL is the URL that contains documentation on how to resolve triggered alerts
	PlaybookURL string
	// DashboardURLTemplate is a template for grafana dashboard URL that shows router metrics.
	// The template accepts go-template format and will be executed with dashboardURLValue which has
	// the following fields: Environment, Cluster, Project, Router, Version.
	DashboardURLTemplate string
}

type AuthorizationConfig 

type AuthorizationConfig struct {
	Enabled bool
	URL     string
}

AuthorizationConfig captures the config for auth using mlp authz

type BatchEnsemblingConfig 

type BatchEnsemblingConfig struct {
	// Unfortunately if Enabled is false and user sets JobConfig/RunnerConfig/ImageBuildingConfig wrongly
	// it will still error out.
	Enabled             bool
	JobConfig           *JobConfig           `validate:"required_if=Enabled True"`
	RunnerConfig        *RunnerConfig        `validate:"required_if=Enabled True"`
	ImageBuildingConfig *ImageBuildingConfig `validate:"required_if=Enabled True"`
	LoggingURLFormat    *string
	MonitoringURLFormat *string
}

BatchEnsemblingConfig captures the config related to the running of batch runners

type ClusterConfig 

type ClusterConfig struct {
	// InClusterConfig is a flag if the service account is provided in Kubernetes
	// and has the relevant credentials to handle all cluster operations.
	InClusterConfig bool
	// VaultConfig is required if InClusterConfig is false.
	VaultConfig *VaultConfig `validate:"required_without=InClusterConfig"`
}

ClusterConfig contains the cluster controller information. Supported features are in cluster configuration and Kubernetes client CA certificates.

type Config 

type Config struct {
	Port                          int `validate:"required"`
	LogLevel                      string
	AllowedOrigins                []string
	AuthConfig                    *AuthorizationConfig
	BatchEnsemblingConfig         BatchEnsemblingConfig         `validate:"required"`
	EnsemblerServiceBuilderConfig EnsemblerServiceBuilderConfig `validate:"required"`
	DbConfig                      *DatabaseConfig               `validate:"required"`
	DeployConfig                  *DeploymentConfig             `validate:"required"`
	SparkAppConfig                *SparkAppConfig               `validate:"required"`
	RouterDefaults                *RouterDefaults               `validate:"required"`
	KubernetesLabelConfigs        *KubernetesLabelConfigs       `validate:"required"`
	KnativeServiceDefaults        *KnativeServiceDefaults
	NewRelicConfig                newrelic.Config
	Sentry                        sentry.Config
	ClusterConfig                 ClusterConfig `validate:"required"`
	TuringEncryptionKey           string        `validate:"required"`
	AlertConfig                   *AlertConfig
	MLPConfig                     *MLPConfig `validate:"required"`
	TuringUIConfig                *SinglePageApplicationConfig
	OpenapiConfig                 *OpenapiConfig
	// Experiment specifies the JSON configuration to set up experiment managers and runners.
	//
	// The configuration follows the following format to support different experiment engines
	// and to allow custom configuration for different engines:
	// { "<experiment_engine>": <experiment_engine_config>, ... }
	//
	// For example:
	// { "experiment_engine_a": {"client": "foo"}, "experiment_engine_b": {"apikey": 12} }
	Experiment map[string]EngineConfig
}

Config is used to parse and store the environment configs

func Load 

func Load(filepaths ...string) (*Config, error)

Load creates a Config object from default config values, config files and environment variables. Load accepts config files as the argument. JSON and YAML format are both supported.

If multiple config files are provided, the subsequent config files will override the config values from the config files loaded earlier.

These config files will override the default config values (refer to setDefaultValues function) and can be overridden by the values from environment variables. Nested keys in the config can be set from environment variable name separed by "_". For instance the config value for "DbConfig.Port" can be overridden by environment variable name "DBCONFIG_PORT". Note that all environment variable names must be upper case.

If no config file is provided, only the default config values and config values from environment varibales will be loaded.

Refer to example.yaml for an example of config file.

func (*Config) ListenAddress 

func (c *Config) ListenAddress() string

ListenAddress returns the Turing Api app's port

func (*Config) Validate 

func (c *Config) Validate() error

type DatabaseConfig 

type DatabaseConfig struct {
	Host             string `validate:"required"`
	Port             int    `validate:"required"`
	User             string `validate:"required"`
	Password         string `validate:"required"`
	Database         string `validate:"required"`
	MigrationsFolder string `validate:"required"`
}

DatabaseConfig config captures the Turing database config

type DefaultEnsemblingJobConfigurations 

type DefaultEnsemblingJobConfigurations struct {
	// BatchEnsemblingJobResources contains the resources delared to run the ensembling job.
	BatchEnsemblingJobResources openapi.EnsemblingResources
	// SparkConfigAnnotations contains the Spark configurations
	SparkConfigAnnotations map[string]string
}

DefaultEnsemblingJobConfigurations contains the default configurations applied to the ensembling job.

type DeploymentConfig 

type DeploymentConfig struct {
	EnvironmentType string        `validate:"required"`
	Timeout         time.Duration `validate:"required"`
	DeletionTimeout time.Duration `validate:"required"`
	MaxCPU          Quantity      `validate:"required"`
	MaxMemory       Quantity      `validate:"required"`
}

DeploymentConfig captures the config related to the deployment of the turing routers

type EngineConfig 

type EngineConfig map[string]interface{}

type EnsemblerServiceBuilderConfig 

type EnsemblerServiceBuilderConfig struct {
	DefaultEnvironment  string               `validate:"required"`
	ImageBuildingConfig *ImageBuildingConfig `validate:"required"`
}

EnsemblerServiceConfig captures the config related to the build and running of ensembler services (real-time)

type ExperimentEnginePluginConfig 

type ExperimentEnginePluginConfig struct {
	Image string `json:"image" validate:"required"`
}

type FluentdConfig 

type FluentdConfig struct {
	// Image to use for fluentd deployments, in the format registry/repository:version.
	Image string
	// Fluentd tag for logs
	Tag string
	// Flush interval seconds - value determined by load job frequency to BQ
	FlushIntervalSeconds int
}

FluentdConfig captures the defaults used by the Turing Router when Fluentd is enabled

type GitlabConfig 

type GitlabConfig struct {
	BaseURL    string
	Token      string
	ProjectID  string
	Branch     string
	PathPrefix string
}

type ImageBuildingConfig 

type ImageBuildingConfig struct {
	// BuildNamespace contains the Kubernetes namespace it should be built in.
	BuildNamespace string `validate:"required"`
	// BuildTimeoutDuration is the Kubernetes Job timeout duration.
	BuildTimeoutDuration time.Duration `validate:"required"`
	// DestinationRegistry is the registry of the newly built ensembler image.
	DestinationRegistry string `validate:"required"`
	// BaseImageRef is the image name of the base ensembler image built from the
	// engines/pyfunc-ensembler-*/Dockerfile. It's a map of image names, per
	// minor python version supported by the SDK.
	BaseImageRef map[string]string `validate:"required"`
	// KanikoConfig contains the configuration related to the kaniko executor image builder.
	KanikoConfig KanikoConfig `validate:"required"`
}

ImageBuildingConfig contains the information regarding the image builder and the image buildee.

type JobConfig 

type JobConfig struct {
	// DefaultEnvironment is the environment used for image building and running the batch ensemblers.
	DefaultEnvironment string `validate:"required"`
	// DefaultConfigurations contains the default configurations applied to the ensembling job.
	// The user (the person who calls the API) is free to override/append the default values.
	DefaultConfigurations DefaultEnsemblingJobConfigurations `validate:"required"`
}

JobConfig captures the config related to the ensembling batch jobs.

type KafkaConfig 

type KafkaConfig struct {
	// Producer Config - Max message byte to send to broker
	MaxMessageBytes int
	// Producer Config - Compression Type of message
	CompressionType string
}

KafkaConfig captures the defaults used by Turing Router when result logger is set to kafka

type KanikoConfig 

type KanikoConfig struct {
	// BuildContextURI contains the image build context, which should be engines/batch-ensembler/
	// The forms supported are listed here https://github.com/GoogleContainerTools/kaniko#kaniko-build-contexts
	BuildContextURI string `validate:"required"`
	// DockerfileFilePath contains where the Dockerfile is
	DockerfileFilePath string `validate:"required"`
	// Image is the Kaniko image
	Image string `validate:"required"`
	// ImageVersion is the version tag of the Kaniko image
	ImageVersion string `validate:"required"`
	// ResourceRequestsLimits is the resources required by Kaniko executor.
	ResourceRequestsLimits ResourceRequestsLimits `validate:"required"`
}

KanikoConfig provides the configuration used for the Kaniko image.

type KnativeServiceDefaults 

type KnativeServiceDefaults struct {
	TargetConcurrency               int
	QueueProxyResourcePercentage    int
	UserContainerLimitRequestFactor float64
}

KnativeServiceDefaults captures some of the configurable defaults specific to Knative services

type KubernetesLabelConfigs 

type KubernetesLabelConfigs struct {
	// LabelPrefix is the prefix used for tagging kubernetes components.
	// Default is an empty string which means your tags will look something like this:
	//   team: teen-titans
	//   stream: nile
	//   environment: dev
	//   orchestrator: turing
	//   app: my-model-app
	LabelPrefix string
	// Environment is the value for the environment label
	Environment string `validate:"required"`
}

KubernetesLabelConfigs are the configurations for labeling

type MLPConfig 

type MLPConfig struct {
	MerlinURL        string `validate:"required"`
	MLPURL           string `validate:"required"`
	MLPEncryptionKey string `validate:"required"`
}

MLPConfig captures the configuration used to connect to the Merlin/MLP API servers

type OpenapiConfig 

type OpenapiConfig struct {
	// ValidationEnabled specifies whether to use OpenAPI validation middleware,
	// which validates HTTP requests against the spec.
	ValidationEnabled bool
	// SpecFile specifies the file path containing OpenAPI v3 spec
	SpecFile string
	// Config file to be used for serving swagger.
	SwaggerUIConfig *SinglePageApplicationConfig
	// Where the merged spec file yaml should be saved.
	MergedSpecFile string
	// Optional. Overrides the file before running the Swagger UI.
	SpecOverrideFile *string
}

OpenapiConfig contains the settings for the OpenAPI specs used for validation and Swagger UI

func (*OpenapiConfig) GenerateSpecFile 

func (c *OpenapiConfig) GenerateSpecFile() error

GenerateSpecFile gets the spec data and writes to a file

func (*OpenapiConfig) SpecData 

func (c *OpenapiConfig) SpecData() ([]byte, error)

SpecData returns a byte slice of the combined yaml files

type Quantity 

type Quantity resource.Quantity

Quantity is an alias for resource.Quantity

func (*Quantity) Decode 

func (qty *Quantity) Decode(value string) (err error)

Decode parses the Quantity config and checks that it is non-empty and can be converted into a valid resource.Quantity.

func (*Quantity) MarshalJSON 

func (qty *Quantity) MarshalJSON() ([]byte, error)

MarshalJSON implements the json.Marshaller interface so that we will get the expected JSON string representation when marshalling a quantity object.

type Resource 

type Resource struct {
	CPU    string `validate:"required"`
	Memory string `validate:"required"`
}

Resource contains the Kubernetes resource request and limits

type ResourceRequestsLimits 

type ResourceRequestsLimits struct {
	Requests Resource `validate:"required"`
	Limits   Resource `validate:"required"`
}

ResourceRequestsLimits contains the Kubernetes resource request and limits for kaniko

type RouterDefaults 

type RouterDefaults struct {
	// Turing router image, in the format registry/repository:version.
	Image string `validate:"required"`
	// Enable Fiber debug logging
	FiberDebugLogEnabled bool
	// Enable router custom metrics
	CustomMetricsEnabled bool
	// Enable Jaeger Tracing
	JaegerEnabled bool
	// Jaeger collector endpoint. If JaegerEnabled is true, this value
	// must be set.
	JaegerCollectorEndpoint string
	// Router log level
	LogLevel string `validate:"required"`
	// Fluentd config for the router
	FluentdConfig       *FluentdConfig
	MonitoringURLFormat *string
	// Configuration of experiment engine plugins, that consists of experiment engine name
	// and the image that contains the plugin implementation.
	//
	// Example:
	// ExperimentEnginePlugins:
	// 	red-exp-engine:
	//	  Image: ghcr.io/myproject/red-exp-engine-plugin:v0.0.1
	// 	blue-exp-engine:
	//	  Image: ghcr.io/myproject/blue-exp-engine-plugin:v0.0.1
	ExperimentEnginePlugins map[string]*ExperimentEnginePluginConfig `validate:"dive"`
	// Kafka Configuration. If result logging is using Kafka
	KafkaConfig *KafkaConfig
}

RouterDefaults contains default configuration for routers deployed by this instance of the Turing API.

type RunnerConfig 

type RunnerConfig struct {
	// TimeInterval is the interval between job firings
	TimeInterval time.Duration `validate:"required"`
	// RecordsToProcessInOneIteration dictates the number of batch ensembling jobs to be queried at once.
	RecordsToProcessInOneIteration int `validate:"required"`
	// MaxRetryCount is the number of retries the batch ensembler runner should try before giving up.
	MaxRetryCount int `validate:"required"`
}

RunnerConfig contains the batch runner configurations

type SinglePageApplicationConfig 

type SinglePageApplicationConfig struct {
	// Specifies the directory, that contains static files that will be served as an SPA
	ServingDirectory string
	// Defines the relative path under which the application will be accessible.
	ServingPath string
}

SinglePageApplicationConfig holds configuration required for serving SPAs

type SparkAppConfig 

type SparkAppConfig struct {
	NodeSelector                   map[string]string
	CorePerCPURequest              float64 `validate:"required"`
	CPURequestToCPULimit           float64 `validate:"required"`
	SparkVersion                   string  `validate:"required"`
	TolerationName                 *string
	SubmissionFailureRetries       int32  `validate:"required"`
	SubmissionFailureRetryInterval int64  `validate:"required"`
	FailureRetries                 int32  `validate:"required"`
	FailureRetryInterval           int64  `validate:"required"`
	PythonVersion                  string `validate:"required"`
	TTLSecond                      int64  `validate:"required"`
}

SparkAppConfig contains the infra configurations that is unique to the user's Kubernetes

type VaultConfig 

type VaultConfig struct {
	Address string `validate:"required"`
	Token   string `validate:"required"`
}

VaultConfig captures the config for connecting to the Vault server

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.