serviceenv

Documentation

Index

• type Configuration
  • func NewConfiguration(config interface{}) *Configuration
• type FeatureFlags
• type GlobalConfiguration
• type PachdFullConfiguration
• type PachdSpecificConfiguration
• type ServiceEnv
  • func InitPachOnlyEnv(config *Configuration) *ServiceEnv
  • func InitServiceEnv(config *Configuration) *ServiceEnv
  • func InitWithKube(config *Configuration) *ServiceEnv
  • func (env *ServiceEnv) GetEtcdClient() *etcd.Client
  • func (env *ServiceEnv) GetKubeClient() *kube.Clientset
  • func (env *ServiceEnv) GetPachClient(ctx context.Context) *client.APIClient
• type StorageConfiguration
• type WorkerFullConfiguration
• type WorkerSpecificConfiguration

Types

type Configuration

type Configuration struct {
	*GlobalConfiguration
	*PachdSpecificConfiguration
	*WorkerSpecificConfiguration
}

Configuration is the generic configuration structure used to access configuration fields.

func NewConfiguration

func NewConfiguration(config interface{}) *Configuration

NewConfiguration creates a generic configuration from a specific type of configuration.

type FeatureFlags

type FeatureFlags struct {
	NewStorageLayer bool `env:"NEW_STORAGE_LAYER,default=false"`
}

FeatureFlags contains the configuration for feature flags.

type GlobalConfiguration

type GlobalConfiguration struct {
	FeatureFlags
	EtcdHost      string `env:"ETCD_SERVICE_HOST,required"`
	EtcdPort      string `env:"ETCD_SERVICE_PORT,required"`
	PPSWorkerPort uint16 `env:"PPS_WORKER_GRPC_PORT,default=80"`
	Port          uint16 `env:"PORT,default=650"`
	HTTPPort      uint16 `env:"HTTP_PORT,default=652"`
	PeerPort      uint16 `env:"PEER_PORT,default=653"`
	PPSEtcdPrefix string `env:"PPS_ETCD_PREFIX,default=pachyderm_pps"`
	Namespace     string `env:"NAMESPACE,default=default"`
	StorageRoot   string `env:"PACH_ROOT,default=/pach"`
}

GlobalConfiguration contains the global configuration.

type PachdFullConfiguration

type PachdFullConfiguration struct {
	GlobalConfiguration
	PachdSpecificConfiguration
}

PachdFullConfiguration contains the full pachd configuration.

type PachdSpecificConfiguration

type PachdSpecificConfiguration struct {
	StorageConfiguration
	NumShards                  uint64 `env:"NUM_SHARDS,default=32"`
	StorageBackend             string `env:"STORAGE_BACKEND,default="`
	StorageHostPath            string `env:"STORAGE_HOST_PATH,default="`
	EtcdPrefix                 string `env:"ETCD_PREFIX,default="`
	PFSEtcdPrefix              string `env:"PFS_ETCD_PREFIX,default=pachyderm_pfs"`
	AuthEtcdPrefix             string `env:"PACHYDERM_AUTH_ETCD_PREFIX,default=pachyderm_auth"`
	EnterpriseEtcdPrefix       string `env:"PACHYDERM_ENTERPRISE_ETCD_PREFIX,default=pachyderm_enterprise"`
	KubeAddress                string `env:"KUBERNETES_PORT_443_TCP_ADDR,required"`
	Metrics                    bool   `env:"METRICS,default=true"`
	Init                       bool   `env:"INIT,default=false"`
	BlockCacheBytes            string `env:"BLOCK_CACHE_BYTES,default=1G"`
	PFSCacheSize               string `env:"PFS_CACHE_SIZE,default=0"`
	WorkerImage                string `env:"WORKER_IMAGE,default="`
	WorkerSidecarImage         string `env:"WORKER_SIDECAR_IMAGE,default="`
	WorkerImagePullPolicy      string `env:"WORKER_IMAGE_PULL_POLICY,default="`
	LogLevel                   string `env:"LOG_LEVEL,default=info"`
	IAMRole                    string `env:"IAM_ROLE,default="`
	ImagePullSecret            string `env:"IMAGE_PULL_SECRET,default="`
	NoExposeDockerSocket       bool   `env:"NO_EXPOSE_DOCKER_SOCKET,default=false"`
	ExposeObjectAPI            bool   `env:"EXPOSE_OBJECT_API,default=false"`
	MemoryRequest              string `env:"PACHD_MEMORY_REQUEST,default=1T"`
	WorkerUsesRoot             bool   `env:"WORKER_USES_ROOT,default=true"`
	S3GatewayPort              uint16 `env:"S3GATEWAY_PORT,default=600"`
	DeploymentID               string `env:"CLUSTER_DEPLOYMENT_ID,default="`
	RequireCriticalServersOnly bool   `env:"REQUIRE_CRITICAL_SERVERS_ONLY",default=false"`
}

PachdSpecificConfiguration contains the pachd specific configuration.

type ServiceEnv

type ServiceEnv struct {
	*Configuration
	// contains filtered or unexported fields
}

ServiceEnv is a struct containing connections to other services in the cluster. In pachd, there is only one instance of this struct, but tests may create more, if they want to create multiple pachyderm "clusters" served in separate goroutines.

func InitPachOnlyEnv

func InitPachOnlyEnv(config *Configuration) *ServiceEnv

InitPachOnlyEnv initializes this service environment. This dials a GRPC connection to pachd only (in a background goroutine), and creates the template pachClient used by future calls to GetPachClient.

This call returns immediately, but GetPachClient will block until the client is ready.

func InitServiceEnv

func InitServiceEnv(config *Configuration) *ServiceEnv

InitServiceEnv initializes this service environment. This dials a GRPC connection to pachd and etcd (in a background goroutine), and creates the template pachClient used by future calls to GetPachClient.

This call returns immediately, but GetPachClient and GetEtcdClient block until their respective clients are ready.

func InitWithKube

func InitWithKube(config *Configuration) *ServiceEnv

InitWithKube is like InitServiceEnv, but also assumes that it's run inside a kubernetes cluster and tries to connect to the kubernetes API server.

func (*ServiceEnv) GetEtcdClient

func (env *ServiceEnv) GetEtcdClient() *etcd.Client

GetEtcdClient returns the already connected etcd client without modification.

func (*ServiceEnv) GetKubeClient

func (env *ServiceEnv) GetKubeClient() *kube.Clientset

GetKubeClient returns the already connected Kubernetes API client without modification.

func (*ServiceEnv) GetPachClient

func (env *ServiceEnv) GetPachClient(ctx context.Context) *client.APIClient

GetPachClient returns a pachd client with the same authentication credentials and cancellation as 'ctx' (ensuring that auth credentials are propagated through downstream RPCs).

Functions that receive RPCs should call this to convert their RPC context to a Pachyderm client, and internal Pachyderm calls should accept clients returned by this call.

(Warning) Do not call this function during server setup unless it is in a goroutine. A Pachyderm client is not available until the server has been setup.

type StorageConfiguration

type StorageConfiguration struct {
	StorageMemoryThreshold        int64 `env:"STORAGE_MEMORY_THRESHOLD"`
	StorageShardThreshold         int64 `env:"STORAGE_SHARD_THRESHOLD"`
	StorageLevelZeroSize          int64 `env:"STORAGE_LEVEL_ZERO_SIZE"`
	StorageLevelSizeBase          int   `env:"STORAGE_LEVEL_SIZE_BASE"`
	StorageUploadConcurrencyLimit int   `env:"STORAGE_UPLOAD_CONCURRENCY_LIMIT,default=100"`
}

StorageConfiguration contains the storage configuration.

type WorkerFullConfiguration

type WorkerFullConfiguration struct {
	GlobalConfiguration
	WorkerSpecificConfiguration
}

WorkerFullConfiguration contains the full worker configuration.

type WorkerSpecificConfiguration

type WorkerSpecificConfiguration struct {
	// Worker gets its own IP here, via the k8s downward API. It then writes that
	// IP back to etcd so that pachd can discover it
	PPSWorkerIP string `env:"PPS_WORKER_IP,required"`
	// The name of the pipeline that this worker belongs to
	PPSPipelineName string `env:"PPS_PIPELINE_NAME,required"`
	// The ID of the commit that contains the pipeline spec.
	PPSSpecCommitID string `env:"PPS_SPEC_COMMIT,required"`
	// The name of this pod
	PodName string `env:"PPS_POD_NAME,required"`
}

WorkerSpecificConfiguration contains the worker specific configuration.