storage

Documentation

Index

• Constants
• Variables
• func Register(name string, factory DataStoreFactory)
• type DataStore
  • func CreateDataStore(dataStoreName string, conf map[string]string) (DataStore, error)
  • func NewInMemObjectStore(conf map[string]string) (DataStore, error)
  • func NewNotImplementedStorage(conf map[string]string) (DataStore, error)
  • func NewS3ObjectStore(conf map[string]string) (DataStore, error)
  • func NewSwiftObjectStore(conf map[string]string) (DataStore, error)
  • func NewVolumeMountStorage(conf map[string]string) (DataStore, error)
• type DataStoreFactory

Constants

const (

	// UsernameKey is the key for the user name
	UsernameKey = "user_name"
	// PasswordKey is the key for the password
	PasswordKey = "password"
	// AuthURLKey is the key for the authentication URL
	AuthURLKey = "auth_url"
	// DomainKey is the key for the domain name
	DomainKey = "domain_name"
	// RegionKey is the key for the region name
	RegionKey = "region"
	// ProjectKey is the key for the project ID
	ProjectKey = "project_id"
	// StorageType is the key for the storage type
	StorageType = "type"
)

const (
	// DataStoreTypeS3 is the type string for the S3-based object store.
	DataStoreTypeMountVolume = "mount_volume"
	DataStoreTypeMountCOSS3  = "mount_cos"
	DataStoreTypeS3          = "s3_datastore"
)

const (
	// DataStoreTypeInMemory is the name that represents an in-memory data store
	DataStoreTypeInMemory = "inmem_datastore"
)

const (
	// DataStoreTypeNotImplemented is the name that represents an in-memory data store
	DataStoreTypeNotImplemented = "not_implemented_datastore"
)

const (
	// DataStoreTypeSwift is the type string for the Swift-based object store.
	DataStoreTypeSwift = "swift_datastore"
)

Variables

var ErrNotConnected = errors.New("not connected to object store")

ErrNotConnected is thrown when the ObjectStore is not connected (i.e., authenticated)

Functions

func Register

func Register(name string, factory DataStoreFactory)

Register enables registration of available data stores.

Types

type DataStore

type DataStore interface {
	Connect() error
	Disconnect()
	ContainerExists(name string) (bool, error)
	UploadArchive(container string, object string, payload []byte) error
	DownloadArchive(container string, object string) ([]byte, error)
	DeleteArchive(container string, object string) error
	GetTrainedModelSize(path string, numLearners int32) (int64, error)
	DownloadTrainedModelAsZipStream(path string, numLearners int32, writer io.Writer) error
	DownloadTrainedModelLogFile(path string, numLearners int32, learnerIndex int32, filename string, writer io.Writer) error
}

DataStore is a minimal interface for interacting with data stores such as IBM ObjectStore or other backend for uploading and downloading DL models, logs, etc.

func CreateDataStore

func CreateDataStore(dataStoreName string, conf map[string]string) (DataStore, error)

CreateDataStore is a factory for instantiating the according to the configuration.

func NewInMemObjectStore

func NewInMemObjectStore(conf map[string]string) (DataStore, error)

NewInMemObjectStore implements an in-memor object store for testing.

func NewNotImplementedStorage

func NewNotImplementedStorage(conf map[string]string) (DataStore, error)

NewNotImplementedStorage implements an in-memor object store for testing.

func NewS3ObjectStore

func NewS3ObjectStore(conf map[string]string) (DataStore, error)

NewS3ObjectStore creates a new connector for the IBM S3 based object store (https://ibm-public-cos.github.io). If `session` is set to nil, the configuration is read via viper. If conn is provided, it will use it for connecting to the provided object store.

func NewSwiftObjectStore

func NewSwiftObjectStore(conf map[string]string) (DataStore, error)

NewSwiftObjectStore creates a new connector for the object store. If `conn` is set to nil, the configuration is read via viper. If conn is provided, it will use it for connecting to the provided object store.

func NewVolumeMountStorage

func NewVolumeMountStorage(conf map[string]string) (DataStore, error)

NewVolumeMountStorage implements an in-memor object store for testing.

type DataStoreFactory

type DataStoreFactory func(conf map[string]string) (DataStore, error)

DataStoreFactory is a simple factory for creating a specific