datastore

package
v0.0.0-...-e545f54 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 6, 2014 License: BSD-3-Clause Imports: 11 Imported by: 0

Documentation

Overview

Package datastore provides versioning and persisting supported data types using one of the supported storage engines.

Index

Constants

View Source 
const DataKeyIndexOffset = dvid.LocalIDSize*2 + dvid.LocalID32Size + 1

The offset to the Index in bytes of a DataKey bytes representation

View Source 
const (
	Version = "0.7"
)

Variables

View Source 
var CompiledTypes = map[UrlString]TypeService{}

CompiledTypes is the set of registered data types compiled into DVID and held as a global variable initialized at runtime.

View Source 
var (
	HelpRequest = Request{Command: []string{"help"}}
)

Functions

func CompiledTypeChart 

func CompiledTypeChart() string

CompiledTypeChart returns a chart (names/urls) of data types compiled into this DVID.

func CompiledTypeNames 

func CompiledTypeNames() string

CompiledTypeNames returns a list of data type names compiled into this DVID.

func CompiledTypeUrls 

func CompiledTypeUrls() string

CompiledTypeUrls returns a list of data type urls supported by this DVID.

func Init 

func Init(directory string, create bool, config dvid.Config) error

Init creates a key-value datastore using default arguments. Datastore configuration is stored as key/values in the datastore and also in a human-readable config file in the datastore directory.

func KeyToChunkIndexer 

func KeyToChunkIndexer(key storage.Key) (dvid.ChunkIndexer, error)

KeyToChunkIndexer takes a Key and returns an implementation of a ChunkIndexer if possible.

func MaxDatasetKey 

func MaxDatasetKey() storage.Key

func MinDatasetKey 

func MinDatasetKey() storage.Key

func Open 

func Open(path string) (s *Service, openErr *OpenError)

Open opens a DVID datastore at the given path (directory, url, etc) and returns a Service that allows operations on that datastore.

func RegisterDatatype 

func RegisterDatatype(t TypeService)

RegisterDatatype registers a data type for DVID use.

func Versions 

func Versions() string

Versions returns a chart of version identifiers for data types and and DVID's datastore fixed at compile-time for this DVID executable

Types

type Data 

type Data struct {
	*DataID
	TypeService

	// Compression of serialized data, e.g., the value in a key-value.
	Compression dvid.Compression

	// Checksum approach for serialized data.
	Checksum dvid.Checksum

	// If false (default), we allow changes along nodes.
	Unversioned bool
}

Data is an instance of a data type with some identifiers and it satisfies a DataService interface. Each Data is dataset-specific.

func NewDataService 

func NewDataService(id *DataID, t TypeService, config dvid.Config) (*Data, error)

NewDataService returns a Data instance. If the configuration doesn't explicitly set compression and checksum, LZ4 and the default checksum (chosen by -crc32 flag) is used.

func (*Data) DataKey 

func (d *Data) DataKey(versionID dvid.VersionLocalID, index dvid.Index) *DataKey

DataKey returns a DataKey for this data given a local version and a data-specific Index.

func (*Data) IsVersioned 

func (d *Data) IsVersioned() bool

func (*Data) ModifyConfig 

func (d *Data) ModifyConfig(config dvid.Config) error

func (*Data) UnknownCommand 

func (d *Data) UnknownCommand(request Request) error

func (*Data) UseChecksum 

func (d *Data) UseChecksum() dvid.Checksum

func (*Data) UseCompression 

func (d *Data) UseCompression() dvid.Compression

func (*Data) VersionMutex 

func (d *Data) VersionMutex(versionID dvid.VersionLocalID) *sync.Mutex

VersionMutex returns a Mutex that is specific for data at a particular version.

type DataAvail 

type DataAvail int

DataAvail gives the availability of data within a node or whether parent nodes must be traversed to check for key/value pairs. 

const (
	// All key/value pairs are available within this node.
	DataComplete DataAvail = iota

	// For any range query, we must also traverse this node's ancestors in the DAG
	// up to any NodeComplete ancestor.  Used if a node is marked as archived.
	DataDelta

	// Queries are redirected to Root since this is unversioned.
	DataRoot

	// Data has been explicitly deleted at this node and is no longer available.
	DataDeleted
)

type DataID 

type DataID struct {
	Name   dvid.DataString
	ID     dvid.DataLocalID
	DsetID dvid.DatasetLocalID
}

DataID identifies data within a DVID server.

func (DataID) DataName 

func (id DataID) DataName() dvid.DataString

func (DataID) DatasetID 

func (id DataID) DatasetID() dvid.DatasetLocalID

func (DataID) LocalID 

func (id DataID) LocalID() dvid.DataLocalID

type DataKey 

type DataKey struct {
	// The DVID server-specific 32-bit ID for a dataset.
	Dataset dvid.DatasetLocalID

	// The DVID server-specific data index that is unique per dataset.
	Data dvid.DataLocalID

	// The DVID server-specific version index that is fewer bytes than a
	// complete UUID and unique per dataset.
	Version dvid.VersionLocalID

	// The datatype-specific (usually spatiotemporal) index that allows partitioning
	// of the data.  In the case of voxels, this could be a (x, y, z) coordinate
	// packed into a slice of bytes.
	Index dvid.Index
}

DataKey holds DVID-centric data like shortened version/UUID, data set, and index identifiers and that follow a convention of how to collapse those identifiers into a []byte key. Ideally, we'd like to keep Key within the datastore package and have storage independent of DVID concepts, but in order to optimize the layout of data in some storage engines, the backend drivers need the additional DVID information. For example, Couchbase allows configuration at the bucket level (RAM cache, CPUs) and datasets could be placed in different buckets.

func (*DataKey) Bytes 

func (key *DataKey) Bytes() (b []byte)

Bytes returns a slice of bytes derived from the concatenation of the key elements.

func (*DataKey) BytesString 

func (key *DataKey) BytesString() string

Bytes returns a string derived from the concatenation of the key elements.

func (*DataKey) BytesToKey 

func (key *DataKey) BytesToKey(b []byte) (storage.Key, error)

BytesToKey returns a DataKey given a slice of bytes

func (*DataKey) KeyType 

func (key *DataKey) KeyType() storage.KeyType

func (*DataKey) String 

func (key *DataKey) String() string

String returns a hexadecimal representation of the bytes encoding a key so it is readable on a terminal.

type DataService 

type DataService interface {
	TypeService

	// DataName returns the name of the data (e.g., grayscale data that is grayscale8 data type).
	DataName() dvid.DataString

	// IsVersioned returns true if this data can be mutated across versions.  If the data is
	// not versioned, only one copy of data is kept across all versions nodes in a dataset.
	IsVersioned() bool

	// ModifyConfig modifies a configuration in a type-specific way.
	ModifyConfig(config dvid.Config) error

	// DoRPC handles command line and RPC commands specific to a data type
	DoRPC(request Request, reply *Response) error

	// DoHTTP handles HTTP requests specific to a data type
	DoHTTP(uuid dvid.UUID, w http.ResponseWriter, r *http.Request) error

	// Returns standard error response for unknown commands
	UnknownCommand(r Request) error
}

DataService is an interface for operations on arbitrary data that use a supported TypeService. Chunk handlers are allocated at this level, so an implementation can own a number of goroutines.

DataService operations are completely type-specific, and each datatype handles operations through RPC (DoRPC) and HTTP (DoHTTP). TODO -- Add SPDY as wrapper to HTTP.

type Dataset 

type Dataset struct {
	*VersionDAG

	// Alias is an optional user-supplied string to identify this dataset
	// in a more friendly way than a UUID.  There are no guarantees that
	// this string is unique across all datasets.
	Alias string

	// DatasetID is the 32-bit identifier that is DVID server-specific.
	DatasetID dvid.DatasetLocalID

	// DataMap keeps the dataset-specific names for instances of data types
	// in this dataset.  Although this is public, access should be through
	// the DataService(name) function to also match possible prefix data names,
	// e.g., multichannel types.
	DataMap map[dvid.DataString]DataService
}

Dataset is a set of Data with an associated version DAG.

func (*Dataset) DataService 

func (dset *Dataset) DataService(name dvid.DataString) (dataservice DataService, err error)

DataService returns a DataService for data of a given name.

func (*Dataset) JSONString 

func (dset *Dataset) JSONString() (jsonStr string, err error)

JSONString returns the JSON for this Data's configuration

func (*Dataset) Key 

func (dset *Dataset) Key() storage.Key

Key returns a Key for this Dataset

func (*Dataset) Put 

func (dset *Dataset) Put(db storage.OrderedKeyValueSetter) error

Put stores a Dataset into a storage engine, overwriting whatever was there before.

func (*Dataset) TypeService 

func (dset *Dataset) TypeService(name dvid.DataString) (t TypeService, err error)

TypeService returns the TypeService underlying data of a given name.

type DatasetKey 

type DatasetKey struct {
	Dataset dvid.DatasetLocalID
}

DatasetKey is an implementation of storage.Key for Dataset persistence.

func (DatasetKey) Bytes 

func (k DatasetKey) Bytes() (b []byte)

func (DatasetKey) BytesString 

func (k DatasetKey) BytesString() string

func (DatasetKey) BytesToKey 

func (k DatasetKey) BytesToKey(b []byte) (storage.Key, error)

func (DatasetKey) KeyType 

func (k DatasetKey) KeyType() storage.KeyType

func (DatasetKey) String 

func (k DatasetKey) String() string

type Datasets 

type Datasets struct {
	// contains filtered or unexported fields
}

Datasets holds information on all the Dataset available.

func (*Datasets) AllJSON 

func (dsets *Datasets) AllJSON() (m []byte, err error)

AllJSON returns JSON of all the datasets information.

func (*Datasets) DataServiceByLocalID 

func (dsets *Datasets) DataServiceByLocalID(id dvid.DatasetLocalID, name dvid.DataString) (DataService, error)

DataServiceByLocalID returns a service for data of a given name under a Dataset referenced by local ID.

func (*Datasets) DataServiceByUUID 

func (dsets *Datasets) DataServiceByUUID(u dvid.UUID, name dvid.DataString) (DataService, error)

DataServiceByUUID returns a service for data of a given name under a Dataset referenced by UUID.

func (*Datasets) DatasetFromLocalID 

func (dsets *Datasets) DatasetFromLocalID(id dvid.DatasetLocalID) (*Dataset, error)

DatasetFromLocalID returns a dataset from a local dataset ID.

func (*Datasets) DatasetFromString 

func (dsets *Datasets) DatasetFromString(str string) (dataset *Dataset, u dvid.UUID, err error)

DatasetFromString returns a dataset from a UUID string. Partial matches are accepted as long as they are unique for a datastore. So if a datastore has nodes with UUID strings 3FA22..., 7CD11..., and 836EE..., we can still find a match even if given the minimum 3 letters. (We don't allow UUID strings of less than 3 letters just to prevent mistakes.)

func (*Datasets) DatasetFromUUID 

func (dsets *Datasets) DatasetFromUUID(u dvid.UUID) (*Dataset, error)

DatasetFromUUID returns a dataset given a UUID.

func (*Datasets) Datatypes 

func (dsets *Datasets) Datatypes() map[UrlString]TypeService

Datatypes returns a map of all unique data types where the key is the unique URL identifying the data type. Since type names can collide across datasets, we do not return the abbreviated data type names.

func (*Datasets) Load 

func (dsets *Datasets) Load(db storage.OrderedKeyValueGetter) (err error)

Load retrieves Datasets and all referenced Dataset from the storage engine.

func (*Datasets) MarshalBinary 

func (dsets *Datasets) MarshalBinary() ([]byte, error)

MarshalBinary fulfills the encoding.BinaryMarshaler interface.

func (*Datasets) MarshalJSON 

func (dsets *Datasets) MarshalJSON() (m []byte, err error)

MarshalJSON returns the JSON of just the list of Dataset.

func (*Datasets) Put 

func (dsets *Datasets) Put(db storage.OrderedKeyValueSetter) error

Put stores Datasets, overwriting whatever was there before.

func (*Datasets) VerifyCompiledTypes 

func (dsets *Datasets) VerifyCompiledTypes() error

VerifyCompiledTypes will return an error if any required data type in the datastore configuration was not compiled into DVID executable. Check is done by more exact URL and not the data type name.

type DatasetsKey 

type DatasetsKey struct{}

DatasetsKey is an implementation of storage.Key for Datasets persistence

func (DatasetsKey) Bytes 

func (k DatasetsKey) Bytes() []byte

func (DatasetsKey) BytesString 

func (k DatasetsKey) BytesString() string

func (DatasetsKey) BytesToKey 

func (k DatasetsKey) BytesToKey(b []byte) (storage.Key, error)

func (DatasetsKey) KeyType 

func (k DatasetsKey) KeyType() storage.KeyType

func (DatasetsKey) String 

func (k DatasetsKey) String() string

type Datatype 

type Datatype struct {
	*DatatypeID

	// A list of interface requirements for the backend datastore
	Requirements *storage.Requirements
}

Datatype is the base struct that satisfies a TypeService and can be embedded in other data types.

func (*Datatype) Help 

func (datatype *Datatype) Help() string

type DatatypeID 

type DatatypeID struct {
	// Data type name and may not be unique.
	Name dvid.TypeString

	// The unique package name that fulfills the DVID Data interface
	Url UrlString

	// The version identifier of this data type code
	Version string
}

DatatypeID uniquely identifies a DVID-supported data type and provides a shorthand name.

func MakeDatatypeID 

func MakeDatatypeID(name dvid.TypeString, url UrlString, version string) *DatatypeID

func (*DatatypeID) DatatypeName 

func (id *DatatypeID) DatatypeName() dvid.TypeString

func (*DatatypeID) DatatypeUrl 

func (id *DatatypeID) DatatypeUrl() UrlString

func (*DatatypeID) DatatypeVersion 

func (id *DatatypeID) DatatypeVersion() string

type Node 

type Node struct {
	*NodeVersion
	*NodeText

	// Avail is used for data compression/deltas in version DAG, depending on
	// type of data (e.g., versioned) and whether nodes are archived or not.
	// If there is no map or data availability is not explicitly set, we use
	// the default for that data, e.g., DataComplete if versioned or DataRoot
	// if unversioned.
	Avail map[dvid.DataString]DataAvail
	// contains filtered or unexported fields
}

Node contains all information needed at each node of the version DAG

type NodeText 

type NodeText struct {
	// Note holds general information on this node.
	Note string

	// Provenance describes the operations performed between the locking of
	// this node's parents and its current state.
	Provenance string
}

NodeText holds provenance and other information useful for analysis. It's possible that these structs could get large if useful provenance is large.

type NodeVersion 

type NodeVersion struct {
	// GlobalID is a globally unique id.
	GlobalID dvid.UUID

	// VersionID is a Dataset-specific id for each UUID, so we can compress the UUIDs.
	VersionID dvid.VersionLocalID

	// Locked nodes are read-only and can be branched.
	Locked bool

	// Parents is an ordered list of parent nodes.
	Parents []dvid.UUID

	// Children is a list of child nodes.
	Children []dvid.UUID

	Created time.Time
	Updated time.Time
}

NodeVersion contains all information for a node in the version DAG like its parents, children, and provenance.

type OpenError 

type OpenError struct {
	ErrorType OpenErrorType
	// contains filtered or unexported fields
}

type OpenErrorType 

type OpenErrorType int
const (
	ErrorOpening OpenErrorType = iota
	ErrorDatasets
	ErrorDatatypeUnavailable
)

type Request 

type Request struct {
	dvid.Command
	Input []byte
}

Request supports requests to the DVID server.

type Response 

type Response struct {
	dvid.Response
	Output []byte
}

Response supports responses from DVID.

func (*Response) Write 

func (r *Response) Write(w io.Writer) error

Writes a response to a writer.

type Service 

type Service struct {
	*Datasets
	// contains filtered or unexported fields
}

Service couples an open DVID storage engine and DVID datasets.

func (*Service) About 

func (s *Service) About() string

About returns a chart of the code versions of compile-time DVID datastore and the runtime data types.

func (*Service) Batcher 

func (s *Service) Batcher() (db storage.Batcher, err error)

Batcher returns an interface that can create a new batch write.

func (*Service) CurrentTypesJSON 

func (s *Service) CurrentTypesJSON() (jsonStr string, err error)

CurrentTypesJSON returns the components and versions of DVID software associated with the current datasets in the service.

func (*Service) DataChart 

func (s *Service) DataChart() string

DataChart returns a text chart of data names and their types for this DVID server.

func (*Service) DatasetJSON 

func (s *Service) DatasetJSON(root dvid.UUID) (stringJSON string, err error)

DatasetJSON returns JSON for a particular dataset referenced by a uuid.

func (*Service) DatasetsAllJSON 

func (s *Service) DatasetsAllJSON() (stringJSON string, err error)

DatasetsAllJSON returns JSON of a list of datasets.

func (*Service) DatasetsListJSON 

func (s *Service) DatasetsListJSON() (stringJSON string, err error)

DatasetsListJSON returns JSON of a list of datasets.

func (*Service) LocalIDFromUUID 

func (s *Service) LocalIDFromUUID(u dvid.UUID) (dID dvid.DatasetLocalID, vID dvid.VersionLocalID, err error)

LocalIDFromUUID when supplied a UUID string, returns smaller sized local IDs that identify a dataset and a version.

func (*Service) Lock 

func (s *Service) Lock(u dvid.UUID) error

Locks the node with the given UUID.

func (*Service) ModifyData 

func (s *Service) ModifyData(u dvid.UUID, dataname dvid.DataString, config dvid.Config) error

ModifyData modifies data of given name in dataset specified by a UUID.

func (*Service) NewData 

func (s *Service) NewData(u dvid.UUID, typename dvid.TypeString, dataname dvid.DataString, config dvid.Config) error

NewData adds data of given name and type to a dataset specified by a UUID.

func (*Service) NewDataset 

func (s *Service) NewDataset() (root dvid.UUID, datasetID dvid.DatasetLocalID, err error)

NewDataset creates a new dataset.

func (*Service) NewVersion 

func (s *Service) NewVersion(parent dvid.UUID) (u dvid.UUID, err error)

NewVersions creates a new version (child node) off of a LOCKED parent node. Will return an error if the parent node has not been locked.

func (*Service) NodeIDFromString 

func (s *Service) NodeIDFromString(str string) (u dvid.UUID, dID dvid.DatasetLocalID,
	vID dvid.VersionLocalID, err error)

NodeIDFromString when supplied a UUID string, returns the matched UUID as well as more compact local IDs that identify the dataset and a version. Partial matches are allowed, similar to DatasetFromString.

func (*Service) OrderedKeyValueDB 

func (s *Service) OrderedKeyValueDB() (storage.OrderedKeyValueDB, error)

OrderedKeyValueDB returns a a key-value database interface.

func (*Service) OrderedKeyValueGetter 

func (s *Service) OrderedKeyValueGetter() (storage.OrderedKeyValueGetter, error)

OrderedKeyValueGetter returns a a key-value getter interface.

func (*Service) OrderedKeyValueSetter 

func (s *Service) OrderedKeyValueSetter() (storage.OrderedKeyValueSetter, error)

OrderedKeyValueSetter returns a a key-value setter interface.

func (*Service) SaveDataset 

func (s *Service) SaveDataset(u dvid.UUID) error

SaveDataset forces this service to persist the dataset with given UUID. It is useful when modifying datasets internally.

func (*Service) Shutdown 

func (s *Service) Shutdown()

Shutdown closes a DVID datastore.

func (*Service) StorageEngine 

func (s *Service) StorageEngine() storage.Engine

StorageEngine returns a a key-value database interface.

func (*Service) SupportedDataChart 

func (s *Service) SupportedDataChart() string

SupportedDataChart returns a chart (names/urls) of data referenced by this datastore

func (*Service) TypesJSON 

func (s *Service) TypesJSON() (jsonStr string, err error)

TypesJSON returns the components and versions of DVID software available in this DVID server.

type Subsetter 

type Subsetter interface {
	// MaximumExtents returns a range of indices for which data is available at
	// some DVID server.
	MaximumExtents() dvid.IndexRange

	// AvailableExtents returns a range of indices for which data is available
	// at this DVID server.  It is the currently available extents.
	AvailableExtents() dvid.IndexRange
}

Subsetter is a type that can tell us its range of Index and how much it has actually available in this server. It's used to implement limited cloning, e.g., only cloning a quarter of an image volume. TODO: Fulfill implementation for voxels data type.

type TypeID 

type TypeID interface {
	// TypeName is an abbreviated type name.
	DatatypeName() dvid.TypeString

	// TypeUrl returns the unique package name that fulfills the DVID Data interface
	DatatypeUrl() UrlString

	// TypeVersion describes the version identifier of this data type code
	DatatypeVersion() string
}

TypeID provides methods for determining the identity of a data type.

type TypeService 

type TypeService interface {
	TypeID

	// Help returns a string explaining how to use a data type's service
	Help() string

	// Create Data that is an instance of this data type in the given Dataset
	NewDataService(id *DataID, config dvid.Config) (service DataService, err error)
}

TypeService is an interface for operations using arbitrary data types.

func TypeServiceByName 

func TypeServiceByName(typeName dvid.TypeString) (typeService TypeService, err error)

TypeServiceByName returns a type-specific service given a type name.

type UrlString 

type UrlString string

type VersionDAG 

type VersionDAG struct {
	Root  dvid.UUID
	Nodes map[dvid.UUID]*Node

	// VersionMap is used to accelerate mapping global UUID to DVID server-specific
	// and smaller ID for a version.
	VersionMap map[dvid.UUID]dvid.VersionLocalID

	NewVersionID dvid.VersionLocalID
	NewDataID    dvid.DataLocalID
	// contains filtered or unexported fields
}

VersionDAG is the directed acyclic graph of NodeVersion and an index by UUID into the graph.

func NewVersionDAG 

func NewVersionDAG() *VersionDAG

NewVersionDAG creates a version DAG and initializes the first unlocked node, assigning its UUID.

func (*VersionDAG) Lock 

func (dag *VersionDAG) Lock(u dvid.UUID) error

Lock locks a node. This is an irreversible operation since some nodes can be cloned externally.

func (*VersionDAG) LogInfo 

func (dag *VersionDAG) LogInfo() string

LogInfo returns provenance information for all the version nodes.

func (*VersionDAG) Versions 

func (dag *VersionDAG) Versions() []dvid.UUID

Versions returns a slice of UUID within this version DAG.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.