datastore

package
v0.0.0-...-2551dc1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 25, 2013 License: BSD-3-Clause Imports: 13 Imported by: 0

Documentation

Overview

Package datastore provides a DVID-specific API for storage and retrieval of supported data types, versioning, and maintenance of volume configuration data like volume extents and resolution.

Data types are declared in the main dvid.go file under imports. An example: 

import (
	// Declare the data types this DVID executable will support
	_ "github.com/janelia-flyem/dvid/datatype/grayscale8"
	_ "github.com/janelia-flyem/dvid/datatype/label64"
	_ "randomsite.org/datatypes/myfoo"
)

Index

Constants

View Source 
const (
	// Default number of block handlers to use per dataset.
	DefaultNumChunkHandlers = 8

	// Number of chunk write requests that can be buffered on each block handler
	// before sender is blocked.
	ChunkHandlerBufferSize = 100000
)

Constants that allow tuning of DVID for a particular target computer.

View Source 
const (
	// ConfigFilename is name of a file with datastore configuration data
	// just for human inspection.
	ConfigFilename = "dvid-config.json"
)
View Source 
const (
	Version = "0.6"
)

Variables

View Source 
var (
	// KeyConfig is the key for a DVID configuration
	KeyConfig = storage.Key{
		Dataset: storage.KeyDatasetGlobal,
		Version: storage.KeyVersionGlobal,
		Index:   []byte{0x01},
	}

	// KeyVersionDAG is the key for a Version DAG.
	KeyVersionDAG = storage.Key{
		Dataset: storage.KeyDatasetGlobal,
		Version: storage.KeyVersionGlobal,
		Index:   []byte{0x02},
	}
)
View Source 
var CompiledTypes = map[UrlString]TypeService{}

CompiledTypes is the set of registered data types compiled into DVID and held as a global variable initialized at runtime.

View Source 
var (

	// DiskAccess is a mutex to make sure we don't have goroutines simultaneously trying
	// to access the key-value database on disk.
	// TODO: Reexamine this in the context of parallel disk drives during cluster use.
	DiskAccess sync.Mutex
)

Functions

func ChunkLoadJSON 

func ChunkLoadJSON() (jsonStr string, err error)

ChunkLoadJSON returns a JSON description of the chunk op requests for each dataset.

func CompiledTypeChart 

func CompiledTypeChart() string

CompiledTypeChart returns a chart (names/urls) of data types compiled into this DVID.

func CompiledTypeNames 

func CompiledTypeNames() string

CompiledTypeNames returns a list of data type names compiled into this DVID.

func CompiledTypeUrls 

func CompiledTypeUrls() string

CompiledTypeUrls returns a list of data type urls supported by this DVID.

func Map 

func Map(op Operation) error

Map performs any sequential read, then breaks down an operation by sending chunks across a range of datatype-specific handlers that concurrently process the data. Each handler is assigned the same chunks based on its index.

func Open 

func Open(directory string) (s *Service, openErr *OpenError)

Open opens a DVID datastore at the given directory and returns a Service that allows operations on that datastore.

func RegisterDatatype 

func RegisterDatatype(t TypeService)

RegisterDatatype registers a data type for DVID use.

func StartDatasetChunkHandlers 

func StartDatasetChunkHandlers(dataset DatasetService)

StartDatasetChunkHandlers makes sure we have chunk handler goroutines for each data set. Chunks are routed to the same handler each time, so concurrent access to a chunk by multiple requests are funneled sequentially into a channel reserved for that chunk's handler.

func Versions 

func Versions() string

Versions returns a chart of version identifiers for data types and and DVID's datastore fixed at compile-time for this DVID executable

Types

type ArbitraryInput 

type ArbitraryInput interface{}

type ArbitraryOutput 

type ArbitraryOutput interface{}

type BlockZYX 

type BlockZYX struct {
	dvid.BlockCoord
}

BlockZYX implements the BlockIndex interface and provides blocking of a 3d coordinate space.

func (BlockZYX) MakeIndex 

func (bc BlockZYX) MakeIndex() (i Index, err error)

MakeIndex returns a 1d Index from a 3d ZYX coordinate.

type ChunkChannels 

type ChunkChannels map[DatasetString]([]chan *ChunkOp)

Each dataset has a pool of channels to communicate with gouroutine handlers.

type ChunkIndex 

type ChunkIndex interface {
	// MakeIndex returns a one-dimensional index.
	MakeIndex() (Index, error)

	// String returns a human-readable representation of the block index.
	String() string
}

ChunkIndex represents an index to a datatype-specific partition of data. For example, a voxels data type would have a 3d block coordinate. Chunk indices do *not* have to be 3d coordinates but can be optimized for a data type's internal data structure.

type ChunkOp 

type ChunkOp struct {
	Operation
	Chunk []byte
	Key   storage.Key
	// contains filtered or unexported fields
}

type Dataset 

type Dataset struct {
	DatasetID

	// Underlying implementation of this dataset is a Datatype
	Datatype TypeService
}

Dataset is an instance of a data type and has an associated datastore.

func NewDataset 

func NewDataset(id DatasetID, t TypeService) *Dataset

func (*Dataset) DatatypeName 

func (d *Dataset) DatatypeName() string

func (*Dataset) DatatypeUrl 

func (d *Dataset) DatatypeUrl() UrlString

func (*Dataset) DatatypeVersion 

func (d *Dataset) DatatypeVersion() string

func (*Dataset) Help 

func (d *Dataset) Help() string

func (*Dataset) NewDataset 

func (d *Dataset) NewDataset(id DatasetID, config dvid.Config) DatasetService

func (*Dataset) UnknownCommand 

func (d *Dataset) UnknownCommand(request Request) error

type DatasetID 

type DatasetID struct {
	Name    DatasetString
	LocalID dvid.LocalID
}

DatasetID identifies a dataset within a DVID instance.

func (DatasetID) DatasetLocalID 

func (id DatasetID) DatasetLocalID() dvid.LocalID

func (DatasetID) DatasetName 

func (id DatasetID) DatasetName() DatasetString

type DatasetService 

type DatasetService interface {
	TypeService

	// DatasetName returns the name of the dataset.
	DatasetName() DatasetString

	// DatasetLocalID returns a DVID instance-specific id for this dataset,
	// which can be held in a relatively small number of bytes and is useful
	// as a key component.
	DatasetLocalID() dvid.LocalID

	// NumChunkHandlers returns the number of chunk handlers that should be
	// used for this dataset.
	NumChunkHandlers() int

	// ChunkHandler processes a chunk of data in a mapped operation.
	ChunkHandler(op *ChunkOp)

	// Handle iteration through a dataset in abstract way.
	NewIndexIterator(extents interface{}) IndexIterator

	// DoRPC handles command line and RPC commands specific to a data type
	DoRPC(request Request, reply *Response) error

	// DoHTTP handles HTTP requests specific to a data type
	DoHTTP(w http.ResponseWriter, r *http.Request) error

	// Returns standard error response for unknown commands
	UnknownCommand(r Request) error
}

DatasetService is an interface for operations on arbitrary datasets that use a supported TypeService. Block handlers can be allocated at this level, so an implementation can own a number of goroutines.

DatasetService operations are completely type-specific, and each datatype handles operations through RPC (DoRPC) and HTTP (DoHTTP). TODO -- Add SPDY as wrapper to HTTP.

type DatasetString 

type DatasetString string

DatasetString is a string that is the name of a DVID data set. This gets its own type for documentation and also provide static error checks to prevent conflation of type name from data set name.

type Datatype 

type Datatype struct {
	DatatypeID

	// A list of interface requirements for the backend datastore
	Requirements storage.Requirements
}

Datatype is the base struct that satisfies a TypeService and can be embedded in other data types.

func (*Datatype) Help 

func (datatype *Datatype) Help() string

type DatatypeID 

type DatatypeID struct {
	// Data type name and may not be unique.
	Name string

	// The unique package name that fulfills the DVID Data interface
	Url UrlString

	// The version identifier of this data type code
	Version string
}

DatatypeID uniquely identifies a DVID-supported data type and provides a shorthand name.

func MakeDatatypeID 

func MakeDatatypeID(name string, url UrlString, version string) DatatypeID

func (*DatatypeID) DatatypeName 

func (id *DatatypeID) DatatypeName() string

func (*DatatypeID) DatatypeUrl 

func (id *DatatypeID) DatatypeUrl() UrlString

func (*DatatypeID) DatatypeVersion 

func (id *DatatypeID) DatatypeVersion() string

type Index 

type Index interface {
	// Bytes gives a slice of bytes.
	Bytes() []byte

	// InvertIndex returns a chunk index given a one-dimensional Index.
	// It is the inverse of MakeIndex().
	InvertIndex() ChunkIndex

	// Hash provides a consistent mapping from an Index to an integer (0,n]
	Hash(n int) int

	// Scheme returns a string describing the indexing scheme.
	Scheme() string

	// String returns a hexadecimal string representation
	String() string
}

Index is a one-dimensional index, typically constructed using some sort of spatiotemporal indexing scheme. For example, Z-curves map n-D space to a 1-D index. It is assumed that implementations for this interface are castable to []byte.

type IndexHilbert 

type IndexHilbert []byte

TODO -- Hilbert curve

func (IndexHilbert) Scheme 

func (i IndexHilbert) Scheme() string

type IndexIterator 

type IndexIterator func() Index

IndexIterator is a function that returns a sequence of indices and ends with nil.

type IndexIteratorMaker 

type IndexIteratorMaker interface {
	NewIndexIterator() IndexIterator
}

IndexIteratorMakers can make new IndexIterators.

type IndexMorton 

type IndexMorton []byte

TODO -- Morton (Z-order) curve

func (IndexMorton) Scheme 

func (i IndexMorton) Scheme() string

type IndexZYX 

type IndexZYX []byte

IndexZYX implements the Index interface and provides simple indexing on Z, then Y, then X.

func (IndexZYX) Bytes 

func (i IndexZYX) Bytes() []byte

func (IndexZYX) Hash 

func (i IndexZYX) Hash(n int) int

Hash returns an integer [0, n) where the returned values should be reasonably spread among the range of returned values.

func (IndexZYX) InvertIndex 

func (i IndexZYX) InvertIndex() ChunkIndex

InvertIndex decodes a spatial index into a block coordinate

func (IndexZYX) OffsetToBlock 

func (i IndexZYX) OffsetToBlock(blockSize dvid.Point3d) (coord dvid.VoxelCoord)

OffsetToBlock returns the voxel coordinate at the top left corner of the block corresponding to the index.

func (IndexZYX) Scheme 

func (i IndexZYX) Scheme() string

func (IndexZYX) String 

func (i IndexZYX) String() string

type OpenError 

type OpenError struct {
	ErrorType OpenErrorType
	// contains filtered or unexported fields
}

type OpenErrorType 

type OpenErrorType int
const (
	ErrorOpening OpenErrorType = iota
	ErrorRuntimeConfig
	ErrorVersionDAG
)

type Operation 

type Operation interface {
	DatasetService

	DatastoreService() *Service

	VersionLocalID() dvid.LocalID

	// Does this operation only read data?
	IsReadOnly() bool

	// Returns an IndexIterator that steps through all chunks for this operation.
	IndexIterator() IndexIterator

	// Map traverses chunks in an operation and maps it to dataset-specific chunk handlers.
	// Types that implement the Operation interface can use datastore.Map() to fulfill
	// this method or implement their own mapping scheme.
	Map() error

	// WaitAdd increments an internal WaitGroup
	WaitAdd()

	// Wait blocks until the operation is complete
	Wait()
}

Operation provides an abstract interface to datatype-specific operations. Each data type will implement operations that are mostly opaque at this level.

type Request 

type Request struct {
	dvid.Command
	Input ArbitraryInput
}

Request supports requests to the DVID server. Since input and reply payloads are different depending on the command and the data type, we use an ArbitraryInput (empty interface) for the payload.

type Response 

type Response struct {
	dvid.Response
	Output ArbitraryOutput
}

Response supports responses from DVID.

type Service 

type Service struct {

	// Holds all version data including map of UUIDs to nodes
	VersionDAG
	// contains filtered or unexported fields
}

Service encapsulates an open DVID datastore, available for operations.

func (*Service) About 

func (config *Service) About() string

About returns a chart of the code versions of compile-time DVID datastore and the runtime data types.

func (*Service) AboutJSON 

func (config *Service) AboutJSON() (jsonStr string, err error)

AboutJSON returns the components and versions of DVID software.

func (*Service) Batcher 

func (s *Service) Batcher() (db storage.Batcher, err error)

Batcher returns an interface that can create a new batch write.

func (*Service) ConfigJSON 

func (config *Service) ConfigJSON() (jsonStr string, err error)

ConfigJSON returns configuration data in JSON format.

func (*Service) DataChart 

func (config *Service) DataChart() string

DataChart returns a chart of data set names and their types for this runtime configuration.

func (*Service) DatasetService 

func (s *Service) DatasetService(name DatasetString) (datasetService DatasetService, err error)

DatasetService returns a type-specific service given a dataset name that should be specific to a DVID instance.

func (*Service) DeleteDataset 

func (s *Service) DeleteDataset(name DatasetString) error

DeleteDataset deletes a data set from the datastore and is considered a DANGEROUS operation. Only to be used during debugging and development. TODO -- Deleted dataset data should also be expunged.

func (*Service) GetHeadUuid 

func (s *Service) GetHeadUuid() UUID

GetHeadUuid returns the UUID of the HEAD node, i.e., the version that is considered the most current.

func (*Service) IteratorMaker 

func (s *Service) IteratorMaker() (db storage.IteratorMaker, err error)

IteratorMaker returns an interface that can create a new iterator.

func (*Service) KeyValueDB 

func (s *Service) KeyValueDB() storage.KeyValueDB

KeyValueDB returns a a key-value database interface.

func (*Service) NewDataset 

func (s *Service) NewDataset(name DatasetString, typeName string, config dvid.Config) error

NewDataset registers a data set name of a given data type with this datastore. TODO -- Will have to pass in initialization parameters like block size, block resolution, units, etc. Then, within the data type, for each dataset we create a blocksize that's tailored for this DVID instance's volume, after taking into account units, resolution, etc.

func (*Service) Shutdown 

func (s *Service) Shutdown()

Shutdown closes a DVID datastore.

func (*Service) StartChunkHandlers 

func (s *Service) StartChunkHandlers()

StartChunkHandlers starts goroutines that handle chunks for every dataset in the service.

func (*Service) SupportedDataChart 

func (s *Service) SupportedDataChart() string

SupportedDataChart returns a chart (names/urls) of data referenced by this datastore

func (*Service) TypeService 

func (s *Service) TypeService(typeName string) (typeService TypeService, err error)

TypeService returns a type-specific service given a type name.

func (*Service) VerifyCompiledTypes 

func (config *Service) VerifyCompiledTypes() error

VerifyCompiledTypes will return an error if any required data type in the datastore configuration was not compiled into DVID executable. Check is done by more exact URL and not the data type name.

func (*Service) VersionBytes 

func (s *Service) VersionBytes(uuidStr string) (b []byte, err error)

VersionBytes returns a slice of bytes representing the VersionID corresponding to the supplied UUID string. The UUID string can be incomplete as long as it contains a unique prefix of a valid UUID.

func (*Service) VersionsJSON 

func (s *Service) VersionsJSON() (jsonStr string, err error)

VersionsJSON returns the version DAG data in JSON format.

type TypeID 

type TypeID interface {
	// TypeName describes a data type and may not be unique.
	DatatypeName() string

	// TypeUrl returns the unique package name that fulfills the DVID Data interface
	DatatypeUrl() UrlString

	// TypeVersion describes the version identifier of this data type code
	DatatypeVersion() string
}

TypeID provides methods for determining the identity of a data type.

type TypeInfo 

type TypeInfo struct {
	Name    string
	Url     string
	Version string
	Help    string
}

TypeInfo contains data type information reformatted for easy consumption by clients.

type TypeService 

type TypeService interface {
	TypeID

	// Help returns a string explaining how to use a data type's service
	Help() string

	// Create a Dataset of this data type
	NewDataset(id DatasetID, config dvid.Config) DatasetService
}

TypeService is an interface for operations using arbitrary data types.

type UUID 

type UUID string

UUID is a 32 character hexidecimal string ("" if invalid) that uniquely identifies nodes in a datastore's DAG. We need universally unique identifiers to prevent collisions during creation of child nodes by distributed DVIDs: http://en.wikipedia.org/wiki/Universally_unique_identifier

func Init 

func Init(directory string, create bool) (uuid UUID)

Init creates a key-value datastore using default arguments. Datastore configuration is stored as key/values in the datastore and also in a human-readable config file in the datastore directory.

func NewUUID 

func NewUUID() UUID

NewUUID returns a UUID

type UrlString 

type UrlString string

type VersionDAG 

type VersionDAG struct {
	// NewID keeps track of the VersionID for any new version.
	NewID dvid.LocalID

	Head       UUID
	Nodes      []VersionNode
	VersionMap map[UUID]dvid.LocalID
}

VersionDAG is the directed acyclic graph of VersionNode and an index by UUID into the graph.

func NewVersionDAG 

func NewVersionDAG() *VersionDAG

NewVersionDAG creates a version DAG and initializes the first unlocked node, assigning its UUID.

func (*VersionDAG) Deserialize 

func (dag *VersionDAG) Deserialize(s dvid.Serialization) (err error)

Deserialize converts a serialization to a VersionDAG

func (*VersionDAG) Get 

func (dag *VersionDAG) Get(db storage.KeyValueDB) (err error)

Get retrieves a configuration from a KeyValueDB.

func (*VersionDAG) LogInfo 

func (dag *VersionDAG) LogInfo() string

LogInfo returns provenance information for all the version nodes.

func (*VersionDAG) NewVersion 

func (dag *VersionDAG) NewVersion(parents []dvid.LocalID) (node *VersionNode, err error)

NewVersion creates a new version node with the given parents.

func (*VersionDAG) Put 

func (dag *VersionDAG) Put(db storage.KeyValueDB) (err error)

Put stores a configuration into a KeyValueDB.

func (*VersionDAG) Serialize 

func (dag *VersionDAG) Serialize() (s dvid.Serialization, err error)

Serialize returns a serialization of VersionDAG with Snappy compression and CRC32 checksum.

func (*VersionDAG) VersionIDFromString 

func (dag *VersionDAG) VersionIDFromString(str string) (id dvid.LocalID, err error)

VersionIDFromString returns a UUID index given its string representation. Partial matches are accepted as long as they are unique for a datastore. So if a datastore has nodes with UUID strings 3FA22..., 7CD11..., and 836EE..., we can still find a match even if given the minimum 3 letters. (We don't allow UUID strings of less than 3 letters just to prevent mistakes.)

func (*VersionDAG) VersionIDFromUUID 

func (dag *VersionDAG) VersionIDFromUUID(uuid UUID) (id dvid.LocalID, err error)

VersionIDFromUUID returns a local version ID given a UUID. This method does not do partial matching, unlike VersionIDFromString().

type VersionNode 

type VersionNode struct {
	// Id is a globally unique id.
	GlobalID UUID

	// VersionID is a DVID instance-specific id per datastore.
	VersionID dvid.LocalID

	// Locked nodes are read-only and can be branched.
	Locked bool

	// Archived nodes only have delta blocks from its parents, and earlier parents
	// in the parent list have priority in determining the delta.
	Archived bool

	// Parents is an ordered list of parent nodes.
	Parents []UUID

	// Children is a list of child nodes.
	Children []UUID

	// Note holds general information on this node.
	Note string

	// Provenance describes the operations performed between the locking of
	// this node's parents and its current state.
	Provenance string

	Created time.Time
	Updated time.Time
}

VersionNode contains all information for a node in the version DAG like its parents, children, and provenance.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.