server

Documentation

Overview

Package server configures and launches http/rpc server and storage engines specific to the type of DVID platform: local (e.g., running on MacBook Pro), clustered, or using cloud-based services like Google Cloud.

Datatypes can use any of the three tiers of storage (MetaData, Mutable, Immutable) that provide a layer of storage semantics (latency, mutability, etc) on top of underlying storage engines.

The DVID web client is also managed from this package. For a DVID web console, see the repo:

https://github.com/janelia-flyem/dvid-console

The goal of a DVID web console is to provide a GUI for monitoring and performing a subset of operations in a nicely formatted view.

DVID command line interaction occurs via the rpc interface to a running server. Please see the main DVID documentation:

http://godoc.org/github.com/janelia-flyem/dvid

Index

• Constants
• Variables
• func About() string
• func AboutJSON() (jsonStr string, err error)
• func AllowTiming() bool
• func BadAPIRequest(w http.ResponseWriter, r *http.Request, d dvid.Data)
• func BadRequest(w http.ResponseWriter, r *http.Request, format interface{}, ...)
• func BlockOnInteractiveRequests(caller ...string)
• func CacheSize(id string) int
• func CheckChunkThrottling()
• func CloseTest()
• func ConfigLocation() string
• func CreateTestInstance(t *testing.T, uuid dvid.UUID, typename, name string, config dvid.Config)
• func CreateTestReplaceSync(t *testing.T, uuid dvid.UUID, name string, syncs ...string)
• func CreateTestSync(t *testing.T, uuid dvid.UUID, name string, syncs ...string)
• func DatastoreConfig() datastore.Config
• func DecodeJSON(r *http.Request) (dvid.Config, error)
• func GitVersion() string
• func GotInteractiveRequest()
• func HTTPAddress() string
• func Host() string
• func InitBackend() (backend *storage.Backend, err error)
• func Initialize() error
• func KafkaActivityTopic() string
• func KafkaAvailable() bool
• func KafkaPrefixTopic() string
• func KafkaServers() []string
• func LoadConfig(filename string) error
• func LogHTTPMutation(versionID, dataID dvid.UUID, r *http.Request, data []byte) (err error)
• func LogJSONMutation(versionID, dataID dvid.UUID, jsondata []byte) error
• func MutcachePath(name dvid.InstanceName) string
• func NoLabelmapSplit() bool
• func Note() string
• func OpenTest(configs ...TestConfig) error
• func RPCAddress() string
• func ReadJSONMutations(w io.Writer, versionID, dataID dvid.UUID) error
• func SendRPC(addr string, req datastore.Request) error
• func Serve()
• func ServeSingleHTTP(w http.ResponseWriter, r *http.Request)
• func SetAdminToken(token string)
• func SetFullWrite(on bool)
• func SetMonitor(on bool)
• func SetReadOnly(on bool)
• func Shutdown()
• func Stores() (map[storage.Alias]dvid.StoreConfig, error)
• func TestBadHTTP(t *testing.T, method, urlStr string, payload io.Reader)
• func TestHTTP(t *testing.T, method, urlStr string, payload io.Reader) []byte
• func TestHTTPError(t *testing.T, method, urlStr string, payload io.Reader) (retbytes []byte, err error)
• func TestHTTPResponse(t *testing.T, method, urlStr string, payload io.Reader) *httptest.ResponseRecorder
• func ThrottledHTTP(w http.ResponseWriter) bool
• func ThrottledOpDone()
• func WebClient() string
• func WebDefaultFile() string
• func WebRedirectPath() string
• func WebServer() string
• type MutationsConfig
  • func MutationLogSpec() MutationsConfig
• type TestConfig

Constants

const (
	// DefaultWebAddress is the default URL of the DVID web server
	DefaultWebAddress = "localhost:8000"

	// DefaultRPCAddress is the default RPC address for command-line use of a remote DVID server
	DefaultRPCAddress = "localhost:8001"

	// ErrorLogFilename is the name of the server error log, stored in the datastore directory.
	ErrorLogFilename = "dvid-errors.log"
)

const (
	// WebAPIVersion is the string version of the API.  Once DVID is somewhat stable,
	// this will be "v1/", "v2/", etc.
	WebAPIVersion = ""

	// WebAPIPath is the relative URL path to our Level 2 REST API
	WebAPIPath = "/api/" + WebAPIVersion

	// WriteTimeout is the maximum time in seconds DVID will wait to write data down HTTP connection.
	WriteTimeout = 600 * time.Second

	// ReadTimeout is the maximum time in seconds DVID will wait to read data from HTTP connection.
	ReadTimeout = 600 * time.Second
)

const SecretKeyVarName = "DVID_JWT_SECRET_KEY"

SecretKeyVarName is the environment variable holding the secret key for JWT support.

Variables

var (
	// MaxDataRequest sets the limit on the amount of data that could be returned for a request
	MaxDataRequest = int64(3) * dvid.Giga

	// InteractiveOpsPer2Min gives the number of interactive-level requests
	// received over the last 2 minutes.  This is useful for throttling "batch"
	// operations on a single DVID server.  Note that this metric is an lower
	// bound on the number of interactive requests over the last minute since
	// we do non-blocking reports.
	InteractiveOpsPer2Min int

	// ActiveHandlers is maximum number of active handlers over last second.
	ActiveHandlers int

	// MaxChunkHandlers sets the maximum number of chunk handlers (goroutines) that
	// can be multiplexed onto available cores.  (See -numcpu setting in dvid.go)
	MaxChunkHandlers = 10 * runtime.NumCPU()

	// HandlerToken is buffered channel to limit spawning of goroutines.
	// See ProcessChunk() in datatype/imageblk for example.
	HandlerToken = make(chan int, MaxChunkHandlers)

	// LargeMutationMutex is a global lock for compute-intense processes that want to
	// spawn goroutines that consume handler tokens.  This lets processes capture most
	// if not all available handler tokens in a FIFO basis rather than have multiple
	// concurrent requests launch a few goroutines each.
	LargeMutationMutex sync.Mutex

	// TimeoutSecs specifies the seconds waiting to open a datastore for exclusive access.
	TimeoutSecs int
)

var (
	// DefaultHost is the default most understandable alias for this server.
	DefaultHost = "localhost"
)

Functions

func About

func About() string

About returns a chart of version identifiers for the DVID source code, DVID datastore, and all component data types for this executable.

func AboutJSON

func AboutJSON() (jsonStr string, err error)

AboutJSON returns a JSON string describing the properties of this server.

func AllowTiming

func AllowTiming() bool

func BadAPIRequest

func BadAPIRequest(w http.ResponseWriter, r *http.Request, d dvid.Data)

BadAPIRequest writes a standard error message to http.ResponseWriter for a badly formatted API call.

func BadRequest

func BadRequest(w http.ResponseWriter, r *http.Request, format interface{}, args ...interface{})

BadRequest writes an error message out to the http.ResponseWriter using format similar to fmt.Printf.

func BlockOnInteractiveRequests

func BlockOnInteractiveRequests(caller ...string)

BlockOnInteractiveRequests will block this goroutine until the number of interactive requests dips below InteractiveOpsBeforeBlock in server configuration.

func CacheSize

func CacheSize(id string) int

CacheSize returns the number oF bytes reserved for the given identifier. If unset, will return 0.

func CheckChunkThrottling

func CheckChunkThrottling()

func CloseTest

func CloseTest()

CloseTest shuts down server for testing.

func ConfigLocation

func ConfigLocation() string

func CreateTestInstance

func CreateTestInstance(t *testing.T, uuid dvid.UUID, typename, name string, config dvid.Config)

CreateTestInstance posts a new data instance for testing.

func CreateTestReplaceSync

func CreateTestReplaceSync(t *testing.T, uuid dvid.UUID, name string, syncs ...string)

CreateTestReplaceSync replaces syncs for a given data instance / uuid.

func CreateTestSync

func CreateTestSync(t *testing.T, uuid dvid.UUID, name string, syncs ...string)

CreateTestSync posts new syncs for a given data instance / uuid.

func DatastoreConfig

func DatastoreConfig() datastore.Config

DatastoreConfig returns data instance configuration necessary to handle id generation.

func DecodeJSON

func DecodeJSON(r *http.Request) (dvid.Config, error)

DecodeJSON decodes JSON passed in a request into a dvid.Config.

func GitVersion

func GitVersion() string

GitVersion returns a git-derived string that allows recovery of the exact source code used for this DVID server.

func GotInteractiveRequest

func GotInteractiveRequest()

GotInteractiveRequest can be called to track the # of interactive requests that require some amount of computation. Don't use this to track simple polling APIs. This routine will not block.

func HTTPAddress

func HTTPAddress() string

func Host

func Host() string

Host returns the most understandable host alias + any port.

func InitBackend

func InitBackend() (backend *storage.Backend, err error)

InitBackend creates a new backend from current configuration.

func Initialize

func Initialize() error

Initialize sets up the server (runs webhook, kafka goroutine, etc) and presumes that LoadConfig() or another method was already used to configure the server.

func KafkaActivityTopic

func KafkaActivityTopic() string

func KafkaAvailable

func KafkaAvailable() bool

KafkaAvailable returns true if kafka servers are initialized.

func KafkaPrefixTopic

func KafkaPrefixTopic() string

func KafkaServers

func KafkaServers() []string

func LoadConfig

func LoadConfig(filename string) error

LoadConfig loads DVID server configuration from a TOML file.

func LogHTTPMutation

func LogHTTPMutation(versionID, dataID dvid.UUID, r *http.Request, data []byte) (err error)

LogHTTPMutation logs a HTTP mutation request to the mutation log specific in the config.

func LogJSONMutation

func LogJSONMutation(versionID, dataID dvid.UUID, jsondata []byte) error

LogJSONMutation logs a JSON mutation record to the Jsonstore directory in the config.

func MutcachePath

func MutcachePath(name dvid.InstanceName) string

MutcachePath returns path to where mutation cache should be placed for the given labelmap instance. If nothing specified, empty string returned.

func NoLabelmapSplit

func NoLabelmapSplit() bool

func Note

func Note() string

func OpenTest

func OpenTest(configs ...TestConfig) error

OpenTest initializes the server for testing, setting up caching, datastore, etc. Later configurations will override earlier ones.

func RPCAddress

func RPCAddress() string

func ReadJSONMutations

func ReadJSONMutations(w io.Writer, versionID, dataID dvid.UUID) error

ReadJSONMutations streams a JSON of mutation records to the writer

func SendRPC

func SendRPC(addr string, req datastore.Request) error

SendRPC sends a request to a remote DVID.

func Serve

func Serve()

Serve starts HTTP and RPC servers.

func ServeSingleHTTP

func ServeSingleHTTP(w http.ResponseWriter, r *http.Request)

ServeSingleHTTP fulfills one request using the default web Mux.

func SetAdminToken

func SetAdminToken(token string)

SetAdminToken sets what can be passed in query string under "admintoken" to get admin (fullwrite) privileges.

func SetFullWrite

func SetFullWrite(on bool)

SetFullWrite allows mutations on any version.

func SetMonitor

func SetMonitor(on bool)

SetMonitor can put server in monitor mode (writes load stats to debug if activity).

func SetReadOnly

func SetReadOnly(on bool)

SetReadOnly can put the server in a read-only mode.

func Shutdown

func Shutdown()

Shutdown handles graceful cleanup of server functions before exiting DVID. This may not be so graceful if the chunk handler uses cgo since the interrupt may be caught during cgo execution.

func Stores

func Stores() (map[storage.Alias]dvid.StoreConfig, error)

func TestBadHTTP

func TestBadHTTP(t *testing.T, method, urlStr string, payload io.Reader)

TestBadHTTP expects a HTTP response with an error status code.

func TestHTTP

func TestHTTP(t *testing.T, method, urlStr string, payload io.Reader) []byte

TestHTTP returns the response body bytes for a test request, making sure any response has status OK.

func TestHTTPError

func TestHTTPError(t *testing.T, method, urlStr string, payload io.Reader) (retbytes []byte, err error)

TestHTTPError returns the response body bytes for a test request, making sure any response has status OK.

func TestHTTPResponse

func TestHTTPResponse(t *testing.T, method, urlStr string, payload io.Reader) *httptest.ResponseRecorder

TestHTTPResponse returns a response from a test run of the DVID server. Use TestHTTP if you just want the response body bytes.

func ThrottledHTTP

func ThrottledHTTP(w http.ResponseWriter) bool

ThrottledHTTP checks if a request can continue under throttling. If so, it returns false. If it cannot (throttled state), it sends a http.StatusServiceUnavailable and returns true. Throttling is controlled by

func ThrottledOpDone

func ThrottledOpDone()

ThrottledOpDone marks the end of a throttled operation, allowing another op blocked by ThrottledHTTP() to succeed.

func WebClient

func WebClient() string

func WebDefaultFile

func WebDefaultFile() string

func WebRedirectPath

func WebRedirectPath() string

func WebServer

func WebServer() string

WebServer returns the configured host name from TOML file or, if not specified, the retrieved hostname plus the port of the web server.

Types

type MutationsConfig

type MutationsConfig struct {
	Httpstore string        // examples: "kafka:mytopic", "logstore:myStoreAlias"
	Jsonstore string        // path to directory of protolog files split by Data+Version UUID
	Blobstore storage.Alias // alias to a store
}

MutationsConfig specifies handling of mutation logs, which are composed of an append-only logs of mutations. There are several types of mutation logs. Httpstore is where all mutation HTTP requests are logged with large binary payloads being stored in the designated Blobstore. Jsonstore is where kafka-like JSON mutation records are logged.

func MutationLogSpec

func MutationLogSpec() MutationsConfig

type TestConfig

type TestConfig struct {
	KVStoresMap  storage.DataMap
	LogStoresMap storage.DataMap
	CacheSize    map[string]int // MB for caches
}

TestConfig specifies configuration for testing servers.