keepclient

package
v0.0.0-...-c92af4d Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 1, 2021 License: AGPL-3.0, Apache-2.0, CC-BY-SA-3.0 Imports: 25 Imported by: 2

Documentation

Overview

Package keepclient provides low-level Get/Put primitives for accessing Arvados Keep blocks.

Index

Constants

View Source 
const (
	XKeepDesiredReplicas         = "X-Keep-Desired-Replicas"
	XKeepReplicasStored          = "X-Keep-Replicas-Stored"
	XKeepStorageClasses          = "X-Keep-Storage-Classes"
	XKeepStorageClassesConfirmed = "X-Keep-Storage-Classes-Confirmed"
)
View Source 
const BLOCKSIZE = 64 * 1024 * 1024

BLOCKSIZE defines the length of a Keep "block", which is 64MB.

Variables

View Source 
var (
	DefaultRequestTimeout      = 20 * time.Second
	DefaultConnectTimeout      = 2 * time.Second
	DefaultTLSHandshakeTimeout = 4 * time.Second
	DefaultKeepAlive           = 180 * time.Second

	DefaultProxyRequestTimeout      = 300 * time.Second
	DefaultProxyConnectTimeout      = 30 * time.Second
	DefaultProxyTLSHandshakeTimeout = 10 * time.Second
	DefaultProxyKeepAlive           = 120 * time.Second
)
View Source 
var (
	ErrSignatureExpired = arvados.ErrSignatureExpired
	ErrSignatureInvalid = arvados.ErrSignatureInvalid
	ErrSignatureMissing = arvados.ErrSignatureMissing
	SignLocator         = arvados.SignLocator
	SignedLocatorRe     = arvados.SignedLocatorRe
	VerifySignature     = arvados.VerifySignature
)
View Source 
var BadChecksum = errors.New("Reader failed checksum")
View Source 
var BlockNotFound = &ErrNotFound{multipleResponseError{
	error:  errors.New("Block not found"),
	isTemp: false,
}}

BlockNotFound is a multipleResponseError where isTemp is false

View Source 
var DebugPrintf = func(string, ...interface{}) {}

DebugPrintf emits debug messages. The easiest way to enable keepclient debug messages in your application is to assign log.Printf to DebugPrintf.

View Source 
var DefaultBlockCache = &BlockCache{}
View Source 
var ErrIncompleteIndex = errors.New("Got incomplete index")

ErrIncompleteIndex is returned when the Index response does not end with a new empty line

View Source 
var ErrNoManifest = errors.New("Collection has no manifest")

ErrNoManifest indicates the given collection has no manifest information (e.g., manifest_text was excluded by a "select" parameter when retrieving the collection record).

View Source 
var ErrNoSuchKeepServer = errors.New("No keep server matching the given UUID is found")

ErrNoSuchKeepServer is returned when GetIndex is invoked with a UUID with no matching keep server

View Source 
var ErrOversizeBlock = OversizeBlockError(errors.New("Exceeded maximum block size (" + strconv.Itoa(BLOCKSIZE) + ")"))
View Source 
var InvalidLocatorError = errors.New("Invalid locator")
View Source 
var MissingArvadosApiHost = errors.New("Missing required environment variable ARVADOS_API_HOST")
View Source 
var MissingArvadosApiToken = errors.New("Missing required environment variable ARVADOS_API_TOKEN")

Functions

func Md5String 

func Md5String(s string) string

Md5String returns md5 hash for the bytes in the given string

func RefreshServiceDiscovery 

func RefreshServiceDiscovery()

RefreshServiceDiscovery clears the Keep service discovery cache.

func RefreshServiceDiscoveryOnSIGHUP 

func RefreshServiceDiscoveryOnSIGHUP()

RefreshServiceDiscoveryOnSIGHUP installs a signal handler that calls RefreshServiceDiscovery when SIGHUP is received.

Types

type BlockCache 

type BlockCache struct {
	// Maximum number of blocks to keep in the cache. If 0, a
	// default size (currently 4) is used instead.
	MaxBlocks int
	// contains filtered or unexported fields
}

func (*BlockCache) Clear 

func (c *BlockCache) Clear()

func (*BlockCache) Get 

func (c *BlockCache) Get(kc *KeepClient, locator string) ([]byte, error)

Get returns data from the cache, first retrieving it from Keep if necessary.

func (*BlockCache) ReadAt 

func (c *BlockCache) ReadAt(kc *KeepClient, locator string, p []byte, off int) (int, error)

ReadAt returns data from the cache, first retrieving it from Keep if necessary.

func (*BlockCache) Sweep 

func (c *BlockCache) Sweep()

Sweep deletes the least recently used blocks from the cache until there are no more than MaxBlocks left.

type ErrNotFound 

type ErrNotFound struct {
	// contains filtered or unexported fields
}

ErrNotFound is a multipleResponseError where isTemp can be true or false

func (*ErrNotFound) Temporary 

func (e *ErrNotFound) Temporary() bool

type Error 

type Error interface {
	error
	Temporary() bool
}

Error interface with an error and boolean indicating whether the error is temporary

type HTTPClient 

type HTTPClient interface {
	Do(*http.Request) (*http.Response, error)
}

type HashCheckingReader 

type HashCheckingReader struct {
	// The underlying data source
	io.Reader

	// The hash function to use
	hash.Hash

	// The hash value to check against.  Must be a hex-encoded lowercase string.
	Check string
}

HashCheckingReader is an io.ReadCloser that checks the contents read from the underlying io.Reader against the provided hash.

func (HashCheckingReader) Close 

func (hcr HashCheckingReader) Close() (err error)

Close reads all remaining data from the underlying Reader and returns BadChecksum if the checksum doesn't match. It also closes the underlying Reader if it implements io.ReadCloser.

func (HashCheckingReader) Read 

func (hcr HashCheckingReader) Read(p []byte) (n int, err error)

Reads from the underlying reader, update the hashing function, and pass the results through. Returns BadChecksum (instead of EOF) on the last read if the checksum doesn't match.

func (HashCheckingReader) WriteTo 

func (hcr HashCheckingReader) WriteTo(dest io.Writer) (written int64, err error)

WriteTo writes the entire contents of hcr.Reader to dest. Returns BadChecksum if writing is successful but the checksum doesn't match.

type InsufficientReplicasError 

type InsufficientReplicasError error

type KeepClient 

type KeepClient struct {
	Arvados       *arvadosclient.ArvadosClient
	Want_replicas int

	HTTPClient     HTTPClient
	Retries        int
	BlockCache     *BlockCache
	RequestID      string
	StorageClasses []string
	// contains filtered or unexported fields
}

KeepClient holds information about Arvados and Keep servers.

func MakeKeepClient 

func MakeKeepClient(arv *arvadosclient.ArvadosClient) (*KeepClient, error)

MakeKeepClient creates a new KeepClient, calls DiscoverKeepServices(), and returns when the client is ready to use.

func New 

func New(arv *arvadosclient.ArvadosClient) *KeepClient

New creates a new KeepClient. Service discovery will occur on the next read/write operation.

func (*KeepClient) Ask 

func (kc *KeepClient) Ask(locator string) (int64, string, error)

Ask verifies that a block with the given hash is available and readable, according to at least one Keep service. Unlike Get, it does not retrieve the data or verify that the data content matches the hash specified by the locator.

Returns the data size (content length) reported by the Keep service and the URI reporting the data size.

func (*KeepClient) ClearBlockCache 

func (kc *KeepClient) ClearBlockCache()

func (*KeepClient) CollectionFileReader 

func (kc *KeepClient) CollectionFileReader(collection map[string]interface{}, filename string) (arvados.File, error)

CollectionFileReader returns a Reader that reads content from a single file in the collection. The filename must be relative to the root of the collection. A leading prefix of "/" or "./" in the filename is ignored.

func (*KeepClient) GatewayRoots 

func (kc *KeepClient) GatewayRoots() map[string]string

GatewayRoots returns the map of Keep remote gateway services: uuid -> baseURI.

func (*KeepClient) Get 

func (kc *KeepClient) Get(locator string) (io.ReadCloser, int64, string, error)

Get retrieves a block, given a locator. Returns a reader, the expected data length, the URL the block is being fetched from, and an error.

If the block checksum does not match, the final Read() on the reader returned by this method will return a BadChecksum error instead of EOF.

func (*KeepClient) GetIndex 

func (kc *KeepClient) GetIndex(keepServiceUUID, prefix string) (io.Reader, error)

GetIndex retrieves a list of blocks stored on the given server whose hashes begin with the given prefix. The returned reader will return an error (other than EOF) if the complete index cannot be retrieved.

This is meant to be used only by system components and admin tools. It will return an error unless the client is using a "data manager token" recognized by the Keep services.

func (*KeepClient) LoadKeepServicesFromJSON 

func (kc *KeepClient) LoadKeepServicesFromJSON(services string) error

LoadKeepServicesFromJSON gets list of available keep services from given JSON and disables automatic service discovery.

func (*KeepClient) LocalLocator 

func (kc *KeepClient) LocalLocator(locator string) (string, error)

LocalLocator returns a locator equivalent to the one supplied, but with a valid signature from the local cluster. If the given locator already has a local signature, it is returned unchanged.

func (*KeepClient) LocalRoots 

func (kc *KeepClient) LocalRoots() map[string]string

LocalRoots returns the map of local (i.e., disk and proxy) Keep services: uuid -> baseURI.

func (*KeepClient) ManifestFileReader 

func (kc *KeepClient) ManifestFileReader(m manifest.Manifest, filename string) (arvados.File, error)

func (*KeepClient) PutB 

func (kc *KeepClient) PutB(buffer []byte) (string, int, error)

PutB writes a block to Keep. It computes the hash itself.

Return values are the same as for PutHR.

func (*KeepClient) PutHB 

func (kc *KeepClient) PutHB(hash string, buf []byte) (string, int, error)

PutHB writes a block to Keep. The hash of the bytes is given in hash, and the data is given in buf.

Return values are the same as for PutHR.

func (*KeepClient) PutHR 

func (kc *KeepClient) PutHR(hash string, r io.Reader, dataBytes int64) (string, int, error)

PutHR puts a block given the block hash, a reader, and the number of bytes to read from the reader (which must be between 0 and BLOCKSIZE).

Returns the locator for the written block, the number of replicas written, and an error.

Returns an InsufficientReplicasError if 0 <= replicas < kc.Wants_replicas.

func (*KeepClient) PutR 

func (kc *KeepClient) PutR(r io.Reader) (locator string, replicas int, err error)

PutR writes a block to Keep. It first reads all data from r into a buffer in order to compute the hash.

Return values are the same as for PutHR.

If the block hash and data size are known, PutHR is more efficient.

func (*KeepClient) ReadAt 

func (kc *KeepClient) ReadAt(locator string, p []byte, off int) (int, error)

ReadAt retrieves a portion of block from the cache if it's present, otherwise from the network.

func (*KeepClient) RefreshServiceDiscovery 

func (kc *KeepClient) RefreshServiceDiscovery()

func (*KeepClient) SetServiceRoots 

func (kc *KeepClient) SetServiceRoots(locals, writables, gateways map[string]string)

SetServiceRoots disables service discovery and updates the localRoots and gatewayRoots maps, without disrupting operations that are already in progress.

The supplied maps must not be modified after calling SetServiceRoots.

func (*KeepClient) WritableLocalRoots 

func (kc *KeepClient) WritableLocalRoots() map[string]string

WritableLocalRoots returns the map of writable local Keep services: uuid -> baseURI.

type Locator 

type Locator struct {
	Hash  string
	Size  int      // -1 if data size is not known
	Hints []string // Including the size hint, if any
}

func MakeLocator 

func MakeLocator(path string) (*Locator, error)

func (*Locator) String 

func (loc *Locator) String() string

type OversizeBlockError 

type OversizeBlockError error

type RootSorter 

type RootSorter struct {
	// contains filtered or unexported fields
}

func NewRootSorter 

func NewRootSorter(serviceRoots map[string]string, hash string) *RootSorter

func (RootSorter) GetSortedRoots 

func (rs RootSorter) GetSortedRoots() []string

func (RootSorter) Len 

func (rs RootSorter) Len() int

func (RootSorter) Less 

func (rs RootSorter) Less(i, j int) bool

Less is really More here: the heaviest root will be at the front of the list.

func (RootSorter) Swap 

func (rs RootSorter) Swap(i, j int)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.