blobclient

package
Version: v0.1.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 6, 2020 License: Apache-2.0 Imports: 22 Imported by: 13

Documentation

Overview

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

Constants

This section is empty.

Variables

View Source
var ErrBlobNotFound = errors.New("blob not found")

ErrBlobNotFound is returned when a blob is not found on origin.

Functions

func Locations

func Locations(p Provider, cluster hostlist.List, d core.Digest) (locs []string, err error)

Locations queries cluster for the locations of d.

func Poll

func Poll(
	r ClientResolver, b backoff.BackOff, d core.Digest, makeRequest func(Client) error) error

Poll wraps requests for endpoints which require polling, due to a blob being asynchronously fetched from remote storage in the origin cluster.

Types

type Client

type Client interface {
	Addr() string

	Locations(d core.Digest) ([]string, error)
	DeleteBlob(d core.Digest) error
	TransferBlob(d core.Digest, blob io.Reader) error

	Stat(namespace string, d core.Digest) (*core.BlobInfo, error)
	StatLocal(namespace string, d core.Digest) (*core.BlobInfo, error)

	GetMetaInfo(namespace string, d core.Digest) (*core.MetaInfo, error)
	OverwriteMetaInfo(d core.Digest, pieceLength int64) error

	UploadBlob(namespace string, d core.Digest, blob io.Reader) error
	DuplicateUploadBlob(namespace string, d core.Digest, blob io.Reader, delay time.Duration) error

	DownloadBlob(namespace string, d core.Digest, dst io.Writer) error

	ReplicateToRemote(namespace string, d core.Digest, remoteDNS string) error

	GetPeerContext() (core.PeerContext, error)

	ForceCleanup(ttl time.Duration) error
}

Client provides a wrapper around all Server HTTP endpoints.

type ClientResolver

type ClientResolver interface {
	// Resolve must return an ordered, stable list of Clients for origins owning d.
	Resolve(d core.Digest) ([]Client, error)
}

ClientResolver resolves digests into Clients of origins.

func NewClientResolver

func NewClientResolver(p Provider, cluster hostlist.List) ClientResolver

NewClientResolver returns a new client resolver.

type ClusterClient

type ClusterClient interface {
	UploadBlob(namespace string, d core.Digest, blob io.Reader) error
	DownloadBlob(namespace string, d core.Digest, dst io.Writer) error
	GetMetaInfo(namespace string, d core.Digest) (*core.MetaInfo, error)
	Stat(namespace string, d core.Digest) (*core.BlobInfo, error)
	OverwriteMetaInfo(d core.Digest, pieceLength int64) error
	Owners(d core.Digest) ([]core.PeerContext, error)
	ReplicateToRemote(namespace string, d core.Digest, remoteDNS string) error
}

ClusterClient defines a top-level origin cluster client which handles blob location resolution and retries.

func NewClusterClient

func NewClusterClient(r ClientResolver) ClusterClient

NewClusterClient returns a new ClusterClient.

type ClusterProvider

type ClusterProvider interface {
	Provide(dns string) (ClusterClient, error)
}

ClusterProvider creates ClusterClients from dns records.

type DuplicateCommitUploadRequest

type DuplicateCommitUploadRequest struct {
	Delay time.Duration `yaml:"delay"`
}

DuplicateCommitUploadRequest defines HTTP request body.

type HTTPClient

type HTTPClient struct {
	// contains filtered or unexported fields
}

HTTPClient defines the Client implementation.

func New

func New(addr string, opts ...Option) *HTTPClient

New returns a new HTTPClient scoped to addr.

func (*HTTPClient) Addr

func (c *HTTPClient) Addr() string

Addr returns the address of the server the client is provisioned for.

func (*HTTPClient) DeleteBlob

func (c *HTTPClient) DeleteBlob(d core.Digest) error

DeleteBlob deletes the blob corresponding to d.

func (*HTTPClient) DownloadBlob

func (c *HTTPClient) DownloadBlob(namespace string, d core.Digest, dst io.Writer) error

DownloadBlob downloads blob for d. If the blob of d is not available yet (i.e. still downloading), returns 202 httputil.StatusError, indicating that the request shoudl be retried later. If not blob exists for d, returns a 404 httputil.StatusError.

func (*HTTPClient) DuplicateUploadBlob

func (c *HTTPClient) DuplicateUploadBlob(
	namespace string, d core.Digest, blob io.Reader, delay time.Duration) error

DuplicateUploadBlob duplicates an blob upload request, which will attempt to write-back at the given delay.

func (*HTTPClient) ForceCleanup

func (c *HTTPClient) ForceCleanup(ttl time.Duration) error

ForceCleanup forces cache cleanup to run.

func (*HTTPClient) GetMetaInfo

func (c *HTTPClient) GetMetaInfo(namespace string, d core.Digest) (*core.MetaInfo, error)

GetMetaInfo returns metainfo for d. If the blob of d is not available yet (i.e. still downloading), returns a 202 httputil.StatusError, indicating that the request should be retried later. If no blob exists for d, returns a 404 httputil.StatusError.

func (*HTTPClient) GetPeerContext

func (c *HTTPClient) GetPeerContext() (core.PeerContext, error)

GetPeerContext gets the PeerContext of the p2p client running alongside the Server.

func (*HTTPClient) Locations

func (c *HTTPClient) Locations(d core.Digest) ([]string, error)

Locations returns the origin server addresses which d is sharded on.

func (*HTTPClient) OverwriteMetaInfo

func (c *HTTPClient) OverwriteMetaInfo(d core.Digest, pieceLength int64) error

OverwriteMetaInfo overwrites existing metainfo for d with new metainfo configured with pieceLength. Primarily intended for benchmarking purposes.

func (*HTTPClient) ReplicateToRemote

func (c *HTTPClient) ReplicateToRemote(namespace string, d core.Digest, remoteDNS string) error

ReplicateToRemote replicates the blob of d to a remote origin cluster. If the blob of d is not available yet, returns 202 httputil.StatusError, indicating that the request should be retried later.

func (*HTTPClient) Stat

func (c *HTTPClient) Stat(namespace string, d core.Digest) (*core.BlobInfo, error)

Stat returns blob info. It returns error if the origin does not have a blob for d.

func (*HTTPClient) StatLocal

func (c *HTTPClient) StatLocal(namespace string, d core.Digest) (*core.BlobInfo, error)

StatLocal returns blob info. It returns error if the origin does not have a blob for d locally.

func (*HTTPClient) TransferBlob

func (c *HTTPClient) TransferBlob(d core.Digest, blob io.Reader) error

TransferBlob uploads a blob to a single origin server. Unlike its cousin UploadBlob, TransferBlob is an internal API which does not replicate the blob.

func (*HTTPClient) UploadBlob

func (c *HTTPClient) UploadBlob(namespace string, d core.Digest, blob io.Reader) error

UploadBlob uploads and replicates blob to the origin cluster, asynchronously backing the blob up to the remote storage configured for namespace.

type HTTPClusterProvider

type HTTPClusterProvider struct {
	// contains filtered or unexported fields
}

HTTPClusterProvider provides ClusterClients backed by HTTP. Does not include health checks.

func NewClusterProvider

func NewClusterProvider(opts ...Option) HTTPClusterProvider

NewClusterProvider returns a new HTTPClusterProvider.

func (HTTPClusterProvider) Provide

func (p HTTPClusterProvider) Provide(dns string) (ClusterClient, error)

Provide creates a new ClusterClient.

type HTTPProvider

type HTTPProvider struct {
	// contains filtered or unexported fields
}

HTTPProvider provides HTTPClients.

func NewProvider

func NewProvider(opts ...Option) HTTPProvider

NewProvider returns a new HTTPProvider.

func (HTTPProvider) Provide

func (p HTTPProvider) Provide(addr string) Client

Provide implements ClientProvider's Provide. TODO(codyg): Make this return error.

type Option

type Option func(*HTTPClient)

Option allows setting optional HTTPClient parameters.

func WithChunkSize

func WithChunkSize(s uint64) Option

WithChunkSize configures an HTTPClient with a custom chunk size for uploads.

func WithTLS

func WithTLS(tls *tls.Config) Option

WithTLS configures an HTTPClient with tls configuration.

type Provider

type Provider interface {
	Provide(addr string) Client
}

Provider defines an interface for creating Client scoped to an origin addr.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL