blobclient

package
v0.1.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 26, 2019 License: Apache-2.0 Imports: 22 Imported by: 26

Documentation

Overview

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

Constants

This section is empty.

Variables

View Source 
var ErrBlobNotFound = errors.New("blob not found")

ErrBlobNotFound is returned when a blob is not found on origin.

Functions

func Locations 

func Locations(p Provider, cluster hostlist.List, d core.Digest) (locs []string, err error)

Locations queries cluster for the locations of d.

func Poll 

func Poll(
	r ClientResolver, b backoff.BackOff, d core.Digest, makeRequest func(Client) error) error

Poll wraps requests for endpoints which require polling, due to a blob being asynchronously fetched from remote storage in the origin cluster.

Types

type Client 

type Client interface {
	Addr() string

	Locations(d core.Digest) ([]string, error)
	DeleteBlob(d core.Digest) error
	TransferBlob(d core.Digest, blob io.Reader) error

	Stat(namespace string, d core.Digest) (*core.BlobInfo, error)
	StatLocal(namespace string, d core.Digest) (*core.BlobInfo, error)

	GetMetaInfo(namespace string, d core.Digest) (*core.MetaInfo, error)
	OverwriteMetaInfo(d core.Digest, pieceLength int64) error

	UploadBlob(namespace string, d core.Digest, blob io.Reader) error
	DuplicateUploadBlob(namespace string, d core.Digest, blob io.Reader, delay time.Duration) error

	DownloadBlob(namespace string, d core.Digest, dst io.Writer) error

	ReplicateToRemote(namespace string, d core.Digest, remoteDNS string) error

	GetPeerContext() (core.PeerContext, error)

	ForceCleanup(ttl time.Duration) error
}

Client provides a wrapper around all Server HTTP endpoints.

type ClientResolver 

type ClientResolver interface {
	// Resolve must return an ordered, stable list of Clients for origins owning d.
	Resolve(d core.Digest) ([]Client, error)
}

ClientResolver resolves digests into Clients of origins.

func NewClientResolver 

func NewClientResolver(p Provider, cluster hostlist.List) ClientResolver

NewClientResolver returns a new client resolver.

type ClusterClient 

type ClusterClient interface {
	UploadBlob(namespace string, d core.Digest, blob io.Reader) error
	DownloadBlob(namespace string, d core.Digest, dst io.Writer) error
	GetMetaInfo(namespace string, d core.Digest) (*core.MetaInfo, error)
	Stat(namespace string, d core.Digest) (*core.BlobInfo, error)
	OverwriteMetaInfo(d core.Digest, pieceLength int64) error
	Owners(d core.Digest) ([]core.PeerContext, error)
	ReplicateToRemote(namespace string, d core.Digest, remoteDNS string) error
}

ClusterClient defines a top-level origin cluster client which handles blob location resolution and retries.

func NewClusterClient 

func NewClusterClient(r ClientResolver) ClusterClient

NewClusterClient returns a new ClusterClient.

type ClusterProvider 

type ClusterProvider interface {
	Provide(dns string) (ClusterClient, error)
}

ClusterProvider creates ClusterClients from dns records.

type DuplicateCommitUploadRequest 

type DuplicateCommitUploadRequest struct {
	Delay time.Duration `yaml:"delay"`
}

DuplicateCommitUploadRequest defines HTTP request body.

type HTTPClient 

type HTTPClient struct {
	// contains filtered or unexported fields
}

HTTPClient defines the Client implementation.

func New 

func New(addr string, opts ...Option) *HTTPClient

New returns a new HTTPClient scoped to addr.

func (*HTTPClient) Addr 

func (c *HTTPClient) Addr() string

Addr returns the address of the server the client is provisioned for.

func (*HTTPClient) DeleteBlob 

func (c *HTTPClient) DeleteBlob(d core.Digest) error

DeleteBlob deletes the blob corresponding to d.

func (*HTTPClient) DownloadBlob 

func (c *HTTPClient) DownloadBlob(namespace string, d core.Digest, dst io.Writer) error

DownloadBlob downloads blob for d. If the blob of d is not available yet (i.e. still downloading), returns 202 httputil.StatusError, indicating that the request shoudl be retried later. If not blob exists for d, returns a 404 httputil.StatusError.

func (*HTTPClient) DuplicateUploadBlob 

func (c *HTTPClient) DuplicateUploadBlob(
	namespace string, d core.Digest, blob io.Reader, delay time.Duration) error

DuplicateUploadBlob duplicates an blob upload request, which will attempt to write-back at the given delay.

func (*HTTPClient) ForceCleanup 

func (c *HTTPClient) ForceCleanup(ttl time.Duration) error

ForceCleanup forces cache cleanup to run.

func (*HTTPClient) GetMetaInfo 

func (c *HTTPClient) GetMetaInfo(namespace string, d core.Digest) (*core.MetaInfo, error)

GetMetaInfo returns metainfo for d. If the blob of d is not available yet (i.e. still downloading), returns a 202 httputil.StatusError, indicating that the request should be retried later. If no blob exists for d, returns a 404 httputil.StatusError.

func (*HTTPClient) GetPeerContext 

func (c *HTTPClient) GetPeerContext() (core.PeerContext, error)

GetPeerContext gets the PeerContext of the p2p client running alongside the Server.

func (*HTTPClient) Locations 

func (c *HTTPClient) Locations(d core.Digest) ([]string, error)

Locations returns the origin server addresses which d is sharded on.

func (*HTTPClient) OverwriteMetaInfo 

func (c *HTTPClient) OverwriteMetaInfo(d core.Digest, pieceLength int64) error

OverwriteMetaInfo overwrites existing metainfo for d with new metainfo configured with pieceLength. Primarily intended for benchmarking purposes.

func (*HTTPClient) ReplicateToRemote 

func (c *HTTPClient) ReplicateToRemote(namespace string, d core.Digest, remoteDNS string) error

ReplicateToRemote replicates the blob of d to a remote origin cluster. If the blob of d is not available yet, returns 202 httputil.StatusError, indicating that the request should be retried later.

func (*HTTPClient) Stat 

func (c *HTTPClient) Stat(namespace string, d core.Digest) (*core.BlobInfo, error)

Stat returns blob info. It returns error if the origin does not have a blob for d.

func (*HTTPClient) StatLocal 

func (c *HTTPClient) StatLocal(namespace string, d core.Digest) (*core.BlobInfo, error)

StatLocal returns blob info. It returns error if the origin does not have a blob for d locally.

func (*HTTPClient) TransferBlob 

func (c *HTTPClient) TransferBlob(d core.Digest, blob io.Reader) error

TransferBlob uploads a blob to a single origin server. Unlike its cousin UploadBlob, TransferBlob is an internal API which does not replicate the blob.

func (*HTTPClient) UploadBlob 

func (c *HTTPClient) UploadBlob(namespace string, d core.Digest, blob io.Reader) error

UploadBlob uploads and replicates blob to the origin cluster, asynchronously backing the blob up to the remote storage configured for namespace.

type HTTPClusterProvider 

type HTTPClusterProvider struct {
	// contains filtered or unexported fields
}

HTTPClusterProvider provides ClusterClients backed by HTTP. Does not include health checks.

func NewClusterProvider 

func NewClusterProvider(opts ...Option) HTTPClusterProvider

NewClusterProvider returns a new HTTPClusterProvider.

func (HTTPClusterProvider) Provide 

func (p HTTPClusterProvider) Provide(dns string) (ClusterClient, error)

Provide creates a new ClusterClient.

type HTTPProvider 

type HTTPProvider struct {
	// contains filtered or unexported fields
}

HTTPProvider provides HTTPClients.

func NewProvider 

func NewProvider(opts ...Option) HTTPProvider

NewProvider returns a new HTTPProvider.

func (HTTPProvider) Provide 

func (p HTTPProvider) Provide(addr string) Client

Provide implements ClientProvider's Provide. TODO(codyg): Make this return error.

type Option 

type Option func(*HTTPClient)

Option allows setting optional HTTPClient parameters.

func WithChunkSize 

func WithChunkSize(s uint64) Option

WithChunkSize configures an HTTPClient with a custom chunk size for uploads.

func WithTLS 

func WithTLS(tls *tls.Config) Option

WithTLS configures an HTTPClient with tls configuration.

type Provider 

type Provider interface {
	Provide(addr string) Client
}

Provider defines an interface for creating Client scoped to an origin addr.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.