request

package module
v0.0.0-...-35b8a9d Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 19, 2026 License: Apache-2.0 Imports: 28 Imported by: 0

README

client-request (Go)

Go Reference LICENSE

Request library for the Dragonfly client. It sends requests to remote servers via the Dragonfly P2P network, supporting streaming and buffered GET requests and preheating files or OCI images through seed peers.

It is the Go implementation of the Rust crate dragonfly-client-request and generates identical task ids and seed peer selections, pinned by shared cross-language test vectors.

Install

go get d7y.io/dragonfly-sdk/client-request/go

Usage

import (
    "context"

    request "d7y.io/dragonfly-sdk/client-request/go"
)

func main() {
    ctx := context.Background()
    proxy, err := request.New(ctx, "http://127.0.0.1:8002")
    if err != nil {
        panic(err)
    }
    defer proxy.Close()

    resp, err := proxy.Get(ctx, request.NewGetRequest("https://example.com/file.txt"))
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()
    // Read resp.Body...
}

Optional request parameters are set with With* options:

req := request.NewGetRequest(
    "https://example.com/file.txt",
    request.WithGetRequestTag("tag"),
    request.WithGetRequestApplication("app"),
    request.WithGetRequestTimeout(30*time.Second),
)

Preheat a file or an OCI image to the seed peers:

if err := proxy.Preheat(ctx, request.NewPreheatRequest("https://example.com/file.txt")); err != nil {
    panic(err)
}

if err := proxy.PreheatImage(ctx, request.NewPreheatImageRequest("docker.io/library/nginx:latest")); err != nil {
    panic(err)
}

Preheat with multiple replicas and scatter downloads across them. Preheating writes the file to the given number of distinct seed peers, and downloading scatters each request across those replicas by picking a random one, retrying on the others up to the max retries. Preheating fails when the available seed peers are fewer than the replicas, while downloading clamps the replicas to the available seed peers. The default replicas is 2:

if err := proxy.Preheat(ctx, request.NewPreheatRequest("https://example.com/file.txt", request.WithPreheatRequestReplicas(3))); err != nil {
    panic(err)
}

resp, err := proxy.Get(ctx, request.NewGetRequest("https://example.com/file.txt", request.WithGetRequestReplicas(3)))
if err != nil {
    panic(err)
}
defer resp.Body.Close()

Look up the endpoints of the seed peers serving a request, then create a proxy bound to those endpoints and download from them directly, scattering the request across them. The endpoints proxy keeps a client with a reusable connection pool per endpoint and doesn't sync seed peers from the scheduler:

req := request.NewGetRequest("https://example.com/file.txt")
endpoints, err := proxy.LookupEndpoints(ctx, req)
if err != nil {
    panic(err)
}

proxyWithEndpoints, err := request.NewWithEndpoints(endpoints)
if err != nil {
    panic(err)
}

resp, err := proxyWithEndpoints.Get(ctx, req)
if err != nil {
    panic(err)
}
defer resp.Body.Close()

// Or write the response body directly into a writer:
// resp, err := proxyWithEndpoints.GetInto(ctx, req, w)

See examples for runnable examples.

Documentation

You can find the full documentation on d7y.io.

LICENSE

Apache 2.0 License. Please see LICENSE for more information.

Documentation

Overview

Package request sends requests to remote servers via the Dragonfly P2P network. It is the Go implementation of the Rust crate dragonfly-client-request and generates identical task ids and seed peer selections.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrRequestTimeout indicates the request timed out.
	ErrRequestTimeout = errors.New("request timeout")

	// ErrInvalidArgument indicates an invalid argument.
	ErrInvalidArgument = errors.New("invalid argument")

	// ErrInternal indicates a request internal error.
	ErrInternal = errors.New("request internal error")
)

Functions

This section is empty.

Types

type BackendError 

type BackendError struct {
	// Message is the backend error message.
	Message string

	// Header is the backend HTTP response header.
	Header http.Header

	// StatusCode is the backend HTTP status code.
	StatusCode int
}

BackendError is the error detail returned by the backend server.

func (*BackendError) Error 

func (e *BackendError) Error() string

Error implements the error interface.

type DfdaemonError 

type DfdaemonError struct {
	// Message is the dfdaemon error message.
	Message string
}

DfdaemonError is the error detail returned by the dfdaemon.

func (*DfdaemonError) Error 

func (e *DfdaemonError) Error() string

Error implements the error interface.

type GetRequest 

type GetRequest struct {
	// contains filtered or unexported fields
}

GetRequest represents a GET request to be sent via the Dragonfly. Construct it with NewGetRequest and set the optional parameters with GetRequestOption.

func NewGetRequest 

func NewGetRequest(url string, opts ...GetRequestOption) *GetRequest

NewGetRequest returns a GetRequest for the url with default values: the default filtered query params, blob digest based task id enabled and a 30 minutes timeout.

type GetRequestOption 

type GetRequestOption func(r *GetRequest)

GetRequestOption configures the GetRequest.

func WithGetRequestApplication 

func WithGetRequestApplication(application string) GetRequestOption

WithGetRequestApplication sets the application that identifies different tasks for the same url.

func WithGetRequestCertificates 

func WithGetRequestCertificates(certs []*x509.Certificate) GetRequestOption

WithGetRequestCertificates sets the client certificates for the request. TODO(chlins): Support client certificates.

func WithGetRequestContentForCalculatingTaskID 

func WithGetRequestContentForCalculatingTaskID(content string) GetRequestOption

WithGetRequestContentForCalculatingTaskID sets the content for calculating the task id. This is used when the task id cannot be calculated based on the url and other parameters, such as when the url contains dynamic query parameters that cannot be filtered out.

func WithGetRequestEnableTaskIDBasedBlobDigest 

func WithGetRequestEnableTaskIDBasedBlobDigest(enable bool) GetRequestOption

WithGetRequestEnableTaskIDBasedBlobDigest sets whether to use the blob digest for task id calculation when downloading from OCI registries. When enabled for OCI blob urls (e.g., /v2/<name>/blobs/sha256:<digest>), the task id is derived from the blob digest rather than the full url. This enables deduplication across registries.

func WithGetRequestFilteredQueryParams 

func WithGetRequestFilteredQueryParams(params []string) GetRequestOption

WithGetRequestFilteredQueryParams sets the filtered query params to generate the task id. When the filter is ["Signature", "Expires", "ns"], for example: http://example.com/xyz?Expires=e1&Signature=s1&ns=docker.io and http://example.com/xyz?Expires=e2&Signature=s2&ns=docker.io will generate the same task id.

func WithGetRequestHeader 

func WithGetRequestHeader(header http.Header) GetRequestOption

WithGetRequestHeader sets the headers of the request.

func WithGetRequestPieceLength 

func WithGetRequestPieceLength(pieceLength uint64) GetRequestOption

WithGetRequestPieceLength sets the task piece length.

func WithGetRequestPriority 

func WithGetRequestPriority(priority int32) GetRequestOption

WithGetRequestPriority sets the task priority, refer to https://github.com/dragonflyoss/api/blob/main/proto/common.proto#L67.

func WithGetRequestReplicas 

func WithGetRequestReplicas(replicas int) GetRequestOption

WithGetRequestReplicas sets the number of seed peers serving the task, default is 2. The request is scattered across the replicas.

func WithGetRequestTag 

func WithGetRequestTag(tag string) GetRequestOption

WithGetRequestTag sets the tag that identifies different tasks for the same url.

func WithGetRequestTimeout 

func WithGetRequestTimeout(timeout time.Duration) GetRequestOption

WithGetRequestTimeout sets the timeout of the request.

type GetResponse 

type GetResponse struct {
	// Success indicates whether the response is successful.
	Success bool

	// Header is the headers of the response.
	Header http.Header

	// StatusCode is the status code of the response.
	StatusCode int

	// Body is the content of the response. It is nil for GetInto and must be
	// closed by the caller for Get.
	Body io.ReadCloser
}

GetResponse represents a GET response received via the Dragonfly.

type PreheatImageRequest 

type PreheatImageRequest struct {
	// contains filtered or unexported fields
}

PreheatImageRequest represents a request to preheat an OCI image through the Dragonfly seed client. The preheat downloads all blobs (config and layers) of the specified image via the Dragonfly proxy. Construct it with NewPreheatImageRequest and set the optional parameters with PreheatImageRequestOption.

func NewPreheatImageRequest 

func NewPreheatImageRequest(image string, opts ...PreheatImageRequestOption) *PreheatImageRequest

NewPreheatImageRequest returns a PreheatImageRequest for the image with default values.

type PreheatImageRequestOption 

type PreheatImageRequestOption func(r *PreheatImageRequest)

PreheatImageRequestOption configures the PreheatImageRequest.

func WithPreheatImageRequestApplication 

func WithPreheatImageRequestApplication(application string) PreheatImageRequestOption

WithPreheatImageRequestApplication sets the application that identifies different tasks for the same url.

func WithPreheatImageRequestAuth 

func WithPreheatImageRequestAuth(username, password string) PreheatImageRequestOption

WithPreheatImageRequestAuth sets the username and password for registry authentication. If not provided, anonymous access is used.

func WithPreheatImageRequestCertificates 

func WithPreheatImageRequestCertificates(certs []*x509.Certificate) PreheatImageRequestOption

WithPreheatImageRequestCertificates sets the client certificates for the request. TODO(chlins): Support client certificates.

func WithPreheatImageRequestConcurrentTaskCount 

func WithPreheatImageRequestConcurrentTaskCount(count int) PreheatImageRequestOption

WithPreheatImageRequestConcurrentTaskCount sets the number of blobs to preheat concurrently, default is 4.

func WithPreheatImageRequestContentForCalculatingTaskID 

func WithPreheatImageRequestContentForCalculatingTaskID(content string) PreheatImageRequestOption

WithPreheatImageRequestContentForCalculatingTaskID sets the content for calculating the task id.

func WithPreheatImageRequestEnableTaskIDBasedBlobDigest 

func WithPreheatImageRequestEnableTaskIDBasedBlobDigest(enable bool) PreheatImageRequestOption

WithPreheatImageRequestEnableTaskIDBasedBlobDigest sets whether to use the blob digest for task id calculation when downloading from OCI registries.

func WithPreheatImageRequestFilteredQueryParams 

func WithPreheatImageRequestFilteredQueryParams(params []string) PreheatImageRequestOption

WithPreheatImageRequestFilteredQueryParams sets the filtered query params to generate the task id.

func WithPreheatImageRequestPieceLength 

func WithPreheatImageRequestPieceLength(pieceLength uint64) PreheatImageRequestOption

WithPreheatImageRequestPieceLength sets the task piece length.

func WithPreheatImageRequestPlatform 

func WithPreheatImageRequestPlatform(platform string) PreheatImageRequestOption

WithPreheatImageRequestPlatform sets the target platform in the format "os/arch" (e.g., "linux/amd64", "linux/arm64"). This is used to select the correct manifest from a multi-platform image index, default is current platform.

func WithPreheatImageRequestPriority 

func WithPreheatImageRequestPriority(priority int32) PreheatImageRequestOption

WithPreheatImageRequestPriority sets the task priority.

func WithPreheatImageRequestReplicas 

func WithPreheatImageRequestReplicas(replicas int) PreheatImageRequestOption

WithPreheatImageRequestReplicas sets the number of seed peers to preheat each blob task to, default is 2.

func WithPreheatImageRequestTag 

func WithPreheatImageRequestTag(tag string) PreheatImageRequestOption

WithPreheatImageRequestTag sets the tag that identifies different tasks for the same url.

func WithPreheatImageRequestTimeout 

func WithPreheatImageRequestTimeout(timeout time.Duration) PreheatImageRequestOption

WithPreheatImageRequestTimeout sets the timeout for each blob download request.

type PreheatRequest 

type PreheatRequest struct {
	// contains filtered or unexported fields
}

PreheatRequest represents a request to preheat a file through the Dragonfly seed client. The preheat downloads the specified file via the Dragonfly proxy, effectively caching it in the P2P network for faster downloading. Construct it with NewPreheatRequest and set the optional parameters with PreheatRequestOption.

func NewPreheatRequest 

func NewPreheatRequest(url string, opts ...PreheatRequestOption) *PreheatRequest

NewPreheatRequest returns a PreheatRequest for the url with default values.

type PreheatRequestOption 

type PreheatRequestOption func(r *PreheatRequest)

PreheatRequestOption configures the PreheatRequest.

func WithPreheatRequestApplication 

func WithPreheatRequestApplication(application string) PreheatRequestOption

WithPreheatRequestApplication sets the application that identifies different tasks for the same url.

func WithPreheatRequestCertificates 

func WithPreheatRequestCertificates(certs []*x509.Certificate) PreheatRequestOption

WithPreheatRequestCertificates sets the client certificates for the request. TODO(chlins): Support client certificates.

func WithPreheatRequestContentForCalculatingTaskID 

func WithPreheatRequestContentForCalculatingTaskID(content string) PreheatRequestOption

WithPreheatRequestContentForCalculatingTaskID sets the content for calculating the task id.

func WithPreheatRequestEnableTaskIDBasedBlobDigest 

func WithPreheatRequestEnableTaskIDBasedBlobDigest(enable bool) PreheatRequestOption

WithPreheatRequestEnableTaskIDBasedBlobDigest sets whether to use the blob digest for task id calculation when downloading from OCI registries.

func WithPreheatRequestFilteredQueryParams 

func WithPreheatRequestFilteredQueryParams(params []string) PreheatRequestOption

WithPreheatRequestFilteredQueryParams sets the filtered query params to generate the task id.

func WithPreheatRequestHeader 

func WithPreheatRequestHeader(header http.Header) PreheatRequestOption

WithPreheatRequestHeader sets the headers of the request.

func WithPreheatRequestPieceLength 

func WithPreheatRequestPieceLength(pieceLength uint64) PreheatRequestOption

WithPreheatRequestPieceLength sets the task piece length.

func WithPreheatRequestPriority 

func WithPreheatRequestPriority(priority int32) PreheatRequestOption

WithPreheatRequestPriority sets the task priority.

func WithPreheatRequestReplicas 

func WithPreheatRequestReplicas(replicas int) PreheatRequestOption

WithPreheatRequestReplicas sets the number of seed peers to preheat the task to, default is 2.

func WithPreheatRequestTag 

func WithPreheatRequestTag(tag string) PreheatRequestOption

WithPreheatRequestTag sets the tag that identifies different tasks for the same url.

func WithPreheatRequestTimeout 

func WithPreheatRequestTimeout(timeout time.Duration) PreheatRequestOption

WithPreheatRequestTimeout sets the timeout of the request.

type Proxy 

type Proxy struct {
	// contains filtered or unexported fields
}

Proxy is the HTTP proxy client that sends requests via Dragonfly.

func New 

func New(ctx context.Context, schedulerEndpoint string, opts ...ProxyOption) (*Proxy, error)

New creates a Proxy that connects to the given scheduler endpoint, e.g. "http://scheduler-service:8002".

func (*Proxy) Close 

func (p *Proxy) Close() error

Close stops the background seed peer refresh and closes the scheduler connection.

func (*Proxy) Get 

func (p *Proxy) Get(ctx context.Context, req *GetRequest) (*GetResponse, error)

Get sends a GET request to a remote server via the Dragonfly and returns a response with a streaming body. The caller must close the body.

func (*Proxy) GetInto 

func (p *Proxy) GetInto(ctx context.Context, req *GetRequest, w io.Writer) (*GetResponse, error)

GetInto sends a GET request to a remote server via the Dragonfly and writes the response body directly into the provided writer.

func (*Proxy) LookupEndpoints 

func (p *Proxy) LookupEndpoints(ctx context.Context, req *GetRequest) ([]string, error)

LookupEndpoints looks up the endpoints (e.g., "http://127.0.0.1:4000") of the seed peers serving the request, in the consistent hash ring selection order for the request's task id. It returns up to the replicas of the request distinct endpoints, clamped to the number of available seed peers.

func (*Proxy) Preheat 

func (p *Proxy) Preheat(ctx context.Context, req *PreheatRequest) error

Preheat preheats a file by downloading it to the replicas of seed peers via the Dragonfly. It triggers every replica seed peer to download the file by the dfdaemon download task API, without streaming the file content back to the client. It fails when the available seed peers are fewer than the replicas of the request.

func (*Proxy) PreheatImage 

func (p *Proxy) PreheatImage(ctx context.Context, req *PreheatImageRequest) error

PreheatImage preheats an OCI image by downloading all its blobs via the Dragonfly. It parses the image reference, authenticates with the OCI registry, resolves the image manifest (including multi-platform image indexes), and triggers the seed client to download each blob (config and layers).

type ProxyError 

type ProxyError struct {
	// Message is the proxy error message.
	Message string

	// Header is the proxy HTTP response header.
	Header http.Header

	// StatusCode is the proxy HTTP status code.
	StatusCode int
}

ProxyError is the error detail returned by the proxy server.

func (*ProxyError) Error 

func (e *ProxyError) Error() string

Error implements the error interface.

type ProxyOption 

type ProxyOption func(p *Proxy)

ProxyOption configures the Proxy.

func WithProxyHealthCheckInterval 

func WithProxyHealthCheckInterval(interval time.Duration) ProxyOption

WithProxyHealthCheckInterval sets the interval of health check for seed peers.

func WithProxyMaxRetries 

func WithProxyMaxRetries(retries uint8) ProxyOption

WithProxyMaxRetries sets the maximum number of retries.

func WithProxySchedulerRequestTimeout 

func WithProxySchedulerRequestTimeout(timeout time.Duration) ProxyOption

WithProxySchedulerRequestTimeout sets the timeout of requests to the scheduler service.

type ProxyWithEndpoints 

type ProxyWithEndpoints struct {
	// contains filtered or unexported fields
}

ProxyWithEndpoints is the HTTP proxy client that sends requests via the fixed seed peer endpoints of the Dragonfly given at construction, without selecting seed peers by the consistent hash ring or syncing them from the scheduler.

func NewWithEndpoints 

func NewWithEndpoints(endpoints []string, opts ...ProxyWithEndpointsOption) (*ProxyWithEndpoints, error)

NewWithEndpoints creates a ProxyWithEndpoints that sends requests via the given seed peer endpoints of the Dragonfly (e.g., the ones returned by Proxy.LookupEndpoints), e.g. "http://127.0.0.1:4001". Each endpoint gets its own client with a reusable connection pool.

func (*ProxyWithEndpoints) Get 

func (p *ProxyWithEndpoints) Get(ctx context.Context, req *GetRequest) (*GetResponse, error)

Get sends a GET request to a remote server via the seed peer endpoints of the Dragonfly and returns a response with a streaming body. The request is sent to a randomly picked endpoint and retried on the others up to the max retries. The caller must close the body.

func (*ProxyWithEndpoints) GetInto 

func (p *ProxyWithEndpoints) GetInto(ctx context.Context, req *GetRequest, w io.Writer) (*GetResponse, error)

GetInto sends a GET request to a remote server via the seed peer endpoints of the Dragonfly and writes the response body directly into the provided writer. The request is sent to a randomly picked endpoint and retried on the others up to the max retries.

type ProxyWithEndpointsOption 

type ProxyWithEndpointsOption func(p *ProxyWithEndpoints)

ProxyWithEndpointsOption configures the ProxyWithEndpoints.

func WithProxyWithEndpointsMaxRetries 

func WithProxyWithEndpointsMaxRetries(retries uint8) ProxyWithEndpointsOption

WithProxyWithEndpointsMaxRetries sets the maximum number of retries.

type Request 

type Request interface {
	// Get sends a GET request to a remote server via the Dragonfly and returns
	// a response with a streaming body. The caller must close the body.
	Get(ctx context.Context, req *GetRequest) (*GetResponse, error)

	// GetInto sends a GET request to a remote server via the Dragonfly and
	// writes the response body directly into the provided writer.
	GetInto(ctx context.Context, req *GetRequest, w io.Writer) (*GetResponse, error)

	// Preheat preheats a file by downloading it to the replicas of seed peers
	// via the Dragonfly, without streaming the file content back to the
	// client. It fails when the available seed peers are fewer than the
	// replicas of the request.
	Preheat(ctx context.Context, req *PreheatRequest) error

	// PreheatImage preheats an OCI image by downloading all its blobs via the
	// Dragonfly. It resolves the image manifest (including multi-platform
	// image indexes) and triggers the seed client to download each blob.
	PreheatImage(ctx context.Context, req *PreheatImageRequest) error

	// LookupEndpoints looks up the endpoints of the seed peers serving the
	// request, in the consistent hash ring selection order for the request's
	// task id. It returns up to the replicas of the request distinct
	// endpoints, clamped to the number of available seed peers.
	LookupEndpoints(ctx context.Context, req *GetRequest) ([]string, error)
}

Request is the interface for sending requests via the Dragonfly.

It enables interaction with remote servers through the Dragonfly, shielding the complex request logic between the client and the Dragonfly seed client's proxy.

type RequestWithEndpoints 

type RequestWithEndpoints interface {
	// Get sends a GET request to a remote server via the seed peer endpoints
	// of the Dragonfly and returns a response with a streaming body. The
	// request is sent to a randomly picked endpoint and retried on the others
	// up to the max retries. The caller must close the body.
	Get(ctx context.Context, req *GetRequest) (*GetResponse, error)

	// GetInto sends a GET request to a remote server via the seed peer
	// endpoints of the Dragonfly and writes the response body directly into
	// the provided writer. The request is sent to a randomly picked endpoint
	// and retried on the others up to the max retries.
	GetInto(ctx context.Context, req *GetRequest, w io.Writer) (*GetResponse, error)
}

RequestWithEndpoints is the interface for sending requests via fixed seed peer endpoints of the Dragonfly.

Unlike Request, it sends requests to the seed peer endpoints given at construction (e.g., the ones returned by Request.LookupEndpoints), without selecting seed peers by the consistent hash ring or syncing them from the scheduler.

Source Files

View all Source files

Directories

Path Synopsis
examples
get command
Command get downloads a file via the Dragonfly and writes it to stdout.
 Command get downloads a file via the Dragonfly and writes it to stdout.
get-with-endpoints command
Command get-with-endpoints looks up the endpoints of the seed peers serving the url via the Dragonfly, then downloads the file from a randomly picked endpoint and writes it to stdout.
 Command get-with-endpoints looks up the endpoints of the seed peers serving the url via the Dragonfly, then downloads the file from a randomly picked endpoint and writes it to stdout.
lookup command
Command lookup looks up the endpoints of the seed peers serving the url via the Dragonfly and prints them to stdout.
 Command lookup looks up the endpoints of the seed peers serving the url via the Dragonfly and prints them to stdout.
preheat command
Command preheat preheats a file or an OCI image to the seed peers via the Dragonfly.
 Command preheat preheats a file or an OCI image to the seed peers via the Dragonfly.
replicas command
Command replicas preheats the url to three replicas of seed peers via the Dragonfly, then downloads it with the request scattered across the replicas, writing the content to stdout.
 Command replicas preheats the url to three replicas of seed peers via the Dragonfly, then downloads it with the request scattered across the replicas, writing the content to stdout.
internal
hashring
Package hashring provides a consistent hash ring with virtual nodes that is bit-for-bit compatible with the Rust client's VNodeHashRing (dragonfly-client-util), which wraps the hashring crate with SipHash-2-4 and zero keys.
 Package hashring provides a consistent hash ring with virtual nodes that is bit-for-bit compatible with the Rust client's VNodeHashRing (dragonfly-client-util), which wraps the hashring crate with SipHash-2-4 and zero keys.
pool
Package pool provides a client pool for managing reusable HTTP client instances with automatic cleanup, ported from the Rust client's pool (dragonfly-client-util).
 Package pool provides a client pool for managing reusable HTTP client instances with automatic cleanup, ported from the Rust client's pool (dragonfly-client-util).
selector
Package selector selects seed peers from the scheduler service with a consistent hash ring, ported from the Rust client's selector (dragonfly-client-request).
 Package selector selects seed peers from the scheduler service with a consistent hash ring, ported from the Rust client's selector (dragonfly-client-request).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.