request

README

GoCryptoTrader package Request

This request package is part of the GoCryptoTrader codebase.

This is still in active development

You can track ideas, planned features and what's in progress on our GoCryptoTrader Kanban board.

Join our slack to discuss all things related to GoCryptoTrader! GoCryptoTrader Slack

Current Features for request

• This package services the exchanges package with request handling.
  • Throttling of requests for an individual exchange

Donations

If this framework helped you in any way, or you would like to support the developers working on it, please donate Bitcoin to:

bc1qk0jareu4jytc0cfrhr5wgshsq8282awpavfahc

Documentation

Index

• Constants
• Variables
• func AbortRateLimitBarrier(ctx context.Context)
• func DefaultRetryPolicy(resp *http.Response, err error) (bool, error)
• func IsVerbose(ctx context.Context, verbose bool) bool
• func NewRateLimit(interval time.Duration, actions int) *rate.Limiter
• func NewRateLimitBarrierContexts(ctx context.Context, participants uint) ([]context.Context, error)
• func RetryAfter(resp *http.Response, now time.Time) time.Duration
• func SetupGlobalReporter(r Reporter)
• func WaitForRateLimitBarrier(ctx context.Context) error
• func WithDelayNotAllowed(ctx context.Context) context.Context
• func WithHeaders(ctx context.Context, headers http.Header) context.Context
• func WithRetryNotAllowed(ctx context.Context) context.Context
• func WithVerbose(ctx context.Context) context.Context
• type AuthType
• type Backoff
  • func DefaultBackoff() Backoff
  • func LinearBackoff(base, maxDuration time.Duration) Backoff
• type EndpointLimit
• type Generate
• type Item
• type RateLimitDefinitions
  • func NewBasicRateLimit(interval time.Duration, actions int, weight Weight) RateLimitDefinitions
• type RateLimiterWithWeight
  • func GetRateLimiterWithWeight(l *rate.Limiter, weight Weight) *RateLimiterWithWeight
  • func NewRateLimitWithWeight(interval time.Duration, actions int, weight Weight) *RateLimiterWithWeight
  • func NewWeightedRateLimitByDuration(interval time.Duration) *RateLimiterWithWeight
  • func (r *RateLimiterWithWeight) RateLimit(ctx context.Context) error
  • func (r *RateLimiterWithWeight) Weight() Weight
• type Reporter
• type Requester
  • func New(name string, httpRequester *http.Client, opts ...RequesterOption) (*Requester, error)
  • func (r *Requester) DisableRateLimiter() error
  • func (r *Requester) EnableRateLimiter() error
  • func (r *Requester) GetHTTPClientUserAgent() (string, error)
  • func (r *Requester) GetNonce(set nonce.Setter) nonce.Value
  • func (r *Requester) GetRateLimiterDefinitions() RateLimitDefinitions
  • func (r *Requester) InitiateRateLimit(ctx context.Context, e EndpointLimit) error
  • func (r *Requester) SendPayload(ctx context.Context, ep EndpointLimit, newRequest Generate, ...) error
  • func (r *Requester) SetHTTPClient(newClient *http.Client) error
  • func (r *Requester) SetHTTPClientTimeout(timeout time.Duration) error
  • func (r *Requester) SetHTTPClientUserAgent(userAgent string) error
  • func (r *Requester) SetProxy(p *url.URL) error
  • func (r *Requester) Shutdown() error
• type RequesterOption
  • func WithBackoff(b Backoff) RequesterOption
  • func WithLimiter(def RateLimitDefinitions) RequesterOption
  • func WithReporter(rep Reporter) RequesterOption
  • func WithRetryPolicy(p RetryPolicy) RequesterOption
• type RetryPolicy
• type Weight

Constants

const (
	// UnsetRequest is an unset request authentication level
	UnsetRequest AuthType = 0
	// UnauthenticatedRequest denotes a request with no credentials
	UnauthenticatedRequest = iota << 1
	// AuthenticatedRequest denotes a request using API credentials
	AuthenticatedRequest
)

const (
	DefaultMaxRetryAttempts = 3
	DefaultMutexLockTimeout = 50 * time.Millisecond
)

Const vars for rate limiter

Variables

var (
	// ErrInvalidRateLimitBarrierParticipants is returned when a barrier cannot coordinate at least two requests.
	ErrInvalidRateLimitBarrierParticipants = errors.New("rate limit barrier requires at least two participants")
	// ErrRateLimitBarrierParticipantUsed is returned when a participant context reaches a limiter again before its barrier resolves.
	ErrRateLimitBarrierParticipantUsed = errors.New("rate limit barrier participant already used")
	// ErrRateLimitBarrierRejected is returned when a peer aborts or is cancelled before group admission.
	ErrRateLimitBarrierRejected = errors.New("rate limit barrier rejected")
)

var (
	ErrRateLimiterAlreadyDisabled = errors.New("rate limiter already disabled")
	ErrRateLimiterAlreadyEnabled  = errors.New("rate limiter already enabled")
	ErrDelayNotAllowed            = errors.New("delay not allowed")
)

Rate limiting errors.

var (
	ErrRequestSystemIsNil = errors.New("request system is nil")
	ErrAuthRequestFailed  = errors.New("authenticated request failed")
	ErrBadStatus          = errors.New("unsuccessful HTTP status code")
)

Public errors

var (
	MaxRetryAttempts = DefaultMaxRetryAttempts
)

Vars for rate limiter

Functions

func AbortRateLimitBarrier

func AbortRateLimitBarrier(ctx context.Context)

AbortRateLimitBarrier rejects a participant that failed before reaching RateLimit. It is a no-op after the participant is consumed.

func DefaultRetryPolicy

func DefaultRetryPolicy(resp *http.Response, err error) (bool, error)

DefaultRetryPolicy determines whether the request should be retried, implemented with a default strategy.

func IsVerbose

func IsVerbose(ctx context.Context, verbose bool) bool

IsVerbose checks main verbosity first then checks context verbose values for specific request verbosity.

func NewRateLimit

func NewRateLimit(interval time.Duration, actions int) *rate.Limiter

NewRateLimit creates a new RateLimit based of time interval and how many actions allowed and breaks it down to an actions-per-second basis -- Burst rate is kept as one as this is not supported for out-bound requests.

func NewRateLimitBarrierContexts

func NewRateLimitBarrierContexts(ctx context.Context, participants uint) ([]context.Context, error)

NewRateLimitBarrierContexts returns distinct contexts whose first rate-limit calls proceed only when every participant can proceed immediately. Each request owner must defer AbortRateLimitBarrier so a failure before reaching the limiter releases the other participants. The final arrival atomically reserves capacity for the group; shared limiters must have enough burst capacity for all participants, and each weight must fit its limiter's burst. After acceptance, subsequent rate-limit calls using a participant context proceed normally, including retries. Coordination covers only the first gate, not request execution: additional rate-limited calls under the same participant context can fail independently and lose the group's all-or-nothing admission guarantee. Cancellation or transport errors after admission can also split execution. NewRateLimit uses burst 1, so participants using its finite limiters must use distinct limiter instances and weight 1. Larger groups sharing a limiter or larger weights require sufficient burst capacity, for example via GetRateLimiterWithWeight with a custom limiter.

func RetryAfter

func RetryAfter(resp *http.Response, now time.Time) time.Duration

RetryAfter parses the Retry-After header in the response to determine the minimum duration needed to wait before retrying.

func SetupGlobalReporter

func SetupGlobalReporter(r Reporter)

SetupGlobalReporter sets a reporter interface to be used for all exchange requests

func WaitForRateLimitBarrier

func WaitForRateLimitBarrier(ctx context.Context) error

WaitForRateLimitBarrier marks a request without an active limiter as immediately available and waits for its peers.

func WithDelayNotAllowed

func WithDelayNotAllowed(ctx context.Context) context.Context

WithDelayNotAllowed adds a value to the context that indicates that no delay is allowed for rate limiting.

func WithHeaders

func WithHeaders(ctx context.Context, headers http.Header) context.Context

WithHeaders adds outbound HTTP header overrides to the context. These values replace matching generated headers, including authentication headers.

func WithRetryNotAllowed

func WithRetryNotAllowed(ctx context.Context) context.Context

WithRetryNotAllowed adds a value to the context that indicates that no retries are allowed for requests.

func WithVerbose

func WithVerbose(ctx context.Context) context.Context

WithVerbose adds verbosity to a request context so that specific requests can have distinct verbosity without impacting all requests.

Types

type AuthType

type AuthType uint8

AuthType helps distinguish the purpose of a HTTP request

type Backoff

type Backoff func(n int) time.Duration

Backoff determines how long to wait between request attempts.

func DefaultBackoff

func DefaultBackoff() Backoff

DefaultBackoff is a default strategy for backoff after a retryable request failure.

func LinearBackoff

func LinearBackoff(base, maxDuration time.Duration) Backoff

LinearBackoff applies a backoff increasing by a base amount with each retry capped at a maximum duration.

type EndpointLimit

type EndpointLimit uint16

EndpointLimit defines individual endpoint rate limits.

const (
	Unset EndpointLimit = iota
	Auth
	UnAuth
)

Const here define individual functionality sub types for rate limiting.

type Generate

type Generate func() (*Item, error)

Generate defines a closure for functionality outside the requester to generate a new *http.Request on every attempt. This minimises the chance of being outside the receive window if application rate limiting reduces outbound requests.

type Item

type Item struct {
	Method                 string
	Path                   string
	Headers                map[string]string
	Body                   io.Reader
	Result                 any
	NonceEnabled           bool
	Verbose                bool
	HTTPDebugging          bool
	HTTPRecording          bool
	HTTPMockDataSliceLimit int // Limits slices per HTTP record to reduce mock data size.
	IsReserved             bool
	// HeaderResponse for inspection of header contents package side useful for
	// pagination
	HeaderResponse *http.Header
}

Item is a temp item for requests

type RateLimitDefinitions

type RateLimitDefinitions map[any]*RateLimiterWithWeight

RateLimitDefinitions is a map of endpoint limits to rate limiters.

func NewBasicRateLimit

func NewBasicRateLimit(interval time.Duration, actions int, weight Weight) RateLimitDefinitions

NewBasicRateLimit returns an object that implements the limiter interface for basic rate limit.

type RateLimiterWithWeight

type RateLimiterWithWeight struct {
	// contains filtered or unexported fields
}

RateLimiterWithWeight is a rate limiter coupled with a weight which refers to the number or weighting of the request. This is used to define the rate limit for a specific endpoint.

var RateLimitNotRequired *RateLimiterWithWeight

RateLimitNotRequired is a no-op rate limiter.

func GetRateLimiterWithWeight

func GetRateLimiterWithWeight(l *rate.Limiter, weight Weight) *RateLimiterWithWeight

GetRateLimiterWithWeight couples a rate limiter with a weight count into an accepted defined rate limiter with weight struct.

func NewRateLimitWithWeight

func NewRateLimitWithWeight(interval time.Duration, actions int, weight Weight) *RateLimiterWithWeight

NewRateLimitWithWeight creates a new RateLimit based of time interval and how many actions allowed. This also has a weight count which refers to the number or weighting of the request. This is used to define the rate limit for a specific endpoint.

func NewWeightedRateLimitByDuration

func NewWeightedRateLimitByDuration(interval time.Duration) *RateLimiterWithWeight

NewWeightedRateLimitByDuration creates a new RateLimit based of time interval. This equates to 1 action per interval. The weight is set to 1.

func (*RateLimiterWithWeight) RateLimit

func (r *RateLimiterWithWeight) RateLimit(ctx context.Context) error

RateLimit throttles a request based on weight, delaying the request. Errors if no delay is permitted via the context and a delay is required.

func (*RateLimiterWithWeight) Weight

func (r *RateLimiterWithWeight) Weight() Weight

Weight returns the number of reservations consumed by each request.

type Reporter

type Reporter interface {
	Latency(name, method, path string, t time.Duration)
}

Reporter interface groups observability functionality over HTTP request latency.

type Requester

type Requester struct {
	Nonce nonce.Nonce
	// contains filtered or unexported fields
}

Requester struct for the request client

func New

func New(name string, httpRequester *http.Client, opts ...RequesterOption) (*Requester, error)

New returns a new Requester

func (*Requester) DisableRateLimiter

func (r *Requester) DisableRateLimiter() error

DisableRateLimiter disables the rate limiting system for the exchange.

func (*Requester) EnableRateLimiter

func (r *Requester) EnableRateLimiter() error

EnableRateLimiter enables the rate limiting system for the exchange.

func (*Requester) GetHTTPClientUserAgent

func (r *Requester) GetHTTPClientUserAgent() (string, error)

GetHTTPClientUserAgent gets the exchanges HTTP user agent

func (*Requester) GetNonce

func (r *Requester) GetNonce(set nonce.Setter) nonce.Value

GetNonce returns a nonce for requests. This locks and enforces concurrent nonce FIFO on the buffered job channel

func (*Requester) GetRateLimiterDefinitions

func (r *Requester) GetRateLimiterDefinitions() RateLimitDefinitions

GetRateLimiterDefinitions returns the rate limiter definitions for the requester.

func (*Requester) InitiateRateLimit

func (r *Requester) InitiateRateLimit(ctx context.Context, e EndpointLimit) error

InitiateRateLimit sleeps for designated end point rate limits.

func (*Requester) SendPayload

func (r *Requester) SendPayload(ctx context.Context, ep EndpointLimit, newRequest Generate, requestType AuthType) error

SendPayload handles sending HTTP/HTTPS requests

func (*Requester) SetHTTPClient

func (r *Requester) SetHTTPClient(newClient *http.Client) error

SetHTTPClient sets exchanges HTTP client

func (*Requester) SetHTTPClientTimeout

func (r *Requester) SetHTTPClientTimeout(timeout time.Duration) error

SetHTTPClientTimeout sets the timeout value for the exchanges HTTP Client and also the underlying transports idle connection timeout

func (*Requester) SetHTTPClientUserAgent

func (r *Requester) SetHTTPClientUserAgent(userAgent string) error

SetHTTPClientUserAgent sets the exchanges HTTP user agent

func (*Requester) SetProxy

func (r *Requester) SetProxy(p *url.URL) error

SetProxy sets a proxy address for the client transport

func (*Requester) Shutdown

func (r *Requester) Shutdown() error

Shutdown releases persistent memory for garbage collection.

type RequesterOption

type RequesterOption func(*Requester)

RequesterOption is a function option that can be applied to configure a Requester when creating it.

func WithBackoff

func WithBackoff(b Backoff) RequesterOption

WithBackoff configures the backoff strategy for a Requester.

func WithLimiter

func WithLimiter(def RateLimitDefinitions) RequesterOption

WithLimiter configures the rate limiter for a Requester.

func WithReporter

func WithReporter(rep Reporter) RequesterOption

WithReporter configures the reporter for a Requester.

func WithRetryPolicy

func WithRetryPolicy(p RetryPolicy) RequesterOption

WithRetryPolicy configures the retry policy for a Requester.

type RetryPolicy

type RetryPolicy func(resp *http.Response, err error) (bool, error)

RetryPolicy determines whether the request should be retried.

type Weight

type Weight uint8

Weight defines the number of reservations to be used. This is a generalised weight for rate limiting. e.g. n weight = n request. i.e. 50 Weight = 50 requests.