The highest tagged major version is
Documentation
¶
Overview ¶
Package httpx is WaxTap's internal HTTP client wrapper. It adds retry, exponential backoff, rate-limit handling, and an optional per-host limiter on top of an injected *http.Client.
Design notes:
- No global timeout is imposed here. Callers set per-operation deadlines on request contexts, so a long legitimate download is not killed by an unrelated global cap while a hung socket still cannot leak a goroutine.
- Retry-After is honored but capped by MaxRetryWait: YouTube can demand hours on an IP ban, so beyond the cap we fail fast with *waxerr.RateLimitError instead of sleeping a worker.
- Retried requests must be replayable: a request with a body but no GetBody is attempted exactly once.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Sleep ¶
Sleep waits for d or until ctx is done, whichever comes first, returning the context error on cancellation. A non-positive d returns immediately.
func WithThrottleHook ¶
func WithThrottleHook(ctx context.Context, h ThrottleHook) context.Context
WithThrottleHook returns a context that reports throttle events to h. A nil hook clears an inherited hook.
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client performs HTTP requests with retry, backoff, and rate-limit handling.
func (*Client) Do ¶
Do executes req with retry/backoff and rate-limit handling. The request context governs cancellation and per-operation deadlines; a context error is never retried. On a capped-out Retry-After it returns *waxerr.RateLimitError.
On success the caller owns the returned response body. On retried or rate-limited responses the intermediate bodies are drained and closed here.
type Config ¶
type Config struct {
// HTTPClient is the underlying client. It should set a DialContext and a
// conservative Timeout (or rely on per-request context deadlines). If nil,
// http.DefaultClient is used.
HTTPClient *http.Client
// Logger receives debug logs for retries/backoff. If nil, logs are discarded.
Logger *slog.Logger
// Limiter gates requests per host. If nil, no limiting is applied.
Limiter Limiter
// MaxRetries is the number of additional attempts after the first.
MaxRetries int
// BaseBackoff is the base of the exponential backoff schedule.
BaseBackoff time.Duration
// MaxBackoff caps a single backoff sleep.
MaxBackoff time.Duration
// MaxRetryWait caps an honored Retry-After. Beyond it, Do fails fast with
// *waxerr.RateLimitError rather than sleeping.
MaxRetryWait time.Duration
// Cooldown is the minimum host penalty after a rate-limit response. A longer
// Retry-After value takes precedence, up to MaxRetryWait. Zero disables the
// base cooldown.
Cooldown time.Duration
}
Config configures a Client. The zero value is usable; New fills sane defaults.
type HostLimiter ¶
type HostLimiter struct {
// contains filtered or unexported fields
}
HostLimiter spaces requests and applies cooldowns independently per host.
Requests are admitted at most once every 1/qps seconds; bursts are not allowed. HostLimiter.Penalize pauses a host even when qps is non-positive.
func NewHostLimiter ¶
func NewHostLimiter(qps float64) *HostLimiter
NewHostLimiter returns a per-host limiter that admits qps requests per second. A non-positive qps disables spacing but still permits cooldowns.
func (*HostLimiter) Penalize ¶
func (l *HostLimiter) Penalize(host string, d time.Duration)
Penalize pauses requests to host for at least d. It also moves the spacing schedule forward so requests remain spaced after the cooldown. A non-positive duration has no effect.
func (*HostLimiter) Wait ¶
func (l *HostLimiter) Wait(ctx context.Context, host string) error
Wait blocks until a request to host may proceed or ctx is done.
Wait rechecks the host cooldown after each timer wakeup. On cancellation, it rolls back the request's spacing reservation when no later request depends on it.
type Limiter ¶
type Limiter interface {
// Wait blocks until host may receive another request or ctx is canceled.
Wait(ctx context.Context, host string) error
// Penalize pauses requests to host for at least d.
Penalize(host string, d time.Duration)
}
Limiter gates outbound requests. Wait blocks until a request may proceed or ctx is done. Penalize pauses requests to one host for at least d.
type NopLimiter ¶
type NopLimiter struct{}
NopLimiter performs no rate limiting.
type ThrottleEvent ¶
type ThrottleEvent struct {
Host string // request host
StatusCode int // rate-limit response status
RetryAfter time.Duration // parsed Retry-After; zero if missing or invalid
Penalty time.Duration // duration passed to Limiter.Penalize
Phase ThrottlePhase // point in the throttle lifecycle
}
ThrottleEvent describes rate limiting by one host.
type ThrottleHook ¶
type ThrottleHook func(ThrottleEvent)
ThrottleHook receives throttle events. It may be called from parallel request workers, so it must be safe for concurrent use.
type ThrottlePhase ¶
type ThrottlePhase int
ThrottlePhase identifies when a throttle event occurred.
const ( // ThrottleDetected is reported after a rate-limit response is received and // the host is penalized. ThrottleDetected ThrottlePhase = iota // ThrottleRetryStarted is reported when a retry begins after backoff. ThrottleRetryStarted )
Source Files