preset

Documentation

Overview

Package preset provides pre-configured HTTP client builders with common middleware combinations.

Index

• func GetAuthHTTPClient(token string) *http.Client
• func GetAuthHTTPClientWithProxy(proxyURL, token string) *http.Client
• func GetAuthHTTPClientWithXdailiProxy(proxyOrder, proxySecret, token string) *http.Client
• func GetHTTPClientForDebug(log *zap.Logger) *http.Client
• func GetHTTPClientWithProxy(proxyURL string) *http.Client
• func GetHTTPClientWithXdailiProxy(proxyOrder, proxySecret string) *http.Client
• func NewHTTPClient(opts ...ClientOption) *http.Client
• func NewHonuClient(proxyURL, token string, qps float64, opts ...ClientOption) *http.Client
• func NewHonuClientWithDebug(proxyURL, token string, qps float64, logger *zap.Logger, opts ...ClientOption) *http.Client
• type ClientOption
  • func WithAuth(token string) ClientOption
  • func WithBaseTransport(transport http.RoundTripper) ClientOption
  • func WithContextAuth() ClientOption
  • func WithDebug(logger *zap.Logger, config mw.DebugConfig) ClientOption
  • func WithDebugSimple(logger *zap.Logger) ClientOption
  • func WithDecorator(name string, d decorator.Decorator) ClientOption
  • func WithProxy(proxyURL string) ClientOption
  • func WithProxyAndTLSVerify(proxyURL string) ClientOption
  • func WithRateLimit(requestsPerSecond float64, burst int) ClientOption
  • func WithRateLimiterManager(manager *mw.RateLimiterManager) ClientOption
  • func WithRetry(count uint, timeout time.Duration) ClientOption
  • func WithRetryOnBadProxy(count uint) ClientOption
  • func WithUserAgent() ClientOption

Functions

func GetAuthHTTPClient

func GetAuthHTTPClient(token string) *http.Client

GetAuthHTTPClient returns a HTTP client with TokenAuth + UserAgent middlewares.

func GetAuthHTTPClientWithProxy

func GetAuthHTTPClientWithProxy(proxyURL, token string) *http.Client

GetAuthHTTPClientWithProxy returns a HTTP client with TokenAuth + UserAgent + Proxy middlewares.

func GetAuthHTTPClientWithXdailiProxy

func GetAuthHTTPClientWithXdailiProxy(proxyOrder, proxySecret, token string) *http.Client

GetAuthHTTPClientWithXdailiProxy returns a HTTP client with TokenAuth + UserAgent + XdailiProxy middlewares.

func GetHTTPClientForDebug

func GetHTTPClientForDebug(log *zap.Logger) *http.Client

GetHTTPClientForDebug returns a HTTP client configured for comprehensive debugging. It provides detailed logging of both the initial request and the final request state after all middleware modifications.

The client includes:

• DebugInit: Initializes logging context and logs request start
• RetryOnTimeout: Retries on timeout errors (5 attempts, 60s timeout)
• UserAgent: Adds randomized user agent
• DumpRequest/DumpResponse: Saves requests/responses to log/dump directory
• DebugFinal: Logs final request state with all headers before sending

This setup provides complete visibility into:

• Initial request state
• All middleware modifications (headers added by UserAgent, retry attempts, etc.)
• Final request state just before sending
• Response details and timing

On first request, the decorator chain is logged for easy debugging.

Example:

logger, _ := zap.NewDevelopment()
client := preset.GetHTTPClientForDebug(logger)
resp, err := client.Get("https://example.com")

func GetHTTPClientWithProxy

func GetHTTPClientWithProxy(proxyURL string) *http.Client

GetHTTPClientWithProxy returns a HTTP client with UserAgent + Proxy middlewares.

func GetHTTPClientWithXdailiProxy

func GetHTTPClientWithXdailiProxy(proxyOrder, proxySecret string) *http.Client

GetHTTPClientWithXdailiProxy returns a HTTP client with UserAgent + XdailiProxy middlewares.

func NewHTTPClient

func NewHTTPClient(opts ...ClientOption) *http.Client

NewHTTPClient creates a new HTTP client with the given options. This is the main entry point for building customized HTTP clients.

Example:

client := preset.NewHTTPClient(
    preset.WithProxy("http://proxy:8080"),
    preset.WithRateLimit(10, 20),
    preset.WithAuth("token123"),
    preset.WithUserAgent(),
)

The middleware is applied in the order specified, so order matters:

• DebugInit middleware (if enabled) is always applied first (HEAD)
• Rate limiter (if enabled)
• Retry middleware (if enabled)
• Authentication (if enabled)
• User agent (if enabled)
• Custom decorators (if any)
• DebugFinal middleware (if enabled) is always applied last (TAIL)

func NewHonuClient

func NewHonuClient(proxyURL, token string, qps float64, opts ...ClientOption) *http.Client

NewHonuClient creates a specialized HTTP client designed for reliable scraping or API interaction through a proxy.

The name "Honu" (Hawaiian for "sea turtle") represents the philosophy behind this preset: slow, steady, and wise navigation through the web. In Hawaiian culture, the honu is a symbol of good luck, endurance, and long life.

It is pre-configured for:

• Proxy usage (with authentication support in the URL)
• Low-rate requests (burst = 1) to avoid detection
• Robust retries (5 attempts) with relaxed timeouts (60s) for slower proxy connections
• Consistent User-Agent generation (when used with middleware.UserAgentRandomSeed)
• Retry on bad proxy connections (5 attempts)

Parameters:

• proxyURL: The full proxy URL (e.g., "http://user:pass@proxy.com:8080")
• token: The authentication token (Bearer token)
• qps: Requests per second (usually low, e.g., 0.1 or 1.0)

Usage:

client := preset.NewHonuClient("http://user:pass@proxy:8080", "my-token", 0.5)

func NewHonuClientWithDebug

func NewHonuClientWithDebug(proxyURL, token string, qps float64, logger *zap.Logger, opts ...ClientOption) *http.Client

NewHonuClientWithDebug is the debug-enabled version of NewHonuClient. It allows passing a logger and optional extra options.

The middleware order keeps Debug outermost, then rate limiting, then retries. This matches the standard NewHTTPClient chain and ensures RequestStats are available to downstream middleware (including ZapVerboseLogger when added).

Default debug configuration is used unless overridden by passing WithDebug() in opts.

Types

type ClientOption

type ClientOption func(*clientConfig)

ClientOption is a functional option for configuring an HTTP client.

func WithAuth

func WithAuth(token string) ClientOption

WithAuth configures token-based authentication.

Example:

client := preset.NewHTTPClient(preset.WithAuth("Bearer token123"))

func WithBaseTransport

func WithBaseTransport(transport http.RoundTripper) ClientOption

WithBaseTransport sets a custom base transport. This is useful if you need to customize the underlying HTTP transport.

Example:

transport := &http.Transport{
    MaxIdleConns: 100,
    IdleConnTimeout: 90 * time.Second,
}
client := preset.NewHTTPClient(preset.WithBaseTransport(transport))

func WithContextAuth

func WithContextAuth() ClientOption

WithContextAuth enables token-based authentication that is resolved per request via context. Use middleware.WithToken to attach the token to the request context.

Example:

client := preset.NewHTTPClient(preset.WithContextAuth())
ctx := middleware.WithToken(context.Background(), "token-from-vault")
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)

func WithDebug

func WithDebug(logger *zap.Logger, config mw.DebugConfig) ClientOption

WithDebug enables debug logging with the specified logger and configuration. This adds DebugInit middleware at the head (outermost) and DebugFinal middleware at the tail (innermost) of the chain for complete request/response visibility.

DebugInit initializes the logging context and logs the initial request state. DebugFinal logs the final request state after all middleware modifications.

Example:

logger, _ := zap.NewDevelopment()
client := preset.NewHTTPClient(
    preset.WithDebug(logger, middleware.DefaultDebugConfig()),
    preset.WithRateLimit(10, 20),
)

func WithDebugSimple

func WithDebugSimple(logger *zap.Logger) ClientOption

WithDebugSimple enables debug logging with default configuration. This is a convenience function for quick debugging during development. Like WithDebug, it adds both DebugInit (HEAD) and DebugFinal (TAIL) middleware.

Example:

logger, _ := zap.NewDevelopment()
client := preset.NewHTTPClient(preset.WithDebugSimple(logger))

func WithDecorator

func WithDecorator(name string, d decorator.Decorator) ClientOption

WithDecorator adds a custom decorator to the middleware chain.

Parameters:

• name: Display name for the decorator (shown in chain visualization)
• d: The decorator function

Custom decorators are placed after built-in middleware. This allows your custom middleware to see the request after built-in middleware (like auth and retry) have processed it.

Execution order:

DebugInit -> RateLimit -> Retry -> Auth -> UserAgent -> Your Custom Decorator -> DebugFinal -> Transport

Example:

myMiddleware := func(rt http.RoundTripper) http.RoundTripper {
    return decorator.TransportFunc(func(req *http.Request) (*http.Response, error) {
        log.Println("Request after built-in middleware:", req.URL)
        return rt.RoundTrip(req)
    })
}
client := preset.NewHTTPClient(preset.WithDecorator("MyLogger", myMiddleware))

func WithProxy

func WithProxy(proxyURL string) ClientOption

WithProxy configures the client to use a proxy. By default, TLS verification is disabled for proxy connections.

Example:

client := preset.NewHTTPClient(preset.WithProxy("http://proxy.example.com:8080"))

func WithProxyAndTLSVerify

func WithProxyAndTLSVerify(proxyURL string) ClientOption

WithProxyAndTLSVerify configures the client to use a proxy with TLS verification enabled.

Example:

client := preset.NewHTTPClient(preset.WithProxyAndTLSVerify("https://proxy:8080"))

func WithRateLimit

func WithRateLimit(requestsPerSecond float64, burst int) ClientOption

WithRateLimit configures rate limiting. You must set the rate limiter key in the request context to enable rate limiting.

Parameters:

• requestsPerSecond: maximum requests per second (e.g., 10.0 for 10 req/s, 0.5 for 1 req per 2s)
• burst: maximum burst size (number of tokens that can be consumed at once)

Example:

client := preset.NewHTTPClient(preset.WithRateLimit(10, 20))
// In your request code:
ctx := context.WithValue(ctx, middleware.RateLimiterKey, "user123")
req := req.WithContext(ctx)

func WithRateLimiterManager

func WithRateLimiterManager(manager *mw.RateLimiterManager) ClientOption

WithRateLimiterManager sets a shared RateLimiterManager. This is useful when you want to share rate limits across multiple HTTP clients. If this is set, WithRateLimit settings (RPS/burst) are ignored in favor of the manager's config.

func WithRetry

func WithRetry(count uint, timeout time.Duration) ClientOption

WithRetry configures retry behavior for timeout errors.

Parameters:

• count: number of retry attempts
• timeout: timeout duration for each attempt

Example:

client := preset.NewHTTPClient(preset.WithRetry(5, 60*time.Second))

func WithRetryOnBadProxy

func WithRetryOnBadProxy(count uint) ClientOption

WithRetryOnBadProxy configures retry behavior for proxy errors. This is automatically enabled when using WithProxy, with a default retry count.

Example:

client := preset.NewHTTPClient(
    preset.WithProxy("http://proxy:8080"),
    preset.WithRetryOnBadProxy(6),
)

func WithUserAgent

func WithUserAgent() ClientOption

WithUserAgent enables the user agent middleware. This adds a randomized user agent to requests.

Example:

client := preset.NewHTTPClient(preset.WithUserAgent())