http

Documentation

Index

• func NormalizeURL(url string) string
• func SplitLines(data []byte) []string
• type Config
  • func (c Config) NewClient(timeout time.Duration) *HTTPClientManager
• type HTTPClientManager
  • func NewHTTPClient(timeout time.Duration, headers []string, proxyURL string, rps int, ...) *HTTPClientManager
  • func NewHTTPClientFromExternal(externalClient *http.Client, headers []string, rps int) *HTTPClientManager
  • func (h *HTTPClientManager) EnableKeepAlives(maxConnsPerHost int)
  • func (h *HTTPClientManager) Get(url string) (string, error)
  • func (h *HTTPClientManager) GetPartialWithContext(ctx context.Context, url string, maxBytes int) (string, error)
  • func (h *HTTPClientManager) GetStatusAndBody(ctx context.Context, url string) (int, string, error)
  • func (h *HTTPClientManager) GetWithContext(ctx context.Context, url string) (string, error)
  • func (h *HTTPClientManager) HeadWithContext(ctx context.Context, url string) (int, error)
• type RateLimiter
  • func NewRateLimiter(rps int) *RateLimiter
  • func (rl *RateLimiter) Wait()

Functions

func NormalizeURL

func NormalizeURL(url string) string

func SplitLines

func SplitLines(data []byte) []string

Types

type Config

type Config struct {
	Headers        []string
	Proxy          string
	RateLimit      int             // Requests per second (0 = unlimited)
	MaxRedirects   int             // Maximum redirects to follow (0 = disable, -1 = default: 10)
	ExternalClient *stdhttp.Client // External HTTP client (optional, for connection pooling)
	SharedLimiter  *RateLimiter    // Shared rate limiter across all clients (set once, reused)
}

Config contains HTTP-related configuration for making requests.

func (Config) NewClient

func (c Config) NewClient(timeout time.Duration) *HTTPClientManager

NewClient creates an HTTPClientManager from this config. If SharedLimiter is set, all clients share that limiter. Otherwise a new per-client limiter is created from RateLimit.

type HTTPClientManager

type HTTPClientManager struct {
	// contains filtered or unexported fields
}

func NewHTTPClient

func NewHTTPClient(timeout time.Duration, headers []string, proxyURL string, rps int, maxRedirects int) *HTTPClientManager

func NewHTTPClientFromExternal

func NewHTTPClientFromExternal(externalClient *http.Client, headers []string, rps int) *HTTPClientManager

NewHTTPClientFromExternal wraps an external http.Client (e.g., from a connection pool). This allows reusing an existing client instead of creating a new one. The external client is used as-is; headers and rate limiting are still applied.

func (*HTTPClientManager) EnableKeepAlives

func (h *HTTPClientManager) EnableKeepAlives(maxConnsPerHost int)

EnableKeepAlives enables HTTP connection reuse. Use this when making many requests to the same host (e.g. bruteforce scanning).

func (*HTTPClientManager) Get

func (h *HTTPClientManager) Get(url string) (string, error)

func (*HTTPClientManager) GetPartialWithContext

func (h *HTTPClientManager) GetPartialWithContext(ctx context.Context, url string, maxBytes int) (string, error)

GetPartialWithContext sends a GET request and reads at most maxBytes of the response body. Useful for fetching just the header portion of large files (e.g. CSS theme version).

func (*HTTPClientManager) GetStatusAndBody

func (h *HTTPClientManager) GetStatusAndBody(ctx context.Context, url string) (int, string, error)

GetStatusAndBody sends a GET request and returns both the status code and body. Unlike GetWithContext, it does not treat non-2xx responses as errors.

func (*HTTPClientManager) GetWithContext

func (h *HTTPClientManager) GetWithContext(ctx context.Context, url string) (string, error)

func (*HTTPClientManager) HeadWithContext

func (h *HTTPClientManager) HeadWithContext(ctx context.Context, url string) (int, error)

HeadWithContext sends a HEAD request and returns the HTTP status code. No body is read. Useful for fast existence checks (e.g. 403 vs 404).

type RateLimiter

type RateLimiter struct {
	// contains filtered or unexported fields
}

RateLimiter implements a token bucket rate limiter.

func NewRateLimiter

func NewRateLimiter(rps int) *RateLimiter

NewRateLimiter creates a new rate limiter with the specified requests per second. If rps is 0 or negative, returns nil (no rate limiting).

func (*RateLimiter) Wait

func (rl *RateLimiter) Wait()

Wait blocks until a token is available. If the rate limiter is nil (disabled), returns immediately.