proxator

package module
v0.0.0-...-6541d63 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 24, 2026 License: MIT Imports: 22 Imported by: 0

README

go-proxator

Go Reference CI

go-proxator is a proxy-pool client for Go with performance-weighted routing, per-domain block memory, retries, and automatic endpoint recovery. It uses Go's standard HTTP transport by default and also supports an opt-in browser-like TLS transport and custom session implementations.

Features

  • Routes more traffic through fast, reliable endpoints without starving the rest.
  • Applies an immediate weight penalty only for the target domain that blocked an endpoint.
  • Reuses transport sessions and supports per-endpoint rate limits.
  • Escalates cooldown after repeated failures and recovers endpoints automatically.
  • Exposes pool statistics, structured logging, and manual endpoint controls.

Requirements and installation

The module requires Go 1.24 or newer. From an existing Go module, run:

go get github.com/cpouldev/go-proxator

Quick start

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	"github.com/cpouldev/go-proxator"
)

func main() {
	client, err := proxator.New(proxator.Config{
		Pools: []proxator.PoolConfig{{
			Name:            "main",
			Endpoints: []proxator.EndpointConfig{{
				Type:     "http",
				Username: "user",
				Password: "pass",
				Host:     "gate.example.net",
				Port:     8000,
			}},
			SessionPoolSize: 1,
		}},
	})
	if err != nil {
		log.Fatal(err)
	}
	defer client.Close()

	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()

	resp, err := client.Get(ctx, "main", "https://example.com")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(resp.Status, len(resp.Body))
}

Replace the sample proxy URL with your provider's endpoint. To try it without writing a file, run the maintained example against a real proxy; TARGET_URL is optional and defaults to https://example.com:

PROXY_TYPE=http PROXY_HOST=proxy.example PROXY_PORT=8080 \
  PROXY_USERNAME=user PROXY_PASSWORD=pass go run ./examples/basic

See examples/basic/main.go for the complete setup, timeout, request, and cleanup flow.

Configure endpoints

Every PoolConfig needs a unique, non-empty Name and at least one EndpointConfig in Endpoints. Each entry specifies the proxy Type, credentials, Host, and numeric Port.

proxator does not transform usernames or generate provider-specific sticky-session URLs. Build each endpoint's credentials in your own configuration layer, where provider conventions belong, then pass the explicit endpoint records to the pool.

Proxy scheme support depends on the selected transport. Do not commit real proxy credentials; use environment variables or a secret manager.

Use one or multiple pools

Pool names can represent providers, regions, or proxy tiers. Pass the configured name to Get, Post, Do, and other request methods. When exactly one pool is configured, those methods may receive "" and resolve that pool automatically; PoolConfig.Name itself is still required.

The ExampleClient_multiplePools example routes requests through separate eu and us pools.

Choose a transport

Configuration Proxy schemes Use when
SessionFactory: nil HTTP, HTTPS Standard net/http behavior is sufficient.
SessionFactory: proxator.AzureTLSFactory{} HTTP, HTTPS, SOCKS5 A target requires a Chrome-like TLS fingerprint.
Custom SessionFactory and Session Implementation-defined You need another protocol, client, or test double.

The client creates SessionPoolSize sessions for each endpoint during New and closes owned sessions in Client.Close. Always defer client.Close() after successful construction.

Routing, retries, and blocking

Endpoint selection combines success rate and average latency, with a warmup period for new endpoints. Blocking responses and recognized proxy errors lower only that endpoint's weight for the target domain. Repeated failures move the endpoint into an escalating cooldown; a later success resets the escalation.

RetryConfig.MaxAttempts defaults to 3 total attempts, including the initial request. The pool selects an endpoint again before each retry. Retries have no separate elapsed-time budget, so bound the whole operation with context.WithTimeout or a deadline. Cancellation stops further attempts; custom RequestFunc callbacks must pass that context to Session.Do, and custom sessions must honor it.

Common pool settings

Setting Default Behavior
SessionPoolSize 15 Concurrent reusable sessions per endpoint; also the rate-limit burst size.
RequestsPerSecond 0 Sustained per-endpoint limit; zero disables limiting.
DomainBlockTTL 5m How long a target-domain penalty remains active.
DomainBlockPenalty 0.1 Weight multiplier for a recently blocked endpoint/domain pair.
HealthCheckTimeout 15s Timeout for one enabled health check.

Health checks stay disabled unless HealthCheckInterval is positive. When enabled, HealthCheckURL is required; choose an endpoint you control or are comfortable polling.

Send requests

Use Get, GetWithHeaders, Post, or PostWithHeaders for common requests; these methods derive the target domain from the URL. Use DoForDomain with a RequestFunc when a custom request should retain domain-specific penalties, or Do when no target domain is available. The complete ExampleClient_Do shows a custom POST with ordered headers.

The built-in transports send string, []byte, and io.Reader bodies as raw data and JSON-encode other supported values. Recreate one-shot readers inside the callback when a request can be retried.

When a blocking response exhausts the configured attempts, these methods return both that last response and a non-nil error, so inspect both values:

resp, err := client.Get(ctx, "main", targetURL)
if err != nil {
	if resp != nil {
		log.Printf("giving up on %s, last status %s", targetURL, resp.Status)
	}
	return err
}

Observe and control pools

Client.Stats() reports alive, cooldown, and dead counts plus per-endpoint latency, success rate, session availability, and cooldown tier. Operational events use Config.Logger, which defaults to slog.Default().

Use MarkEndpointDead(pool, index) to retire an endpoint and ResetEndpoint(pool, index) to restore it. Endpoint indexes are available in Stats; see ExampleClient_Stats.

Repository layout

.
├── client.go, pool.go, endpoint.go   # routing, pooling, and endpoint lifecycle
├── config.go, detector.go, transient.go
├── transport*.go                     # session contracts and built-in adapters
├── doc.go, example_test.go           # package docs and runnable examples
├── examples/basic/                   # executable getting-started example
├── internal/domain/                  # per-domain block tracking and tests
└── .github/workflows/ci.yml          # build, test, vet, format, and lint checks

The exported API lives in the root package so that every type, field, and method is documented on pkg.go.dev. Only genuinely private state lives under internal/.

Development

go mod verify
go build ./...
go test ./...
go test -race -coverprofile=coverage.txt ./...
go vet ./...
test -z "$(gofmt -l .)"
golangci-lint run

CI runs dependency verification, formatting, vet, and race-enabled tests with coverage on Go 1.24 and 1.25. A separate job runs golangci-lint.

Status

The API may change before v1. See the Go Reference for exported types and methods.

License

MIT

Documentation

Overview

Package proxator routes HTTP requests through pools of rotating proxies, scoring each endpoint on live performance and taking failing ones out of rotation automatically.

By default it uses Go's standard HTTP transport with normal TLS certificate verification. Callers that need a Chrome-like TLS fingerprint can select AzureTLSFactory, or provide their own SessionFactory.

Selection

Endpoints are chosen by weighted random selection rather than round-robin. An endpoint's weight is its success rate divided by its average latency, so faster and more reliable endpoints receive proportionally more traffic. For the first few requests the weight blends toward a neutral baseline, which stops a newly added endpoint from either monopolising traffic or being written off because its first sample happened to be slow.

Domain-aware blocking

Blocks are recorded per endpoint *and* per target domain. If one host starts serving challenge pages to endpoint 3, endpoint 3 is deprioritised for that host while keeping full weight everywhere else — a per-endpoint circuit breaker would have removed it from rotation completely. PoolConfig controls the penalty multiplier and expiry through DomainBlockPenalty and DomainBlockTTL.

Cooldown

After a configurable number of consecutive failures an endpoint enters cooldown. Each consecutive cooldown doubles the previous duration, capped at 16x the base period and at RetryConfig.MaxCooldown. A single success resets the escalation.

Transports

Config.SessionFactory defaults to HTTPFactory, which supports HTTP and HTTPS proxy URLs, uses normal certificate verification, and does not guarantee header field order on the wire. AzureTLSFactory supports HTTP and HTTPS proxies and is tested with SOCKS5 proxies. Proxy scheme support is otherwise factory-specific.

Client construction eagerly creates SessionPoolSize sessions for each endpoint. The Client owns and closes every session returned successfully by the factory. Custom Session implementations must honor the context passed to Do, fully buffer response bodies, and return a non-nil Response with a nil error.

Retries and cancellation

RetryConfig.MaxAttempts includes the initial request. Every callback error is generally eligible for retry, but only errors and responses classified as blocked affect domain penalties and endpoint cooldown. Each attempt selects an endpoint again, though weighted selection can choose the same endpoint.

Retries have no separate elapsed-time budget. Use context.WithTimeout to bound the operation, and pass that context to Session.Do from a RequestFunc. If attempts are exhausted after a blocking response, Client.Do can return both the Response and a non-nil error.

Usage 

client, err := proxator.New(proxator.Config{
    Pools: []proxator.PoolConfig{{
        Name:              "main",
        Endpoints: []proxator.EndpointConfig{
            {Type: "http", Username: "user-1", Password: "pass", Host: "gate.example.net", Port: 8000},
            {Type: "http", Username: "user-2", Password: "pass", Host: "gate.example.net", Port: 8000},
        },
        RequestsPerSecond: 5,
    }},
})
if err != nil {
    return err
}
defer client.Close()

resp, err := client.Get(ctx, "main", "https://example.com")

With exactly one pool configured the name may be omitted: 

resp, err := client.Get(ctx, "", "https://example.com")

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	// ErrAllProxiesBlocked is returned when every endpoint in the selected
	// pool is dead or in cooldown. It is not retried.
	ErrAllProxiesBlocked = errors.New("proxator: all proxies are blocked or unavailable")

	// ErrPoolNotFound is returned when the named pool does not exist.
	ErrPoolNotFound = errors.New("proxator: pool not found")

	// ErrNoPools is returned by New when Config contains no pools.
	ErrNoPools = errors.New("proxator: no pools configured")

	// ErrAmbiguousPool is returned when a pool name is omitted but more than
	// one pool is configured.
	ErrAmbiguousPool = errors.New("proxator: pool name required when multiple pools are configured")

	// ErrClientClosed is returned when work is attempted after the endpoint
	// sessions have been closed.
	ErrClientClosed = errors.New("proxator: client is closed")
)

Sentinel errors returned by the package.

View Source 
var DefaultAmbiguousPatterns = []string{
	"failed to dial",
	"connection refused",
	"connection reset",
	"i/o timeout",
	"tls handshake timeout",
	"no such host",
	"502 bad gateway",
	"503 service unavailable",
	"429 too many requests",
}

DefaultAmbiguousPatterns are generic network failures. Any TCP connection in the process can produce these — your database, your cache, your queue — so they are only transient when the error also carries proxy context.

View Source 
var DefaultBlockedStatusCodes = []int{
	401,
	403,
	407,
	429,
	503,
}

DefaultBlockedStatusCodes are HTTP status codes treated as definitive blocks without inspecting the response body.

View Source 
var DefaultBlockingPatterns = []string{
	"just a moment",
	"cloudflare",
	"captcha",
	"status 403",
	"status 407",
	"status 429",
	"status 502",
	"status 503",
}

DefaultBlockingPatterns indicate target-site blocking. Like the ambiguous set, they require proxy context: a bare "status 403" is just as likely to be an expired API key as a bot wall.

View Source 
var DefaultBodyPatterns = []string{
	`(?i)cloudflare`,
	`(?i)captcha`,
	`(?i)access.denied`,
	`(?i)forbidden`,
	`(?i)rate.limit`,
	`(?i)too.many.requests`,
	`(?i)please.verify`,
	`(?i)security.check`,
	`(?i)bot.detection`,
	`(?i)suspicious.activity`,
	`(?i)cf-ray`,
	`(?i)__cf_chl`,
	`(?i)hcaptcha`,
	`(?i)recaptcha`,
	`(?i)turnstile`,
	`(?i)just.a.moment`,
	`(?i)checking.your.browser`,
	`(?i)ddos.protection`,
	`(?i)attention.required`,
}

DefaultBodyPatterns are regular expressions matched against the body of non-2xx responses to spot interstitial challenge and bot-wall pages.

View Source 
var DefaultContextPatterns = []string{
	"proxy",
	"proxator",
}

DefaultContextPatterns confirm that an error came from proxied traffic.

Keep these tight. Every string here widens what the ambiguous and blocking tiers will swallow, and the whole point of the tiering is to avoid misclassifying a database outage as a proxy hiccup. Add the names of your own scraping call sites here if their errors do not already mention a proxy.

View Source 
var DefaultErrorPatterns = []string{
	"403",
	"forbidden",
	"blocked",
	"captcha",
	"challenge",
	"rate limit",
	"too many requests",
}

DefaultErrorPatterns are lowercase substrings matched against transport error strings to spot blocking that surfaced as an error rather than a response.

View Source 
var DefaultUnambiguousPatterns = []string{
	"proxy tunnel failed",
	"all proxies are blocked",
	"blocking response",
}

DefaultUnambiguousPatterns are error substrings that can only originate from proxy infrastructure. They are classified as transient on sight.

Functions

func IsTransient 

func IsTransient(err error) bool

IsTransient reports whether err is a transient external failure, using the Default pattern sets as they are configured at call time. Build a TransientClassifier once with NewTransientClassifier when classifying errors in bulk, or NewTransientClassifierWith to tune the pattern sets.

Example 
package main

import (
	"errors"
	"fmt"

	proxator "github.com/cpouldev/go-proxator"
)

func main() {
	err := errors.New("proxy tunnel failed: connection reset")
	fmt.Println(proxator.IsTransient(err))
}

Output:
true

Types

type AzureTLSFactory 

type AzureTLSFactory struct{}

AzureTLSFactory creates sessions with a Chrome-like TLS fingerprint.

Use it when a target requires browser-like TLS behavior. HTTPFactory is the package default for standard, transport-agnostic HTTP proxying.

func (AzureTLSFactory) New 

func (AzureTLSFactory) New(proxyURL string) (Session, error)

New creates an Azure TLS session routed through proxyURL.

type BlockDetector 

type BlockDetector struct {
	// contains filtered or unexported fields
}

BlockDetector classifies responses and transport errors as blocking. A zero-value BlockDetector is not usable — construct one with NewBlockDetector or NewBlockDetectorWith.

func NewBlockDetector 

func NewBlockDetector() *BlockDetector

NewBlockDetector returns a detector using the package defaults.

func NewBlockDetectorWith 

func NewBlockDetectorWith(cfg DetectorConfig) (*BlockDetector, error)

NewBlockDetectorWith returns a detector built from cfg, falling back to the package defaults for any field left empty.

func (*BlockDetector) IsBlocked 

func (d *BlockDetector) IsBlocked(resp *Response) bool

IsBlocked reports whether a response indicates blocking.

Successful (2xx) responses short-circuit before any body scan, so the common path costs one map lookup rather than running every pattern over the body.

func (*BlockDetector) IsBlockedError 

func (d *BlockDetector) IsBlockedError(err error) bool

IsBlockedError reports whether a transport error indicates blocking.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client routes requests across one or more named pools of rotating proxies.

A Client is safe for concurrent use. Call Close when finished to release every pooled transport session and stop background health checks.

Example (MultiplePools) 
package main

import (
	"context"
	"fmt"
	"net/http"
	"net/http/httptest"
	"net/url"
	"strconv"

	proxator "github.com/cpouldev/go-proxator"
)

func main() {
	euProxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		_, _ = w.Write([]byte("eu"))
	}))
	defer euProxy.Close()
	usProxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		_, _ = w.Write([]byte("us"))
	}))
	defer usProxy.Close()

	client, err := proxator.New(proxator.Config{
		SessionFactory: proxator.HTTPFactory{},
		Pools: []proxator.PoolConfig{
			{Name: "eu", Endpoints: []proxator.EndpointConfig{endpointConfig(euProxy.URL)}},
			{Name: "us", Endpoints: []proxator.EndpointConfig{endpointConfig(usProxy.URL)}},
		},
	})
	if err != nil {
		panic(err)
	}
	defer client.Close()

	ctx := context.Background()
	euResp, err := client.Get(ctx, "eu", "http://target.example/items")
	if err != nil {
		panic(err)
	}
	usResp, err := client.Get(ctx, "us", "http://target.example/items")
	if err != nil {
		panic(err)
	}

	fmt.Println(string(euResp.Body), string(usResp.Body))
}

func endpointConfig(rawURL string) proxator.EndpointConfig {
	parsed, err := url.Parse(rawURL)
	if err != nil {
		panic(err)
	}
	port, err := strconv.Atoi(parsed.Port())
	if err != nil {
		panic(err)
	}
	password, _ := parsed.User.Password()
	return proxator.EndpointConfig{
		Type:     parsed.Scheme,
		Username: parsed.User.Username(),
		Password: password,
		Host:     parsed.Hostname(),
		Port:     port,
	}
}

Output:
eu us

func New 

func New(cfg Config) (*Client, error)

New creates a Client from cfg. It eagerly constructs every configured pool and closes any sessions created successfully if construction fails.

func (*Client) Close 

func (c *Client) Close()

Close releases every pool. It is safe to call more than once.

func (*Client) Do 

func (c *Client) Do(ctx context.Context, pool string, fn RequestFunc) (*Response, error)

Do runs fn against a rotating endpoint of the named pool. It retries when fn returns an error or the response is classified as blocking. It selects an endpoint again before each attempt, but weighted selection may choose the same endpoint. MaxAttempts limits attempt count, not elapsed time; ctx bounds the complete operation.

Pass an empty pool name when exactly one pool is configured. Use DoForDomain when fn targets a known host and you want domain-aware endpoint selection.

Example 
package main

import (
	"context"
	"fmt"
	"net/http"
	"net/http/httptest"
	"net/url"
	"strconv"

	proxator "github.com/cpouldev/go-proxator"
)

func main() {
	proxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusCreated)
		_, _ = fmt.Fprintf(w, "%s %s", r.Method, r.Header.Get("X-Trace"))
	}))
	defer proxy.Close()

	client, err := proxator.New(proxator.Config{
		SessionFactory: proxator.HTTPFactory{},
		Pools: []proxator.PoolConfig{{
			Name:            "main",
			Endpoints:       []proxator.EndpointConfig{endpointConfig(proxy.URL)},
			SessionPoolSize: 1,
		}},
	})
	if err != nil {
		panic(err)
	}
	defer client.Close()

	ctx := context.Background()
	resp, err := client.Do(ctx, "", func(session proxator.Session) (*proxator.Response, error) {
		return session.Do(ctx, proxator.Request{
			Method:  http.MethodPost,
			URL:     "http://target.example/jobs",
			Headers: proxator.OrderedHeaders{{"X-Trace", "example"}},
		})
	})
	if err != nil {
		panic(err)
	}

	fmt.Printf("%d %s\n", resp.StatusCode, resp.Body)
}

func endpointConfig(rawURL string) proxator.EndpointConfig {
	parsed, err := url.Parse(rawURL)
	if err != nil {
		panic(err)
	}
	port, err := strconv.Atoi(parsed.Port())
	if err != nil {
		panic(err)
	}
	password, _ := parsed.User.Password()
	return proxator.EndpointConfig{
		Type:     parsed.Scheme,
		Username: parsed.User.Username(),
		Password: password,
		Host:     parsed.Hostname(),
		Port:     port,
	}
}

Output:
201 POST example

func (*Client) DoForDomain 

func (c *Client) DoForDomain(
	ctx context.Context,
	poolName string,
	domain string,
	fn RequestFunc,
) (*Response, error)

DoForDomain is Do with an explicit domain hint. Endpoints recently blocked for that domain are deprioritised while keeping full weight elsewhere.

func (*Client) Get 

func (c *Client) Get(ctx context.Context, pool, targetURL string) (*Response, error)

Get performs a GET request, deriving the domain hint from targetURL.

Example 
package main

import (
	"context"
	"fmt"
	"net/http"
	"net/http/httptest"
	"net/url"
	"strconv"

	proxator "github.com/cpouldev/go-proxator"
)

func main() {
	proxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		_, _ = w.Write([]byte("proxied"))
	}))
	defer proxy.Close()

	client, err := proxator.New(proxator.Config{
		SessionFactory: proxator.HTTPFactory{},
		Pools: []proxator.PoolConfig{{
			Name:      "main",
			Endpoints: []proxator.EndpointConfig{endpointConfig(proxy.URL)},
		}},
	})
	if err != nil {
		panic(err)
	}
	defer client.Close()

	resp, err := client.Get(context.Background(), "", "http://target.example/items")
	if err != nil {
		panic(err)
	}
	fmt.Printf("%d %s\n", resp.StatusCode, resp.Body)
}

func endpointConfig(rawURL string) proxator.EndpointConfig {
	parsed, err := url.Parse(rawURL)
	if err != nil {
		panic(err)
	}
	port, err := strconv.Atoi(parsed.Port())
	if err != nil {
		panic(err)
	}
	password, _ := parsed.User.Password()
	return proxator.EndpointConfig{
		Type:     parsed.Scheme,
		Username: parsed.User.Username(),
		Password: password,
		Host:     parsed.Hostname(),
		Port:     port,
	}
}

Output:
200 proxied

func (*Client) GetWithHeaders 

func (c *Client) GetWithHeaders(
	ctx context.Context,
	pool, targetURL string,
	headers OrderedHeaders,
) (*Response, error)

GetWithHeaders performs a GET request with headers represented in a stable order. Whether that order is preserved on the wire is transport-specific.

func (*Client) HasPool 

func (c *Client) HasPool(name string) bool

HasPool reports whether the named pool exists and holds any endpoints.

func (*Client) MarkEndpointDead 

func (c *Client) MarkEndpointDead(pool string, index int) error

MarkEndpointDead permanently retires one endpoint. Dead endpoints never recover on their own; call ResetEndpoint to bring one back.

func (*Client) Ping 

func (c *Client) Ping(ctx context.Context, pool, probeURL string) error

Ping fetches probeURL through one endpoint of the named pool and reports whether it answered with a non-error status.

probeURL is required: the library will not pick a third-party endpoint to poll on your behalf.

func (*Client) Pool 

func (c *Client) Pool(name string) *Pool

Pool returns the named pool, or nil when it does not exist.

func (*Client) PoolNames 

func (c *Client) PoolNames() []string

PoolNames returns the configured pool names in sorted order.

func (*Client) Post 

func (c *Client) Post(ctx context.Context, pool, targetURL string, body any) (*Response, error)

Post performs a POST request, deriving the domain hint from targetURL. When body is a consumable io.Reader, retries reuse it after the first attempt; use Do with a RequestFunc that creates a fresh reader when necessary.

func (*Client) PostWithHeaders 

func (c *Client) PostWithHeaders(
	ctx context.Context,
	pool, targetURL string,
	body any,
	headers OrderedHeaders,
) (*Response, error)

PostWithHeaders performs a POST request with headers represented in a stable order. Whether that order is preserved on the wire is transport-specific. When body is a consumable io.Reader, use Do to recreate it for each attempt.

func (*Client) ResetEndpoint 

func (c *Client) ResetEndpoint(pool string, index int) error

ResetEndpoint returns one endpoint to service and clears its failure history.

func (*Client) Stats 

func (c *Client) Stats() map[string]PoolStats

Stats returns a snapshot of every pool, keyed by pool name.

Example 
package main

import (
	"fmt"

	proxator "github.com/cpouldev/go-proxator"
)

func main() {
	client, err := proxator.New(proxator.Config{
		Pools: []proxator.PoolConfig{{
			Name: "main",
			Endpoints: []proxator.EndpointConfig{{
				Type: "http",
				Host: "proxy.example",
				Port: 8080,
			}},
			SessionPoolSize: 1,
		}},
	})
	if err != nil {
		panic(err)
	}
	defer client.Close()

	stats := client.Stats()["main"]
	fmt.Printf("%s: %d alive, %d total\n", stats.Name, stats.Alive, stats.Total)
}

Output:
main: 1 alive, 1 total

type Config 

type Config struct {
	// Pools are the named proxy pools. At least one pool is required and
	// every pool must carry a unique, non-empty Name.
	Pools []PoolConfig

	// Retry governs attempts, backoff, and endpoint cooldown across all pools.
	// Each non-positive field is replaced by its value from DefaultRetryConfig.
	Retry RetryConfig

	// Detector classifies responses and transport errors as blocked.
	// Defaults to NewBlockDetector.
	Detector *BlockDetector

	// SessionFactory creates the sessions held by every endpoint. It defaults
	// to HTTPFactory. Client construction calls the factory
	// SessionPoolSize times per endpoint and owns every session returned
	// successfully. Supply another factory to use a browser-like TLS adapter,
	// a custom transport, or an in-memory test implementation.
	SessionFactory SessionFactory

	// Logger receives operational events: cooldown entry and recovery, dead
	// endpoints, and health-check failures. Defaults to slog.Default.
	Logger *slog.Logger
}

Config configures a Client.

type DetectorConfig 

type DetectorConfig struct {
	// StatusCodes replaces DefaultBlockedStatusCodes when non-empty.
	StatusCodes []int

	// BodyPatterns replaces DefaultBodyPatterns when non-empty. Entries are
	// regular expressions; an invalid one makes NewBlockDetectorWith fail.
	BodyPatterns []string

	// ErrorPatterns replaces DefaultErrorPatterns when non-empty. Entries are
	// matched as lowercase substrings.
	ErrorPatterns []string

	// DisableCloudflareHeaderCheck turns off the heuristic that treats a >=400
	// response carrying genuine Cloudflare headers as a block.
	DisableCloudflareHeaderCheck bool
}

DetectorConfig customises block detection.

type EndpointConfig 

type EndpointConfig struct {
	// Type is the proxy protocol, such as "http", "https", or "socks5".
	// Supported types depend on Config.SessionFactory.
	Type string

	// Username and Password are the credentials for this endpoint. Both are
	// optional; Password may be set when the proxy accepts an empty username.
	Username string
	Password string

	// Host is the proxy host without a port. IPv6 literals may be supplied with
	// or without brackets.
	Host string

	// Port is the proxy port. It must be between 1 and 65535.
	Port int
}

EndpointConfig describes one proxy endpoint in a pool.

Keep provider-specific session conventions outside proxator: specify the exact credentials for each endpoint rather than relying on generated names.

type EndpointStats 

type EndpointStats struct {
	Index             int
	State             State
	FailCount         int
	AvailableSessions int
	TotalSessions     int
	AvgLatency        time.Duration
	TotalRequests     int64
	SuccessRequests   int64
	SuccessRate       float64
	CooldownTier      int
}

EndpointStats is a snapshot of one endpoint's health.

type HTTPFactory 

type HTTPFactory struct{}

HTTPFactory creates sessions backed by Go's standard net/http transport. It uses the standard TLS certificate verification behavior. Headers are applied through http.Header, so their field order is not guaranteed on the wire. Request bodies supplied as string, []byte, or io.Reader are sent raw; other values are JSON-encoded and receive an application/json content type when no content type was supplied. Use it when browser TLS fingerprinting is unnecessary.

func (HTTPFactory) New 

func (HTTPFactory) New(proxyURL string) (Session, error)

New creates a standard HTTP session routed through an HTTP or HTTPS proxy URL. Other proxy schemes, including SOCKS5, are rejected.

type OrderedHeaders 

type OrderedHeaders [][]string

OrderedHeaders represents headers as ordered rows. Each row contains a field name followed by one or more values. Adapters that support ordered headers can preserve the row order; HTTPFactory applies the rows to http.Header and does not guarantee their order on the wire.

type Pool 

type Pool struct {
	// contains filtered or unexported fields
}

Pool manages one named set of interchangeable proxy endpoints.

func (*Pool) Close 

func (p *Pool) Close()

Close releases every session and stops the health checker.

func (*Pool) Name 

func (p *Pool) Name() string

Name returns the pool's configured name.

func (*Pool) Stats 

func (p *Pool) Stats() PoolStats

Stats returns a point-in-time snapshot of the pool.

type PoolConfig 

type PoolConfig struct {
	// Name identifies the pool in Client calls and in Stats. Required.
	Name string

	// Endpoints are the explicit proxy endpoints in the pool. At least one is
	// required.
	Endpoints []EndpointConfig

	// SessionPoolSize is the number of concurrent transport sessions held per
	// endpoint. Defaults to 15. This is also the rate limiter's burst size.
	SessionPoolSize int

	// RequestsPerSecond caps the sustained request rate per endpoint.
	// Zero means unlimited.
	RequestsPerSecond float64

	// HealthCheckInterval enables background health checks when greater than
	// zero. HealthCheckURL is then required.
	HealthCheckInterval time.Duration

	// HealthCheckURL is fetched through every alive endpoint on each
	// health-check tick.
	//
	// There is deliberately no default. Pick a URL you control or are content
	// to poll indefinitely — a library that silently polls a third party on
	// every user's behalf is a bad neighbour.
	HealthCheckURL string

	// HealthCheckTimeout bounds a single health-check request. Defaults to 15s.
	HealthCheckTimeout time.Duration

	// DomainBlockTTL controls how long a block lowers an endpoint's weight for
	// one target domain. Defaults to 5 minutes.
	DomainBlockTTL time.Duration

	// DomainBlockPenalty is the weight multiplier for an endpoint recently
	// blocked by one domain. It must be greater than 0 and at most 1. The zero
	// value selects the default, 0.1.
	DomainBlockPenalty float64
}

PoolConfig describes one named pool of interchangeable proxy endpoints.

type PoolStats 

type PoolStats struct {
	Name            string
	Total           int
	Alive           int
	Dead            int
	Cooldown        int
	SessionPoolSize int
	Endpoints       []EndpointStats
}

PoolStats is a snapshot of one pool's health.

type Request 

type Request struct {
	// Method is the HTTP request method.
	Method string

	// URL is the absolute target URL, not the proxy URL.
	URL string

	// Body is the request payload. The built-in adapters send nil, string,
	// []byte, and io.Reader values without JSON encoding and JSON-encode other
	// supported values. Exact support is adapter-specific. A RequestFunc that
	// may be retried should recreate a consumable io.Reader for every attempt.
	Body any

	// Headers are the request header rows.
	Headers OrderedHeaders
}

Request is the transport-neutral form of one proxied HTTP request.

type RequestFunc 

type RequestFunc func(session Session) (*Response, error)

RequestFunc issues a request on a borrowed session. It must not retain the session beyond the call and must return a non-nil Response with a nil error. To observe cancellation, capture the request context and pass it to Session.Do.

type Response 

type Response struct {
	// StatusCode is the numeric HTTP status code.
	StatusCode int

	// Status is the complete HTTP status, such as "200 OK".
	Status string

	// Body is the fully buffered response payload.
	Body []byte

	// Header contains the response headers.
	Header http.Header
}

Response contains the response data used by routing, block detection, and callers. Transport adapters must fully buffer Body before returning. Client methods may return both a Response and an error when retries end on a blocking response.

type RetryConfig 

type RetryConfig struct {
	// MaxAttempts is the total number of attempts per request, including the
	// initial attempt. An endpoint is selected again before every attempt. A
	// non-positive value selects the default, 3.
	MaxAttempts int

	// InitialDelay is the base delay between attempts. Combined with
	// exponential backoff and jitter.
	InitialDelay time.Duration

	// MaxDelay caps the delay between attempts.
	MaxDelay time.Duration

	// CooldownPeriod is the base cooldown applied when an endpoint crosses
	// FailThreshold. Each consecutive cooldown doubles it.
	CooldownPeriod time.Duration

	// MaxCooldown caps the escalated cooldown. The effective duration never
	// exceeds the lower of MaxCooldown and 16 times CooldownPeriod. A
	// non-positive value selects the default, 5 minutes.
	MaxCooldown time.Duration

	// FailThreshold is the number of consecutive failures before an endpoint
	// enters cooldown. Defaults to 3.
	FailThreshold int32
}

RetryConfig configures retry, backoff and endpoint cooldown behaviour.

func DefaultRetryConfig 

func DefaultRetryConfig() RetryConfig

DefaultRetryConfig returns sensible defaults for retry configuration.

type Session 

type Session interface {
	Do(context.Context, Request) (*Response, error)
	Close()
}

Session sends requests through one proxy endpoint. A Session is borrowed by one goroutine at a time and must not be retained after a RequestFunc returns. Do must honor ctx and return a non-nil Response whenever it returns a nil error. The Client owns the Session and calls Close when finished.

type SessionFactory 

type SessionFactory interface {
	New(proxyURL string) (Session, error)
}

SessionFactory creates a transport session configured for proxyURL. New calls it eagerly SessionPoolSize times for each endpoint. A successful call must return a distinct Session and transfers its ownership to the Client; an error must not transfer ownership. Supported proxy URL schemes are implementation-specific. Config.SessionFactory defaults to HTTPFactory; AzureTLSFactory is available as an explicit browser-like TLS option.

type State 

type State int32

State is the lifecycle state of a single proxy endpoint. 

const (
	// StateAlive means the endpoint is eligible for selection.
	StateAlive State = iota
	// StateDead means the endpoint was permanently retired and will not
	// recover without an explicit ResetEndpoint call.
	StateDead
	// StateCooldown means the endpoint failed recently and is temporarily
	// excluded. It returns to StateAlive when its cooldown elapses.
	StateCooldown
)

func (State) String 

func (s State) String() string

type TransientClassifier 

type TransientClassifier struct {
	// contains filtered or unexported fields
}

TransientClassifier decides whether an error is a transient external failure that will clear on its own, and therefore should be retried and kept out of your error tracker rather than paged on.

It works in three tiers, because a flat substring list produces false positives that are worse than the noise it removes:

  1. Unambiguous proxy failures — always transient.
  2. Generic network failures — transient only alongside proxy context.
  3. Blocking and rate-limit signals — transient only alongside proxy context.

Without tier 2's context requirement, a "connection refused" from your primary database would be silently classified as a proxy blip.

func NewTransientClassifier 

func NewTransientClassifier() *TransientClassifier

NewTransientClassifier returns a classifier using the package defaults.

func NewTransientClassifierWith 

func NewTransientClassifierWith(cfg TransientConfig) *TransientClassifier

NewTransientClassifierWith returns a classifier built from cfg, falling back to the package defaults for any field left empty.

func (*TransientClassifier) IsTransient 

func (c *TransientClassifier) IsTransient(err error) bool

IsTransient reports whether err is a transient external failure.

type TransientConfig 

type TransientConfig struct {
	UnambiguousPatterns []string
	AmbiguousPatterns   []string
	BlockingPatterns    []string
	ContextPatterns     []string
}

TransientConfig customises a TransientClassifier. Empty fields fall back to the corresponding package defaults.

Source Files

View all Source files

Directories

Path Synopsis
examples
basic command
Command basic sends one request through a configured proxy endpoint.
 Command basic sends one request through a configured proxy endpoint.
internal
domain
Package domain contains domain-aware routing state used by proxator.
 Package domain contains domain-aware routing state used by proxator.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.