README
¶
proxyhive
Go proxy pool manager with health checking, latency scoring, rotation strategies, geo-targeting, and sticky sessions.
Feed it a list of proxies and it handles the rest — background health checks, EWMA latency tracking, automatic backoff on failures, and smart rotation via Power-of-Two-Choices, round-robin, least-latency, or random selection.
Features
- Multiple protocols — HTTP CONNECT, SOCKS5, SOCKS4
- Rotation strategies — P2C (default), round-robin, random, least-latency
- Health checking — background checks with configurable interval, EWMA latency tracking
- Automatic backoff — exponential backoff on failures, automatic recovery
- Geo-filtering — filter by country, city, ASN
- Sticky sessions — pin a key to a specific proxy for a TTL
http.RoundTripper— drop-in transport forhttp.ClientDialContext— compatible withnet.Dialerfor raw connections
Install
go get github.com/divinedev111/proxyhive
Quick Start
package main
import (
"fmt"
"log"
"github.com/divinedev111/proxyhive"
)
func main() {
pool := proxyhive.New(
proxyhive.WithStrategy(proxyhive.P2C()),
proxyhive.WithHealthCheckURL("https://httpbin.org/ip"),
)
defer pool.Close()
pool.Add("http://proxy1.example.com:8080", proxyhive.InCountry("US"))
pool.Add("socks5://user:pass@proxy2.example.com:1080", proxyhive.InCountry("DE"))
proxy, err := pool.Get(proxyhive.WithCountry("US"))
if err != nil {
log.Fatal(err)
}
fmt.Printf("Selected: %s (%s)\n", proxy.Addr(), proxy.Protocol)
}
Usage
As an HTTP Transport
pool := proxyhive.New()
pool.Add("http://proxy1:8080")
pool.Add("http://proxy2:8080")
client := &http.Client{
Transport: pool.RoundTripper(),
}
resp, err := client.Get("https://example.com")
With Geo-Filtering
proxy, err := pool.Get(
proxyhive.WithCountry("US", "CA"),
proxyhive.WithMaxLatency(500 * time.Millisecond),
)
With Sticky Sessions
Pin requests to the same proxy by key:
transport := pool.RoundTripper(
proxyhive.WithCountry("US"),
proxyhive.Sticky("session-abc", 10*time.Minute),
)
client := &http.Client{Transport: transport}
// All requests use the same proxy for 10 minutes
As a Dialer
dialer := pool.NewDialer(proxyhive.WithProtocol(proxyhive.SOCKS5))
conn, err := dialer.DialContext(ctx, "tcp", "example.com:443")
Pool Stats
stats := pool.Stats()
fmt.Printf("Total: %d, Alive: %d, Dead: %d\n", stats.Total, stats.Alive, stats.Dead)
fmt.Printf("Avg latency: %v\n", stats.AvgLatency)
fmt.Printf("Countries: %v\n", stats.Countries)
Configuration
pool := proxyhive.New(
proxyhive.WithStrategy(proxyhive.LeastLatency()), // rotation strategy
proxyhive.WithHealthCheck(30*time.Second, 10*time.Second), // interval, timeout
proxyhive.WithHealthCheckURL("https://httpbin.org/ip"), // health check target
proxyhive.WithEWMADecay(0.3), // latency smoothing factor
proxyhive.WithMaxBackoff(time.Hour), // max backoff for dead proxies
)
Strategies
| Strategy | Description |
|---|---|
P2C() |
Power of Two Choices — pick 2 random, use the faster one. Default. |
RoundRobin() |
Cycle through proxies sequentially |
Random() |
Uniform random selection |
LeastLatency() |
Always pick the proxy with lowest EWMA latency |
Proxy Options
pool.Add("http://proxy:8080",
proxyhive.InCountry("US"),
proxyhive.InCity("New York"),
proxyhive.WithASNTag("AS13335"),
)
Architecture
proxyhive.go Pool constructor
pool.go Pool with Add/Get/Remove/Stats and filtering
proxy.go Proxy type with EWMA latency and backoff
strategy.go Rotation strategies (P2C, RoundRobin, Random, LeastLatency)
health.go Background health checker
transport.go http.RoundTripper with sticky sessions
dialer.go net.Dialer-compatible DialContext
geo.go Geo/protocol/latency filters
parse.go Proxy URL parser
options.go Configuration options
dial/
http.go HTTP CONNECT dialer
socks5.go SOCKS5 dialer
socks4.go SOCKS4 dialer
License
MIT
Documentation
¶
Overview ¶
Package proxyhive manages proxy pools with health checking, rotation strategies, latency scoring, geo-filtering, and sticky sessions.
Index ¶
- type Dialer
- type Filter
- type HealthChecker
- type Option
- type Pool
- func (p *Pool) Add(rawURL string, opts ...ProxyOption) error
- func (p *Pool) AddMany(rawURLs []string, opts ...ProxyOption) error
- func (p *Pool) Alive() []*Proxy
- func (p *Pool) Close()
- func (p *Pool) DialContext(ctx context.Context, network, addr string) (net.Conn, error)
- func (p *Pool) Get(filters ...Filter) (*Proxy, error)
- func (p *Pool) NewDialer(filters ...Filter) *Dialer
- func (p *Pool) Remove(rawURL string)
- func (p *Pool) RoundTripper(filters ...Filter) http.RoundTripper
- func (p *Pool) Stats() PoolStats
- type PoolStats
- type Protocol
- type Proxy
- type ProxyOption
- type Strategy
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Dialer ¶
type Dialer struct {
// contains filtered or unexported fields
}
Dialer wraps a Pool to provide a DialContext compatible with net.Dialer.
type Filter ¶
Filter narrows proxy selection by returning true for matching proxies.
func Sticky ¶
Sticky returns a special filter that enables sticky sessions. The given key maps to a specific proxy for the duration of ttl.
func WithAlive ¶
func WithAlive() Filter
WithAlive returns a filter that matches only alive proxies. Applied by default.
func WithCity ¶
WithCity returns a filter that matches proxies in any of the given cities (case-insensitive).
func WithCountry ¶
WithCountry returns a filter that matches proxies in any of the given countries (ISO 3166-1 alpha-2).
func WithMaxLatency ¶
WithMaxLatency returns a filter that excludes proxies with EWMA latency above d.
func WithProtocol ¶
WithProtocol returns a filter that matches proxies with the given protocol.
type HealthChecker ¶
type HealthChecker struct {
// contains filtered or unexported fields
}
HealthChecker runs background health checks against proxies in a pool.
type Option ¶
type Option func(*poolConfig)
Option configures a Pool.
func WithEWMADecay ¶
WithEWMADecay sets the alpha factor for EWMA latency calculation.
func WithHealthCheck ¶
WithHealthCheck sets the interval and timeout for background health checks.
func WithHealthCheckURL ¶
WithHealthCheckURL sets the URL used for health check validation.
func WithMaxBackoff ¶
WithMaxBackoff sets the maximum backoff duration for failed proxies.
func WithStrategy ¶
WithStrategy sets the proxy rotation strategy.
type Pool ¶
type Pool struct {
// contains filtered or unexported fields
}
Pool manages a set of proxies with health checking and rotation.
func (*Pool) Add ¶
func (p *Pool) Add(rawURL string, opts ...ProxyOption) error
Add parses and adds a single proxy to the pool.
func (*Pool) AddMany ¶
func (p *Pool) AddMany(rawURLs []string, opts ...ProxyOption) error
AddMany parses and adds multiple proxies to the pool.
func (*Pool) Close ¶
func (p *Pool) Close()
Close stops the background health checker and releases resources.
func (*Pool) DialContext ¶
DialContext connects to the target address through a pool proxy.
func (*Pool) Get ¶
Get selects a proxy from the pool using the configured strategy. WithAlive is always applied. Additional filters narrow the candidates.
func (*Pool) RoundTripper ¶
func (p *Pool) RoundTripper(filters ...Filter) http.RoundTripper
RoundTripper returns an http.RoundTripper that routes requests through the pool.
type PoolStats ¶
type PoolStats struct {
Total int
Alive int
Dead int
AvgLatency time.Duration
Protocols map[Protocol]int
Countries map[string]int
}
PoolStats contains aggregate statistics about the pool.
type Proxy ¶
type Proxy struct {
URL *url.URL
Protocol Protocol
Country string
City string
ASN string
// contains filtered or unexported fields
}
Proxy represents a single proxy with metadata and health stats.
func ParseProxy ¶
ParseProxy parses a raw proxy string into a Proxy. Supported formats: http://host:port, socks5://user:pass@host:port, socks4://host:port, or bare host:port (defaults to HTTP).
func ParseProxyList ¶
ParseProxyList reads proxies from a reader, one per line. Empty lines and lines starting with # are skipped.
func (*Proxy) SuccessCount ¶
SuccessCount returns the total successful checks.
type ProxyOption ¶
type ProxyOption func(*Proxy)
ProxyOption configures per-proxy metadata when adding.
func WithASNTag ¶
func WithASNTag(asn string) ProxyOption
WithASNTag sets the proxy's ASN identifier.
type Strategy ¶
Strategy selects a proxy from a list of candidates.
func LeastLatency ¶
func LeastLatency() Strategy
LeastLatency returns a strategy that always picks the lowest latency proxy. Warning: may cause thundering herd if many clients use the same pool.
func P2C ¶
func P2C() Strategy
P2C returns a Power of Two Choices strategy. Picks 2 random proxies and returns the one with lower latency.
func Random ¶
func Random() Strategy
Random returns a strategy that selects a proxy uniformly at random.
func RoundRobin ¶
func RoundRobin() Strategy
RoundRobin returns a strategy that cycles through proxies sequentially.
Source Files
Directories