README
¶
proxypool
A rotating proxy pool with cooldowns, for Go.
No dependencies. No background goroutines. Safe for concurrent use.
Why
Every project that runs more than one proxy ends up rewriting the same four things, each of them small and collectively a morning:
- providers emit at least four different line formats, and none of them is the URL an HTTP client accepts
- a blocked address should stop being handed out for a while, then come back on its own
- credentials must never reach a log
- when every address is cooling down, the caller needs to be told rather than handed one anyway
That's the whole scope. There are no health checks and no scoring, because whether an address is usable is something only your own traffic can answer. A liveness probe that passes proves nothing about the request you actually care about.
Install
go get github.com/Mootjelh/proxypool
Use
pool, err := proxypool.Load("proxies.txt")
if err != nil {
return err
}
for {
p, healthy := pool.Next()
if !healthy {
// Every address is cooling down. p is the one recovering soonest.
time.Sleep(5 * time.Second)
continue
}
transport, err := p.Transport()
if err != nil {
return err
}
client := &http.Client{Transport: transport, Timeout: 20 * time.Second}
resp, err := client.Get(target)
if err != nil || resp.StatusCode == http.StatusTooManyRequests {
log.Printf("blocked on %s", p) // credentials stripped, index kept
pool.Block(p, 10*time.Minute)
continue
}
...
}
Input formats
Normalize takes what providers hand out:
| input | result |
|---|---|
1.2.3.4:8080 |
http://1.2.3.4:8080 |
gw.example.com:7000:user:pass |
http://user:pass@gw.example.com:7000 |
user:pass@1.2.3.4:8080 |
http://user:pass@1.2.3.4:8080 |
socks5://user:pass@host:1080 |
unchanged |
Load reads a file of them, skipping blank lines and # comments. A line it can't parse is an error naming the line number rather than a silent skip. A proxy list is configuration, and quietly running on nine of ten addresses is worse than not starting at all.
Changing the list while it runs
Providers hand out a fresh list every so often. Replace swaps it and keeps the cooldown of every address that survives:
if err := pool.Replace(newList); err != nil {
return err
}
Building a second Pool would swap the list too, and would forget that four of those addresses were blocked a minute ago. They'd go straight back into rotation, be refused again, and the pool would relearn the same thing on every refresh.
Add appends and skips addresses already present, because a duplicate takes a double share of the traffic under round robin. Remove drops one. Both refuse a batch containing an unparseable address rather than applying part of it.
Seeing what the pool is doing
Len and Healthy give you two numbers, which is not enough to tune anything. Stats returns a snapshot per address:
for _, s := range pool.Stats() {
log.Println(s)
}
pool#0 gw.example.com:5555 out=412 blocked=3
pool#1 gw.example.com:5555 out=409 blocked=87 cooling 4m20s
pool#2 gw.example.com:5555 out=0 blocked=0 never used
Providers differ enormously in how often their addresses get refused, so a pool mixing two of them looks fine in aggregate while half of it fails. The middle line above is that half.
never used is worth grepping for. An address that has never been handed out on a pool that has been running for hours usually means the rotation is not reaching it.
Two details that are easy to get wrong
Logging a pool whose entries share a host
Credentials have to be stripped from logs. But sticky residential pools give every entry the same host:port and vary only the username, which is what encodes the session. Strip the credentials and a pool of a thousand prints as one line repeated, so a log showing a healthy rotation looks identical to a log showing every request pinned to one address.
Proxy.String() keeps the index for that reason:
pool#0 gw.example.com:5555
pool#1 gw.example.com:5555
pool#2 gw.example.com:5555
The BOM in your proxy list
PowerShell's Out-File and Set-Content write UTF-8 with a BOM by default. Those three invisible bytes attach to the first address only, so gw.example.com becomes a hostname that doesn't resolve, and it presents as that one proxy being dead. Not the sort of thing you'd think to check.
Load strips it, and there's a test for it.
Why the parser checks field count first
A password can legitimately contain an @. So host:port:user:p@ss is checked by field count before anything searches for an @, otherwise it reads as user:pass@host:port and yields a URL pointing at a host that was never in your file.
Field count is unambiguous where a character search isn't. There's a test on this one, because it's the sort of thing a later refactor simplifies straight back into a bug.
What it doesn't do
- No health checking. Use your own traffic.
- No per-request retry. That's your control flow, not the pool's.
- No SOCKS dialer.
Proxy.Transport()handssocks5://URLs tonet/http, which supports them. Anything more exotic is yours to build fromProxy.URL. - No rotation strategies beyond round robin. If you want one address pinned per long-lived session, use
Pool.All()and index it yourself. That's a different strategy, and stacking it onNext()gets you neither.
License
MIT
Documentation
¶
Overview ¶
Package proxypool hands out proxies round-robin and sidelines the ones that got blocked.
It solves the small set of problems that show up every time you run more than one proxy, and that are individually trivial and collectively tedious:
- providers emit at least four different line formats, and none of them is the URL an HTTP client wants
- a blocked address should stop being handed out for a while, then come back on its own
- credentials must never reach a log
- when every address is cooling down, the caller needs to know that rather than be handed one silently
It is deliberately small: no health checks, no scoring, no background goroutines. A Pool is a picker with a cooldown, and whether an address is healthy is something only the caller's own traffic can answer.
All methods are safe for concurrent use.
Example ¶
The usual loop: take an address, use it, sideline it when the far end complains, and let it come back on its own.
Note what the log lines look like. All three entries share one host:port and differ only in the username, which is how sticky residential pools work, so the index is the only thing that tells them apart once the credentials are stripped.
package main
import (
"fmt"
"time"
"github.com/Mootjelh/proxypool"
)
func main() {
pool := proxypool.New([]string{
"http://sess-a:pw@gw.example.com:5555",
"http://sess-b:pw@gw.example.com:5555",
"http://sess-c:pw@gw.example.com:5555",
})
for i := 0; i < 4; i++ {
p, healthy := pool.Next()
if !healthy {
fmt.Println("every address is cooling down; soonest back:", p)
break
}
fmt.Println("request", i, "via", p)
if i == 1 {
// Suppose that one came back 429.
pool.Block(p, time.Hour)
}
}
fmt.Printf("healthy: %d of %d\n", pool.Healthy(), pool.Len())
}
Output: request 0 via pool#0 gw.example.com:5555 request 1 via pool#1 gw.example.com:5555 request 2 via pool#2 gw.example.com:5555 request 3 via pool#0 gw.example.com:5555 healthy: 2 of 3
Index ¶
- func Mask(raw string) string
- func Normalize(raw string) (string, error)
- type Pool
- func (p *Pool) Add(raw ...string) (int, error)
- func (p *Pool) All() []Proxy
- func (p *Pool) Block(pr Proxy, d time.Duration)
- func (p *Pool) Healthy() int
- func (p *Pool) Len() int
- func (p *Pool) Next() (Proxy, bool)
- func (p *Pool) Remove(raw string) bool
- func (p *Pool) Replace(raw []string) error
- func (p *Pool) Stats() []Stats
- func (p *Pool) URLs() []string
- type Proxy
- type Stats
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Mask ¶
Mask strips credentials and scheme from a proxy URL so logins never reach a log. It is what Proxy.String uses.
Prefer logging the Proxy itself: on a pool whose entries share a host, a masked URL alone cannot tell them apart.
func Normalize ¶
Normalize converts the common proxy list formats into a URL an HTTP client accepts:
host:port -> http://host:port host:port:user:pass -> http://user:pass@host:port user:pass@host:port -> http://user:pass@host:port scheme://... -> unchanged
The third form is what most residential providers emit by default, and it is the one no HTTP client takes.
Example ¶
Normalize accepts the formats providers actually hand out, none of which is the URL an HTTP client wants.
package main
import (
"fmt"
"github.com/Mootjelh/proxypool"
)
func main() {
for _, raw := range []string{
"1.2.3.4:8080",
"gw.example.com:7000:user:pass",
"user:pass@1.2.3.4:8080",
"socks5://user:pass@host:1080",
} {
u, err := proxypool.Normalize(raw)
if err != nil {
fmt.Println("error:", err)
continue
}
fmt.Println(u)
}
}
Output: http://1.2.3.4:8080 http://user:pass@gw.example.com:7000 http://user:pass@1.2.3.4:8080 socks5://user:pass@host:1080
Types ¶
type Pool ¶
type Pool struct {
// contains filtered or unexported fields
}
Pool hands out proxies round-robin and sidelines ones that got blocked. The zero Pool is not usable; build one with New or Load.
func Load ¶
Load reads a proxy list from disk, one entry per line, and normalises each one. Blank lines and lines starting with # are ignored.
An unparseable line is an error naming the line number, rather than a silent skip: a proxy list is configuration, and quietly running on nine of ten addresses is worse than not starting.
func New ¶
New builds a pool from already-normalised proxy URLs.
An empty slice, or a slice containing one empty string, means "connect directly", so a caller that has no proxies configured can use the same code path as one that does, rather than branching everywhere.
func (*Pool) Add ¶ added in v0.2.0
Add appends addresses, normalising each one the way Load does, and reports how many were new.
Addresses already in the pool are skipped rather than appended twice. A duplicate is not harmless: round robin would hand it out twice per cycle, so that address would quietly take a double share of the traffic and reach a rate limit first.
If any address cannot be parsed, nothing is added and the error names it. Adding nine of ten silently is the failure mode Load already refuses.
func (*Pool) All ¶
All returns every proxy in the pool, in file order, regardless of cooldown.
Use it when you want the list rather than the picker, for instance to pin one address per long-lived session, which is a different rotation strategy from this pool's per-request round robin and should not be built on top of Pool.Next.
func (*Pool) Block ¶
Block sidelines a proxy for the given duration after it was throttled.
A shorter Block never shortens a cooldown already in place. Two failures on one address should not leave it less penalised than one.
func (*Pool) Next ¶
Next returns the proxy to use for the next request.
The bool reports whether it is currently healthy. When every proxy is cooling down it returns the one that recovers soonest, with false. The caller is better placed than the pool to decide between waiting, proceeding anyway and giving up.
func (*Pool) Remove ¶ added in v0.2.0
Remove drops an address and reports whether it was there. It can be passed in any format Normalize accepts.
It also matches the string verbatim, and that is not belt and braces. New takes its addresses as given, so a pool can hold something Normalize would reject, and matching only the normalised form would leave that entry in the pool with no way to take it out.
Removing the last entry leaves an empty pool, and Pool.Next then reports unhealthy rather than panicking.
func (*Pool) Replace ¶ added in v0.2.0
Replace swaps the whole list, keeping the cooldown of every address that survives the swap.
This is what a provider handing out a fresh list every so often needs. Building a second Pool would do the swap too, and would forget that four of those addresses were blocked a minute ago: they would go straight back into rotation, be refused again, and the pool would relearn the same thing every time the list refreshed.
If any address cannot be parsed, the pool is left untouched.
func (*Pool) Stats ¶ added in v0.3.0
Stats returns a snapshot per address, in file order.
Two counts and a total is not enough to tune a pool. Providers differ enormously in how often their addresses get refused, so a pool mixing two of them looks fine in aggregate while half of it fails. This is the per-address view that shows which half.
type Proxy ¶
type Proxy struct {
// URL is the full proxy URL, credentials included. Pass it to your HTTP
// client, and do not log it. Use the Proxy itself for that.
URL string
// Index is the address's stable position in the pool, counting from 0.
Index int
}
Proxy is one address from the pool.
func (Proxy) String ¶
String renders the proxy for logs: credentials stripped, index kept.
The index is not decoration. Rotating residential pools typically give every entry the SAME host:port and vary only the username, which encodes the sticky session. Strip the credentials, which you must, and a pool of a thousand addresses prints as one, so a log that says the rotation is working looks exactly like a log that says it is not.
func (Proxy) Transport ¶
Transport returns an *http.Transport routed through this proxy, or a plain one when the entry is direct.
Example ¶
Transport is the bridge to net/http. Build a fresh client per address rather than swapping the proxy on a shared transport: a transport pools connections, and a pooled connection is already through the old proxy.
package main
import (
"fmt"
"net/http"
"time"
"github.com/Mootjelh/proxypool"
)
func main() {
pool := proxypool.New([]string{"http://user:pw@gw.example.com:5555"})
p, _ := pool.Next()
transport, err := p.Transport()
if err != nil {
fmt.Println(err)
return
}
client := &http.Client{Transport: transport, Timeout: 20 * time.Second}
_ = client
fmt.Println("routing through", p)
}
Output: routing through pool#0 gw.example.com:5555
type Stats ¶ added in v0.3.0
type Stats struct {
Proxy Proxy
// HandedOut counts every time [Pool.Next] returned this address, including
// the times it came back with healthy false. The caller receives it either
// way, and leaving those out would understate the address carrying an
// exhausted pool.
HandedOut int
// Blocked counts every [Pool.Block] that resolved to this address, whether
// or not it extended a cooldown already in place.
Blocked int
// LastUsed is when Next last returned it. Zero means never, which is worth
// looking for: an address that has never been handed out on a pool that has
// been running for hours is usually a sign the rotation is not reaching it.
LastUsed time.Time
// CoolingUntil is when the current cooldown expires. Zero means healthy.
CoolingUntil time.Time
}
Stats is what one address has done. It is a copy, so it can be held, sorted and logged without touching the pool again.
Source Files