README
¶
proxypool
A rotating proxy pool with cooldowns, for Go.
No dependencies. No background goroutines. Safe for concurrent use.
Why
Every project that runs more than one proxy ends up rewriting the same four things, each of them small and collectively a morning:
- providers emit at least four different line formats, and none of them is the URL an HTTP client accepts
- a blocked address should stop being handed out for a while, then come back on its own
- credentials must never reach a log
- when every address is cooling down, the caller needs to be told rather than handed one anyway
That's the whole scope. There are no health checks and no scoring, because whether an address is usable is something only your own traffic can answer. A liveness probe that passes proves nothing about the request you actually care about.
Install
go get github.com/Mootjelh/proxypool
Use
pool, err := proxypool.Load("proxies.txt")
if err != nil {
return err
}
for {
p, healthy := pool.Next()
if !healthy {
// Every address is cooling down. p is the one recovering soonest.
time.Sleep(5 * time.Second)
continue
}
transport, err := p.Transport()
if err != nil {
return err
}
client := &http.Client{Transport: transport, Timeout: 20 * time.Second}
resp, err := client.Get(target)
if err != nil || resp.StatusCode == http.StatusTooManyRequests {
log.Printf("blocked on %s", p) // credentials stripped, index kept
pool.Block(p, 10*time.Minute)
continue
}
...
}
Input formats
Normalize takes what providers hand out:
| input | result |
|---|---|
1.2.3.4:8080 |
http://1.2.3.4:8080 |
gw.example.com:7000:user:pass |
http://user:pass@gw.example.com:7000 |
user:pass@1.2.3.4:8080 |
http://user:pass@1.2.3.4:8080 |
socks5://user:pass@host:1080 |
unchanged |
Load reads a file of them, skipping blank lines and # comments. A line it can't parse is an error naming the line number rather than a silent skip. A proxy list is configuration, and quietly running on nine of ten addresses is worse than not starting at all.
Two details that are easy to get wrong
Logging a pool whose entries share a host
Credentials have to be stripped from logs. But sticky residential pools give every entry the same host:port and vary only the username, which is what encodes the session. Strip the credentials and a pool of a thousand prints as one line repeated, so a log showing a healthy rotation looks identical to a log showing every request pinned to one address.
Proxy.String() keeps the index for that reason:
pool#0 gw.example.com:5555
pool#1 gw.example.com:5555
pool#2 gw.example.com:5555
The BOM in your proxy list
PowerShell's Out-File and Set-Content write UTF-8 with a BOM by default. Those three invisible bytes attach to the first address only, so gw.example.com becomes a hostname that doesn't resolve, and it presents as that one proxy being dead. Not the sort of thing you'd think to check.
Load strips it, and there's a test for it.
Why the parser checks field count first
A password can legitimately contain an @. So host:port:user:p@ss is checked by field count before anything searches for an @, otherwise it reads as user:pass@host:port and yields a URL pointing at a host that was never in your file.
Field count is unambiguous where a character search isn't. There's a test on this one, because it's the sort of thing a later refactor simplifies straight back into a bug.
What it doesn't do
- No health checking. Use your own traffic.
- No per-request retry. That's your control flow, not the pool's.
- No SOCKS dialer.
Proxy.Transport()handssocks5://URLs tonet/http, which supports them. Anything more exotic is yours to build fromProxy.URL. - No rotation strategies beyond round robin. If you want one address pinned per long-lived session, use
Pool.All()and index it yourself. That's a different strategy, and stacking it onNext()gets you neither.
License
MIT
Documentation
¶
Overview ¶
Package proxypool hands out proxies round-robin and sidelines the ones that got blocked.
It solves the small set of problems that show up every time you run more than one proxy, and that are individually trivial and collectively tedious:
- providers emit at least four different line formats, and none of them is the URL an HTTP client wants
- a blocked address should stop being handed out for a while, then come back on its own
- credentials must never reach a log
- when every address is cooling down, the caller needs to know that rather than be handed one silently
It is deliberately small: no health checks, no scoring, no background goroutines. A Pool is a picker with a cooldown, and whether an address is healthy is something only the caller's own traffic can answer.
All methods are safe for concurrent use.
Example ¶
The usual loop: take an address, use it, sideline it when the far end complains, and let it come back on its own.
Note what the log lines look like. All three entries share one host:port and differ only in the username, which is how sticky residential pools work, so the index is the only thing that tells them apart once the credentials are stripped.
package main
import (
"fmt"
"time"
"github.com/Mootjelh/proxypool"
)
func main() {
pool := proxypool.New([]string{
"http://sess-a:pw@gw.example.com:5555",
"http://sess-b:pw@gw.example.com:5555",
"http://sess-c:pw@gw.example.com:5555",
})
for i := 0; i < 4; i++ {
p, healthy := pool.Next()
if !healthy {
fmt.Println("every address is cooling down; soonest back:", p)
break
}
fmt.Println("request", i, "via", p)
if i == 1 {
// Suppose that one came back 429.
pool.Block(p, time.Hour)
}
}
fmt.Printf("healthy: %d of %d\n", pool.Healthy(), pool.Len())
}
Output: request 0 via pool#0 gw.example.com:5555 request 1 via pool#1 gw.example.com:5555 request 2 via pool#2 gw.example.com:5555 request 3 via pool#0 gw.example.com:5555 healthy: 2 of 3
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Mask ¶
Mask strips credentials and scheme from a proxy URL so logins never reach a log. It is what Proxy.String uses.
Prefer logging the Proxy itself: on a pool whose entries share a host, a masked URL alone cannot tell them apart.
func Normalize ¶
Normalize converts the common proxy list formats into a URL an HTTP client accepts:
host:port -> http://host:port host:port:user:pass -> http://user:pass@host:port user:pass@host:port -> http://user:pass@host:port scheme://... -> unchanged
The third form is what most residential providers emit by default, and it is the one no HTTP client takes.
Example ¶
Normalize accepts the formats providers actually hand out, none of which is the URL an HTTP client wants.
package main
import (
"fmt"
"github.com/Mootjelh/proxypool"
)
func main() {
for _, raw := range []string{
"1.2.3.4:8080",
"gw.example.com:7000:user:pass",
"user:pass@1.2.3.4:8080",
"socks5://user:pass@host:1080",
} {
u, err := proxypool.Normalize(raw)
if err != nil {
fmt.Println("error:", err)
continue
}
fmt.Println(u)
}
}
Output: http://1.2.3.4:8080 http://user:pass@gw.example.com:7000 http://user:pass@1.2.3.4:8080 socks5://user:pass@host:1080
Types ¶
type Pool ¶
type Pool struct {
// contains filtered or unexported fields
}
Pool hands out proxies round-robin and sidelines ones that got blocked. The zero Pool is not usable; build one with New or Load.
func Load ¶
Load reads a proxy list from disk, one entry per line, and normalises each one. Blank lines and lines starting with # are ignored.
An unparseable line is an error naming the line number, rather than a silent skip: a proxy list is configuration, and quietly running on nine of ten addresses is worse than not starting.
func New ¶
New builds a pool from already-normalised proxy URLs.
An empty slice, or a slice containing one empty string, means "connect directly", so a caller that has no proxies configured can use the same code path as one that does, rather than branching everywhere.
func (*Pool) All ¶
All returns every proxy in the pool, in file order, regardless of cooldown.
Use it when you want the list rather than the picker, for instance to pin one address per long-lived session, which is a different rotation strategy from this pool's per-request round robin and should not be built on top of Pool.Next.
func (*Pool) Block ¶
Block sidelines a proxy for the given duration after it was throttled.
A shorter Block never shortens a cooldown already in place. Two failures on one address should not leave it less penalised than one.
func (*Pool) Next ¶
Next returns the proxy to use for the next request.
The bool reports whether it is currently healthy. When every proxy is cooling down it returns the one that recovers soonest, with false. The caller is better placed than the pool to decide between waiting, proceeding anyway and giving up.
type Proxy ¶
type Proxy struct {
// URL is the full proxy URL, credentials included. Pass it to your HTTP
// client, and do not log it. Use the Proxy itself for that.
URL string
// Index is the address's stable position in the pool, counting from 0.
Index int
}
Proxy is one address from the pool.
func (Proxy) String ¶
String renders the proxy for logs: credentials stripped, index kept.
The index is not decoration. Rotating residential pools typically give every entry the SAME host:port and vary only the username, which encodes the sticky session. Strip the credentials, which you must, and a pool of a thousand addresses prints as one, so a log that says the rotation is working looks exactly like a log that says it is not.
func (Proxy) Transport ¶
Transport returns an *http.Transport routed through this proxy, or a plain one when the entry is direct.
Example ¶
Transport is the bridge to net/http. Build a fresh client per address rather than swapping the proxy on a shared transport: a transport pools connections, and a pooled connection is already through the old proxy.
package main
import (
"fmt"
"net/http"
"time"
"github.com/Mootjelh/proxypool"
)
func main() {
pool := proxypool.New([]string{"http://user:pw@gw.example.com:5555"})
p, _ := pool.Next()
transport, err := p.Transport()
if err != nil {
fmt.Println(err)
return
}
client := &http.Client{Transport: transport, Timeout: 20 * time.Second}
_ = client
fmt.Println("routing through", p)
}
Output: routing through pool#0 gw.example.com:5555
Source Files