README
¶
unblock
A Go HTTP client that looks like a browser at the network layer, and knows when it has been blocked.
c := unblock.New(unblock.Options{})
resp, err := c.Get(ctx, "https://example.com")
if b := resp.Blocked(); b != nil {
fmt.Println(b.Kind, b.Hint) // CF_CHALLENGE / CF_BLOCK_1020 / WAF_GENERIC / ...
}
Three things are handled, and one deliberately is not.
1. TLS/JA3 and HTTP/2 fingerprints
net/http, requests, and httpx all share a recognisable fingerprint. On an HTTPS connection the client sends a TLS ClientHello carrying cipher suites, extension order, elliptic curves, and ALPN — WAFs hash that into JA3/JA4. On top of it sits the HTTP/2 fingerprint: SETTINGS values, WINDOW_UPDATE, pseudo-header order.
unblock uses bogdanfinn/tls-client (ClientHello via uTLS, h2 via fhttp) so the fingerprint matches a real Chrome, Safari, or Firefox:
c := unblock.New(unblock.Options{
Profile: "chrome_133", // empty = newest supported Chrome profile
})
Conceptually the same as curl_cffi in Python, without a separate process.
2. Automatic HTTP/1.1 fallback
Some servers reject HTTP/2 negotiation with RST_STREAM (stream ID 1; INTERNAL_ERROR) even though TLS succeeded. unblock detects that error shape and retries once over HTTP/1.1 with the same TLS fingerprint.
The first attempt only gets half the time budget, so the retry is not starved by a hanging first try. Response.Attempts and Response.HTTP1 report which path was used.
resp, _ := c.Get(ctx, url)
fmt.Println(resp.Attempts) // 1, or 2 when the fallback was used
Force HTTP/1.1 always with Options.ForceHTTP1.
3. Block classification
Four different situations hide behind HTTP 403, and each needs a different action. Classify tells them apart from headers and body, with no extra requests:
Kind |
Marker | What to do |
|---|---|---|
CF_CHALLENGE |
cf-mitigated: challenge, _cf_chl_opt, Just a moment |
needs a browser that executes JS |
CF_BLOCK_1010 |
Error 1010 |
browser signature banned: use a TLS profile, not a spoofed UA |
CF_BLOCK_1009 |
Error 1009 |
country block: needs a local proxy |
CF_BLOCK_1015 |
Error 1015 |
rate limited: add a cooldown |
CF_BLOCK_1020 |
Error 1020 |
firewall rule rejected the ASN/datacenter |
WAF_GENERIC |
x-sucuri-id, x-datadome, x-akamai-request-id, x-iinfo |
third-party WAF |
b := unblock.Classify(resp)
if b != nil {
log.Printf("blocked: %s (ray %s) — %s", b.Kind, b.RayID, b.Hint)
}
if b.Challenge() {
// the only category a browser can resolve
}
A plain 403 from the origin (a normal "forbidden" page, a login wall) is intentionally not treated as a block — that is not an anti-bot concern.
Per-host cooldown
Cloudflare blocks are sticky: hammering the same host within a short window tends to escalate the block. Cooldown buys distance:
cd := unblock.NewCooldown(6 * time.Hour)
host := unblock.HostOf(url)
if !cd.Allow(host) {
return fmt.Errorf("host %s is cooling down for %s", host, cd.Remaining(host))
}
resp, err := c.Get(ctx, url)
if b := resp.Blocked(); b != nil {
cd.Mark(host)
}
What this does NOT do
It does not execute JavaScript. A managed Cloudflare challenge (Turnstile, the "Checking your browser" page that computes a token) still cannot be bypassed — fingerprints only win against rules that are purely TLS/h2 based.
The package tells you that this is what happened (CF_CHALLENGE) instead of pretending to solve it. For that you need a real browser: FlareSolverr, go-rod, or the end user's own browser.
Also out of scope: IP reputation and behaviour. A datacenter IP is often blocked even with a perfect fingerprint — use Options.ProxyURL to leave that IP behind.
Install
go get github.com/nyelonong/unblock
Requires Go 1.27+ (tracking tls-client).
API
func New(opts Options) *Client
func (c *Client) Get(ctx context.Context, url string) (*Response, error)
func (c *Client) Head(ctx context.Context, url string) (*Response, error)
func (c *Client) Do(ctx context.Context, method, url string) (*Response, error)
func Classify(r *Response) *Block
func HostOf(rawURL string) string
func NewCooldown(window time.Duration) *Cooldown
Options: Profile, ProxyURL, Timeout, MaxBody, Headers, ForceHTTP1.
Response: StatusCode, Header, Body, FinalURL, Attempts, HTTP1, plus Get(header) and Blocked().
Test
go test ./...
Covers a real httptest server, body limits, custom headers, ForceHTTP1, 9 block-classification variants, and cooldown behaviour.
License
MIT — see LICENSE.
Documentation
¶
Overview ¶
Package unblock is an HTTP client for Go that looks like a browser at the network layer (Chrome-like TLS ClientHello and HTTP/2 fingerprint) and knows when it has been blocked.
Three things are handled:
- TLS/JA3 and HTTP/2 fingerprints — via bogdanfinn/tls-client, so WAF rules based on fingerprints do not reject the request.
- Automatic HTTP/1.1 fallback when a server rejects HTTP/2 (RST_STREAM), reusing the same TLS fingerprint.
- Block classification — Cloudflare challenges, errors 1009/1010/1015/1020, and third-party WAFs are told apart from one another, because each needs a different follow-up action.
This package does NOT execute JavaScript. A Cloudflare challenge that requires JS cannot be bypassed here; Classify only reports that this is what happened.
Index ¶
Examples ¶
Constants ¶
const DefaultCooldownWindow = 6 * time.Hour
DefaultCooldownWindow is used when NewCooldown receives a zero duration.
const DefaultMaxBody = 5 << 20
DefaultMaxBody caps how many body bytes are read (5 MiB).
const DefaultTimeout = 20 * time.Second
DefaultTimeout is the per-attempt deadline when Options.Timeout is zero.
Variables ¶
var ErrNoBody = errors.New("unblock: empty body")
ErrNoBody is returned when a response carries no usable payload.
Functions ¶
Types ¶
type Block ¶
type Block struct {
// Kind is the category: CF_CHALLENGE, CF_BLOCK_1009, CF_BLOCK_1010,
// CF_BLOCK_1015, CF_BLOCK_1020, CF_BLOCK_GENERIC, WAF_GENERIC.
Kind string `json:"kind"`
// Code is the Cloudflare error code when present (e.g. "1020").
Code string `json:"code,omitempty"`
// RayID helps when contacting WAF support or the site owner.
RayID string `json:"ray_id,omitempty"`
// Server is the value of the Server header.
Server string `json:"server,omitempty"`
// Hint suggests the next action.
Hint string `json:"hint,omitempty"`
}
Block describes a single blocking event.
func Classify ¶
Classify inspects a response and reports whether it is a WAF/anti-bot block. It returns nil when the response is not a block.
Four different situations hide behind HTTP 403, and each needs a different response — never treat them as one case.
Example ¶
Tell block types apart.
package main
import (
"fmt"
"net/http"
"github.com/nyelonong/unblock"
)
func main() {
h := http.Header{}
h.Set("Server", "cloudflare")
h.Set("Cf-Ray", "8a1b2c3d-JKT")
resp := &unblock.Response{
StatusCode: 403,
Header: h,
Body: []byte("<title>Attention Required!</title><h2>Error 1020</h2>"),
}
b := unblock.Classify(resp)
fmt.Println(b.Kind, b.Code, b.Challenge())
fmt.Println(b.Hint)
}
Output: CF_BLOCK_1020 1020 false firewall rule rejected the ASN/datacenter: needs a residential proxy
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client is a fetcher with a browser fingerprint.
func New ¶
New builds a Client. An unusable configuration does not make New fail: the client falls back to plain net/http so callers never have to handle an error on the construction path.
func (*Client) Do ¶
Do performs the request and returns the full body.
When the first attempt fails because of an HTTP/2 problem (RST_STREAM, INTERNAL_ERROR), the request is retried once over HTTP/1.1 with the same TLS fingerprint. Half of the time budget is reserved for that second attempt.
func (*Client) Get ¶
Get fetches a URL. Redirects are followed.
Example ¶
Fetch a page with a browser fingerprint.
package main
import (
"context"
"fmt"
"net/http"
"net/http/httptest"
"time"
"github.com/nyelonong/unblock"
)
func main() {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte("<html>hello</html>"))
}))
defer srv.Close()
c := unblock.New(unblock.Options{Profile: "chrome_133", Timeout: 5 * time.Second})
resp, err := c.Get(context.Background(), srv.URL)
if err != nil {
fmt.Println("error:", err)
return
}
fmt.Println(resp.StatusCode, len(resp.Body) > 0, resp.Blocked() == nil)
}
Output: 200 true true
type Cooldown ¶
type Cooldown struct {
// contains filtered or unexported fields
}
Cooldown records hosts that just blocked us.
Cloudflare blocks are sticky: repeating a request to the same host within a short window tends to escalate the block (challenge becomes block). A cooldown buys distance before that host is attempted again.
Example ¶
Use a per-host cooldown.
package main
import (
"fmt"
"time"
"github.com/nyelonong/unblock"
)
func main() {
cd := unblock.NewCooldown(time.Hour)
host := unblock.HostOf("https://example.com/page")
fmt.Println(cd.Allow(host))
cd.Mark(host)
fmt.Println(cd.Allow(host))
}
Output: true false
func NewCooldown ¶
NewCooldown creates a Cooldown with the given window. Zero means DefaultCooldownWindow.
type Options ¶
type Options struct {
// Profile names a TLS preset, e.g. "chrome_133" or "firefox_135".
// Empty means the package default Chrome profile (the newest supported one).
// See profiles.MappedTLSClients for the full list.
Profile string
// ProxyURL is optional: "http://127.0.0.1:8080" or "socks5://10.0.0.2:1080".
ProxyURL string
// Timeout per attempt. Zero means DefaultTimeout.
Timeout time.Duration
// MaxBody caps the number of body bytes read. Zero means DefaultMaxBody.
MaxBody int64
// Headers are added to every request. Values here override whatever the TLS
// profile sets (user-agent, accept, and so on).
Headers map[string]string
// ForceHTTP1 disables HTTP/2 negotiation entirely. Useful for servers that
// tarpit or consistently reject h2.
ForceHTTP1 bool
}
Options configures a Client.
type Response ¶
type Response struct {
StatusCode int
Header http.Header
Body []byte
FinalURL string
// Attempts is 1 for the first try, 2 when the HTTP/1.1 fallback was used.
Attempts int
// HTTP1 is true when the response came from the HTTP/1.1 fallback path.
HTTP1 bool
}
Response is a fully-read response, truncated at MaxBody.
Source Files