Help us shape the future of Go! Take the 2026 Go Developer Survey

unblock

package module
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 26, 2026 License: MIT Imports: 16 Imported by: 0

README

unblock

A Go HTTP client that looks like a browser at the network layer, and knows when it has been blocked.

c := unblock.New(unblock.Options{})
resp, err := c.Get(ctx, "https://example.com")
if b := resp.Blocked(); b != nil {
    fmt.Println(b.Kind, b.Hint) // CF_CHALLENGE / CF_BLOCK_1020 / WAF_GENERIC / ...
}

Three things are handled, and one deliberately is not.

1. TLS/JA3 and HTTP/2 fingerprints

net/http, requests, and httpx all share a recognisable fingerprint. On an HTTPS connection the client sends a TLS ClientHello carrying cipher suites, extension order, elliptic curves, and ALPN — WAFs hash that into JA3/JA4. On top of it sits the HTTP/2 fingerprint: SETTINGS values, WINDOW_UPDATE, pseudo-header order.

unblock uses bogdanfinn/tls-client (ClientHello via uTLS, h2 via fhttp) so the fingerprint matches a real Chrome, Safari, or Firefox:

c := unblock.New(unblock.Options{
    Profile: "chrome_133", // empty = newest supported Chrome profile
})

Conceptually the same as curl_cffi in Python, without a separate process.

2. Headers follow the fingerprint

tls-client fingerprints the handshake and the HTTP/2 frames, but it leaves the headers alone. A bare client therefore sends Go-http-client/2.0 with a Chrome TLS fingerprint — a mismatch any WAF will flag, and the kind of thing you only notice by looking at what the server actually receives.

unblock derives a consistent header set from the profile:

Header Source
user-agent the profile's browser and version
sec-ch-ua, sec-ch-ua-mobile, sec-ch-ua-platform same version, Chromium-family only
accept, accept-language, accept-encoding navigation defaults
sec-fetch-dest/mode/site/user, upgrade-insecure-requests navigation defaults

So no manual user-agent. unblock.DefaultHeaders("chrome_133") is exported if you need the set. Override through Options.Headers only when the value stays consistent with the profile — accept-language is safe; a hand-written user-agent that disagrees with sec-ch-ua and the TLS fingerprint is worse than sending nothing at all.

3. Automatic HTTP/1.1 fallback

Some servers reject HTTP/2 negotiation with RST_STREAM (stream ID 1; INTERNAL_ERROR) even though TLS succeeded. unblock detects that error shape and retries once over HTTP/1.1 with the same TLS fingerprint.

The first attempt only gets half the time budget, so the retry is not starved by a hanging first try. Response.Attempts and Response.HTTP1 report which path was used.

resp, _ := c.Get(ctx, url)
fmt.Println(resp.Attempts) // 1, or 2 when the fallback was used

Force HTTP/1.1 always with Options.ForceHTTP1.

4. HTTP/3 protocol racing

Rules that block the HTTP/2 fingerprint usually do not apply to HTTP/3, which has its own fingerprint surface. ProtocolRacing starts QUIC and TCP in parallel and keeps whichever connects first, then reuses it per host:

c := unblock.New(unblock.Options{ProtocolRacing: true})

Measured on a Cloudflare-protected page where every recent profile was blocked over HTTP/2 (403), protocol racing returned 200 over HTTP/3 — with chrome_133 and with chrome_150. It is the single strongest lever here: stronger than changing profiles, and stronger than forcing HTTP/1.1.

Needs UDP egress. If QUIC is blocked, the client falls back to TCP; Response.Proto reports what actually answered (HTTP/3.0, HTTP/2.0, HTTP/1.1).

On Linux, quic-go warns about a small UDP receive buffer. Raise it for stable throughput:

sysctl -w net.core.rmem_max=7500000 net.core.wmem_max=7500000
HTTP/3 cannot travel through a proxy

HTTP CONNECT tunnels TCP. QUIC runs on UDP, so a racing rung with ProxyURL set does not degrade to h2 over the proxy — it connects directly and leaves your real exit IP exposed.

unblock therefore ignores ProtocolRacing when a proxy is configured, and a ladder records such rungs as skipped instead of running them.

Need h3 and an egress? The tunnel has to carry UDP: WireGuard, SOCKS5 UDP ASSOCIATE, or MASQUE (CONNECT-UDP). An HTTP proxy cannot do it, no matter how it is configured.

4. The ladder: one call, several attempts

A single fetch is a gamble on one fingerprint. Ladder walks an ordered list of attempts and stops at the first rung that gets an answer:

l := unblock.NewLadder(unblock.Options{Timeout: 20 * time.Second}).
    WithMaxAttempts(4).
    WithCooldown(unblock.NewCooldown(6 * time.Hour))

resp, err := l.Get(ctx, url)
resp, err = l.Post(ctx, apiURL, "application/json", body) // read-only APIs only

for _, a := range l.Last() {          // what each rung did
    fmt.Println(a.String(), a.Proto, a.HTTPStatus, a.BlockKind)
}

Default order (DefaultAttempts): chrome_133 over HTTP/2 → the same profile racing HTTP/3 → the same profile over HTTP/1.1 → firefox_123 over HTTP/1.1.

Rules that make it predictable:

  • A WAF block means "try the next rung".
  • Any answer from the origin — 200, 404, even 500 — ends the walk. Changing fingerprints cannot fix a page that does not exist.
  • All rungs blocked → *BlockedError (wraps ErrBlocked) carrying the last block and the full trail. The ladder never invents a verdict.
  • WithCooldown parks a host after a block; the next call returns *CooldownError instead of hammering.
  • Winner() returns the fetcher that worked, so follow-up requests reuse the same fingerprint.
  • WithFetcherFactory swaps how a rung is served (tests). For Post, the fetcher must also implement Poster.
  • Post walks the same rungs and shares the cooldown. The next rung resends the body after a block, so use it only for requests that are safe to repeat.

Cost: a healthy site costs 1 request; a WAF-protected one usually 2 (HTTP/3 wins); a network without UDP needs 3.

6. Block classification

Four different situations hide behind HTTP 403, and each needs a different action. Classify tells them apart from headers and body, with no extra requests:

Kind Marker What to do
CF_CHALLENGE cf-mitigated: challenge, _cf_chl_opt, Just a moment needs a browser that executes JS
CF_BLOCK_1010 Error 1010 browser signature banned: use a TLS profile, not a spoofed UA
CF_BLOCK_1009 Error 1009 country block: needs a local proxy
CF_BLOCK_1015 Error 1015 rate limited: add a cooldown
CF_BLOCK_1020 Error 1020 firewall rule rejected the ASN/datacenter
WAF_GENERIC x-sucuri-id, x-datadome, x-akamai-request-id, x-iinfo third-party WAF

Note that a block is a statement about this request: same URL, same profile, different protocol can answer 200. Before concluding a host is unreachable, try ProtocolRacing (HTTP/3) and ForceHTTP1.

b := unblock.Classify(resp)
if b != nil {
    log.Printf("blocked: %s (ray %s) — %s", b.Kind, b.RayID, b.Hint)
}
if b.Challenge() {
    // the only category a browser can resolve
}

A plain 403 from the origin (a normal "forbidden" page, a login wall) is intentionally not treated as a block — that is not an anti-bot concern.

Per-host cooldown

Cloudflare blocks are sticky: hammering the same host within a short window tends to escalate the block. Cooldown buys distance:

cd := unblock.NewCooldown(6 * time.Hour)
host := unblock.HostOf(url)

if !cd.Allow(host) {
    return fmt.Errorf("host %s is cooling down for %s", host, cd.Remaining(host))
}
resp, err := c.Get(ctx, url)
if b := resp.Blocked(); b != nil {
    cd.Mark(host)
}

Fetching URLs your users typed

A server that fetches a user-supplied URL must not let it reach internal addresses (SSRF). Pass a net.Dialer whose Control hook refuses them; it runs after DNS resolution on every TCP connection, redirects included:

c := unblock.New(unblock.Options{
    Dialer: &net.Dialer{Control: func(_, address string, _ syscall.RawConn) error {
        host, _, _ := net.SplitHostPort(address)
        ip := net.ParseIP(host)
        if ip == nil || ip.IsPrivate() || ip.IsLoopback() || ip.IsLinkLocalUnicast() ||
            ip.IsLinkLocalMulticast() || ip.IsMulticast() || ip.IsUnspecified() {
            return fmt.Errorf("%s is not public", host)
        }
        return nil
    }},
})

QUIC does not go through the dialer, so a Dialer disables HTTP/3: the client ignores ProtocolRacing, and a ladder records racing rungs as skipped.

With ProxyURL set, Control sees only the proxy's address. The proxy resolves and connects to the target itself, so the guard has to live in the proxy.

What this does NOT do

It does not execute JavaScript. A managed Cloudflare challenge (Turnstile, the "Checking your browser" page that computes a token) still cannot be bypassed — fingerprints only win against rules that are purely TLS/h2 based.

The package tells you that this is what happened (CF_CHALLENGE) instead of pretending to solve it. For that you need a real browser: FlareSolverr, go-rod, or the end user's own browser.

Also out of scope: IP reputation and behaviour. A datacenter IP is often blocked even with a perfect fingerprint — use Options.ProxyURL to leave that IP behind.

Install

go get github.com/nyelonong/unblock

Requires Go 1.25+ (tls-client v1.16.0 or newer).

API

func New(opts Options) *Client
func (c *Client) Get(ctx context.Context, url string) (*Response, error)
func (c *Client) Head(ctx context.Context, url string) (*Response, error)
func (c *Client) Do(ctx context.Context, method, url string) (*Response, error)
func (c *Client) Post(ctx context.Context, url, contentType string, body []byte) (*Response, error)
func (c *Client) CloseIdleConnections()

func DefaultHeaders(profile string) map[string]string
func Classify(r *Response) *Block
func HostOf(rawURL string) string
func NewCooldown(window time.Duration) *Cooldown

func NewLadder(base Options, attempts ...Attempt) *Ladder
func DefaultAttempts() []Attempt
func (l *Ladder) WithMaxAttempts(n int) *Ladder
func (l *Ladder) WithCooldown(c *Cooldown) *Ladder
func (l *Ladder) WithFetcherFactory(f func(Attempt) Fetcher) *Ladder
func (l *Ladder) Get(ctx context.Context, url string) (*Response, error)
func (l *Ladder) Head(ctx context.Context, url string) (*Response, error)
func (l *Ladder) Post(ctx context.Context, url, contentType string, body []byte) (*Response, error)
func (l *Ladder) Winner() Fetcher
func (l *Ladder) Last() []AttemptResult
func (l *Ladder) CloseIdleConnections()

Options: Profile, ProxyURL, Timeout, MaxBody, Headers, ForceHTTP1, ProtocolRacing, InsecureSkipVerify, Dialer.

Response: StatusCode, Header, Body, FinalURL, Proto, Attempts, HTTP1, plus Get(header) and Blocked().

Test

go test ./...

Covers a real httptest server, body limits, custom headers, ForceHTTP1, 9 block-classification variants, and cooldown behaviour.

License

MIT — see LICENSE.

Documentation

Overview

Package unblock is an HTTP client for Go that looks like a browser at the network layer (Chrome-like TLS ClientHello and HTTP/2 fingerprint) and knows when it has been blocked.

Three things are handled:

  1. TLS/JA3 and HTTP/2 fingerprints — via bogdanfinn/tls-client, so WAF rules based on fingerprints do not reject the request.
  2. Automatic HTTP/1.1 fallback when a server rejects HTTP/2 (RST_STREAM), reusing the same TLS fingerprint.
  3. Block classification — Cloudflare challenges, errors 1009/1010/1015/1020, and third-party WAFs are told apart from one another, because each needs a different follow-up action.

This package does NOT execute JavaScript. A Cloudflare challenge that requires JS cannot be bypassed here; Classify only reports that this is what happened.

Index

Examples

Constants

View Source 
const DefaultCooldownWindow = 6 * time.Hour

DefaultCooldownWindow is used when NewCooldown receives a zero duration.

View Source 
const DefaultMaxBody = 5 << 20

DefaultMaxBody caps how many body bytes are read (5 MiB).

View Source 
const DefaultTimeout = 20 * time.Second

DefaultTimeout is the per-attempt deadline when Options.Timeout is zero.

Variables

View Source 
var ErrBlocked = errors.New("unblock: every attempt was blocked")

ErrBlocked is returned when every rung was blocked by a WAF.

View Source 
var ErrNoBody = errors.New("unblock: empty body")

ErrNoBody is returned when a response carries no usable payload.

Functions

func DefaultHeaders added in v0.4.0 

func DefaultHeaders(profile string) map[string]string

DefaultHeaders returns a browser-consistent header set for a TLS profile.

tls-client fingerprints the handshake and the HTTP/2 frames but leaves the headers alone, so without this the wire says "Go-http-client/2.0" while the fingerprint claims Chrome — a mismatch any WAF will flag. The browser name and version are read from the profile (GetClientHelloId), never guessed from the profile string.

accept-encoding is deliberately absent: the transport adds its own and decompresses the body. Setting it here leaves the response compressed, which silently shrinks the HTML and breaks substring checks.

func HostOf 

func HostOf(raw string) string

HostOf returns the host (without port) of a URL, or the input unchanged when it cannot be parsed.

Types

type Attempt added in v0.3.0 

type Attempt struct {
	Profile        string `json:"profile"`
	ForceHTTP1     bool   `json:"http1,omitempty"`
	ProtocolRacing bool   `json:"http3,omitempty"`
}

Attempt is one fetch strategy: a TLS profile plus the protocol controls that matter when a WAF rejects a fingerprint.

func DefaultAttempts added in v0.3.0 

func DefaultAttempts() []Attempt

DefaultAttempts is the measured starting order.

Basis: 17 profiles, one WAF-protected page, single GET each.

  • HTTP/2 was rejected for every recent profile.
  • Racing HTTP/3 answered 200 for chrome_133 and chrome_150 while HTTP/2 was blocked for both, so the protocol is a stronger lever than the profile.
  • HTTP/1.1 carried chrome_133, chrome_144, firefox_123, and brave_146.
  • chrome_144 and chrome_150 share a JA3N hash yet behaved differently, so the HTTP/2 settings fingerprint is part of what the WAF sees.

Order: mainstream Chrome over h2, the same profile racing h3, the same profile over h1.1, then another engine for when Chrome-shaped traffic is blocked wholesale. Ties break on how common the browser version is in the wild, not on release date.

func (Attempt) String added in v0.3.0 

func (a Attempt) String() string

String renders one attempt for logs: profile plus protocol.

type AttemptResult added in v0.3.0 

type AttemptResult struct {
	Attempt
	HTTPStatus int    `json:"http_status,omitempty"`
	Proto      string `json:"proto,omitempty"`
	BlockKind  string `json:"block_kind,omitempty"`
	Error      string `json:"error,omitempty"`
	OK         bool   `json:"ok,omitempty"`
}

AttemptResult records what one rung did.

type Block 

type Block struct {
	// Kind is the category: CF_CHALLENGE, CF_BLOCK_1009, CF_BLOCK_1010,
	// CF_BLOCK_1015, CF_BLOCK_1020, CF_BLOCK_GENERIC, WAF_GENERIC.
	Kind string `json:"kind"`
	// Code is the Cloudflare error code when present (e.g. "1020").
	Code string `json:"code,omitempty"`
	// RayID helps when contacting WAF support or the site owner.
	RayID string `json:"ray_id,omitempty"`
	// Server is the value of the Server header.
	Server string `json:"server,omitempty"`
	// Hint suggests the next action.
	Hint string `json:"hint,omitempty"`
}

Block describes a single blocking event.

func Classify 

func Classify(r *Response) *Block

Classify inspects a response and reports whether it is a WAF/anti-bot block. It returns nil when the response is not a block.

Four different situations hide behind HTTP 403, and each needs a different response — never treat them as one case.

Example

Tell block types apart. 

package main

import (
	"fmt"
	"net/http"

	"github.com/nyelonong/unblock"
)

func main() {
	h := http.Header{}
	h.Set("Server", "cloudflare")
	h.Set("Cf-Ray", "8a1b2c3d-JKT")

	resp := &unblock.Response{
		StatusCode: 403,
		Header:     h,
		Body:       []byte("<title>Attention Required!</title><h2>Error 1020</h2>"),
	}

	b := unblock.Classify(resp)
	fmt.Println(b.Kind, b.Code, b.Challenge())
	fmt.Println(b.Hint)

}

Output:
CF_BLOCK_1020 1020 false
firewall rule rejected the ASN/datacenter: needs a residential proxy

func (*Block) Challenge 

func (b *Block) Challenge() bool

Challenge reports whether this block can be resolved by a browser that executes JavaScript (FlareSolverr, go-rod, or the end user's browser).

type BlockedError added in v0.3.0 

type BlockedError struct {
	Block    *Block
	Attempts []AttemptResult
}

BlockedError carries the last block plus the full trail.

func (*BlockedError) Error added in v0.3.0 

func (e *BlockedError) Error() string

func (*BlockedError) Unwrap added in v0.3.0 

func (e *BlockedError) Unwrap() error

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client is a fetcher with a browser fingerprint.

Example (ForceHTTP1)

Combine both levers: force HTTP/1.1 when QUIC is unavailable in the network and HTTP/2 is blocked. The TLS fingerprint is unchanged. 

package main

import (
	"context"
	"fmt"
	"net/http"
	"net/http/httptest"
	"time"

	"github.com/nyelonong/unblock"
)

func main() {
	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		_, _ = w.Write([]byte("ok"))
	}))
	defer srv.Close()

	c := unblock.New(unblock.Options{Timeout: 5 * time.Second, ForceHTTP1: true})
	resp, err := c.Get(context.Background(), srv.URL)
	if err != nil {
		fmt.Println("error:", err)
		return
	}

	fmt.Println(resp.StatusCode, resp.HTTP1)

}

Output:
200 true
Example (ProtocolRacing)

Race HTTP/3 against HTTP/2 and read the protocol that actually answered.

Many anti-bot rules are scoped to the HTTP/2 fingerprint. Racing QUIC in parallel with TCP often gets an answer where HTTP/2 alone is blocked, and the same TLS profile is reused either way.

Response.Proto tells you which one won, so a "blocked" result is never ambiguous about the transport that produced it. 

package main

import (
	"context"
	"fmt"
	"net/http"
	"net/http/httptest"
	"time"

	"github.com/nyelonong/unblock"
)

func main() {
	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		_, _ = w.Write([]byte("<html>hello</html>"))
	}))
	defer srv.Close()

	c := unblock.New(unblock.Options{
		Profile:        "chrome_133",
		Timeout:        5 * time.Second,
		ProtocolRacing: true,
	})

	resp, err := c.Get(context.Background(), srv.URL)
	if err != nil {
		fmt.Println("error:", err)
		return
	}

	fmt.Println(resp.StatusCode, resp.Proto)

}

Output:
200 HTTP/1.1

func New 

func New(opts Options) *Client

New builds a Client. An unusable configuration does not make New fail: the client falls back to plain net/http so callers never have to handle an error on the construction path.

func (*Client) CloseIdleConnections added in v0.8.0 

func (c *Client) CloseIdleConnections()

CloseIdleConnections closes the keep-alive connections the client holds. Call it when the client is done, so a short-lived client does not keep sockets open until the idle timeout.

func (*Client) Do 

func (c *Client) Do(ctx context.Context, method, url string) (*Response, error)

Do performs the request and returns the full body.

When the first attempt fails because of an HTTP/2 problem (RST_STREAM, INTERNAL_ERROR), the request is retried once over HTTP/1.1 with the same TLS fingerprint. Half of the time budget is reserved for that second attempt.

func (*Client) Get 

func (c *Client) Get(ctx context.Context, url string) (*Response, error)

Get fetches a URL. Redirects are followed.

Example

Fetch a page with a browser fingerprint. 

package main

import (
	"context"
	"fmt"
	"net/http"
	"net/http/httptest"
	"time"

	"github.com/nyelonong/unblock"
)

func main() {
	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		_, _ = w.Write([]byte("<html>hello</html>"))
	}))
	defer srv.Close()

	c := unblock.New(unblock.Options{Profile: "chrome_133", Timeout: 5 * time.Second})

	resp, err := c.Get(context.Background(), srv.URL)
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	fmt.Println(resp.StatusCode, len(resp.Body) > 0, resp.Blocked() == nil)

}

Output:
200 true true

func (*Client) Head 

func (c *Client) Head(ctx context.Context, url string) (*Response, error)

Head sends a HEAD request. Some servers reject HEAD; callers that need a body should fall back to Get themselves.

func (*Client) Post added in v0.7.0 

func (c *Client) Post(ctx context.Context, url, contentType string, body []byte) (*Response, error)

Post sends a request with a body, which is what JSON endpoints behind a WAF need (for example YouTube's continuation calls). The fingerprint is the same as for Get.

type Cooldown 

type Cooldown struct {
	// contains filtered or unexported fields
}

Cooldown records hosts that just blocked us.

Cloudflare blocks are sticky: repeating a request to the same host within a short window tends to escalate the block (challenge becomes block). A cooldown buys distance before that host is attempted again.

Example

Use a per-host cooldown. 

package main

import (
	"fmt"
	"time"

	"github.com/nyelonong/unblock"
)

func main() {
	cd := unblock.NewCooldown(time.Hour)
	host := unblock.HostOf("https://example.com/page")

	fmt.Println(cd.Allow(host))
	cd.Mark(host)
	fmt.Println(cd.Allow(host))

}

Output:
true
false

func NewCooldown 

func NewCooldown(window time.Duration) *Cooldown

NewCooldown creates a Cooldown with the given window. Zero means DefaultCooldownWindow.

func (*Cooldown) Allow 

func (c *Cooldown) Allow(host string) bool

Allow reports whether the host may be attempted now.

func (*Cooldown) Mark 

func (c *Cooldown) Mark(host string)

Mark parks the host for the cooldown window.

func (*Cooldown) Remaining 

func (c *Cooldown) Remaining(host string) time.Duration

Remaining returns how long the host stays parked (0 when it is allowed).

func (*Cooldown) Reset 

func (c *Cooldown) Reset(host string)

Reset clears the cooldown marker for a host.

type CooldownError added in v0.3.0 

type CooldownError struct {
	Host      string
	Remaining time.Duration
}

CooldownError is returned when the host is still parked.

func (*CooldownError) Error added in v0.3.0 

func (e *CooldownError) Error() string

type Fetcher added in v0.3.0 

type Fetcher interface {
	Get(ctx context.Context, url string) (*Response, error)
	Head(ctx context.Context, url string) (*Response, error)
}

Fetcher is the surface the ladder needs. *Client satisfies it.

type Ladder added in v0.3.0 

type Ladder struct {
	// contains filtered or unexported fields
}

Ladder walks attempts in order and stops at the first answer that is not a WAF block. A block means "try the next rung"; any answer from the origin — 200, 404, even 500 — means the site responded, so the ladder stops.

func NewLadder added in v0.3.0 

func NewLadder(base Options, attempts ...Attempt) *Ladder

NewLadder builds a Ladder on top of base options. Attempts that carry an empty Profile inherit nothing from base: a ladder rung is a complete strategy, so set Profile explicitly, or start from DefaultAttempts.

func (*Ladder) CloseIdleConnections added in v0.8.0 

func (l *Ladder) CloseIdleConnections()

CloseIdleConnections closes the keep-alive connections of every rung's client. Call it when a short-lived ladder is done.

func (*Ladder) Get added in v0.3.0 

func (l *Ladder) Get(ctx context.Context, url string) (*Response, error)

Get fetches a URL, walking the ladder.

Example

One call walks the whole ladder and stops at the first rung that gets an answer: HTTP/2, then HTTP/3 racing, then HTTP/1.1, then another engine.

A block means "try the next rung". Any answer from the origin — 200, 404, even 500 — ends the walk, because changing fingerprints cannot fix a page that does not exist. 

package main

import (
	"context"
	"errors"
	"fmt"
	"time"

	"github.com/nyelonong/unblock"
)

func main() {
	l := unblock.NewLadder(unblock.Options{
		Timeout: 20 * time.Second,
		Headers: map[string]string{"accept-language": "en-US,en;q=0.9"},
	}).WithMaxAttempts(4)

	resp, err := l.Get(context.Background(), "https://example.com/")
	if err != nil {
		var blocked *unblock.BlockedError
		if errors.As(err, &blocked) {
			fmt.Println("every rung blocked, last kind:", blocked.Block.Kind)
			return
		}
		fmt.Println("request failed:", err)
		return
	}

	fmt.Println(resp.StatusCode, resp.Proto)
	for _, a := range l.Last() {
		fmt.Printf("  %-32s %-9s %d\n", a.String(), a.Proto, a.HTTPStatus)
	}
}

func (*Ladder) Head added in v0.3.0 

func (l *Ladder) Head(ctx context.Context, url string) (*Response, error)

Head sends a HEAD request, walking the ladder.

func (*Ladder) Last added in v0.3.0 

func (l *Ladder) Last() []AttemptResult

Last returns the trail of the most recent walk.

func (*Ladder) Post added in v0.9.0 

func (l *Ladder) Post(ctx context.Context, url, contentType string, body []byte) (*Response, error)

Post sends a request with a body, walking the ladder, with the same cooldown and trail as Get.

The next rung resends the same body after a WAF block. A block is answered by the WAF before the origin sees the request, but use Post only for requests that are safe to repeat, such as read-only JSON APIs: the ladder cannot tell whether a request that failed in transit reached the origin, and Client already resends over HTTP/1.1 after an HTTP/2 failure.

func (*Ladder) Winner added in v0.3.0 

func (l *Ladder) Winner() Fetcher

Winner returns the fetcher that produced the last successful response, so follow-up requests reuse the same fingerprint. For a follow-up POST, assert it to Poster; a *Client winner satisfies it.

func (*Ladder) WithCooldown added in v0.3.0 

func (l *Ladder) WithCooldown(c *Cooldown) *Ladder

WithCooldown parks a host after a block, so repeated checks do not escalate it.

Example

Park a host after a block so repeated checks do not escalate it. 

package main

import (
	"context"
	"errors"
	"fmt"
	"time"

	"github.com/nyelonong/unblock"
)

func main() {
	cd := unblock.NewCooldown(6 * time.Hour)

	l := unblock.NewLadder(unblock.Options{Timeout: 20 * time.Second}).
		WithCooldown(cd).
		WithMaxAttempts(3)

	if _, err := l.Get(context.Background(), "https://example.com/"); err != nil {
		var cooling *unblock.CooldownError
		if errors.As(err, &cooling) {
			fmt.Println("still parked for", cooling.Remaining)
		}
	}

	fmt.Println("cooling down:", !cd.Allow(unblock.HostOf("https://example.com/")))
}

func (*Ladder) WithFetcherFactory added in v0.3.0 

func (l *Ladder) WithFetcherFactory(f func(Attempt) Fetcher) *Ladder

WithFetcherFactory overrides how a rung is served. Intended for tests.

func (*Ladder) WithMaxAttempts added in v0.3.0 

func (l *Ladder) WithMaxAttempts(n int) *Ladder

WithMaxAttempts caps how many rungs are tried.

type Options 

type Options struct {
	// Profile names a TLS preset, e.g. "chrome_133" or "firefox_135".
	// Empty means the package default Chrome profile (the newest supported one).
	// See profiles.MappedTLSClients for the full list.
	Profile string

	// ProxyURL is optional: "http://127.0.0.1:8080" or "socks5://10.0.0.2:1080".
	ProxyURL string

	// Timeout per attempt. Zero means DefaultTimeout.
	Timeout time.Duration

	// MaxBody caps the number of body bytes read. Zero means DefaultMaxBody.
	MaxBody int64

	// Headers override the browser-consistent defaults derived from Profile
	// (user-agent, sec-ch-ua, accept, sec-fetch-*). Set one only when you know
	// the value stays consistent with the profile's fingerprint.
	Headers map[string]string

	// ForceHTTP1 disables HTTP/2 negotiation entirely. Useful for servers that
	// tarpit or consistently reject h2.
	ForceHTTP1 bool

	// ProtocolRacing races HTTP/3 (QUIC) and HTTP/2 (TCP) and uses whichever
	// connects first, similar to Chrome's Happy Eyeballs. The client then reuses
	// the winning protocol per host.
	//
	// This is the strongest lever against anti-bot rules that target the HTTP/2
	// fingerprint: measured on a Cloudflare-protected page where every recent
	// profile was blocked over HTTP/2, HTTP/3 answered 200 with both
	// chrome_133 and chrome_150. Requires UDP egress; if QUIC is blocked the
	// client falls back to TCP.
	ProtocolRacing bool

	// InsecureSkipVerify disables certificate verification. Only for internal
	// hosts with self-signed certificates and for tests; it does not affect the
	// fingerprint, only trust.
	InsecureSkipVerify bool

	// Dialer opens every TCP connection, so its Control hook sees each resolved
	// address, redirects included. Use it to refuse private networks when the URL
	// comes from a user. QUIC does not use it, so setting Dialer disables HTTP/3
	// and ProtocolRacing. With ProxyURL set, Control sees only the proxy's
	// address: the proxy resolves and connects to the target, so the guard no
	// longer applies to it.
	Dialer *net.Dialer
}

Options configures a Client.

type Poster added in v0.9.0 

type Poster interface {
	Post(ctx context.Context, url, contentType string, body []byte) (*Response, error)
}

Poster is the extra surface Ladder.Post needs from each rung. *Client satisfies it. It is separate from Fetcher so a fetcher that only reads still works everywhere else.

type Response 

type Response struct {
	StatusCode int
	Header     http.Header
	Body       []byte
	FinalURL   string
	// Proto is the negotiated protocol, e.g. "HTTP/2.0" or "HTTP/3.0".
	Proto string

	// Attempts is 1 for the first try, 2 when the HTTP/1.1 fallback was used.
	Attempts int
	// HTTP1 is true when the response came from the HTTP/1.1 fallback path.
	HTTP1 bool
}

Response is a fully-read response, truncated at MaxBody.

func (*Response) Blocked 

func (r *Response) Blocked() *Block

Blocked returns block details, or nil when the response is not a block.

func (*Response) Get 

func (r *Response) Get(name string) string

Get returns the value of a response header.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.