cloudfetch

README

cloudfetch

Go HTTP client that bypasses Cloudflare bot detection by mimicking a real browser's TLS fingerprint.

How it works

Cloudflare identifies bots partly by their TLS Client Hello fingerprint (JA3/JA4). Go's default net/http has a distinctive fingerprint that gets blocked. cloudfetch uses uTLS to reproduce a real browser's TLS handshake, sends that browser's headers, and keeps a cookie jar.

• TLS fingerprint — Chrome 133, Firefox 120, Safari 16 or Edge 85 via uTLS
• Matching headers — User-Agent, Sec-CH-UA, Sec-Fetch-*, etc. for the same browser and version as the TLS fingerprint
• JS challenge solver — solves the classic IUAM ("Checking your browser") challenge using goja (pure Go JS interpreter)
• Auto-retry — detects challenge pages and retries after solving (up to 3 attempts)
• HTTP/2 — automatic ALPN negotiation and h2 support
• Compression — transparent brotli/gzip/deflate/zstd decompression
• Cookie jar — maintains cookies across requests
• Proxy — HTTP CONNECT proxies, with optional basic auth
• Browser handoff — optionally passes challenges it can't solve to headless Chrome once, then reuses the clearance cookie (see Browser handoff)

Limitations

• Header order and HTTP/2 settings come from Go's net/http, not the browser. Sites that fingerprint those can still tell cloudfetch apart from a real browser.
• Newer JS challenges (challenge-platform) are detected, but without a solver cloudfetch only submits the page's form fields without running its script, so they usually fail.
• Managed and Interactive Challenges (Turnstile) need a real browser. Without a solver, cloudfetch returns ErrManagedChallenge; with chromesolver, Chrome passes them, clicking the checkbox when asked.
• CAPTCHAs (hCaptcha, reCAPTCHA) aren't solved.

Install

go get github.com/luinor223/cloudfetch

Usage

package main

import (
    "fmt"
    "log"

    "github.com/luinor223/cloudfetch"
)

func main() {
    client := cloudfetch.New()

    resp, err := client.Get("https://example.com")
    if err != nil {
        log.Fatal(err)
    }

    text, err := resp.Text()
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("Status: %d, Length: %d\n", resp.StatusCode, len(text))
}

Options

// Firefox fingerprint instead of Chrome
client := cloudfetch.New(cloudfetch.WithBrowser(cloudfetch.Firefox))

// Per-request timeout. To bound a whole call, including challenge solving,
// give the request's context a deadline.
client := cloudfetch.New(cloudfetch.WithTimeout(60 * time.Second))

// Challenge solve attempts per request (default 3, 0 disables solving)
client := cloudfetch.New(cloudfetch.WithMaxRetries(1))

// HTTP proxy (CONNECT), optionally with basic auth
client := cloudfetch.New(cloudfetch.WithProxy("http://user:pass@127.0.0.1:8080"))

// Header for every request (request headers still take precedence)
client := cloudfetch.New(cloudfetch.WithHeader("Referer", "https://google.com"))

Methods

resp, err := client.Get(url)
resp, err := client.Post(url, "application/json", body)
resp, err := client.Do(req)  // *http.Request: per-request headers, context

text, err := resp.Text()    // string, auto-decompressed
bytes, err := resp.Bytes()  // []byte, auto-decompressed

cookies := client.Cookies(url)
client.CloseIdleConnections()
httpClient := client.Underlying()  // raw *http.Client: fingerprint only, no headers or solving

Browser handoff

Managed Challenges and newer JS challenges need a real browser. Set a Solver and cloudfetch hands those pages to it (and classic IUAM pages once its own attempts fail) once per request. It stores the cookies it gets (including cf_clearance), and sends the browser's identity headers (User-Agent, Sec-CH-UA*) to that host from then on, so the requests match the browser that passed. The chromesolver module does this with headless Chrome. It's a separate module so the core library doesn't pull in chromedp; installing it also installs cloudfetch:

go get github.com/luinor223/cloudfetch/chromesolver

client := cloudfetch.New(cloudfetch.WithSolver(chromesolver.New()))
resp, err := client.Get(url)  // first request starts Chrome (~6s), later ones don't

It needs Chrome or Chromium installed. Cloudflare ties cf_clearance to the IP address and User-Agent, so if you use a proxy, give Chrome the same one with chromesolver.WithAllocatorOptions(chromedp.ProxyServer(...)). When the cookie expires, the next challenge runs the solver again.

Errors

resp, err := client.Get(url)
switch {
case errors.Is(err, cloudfetch.ErrManagedChallenge): // Turnstile: needs a real browser (see WithSolver)
case errors.Is(err, cloudfetch.ErrCaptcha):          // CAPTCHA
case errors.Is(err, cloudfetch.ErrChallengeFailed):  // JS challenge not solved after retries
}

License

MIT

Documentation

Overview

Package cloudfetch is an HTTP client that gets past Cloudflare bot detection by sending a real browser's TLS fingerprint and headers. It solves classic IUAM challenges itself and hands harder ones to an optional Solver.

Index

• Variables
• type Browser
• type Clearance
• type Client
  • func New(opts ...Option) *Client
  • func (c *Client) CloseIdleConnections()
  • func (c *Client) Cookies(rawURL string) []*http.Cookie
  • func (c *Client) Do(req *http.Request) (*Response, error)
  • func (c *Client) Get(url string) (*Response, error)
  • func (c *Client) Post(url, contentType string, body io.Reader) (*Response, error)
  • func (c *Client) Underlying() *http.Client
• type Option
  • func WithBrowser(b Browser) Option
  • func WithHeader(key, value string) Option
  • func WithMaxRetries(n int) Option
  • func WithProxy(proxyURL string) Option
  • func WithSolver(s Solver) Option
  • func WithTimeout(d time.Duration) Option
• type Response
  • func (r *Response) Bytes() ([]byte, error)
  • func (r *Response) Text() (string, error)
• type Solver

Variables

var (
	// Chrome is Chrome 133 on Windows, the default.
	Chrome = Browser{utls.HelloChrome_133, [][2]string{
		{"sec-ch-ua", `"Not(A:Brand";v="99", "Google Chrome";v="133", "Chromium";v="133"`},
		{"sec-ch-ua-mobile", "?0"},
		{"sec-ch-ua-platform", `"Windows"`},
		{"Upgrade-Insecure-Requests", "1"},
		{"User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"},
		{"Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"},
		{"Sec-Fetch-Site", "none"},
		{"Sec-Fetch-Mode", "navigate"},
		{"Sec-Fetch-User", "?1"},
		{"Sec-Fetch-Dest", "document"},
		{"Accept-Encoding", "gzip, deflate, br, zstd"},
		{"Accept-Language", "en-US,en;q=0.9"},
		{"Priority", "u=0, i"},
	}}

	// Firefox is Firefox 120 on Windows.
	Firefox = Browser{utls.HelloFirefox_120, [][2]string{
		{"User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0"},
		{"Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"},
		{"Accept-Language", "en-US,en;q=0.5"},
		{"Accept-Encoding", "gzip, deflate, br"},
		{"Upgrade-Insecure-Requests", "1"},
		{"Sec-Fetch-Dest", "document"},
		{"Sec-Fetch-Mode", "navigate"},
		{"Sec-Fetch-Site", "none"},
		{"Sec-Fetch-User", "?1"},
	}}

	// Safari is Safari 16.0 on macOS.
	Safari = Browser{utls.HelloSafari_16_0, [][2]string{
		{"Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"},
		{"User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15"},
		{"Accept-Language", "en-US,en;q=0.9"},
		{"Accept-Encoding", "gzip, deflate, br"},
	}}

	// Edge is Edge 85 on Windows.
	Edge = Browser{utls.HelloEdge_85, [][2]string{
		{"Upgrade-Insecure-Requests", "1"},
		{"User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36 Edg/85.0.564.51"},
		{"Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"},
		{"Sec-Fetch-Site", "none"},
		{"Sec-Fetch-Mode", "navigate"},
		{"Sec-Fetch-User", "?1"},
		{"Sec-Fetch-Dest", "document"},
		{"Accept-Encoding", "gzip, deflate, br"},
		{"Accept-Language", "en-US,en;q=0.9"},
	}}
)

Browsers to impersonate with WithBrowser.

var (
	// ErrManagedChallenge means a Managed or Interactive Challenge that needs
	// a real browser, and no Solver got past it.
	ErrManagedChallenge = errors.New("cloudfetch: managed challenge (Turnstile) needs a real browser")

	// ErrCaptcha means the site served a CAPTCHA.
	ErrCaptcha = errors.New("cloudfetch: CAPTCHA challenge needs an external solving service")

	// ErrChallengeFailed means a JS challenge wasn't solved within
	// WithMaxRetries attempts or by the Solver.
	ErrChallengeFailed = errors.New("cloudfetch: challenge not solved")
)

Errors returned by Client when a challenge can't be solved.

Types

type Browser

type Browser struct {
	// contains filtered or unexported fields
}

Browser is a browser to impersonate: its TLS fingerprint and the default headers it sends, both for the same version.

type Clearance

type Clearance struct {
	Cookies []*http.Cookie
	Header  http.Header
}

Clearance holds the cookies a Solver earned (typically cf_clearance) and the identity headers it sent (User-Agent, Sec-CH-UA*). Cloudflare ties the cookie to them, so the Client sends them to that host from then on.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is an HTTP client with browser TLS fingerprinting. It is safe for concurrent use.

func New

func New(opts ...Option) *Client

New creates a new Client with the given options.

func (*Client) CloseIdleConnections

func (c *Client) CloseIdleConnections()

CloseIdleConnections closes connections kept open by earlier requests that are not in use. It does not interrupt requests in progress.

func (*Client) Cookies

func (c *Client) Cookies(rawURL string) []*http.Cookie

Cookies returns the cookies stored for the given URL.

func (*Client) Do

func (c *Client) Do(req *http.Request) (*Response, error)

Do sends req, solving Cloudflare challenges on the way, and fills in headers req doesn't set without modifying it. If a challenge can't be solved it returns ErrManagedChallenge, ErrCaptcha or ErrChallengeFailed; otherwise the caller must close the body, directly or via Text or Bytes.

func (*Client) Get

func (c *Client) Get(url string) (*Response, error)

Get issues a GET request to url.

func (*Client) Post

func (c *Client) Post(url, contentType string, body io.Reader) (*Response, error)

Post issues a POST request to url with the given body and Content-Type.

func (*Client) Underlying

func (c *Client) Underlying() *http.Client

Underlying returns the *http.Client behind c: fingerprint and cookie jar, but no default headers or challenge solving.

type Option

type Option func(*Client)

Option configures a Client.

func WithBrowser

func WithBrowser(b Browser) Option

WithBrowser sets the browser to impersonate: its TLS fingerprint and its default headers. Defaults to Chrome.

func WithHeader

func WithHeader(key, value string) Option

WithHeader adds a header to every request, over the browser's default but under headers set on the request.

func WithMaxRetries

func WithMaxRetries(n int) Option

WithMaxRetries sets how many times to solve a challenge for one request before giving up with ErrChallengeFailed. Defaults to 3; 0 disables solving.

func WithProxy

func WithProxy(proxyURL string) Option

WithProxy tunnels connections through an HTTP CONNECT proxy, "http://[user:pass@]host:port". The fingerprint is unchanged.

func WithSolver

func WithSolver(s Solver) Option

WithSolver sets a Solver for challenges cloudfetch can't solve: managed and challenge-platform ones at once, IUAM ones after WithMaxRetries attempts. It runs at most once per request.

func WithTimeout

func WithTimeout(d time.Duration) Option

WithTimeout sets the HTTP client timeout. Defaults to 30s.

type Response

type Response struct {
	*http.Response
	// contains filtered or unexported fields
}

Response is an HTTP response from a Client. Text and Bytes return the decompressed body and can be called repeatedly; Body is raw. Use them for Cloudflare 403/429/503 responses, whose body the Client has already read.

func (*Response) Bytes

func (r *Response) Bytes() ([]byte, error)

Bytes returns the response body as bytes, decompressing if needed.

func (*Response) Text

func (r *Response) Text() (string, error)

Text returns the response body as a string, decompressing if needed.

type Solver

type Solver interface {
	// Solve gets past u's challenge and returns the clearance.
	Solve(ctx context.Context, u *url.URL) (*Clearance, error)
}

A Solver gets past challenges cloudfetch can't, usually with a real browser. See github.com/luinor223/cloudfetch/chromesolver.