cloudfetch

README

cloudfetch

Go HTTP client that bypasses Cloudflare bot detection by mimicking a real browser's TLS fingerprint.

How it works

Cloudflare identifies bots partly by their TLS Client Hello fingerprint (JA3/JA4). Go's default net/http has a distinctive fingerprint that gets blocked. cloudfetch uses uTLS to reproduce a real browser's TLS handshake, sends that browser's headers, and keeps a cookie jar.

• TLS fingerprint — Chrome 133, Firefox 120, Safari 16 or Edge 85 via uTLS
• Matching headers — User-Agent, Sec-CH-UA, Sec-Fetch-*, etc. for the same browser and version as the TLS fingerprint
• JS challenge solver — solves the classic IUAM ("Checking your browser") challenge using goja (pure Go JS interpreter)
• Auto-retry — detects challenge pages and retries after solving (up to 3 attempts)
• HTTP/2 — automatic ALPN negotiation and h2 support
• Compression — transparent brotli/gzip/deflate/zstd decompression
• Cookie jar — maintains cookies across requests
• Proxy — HTTP CONNECT proxies, with optional basic auth

Limitations

• Header order and HTTP/2 settings come from Go's net/http, not the browser. Sites that fingerprint those can still tell cloudfetch apart from a real browser.
• Newer JS challenges (challenge-platform) are detected, but the solver only submits the page's form fields without running its script, so they usually fail.
• Turnstile (managed challenges) and CAPTCHAs need a real browser; no HTTP-only client can solve them.

Install

go get github.com/luinor223/cloudfetch

Usage

package main

import (
    "fmt"
    "log"

    "github.com/luinor223/cloudfetch"
)

func main() {
    client := cloudfetch.New()

    resp, err := client.Get("https://example.com")
    if err != nil {
        log.Fatal(err)
    }

    text, err := resp.Text()
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("Status: %d, Length: %d\n", resp.StatusCode, len(text))
}

Options

// Firefox fingerprint instead of Chrome
client := cloudfetch.New(cloudfetch.WithBrowser(cloudfetch.Firefox))

// Per-request timeout. To bound a whole call, including challenge solving,
// give the request's context a deadline.
client := cloudfetch.New(cloudfetch.WithTimeout(60 * time.Second))

// Challenge solve attempts per request (default 3, 0 disables solving)
client := cloudfetch.New(cloudfetch.WithMaxRetries(1))

// HTTP proxy (CONNECT), optionally with basic auth
client := cloudfetch.New(cloudfetch.WithProxy("http://user:pass@127.0.0.1:8080"))

// Header for every request (request headers still take precedence)
client := cloudfetch.New(cloudfetch.WithHeader("Referer", "https://google.com"))

Methods

resp, err := client.Get(url)
resp, err := client.Post(url, "application/json", body)
resp, err := client.Do(req)  // *http.Request: per-request headers, context

text, err := resp.Text()    // string, auto-decompressed
bytes, err := resp.Bytes()  // []byte, auto-decompressed

cookies := client.Cookies(url)
client.CloseIdleConnections()
httpClient := client.Underlying()  // raw *http.Client: fingerprint only, no headers or solving

Errors

resp, err := client.Get(url)
switch {
case errors.Is(err, cloudfetch.ErrManagedChallenge): // Turnstile: needs a real browser
case errors.Is(err, cloudfetch.ErrCaptcha):          // CAPTCHA
case errors.Is(err, cloudfetch.ErrChallengeFailed):  // JS challenge not solved after retries
}

License

MIT

Documentation

Overview

Package cloudfetch provides an HTTP client that bypasses Cloudflare bot detection by mimicking a real browser's TLS fingerprint and headers.

It uses uTLS to reproduce a browser's TLS Client Hello (JA3/JA4 fingerprint), sends that browser's default headers, keeps a cookie jar across requests, and solves Cloudflare's classic IUAM JS challenge.

Index

• Variables
• type Browser
• type Client
  • func New(opts ...Option) *Client
  • func (c *Client) CloseIdleConnections()
  • func (c *Client) Cookies(rawURL string) []*http.Cookie
  • func (c *Client) Do(req *http.Request) (*Response, error)
  • func (c *Client) Get(url string) (*Response, error)
  • func (c *Client) Post(url, contentType string, body io.Reader) (*Response, error)
  • func (c *Client) Underlying() *http.Client
• type Option
  • func WithBrowser(b Browser) Option
  • func WithHeader(key, value string) Option
  • func WithMaxRetries(n int) Option
  • func WithProxy(proxyURL string) Option
  • func WithTimeout(d time.Duration) Option
• type Response
  • func (r *Response) Bytes() ([]byte, error)
  • func (r *Response) Text() (string, error)

Variables

var (
	// Chrome is Chrome 133 on Windows, the default.
	Chrome = Browser{utls.HelloChrome_133, [][2]string{
		{"sec-ch-ua", `"Not(A:Brand";v="99", "Google Chrome";v="133", "Chromium";v="133"`},
		{"sec-ch-ua-mobile", "?0"},
		{"sec-ch-ua-platform", `"Windows"`},
		{"Upgrade-Insecure-Requests", "1"},
		{"User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"},
		{"Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"},
		{"Sec-Fetch-Site", "none"},
		{"Sec-Fetch-Mode", "navigate"},
		{"Sec-Fetch-User", "?1"},
		{"Sec-Fetch-Dest", "document"},
		{"Accept-Encoding", "gzip, deflate, br, zstd"},
		{"Accept-Language", "en-US,en;q=0.9"},
		{"Priority", "u=0, i"},
	}}

	// Firefox is Firefox 120 on Windows.
	Firefox = Browser{utls.HelloFirefox_120, [][2]string{
		{"User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0"},
		{"Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"},
		{"Accept-Language", "en-US,en;q=0.5"},
		{"Accept-Encoding", "gzip, deflate, br"},
		{"Upgrade-Insecure-Requests", "1"},
		{"Sec-Fetch-Dest", "document"},
		{"Sec-Fetch-Mode", "navigate"},
		{"Sec-Fetch-Site", "none"},
		{"Sec-Fetch-User", "?1"},
	}}

	// Safari is Safari 16.0 on macOS.
	Safari = Browser{utls.HelloSafari_16_0, [][2]string{
		{"Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"},
		{"User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15"},
		{"Accept-Language", "en-US,en;q=0.9"},
		{"Accept-Encoding", "gzip, deflate, br"},
	}}

	// Edge is Edge 85 on Windows.
	Edge = Browser{utls.HelloEdge_85, [][2]string{
		{"Upgrade-Insecure-Requests", "1"},
		{"User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36 Edg/85.0.564.51"},
		{"Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"},
		{"Sec-Fetch-Site", "none"},
		{"Sec-Fetch-Mode", "navigate"},
		{"Sec-Fetch-User", "?1"},
		{"Sec-Fetch-Dest", "document"},
		{"Accept-Encoding", "gzip, deflate, br"},
		{"Accept-Language", "en-US,en;q=0.9"},
	}}
)

Browsers to impersonate with WithBrowser.

var (
	// ErrManagedChallenge means the site served a managed challenge
	// (Turnstile), which needs a real browser.
	ErrManagedChallenge = errors.New("cloudfetch: managed challenge (Turnstile) needs a real browser")

	// ErrCaptcha means the site served a CAPTCHA, which needs an external
	// solving service.
	ErrCaptcha = errors.New("cloudfetch: CAPTCHA challenge needs an external solving service")

	// ErrChallengeFailed means the site still served a JS challenge after the
	// number of attempts set by WithMaxRetries.
	ErrChallengeFailed = errors.New("cloudfetch: challenge not solved")
)

Errors returned by Client when a challenge can't be solved.

Types

type Browser

type Browser struct {
	// contains filtered or unexported fields
}

Browser is a browser to impersonate: its TLS fingerprint and the default headers it sends, both for the same version.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is an HTTP client with browser TLS fingerprinting. It is safe for concurrent use.

func New

func New(opts ...Option) *Client

New creates a new Client with the given options.

func (*Client) CloseIdleConnections

func (c *Client) CloseIdleConnections()

CloseIdleConnections closes connections kept open by earlier requests that are not in use. It does not interrupt requests in progress.

func (*Client) Cookies

func (c *Client) Cookies(rawURL string) []*http.Cookie

Cookies returns the cookies stored for the given URL.

func (*Client) Do

func (c *Client) Do(req *http.Request) (*Response, error)

Do sends req and returns the response, solving Cloudflare challenges along the way. It fills in the headers req doesn't set and leaves req unmodified. req's context also bounds the waits before submitting challenge answers.

If a challenge can't be solved, Do returns ErrManagedChallenge, ErrCaptcha or ErrChallengeFailed. Otherwise the caller must close the response body, directly or by calling Text or Bytes.

func (*Client) Get

func (c *Client) Get(url string) (*Response, error)

Get issues a GET request to url.

func (*Client) Post

func (c *Client) Post(url, contentType string, body io.Reader) (*Response, error)

Post issues a POST request to url with the given body and Content-Type.

func (*Client) Underlying

func (c *Client) Underlying() *http.Client

Underlying returns the *http.Client that sends requests. It keeps the browser's TLS fingerprint and the cookie jar, but requests sent through it directly get none of the browser's default headers and no challenge solving.

type Option

type Option func(*Client)

Option configures a Client.

func WithBrowser

func WithBrowser(b Browser) Option

WithBrowser sets the browser to impersonate: its TLS fingerprint and its default headers. Defaults to Chrome.

func WithHeader

func WithHeader(key, value string) Option

WithHeader adds a header sent with every request. It overrides the browser's default for the same header, and is overridden by headers set on the request itself.

func WithMaxRetries

func WithMaxRetries(n int) Option

WithMaxRetries sets how many times to solve a challenge for one request before giving up with ErrChallengeFailed. Defaults to 3; 0 disables solving.

func WithProxy

func WithProxy(proxyURL string) Option

WithProxy routes connections through an HTTP proxy, given as "http://[user:pass@]host:port", using CONNECT tunnels. The TLS handshake with the target still happens end to end, so the fingerprint is unchanged.

func WithTimeout

func WithTimeout(d time.Duration) Option

WithTimeout sets the HTTP client timeout. Defaults to 30s.

type Response

type Response struct {
	*http.Response
	// contains filtered or unexported fields
}

Response is an HTTP response from a Client.

Text and Bytes read the whole body, decompress it according to its Content-Encoding, close it, and cache the result, so they can be called more than once. Reading Body directly gives the raw, possibly compressed bytes instead. For Cloudflare 403, 429 and 503 responses Client has already read the body to look for a challenge, so use Text or Bytes for those.

func (*Response) Bytes

func (r *Response) Bytes() ([]byte, error)

Bytes returns the response body as bytes, decompressing if needed.

func (*Response) Text

func (r *Response) Text() (string, error)

Text returns the response body as a string, decompressing if needed.