cloudfetch

README

cloudfetch

Go HTTP client that bypasses Cloudflare bot detection by mimicking a real browser's TLS fingerprint.

How it works

Cloudflare identifies bots partly by their TLS Client Hello fingerprint (JA3/JA4). Go's default net/http has a distinctive fingerprint that gets blocked. cloudfetch uses uTLS to reproduce a real browser's TLS handshake, sends that browser's headers, and keeps a cookie jar.

• TLS fingerprint — Chrome 133, Firefox 120, Safari 16 or Edge 85 via uTLS
• Matching headers — User-Agent, Sec-CH-UA, Sec-Fetch-*, etc. for the same browser and version as the TLS fingerprint
• JS challenge solver — solves the classic IUAM ("Checking your browser") challenge using goja (pure Go JS interpreter)
• Auto-retry — detects challenge pages and retries after solving (up to 3 attempts)
• HTTP/2 — automatic ALPN negotiation and h2 support
• Compression — transparent brotli/gzip/deflate/zstd decompression
• Cookie jar — maintains cookies across requests
• Proxy — HTTP CONNECT proxies, with optional basic auth

Limitations

• Header order and HTTP/2 settings come from Go's net/http, not the browser. Sites that fingerprint those can still tell cloudfetch apart from a real browser.
• Newer JS challenges (challenge-platform) are detected, but the solver only submits the page's form fields without running its script, so they usually fail.
• Turnstile (managed challenges) and CAPTCHAs need a real browser; no HTTP-only client can solve them.

Install

go get github.com/luinor223/cloudfetch

Usage

package main

import (
    "fmt"
    "log"

    "github.com/luinor223/cloudfetch"
)

func main() {
    client := cloudfetch.New()

    resp, err := client.Get("https://example.com")
    if err != nil {
        log.Fatal(err)
    }

    text, err := resp.Text()
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("Status: %d, Length: %d\n", resp.StatusCode, len(text))
}

Options

// Firefox fingerprint instead of Chrome
client := cloudfetch.New(cloudfetch.WithBrowser(cloudfetch.Firefox))

// Custom per-request timeout. Solving a challenge waits several seconds and
// makes extra requests; use DoContext with a deadline to bound the whole call.
client := cloudfetch.New(cloudfetch.WithTimeout(60 * time.Second))

// HTTP proxy (CONNECT), optionally with basic auth
client := cloudfetch.New(cloudfetch.WithProxy("http://user:pass@127.0.0.1:8080"))

// Extra headers
client := cloudfetch.New(cloudfetch.WithHeader("Referer", "https://google.com"))

Methods

resp, err := client.Get(url)
resp, err := client.Post(url, body)
resp, err := client.Do(method, url, body)
resp, err := client.DoContext(ctx, method, url, body)  // cancellable, incl. challenge waits

text, err := resp.Text()   // string, auto-decompressed
bytes, err := resp.Bytes()  // []byte, auto-decompressed

cookies := client.Cookies(url)
httpClient := client.Underlying()  // *http.Client for advanced use

License

MIT

Documentation

Overview

Package cloudfetch provides an HTTP client that bypasses Cloudflare bot detection by mimicking a real browser's TLS fingerprint and headers.

It uses uTLS to reproduce a browser's TLS Client Hello (JA3/JA4 fingerprint), sends that browser's default headers, and maintains a cookie jar across requests.

Index

• Variables
• type Client
  • func New(opts ...Option) *Client
  • func (c *Client) Cookies(rawURL string) []*http.Cookie
  • func (c *Client) Do(method, rawURL string, body io.Reader) (*Response, error)
  • func (c *Client) DoContext(ctx context.Context, method, rawURL string, body io.Reader) (*Response, error)
  • func (c *Client) Get(rawURL string) (*Response, error)
  • func (c *Client) Post(rawURL string, body io.Reader) (*Response, error)
  • func (c *Client) Underlying() *http.Client
• type Option
  • func WithBrowser(id utls.ClientHelloID) Option
  • func WithHeader(key, value string) Option
  • func WithProxy(proxyURL string) Option
  • func WithTimeout(d time.Duration) Option
• type Response
  • func (r *Response) Bytes() ([]byte, error)
  • func (r *Response) Text() (string, error)

Variables

var (
	Chrome  = utls.HelloChrome_Auto  // Chrome 133
	Firefox = utls.HelloFirefox_Auto // Firefox 120
	Safari  = utls.HelloSafari_Auto  // Safari 16.0
	Edge    = utls.HelloEdge_Auto    // Edge 85
)

ClientHelloID aliases for convenience. Each comes with request headers matching the same browser and version, so the TLS fingerprint and the User-Agent agree.

var MaxChallengeRetries = 3

MaxChallengeRetries is how many times to attempt solving a challenge.

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is an HTTP client with browser TLS fingerprinting.

func New

func New(opts ...Option) *Client

New creates a new Client with the given options.

func (*Client) Cookies

func (c *Client) Cookies(rawURL string) []*http.Cookie

Cookies returns the cookies stored for the given URL.

func (*Client) Do

func (c *Client) Do(method, rawURL string, body io.Reader) (*Response, error)

Do performs an HTTP request with the given method. If a Cloudflare challenge is detected, it attempts to solve it automatically.

func (*Client) DoContext

func (c *Client) DoContext(ctx context.Context, method, rawURL string, body io.Reader) (*Response, error)

DoContext is like Do but honors ctx for every request and for the waits before submitting challenge answers. WithTimeout limits each HTTP request; give ctx a deadline to bound the whole call, including challenge solving.

func (*Client) Get

func (c *Client) Get(rawURL string) (*Response, error)

Get performs an HTTP GET request.

func (*Client) Post

func (c *Client) Post(rawURL string, body io.Reader) (*Response, error)

Post performs an HTTP POST request.

func (*Client) Underlying

func (c *Client) Underlying() *http.Client

Underlying returns the raw *http.Client for advanced use.

type Option

type Option func(*Client)

Option configures a Client.

func WithBrowser

func WithBrowser(id utls.ClientHelloID) Option

WithBrowser sets the browser to impersonate: its TLS fingerprint and its default headers. Defaults to Chrome. For a utls ClientHelloID other than the package presets, headers follow the preset for the same browser, so set User-Agent with WithHeader if the version differs.

func WithHeader

func WithHeader(key, value string) Option

WithHeader adds a default header sent with every request.

func WithProxy

func WithProxy(proxyURL string) Option

WithProxy routes connections through an HTTP proxy, given as "http://[user:pass@]host:port", using CONNECT tunnels. The TLS handshake with the target still happens end to end, so the fingerprint is unchanged.

func WithTimeout

func WithTimeout(d time.Duration) Option

WithTimeout sets the HTTP client timeout. Defaults to 30s.

type Response

type Response struct {
	*http.Response
	// contains filtered or unexported fields
}

Response wraps http.Response with convenience methods.

func (*Response) Bytes

func (r *Response) Bytes() ([]byte, error)

Bytes returns the response body as bytes, decompressing if needed.

func (*Response) Text

func (r *Response) Text() (string, error)

Text returns the response body as a string, decompressing if needed.