requests

package
v0.0.0-...-2835ce2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 19, 2026 License: MIT Imports: 30 Imported by: 0

README

requests

A Go port of the curl_cffi.requests API: a requests-style HTTP client that impersonates real browsers' TLS/JA3, HTTP/2 and HTTP/3 fingerprints.

The package is pure Go (no cgo). A Session owns a cookie jar and a cache of transports; the module-level helpers run in a throwaway session. Import path: github.com/HashShin/shade/requests.

The module root re-exports all of it, so import . "github.com/HashShin/shade" gives the same names unqualified; see the root README.

Quick start

package main

import (
	"fmt"

	"github.com/HashShin/shade/requests"
)

func main() {
	s := requests.NewSession(
		requests.WithImpersonate(requests.DefaultChrome),
		requests.WithTimeoutSeconds(15),
	)
	defer s.Close()

	rsp, err := s.Post("https://httpbin.org/post",
		requests.WithJSON(map[string]any{"hello": "world"}),
	)
	if err != nil {
		panic(err)
	}
	fmt.Println(rsp.StatusCode, rsp.Text())
}

A one-off request needs no session:

rsp, err := requests.Get("https://tls.browserleaks.com/json",
	requests.WithImpersonate(requests.DefaultChrome),
)

DefaultChrome follows the current Chrome preset; requests.Chrome150 pins that exact version. Either is a compile-checked name, where "chrome" is only checked when the request is made.

WithImpersonate takes a target name. A constant is preferred, because a misspelt one does not compile: requests.DefaultChrome and the other family defaults follow the current preset of each family, the exact targets are requests.Chrome150 and its siblings, and both are re-exported from the impersonate package, which is where they are defined and where Targets() lists them all. native, none, go or an empty name select Go's own stack, and curl reproduces the system curl.

To drop the qualifier at a call site, import the module root instead. It re-exports this package, the targets and the options, so Request, Headers and Chrome146 need no prefix; see the root README. A dot import is what staticcheck reports as ST1001, so silence it deliberately if your setup runs staticcheck.

Module-level helpers

Each helper builds a Session, sends the request and closes the session, so cookies and connections are not reused across calls. The method is upper-cased by Session.Request.

Function Method
Get(rawURL string, opts ...Option) (*Response, error) GET
Post(rawURL string, opts ...Option) (*Response, error) POST
Put(rawURL string, opts ...Option) (*Response, error) PUT
Patch(rawURL string, opts ...Option) (*Response, error) PATCH
Delete(rawURL string, opts ...Option) (*Response, error) DELETE
Head(rawURL string, opts ...Option) (*Response, error) HEAD
Options(rawURL string, opts ...Option) (*Response, error) OPTIONS
Trace(rawURL string, opts ...Option) (*Response, error) TRACE
Do(method, rawURL string, opts ...Option) (*Response, error) caller-supplied

Session

NewSession(opts ...Option) *Session applies its options as defaults for every request; options passed to a request override them for that request only. The session is safe for concurrent use.

Method Purpose
Request(method, rawURL string, opts ...Option) (*Response, error) send a request
Send(req Request) (*Response, error) send a request described by a value
Get / Post / Put / Patch / Delete / Head / Options / Trace verb helpers taking (rawURL string, opts ...Option)
Close() close idle connections; later requests fail with *SessionClosed
Cookies() *Cookies the live session cookie jar
SetCookies(c CookieTypes) replace the session jar
Headers() *Headers the session default headers (mutable)
UserAgent() string the impersonated UA, the curl default for curl, "" for native

Session defaults come from defaultConfig:

Setting Default
timeout 30s
follow redirects true
maximum redirects 30
verify TLS certificates true
trust proxy environment variables true
default response encoding utf-8

A session keeps a transport cache keyed by impersonation name, effective proxy, TLS verification, HTTP version, interface and client certificate, so requests that share those values reuse connections.

Send takes the struct form, which is useful when a request is built from configuration or stored rather than written out at the call site. There is a module-level Send(Request) too, for a one-off that needs no session:

rsp, err := sess.Send(requests.Request{
	Method:      "GET",
	URL:         "https://httpbun.com/get",
	Headers:     requests.Headers{"Accept: application/json"},
	Impersonate: requests.Chrome146, // same package: no second import
	Timeout:     10 * time.Second,
})

Impersonate is a string, so "chrome146" also works, but the constants are re-exported here from the impersonate package so a request needs one import rather than two. A misspelt constant does not compile; a misspelt string fails when the request is made, as an *ImpersonateError. impersonate.Targets() lists every target.

Only URL is required. Method empty means GET, and a zero Timeout or empty Proxy leaves the session's setting in place rather than overriding it with an empty value. JSON wins over Body when both are set. The fields are Method, URL, Headers, Params, Cookies, Body, JSON, Impersonate, Timeout and Proxy.

Response

rsp := requests.NewResponse() // empty: 200 OK, empty headers and cookies
Field Type Notes
URL string final URL after redirects
Content []byte decoded body in buffered mode
StatusCode int status code
Reason string status text without the numeric code
Ok bool 200 <= StatusCode < 400
Headers *Headers response headers
Cookies *Cookies session cookies matching the final URL
Elapsed time.Duration from the first send to the final response
DefaultEncoding string encoding used when the response has no charset
RedirectCount int number of hops, i.e. len(History)
RedirectURL string not populated (always "")
HTTPVersion int HTTP major version (1, 2 or 3)
PrimaryIP, PrimaryPort, LocalIP, LocalPort string, int, string, int not populated
History []*Response redirect hops; their Content/Body are nil
Request *Request the final request
DownloadSize int64 len(Content) in buffered mode; 0 when streaming
UploadSize, HeaderSize, RequestSize int64 not populated
Body io.ReadCloser set with WithStream(true); the caller closes it

Methods:

Method Purpose
Text() string decode Content with Encoding, replacing invalid bytes; cached
JSON(v any) error json.Unmarshal of Content
Encoding() string explicit encoding, else the Content-Type charset, else DefaultEncoding, else utf-8
SetEncoding(enc string) error override the charset; errors once Text has run
CharsetEncoding() string charset parsed from Content-Type, or ""
ContentType() string Content-Type header value
HasHeader(name string) bool header presence test
IsRedirect() bool 301/302/303/307/308 with a Location header
ReadAll() ([]byte, error) consume and return a streaming body; buffered mode returns Content
IterContent() (func() ([]byte, error, bool), error) chunk iterator; the bool marks completion
IterLines(delimiter string) ([]string, error) split the body; empty delimiter splits on \n after normalising \r\n
Len() int body length in bytes
RaiseForStatus() error *HTTPError unless Ok
Close() error close Body if set
String() string "<Response [200]>"

Compression is decoded from Content-Encoding: gzip/x-gzip, deflate, br/brotli and zstd/zstandard, including stacked values.

Request

Request describes the final request and is reachable as Response.Request.

Field Type
URL string
Method string
Headers *Headers
Body []byte

Headers

Headers is an ordered, case-insensitive header collection. Set replaces in place (keeping the key's position), Add appends a second line.

It is a []string of "Name: Value" lines, so it can be written as a literal, which is how headers appear in HTTP and on a command line:

requests.Headers{
	"Accept: application/json",
	"X-Custom: value",
}

NewHeaders(h HeaderTypes) *Headers accepts Headers or a plain []string (same content), *Headers, []HeaderPair, map[string]string or map[string][]string. A HeaderPair is {Name, Value string}. Map inputs are applied in sorted key order, because a map has none. An unsupported type panics, so a caller finds out immediately rather than sending a request with headers silently missing.

A value's leading and trailing spaces are not significant and are dropped when the line is read back, matching how an HTTP parser treats them. Only the first colon separates the name from the value, so a value may contain one.

Method Purpose
Set(name, value string) replace, or append when absent
Add(name, value string) append another value
Del(name string) remove all values for a key
Get(name string) string values joined with ", "
GetList(name string) []string all values in order
Has(name string) bool presence test
Len() int number of header lines
MultiItems() []HeaderPair every line in order, without joining
Items() map[string]string unique keys to joined values
Keys() []string unique names in first-seen order
Update(other HeaderTypes) merge another set, replacing in place
Clone() *Headers deep copy
String() string Headers{...} rendering

Header order is part of the fingerprint. The transport lower-cases names on the wire (HTTP/2 requires it); see Limitations for the HTTP/1.1 effect.

Cookies

Cookies is an ordered, mutex-protected cookie collection. NewCookies(c CookieTypes) *Cookies accepts *Cookies, Cookies, map[string]string or []*Cookie. A Cookie has Name, Value, Domain, Path, Secure, HTTPOnly and Expires fields.

Method Purpose
Set(name, value string) set a bare cookie
SetCookie(ck Cookie) insert or replace, matching name+domain+path
Get(name string) (string, bool) raw value by name
GetCookie(name string) (*Cookie, bool) full cookie by name
Del(name string) remove by name
Items() []Cookie snapshot in insertion order
Len() int count
Map() map[string]string name to value
Update(other CookieTypes) merge another collection
String() string render as a=1; b=2

Expired, secure-on-http and domain/path-mismatched cookies are filtered before a request; Set-Cookie from each hop updates the jar unless WithDiscardCookies is set.

Params

Params is []Param, an ordered query list that preserves duplicates; Param is {Key, Value string} and Params.Encode() renders a query string. WithParams accepts Params, []Param, url.Values, map[string]string or map[string][]string (and panics on any other type).

Options

Option configures a Session or a single request. All options below exist in options.go.

Some are kept only for curl_cffi API compatibility and cannot be honoured by this port's transport. Those are marked refused: setting one makes the request fail with an *UnsupportedOptionError before anything is sent, rather than silently dropping it and sending a fingerprint the caller did not ask for.

Request body
Option Effect
WithData(v any) body from string, []byte, io.Reader, url.Values, Params, map[string]string or map[string][]string; maps are form-encoded with application/x-www-form-urlencoded
WithContent(v any) raw body from string, []byte or io.Reader; other types error
WithJSON(v any) JSON body and application/json, without an added trailing newline
URL and query
Option Effect
WithParams(p any) query parameters, appended to any in the URL
WithBaseURL(u string) base URL for resolving a relative request URL
WithQuote(v any) URL percent-encoding parity hook; refused
Headers and cookies
Option Effect
WithHeaders(h HeaderTypes) merge request headers
WithHeader(name, value string) set one request header
WithCookies(c CookieTypes) set request cookies
WithReferer(r string) set Referer
WithAuth(username, password string) HTTP basic auth; sent only to the original host
WithAcceptEncoding(v string) set Accept-Encoding; an empty value removes it and lets the transport decompress
Transport and connection
Option Effect
WithTimeout(d time.Duration) timeout applied to each network request
WithTimeoutSeconds(s float64) same, in seconds
WithProxy(proxyURL string) one proxy for all schemes
WithProxies(p map[string]string) proxy by "all", "https" or "http", first non-empty in that order
WithProxyAuth(username, password string) proxy credentials; impersonating transport only
WithVerify(v bool) TLS certificate verification
WithCert(certFile, keyFile string) client certificate for mTLS; impersonating transport only
WithInterface(name string) bind to a local IP address; interface names are not resolved
WithHTTPVersion(v string) "v1", "v2" or "v3" (also 1, http/2, h3, auto, ...)
WithTrustEnv(v bool) honour HTTPS_PROXY, https_proxy, ALL_PROXY, all_proxy when no proxy is set
WithDebug(v bool) verbose impersonating-transport logging
WithMaxRecvSpeed(n int) download rate limit; refused
WithDefaultEncoding(enc string) charset for responses without one
TLS and fingerprint
Option Effect
WithImpersonate(name string) select a preset, curl, or native/none/go
WithJA3(s string) custom JA3 string; refused (use WithImpersonate)
WithAkamai(s string) custom Akamai HTTP/2 string; refused
WithExtraFP(fp *ExtraFingerprints) per-request fingerprint overrides; refused
WithDefaultHeaders(v bool) keep only the caller's headers, dropping the preset's default set (false); the fingerprint preset itself is still used

ExtraFingerprints carries TLSMinVersion, TLSGrease, TLSPermuteExtensions, TLSCertCompression, TLSRecordSizeLimit, HTTP2StreamWeight, HTTP2StreamExclusive, HTTP2NoPriority, HeaderOrder and SplitCookies. Preset headers are always merged, with user headers winning.

Streams and callbacks
Option Effect
WithStream(v bool) leave the body unread in Response.Body for incremental reads
WithContentCallback(fn func([]byte) error) call fn for each body chunk; forces buffered reading
Redirects
Option Effect
WithAllowRedirects(v bool) follow redirects (default true)
WithMaxRedirects(n int) limit; -1 means unlimited; exceeding it returns *TooManyRedirects

Redirect handling mirrors libcurl: 303 always becomes GET; 301/302 from a non-GET/HEAD method becomes GET; 307/308 preserve method and body. Each hop's Set-Cookie is applied before the next hop, and Authorization is dropped on a cross-host redirect.

Retries
Option Effect
WithRetry(n int) retry a failed request up to n extra times, with no backoff

Any transport error is retried, including a body that cannot be replayed; there is no retry on HTTP status.

Behaviour
Option Effect
WithDiscardCookies(v bool) do not let server cookies update the session jar
WithRaiseForStatus(v bool) return *HTTPError for 4xx/5xx responses

Errors

RequestException is the base error; it carries Msg, a curl-like numeric Code and an optional *Response. errors.As distinguishes the wrappers:

Type Raised when
*ConnectionError transport failure
*DNSError host resolution fails
*ProxyError proxy failure
*SSLError, *CertificateVerifyError TLS or certificate failure
*Timeout timeout; also matches ErrTimeout via errors.Is and IsTimeout
*TooManyRedirects redirect limit exceeded
*InvalidURL, *InvalidSchema malformed URL or unsupported scheme
*ImpersonateError unknown target or unusable TLS configuration
*SessionClosed request after Close
*UnsupportedOptionError an option set for curl_cffi parity that this port cannot honour
*HTTPError RaiseForStatus / WithRaiseForStatus

NewRequestException builds the most specific wrapper for a transport error string. InterfaceError, IncompleteRead and UnrewindableBodyError are declared for API parity but are never returned.

Tests

The unit tests use httptest servers for parameters, headers, JSON and form bodies, redirects and history, cookie forwarding, gzip decoding, RaiseForStatus, timeouts, session closing and header ordering (session_test.go, headers_test.go). curl_test.go and spec_test.go check the curl ClientHello, the curl HTTP/2 profile and the preset spec builder. integration_test.go's TestLiveFingerprintBaseline compares live JA3N and Akamai hashes against testdata/fingerprint_baseline.json, recorded once from the Python curl_cffi; run it with make test-live.

Limitations

  • Pure Go, no cgo. The impersonating transport is built on github.com/bogdanfinn/fhttp, tls-client and utls.
  • Async sessions are not implemented: Session is synchronous.
  • WebSockets are absent, as are caching backends, DoH and curl_options. The config has an unused dohURL field and no option sets it.
  • HTTP/1.1 header names are lower-cased on the wire by the transport. HTTP/2 (the default for impersonated traffic) is unaffected because its names are lower-case anyway.
  • Response.PrimaryIP, PrimaryPort, LocalIP, LocalPort, UploadSize, HeaderSize and RequestSize are never populated, and RedirectURL is always empty; DownloadSize is only set in buffered mode.
  • Chrome 110+ permutes its TLS extensions per connection. This port emits a fixed valid canonical order, so JA3N/JA4 (which sort extensions) are exact while the raw JA3 byte order is one valid sample rather than re-randomised.
  • WithJA3, WithAkamai, WithExtraFP, WithQuote and WithMaxRecvSpeed are accepted for curl_cffi compatibility but cannot be honoured, so setting any of them makes the request fail with an *UnsupportedOptionError. WithJA3 is the clearest case: a JA3 string lists extension ids, not their contents, and the transport builds a ClientHello from a full spec, so there is nothing faithful to construct from it. WithAkamai and WithExtraFP are plumbed to no profile, and no supported transport can rate-limit for WithMaxRecvSpeed.
  • WithCert and WithProxyAuth only affect the impersonating transport; the native target ignores them. WithInterface binds only when its value parses as a local IP address, not for a named interface.
  • WithProxies reads only the "all", "https" and "http" keys, and the environment lookup uses HTTPS_PROXY/https_proxy/ALL_PROXY/all_proxy (not HTTP_PROXY).
  • requests has no JavaScript engine. For client-rendered pages use the browser package, which runs scripts over the same impersonating transport. A matching fingerprint alone is not sufficient for every site; see the root README.md for measured examples.

Documentation

Overview

Package requests is a Go port of the curl_cffi.requests API. It provides a requests-like Session with browser TLS/HTTP fingerprint impersonation.

Index

Constants

View Source 
const (
	Chrome100        = impersonate.Chrome100
	Chrome101        = impersonate.Chrome101
	Chrome104        = impersonate.Chrome104
	Chrome107        = impersonate.Chrome107
	Chrome110        = impersonate.Chrome110
	Chrome116        = impersonate.Chrome116
	Chrome119        = impersonate.Chrome119
	Chrome120        = impersonate.Chrome120
	Chrome123        = impersonate.Chrome123
	Chrome124        = impersonate.Chrome124
	Chrome131        = impersonate.Chrome131
	Chrome131Android = impersonate.Chrome131Android
	Chrome133a       = impersonate.Chrome133a
	Chrome136        = impersonate.Chrome136
	Chrome142        = impersonate.Chrome142
	Chrome145        = impersonate.Chrome145
	Chrome146        = impersonate.Chrome146
	Chrome150        = impersonate.Chrome150
	Chrome99         = impersonate.Chrome99
	Chrome99Android  = impersonate.Chrome99Android
	Custom           = impersonate.Custom
	Edge101          = impersonate.Edge101
	Edge99           = impersonate.Edge99
	Firefox133       = impersonate.Firefox133
	Firefox135       = impersonate.Firefox135
	Firefox144       = impersonate.Firefox144
	Firefox147       = impersonate.Firefox147
	OkHTTP4Android   = impersonate.OkHTTP4Android
	Safari153        = impersonate.Safari153
	Safari155        = impersonate.Safari155
	Safari170        = impersonate.Safari170
	Safari172Ios     = impersonate.Safari172Ios
	Safari180        = impersonate.Safari180
	Safari180Ios     = impersonate.Safari180Ios
	Safari184        = impersonate.Safari184
	Safari184Ios     = impersonate.Safari184Ios
	Safari260        = impersonate.Safari260
	Safari2601       = impersonate.Safari2601
	Safari260Ios     = impersonate.Safari260Ios
	Tor145           = impersonate.Tor145
)

Impersonation targets, re-exported from the impersonate package so that a request needs one import rather than two: 

sess.Send(requests.Request{
	URL:         "https://httpbun.com/get",
	Impersonate: requests.Chrome146,
})

These are aliases of the constants in impersonate/targets.go, not a second copy of the values, so a preset change reaches both. TestTargetsMatchImpersonate fails if a target is added there and not here.

The Impersonate field is a plain string, so "chrome146" works too. Prefer a constant: a misspelt string is only caught when the request is made, whereas a misspelt constant does not compile. impersonate.Get reports an unknown target as an *ImpersonateError.

View Source 
const (
	DefaultChrome        = impersonate.DefaultChrome
	DefaultEdge          = impersonate.DefaultEdge
	DefaultSafari        = impersonate.DefaultSafari
	DefaultSafariIOS     = impersonate.DefaultSafariIOS
	DefaultSafariBeta    = impersonate.DefaultSafariBeta
	DefaultSafariIOSBeta = impersonate.DefaultSafariIOSBeta
	DefaultChromeAndroid = impersonate.DefaultChromeAndroid
	DefaultFirefox       = impersonate.DefaultFirefox
	DefaultTor           = impersonate.DefaultTor
)

The family defaults: the target each short alias resolves to, so "chrome" resolves to DefaultChrome. These are the better choice for an example or a default setting, because they follow the family as the presets are updated, where naming an exact version pins a browser that will age.

Variables

View Source 
var ErrTimeout = errors.New("requests: timeout")

ErrTimeout is a sentinel matching any Timeout error.

Functions

func IsTimeout 

func IsTimeout(err error) bool

IsTimeout reports whether err is a Timeout.

func NewRequestException 

func NewRequestException(msg string, code int, rsp *Response) error

NewRequestException builds the most specific error for a transport failure.

func UseBrowser 

func UseBrowser(newRenderer func(sess *Session) BrowserRenderer)

UseBrowser installs the renderer that Request.Browser delegates to, one per session, so a page and a plain request on the same session share cookies. The browser package calls it from an init, and there is no reason to call it yourself: importing the browser package is what installs it.

Types

type BasicAuth 

type BasicAuth struct {
	Username string
	Password string
}

BasicAuth holds HTTP basic-auth credentials.

type BrowserRenderer 

type BrowserRenderer interface {
	// Send loads the request's URL and returns the page that settles, with the
	// rendered document as the response body.
	Send(req Request) (*Response, error)
	// Close releases the renderer's browser.
	Close()
}

BrowserRenderer is the browser a Request with Browser set is loaded in. The browser package installs one from its init, which is how this package stays free of a dependency on it: browser imports requests, so the reverse would be an import cycle.

type CertificateVerifyError 

type CertificateVerifyError struct{ *SSLError }

CertificateVerifyError indicates certificate validation failed.

type ClientCert 

type ClientCert struct {
	CertFile string
	KeyFile  string
}

ClientCert holds a client certificate and key file path.

type ConnectionError 

type ConnectionError struct{ *RequestException }

ConnectionError indicates a connection level failure. 

type Cookie struct {
	Name     string
	Value    string
	Domain   string
	Path     string
	Secure   bool
	HTTPOnly bool
	Expires  time.Time
}

Cookie is a single HTTP cookie with the attributes curl_cffi tracks.

type CookieTypes 

type CookieTypes interface{}

CookieTypes is the set of accepted cookie inputs.

type Cookies 

type Cookies struct {
	// contains filtered or unexported fields
}

Cookies is an ordered cookie collection. It mirrors curl_cffi's Cookies mapping (name -> attributes) closely enough for session handling.

func NewCookies 

func NewCookies(c CookieTypes) *Cookies

NewCookies builds a Cookies value from a map or an existing collection.

func (*Cookies) Del 

func (c *Cookies) Del(name string)

Del removes a cookie by name.

func (*Cookies) Get 

func (c *Cookies) Get(name string) (string, bool)

Get returns the raw value of a cookie by name.

func (*Cookies) GetCookie 

func (c *Cookies) GetCookie(name string) (*Cookie, bool)

GetCookie returns the full cookie by name.

func (*Cookies) Items 

func (c *Cookies) Items() []Cookie

Items returns a copy of all cookies in insertion order.

func (*Cookies) Len 

func (c *Cookies) Len() int

Len returns the cookie count.

func (*Cookies) Map 

func (c *Cookies) Map() map[string]string

Map returns name -> value for every cookie.

func (*Cookies) Set 

func (c *Cookies) Set(name, value string)

Set sets a cookie by name, dropping any domain/path attributes.

func (*Cookies) SetCookie 

func (c *Cookies) SetCookie(ck Cookie)

SetCookie inserts or replaces a cookie, matching on name+domain+path.

func (*Cookies) String 

func (c *Cookies) String() string

String renders the cookies as a "a=1; b=2" header value.

func (*Cookies) Update 

func (c *Cookies) Update(other CookieTypes)

Update merges cookies into the collection, replacing by name.

type DNSError 

type DNSError struct{ *ConnectionError }

DNSError indicates the host could not be resolved.

type ExtraFingerprints 

type ExtraFingerprints struct {
	TLSMinVersion        string
	TLSGrease            *bool
	TLSPermuteExtensions *bool
	TLSCertCompression   string
	TLSRecordSizeLimit   int
	HTTP2StreamWeight    *int
	HTTP2StreamExclusive *int
	HTTP2NoPriority      *bool
	HeaderOrder          string
	SplitCookies         *bool
}

ExtraFingerprints carries the per-request fingerprint overrides that curl_cffi exposes through extra_fp. Only the fields understood by this port are represented.

type HTTPError 

type HTTPError struct{ *RequestException }

HTTPError is returned by Response.raise_for_status for 4xx/5xx.

type HeaderPair 

type HeaderPair struct {
	Name  string
	Value string
}

HeaderPair is a single header line.

type HeaderTypes 

type HeaderTypes interface{}

HeaderTypes is the set of accepted header inputs: *Headers, Headers, []HeaderPair, map[string]string, or map[string][]string. Headers itself is a []string of "Name: Value" lines, so a plain []string is accepted too. A map loses ordering, so prefer Headers or []HeaderPair when order matters, as it usually does for impersonation. Anything else panics, so a caller finds out immediately rather than sending a request with headers silently missing.

type Headers 

type Headers []string

Headers is an ordered, case-insensitive HTTP header collection. It mirrors the behaviour of curl_cffi/httpx Headers: insertion order is preserved, lookups are case-insensitive and repeated keys can be retrieved with GetList.

It is a slice of "Name: Value" lines, so it can be written directly as a composite literal, which is how headers appear in HTTP and on a command line: 

requests.Headers{
	"Accept: application/json",
	"X-Custom: value",
}

A value's leading and trailing spaces are not significant and are dropped when the line is read back, which matches how an HTTP parser treats them.

func NewHeaders 

func NewHeaders(h HeaderTypes) *Headers

NewHeaders builds ordered Headers from any supported input.

func (*Headers) Add 

func (h *Headers) Add(name, value string)

Add appends a value, keeping any existing values for the same key.

func (*Headers) Clone 

func (h *Headers) Clone() *Headers

Clone returns a deep copy.

func (*Headers) Del 

func (h *Headers) Del(name string)

Del removes all values for name.

func (*Headers) Get 

func (h *Headers) Get(name string) string

Get returns the values for name concatenated with ", " (RFC 7230 style).

func (*Headers) GetList 

func (h *Headers) GetList(name string) []string

GetList returns all values for name, in order.

func (*Headers) Has 

func (h *Headers) Has(name string) bool

Has reports whether name is present.

func (*Headers) Items 

func (h *Headers) Items() map[string]string

Items returns unique keys mapped to their concatenated values.

func (*Headers) Keys 

func (h *Headers) Keys() []string

Keys returns the unique header names in first-seen order.

func (*Headers) Len 

func (h *Headers) Len() int

Len returns the number of header lines.

func (*Headers) MultiItems 

func (h *Headers) MultiItems() []HeaderPair

MultiItems returns every header line in order, without concatenation.

func (*Headers) Set 

func (h *Headers) Set(name, value string)

Set replaces any existing value for name, keeping its position, or appends. The stored line takes the spelling of name that is passed in.

func (*Headers) String 

func (h *Headers) String() string

func (*Headers) Update 

func (h *Headers) Update(other HeaderTypes)

Update merges another header set, replacing any existing keys in place.

type ImpersonateError 

type ImpersonateError struct{ *RequestException }

ImpersonateError indicates the requested impersonation target is unknown or the TLS configuration failed.

type IncompleteRead 

type IncompleteRead struct{ *HTTPError }

IncompleteRead indicates the body was truncated.

type InterfaceError 

type InterfaceError struct{ *RequestException }

InterfaceError indicates the requested outgoing interface could not be used.

type InvalidSchema 

type InvalidSchema struct{ *RequestException }

InvalidSchema indicates an unsupported URL scheme.

type InvalidURL 

type InvalidURL struct{ *RequestException }

InvalidURL indicates a malformed URL.

type Option 

type Option func(*config)

Option configures a Session or a single request.

func WithAcceptEncoding 

func WithAcceptEncoding(v string) Option

WithAcceptEncoding sets the Accept-Encoding header. Pass an empty string to remove it and let the transport handle decompression.

func WithAkamai 

func WithAkamai(s string) Option

WithAkamai sets a custom Akamai HTTP/2 fingerprint string (advanced).

func WithAllowRedirects 

func WithAllowRedirects(v bool) Option

WithAllowRedirects controls whether redirects are followed.

func WithAuth 

func WithAuth(username, password string) Option

WithAuth sets HTTP basic auth credentials.

func WithBaseURL 

func WithBaseURL(u string) Option

WithBaseURL sets a base URL used to resolve relative request URLs.

func WithBrowser 

func WithBrowser() Option

WithBrowser loads the URL in the full browser instead of the fast client, so the page's scripts run and the body is the document as it settled. It is the option form of Request.Browser, and needs the browser package imported, which is what installs it.

func WithCert 

func WithCert(certFile, keyFile string) Option

WithCert sets a client certificate and key file for mutual TLS.

func WithContent 

func WithContent(v any) Option

WithContent sets the raw request body, allowing streaming io.Reader bodies.

func WithContentCallback 

func WithContentCallback(fn func([]byte) error) Option

WithContentCallback is invoked for every body chunk as it arrives.

func WithCookies 

func WithCookies(cookieTypes CookieTypes) Option

WithCookies sets request cookies.

func WithData 

func WithData(v any) Option

WithData sets the request body. Accepts string, []byte, io.Reader, map[string]string, url.Values or Params (the latter are form-encoded).

func WithDebug 

func WithDebug(v bool) Option

WithDebug enables verbose transport logging.

func WithDefaultEncoding 

func WithDefaultEncoding(enc string) Option

WithDefaultEncoding sets the charset used to decode responses that lack an explicit charset.

func WithDefaultHeaders 

func WithDefaultHeaders(v bool) Option

WithDefaultHeaders controls whether the impersonated browser's default headers are applied. User supplied headers always win.

func WithDiscardCookies 

func WithDiscardCookies(v bool) Option

WithDiscardCookies prevents server cookies from updating the session.

func WithExtraFP 

func WithExtraFP(fp *ExtraFingerprints) Option

WithExtraFP sets extra fingerprint overrides.

func WithHTTPVersion 

func WithHTTPVersion(v string) Option

WithHTTPVersion selects the HTTP version: "v1", "v2", "v3".

func WithHeader 

func WithHeader(name, value string) Option

WithHeader sets a single request header.

func WithHeaders 

func WithHeaders(h HeaderTypes) Option

WithHeaders sets request headers.

func WithImpersonate 

func WithImpersonate(name string) Option

WithImpersonate selects a browser fingerprint preset to mimic.

func WithInterface 

func WithInterface(name string) Option

WithInterface binds the outgoing connection to a network interface or local IP.

func WithJA3 

func WithJA3(s string) Option

WithJA3 sets a custom JA3 fingerprint string (advanced).

func WithJSON 

func WithJSON(v any) Option

WithJSON sets a JSON request body and the application/json content type.

func WithMaxRecvSpeed 

func WithMaxRecvSpeed(n int) Option

WithMaxRecvSpeed limits the download rate in bytes per second.

func WithMaxRedirects 

func WithMaxRedirects(n int) Option

WithMaxRedirects sets the maximum number of redirects (use -1 for unlimited).

func WithParams 

func WithParams(p any) Option

WithParams sets the query string. Accepts Params, url.Values, map[string]string or map[string][]string.

func WithProxies 

func WithProxies(p map[string]string) Option

WithProxies sets per-scheme or per-host proxy URLs, e.g. {"http": "http://...", "https": "http://...", "all": "..."}.

func WithProxy 

func WithProxy(proxyURL string) Option

WithProxy sets a single proxy URL used for all schemes.

func WithProxyAuth 

func WithProxyAuth(username, password string) Option

WithProxyAuth sets proxy credentials.

func WithQuote 

func WithQuote(v any) Option

WithQuote controls URL percent-encoding behaviour (parity with curl_cffi's quote parameter). Pass false to disable automatic encoding.

Not implemented: setting it makes the request fail with an *UnsupportedOptionError rather than silently ignoring it.

func WithRaiseForStatus 

func WithRaiseForStatus(v bool) Option

WithRaiseForStatus makes the request return an error for 4xx/5xx responses.

func WithReferer 

func WithReferer(r string) Option

WithReferer sets the Referer header.

func WithRetry 

func WithRetry(n int) Option

WithRetry sets the number of retries for failed requests.

func WithStream 

func WithStream(v bool) Option

WithStream keeps the response body unread so it can be consumed incrementally.

func WithTimeout 

func WithTimeout(d time.Duration) Option

WithTimeout sets the total request timeout.

func WithTimeoutSeconds 

func WithTimeoutSeconds(s float64) Option

WithTimeoutSeconds sets the total request timeout in seconds.

func WithTrustEnv 

func WithTrustEnv(v bool) Option

WithTrustEnv controls whether http_proxy/https_proxy environment variables are honoured when no explicit proxy is configured.

func WithVerify 

func WithVerify(v bool) Option

WithVerify controls TLS certificate verification.

type Param 

type Param struct {
	Key   string
	Value string
}

Param is one query string key/value pair.

type Params 

type Params []Param

Params is an ordered query parameter list. Use Param to preserve duplicates.

func (Params) Encode 

func (p Params) Encode() string

Encode renders Params as a query string.

type ProxyError 

type ProxyError struct{ *RequestException }

ProxyError indicates a proxy failure.

type Request 

type Request struct {
	// Method is the HTTP method. Empty means GET.
	Method string
	// URL is the target. A URL without a scheme defaults to https.
	URL string

	// Headers accepts anything the WithHeaders option accepts, including
	// []string{"Name: Value"} and map[string]string.
	Headers HeaderTypes
	// Params accepts anything the WithParams option accepts.
	Params any
	// Cookies accepts anything the WithCookies option accepts.
	Cookies CookieTypes

	// Body is the raw request body. JSON wins when both are set.
	Body []byte
	// JSON is marshalled as the request body and sets application/json.
	JSON any

	// Browser loads the URL in the full browser instead of the fast client:
	// scripts run, the page settles, and the response body is the document as
	// it ended up. It needs the browser package linked in, which importing the
	// module root does; a program that imports requests alone is told so.
	//
	// Only a GET of a URL has a browser path, and it is slower by orders of
	// magnitude, so leave it off unless the page needs scripts.
	Browser bool

	// Impersonate selects the fingerprint target, for example
	// impersonate.Chrome131. Empty uses the session default.
	Impersonate string

	// Timeout overrides the session timeout when it is non-zero.
	Timeout time.Duration
	// Proxy overrides the session proxy when it is non-empty.
	Proxy string
}

Request describes an HTTP request as a value. It is what Response.Request reports for a request that was sent, and what Session.Send accepts to send one, so a request can be built, stored and passed around rather than being a list of options at the call site.

Only Method and URL are required; every other field falls back to the session's setting when it is left at its zero value.

func ToRequest 

func ToRequest(v RequestTypes) (Request, error)

ToRequest turns an accepted request input - a Request, a *Request or a URL string - into a Request. It is what Send, Browse and the method helpers run their input through, exported for a caller that needs the value: the browser package opens the page object behind the same input.

type RequestException 

type RequestException struct {
	Msg      string
	Code     int
	Response *Response
}

RequestException is the base error for the requests package. It carries an optional associated *Response and a curl-like numeric code for parity with curl_cffi.

func (*RequestException) Error 

func (e *RequestException) Error() string

func (*RequestException) Unwrap 

func (e *RequestException) Unwrap() error

type RequestTypes 

type RequestTypes interface{}

RequestTypes is the set of accepted request inputs: a Request value, a *Request, or a URL string, which means a GET with the session's defaults.

It exists so the common case needs no literal - Send(url) and Browse(url) are one word each - while a request that sets several fields is still written as one. The other inputs in this package take several shapes the same way: HeaderTypes, CookieTypes and Params.

type Response 

type Response struct {
	URL             string
	Content         []byte
	StatusCode      int
	Reason          string
	Ok              bool
	Headers         *Headers
	Cookies         *Cookies
	Elapsed         time.Duration
	DefaultEncoding string
	RedirectCount   int
	RedirectURL     string
	HTTPVersion     int
	PrimaryIP       string
	PrimaryPort     int
	LocalIP         string
	LocalPort       int
	History         []*Response
	Request         *Request
	DownloadSize    int64
	UploadSize      int64
	HeaderSize      int64
	RequestSize     int64

	// Body is set when the request used Stream=true. It must be closed by the
	// caller (or via Close).
	Body io.ReadCloser
	// contains filtered or unexported fields
}

Response holds everything the server sent back, mirroring curl_cffi.requests.Response.

func Browse 

func Browse(req RequestTypes, opts ...Option) (*Response, error)

Browse loads a request in the browser in a throwaway session, spelled like Send with the path in the name. It needs the browser package imported, which is what installs it: 

rsp, err := Browse("https://quotes.toscrape.com/js/")

func Delete 

func Delete(rawURL string, opts ...Option) (*Response, error)

Delete sends a DELETE request in a throwaway session.

func Do 

func Do(method, rawURL string, opts ...Option) (*Response, error)

Do sends a request in a throwaway session, mirroring curl_cffi's module-level requests.request helper.

func Get 

func Get(rawURL string, opts ...Option) (*Response, error)

Get sends a GET request in a throwaway session. 

func Head(rawURL string, opts ...Option) (*Response, error)

Head sends a HEAD request in a throwaway session.

func NewResponse 

func NewResponse() *Response

NewResponse returns an empty response.

func Options 

func Options(rawURL string, opts ...Option) (*Response, error)

Options sends an OPTIONS request in a throwaway session.

func Patch 

func Patch(rawURL string, opts ...Option) (*Response, error)

Patch sends a PATCH request in a throwaway session.

func Post 

func Post(rawURL string, opts ...Option) (*Response, error)

Post sends a POST request in a throwaway session.

func Put 

func Put(rawURL string, opts ...Option) (*Response, error)

Put sends a PUT request in a throwaway session.

func Send 

func Send(req RequestTypes, opts ...Option) (*Response, error)

Send performs req in a throwaway session, so a Request built as a value can be sent without a Session of its own: 

rsp, err := Send(Request{
	Method:      "GET",
	URL:         "https://httpbun.com/get",
	Impersonate: Chrome146,
})

It reads the whole body before returning, like the other helpers here. Use Session.Send instead when cookies and connections should be reused across requests.

func Trace 

func Trace(rawURL string, opts ...Option) (*Response, error)

Trace sends a TRACE request in a throwaway session.

func (*Response) CharsetEncoding 

func (r *Response) CharsetEncoding() string

CharsetEncoding returns the charset from the Content-Type header, if any.

func (*Response) Close 

func (r *Response) Close() error

Close closes a streaming body, if any.

func (*Response) ContentType 

func (r *Response) ContentType() string

ContentType returns the Content-Type header value.

func (*Response) Encoding 

func (r *Response) Encoding() string

Encoding returns the charset used to decode Text.

func (*Response) HasHeader 

func (r *Response) HasHeader(name string) bool

HasHeader reports whether a response header exists.

func (*Response) IsRedirect 

func (r *Response) IsRedirect() bool

IsRedirect reports whether this is a well-formed redirect response.

func (*Response) IterContent 

func (r *Response) IterContent() (func() ([]byte, error, bool), error)

IterContent yields the response body in chunks. In non-stream mode it yields the buffered content once. In stream mode it reads from the network.

func (*Response) IterLines 

func (r *Response) IterLines(delimiter string) ([]string, error)

IterLines iterates over the response body line by line.

func (*Response) JSON 

func (r *Response) JSON(v any) error

JSON unmarshals the body into v using encoding/json.

func (*Response) Len 

func (r *Response) Len() int

Len returns the response body length in bytes.

func (*Response) RaiseForStatus 

func (r *Response) RaiseForStatus() error

RaiseForStatus returns an *HTTPError if the response status is not in [200, 400), matching requests/curl_cffi.

func (*Response) ReadAll 

func (r *Response) ReadAll() ([]byte, error)

ReadAll consumes and returns the rest of a streaming body.

func (*Response) SetEncoding 

func (r *Response) SetEncoding(enc string) error

SetEncoding overrides the decode charset. It must be called before Text.

func (*Response) String 

func (r *Response) String() string

String implements fmt.Stringer.

func (*Response) Text 

func (r *Response) Text() string

Text decodes Content using Encoding, replacing invalid bytes.

type SSLError 

type SSLError struct{ *ConnectionError }

SSLError indicates a TLS failure.

type Session 

type Session struct {
	// contains filtered or unexported fields
}

Session is a requests-like client with a shared cookie jar and connection pool. It is safe for concurrent use.

func NewSession 

func NewSession(opts ...Option) *Session

NewSession creates a Session. Options set here become defaults for every request and can be overridden per request.

func (*Session) Browse 

func (s *Session) Browse(req RequestTypes, opts ...Option) (*Response, error)

Browse loads a request in the browser, which is Send with Browser set: the name says which path it takes, so the two read alike: 

sess.Send("https://example.com/")    // impersonated HTTP
sess.Browse("https://example.com/")  // the same URL, rendered

It takes anything Send does, and Get with WithBrowser reaches the same page.

func (*Session) Close 

func (s *Session) Close()

Close releases idle connections held by the session.

func (*Session) Cookies 

func (s *Session) Cookies() *Cookies

Cookies returns the live session cookie jar.

func (*Session) Delete 

func (s *Session) Delete(rawURL string, opts ...Option) (*Response, error)

Delete sends a DELETE request.

func (*Session) Get 

func (s *Session) Get(rawURL string, opts ...Option) (*Response, error)

Get sends a GET request.

func (*Session) Head 

func (s *Session) Head(rawURL string, opts ...Option) (*Response, error)

Head sends a HEAD request.

func (*Session) Headers 

func (s *Session) Headers() *Headers

Headers returns the session default headers. Mutating it affects subsequent requests.

func (*Session) Options 

func (s *Session) Options(rawURL string, opts ...Option) (*Response, error)

Options sends an OPTIONS request.

func (*Session) Patch 

func (s *Session) Patch(rawURL string, opts ...Option) (*Response, error)

Patch sends a PATCH request.

func (*Session) Post 

func (s *Session) Post(rawURL string, opts ...Option) (*Response, error)

Post sends a POST request.

func (*Session) Put 

func (s *Session) Put(rawURL string, opts ...Option) (*Response, error)

Put sends a PUT request.

func (*Session) Request 

func (s *Session) Request(method, rawURL string, opts ...Option) (*Response, error)

Request sends an HTTP request using the session defaults plus opts.

func (*Session) Send 

func (s *Session) Send(req RequestTypes, opts ...Option) (*Response, error)

Send performs the request described by req. It is the struct-shaped counterpart to Request(method, url, opts...): the same request can be built as a value, stored, or filled in from configuration, instead of being spread across options at the call site. 

req := requests.Request{
	Method:      "GET",
	URL:         "https://httpbun.com/get",
	Headers:     []string{"Accept: application/json"},
	Impersonate: impersonate.Chrome131,
}
resp, err := sess.Send(req)

Fields left at their zero value fall back to the session's defaults, so an empty Method means GET and an empty Timeout means the session timeout.

func (*Session) SetCookies 

func (s *Session) SetCookies(c CookieTypes)

SetCookies replaces the session cookie jar.

func (*Session) Trace 

func (s *Session) Trace(rawURL string, opts ...Option) (*Response, error)

Trace sends a TRACE request.

func (*Session) UserAgent 

func (s *Session) UserAgent() string

UserAgent reports the User-Agent this session sends: the impersonated browser's, when a preset is selected, and the curl default otherwise. A caller that also exposes a user agent to page scripts uses this so that what the server is told and what navigator.userAgent reports are the same browser, rather than one desktop Chrome to the server and another to the page.

type SessionClosed 

type SessionClosed struct{ *RequestException }

SessionClosed indicates the session was already closed.

type Timeout 

type Timeout struct{ *RequestException }

Timeout indicates the request timed out.

func (*Timeout) Is 

func (e *Timeout) Is(target error) bool

Is lets errors.Is(err, ErrTimeout) succeed for Timeout errors.

type TooManyRedirects 

type TooManyRedirects struct{ *RequestException }

TooManyRedirects indicates the redirect limit was exceeded.

type UnrewindableBodyError 

type UnrewindableBodyError struct{ *RequestException }

UnrewindableBodyError indicates the body could not be replayed for a redirect or retry.

type UnsupportedOptionError 

type UnsupportedOptionError struct{ *RequestException }

UnsupportedOptionError indicates an option that curl_cffi accepts but that this port cannot honour. It is returned rather than ignored on purpose: silently dropping a fingerprint option would send a request that does not match what the caller asked for, which is worse than failing outright.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.