cfscrape

README

Read the docs.

Documentation

Overview

A Golang port(?) of https://github.com/Anorov/cloudflare-scrape

Index

• Variables
• func Get(url string) (resp *http.Response, err error)
• func IsCloudflareChallenge(resp *http.Response) bool
• func NewRoundTripper(options *Options) http.RoundTripper
• func NodeExecute(javascript string) (string, error)
• type ChallengeAnswer
  • func SolveChallenge(hostname string, body []byte, executor JSExecutor) (*ChallengeAnswer, error)
• type CookieFilter
  • func CookiesWithName(names ...string) CookieFilter
• type FilteringJar
  • func NewFilteringJar(filter CookieFilter) *FilteringJar
  • func (j *FilteringJar) SetCookies(u *url.URL, cookies []*http.Cookie)
• type JSExecutor
• type JSExecutorFunc
  • func (f JSExecutorFunc) ExecuteJS(s string) (string, error)
• type Options

Variables

var DefaultClient = &http.Client{Transport: NewRoundTripper(nil)}

var (
	// This error is returned if we could not solve cloudflare's challenge.
	// Most likely, this means Cloudflare changed their "challenge"
	ErrCouldNotSolve = errors.New("Could not solve cloudflare challenge")
)

var NodeExecutor = JSExecutorFunc(NodeExecute)

Functions

func Get

func Get(url string) (resp *http.Response, err error)

func IsCloudflareChallenge

func IsCloudflareChallenge(resp *http.Response) bool

func NewRoundTripper

func NewRoundTripper(options *Options) http.RoundTripper

Creates a RoundTripper that solves Cloudflare "I Am Under Attack" / "anti-bot" page.

To use this: create an http.Client with a Transport field set to the RoundTripper that this function creates.

A nil value for options will set reasonable defaults for some definition of reasonable.

But in the spirit of Go, where it is *actually* encouraged for each package to define their own set of random global variables, you may find an existing http.Client defined already as DefaultClient.

func NodeExecute

func NodeExecute(javascript string) (string, error)

Executes a javascript expression with node. This executor makes no security guarantees.

Types

type ChallengeAnswer

type ChallengeAnswer struct {
	Challenge     string
	ChallengePass string
	JsAnswer      string

	ExecutedJavascript string
}

func SolveChallenge

func SolveChallenge(hostname string, body []byte, executor JSExecutor) (*ChallengeAnswer, error)

Given a hostname, the cloudflare anti bot page, and a javascript executor, this returns the "answer" to cloudflare's challenge.

type CookieFilter

type CookieFilter func([]*http.Cookie) []*http.Cookie

func CookiesWithName

func CookiesWithName(names ...string) CookieFilter

type FilteringJar

type FilteringJar struct {
	Filter CookieFilter
	*cookiejar.Jar
}

func NewFilteringJar

func NewFilteringJar(filter CookieFilter) *FilteringJar

func (*FilteringJar) SetCookies

func (j *FilteringJar) SetCookies(u *url.URL, cookies []*http.Cookie)

type JSExecutor

type JSExecutor interface {
	ExecuteJS(javascript string) (string, error)
}

A JSExecutor executes a javascript expression, and returns the resulting string. Note that the javascript that it must execute can be arbitrary.

type JSExecutorFunc

type JSExecutorFunc func(string) (string, error)

func (JSExecutorFunc) ExecuteJS

func (f JSExecutorFunc) ExecuteJS(s string) (string, error)

type Options

type Options struct {
	// If nil, this uses the default executor, which uses node.js
	Executor JSExecutor

	// If nil, this uses the default http Transport
	RoundTripper http.RoundTripper
}