browser

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 20, 2026 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Overview

Package browser 提供统一的浏览器池抽象层。

支持多种浏览器后端（Rod、Browserless、Surf 等）， 上层调用方通过统一的 Browser 接口操作页面，无需关心底层实现。

架构参考 database/sql 的 Strategy Pattern：

  • 父包定义接口（Browser、Provider）和通用能力（Pool、Fetcher）
  • 子包（rod/、browserless/、surf/）提供具体实现
  • 用户按需 import 子包，配置驱动注册

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrPoolClosed 表示浏览器池已关闭
	ErrPoolClosed = errors.New("browser: pool closed")

	// ErrAcquireTimeout 表示获取浏览器实例超时
	ErrAcquireTimeout = errors.New("browser: acquire timeout")

	// ErrProviderNotFound 表示未注册对应类型的 Provider
	ErrProviderNotFound = errors.New("browser: provider not found")

	// ErrUnsupported 表示当前浏览器实现不支持该操作
	ErrUnsupported = errors.New("browser: operation not supported")

	// ErrAllBlocked 表示所有浏览器类型都被风控拦截
	ErrAllBlocked = errors.New("browser: all browser types blocked")

	// ErrNoHealthyEndpoint 表示没有可用的健康远程节点
	ErrNoHealthyEndpoint = errors.New("browser: no healthy endpoint available")

	// ErrWAFBlocked 表示页面被 WAF/风控系统拦截
	ErrWAFBlocked = errors.New("browser: waf blocked")
)
View Source 
var DefaultPoolConfig = PoolConfig{
	MaxInstances:    8,
	IdleTimeout:     5 * time.Minute,
	AcquireTimeout:  30 * time.Second,
	HealthCheckFreq: 30 * time.Second,
	PingTimeout:     2 * time.Second,
}

DefaultPoolConfig 默认池配置

View Source 
var DefaultViewport = Viewport{
	Width:  1920,
	Height: 1080,
	Scale:  1.0,
	Mobile: false,
}

DefaultViewport 默认桌面视口

View Source 
var DefaultWorkerConfig = WorkerConfig{
	Concurrency: 4,
}

DefaultWorkerConfig 默认 Worker 配置

View Source 
var Locales = []string{
	"en-US", "en-GB", "en-CA", "en-AU",
	"de-DE", "fr-FR", "ja-JP", "zh-CN",
}

Locales 预定义的语言区域列表

View Source 
var Timezones = []string{
	"America/New_York", "America/Chicago", "America/Denver",
	"America/Los_Angeles", "America/Phoenix",
	"Europe/London", "Europe/Paris", "Europe/Berlin",
	"Asia/Tokyo", "Asia/Shanghai", "Asia/Singapore",
	"Australia/Sydney",
}

Timezones 预定义的时区列表

View Source 
var UserAgents = []string{
	"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
	"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
	"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36",
	"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36",
	"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
}

UserAgents 预定义的 User-Agent 列表

View Source 
var Viewports = []Viewport{
	{Width: 1920, Height: 1080, Scale: 1.0},
	{Width: 1366, Height: 768, Scale: 1.0},
	{Width: 1536, Height: 864, Scale: 1.25},
	{Width: 1440, Height: 900, Scale: 1.0},
	{Width: 1680, Height: 1050, Scale: 1.0},
	{Width: 2560, Height: 1440, Scale: 1.0},
	{Width: 1280, Height: 720, Scale: 1.0},
	{Width: 1600, Height: 900, Scale: 1.0},
}

Viewports 预定义的桌面视口列表

Functions

func CheckWAF 

func CheckWAF(ctx context.Context, b Browser, detector BlockDetector) error

CheckWAF 检测当前页面是否被 WAF 拦截

func IsConnectionError 

func IsConnectionError(err error) bool

IsConnectionError 判断是否为底层连接级错误

func PlatformFromUA 

func PlatformFromUA(ua string) string

PlatformFromUA 根据 User-Agent 推断平台标识

func PlatformOverrideScript 

func PlatformOverrideScript(platform string) string

PlatformOverrideScript 生成 navigator.platform 覆盖脚本

func RandomLocale 

func RandomLocale() string

RandomLocale 随机选择一个语言区域

func RandomTimezone 

func RandomTimezone() string

RandomTimezone 随机选择一个时区

func RandomUserAgent 

func RandomUserAgent() string

func TruncateUA 

func TruncateUA(ua string, maxLen int) string

TruncateUA 截断过长的 User-Agent 用于日志输出

Types

type AcquireOpts 

type AcquireOpts struct {
	Type      Type
	Viewport  Viewport
	UserAgent string
	Locale    string
	Proxy     string
}

AcquireOpts 获取浏览器实例的选项

type BlockDetector 

type BlockDetector interface {
	Detect(html, title string, statusCode int) BlockResult
}

BlockDetector 风控检测接口

type BlockResult 

type BlockResult struct {
	Blocked bool   `json:"blocked"`
	Reason  string `json:"reason"`
	Type    string `json:"type"`
}

BlockResult 风控检测结果

type Browser 

type Browser interface {
	Navigate(ctx context.Context, url string) error
	WaitStable(ctx context.Context) error
	HTML(ctx context.Context) (string, error)
	Text(ctx context.Context) (string, error)
	Title(ctx context.Context) (string, error)
	URL(ctx context.Context) (string, error)
	Screenshot(ctx context.Context) ([]byte, error)
	Eval(ctx context.Context, js string) (string, error)
	// EvalDirect 直接通过 CDP 执行 JS，不等待 navigation lifecycle。
	// 使用场景: 页面目标元素已确认存在，只需操作 DOM (SPA 表单交互、DOM 读取)。
	// 不确定目标元素是否存在时，请先用 WaitSelector 或自行轮询确认后再调用。
	EvalDirect(ctx context.Context, js string) (string, error)
	Click(ctx context.Context, selector string) error
	Type(ctx context.Context, selector, text string) error
	WaitSelector(ctx context.Context, selector string) error
	Cookies(ctx context.Context) ([]Cookie, error)
	SetCookies(ctx context.Context, cookies []Cookie) error
	BrowserType() Type
	Close() error
}

Browser 统一浏览器操作接口

所有浏览器后端（Rod、Browserless、Surf 等）都实现此接口。 不支持的操作应返回 ErrUnsupported。 

type Cookie struct {
	Name     string `json:"name"`
	Value    string `json:"value"`
	Domain   string `json:"domain"`
	Path     string `json:"path"`
	Secure   bool   `json:"secure"`
	HTTPOnly bool   `json:"http_only"`
}

Cookie HTTP cookie

type DefaultBlockDetector 

type DefaultBlockDetector struct{}

DefaultBlockDetector 默认风控检测器

func NewBlockDetector 

func NewBlockDetector() *DefaultBlockDetector

func (*DefaultBlockDetector) Detect 

func (d *DefaultBlockDetector) Detect(html, title string, statusCode int) BlockResult

type DetectResult 

type DetectResult struct {
	SuggestedType Type     `json:"suggested_type"`
	IsSSR         bool     `json:"is_ssr"`
	Score         int      `json:"score"`
	Signals       []string `json:"signals"`
}

DetectResult 页面类型探测结果

type FetchAttempt 

type FetchAttempt struct {
	Type     Type          `json:"type"`
	Duration time.Duration `json:"duration"`
	Blocked  bool          `json:"blocked"`
	Reason   string        `json:"reason"`
	Err      error         `json:"-"`
}

FetchAttempt 单次尝试记录

type FetchOption 

type FetchOption func(*fetchCall)

FetchOption 单次 Fetch 调用的选项

func FetchWithProvider 

func FetchWithProvider(t Type) FetchOption

FetchWithProvider 覆盖本次请求使用的浏览器类型

type FetchResult 

type FetchResult struct {
	HTML         string         `json:"html"`
	Title        string         `json:"title"`
	FinalType    Type           `json:"final_type"`
	Attempts     []FetchAttempt `json:"attempts"`
	TotalLatency time.Duration  `json:"total_latency"`
}

FetchResult 抓取结果

type Fetcher 

type Fetcher struct {
	// contains filtered or unexported fields
}

Fetcher 业务层单一入口，封装 Pool + BlockDetector

func NewFetcher 

func NewFetcher(pool *Pool, detector BlockDetector, opts ...FetcherOption) (*Fetcher, error)

func (*Fetcher) Detect 

func (f *Fetcher) Detect(ctx context.Context, url string) (*DetectResult, error)

Detect 探测目标 URL 的页面类型

func (*Fetcher) Fetch 

func (f *Fetcher) Fetch(ctx context.Context, url string, opts ...FetchOption) (*FetchResult, error)

func (*Fetcher) SetPrimary 

func (f *Fetcher) SetPrimary(t Type)

type FetcherOption 

type FetcherOption func(*Fetcher)

FetcherOption 配置 Fetcher 的函数式选项

func WithFallback 

func WithFallback(types ...Type) FetcherOption

func WithPrimary 

func WithPrimary(t Type) FetcherOption

type Metrics 

type Metrics struct {
	AcquireTotal   atomic.Int64
	AcquireSuccess atomic.Int64
	AcquireFail    atomic.Int64
	CreateTotal    atomic.Int64
	ReleaseTotal   atomic.Int64
	EvictTotal     atomic.Int64
	ReuseTotal     atomic.Int64

	FetchTotal    atomic.Int64
	FetchSuccess  atomic.Int64
	FetchBlocked  atomic.Int64
	FetchFallback atomic.Int64
	FetchAllFail  atomic.Int64
	// contains filtered or unexported fields
}

Metrics 浏览器池运行指标

func NewMetrics 

func NewMetrics() *Metrics

func (*Metrics) LogSummary 

func (m *Metrics) LogSummary()

func (*Metrics) RecordBlockByType 

func (m *Metrics) RecordBlockByType(t Type)

func (*Metrics) RecordFetchByType 

func (m *Metrics) RecordFetchByType(t Type)

func (*Metrics) RecordLatency 

func (m *Metrics) RecordLatency(d time.Duration)

func (*Metrics) Snapshot 

func (m *Metrics) Snapshot() MetricsSnapshot

type MetricsSnapshot 

type MetricsSnapshot struct {
	AcquireTotal   int64          `json:"acquire_total"`
	AcquireSuccess int64          `json:"acquire_success"`
	AcquireFail    int64          `json:"acquire_fail"`
	CreateTotal    int64          `json:"create_total"`
	ReleaseTotal   int64          `json:"release_total"`
	EvictTotal     int64          `json:"evict_total"`
	ReuseTotal     int64          `json:"reuse_total"`
	FetchTotal     int64          `json:"fetch_total"`
	FetchSuccess   int64          `json:"fetch_success"`
	FetchBlocked   int64          `json:"fetch_blocked"`
	FetchFallback  int64          `json:"fetch_fallback"`
	FetchAllFail   int64          `json:"fetch_all_fail"`
	FetchByType    map[Type]int64 `json:"fetch_by_type"`
	BlockByType    map[Type]int64 `json:"block_by_type"`
	AvgLatency     time.Duration  `json:"avg_latency"`
	P50Latency     time.Duration  `json:"p50_latency"`
	P95Latency     time.Duration  `json:"p95_latency"`
	P99Latency     time.Duration  `json:"p99_latency"`
	SuccessRate    float64        `json:"success_rate"`
	FallbackRate   float64        `json:"fallback_rate"`
}

MetricsSnapshot 指标快照 

type NavigateOption func(*NavigateOpts)

func WithEarlyWait 

func WithEarlyWait(d time.Duration) NavigateOption

func WithWaitStable 

func WithWaitStable() NavigateOption
type NavigateOpts struct {
	EarlyWait  time.Duration
	WaitStable bool
}

NavigateOpts 导航选项 

type NavigateResult struct {
	URL     string
	Title   string
	Elapsed time.Duration
}

NavigateResult 导航结果 

func NavigateAndCheck(ctx context.Context, b Browser, url string, detector BlockDetector, options ...NavigateOption) (*NavigateResult, error)

NavigateAndCheck 导航到 URL 并检测 WAF 拦截

type Pool 

type Pool struct {
	// contains filtered or unexported fields
}

Pool 浏览器实例池

func NewPool 

func NewPool(cfg PoolConfig) *Pool

func (*Pool) Acquire 

func (p *Pool) Acquire(ctx context.Context, opts AcquireOpts) (Browser, error)

func (*Pool) Close 

func (p *Pool) Close() error

func (*Pool) FetchWithFallback 

func (p *Pool) FetchWithFallback(ctx context.Context, url string, chain []Type, detector BlockDetector) (*FetchResult, error)

FetchWithFallback 智能降级抓取

func (*Pool) HasProvider 

func (p *Pool) HasProvider(t Type) bool

func (*Pool) Metrics 

func (p *Pool) Metrics() *Metrics

func (*Pool) Register 

func (p *Pool) Register(provider Provider)

func (*Pool) Release 

func (p *Pool) Release(b Browser, lastErr error) error

Release 归还浏览器实例到池中

func (*Pool) Stats 

func (p *Pool) Stats() PoolStats

type PoolConfig 

type PoolConfig struct {
	MaxInstances    int
	IdleTimeout     time.Duration
	AcquireTimeout  time.Duration
	HealthCheckFreq time.Duration
	PingTimeout     time.Duration // 复用空闲实例前的存活探测超时，零值取 2s
}

PoolConfig 浏览器池配置

type PoolStats 

type PoolStats struct {
	Total     int
	Available int
	InUse     int
	ByType    map[Type]int
}

PoolStats 浏览器池统计信息

type Provider 

type Provider interface {
	Type() Type
	Create(ctx context.Context, opts AcquireOpts) (Browser, error)
	HealthCheck(ctx context.Context) error
	Close() error
}

Provider 浏览器实例工厂接口

每种浏览器类型实现一个 Provider，负责创建和管理该类型的实例。

type Task 

type Task struct {
	// URL 目标地址
	URL string

	// Chain 降级链 (如 [TypeSurf, TypeRodHeadless])
	Chain []Type
}

Task 抓取任务

type TaskResult 

type TaskResult struct {
	// URL 目标地址
	URL string `json:"url"`

	// FetchResult 抓取结果 (nil if error)
	FetchResult *FetchResult `json:"fetch_result"`

	// Err 错误 (如果有)
	Err error `json:"-"`

	// Duration 执行耗时
	Duration time.Duration `json:"duration"`
}

TaskResult 任务执行结果

type Type 

type Type int

Type 浏览器类型 

const (
	TypeRodHeadless Type = iota // headless Chrome + stealth
	TypeRodHeaded               // 有头 Chrome
	TypeBrowserless             // Browserless v2 集群
	TypeSurf                    // 纯 HTTP TLS 伪装
	TypeCamoufox                // Camoufox 反检测浏览器 (预留)
)

func (Type) String 

func (t Type) String() string

String 返回浏览器类型的可读名称

type Viewport 

type Viewport struct {
	Width  int
	Height int
	Scale  float64
	Mobile bool
}

Viewport 视口配置

func RandomViewport 

func RandomViewport() Viewport

RandomViewport 随机选择一个桌面视口

type WAFBlockedError 

type WAFBlockedError struct {
	Result BlockResult
	URL    string
}

WAFBlockedError 携带 WAF 拦截详情的结构化错误

func (*WAFBlockedError) Error 

func (e *WAFBlockedError) Error() string

func (*WAFBlockedError) Unwrap 

func (e *WAFBlockedError) Unwrap() error

type Worker 

type Worker struct {
	// contains filtered or unexported fields
}

Worker 并发任务调度器

批量执行抓取任务，通过信号量控制并发数，支持速率限制和结果回调。 结果按原始任务顺序返回。

func NewWorker 

func NewWorker(pool *Pool, detector BlockDetector, cfg WorkerConfig) *Worker

NewWorker 创建 Worker

func (*Worker) Run 

func (w *Worker) Run(ctx context.Context, tasks []Task) []TaskResult

Run 并发执行任务列表，返回所有结果 (保持原始顺序)

ctx 取消时，尚未开始的任务不会执行，已开始的任务会通过 ctx 传播取消信号。

type WorkerConfig 

type WorkerConfig struct {
	// Concurrency 并发 worker 数量
	Concurrency int

	// RateLimit 每秒最大请求数 (0 表示不限制)
	RateLimit int

	// OnResult 每完成一个任务的回调 (可选，在 goroutine 中调用)
	// idx 为任务在原始列表中的索引
	OnResult func(idx int, result TaskResult)
}

WorkerConfig Worker 配置

Source Files

View all Source files

Directories

Path Synopsis
Package browserless 提供基于 Browserless v2 的远程浏览器集群 Provider 实现。
 Package browserless 提供基于 Browserless v2 的远程浏览器集群 Provider 实现。
Package rod 提供基于 go-rod 的 Chrome 浏览器 Provider 实现。
 Package rod 提供基于 go-rod 的 Chrome 浏览器 Provider 实现。
Package surf 提供基于 TLS 指纹伪装的纯 HTTP 客户端 Provider 实现。
 Package surf 提供基于 TLS 指纹伪装的纯 HTTP 客户端 Provider 实现。

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.