README
¶
scrape-kdl go-rod adapter
This nested Go module implements scrapekdl.BrowserAdapter with go-rod while keeping the core module free of browser dependencies.
go get github.com/hsblabs/scrape-kdl/adapters/rod@latest
browser := rod.New().MustConnect()
defer browser.MustClose()
adapter, err := rodadapter.NewBrowser(browser)
if err != nil {
return err
}
defer adapter.Close()
result, err := program.Extract(ctx, inputs, scrapekdl.Options{
Browser: adapter,
AllowJavaScript: true,
})
NewBrowser creates and owns one page but does not own the browser. New(page) wraps a caller-owned page and closes neither page nor browser.
The adapter implements BrowserAdapterLease. Concurrent Program.Extract calls using the same adapter are serialized for the complete navigation/workflow/extraction lifecycle. Separate adapters/pages are recommended for parallel throughput.
CLI
cmd/scrape-kdl-rod executes browser-mode extractors end to end:
scrape-kdl-rod -spec extractor.kdl --input race_id=202401010101 --allow-js --json
The command shares the core CLI's automation conventions:
--input NAME=VALUE— runtime input, typed by the extractor's declarations; repeatable;--session-file FILE|-— headers and cookies as JSON ({"headers": {...}, "cookies": [...]}); plaintext--header/--cookieflags are rejected;--timeout,--user-agent,--json,-o/--out FILE|-;--allow-private-hosts— allow an initial navigation target that is not globally reachable, rejected by default;--allow-js,--headless(default true),--version.
--json emits exactly one JSON document on standard output for success, processing failure, or usage failure after the flag is recognized. It cannot be combined with --out FILE; use --out - or omit --out. Help is written to standard output and exits 0. Exit statuses follow the core CLI: 0 success, 1 processing failure, 2 usage error, 130 for SIGINT, and 143 for SIGTERM.
The default URL policy covers the initial navigation target only. Browser redirects, subresources, service workers, and page-initiated traffic require browser-context or host-level network controls. --allow-private-hosts disables even the initial-target safeguard.
See docs/rod-adapter.md for the complete lifecycle, CLI, output, cancellation, and security contract.
Verification
Contract verification without downloading go-rod:
make test-rod-contract
Real dependency verification:
make test-rod
Chromium E2E:
make test-rod-e2e
The module is versioned independently using tags such as adapters/rod/v0.1.0.
Documentation
¶
Overview ¶
Package rodadapter provides a go-rod implementation of scrape-kdl's BrowserAdapter interface.
Index ¶
- type Adapter
- func (a *Adapter) Acquire(ctx context.Context) (func(), error)
- func (a *Adapter) Attribute(ctx context.Context, element scrapekdl.BrowserElement, name string) (string, bool, error)
- func (a *Adapter) Click(ctx context.Context, selector string, timeout time.Duration) error
- func (a *Adapter) Close() error
- func (a *Adapter) Evaluate(ctx context.Context, source string, options scrapekdl.BrowserEvaluateOptions) (any, error)
- func (a *Adapter) Fill(ctx context.Context, selector, value string, timeout time.Duration) error
- func (a *Adapter) HTML(ctx context.Context, element scrapekdl.BrowserElement) (string, error)
- func (a *Adapter) Navigate(ctx context.Context, target string, options scrapekdl.BrowserNavigateOptions) error
- func (a *Adapter) Press(ctx context.Context, selector, key string, timeout time.Duration) error
- func (a *Adapter) QueryAll(ctx context.Context, scope scrapekdl.BrowserElement, selector string) ([]scrapekdl.BrowserElement, error)
- func (a *Adapter) QueryLimit(ctx context.Context, scope scrapekdl.BrowserElement, selector string, ...) ([]scrapekdl.BrowserElement, error)
- func (a *Adapter) Scroll(ctx context.Context, x, y float64) error
- func (a *Adapter) Text(ctx context.Context, element scrapekdl.BrowserElement) (string, error)
- func (a *Adapter) WaitFor(ctx context.Context, selector, state string, timeout time.Duration) error
- func (a *Adapter) WaitForNetworkIdle(ctx context.Context, idle, timeout time.Duration) error
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Adapter ¶
type Adapter struct {
// contains filtered or unexported fields
}
Adapter executes browser-mode extractors against one rod page. A single Adapter must not be used for concurrent extractions.
func New ¶
New creates an adapter around an existing page. The page remains owned by the caller and is not closed by Adapter.Close.
func NewBrowser ¶
NewBrowser creates a fresh about:blank page in an existing connected browser. The adapter owns and closes the page, but not the browser.
func (*Adapter) Acquire ¶
Acquire reserves the page for one complete scrape-kdl extraction. The runtime calls this optional capability before navigation and releases it after output extraction, preventing workflow and read operations from interleaving across concurrent Program.Extract calls.
func (*Adapter) QueryAll ¶
func (a *Adapter) QueryAll(ctx context.Context, scope scrapekdl.BrowserElement, selector string) ([]scrapekdl.BrowserElement, error)
func (*Adapter) QueryLimit ¶
func (a *Adapter) QueryLimit(ctx context.Context, scope scrapekdl.BrowserElement, selector string, limit int) ([]scrapekdl.BrowserElement, error)
Source Files
Directories