Documentation
¶
Overview ¶
Package scraper fetches and parses documentation pages for indexing.
Index ¶
- Constants
- Variables
- func Fetch(ctx context.Context, client *http.Client, src Source) ([]db.Doc, error)
- func IsTransientAgentError(err error) bool
- func ParseMarkdown(libID, sourceName, content string) []db.Doc
- func ParseRST(libID, sourceName, content string) []db.Doc
- type Agent
- type Config
- type FetchOneResult
- func FetchOne(ctx context.Context, client *http.Client, libID, url string) (FetchOneResult, error)
- func FetchOneViaAgent(ctx context.Context, client *http.Client, agent *Agent, libID, url string) (FetchOneResult, error)
- func FetchOneViaGithubRST(ctx context.Context, client *http.Client, libID, url string) (FetchOneResult, error)
- type HTTPStatusError
- type LibrarySource
- type ResolvedSource
- type Source
- type VersionEntry
Constants ¶
const ( EnvAgentEndpoint = "DEADZONE_AGENT_ENDPOINT" EnvAgentModel = "DEADZONE_AGENT_ENDPOINT_MODEL" EnvAgentAPIKey = "DEADZONE_AGENT_ENDPOINT_API_KEY" )
Environment variables consumed by NewAgentFromEnv. Documented in the README "Scraping non-trivial doc sources" section. The agent endpoint is per-user infrastructure (Ollama on localhost, a shared corporate vLLM, OpenAI proper, ...), not project metadata, which is why these live in env rather than libraries_sources.yaml.
const ( // KindGithubMD is the fast path: HTTP GET on raw markdown URLs and // straight into ParseMarkdown. No LLM, no preprocessing. KindGithubMD = "github-md" // KindGithubRST mirrors KindGithubMD for projects that ship docs as // reStructuredText in the source repo (cpython, Django, NumPy, …). // HTTP GET → ParseRST → db.Doc. No LLM. See #95. KindGithubRST = "github-rst" // KindScrapeViaAgent delegates content → clean markdown extraction // to an OpenAI-compatible chat completions endpoint (Ollama, vLLM, // LocalAI, OpenAI, ...). The catch-all path for HTML doc sites and // any other format that isn't trivially raw markdown. See #27. KindScrapeViaAgent = "scrape-via-agent" )
Kind discriminators for LibrarySource.Kind. All branches feed the same downstream pipeline (parse → embed → store); they only differ in how the source markup is obtained and which parser turns it into db.Doc entries.
Variables ¶
var ErrAgentNotConfigured = errors.New("agent endpoint not configured (set " + EnvAgentEndpoint + " and " + EnvAgentModel + ")")
ErrAgentNotConfigured is returned by NewAgentFromEnv when the required env vars are missing. Wrap with errors.Is to detect — the scraper main loop uses it to print a single actionable startup error instead of a generic env-var dump.
var ErrAgentVerificationFailed = errors.New("agent output failed code-block verification")
ErrAgentVerificationFailed is returned by FetchOneViaAgent when the LLM's output contains a fenced code block that does not appear in the source content (likely a hallucination). The doc is dropped and the failure is logged at scraper.agent_verification_failed.
var ErrPDFNotSupportedYet = errors.New("application/pdf input is not supported yet (planned follow-up to #27)")
ErrPDFNotSupportedYet is returned by preprocess for application/pdf content. The slot is reserved deliberately: the architecture in #27 is designed so adding PDF is a single new case in this switch plus a pure-Go text extractor, not a rearchitecture. Until that follow-up lands the scraper refuses PDFs loudly rather than silently routing raw bytes through the LLM.
Functions ¶
func Fetch ¶
Fetch downloads each URL in src and returns the concatenated Docs. Implemented as a thin loop over FetchOne so callers that just want "give me everything" don't have to deal with per-URL bookkeeping.
func IsTransientAgentError ¶
IsTransientAgentError reports whether err represents a transient failure that should soft-skip the URL rather than abort the whole lib. Matches: context.DeadlineExceeded, net.Error.Timeout, ECONNRESET / EPIPE, unexpected EOF during response read, and 5xx HTTP status. Callers in cmd/scraper combine this with ErrAgentVerificationFailed (intentional per-URL drop) under a shared skippedThisLib ceiling.
func ParseMarkdown ¶
ParseMarkdown splits raw markdown content into db.Doc values by H2 headings.
Rules:
- Text before the first H2 is emitted as one Doc, titled from the H1 heading (the first "# …" line) or sourceName if no H1 is found.
- Each H2 section becomes one Doc; the title is the H2 heading text.
- "##" lines inside backtick code fences (``` or ~~~) are ignored.
- Docs with empty content (whitespace-only) are dropped.
func ParseRST ¶
ParseRST chunks a reStructuredText document into db.Doc entries, one per section heading. Mirrors ParseMarkdown's contract for downstream pipeline compatibility — see scraper.go for the markup-agnostic Doc shape.
Rules:
- A section heading is any non-empty line followed by an underline made of repeated rstAdornmentChars (>=3 chars). Overline form is not detected — cpython/Django/NumPy stdlib docs use underline only.
- One Doc per heading; nested subsections become flat sibling Docs (no parent linkage), same as ParseMarkdown's H2 split.
- Doc.Title is the heading text without the underline characters.
- Code blocks (`.. code-block:: lang`, `.. sourcecode::`, `.. code::`, and trailing `::` literal blocks) are preserved verbatim by indentation tracking.
- Sphinx noise: `.. _label:` targets dropped; `:role:` cross-refs collapsed to their visible text.
- Unknown `.. directive::` lines are kept verbatim so a downstream search query for "seealso" still has signal.
- Text before the first heading is emitted as a single Doc titled sourceName, matching ParseMarkdown's preamble behavior.
Types ¶
type Agent ¶
type Agent struct {
// contains filtered or unexported fields
}
Agent drives LLM-backed content extraction via an OpenAI-compatible /v1/chat/completions endpoint. Deadzone does not host an LLM — the user brings their own runtime (Ollama, vLLM, LocalAI, OpenAI, ...). All Agent does is build the request, ship it, and parse the response.
Concurrency: Agent is safe for concurrent use as long as the embedded http.Client is. The default constructor wires a fresh http.Client per Agent, so you get that for free.
func NewAgent ¶
NewAgent is the explicit constructor used by tests (which inject an httptest.Server URL) and by NewAgentFromEnv. Passing client == nil gives you a fresh http.Client with agentDefaultTimeout.
func NewAgentFromEnv ¶
NewAgentFromEnv constructs an Agent from the three DEADZONE_AGENT_* env vars. Endpoint and model are required; the API key is optional (Ollama and most local runtimes don't need one). Returns ErrAgentNotConfigured if either required var is unset, so callers can distinguish "user forgot to set env" from "endpoint is bad".
func (*Agent) Endpoint ¶
Endpoint returns the base URL the agent talks to. Exposed for the startup log line so an operator can see which runtime they wired up.
func (*Agent) Extract ¶
Extract normalizes content into clean Markdown via the LLM.
contentType drives a one-line hint prepended to the user message so smaller models know what they're looking at. The system prompt itself is identical regardless of input type — the rules (preserve code verbatim, drop chrome, no commentary) don't depend on whether the source was HTML or PDF text.
Inputs longer than agentInputMaxChars are truncated to that exact length and a slog.Warn is emitted via slog.Default(). v1 ships a single constant cap; smart chunking is a follow-up listed in #27.
func (*Agent) Ping ¶
Ping sends a trivial chat completion to the configured endpoint to verify it is reachable, the model is loaded, and the API key (if any) is accepted. Called once at scraper startup when at least one source has kind=scrape-via-agent, so the operator finds out about a misconfigured endpoint before any URLs are processed.
Ping uses the same code path as Extract — same URL, same auth, same response shape — so a successful Ping implies the actual extraction calls have a working transport.
type Config ¶
type Config struct {
Libraries []LibrarySource `yaml:"libraries"`
}
Config is the parsed libraries_sources.yaml file.
func LoadConfig ¶
LoadConfig reads, parses, and validates a libraries_sources.yaml file. All validation runs at parse time so a misconfigured registry fails the scraper immediately rather than mid-run.
func (*Config) Resolve ¶
func (c *Config) Resolve(libFilter, versionFilter string) []ResolvedSource
Resolve flattens every entry in the config into ResolvedSources, applying the (libFilter, versionFilter) filter pair introduced in #113:
- libFilter == "" matches every resolved entry (versionFilter is ignored in this case; the caller is expected to reject that combination as a usage error before calling in).
- libFilter != "", versionFilter == "" matches every expanded version of that base (and the base itself for single-version entries). This is the "scrape every version of terraform" knob.
- libFilter != "", versionFilter != "" matches the exactly-one expanded entry whose (BaseLibID, Version) pair equals the filter. This is the "scrape only terraform v1.14" knob.
type FetchOneResult ¶
FetchOneResult bundles what FetchOne produces for a single URL: the parsed docs plus the raw byte count of the response body. Bytes is exposed so callers can log fetch volume per URL without holding on to (or re-reading) the body.
func FetchOne ¶
FetchOne downloads a single markdown URL with the given http.Client and parses it into docs. Errors are wrapped with the URL so callers can include them verbatim in structured logs.
FetchOne is the per-URL primitive used by Fetch and by cmd/scraper, which drives its own URL loop so it can emit per-URL log events (scraper.fetch / scraper.fetch_failed) and time embedding/insertion alongside the fetch.
func FetchOneViaAgent ¶
func FetchOneViaAgent(ctx context.Context, client *http.Client, agent *Agent, libID, url string) (FetchOneResult, error)
FetchOneViaAgent is the per-URL primitive for the scrape-via-agent kind: HTTP GET, content-type-aware preprocessing, LLM extraction, code-block verification, and ParseMarkdown into docs.
Returns ErrAgentVerificationFailed if the LLM emitted a fenced code block that does not appear verbatim in the source content. The caller is expected to log the failure and skip the doc; the rest of the URLs in the same source still get processed.
Like FetchOne, this is the per-URL primitive used by cmd/scraper so the operator log can carry per-URL fetch / verify / index events instead of one summary per source.
func FetchOneViaGithubRST ¶
func FetchOneViaGithubRST(ctx context.Context, client *http.Client, libID, url string) (FetchOneResult, error)
FetchOneViaGithubRST downloads a single raw .rst URL and parses it into docs. Mirrors FetchOne's contract; only the parser differs.
Like the markdown fast path, this path does no content-type check — raw.githubusercontent.com serves text/plain for .rst and the parser is robust to whatever bytes it gets.
type HTTPStatusError ¶
type HTTPStatusError struct {
Status int
URL string
Body string // truncated response body snippet, may be empty
}
HTTPStatusError carries a non-200 HTTP status code from either the agent endpoint (Agent.do) or the source URL fetch (FetchOneViaAgent). Exported so cmd/scraper can classify 5xx as transient-and-soft-fail vs 4xx as likely-misconfiguration-and-hard-fail via errors.As.
func (*HTTPStatusError) Error ¶
func (e *HTTPStatusError) Error() string
type LibrarySource ¶
type LibrarySource struct {
LibID string
Kind string
URLs []string
Ref string
Versions []VersionEntry
}
LibrarySource is a single entry in libraries_sources.yaml.
A LibrarySource with no Versions describes one library directly. A LibrarySource with Versions is a YAML-level shorthand: at Expand() time it produces one ResolvedSource per version, each with its URLs templated and the version surfaced in the dedicated Version field. The base LibID is never mutated — two versions of the same lib share one LibID and differ only in Version. See #113.
Ref pins URLs to a single upstream git tag or commit SHA when URLs contain the literal "{ref}" token. See #103.
func (LibrarySource) Expand ¶
func (l LibrarySource) Expand() []ResolvedSource
Expand turns one LibrarySource into one or more ResolvedSources.
Single-version entries pass through unchanged: LibID == BaseLibID, Version is empty, URLs are copied with {ref} substituted from the top-level Ref (if present).
Multi-version entries produce one ResolvedSource per version with LibID == BaseLibID (the base, e.g. /hashicorp/terraform) and Version set to the version identifier from the `versions:` map. The {ref} placeholder is substituted from the per-version Ref if set, else from the top-level Ref. Post-#120 there is no {version} substitution: the version identifier is a user-facing label only, and per-version URL divergence is expressed either through {ref} (10 libs) or per-version `urls:` overrides (terraform). The "<base>/<version>" concatenation that earlier builds produced here is gone (#113); downstream code treats (LibID, Version) as the canonical slot.
func (*LibrarySource) UnmarshalYAML ¶ added in v0.2.0
func (l *LibrarySource) UnmarshalYAML(node *yaml.Node) error
UnmarshalYAML parses `versions:` as a mapping:
versions: {v1: {ref: tag1}, v2: {ref: tag2}}
Each value is an object with optional `ref:` and `urls:` per-version overrides. The legacy list form (`versions: [v1, v2]`) is rejected — it carried no per-version metadata and was strictly a subset of the map shape, so keeping it meant two parse paths for the same semantics (see #117). All other fields parse via the standard reflection path. Declaration order is preserved so the scrape loop hits versions in a deterministic order.
type ResolvedSource ¶
type ResolvedSource struct {
LibID string
BaseLibID string
Version string
Kind string
Ref string
URLs []string
}
ResolvedSource is one library, post-version-expansion, ready to scrape.
LibID always equals BaseLibID — it is the base lib_id (e.g. /hashicorp/terraform). Version is empty for single-version entries and carries the version tag (e.g. "v1.14") for multi-version entries. The "<base>/<version>" concat that earlier builds produced here is gone (#113); callers that need a (lib_id, version) slot pass them as two fields. Ref is the effective git ref applied to the URLs (per-version Ref if set, else the top-level LibrarySource.Ref).
BaseLibID is retained as a separate field for readability at call sites — it is always == LibID after #113, but the name documents intent ("the unversioned identity of this lib").
type Source ¶
type Source struct {
LibID string // e.g. "/modelcontextprotocol/go-sdk"
URLs []string // raw markdown URLs to fetch
}
Source describes a library's documentation to scrape.
type VersionEntry ¶ added in v0.2.0
VersionEntry is one element of LibrarySource.Versions after parsing.
Entries come from the map shape `versions: {v1: {ref: tag1}, v2: {ref: tag2}}`. Ref, when set, overrides the top-level Ref for this version; when empty the top-level Ref applies (if any). Declaration order is preserved so scrapes are deterministic.
URLs (see #115) is a per-version override of the parent LibrarySource.URLs. When non-nil, Expand uses it verbatim for this version; when nil, the version inherits the top-level URLs. An explicit empty list is rejected at parse time — inheritance is expressed by omitting the field.
Source Files