source

Documentation

Overview

Package source implements the Source interface and its four adapters, decided in https://github.com/cheetahbyte/lore/issues/8: each adapter only resolves identity/versions and fetches raw content — chunking is one shared pipeline stage in the ingest package, not reimplemented per adapter.

Index

• func WithCrawlOptions(ctx context.Context, opts CrawlOptions) context.Context
• type CrawlOptions
• type GitHub
  • func NewGitHub() *GitHub
  • func (g *GitHub) Fetch(ctx context.Context, id identity.ID, version string) ([]RawPage, error)
  • func (g *GitHub) Resolve(ctx context.Context, ref string) (identity.ID, []string, error)
  • func (g *GitHub) Type() identity.SourceType
• type GoPackage
  • func NewGoPackage() *GoPackage
  • func (g *GoPackage) Fetch(ctx context.Context, id identity.ID, version string) ([]RawPage, error)
  • func (g *GoPackage) Resolve(ctx context.Context, ref string) (identity.ID, []string, error)
  • func (g *GoPackage) Type() identity.SourceType
• type LLMsTxt
  • func NewLLMsTxt() *LLMsTxt
  • func (l *LLMsTxt) Fetch(ctx context.Context, id identity.ID, version string) ([]RawPage, error)
  • func (l *LLMsTxt) Resolve(ctx context.Context, ref string) (identity.ID, []string, error)
  • func (l *LLMsTxt) Type() identity.SourceType
• type NPM
  • func NewNPM() *NPM
  • func (n *NPM) Fetch(ctx context.Context, id identity.ID, version string) ([]RawPage, error)
  • func (n *NPM) Resolve(ctx context.Context, ref string) (identity.ID, []string, error)
  • func (n *NPM) Type() identity.SourceType
• type PyPI
  • func NewPyPI() *PyPI
  • func (p *PyPI) Fetch(ctx context.Context, id identity.ID, version string) ([]RawPage, error)
  • func (p *PyPI) Resolve(ctx context.Context, ref string) (identity.ID, []string, error)
  • func (p *PyPI) Type() identity.SourceType
• type RawPage
• type Registry
  • func (r Registry) For(t identity.SourceType) (Source, bool)
• type Source
• type URL
  • func NewURL() *URL
  • func (u *URL) Fetch(ctx context.Context, id identity.ID, version string) ([]RawPage, error)
  • func (u *URL) Resolve(ctx context.Context, ref string) (identity.ID, []string, error)
  • func (u *URL) Type() identity.SourceType

Functions

func WithCrawlOptions

func WithCrawlOptions(ctx context.Context, opts CrawlOptions) context.Context

Types

type CrawlOptions

type CrawlOptions struct {
	Depth   int // hop count from the starting URL; 0 or 1 means "just this page"
	Include []string
	Exclude []string
}

CrawlOptions configures URL.Fetch, sourced from the per-source config decided in issue #3 (config.toml's [sources."url:..."] table). Passed via context since the Source interface itself (issue #8) doesn't carry per-call options — WithCrawlOptions/crawlOptionsFrom are the seam.

type GitHub

type GitHub struct {
	HTTPClient *http.Client
	Token      string
}

GitHub is the github: Source adapter, decided in issue #8. It reads tags/releases for versioning (issue #9) and README/docs markdown files for content, using the unauthenticated GitHub REST API by default (set GITHUB_TOKEN or GH_TOKEN for a higher rate limit).

func NewGitHub

func NewGitHub() *GitHub

func (*GitHub) Fetch

func (g *GitHub) Fetch(ctx context.Context, id identity.ID, version string) ([]RawPage, error)

func (*GitHub) Resolve

func (g *GitHub) Resolve(ctx context.Context, ref string) (identity.ID, []string, error)

func (*GitHub) Type

func (g *GitHub) Type() identity.SourceType

type GoPackage

type GoPackage struct {
	HTTPClient *http.Client
	URLAdapter *URL
}

GoPackage is the pkg.go.dev: Source adapter, decided in issue #8. Versions come from the Go module proxy's @v/list; content is the rendered pkg.go.dev page (HTML), converted the same way the url: adapter converts crawled pages, since pkg.go.dev has no JSON doc API.

func NewGoPackage

func NewGoPackage() *GoPackage

func (*GoPackage) Fetch

func (g *GoPackage) Fetch(ctx context.Context, id identity.ID, version string) ([]RawPage, error)

func (*GoPackage) Resolve

func (g *GoPackage) Resolve(ctx context.Context, ref string) (identity.ID, []string, error)

func (*GoPackage) Type

func (g *GoPackage) Type() identity.SourceType

type LLMsTxt

type LLMsTxt struct {
	HTTPClient *http.Client
}

LLMsTxt is the llms-txt: Source adapter, decided in issue #8. Sites of this type have no version concept, per issue #9 — Resolve always returns exactly one implicit version, "".

func NewLLMsTxt

func NewLLMsTxt() *LLMsTxt

func (*LLMsTxt) Fetch

func (l *LLMsTxt) Fetch(ctx context.Context, id identity.ID, version string) ([]RawPage, error)

func (*LLMsTxt) Resolve

func (l *LLMsTxt) Resolve(ctx context.Context, ref string) (identity.ID, []string, error)

func (*LLMsTxt) Type

func (l *LLMsTxt) Type() identity.SourceType

type NPM

type NPM struct{ HTTPClient *http.Client }

NPM is the npm: Source adapter, decided in issue #8. Versions come from the registry's own version list; per issue #9, the default version prefers the registry's own "latest" dist-tag over a blind semver-max (which would otherwise pick prerelease/canary builds that carry a higher base version than the actual stable release — confirmed against the real react packument, where dist-tags.latest is 19.2.8 but the highest semver-sortable version is a 19.3.0-canary-* build).

func NewNPM

func NewNPM() *NPM

func (*NPM) Fetch

func (n *NPM) Fetch(ctx context.Context, id identity.ID, version string) ([]RawPage, error)

func (*NPM) Resolve

func (n *NPM) Resolve(ctx context.Context, ref string) (identity.ID, []string, error)

func (*NPM) Type

func (n *NPM) Type() identity.SourceType

type PyPI

type PyPI struct{ HTTPClient *http.Client }

PyPI is the pypi: Source adapter, decided in issue #8.

func NewPyPI

func NewPyPI() *PyPI

func (*PyPI) Fetch

func (p *PyPI) Fetch(ctx context.Context, id identity.ID, version string) ([]RawPage, error)

func (*PyPI) Resolve

func (p *PyPI) Resolve(ctx context.Context, ref string) (identity.ID, []string, error)

func (*PyPI) Type

func (p *PyPI) Type() identity.SourceType

type RawPage

type RawPage struct {
	URL         string
	Path        string // file path within a repo, if applicable
	Content     string
	ContentType string // "markdown" | "html" | "text"
}

RawPage is one fetched, not-yet-chunked document.

type Registry

type Registry map[identity.SourceType]Source

Registry looks up a Source by type.

func (Registry) For

func (r Registry) For(t identity.SourceType) (Source, bool)

type Source

type Source interface {
	Type() identity.SourceType

	// Resolve turns a user-typed reference (e.g. "owner/repo" or
	// "react") into a canonical ID and its available versions. Sources
	// with no version concept (llms-txt, url) always return exactly one
	// implicit version: "".
	Resolve(ctx context.Context, ref string) (identity.ID, []string, error)

	// Fetch retrieves the raw content for id at version.
	Fetch(ctx context.Context, id identity.ID, version string) ([]RawPage, error)
}

Source is implemented once per source type (github, npm, pypi, pkg.go.dev, llms-txt, url).

type URL

type URL struct {
	HTTPClient *http.Client
}

URL is the url: Source adapter (crawled sites), decided in issue #8. No version concept, per issue #9 — Resolve always returns one implicit version, "".

func NewURL

func NewURL() *URL

func (*URL) Fetch

func (u *URL) Fetch(ctx context.Context, id identity.ID, version string) ([]RawPage, error)

func (*URL) Resolve

func (u *URL) Resolve(ctx context.Context, ref string) (identity.ID, []string, error)

func (*URL) Type

func (u *URL) Type() identity.SourceType