Documentation
¶
Overview ¶
Package scraper defines the interface for site scrapers and a global registry.
Each supported site implements StudioScraper and registers itself via Register in an init() function. Consumers look up scrapers with ForURL or ForID, then call StudioScraper.ListScenes to stream results.
Scraper packages live under internal/scrapers/ and must be blank-imported to trigger registration.
Index ¶
- func AbsentError(url string, err error) error
- func Debugf(level int, format string, args ...any)
- func HostMatches(rawURL string, hosts ...string) bool
- func Paginate(ctx context.Context, opts ListOpts, siteID string, out chan<- SceneResult, ...)
- func ParseError(url string, err error) error
- func Register(s StudioScraper)
- func ResetWarnDelayBelow()
- func ResetWarnURLFallthrough()
- func SetVerbose(level int)
- func TransportError(url string, err error) error
- func URLHasNonRootPath(rawURL string) bool
- func Verbose() int
- func WarnDelayBelow(siteID string, actual, recommended time.Duration)
- func WarnURLFallthrough(siteID, rawURL string)
- type FailureKind
- type ListOpts
- type PageResult
- type ResultKind
- type SceneResult
- type ScrapeError
- type StudioScraper
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func AbsentError ¶
AbsentError marks err as a resource that is legitimately not there, so it does not count against traversal completeness. Use it for optional sub-listings a site may simply not have — not as a way to silence a fetch that failed.
func Debugf ¶
Debugf prints a debug message to stderr if the current verbosity level is >= the requested level. Format follows fmt.Fprintf conventions.
Levels by convention:
1 — high-level operations (pages fetched, items found, categories discovered) 2 — HTTP requests (URL, method, status, size) 3 — parsing details (regex matches, extracted fields)
func HostMatches ¶
HostMatches reports whether rawURL's host is one of hosts, ignoring a leading "www." on either side.
It compares parsed hosts rather than searching the raw URL text, so a look-alike domain that merely contains one of hosts does not match. See CONTRIBUTING.md § "Host regexes".
func Paginate ¶
func Paginate(ctx context.Context, opts ListOpts, siteID string, out chan<- SceneResult, fetchPage func(ctx context.Context, page int) (PageResult, error), )
Paginate runs a page-numbered pagination loop, handling delay, context cancellation, progress reporting, KnownIDs early-stop, and scene sending. The caller provides a FetchPage callback that fetches and parses a single page; everything else is handled by the loop.
FetchPage receives the 1-based page number and returns a PageResult. Set PageResult.Total on the first page for progress display. Set PageResult.Done to true when there are no more pages (e.g. items < pageSize, page >= totalPages). An empty Scenes slice also stops the loop, unless PageResult.Continue is set — see PageResult.Continue for pages that yield zero scenes but are not the end of the listing.
The siteID string is used only for debug log messages.
Paginate does NOT call defer close(out) — the caller's run() must still do that, since some scrapers do additional work after the pagination loop returns (e.g. worker pool teardown).
Two guards bound the loop against CMSes that never report an end: a hard page cap (paginateSafetyCap) and repeat-page detection (a page whose first scene ID matches the previous page is the CMS echoing its last page, so the loop stops). Both prevent an unbounded --full crawl when a callback forgets to set Done.
func ParseError ¶
ParseError marks err as a page that arrived but could not be understood. This is the one worth reaching for by hand: a parser that quietly returns nothing is indistinguishable from a site with nothing on it, and the difference decides whether an authoritative save may delete scenes.
func Register ¶
func Register(s StudioScraper)
Register adds a scraper to the global registry. Call this from an init() function in each scraper package. Panics if a scraper with the same ID is already registered.
func ResetWarnDelayBelow ¶
func ResetWarnDelayBelow()
ResetWarnDelayBelow clears the once-per-siteID memoisation. Intended for tests that need to re-trigger the warning across cases.
func ResetWarnURLFallthrough ¶
func ResetWarnURLFallthrough()
ResetWarnURLFallthrough clears the once-per-key memoisation. For tests.
func SetVerbose ¶
func SetVerbose(level int)
SetVerbose sets the global debug verbosity level. Level 0 = silent (default), 1+ = increasingly verbose.
func TransportError ¶
TransportError marks err as a page that never arrived. Scrapers rarely need this — Classify already recognises network errors and httpx status errors — but it is here for transports it cannot see into.
func URLHasNonRootPath ¶
URLHasNonRootPath reports whether rawURL carries a path (or query) component beyond the bare site root — i.e. it looks like a filtered view (model, channel, category, tag, DVD) rather than the studio's front page. A trailing "/" and the common root files ("/", "/en/", "/tour/") count as root.
Scrapers whose run() dispatches on URL shape can use this in their default (full-catalogue) branch to detect a filtered URL that fell through to scraping the entire site — see WarnURLFallthrough.
func WarnDelayBelow ¶
WarnDelayBelow prints a one-shot stderr warning when `actual` is strictly below `recommended` for scraper `siteID`. Use it to flag when the operator's chosen delay is below a value the upstream site is known to need to avoid rate-limiting. Unlike Debugf this prints regardless of verbosity, but only once per siteID per process so it doesn't spam.
`recommended` of 0 or `actual` of 0 are no-ops if recommended <= 0; a recommended floor of zero means "no minimum", so nothing to warn about. Callers should pass the package's documented minimum (e.g. 500ms for julesjordan / newsensations).
func WarnURLFallthrough ¶
func WarnURLFallthrough(siteID, rawURL string)
WarnURLFallthrough prints a one-shot stderr warning that a filtered URL (one with a non-root path) fell through to the default full-catalogue scrape for siteID. This is the loud signal for AUDIT_PLAN S3/B3: a model/channel/tag URL that the scraper did not recognise would otherwise silently scrape the whole site under that URL's store key. It is a no-op for root URLs. Prints at most once per (siteID, path) per process.
Types ¶
type FailureKind ¶
type FailureKind int
FailureKind classifies why a scrape step failed.
Scrapers report non-fatal failures as Error results on the channel, and for a long time every one of them was an undifferentiated `error`. Three very different situations were reaching the consumer as the same value:
- the page never arrived — the scenes on it are missing, and a retry might well get them
- the page arrived but could not be understood — the scenes on it are also missing, but the site changed and the parser needs fixing
- the requested thing is legitimately not there — nothing is missing
The distinction is load-bearing rather than cosmetic. `--full` and `--refresh` treat a traversal as authoritative over the studio's whole scene set, so an unreached page has to suppress the destructive delete, while an optional sub-listing that genuinely 404s must not — inflating the error count with expected absences made every such run non-authoritative.
const ( // FailureUnknown is a failure that has not been classified. It is the // conservative default: the plain errors scrapers already return land // here, and are treated as potentially-missing data exactly as they were // before this classification existed. FailureUnknown FailureKind = iota // FailureTransport means the page never arrived — a network error, a // timeout, a cancelled context, or a status that says the server would // not serve it (5xx, 429, 403). FailureTransport // FailureParse means the page arrived intact but could not be read: a // missing block, a selector that matches nothing, an unparseable date. // This is the site-redesign signal, and the one worth acting on — it // will not fix itself on a retry. FailureParse // FailureAbsent means the requested resource is legitimately gone (404, // 410) or the listing is genuinely empty. No data is missing, so a // traversal that only saw these is still complete. FailureAbsent )
func Classify ¶
func Classify(err error) FailureKind
Classify determines the FailureKind of an error.
It looks for an error in the chain that classifies itself (see ScrapeError), then falls back to recognising cancellation and network errors as transport failures. Anything else is FailureUnknown, which callers must treat as possibly-missing data.
A nil error is FailureUnknown; callers should not be classifying success.
func (FailureKind) MissingData ¶
func (k FailureKind) MissingData() bool
MissingData reports whether a failure of this kind means scenes went uncollected, and therefore that a traversal cannot be treated as the studio's complete state.
Everything except FailureAbsent counts, including FailureUnknown: an error nobody classified might have cost us a page, and wrongly believing a traversal complete is what deletes a catalogue.
func (FailureKind) String ¶
func (k FailureKind) String() string
String returns the lowercase name of the failure kind.
type ListOpts ¶
type ListOpts struct {
// Workers sets the number of concurrent detail-page fetchers for scrapers
// that use a worker pool. Zero uses the scraper's default (typically 4).
Workers int
// KnownIDs, when non-empty, signals the scraper to stop pagination as soon
// as it encounters an ID already in the set. Used for incremental runs where
// content is sorted newest-first and trailing pages are already stored.
KnownIDs map[string]bool
// Delay is the duration to sleep between page fetches (and between detail
// fetches for scrapers that use a worker pool). Zero means no delay.
Delay time.Duration
}
ListOpts controls scraping behaviour passed in from the CLI/config.
type PageResult ¶
type PageResult struct {
Scenes []models.Scene
Total int
Done bool
// Continue tells the loop not to treat an empty Scenes slice as
// end-of-listing. Set it when a page legitimately yields zero Scenes but
// more pages may still follow — either because the page's raw items all
// filtered out (videos-only, dedup, details that failed to fetch), or
// because pagination walks a fixed list (DVDs, years) where an empty
// element is not the end. Callbacks that set Continue MUST also set Done at
// the true end, or the loop will not terminate. Leave it false for the
// common case where an empty page means the listing is exhausted.
Continue bool
}
PageResult is returned by a FetchPage callback to the pagination loop.
type ResultKind ¶
type ResultKind int
ResultKind identifies what a SceneResult carries.
const ( // KindScene indicates the result carries a valid Scene. KindScene ResultKind = iota // KindError indicates a non-fatal error. Log and continue. KindError // KindTotal is a progress hint sent once after the first page. KindTotal // KindStoppedEarly signals the scraper hit a known ID and stopped pagination. KindStoppedEarly )
func (ResultKind) String ¶
func (k ResultKind) String() string
type SceneResult ¶
type SceneResult struct {
Kind ResultKind
Scene models.Scene
Err error
Total int
}
SceneResult is a single item sent on the channel returned by ListScenes. Use the Kind field to determine which other fields are populated. Prefer the constructor functions Scene, Error, [Total], StoppedEarly.
func Error ¶
func Error(err error) SceneResult
Error constructs a SceneResult carrying a non-fatal error.
func Progress ¶
func Progress(total int) SceneResult
Progress constructs a SceneResult carrying a total-scenes hint.
func Scene ¶
func Scene(s models.Scene) SceneResult
Scene constructs a SceneResult carrying a scraped scene.
func StoppedEarly ¶
func StoppedEarly() SceneResult
StoppedEarly constructs a SceneResult signalling early pagination stop.
type ScrapeError ¶
type ScrapeError struct {
// Kind is why the step failed.
Kind FailureKind
// URL is the page being fetched or parsed, if known.
URL string
// Err is the underlying error.
Err error
}
ScrapeError annotates an error with a FailureKind and the URL it concerns, so the cmd layer can tell an unreached page from an absent one without pattern-matching on error strings.
Build one with TransportError, ParseError or AbsentError rather than filling the struct directly.
func (*ScrapeError) Error ¶
func (e *ScrapeError) Error() string
Error implements the error interface.
func (*ScrapeError) FailureKind ¶
func (e *ScrapeError) FailureKind() FailureKind
FailureKind reports how this error should be classified. It is what Classify looks for, and any error type may implement the same method to opt in.
func (*ScrapeError) Unwrap ¶
func (e *ScrapeError) Unwrap() error
Unwrap returns the underlying error so errors.Is/As see through the annotation.
type StudioScraper ¶
type StudioScraper interface {
// ID returns a stable lowercase identifier for this scraper (e.g. "manyvids").
ID() string
// Patterns returns the URL patterns this scraper handles.
// Used by `fss list-scrapers` and as documentation. A scraper may declare
// multiple patterns (different URL formats, shared-platform sites, etc.).
Patterns() []string
// MatchesURL returns true if this scraper can handle the given studio URL.
MatchesURL(url string) bool
// ListScenes fetches all scenes for the given studio URL and sends each
// result down the returned channel. The channel is closed when done.
// Implementations should respect ctx cancellation.
ListScenes(ctx context.Context, studioURL string, opts ListOpts) (<-chan SceneResult, error)
}
StudioScraper is implemented once per supported site. Adding a new site means creating internal/scrapers/<site>/<site>.go, implementing this interface, and calling scraper.Register in an init().
func ForID ¶
func ForID(id string) (StudioScraper, error)
ForID returns the registered scraper with the given ID, or an error if none match.
func ForURL ¶
func ForURL(url string) (StudioScraper, error)
ForURL returns the first registered scraper that matches the given URL, or an error if none match. Resolution is first-match-wins by registration (import) order. When more than one scraper matches — e.g. a broad parent regex shadowing a sub-site — the extra matches are reported at debug level so the overlap is visible without changing which scraper is chosen.
Source Files