output

Documentation

Index

• Constants
• Variables
• func CanonicalStudioURL(rawURL string) string
• func LegacySlugify(rawURL string) string
• func Slugify(rawURL string) string
• func SweepStaleTempFiles(dir string, maxAge time.Duration) int
• func WriteCSV(scenes []models.Scene, path string) error
• func WriteJSON(sf models.StudioFile, path string) error

Constants

const MinSweepAge = 10 * time.Minute

MinSweepAge is the shortest age SweepStaleTempFiles will honour.

Variables

var CSVHeaders = []string{
	"id", "siteId", "studioUrl",
	"title", "url", "date", "description",
	"thumbnail", "preview",
	"performers", "director", "studio",
	"tags", "categories",
	"series", "seriesPart",
	"duration", "resolution", "width", "height", "format",
	"views", "likes", "comments",
	"lowestPrice", "lowestPriceDate", "priceHistory",
	"scrapedAt", "deletedAt",

	"firstSeenAt", "externalIds",
}

CSVHeaders defines the column order for CSV exports. Multi-value fields use | as separator. PriceHistory is serialised as a JSON string.

Functions

func CanonicalStudioURL

func CanonicalStudioURL(rawURL string) string

CanonicalStudioURL reduces a studio URL to the form used as its storage identity. Two URLs that address the same catalogue map to the same string:

http://Example.com/studio/  ->  https://example.com/studio
https://example.com/studio  ->  https://example.com/studio

It normalises the scheme to https, lowercases the host, drops a default port, removes any fragment, and strips a trailing slash from the path. The path's case, the query string and everything else are left alone — paths are case-sensitive on most servers, and a query can genuinely select a different catalogue.

**This is identity, not an address.** `studio_url` is a primary-key component *and* what scrapers are asked to fetch; those are different jobs. Callers must keep passing the operator's original URL to the scraper — some sites are http-only, so fetching the canonical form would request a page that does not exist. Only the stored key is canonical.

A URL that cannot be parsed is returned unchanged rather than mangled.

func LegacySlugify

func LegacySlugify(rawURL string) string

LegacySlugify reproduces the pre-hash Slugify output. The Flat store uses it to find and migrate files written before the hash suffix was introduced.

func Slugify

func Slugify(rawURL string) string

Slugify converts a studio URL to a safe filename stem. It appends a short hash of the raw URL so that distinct URLs never collide on the same slug — e.g. "anna_b" and "anna-b" both sanitize to "anna-b" but get different hashes — and so that URLs whose host+path sanitize to the empty string (e.g. purely non-ASCII paths) still produce a unique, non-empty stem. The stem is capped at maxSlugLen bytes.

Changing this output renames Flat-store files; the Flat store migrates the legacy (un-hashed) filename on read — see LegacySlugify.

func SweepStaleTempFiles

func SweepStaleTempFiles(dir string, maxAge time.Duration) int

SweepStaleTempFiles removes leftover `.fss-tmp-*` files in dir whose last-modified time is older than maxAge. These are orphans from a previous `atomicWriteFile` whose process died (SIGKILL, OOM, power loss) between `os.CreateTemp` and the deferred `os.Remove`. The age guard prevents racing with a concurrent live writer — a `--db` SQLite scrape can sit holding an open temp for a long time, so we only remove files clearly stale enough that no in-flight write could own them.

maxAge is clamped up to MinSweepAge: the guard is the only thing keeping this from deleting a temp file a live writer still owns, so a caller passing a shorter age (or zero) gets the floor rather than the race.

Returns the number of files removed. Errors stat'ing or removing a single file are not propagated; this is a best-effort cleanup and a failed remove just means a tiny disk leak, not a correctness issue. Missing directories are not an error — Flat-store first run.

func WriteCSV

func WriteCSV(scenes []models.Scene, path string) error

WriteCSV writes scenes as CSV with a header row, using atomic file replacement.

func WriteJSON

func WriteJSON(sf models.StudioFile, path string) error

WriteJSON writes a StudioFile as indented JSON, using atomic file replacement to prevent corruption on crash.