parseutil

Documentation

Overview

Package parseutil holds the parsing helpers shared across FSS scrapers: duration parsing (colon-separated and ISO 8601), date normalisation, Open Graph meta extraction, and schema.org VideoObject extraction from JSON-LD. Scrapers should reach for these rather than reimplementing them.

Index

• func OpenGraph(body []byte) map[string]string
• func ParseDurationColon(s string) int
• func ParseDurationISO(s string) int
• func StripOrdinalSuffix(s string) string
• func TryParseDate(s string, layouts ...string) (time.Time, error)
• type VideoObject
  • func ExtractVideoObject(body []byte) *VideoObject
  • func ExtractVideoObjects(body []byte) []VideoObject

Functions

func OpenGraph

func OpenGraph(body []byte) map[string]string

OpenGraph extracts every `<meta property="og:*" content="…">` pair from `body` into a map keyed by the full property name (e.g. `"og:title"`, `"og:image"`, `"og:video:duration"`). Both attribute orderings (`property` then `content`, and the reverse) are recognised; this matches what real sites emit in practice.

The returned map values are the raw `content` attribute strings as they appear in the source — HTML entities are NOT decoded, and trailing whitespace is preserved. Callers that need an unescaped string should pass the value through `html.UnescapeString` (and `strings.TrimSpace` where relevant). The values are kept raw so the helper matches what scrapers were doing before extraction without taking on entity-handling decisions they each made differently.

Repeated `og:foo` tags (e.g. multiple `og:image` entries on articles) collapse to the last occurrence in source order. If a caller needs every value, switch to FindAllStringSubmatch directly.

Returns an empty (non-nil) map if no OpenGraph tags are present.

func ParseDurationColon

func ParseDurationColon(s string) int

ParseDurationColon parses colon-separated durations like "MM:SS" or "HH:MM:SS" and returns the total seconds. Handles any number of colon-separated parts. Returns 0 for empty or unparseable input.

func ParseDurationISO

func ParseDurationISO(s string) int

ParseDurationISO parses ISO 8601 durations like "PT1H2M3S" and returns the total seconds. Input case is ignored. Returns 0 for empty or unparseable input.

Only the time components (H/M/S) are understood. A duration carrying a date part ("P1DT2H") is not supported and yields a partial result — no FSS scraper has been seen to emit one, and guessing at day arithmetic would turn a visibly wrong number into a plausibly wrong one.

func StripOrdinalSuffix

func StripOrdinalSuffix(s string) string

StripOrdinalSuffix removes English ordinal suffixes (`st`, `nd`, `rd`, `th`, case-insensitive) immediately following a digit run. Example:

"8th May 2026"        → "8 May 2026"
"May 8th, 2026"       → "May 8, 2026"
"22nd September 2024" → "22 September 2024"

Digits without a suffix are left alone. Use this as a pre-pass before `time.Parse` against a layout that uses bare day numbers.

func TryParseDate

func TryParseDate(s string, layouts ...string) (time.Time, error)

TryParseDate attempts each layout in order and returns the first successful parse. Returns a zero time and an error if none match. Callers choose the layout set — there is no grab-bag of every known format, since mixing ambiguous formats (e.g. M/D/Y vs D/M/Y) would risk silent mis-classification.

Types

type VideoObject

type VideoObject struct {
	URL           string   `json:"url"`
	Name          string   `json:"name"`
	Description   string   `json:"description"`
	ThumbnailURL  string   `json:"thumbnailUrl"`
	ContentURL    string   `json:"contentUrl"`
	Duration      string   `json:"duration"`
	UploadDate    string   `json:"uploadDate"`
	DatePublished string   `json:"datePublished"`
	Actors        []string `json:"-"`
	Director      string   `json:"-"`
	Keywords      string   `json:"keywords"`
	PartOfSeries  string   `json:"-"`
}

VideoObject holds the common fields from a schema.org VideoObject embedded in a page's JSON-LD script block.

func ExtractVideoObject

func ExtractVideoObject(body []byte) *VideoObject

ExtractVideoObject finds the first VideoObject in the page's JSON-LD blocks. Returns nil if none is found. It handles both bare VideoObject blocks and ItemList wrappers (returning the first item). Actor fields are parsed flexibly: arrays of strings, arrays of {"name":"…"} objects, or a single string all work.

func ExtractVideoObjects

func ExtractVideoObjects(body []byte) []VideoObject

ExtractVideoObjects returns all VideoObject entries found in the page's JSON-LD blocks, including those wrapped in an ItemList.