imdb

README

libimdb

A programmatic, CGO-free Go library for creating and querying a custom SQLite catalog from IMDb's public datasets. Module: github.com/odwrtw/libimdb; package name: imdb. Requires Go 1.26.1 or newer.

Go documentation

What it owns

The catalog is authoritative for IMDb identity and available IMDb facts:

• Primary/original titles and alternative titles, with alias language/region.
• Movie/show kind and original IMDb title type.
• Start/end year, runtime in minutes, adult flag, genres, rating, and vote count.
• Dataset provenance, compressed-input checksums, and catalog generation.

Retained title types are movie, tvMovie, tvSeries, and tvMiniSeries. Episodes, shorts, people, plots, images, exact air dates, and TMDB/TVDB IDs are not included. Application state and provider enrichment belong elsewhere. Unknown numeric/boolean fields use pointers; unknown genres are nil. Unknown original titles, alias language, and alias region use empty strings. Go fields use explicit names: TitleType, StartYear, RuntimeMinutes. Name.Source is a NameSource (PrimaryName, OriginalName, or AliasName). Name.IsOriginal is IMDb's alias isOriginalTitle metadata, not a general original-name test; primary/original name records leave it nil.

Update from Go

The same operation creates a missing catalog or refreshes an existing one:

ctx, cancel := context.WithTimeout(context.Background(), 2*time.Hour)
defer cancel()

info, err := imdb.Update(ctx, "imdb.sqlite", imdb.UpdateOptions{
    Attempts: 2,
})

The default source is https://datasets.imdbws.com/, using title.basics.tsv.gz, title.akas.tsv.gz, and title.ratings.tsv.gz.

Data flows directly from HTTP through gzip/TSV parsing into prepared SQLite statements. Neither compressed/extracted dataset temporary files nor whole-file buffers are used. Rows are limited to 1 MiB. Suggestion names are deduplicated by IMDb ID and folded name; trigram inserts are batched per distinct entry. Raw aliases and their metadata remain intact. All three datasets, search indexes, metadata, and checksums commit in one transaction. Removed facts disappear on successful refresh.

Any transfer, parse, checksum, insertion, close, cancellation, or commit failure preserves the previous committed catalog. A failed first creation can leave an empty file; it will not open as a valid catalog and can be retried.

Retries restart the entire transaction for transient network errors, incomplete transfers, HTTP 429, or HTTP 5xx. The default is one attempt, with a maximum of five. There is no incremental/conditional refresh yet. Download speed is coupled to insert speed; caller deadlines and real-dataset throughput measurements are important.

Supply HTTPSource{Client: client, BaseURL: base} to customize HTTP policy, or a Source/SourceFunc to open existing gzip files or other streams. Source.Open returns (io.ReadCloser, SourceMetadata, error). Source metadata supplies the provenance location, ETag, last-modified value and advisory compressed size. Info.Sources contains DatasetReport values combining that metadata with the importer's dataset identity, row count, and computed SHA-256. The digest is not an expected-checksum input or an external integrity assertion.

On successful open, Update owns and closes the stream. On error the source must release any acquired resources. Calls are sequential within an update; retries reopen datasets and require fresh, complete streams starting at byte zero. Sources must honor context cancellation; an arbitrary blocking reader cannot be forcibly interrupted by the library.

Update progress

The library is silent unless UpdateOptions.Progress is supplied:

info, err := imdb.Update(ctx, "imdb.sqlite", imdb.UpdateOptions{
    Progress: func(p imdb.Progress) {
        if p.Dataset != "" {
            fmt.Fprintf(os.Stderr, "attempt %d: %s %s, %d compressed bytes, %d rows\n",
                p.Attempt, p.Phase, p.Dataset, p.BytesRead, p.Rows)
        } else {
            fmt.Fprintf(os.Stderr, "attempt %d: %s\n", p.Attempt, p.Phase)
        }
    },
})

The callback receives synchronous snapshots and must return promptly; slow callbacks slow the import. Do not reenter Update from it. Cancel the update's context to stop. Nil disables reporting. Ordinary counters are throttled to about four notifications per second; phase boundaries and final dataset counters are always delivered. There is no background heartbeat during blocking work.

Phase reports Preparing, Opening, Importing, DatasetDone, Indexing, Validating, Committing, Retrying, Done, or Failed. Only dataset phases carry Dataset and counters; update-wide events have an empty dataset. Done means the catalog committed successfully. DatasetDone follows checksum validation and successful stream closure, not merely reading the last byte. Retrying and Failed carry Err; cancellation is a Failed event.

Attempt is one-based (zero if options are rejected before any attempt), and Attempts is the maximum allowed. Counters reset on each dataset and full-update retry. Rows excludes the TSV header and includes successfully processed filtered records, not just retained catalog rows. BytesRead counts compressed input. Gzip/parser read-ahead means bytes and rows advance independently, and 100% of input bytes is not 100% of the database update. Indexing, validation, and commit have phases but no measurable percentage.

TotalBytes is advisory; zero means unknown. HTTP sources use a known content length. Custom sources can supply SourceMetadata.SizeBytes; known sizes are stored as optional provenance (size_bytes); its absence means the size is unknown.

Database safety and concurrency

• SQLite WAL, shared-memory/journal files, and SQLite-managed scratch storage are distinct from dataset temporary files and may be used.
• Use a local filesystem supporting SQLite locking, not a network filesystem.
• Only one writer is supported. SQLite lock waiting is bounded to one second; contention is an error, not an implicit competing import.
• Read-only catalog handles remain valid during refresh. Each operation reads one transaction snapshot; its next operation sees newly committed data.
• Use Catalog.View for multi-call workflows needing one generation; a separate Info call does not identify a previous operation's snapshot. Resolution, SearchResult, and AmbiguousError also report their own generation.
• Do not open this changing database with SQLite immutable=1.
• Read-only container deployments must mount the catalog directory with the required WAL/SHM files available. Validate the actual mount/checkpoint setup.
• Readers and refreshes require schema v2. Other schemas and foreign or unversioned databases are rejected with ErrSchema, never overwritten. To rebuild an incompatible catalog, use a new path and explicitly point consumers to it.
• SQLite refreshes do not compact the database automatically.

Lookup and safe resolution

catalog, err := imdb.Open(ctx, "imdb.sqlite")
if err != nil {
    return err
}
defer catalog.Close()

match, err := catalog.Resolve(ctx, imdb.Query{
    Title:     "The Matrix",
    Kind:      imdb.Movie,
    StartYear: 1999,
})
if err != nil {
    // errors.Is(err, imdb.ErrNotFound) or imdb.ErrAmbiguous are normal
    // identity outcomes, distinct from catalog/operational failures.
    return err
}
id := match.Title.ID

Resolve returns a Resolution, with matched name, exact/normalized method, and generation, but no ranking score. On ambiguity it returns a zero resolution and *AmbiguousError, which matches ErrAmbiguous via errors.Is:

var ambiguity *imdb.AmbiguousError
if errors.As(err, &ambiguity) {
    // Distinct safe-name conflicts, sorted by ID, never popularity.
    for _, title := range ambiguity.Titles {
        fmt.Println(title.ID, title.PrimaryTitle)
    }
    // At most 20 titles; Truncated indicates further conflicts exist.
    fmt.Println(ambiguity.Generation, ambiguity.Truncated)
}

These conflicts come from the same snapshot and filters as resolution. Broader candidate search is not a replacement for this ambiguity information.

Lookup returns facts for an ID, or ErrNotFound for an absent or unsupported title. IDs are tt followed by one or more ASCII digits. LookupMany returns a snapshot-consistent map for up to 10,000 IDs, omitting absent IDs and collapsing duplicates; empty input returns an empty map. Names returns raw names and metadata in import order, not a preference order or a cross-generation ordering guarantee. Distinct original names are included; an original identical to the primary is not repeated as OriginalName. Info reports provenance and generation.

Query takes a title, not a filename. Parse filenames outside this library. Empty kind searches both kinds; StartYear zero means unspecified. A supplied year is a hard constraint, including for records with unknown years. Adult titles are included by default; ExcludeAdult excludes those explicitly flagged adult.

Resolution normalizes Unicode NFC, Unicode case, and whitespace only. It never removes accents, punctuation, or words. All matching primary/original/alias names participate in ambiguity detection; even an exact primary spelling does not override an identity collision. Repeated aliases for one ID are not ambiguity. Popularity never selects an identity. Queries are limited to 1,024 bytes and 256 normalized Unicode code points.

Consistent multi-call reads

err = catalog.View(ctx, func(view *imdb.View) error {
    resolved, err := view.Resolve(ctx, imdb.Query{Title: "The Matrix"})
    if err != nil {
        return err
    }
    names, err := view.Names(ctx, resolved.Title.ID)
    if err != nil {
        return err
    }
    // resolved and names belong to one generation, even during a refresh.
    fmt.Println(resolved.Generation, names)
    return nil
})

The view also provides Lookup, LookupMany, Candidates, and Info. Its lifetime is the callback, with cleanup on success, error, or panic. Do not copy, retain, or use the view concurrently. Use the view rather than the same catalog inside its callback: each catalog serializes operations on one connection. Results remain usable after the callback; view calls then return ErrViewClosed. Keep callbacks short because snapshots can retain WAL space. The view's parent context governs its lifetime; individual calls can use shorter deadlines.

Invalid input

Invalid queries, IDs, paths, HTTP source settings, and update/search options return *ValidationError, matching ErrInvalidInput. Use errors.As to obtain Field and Problem; do not parse error strings. Operational failures, cancellation, not-found, and ambiguity are separate outcomes.

Broader suggestions

result, err := catalog.Candidates(ctx, imdb.Query{
    Title: "matrix", Kind: imdb.Movie,
}, imdb.SearchOptions{Limit: 20})

Candidate search uses indexed normalized names and accent/punctuation-folded prefixes and substrings. Substring matching starts at three folded Unicode characters, so matrix finds The Matrix and interior fragments work too. One- and two-character queries use exact/prefix matching only. SQL wildcards and operators have no special meaning; punctuation is folded as described above.

When direct results do not fill Limit, queries of at least four folded characters also retrieve whole-title typo suggestions using rune-aware edit distance. Typo search inspects up to three rare trigram posting streams; it does not aggregate every common-gram hit or scan every title. Substring retrieval uses the least frequent query gram and checks literal containment. Kind-aware indexes avoid reading the other kind's prefix/posting entries.

SearchResult.Matches contains Candidate values, a different type from Resolution. Results are deduplicated by IMDb ID, with full title and matched-alias metadata loaded only for the final IDs. Ranking is deterministic: exact, normalized, prefix, substring, fuzzy; then score, votes, ID and matched name. Shorter prefix and substring matches score higher. Scores are ranking values, not confidence probabilities. Lossy accent/punctuation equivalence remains fuzzy.

This is heuristic retrieval, not guaranteed exhaustive search or a globally optimal popularity ranking. Default Budget: 1,000 entries/postings per stage; maximum: 10,000. Exact retrieval counts distinct IDs; prefix/substring retrieval counts compact entries; all selected typo streams share one posting budget. Budgets apply before year/adult and similarity filters. Repeated raw aliases do not consume extra entries, but different folded names and postings can. Each stream may read one extra row to detect clipping. Default output Limit: 20; maximum: 100.

LimitTruncated means retrieved matches were omitted to satisfy Limit; increasing the limit may expose them. BudgetExhausted means a retrieval stage was clipped by Budget, even when filtering leaves no returned matches; increasing the budget may retrieve more. Both flags can be true. Neither measures overall typo recall: a title sharing no selected gram may be missed. Skipping typo fallback because direct results fill the limit is normal search policy, not clipping. Use context deadlines; indexed safe-name ambiguity detection and final alias selection still process raw matching aliases.

Never infer a unique identity from candidates, their rank, or their count. Use Resolve for authoritative automated selection. Accent/punctuation folding and fuzzy matches remain suggestions only.

Search validation and benchmarks

Routine tests and BenchmarkSearch/BenchmarkSearchUpdate use synthetic, duplicate-heavy datasets. Optional real-data checks open an existing v2 catalog read-only and never refresh it:

go test -run '^$' -bench '^(BenchmarkSearch|BenchmarkSearchUpdate)$' -benchmem .
LIBIMDB_BENCH_DB=/path/to/imdb.sqlite go test -run '^TestRealCatalogSearch$' \
    -bench '^BenchmarkRealCatalog$' -benchmem -v .

The real benchmark reports SQLite version, catalog generation and counts. Record hardware, cache state and budgets when comparing runs; its timings are not a latency guarantee. Existing refreshes reuse SQLite pages and do not automatically shrink the database file, even when search postings become smaller.

CLI

GitHub releases provide a static Linux amd64 binary (imdb-linux-amd64). Releases are published after checks pass on pushes to master, with UTC date/time tags and names (YYYYMMDD-HHMMSS).

For a daily refresh using the binary built by ./build.sh, see the systemd user service and timer examples.

go run ./cmd/imdb update -db imdb.sqlite
go run ./cmd/imdb update -db imdb.sqlite -input-dir /path/to/existing/gzip/files
go run ./cmd/imdb update -db imdb.sqlite -progress always  # progress even when redirected
go run ./cmd/imdb update -db imdb.sqlite -progress never   # suppress progress
go run ./cmd/imdb info -db imdb.sqlite
go run ./cmd/imdb query -db imdb.sqlite -title 'The Matrix' -kind movie -year 1999
go run ./cmd/imdb query -db imdb.sqlite -title 'matrix' -kind movie -candidates
go run ./cmd/imdb query -db imdb.sqlite -title 'The Matrx' -kind movie -candidates

Updates default to -progress auto: a live status on terminal stderr (except TERM=dumb), showing the current file, compressed bytes/percentage when size is known, rows, attempts, and database phases. -input-dir uses regular-file sizes. Completed files and retries retain a summary line. Unknown sizes show bytes and rows without an invented percentage.

-progress always enables progress on any stderr destination, using plain newline records instead of terminal escapes when it is not an interactive terminal. -progress never disables it. Progress is best-effort; display-write failures do not abort the update.

Results remain JSON on stdout. Resolutions include generation and no score; title JSON uses start_year and runtime_minutes. Candidate results include limit_truncated and budget_exhausted. JSON errors retain kind and error, with optional ambiguity (titles, generation, truncation) or validation (field, problem) details. -year sets Query.StartYear.

With redirected stderr, default/never errors remain JSON-only; always deliberately mixes human progress lines with the final JSON error. The display ends before result/error output, and existing signal cancellation and exit codes are unchanged.

Documentation

Overview

Package imdb provides a CGO-free SQLite catalog of IMDb titles and facts. Update creates or refreshes the catalog atomically from streaming gzip datasets. Catalog.Resolve returns a Resolution only for unique exact or conservatively normalized names; AmbiguousError reports bounded conflicts without a winner. Catalog.Candidates returns heuristic Candidate suggestions, never identities. Catalog.View scopes multi-call reads to one committed generation. Invalid input returns ValidationError matching ErrInvalidInput. Readers and refreshes require schema v2. IMDb data is subject to separate personal, non-commercial licensing terms.

Index

• Variables
• type AmbiguousError
  • func (e *AmbiguousError) Error() string
  • func (e *AmbiguousError) Unwrap() error
• type Candidate
• type Catalog
  • func Open(ctx context.Context, path string) (*Catalog, error)
  • func (c *Catalog) Candidates(ctx context.Context, q Query, opts SearchOptions) (SearchResult, error)
  • func (c *Catalog) Close() error
  • func (c *Catalog) Info(ctx context.Context) (Info, error)
  • func (c *Catalog) Lookup(ctx context.Context, id string) (Title, error)
  • func (c *Catalog) LookupMany(ctx context.Context, ids []string) (map[string]Title, error)
  • func (c *Catalog) Names(ctx context.Context, id string) ([]Name, error)
  • func (c *Catalog) Resolve(ctx context.Context, q Query) (Resolution, error)
  • func (c *Catalog) View(ctx context.Context, fn func(*View) error) error
• type Counts
• type Dataset
• type DatasetReport
• type HTTPError
  • func (e *HTTPError) Error() string
• type HTTPSource
  • func (s HTTPSource) Open(ctx context.Context, d Dataset) (io.ReadCloser, SourceMetadata, error)
• type Info
  • func Update(ctx context.Context, path string, opts UpdateOptions) (Info, error)
• type Kind
• type MatchMethod
• type Name
• type NameSource
• type Progress
• type ProgressPhase
• type Query
• type Resolution
• type SearchOptions
• type SearchResult
• type Source
• type SourceFunc
  • func (f SourceFunc) Open(ctx context.Context, d Dataset) (io.ReadCloser, SourceMetadata, error)
• type SourceMetadata
• type Title
• type UpdateOptions
• type ValidationError
  • func (e *ValidationError) Error() string
  • func (e *ValidationError) Unwrap() error
• type View
  • func (v *View) Candidates(ctx context.Context, q Query, opts SearchOptions) (SearchResult, error)
  • func (v *View) Info(ctx context.Context) (Info, error)
  • func (v *View) Lookup(ctx context.Context, id string) (Title, error)
  • func (v *View) LookupMany(ctx context.Context, ids []string) (map[string]Title, error)
  • func (v *View) Names(ctx context.Context, id string) ([]Name, error)
  • func (v *View) Resolve(ctx context.Context, q Query) (Resolution, error)

Examples

• Catalog.Candidates
• Catalog.Resolve
• Catalog.View
• ValidationError

Variables

var (
	// ErrNotFound means no retained title satisfies the lookup or resolution.
	ErrNotFound = errors.New("imdb: title not found")
	// ErrAmbiguous is matched by AmbiguousError via errors.Is.
	ErrAmbiguous = errors.New("imdb: ambiguous title")
	// ErrInvalidInput is matched by ValidationError via errors.Is.
	ErrInvalidInput = errors.New("imdb: invalid input")
	// ErrSchema means the database is foreign, uninitialized, or incompatible.
	ErrSchema = errors.New("imdb: unsupported or uninitialized catalog")
	// ErrViewClosed means a View was used outside its callback or is uninitialized.
	ErrViewClosed = errors.New("imdb: read view closed")
)

Types

type AmbiguousError

type AmbiguousError struct {
	Titles     []Title `json:"titles"`
	Truncated  bool    `json:"truncated"`
	Generation string  `json:"generation"`
}

AmbiguousError reports conflicting titles sharing a conservatively normalized name under the query's filters. Titles contains at most 20 distinct titles, ordered by IMDb ID, not popularity. Truncated means more conflicts exist. All titles and Generation belong to the same snapshot. None is selected.

func (*AmbiguousError) Error

func (e *AmbiguousError) Error() string

func (*AmbiguousError) Unwrap

func (e *AmbiguousError) Unwrap() error

type Candidate

type Candidate struct {
	Title  Title       `json:"title"`
	Name   Name        `json:"matched_name"`
	Method MatchMethod `json:"method"`
	Score  float64     `json:"score"`
}

Candidate is a suggestion, never proof of identity, even for Exact matches. Score is ranking similarity, not a confidence probability. Use Resolve for automated identity selection rather than candidate count, method, or score.

type Catalog

type Catalog struct {
	// contains filtered or unexported fields
}

Catalog is a concurrency-safe read-only handle. Each operation reads one transaction snapshot; later operations automatically observe committed updates. Close releases its resources. Do not share the database over a network filesystem.

func Open

func Open(ctx context.Context, path string) (*Catalog, error)

Open opens an existing catalog read-only and validates its schema. It never creates a missing file. Path must be a filesystem path, not a SQLite URI.

func (*Catalog) Candidates

func (c *Catalog) Candidates(ctx context.Context, q Query, opts SearchOptions) (SearchResult, error)

Candidates returns bounded exact, normalized, prefix, substring and whole-title typo suggestions. Substrings require three folded Unicode characters; typos require four and run only when direct matches do not fill Limit. Retrieval is heuristic: clipping flags do not measure recall. Invalid queries or options return *ValidationError. No suggestions returns an empty, non-nil Matches. Use Resolve, never candidate rank or count, for authoritative identity.

Example

package main

import (
	"context"
	"fmt"

	imdb "github.com/odwrtw/libimdb"
)

func main() {
	catalog, err := imdb.Open(context.Background(), "imdb.sqlite")
	if err != nil {
		return
	}
	defer func() { _ = catalog.Close() }()

	result, err := catalog.Candidates(context.Background(), imdb.Query{
		Title: "matrix", Kind: imdb.Movie,
	}, imdb.SearchOptions{Limit: 20})
	if err != nil {
		return
	}
	for _, match := range result.Matches {
		fmt.Println(match.Title.ID, match.Title.PrimaryTitle, match.Method)
	}
	// These are suggestions, never proof of a unique identity.
	fmt.Println("limit clipped:", result.LimitTruncated, "budget exhausted:", result.BudgetExhausted)
}

func (*Catalog) Close

func (c *Catalog) Close() error

Close releases the catalog's resources and prevents new operations. Finish View callbacks before closing the Catalog.

func (*Catalog) Info

func (c *Catalog) Info(ctx context.Context) (Info, error)

Info returns metadata for the operation's committed snapshot.

func (*Catalog) Lookup

func (c *Catalog) Lookup(ctx context.Context, id string) (Title, error)

Lookup returns the retained title with id, or ErrNotFound. IDs must be "tt" followed by one or more ASCII digits; invalid IDs return ValidationError. Valid IDs for unsupported title types are treated as not found.

func (*Catalog) LookupMany

func (c *Catalog) LookupMany(ctx context.Context, ids []string) (map[string]Title, error)

LookupMany returns a non-nil map of found titles, omitting absent IDs and collapsing duplicates. Empty input returns an empty map. All results belong to one snapshot. At most 10000 IDs are allowed; any invalid ID rejects the entire request with ValidationError. ID syntax is the same as for Lookup.

func (*Catalog) Names

func (c *Catalog) Names(ctx context.Context, id string) ([]Name, error)

Names returns primary/original/alias names without deduplication. Ordering is dataset import order within this snapshot, not a preference ranking or stable across generations. An original name identical to the primary is not repeated as an OriginalName. It returns ErrNotFound for an absent title and ValidationError for invalid ID syntax (see Lookup).

func (*Catalog) Resolve

func (c *Catalog) Resolve(ctx context.Context, q Query) (Resolution, error)

Resolve accepts only a unique identity after Unicode NFC, case and whitespace normalization. It never uses fuzzy or popularity ranking. All matching names participate in ambiguity detection, even when a primary name is an exact spelling match. StartYear never falls back to another or unknown year. No match returns ErrNotFound; multiple identities return *AmbiguousError, matching ErrAmbiguous. Invalid queries return *ValidationError. On any error the returned Resolution is zero; ambiguity never selects a winner.

Example

package main

import (
	"context"
	"errors"
	"fmt"

	imdb "github.com/odwrtw/libimdb"
)

func main() {
	// Populate this catalog with imdb.Update before opening it.
	catalog, err := imdb.Open(context.Background(), "imdb.sqlite")
	if err != nil {
		return
	}
	defer func() { _ = catalog.Close() }()

	match, err := catalog.Resolve(context.Background(), imdb.Query{
		Title: "The Matrix", Kind: imdb.Movie, StartYear: 1999,
	})
	var ambiguity *imdb.AmbiguousError
	switch {
	case errors.As(err, &ambiguity):
		for _, title := range ambiguity.Titles {
			fmt.Println(title.ID, title.PrimaryTitle)
		}
		fmt.Println("more conflicts:", ambiguity.Truncated)
	case errors.Is(err, imdb.ErrNotFound):
		fmt.Println("no safe identity match")
	case err != nil:
		fmt.Println("catalog failure:", err)
	default:
		fmt.Println(match.Title.ID, match.Generation)
	}
}

func (*Catalog) View

func (c *Catalog) View(ctx context.Context, fn func(*View) error) error

View runs fn synchronously against one committed generation, releasing the snapshot even on callback error or panic. It returns fn's error, or the context error if fn succeeds after cancellation. A nil fn is invalid input. Use the supplied View for all reads inside fn, not the Catalog. Keep callbacks short: long-lived snapshots can prevent SQLite from reclaiming WAL space.

Example

package main

import (
	"context"
	"fmt"

	imdb "github.com/odwrtw/libimdb"
)

func main() {
	ctx := context.Background()
	catalog, err := imdb.Open(ctx, "imdb.sqlite")
	if err != nil {
		return
	}
	defer func() { _ = catalog.Close() }()

	err = catalog.View(ctx, func(view *imdb.View) error {
		resolved, err := view.Resolve(ctx, imdb.Query{Title: "The Matrix"})
		if err != nil {
			return err
		}
		names, err := view.Names(ctx, resolved.Title.ID)
		if err != nil {
			return err
		}
		// Both results come from one generation, even across a refresh.
		fmt.Println(resolved.Generation, names)
		return nil
	})
	if err != nil {
		fmt.Println(err)
	}
}

type Counts

type Counts struct {
	Movies  int64 `json:"movies"`
	Shows   int64 `json:"shows"`
	Names   int64 `json:"names"`
	Ratings int64 `json:"ratings"`
}

Counts reports retained catalog rows. Names includes repeated raw aliases.

type Dataset

type Dataset string

Dataset identifies one of the three required gzip-compressed IMDb TSV files.

const (
	Basics  Dataset = "title.basics.tsv.gz"
	Aliases Dataset = "title.akas.tsv.gz"
	Ratings Dataset = "title.ratings.tsv.gz"
)

type DatasetReport

type DatasetReport struct {
	SourceMetadata
	Dataset Dataset `json:"dataset"`
	SHA256  string  `json:"sha256"`
	Rows    int64   `json:"rows"`
}

DatasetReport combines source metadata with facts computed by the importer. SHA256 hashes the consumed compressed input; it is not compared to an external expected digest. Rows excludes the header and includes filtered records.

type HTTPError

type HTTPError struct {
	URL        string
	StatusCode int
}

HTTPError reports a non-200 download response. Update may retry 429 and 5xx.

func (*HTTPError) Error

func (e *HTTPError) Error() string

type HTTPSource

type HTTPSource struct {
	BaseURL string
	Client  *http.Client
}

HTTPSource streams official datasets. BaseURL and Client allow test servers or caller-managed HTTP policy. An empty BaseURL uses https://datasets.imdbws.com/; a custom BaseURL must be absolute HTTP(S), without query or fragment. A nil Client uses a two-hour per-request timeout; callers should also supply an overall context deadline.

func (HTTPSource) Open

func (s HTTPSource) Open(ctx context.Context, d Dataset) (io.ReadCloser, SourceMetadata, error)

Open downloads d without transparent HTTP decompression. The caller owns the returned body on success. Invalid datasets or BaseURL return ValidationError; non-200 responses return HTTPError after closing the response body.

type Info

type Info struct {
	SchemaVersion        int             `json:"schema_version"`
	NormalizationVersion int             `json:"normalization_version"`
	Generation           string          `json:"generation"`
	UpdatedAt            time.Time       `json:"updated_at"`
	Counts               Counts          `json:"counts"`
	Sources              []DatasetReport `json:"sources"`
}

Info describes one committed catalog generation and its dataset provenance. UpdatedAt is the UTC time recorded immediately before commit.

func Update

func Update(ctx context.Context, path string, opts UpdateOptions) (Info, error)

Update creates or refreshes a catalog in one transaction, without dataset temporary files or whole-download buffers. The catalog must be owned by this library using the current schema or empty. Foreign, unversioned and incompatible-schema databases are never overwritten. WAL is used for concurrent readers; SQLite's own transaction files are required. A failed update preserves previous committed data; a failed first creation may leave an empty, uninitialized database file.

type Kind

type Kind string

Kind groups retained IMDb title types into movies and series.

const (
	Movie Kind = "movie" // movie and tvMovie
	Show  Kind = "show"  // tvSeries and tvMiniSeries
)

type MatchMethod

type MatchMethod string

MatchMethod describes how a name matched, not whether its identity is unique.

const (
	Exact      MatchMethod = "exact"
	Normalized MatchMethod = "normalized"
	Prefix     MatchMethod = "prefix"
	Substring  MatchMethod = "substring"
	Fuzzy      MatchMethod = "fuzzy"
)

type Name

type Name struct {
	Title      string     `json:"title"`
	Source     NameSource `json:"source"`
	Region     string     `json:"region,omitempty"`
	Language   string     `json:"language,omitempty"`
	Types      []string   `json:"types,omitempty"`
	Attributes []string   `json:"attributes,omitempty"`
	IsOriginal *bool      `json:"is_original"`
}

Name is a primary title, distinct original title, or raw alias. Empty Region and Language and nil Types and Attributes mean unavailable or not applicable. IsOriginal is IMDb's alias isOriginalTitle field; it is nil for primary and original names and is not a general test for whether a name is original.

type NameSource

type NameSource string

NameSource identifies the dataset role of a name, not its language or region.

const (
	PrimaryName  NameSource = "primary"
	OriginalName NameSource = "original"
	AliasName    NameSource = "alias"
)

type Progress

type Progress struct {
	Dataset    Dataset
	Rows       int64 // successfully processed records, excluding header, including filtered records
	Phase      ProgressPhase
	Attempt    int // one-based; zero if options were rejected before any attempt
	Attempts   int // maximum attempts, not necessarily all used
	BytesRead  int64
	TotalBytes int64 // source-supplied compressed size; zero means unknown, totals are advisory
	Err        error // non-nil only for Retrying and Failed, including cancellation
}

Progress is a snapshot of a streaming update. Counters reset for each dataset and full-update attempt. BytesRead counts compressed input, not decompressed TSV bytes; buffering means bytes and rows advance independently. Reaching TotalBytes does not imply dataset validation or catalog commit has completed.

type ProgressPhase

type ProgressPhase string

ProgressPhase identifies the current operation. Only Opening, Importing, and DatasetDone carry a Dataset and per-dataset counters; other phases are update-wide.

const (
	Preparing   ProgressPhase = "preparing"
	Opening     ProgressPhase = "opening"
	Importing   ProgressPhase = "importing"
	DatasetDone ProgressPhase = "dataset_done"
	Indexing    ProgressPhase = "indexing"
	Validating  ProgressPhase = "validating"
	Committing  ProgressPhase = "committing"
	Retrying    ProgressPhase = "retrying"
	Done        ProgressPhase = "done"
	Failed      ProgressPhase = "failed"
)

type Query

type Query struct {
	Title        string
	Kind         Kind
	StartYear    int
	ExcludeAdult bool
}

Query takes a title, not a filename. Title must be valid UTF-8 without NUL, nonempty after normalization, at most 1024 bytes and 256 normalized Unicode code points. Empty Kind searches both kinds. StartYear zero is unspecified; a positive value is a hard constraint excluding unknown years. ExcludeAdult excludes only titles explicitly marked adult, not those with unknown flags.

type Resolution

type Resolution struct {
	Title      Title       `json:"title"`
	Name       Name        `json:"matched_name"`
	Method     MatchMethod `json:"method"`
	Generation string      `json:"generation"`
}

Resolution is a unique identity accepted by Resolve. Method is Exact or Normalized; Name is the matching name. Generation identifies its snapshot.

type SearchOptions

type SearchOptions struct {
	Limit  int // default 20, range 1–100
	Budget int // entries/postings inspected per stage, before filters; default 1000, range 1–10000
}

SearchOptions bounds heuristic retrieval. Zero values select defaults. Budget counts distinct IDs in the exact stage, folded names in prefix and substring stages, and postings across all selected typo streams combined. It applies before year/adult and similarity filters; different names for one title can consume multiple entries. Each stream may read one extra row to detect clipping. Budgets bound these stages, not overall query time.

type SearchResult

type SearchResult struct {
	Matches         []Candidate `json:"matches"`
	LimitTruncated  bool        `json:"limit_truncated"`  // retrieved matches were omitted to satisfy Limit
	BudgetExhausted bool        `json:"budget_exhausted"` // a retrieval stage was clipped by Budget
	Generation      string      `json:"generation"`
}

SearchResult contains distinct-ID suggestions ordered by method, descending score, descending votes, ID, then matched name. Matches is non-nil on success. Neither clipping flag measures recall or promises exhaustive retrieval.

type Source

type Source interface {
	Open(context.Context, Dataset) (io.ReadCloser, SourceMetadata, error)
}

Source supplies complete gzip streams. Open and the returned reader must honor cancellation. On success, Open returns a non-nil reader owned and closed by Update; on error, the source must release any acquired resources. Calls are sequential within an update, but retries reopen datasets and require fresh complete streams starting at byte zero. Shared sources must handle concurrent independent updates if used that way. Source metadata is advisory.

type SourceFunc

type SourceFunc func(context.Context, Dataset) (io.ReadCloser, SourceMetadata, error)

SourceFunc adapts a function to Source.

func (SourceFunc) Open

func (f SourceFunc) Open(ctx context.Context, d Dataset) (io.ReadCloser, SourceMetadata, error)

Open calls f with the requested dataset.

type SourceMetadata

type SourceMetadata struct {
	URL          string `json:"url,omitempty"`
	ETag         string `json:"etag,omitempty"`
	LastModified string `json:"last_modified,omitempty"`
	SizeBytes    int64  `json:"size_bytes,omitempty"`
}

SourceMetadata is supplied by a source, not computed or verified by Update. URL is a provenance location and may be a local path. Empty strings mean unavailable; SizeBytes <= 0 means unknown. SizeBytes is an advisory compressed size for progress, not an integrity assertion. LastModified is opaque metadata.

type Title

type Title struct {
	ID             string   `json:"imdb_id"`
	Kind           Kind     `json:"kind"`
	TitleType      string   `json:"title_type"`
	PrimaryTitle   string   `json:"primary_title"`
	OriginalTitle  string   `json:"original_title,omitempty"`
	StartYear      *int     `json:"start_year"`
	EndYear        *int     `json:"end_year"`
	RuntimeMinutes *int     `json:"runtime_minutes"`
	Adult          *bool    `json:"is_adult"`
	Genres         []string `json:"genres"`
	Rating         *float64 `json:"rating"`
	Votes          *int64   `json:"votes"`
}

Title contains IMDb facts, not provider enrichment or application state. Nil numeric/boolean fields and nil Genres are unavailable. An empty OriginalTitle is unavailable. StartYear is the release or series start year. Only movie, tvMovie, tvSeries, and tvMiniSeries title types are retained.

type UpdateOptions

type UpdateOptions struct {
	Source     Source        // nil uses the official HTTPSource
	Attempts   int           // default 1, maximum 5; retries restart the entire transaction
	RetryDelay time.Duration // default one second
	// Progress receives synchronous snapshots, including lifecycle and retry
	// events. It must return promptly and must not reenter Update. Ordinary
	// counters are throttled to about four events/second; boundaries are always
	// reported. Nil disables reporting. Cancel the context to stop an update.
	Progress func(Progress)
}

UpdateOptions controls source policy, whole-transaction retries, and reporting. Zero values select the documented defaults. Invalid options return ValidationError.

type ValidationError

type ValidationError struct {
	Field   string `json:"field"`
	Problem string `json:"problem"`
}

ValidationError identifies an invalid argument or option. Field is a Go field name (for example, Query.Title or UpdateOptions.Attempts), or an argument name. Problem describes the constraint without echoing potentially sensitive input.

Example

package main

import (
	"context"
	"errors"
	"fmt"

	imdb "github.com/odwrtw/libimdb"
)

func main() {
	_, err := imdb.Open(context.Background(), "")
	var invalid *imdb.ValidationError
	if errors.As(err, &invalid) {
		fmt.Println(invalid.Field)
	}
	fmt.Println(errors.Is(err, imdb.ErrInvalidInput))
}

Output:
path
true

func (*ValidationError) Error

func (e *ValidationError) Error() string

func (*ValidationError) Unwrap

func (e *ValidationError) Unwrap() error

type View

type View struct {
	// contains filtered or unexported fields
}

View is a read-only snapshot scoped to a Catalog.View callback. Its zero value is closed. Do not copy it, retain it, or use it concurrently. Do not call the same Catalog from its callback: a Catalog serializes operations on one connection. Results remain valid after the callback; the View itself does not.

func (*View) Candidates

func (v *View) Candidates(ctx context.Context, q Query, opts SearchOptions) (SearchResult, error)

Candidates is like Catalog.Candidates, using this view's snapshot.

func (*View) Info

func (v *View) Info(ctx context.Context) (Info, error)

Info returns metadata for this view's generation. The returned data can be modified by the caller without changing the view's metadata.

func (*View) Lookup

func (v *View) Lookup(ctx context.Context, id string) (Title, error)

Lookup is like Catalog.Lookup, using this view's snapshot.

func (*View) LookupMany

func (v *View) LookupMany(ctx context.Context, ids []string) (map[string]Title, error)

LookupMany is like Catalog.LookupMany, using this view's snapshot.

func (*View) Names

func (v *View) Names(ctx context.Context, id string) ([]Name, error)

Names is like Catalog.Names, using this view's snapshot.

func (*View) Resolve

func (v *View) Resolve(ctx context.Context, q Query) (Resolution, error)

Resolve is like Catalog.Resolve, using this view's snapshot, including any conflicts returned in AmbiguousError.