imdb2meta

README

imdb2meta

A service for getting movie and TV show metadata for an IMDb ID via HTTP, using the official IMDb datasets

Content

• Content
• Usage
  1. Import data
  2. Run service
  3. Query service
• Protocol buffer generation
• ⚠ Warning

Usage

First you need import the data of the IMDb dataset into a database, then you need to start the web service which is backed by the database and finally you can query it via HTTP.

1. Import data

First you need import the data of the IMDb dataset into a database. We support BadgerDB and bbolt.

Steps:

1. Download the title.basics.tsv.gz dataset from https://datasets.imdbws.com

   • For more info about IMDb datasets see https://www.imdb.com/interfaces/

   • ⚠ Warning: IMDb.com, Inc is the copyright owner of the data in the IMDb datasets. You may only use the data for personal and non-commercial use. For more info see "Can I use IMDb data in my software?" and their copyright/conditions of use statement.

2. Exract the TSV file somewhere

3. Run the import tool with the appropriate CLI arguments

   • Example: imdb2meta-import -tsvPath "/home/john/Downloads/data.tsv" -badgerPath "/home/john/imdb2meta/badger"

Note: The import takes a while (and much longer with bbolt than with BadgerDB), the process requires a lot of memory and the final DB size is fairly big.
With a 6-core, 12-thread CPU and a mid-range SSD, an import of all data (5820284 rows as of 2020-11-21) into BadgerDB takes 2.5 minutes, up to 760 MB memory and the final DB size is 1.14 GB.
When skipping TV episodes and storing only the minimal metadata it takes 45 seconds, up to 510 MB memory and the final DB size is 255 MB.

CLI reference:

Usage of imdb2meta-import:
  -badgerPath string
        Path to the directory with the BadgerDB files
  -boltPath string
        Path to the bbolt DB file
  -limit int
        Limit the number of rows to process (excluding the header row)
  -minimal
        Only store minimal metadata (ID, type, title, release/start year)
  -skipEpisodes
        Skip storing individual TV episodes
  -tsvPath string
        Path to the "data.tsv" file that's inside the "title.basics.tsv.gz" archive

2. Run service

After importing the data you can start the web service.

Example: imdb2meta-service -badgerPath "/home/john/imdb2meta/badger"

CLI reference:

Usage of imdb2meta-service:
  -badgerPath string
        Path to the directory with the BadgerDB files
  -bindAddr string
        Local interface address to bind to. "localhost" only allows access from the local host. "0.0.0.0" binds to all network interfaces. (default "localhost")
  -boltPath string
        Path to the bbolt DB file
  -port int
        Port to listen on (default 8080)

3. Query service

After starting the web service you can query it via HTTP:

Example request: curl "http://localhost:8080/meta/tt1254207

Example response:

{
    "id": "tt1254207",
    "titleType": "SHORT",
    "primaryTitle": "Big Buck Bunny",
    "startYear": 2008,
    "runtime": 10,
    "genres": [
        "Animation",
        "Comedy",
        "Short"
    ]
}

Protocol buffer generation

To re-generate the meta.pg.go file from the meta.proto file, run: protoc -I="./protos" --go_out=./pb --go_opt=paths=source_relative meta.proto

⚠ Warning

IMDb.com, Inc is the copyright owner of the data in the IMDb datasets. You may only use the data for personal and non-commercial use. For more info see "Can I use IMDb data in my software?" and their copyright/conditions of use statement.