go-open-discogs-batch

README

Go OpenDiscogs Batch

Stream Discogs monthly public data dumps into PostgreSQL with bounded memory, durable progress, and idempotent recovery.

Use this importer with Go OpenDiscogs API for new deployments.

This release consumes canonical open-discogs-model v1.0.0. The model now maintains Go packages, SQL migrations, and import contracts. Its v1 migration guide describes the Java publication retirement; the SQL bytes and Go APIs are unchanged. This is an independent project and is not endorsed by Discogs.

The Java Batch repository is deprecated and archived. Existing deployments should follow its Go migration guide.

[!CAUTION] Stop all importers before applying missing canonical migrations. Upgrade each importer you intend to keep using before resuming imports; an older artifact rejects a database whose canonical migration ledger is newer than its bundled model.

• Import safety and recovery
• Performance measurements
• Releases

Quick start

The PostgreSQL database must already exist. The importer creates the selected schema when permitted, applies canonical migrations, resolves and downloads the selected dumps, then imports them.

go-open-discogs-batch \
  --database-url 'postgresql://user:password@localhost:5432/open_discogs' \
  --database-schema open_discogs \
  --entities artist,label,master,release

Omit --dump-month to select the latest dump for each entity independently. Use --dump-month=2026-07 to require every selected entity from that month. Exact-month runs use a complete durable catalog without an upstream request; otherwise discovery performs the bounded request sequence documented in Import safety and recovery.

Import contract

Boundary	Behavior
Source	Monthly public dumps only; no Discogs API, live hydration, or user writes
Memory	Dumps are decompressed and parsed as streams; the full dump is never held in memory
Commit	A root, its supported relations, and durable chunk progress commit atomically; Release backlinks reconcile once after all chunks
Retry	Compatible interrupted runs resume verified chunks and rerun pending finalization; --force restarts the same manifest from zero
Convergence	Supported missing relations are removed; roots absent from a newer dump are not deleted
Visibility	Readers can observe committed chunks; a complete monthly import is not an atomic snapshot switch
Files	Failed-run downloads remain; --cleanup removes only this run's selected files after success

Older dumps require --allow-downgrade. See Import safety and recovery for admission, locking, interruption, and resume rules.

Release data scope

Release data	Current behavior
Imported	Core fields; release artists; labels and catalog numbers; companies; formats; genres and styles; identifiers; top-level tracks; videos; release-level credited artist ID and role
Not imported	Series membership; per-track artists and extra artists; sub-track/index-track hierarchy; anv, join, and credit tracks metadata not represented by the canonical schema
Images	The audited 2026-08 public release dump has no image elements; no separate image source is used

The source code license does not grant rights to third-party data. Review the terms of every data source your application uses. Applications that also call the live Discogs API must review its current Terms of Use.

Database setup

PostgreSQL 15 or newer is required. There is no separate init command, and the importer never creates the database.

Target	Required authority
Missing schema	CREATE on the database; the importer creates the selected schema
Existing schema	USAGE and CREATE on the schema, table writes, and migration DDL authority
Restricted batch role	A DBA prepares the schema, extension, and grants first

Model v1.0.0 migrations are the only schema source of truth. Migration V007 uses CREATE EXTENSION IF NOT EXISTS pg_trgm; allow the migration role to install this trusted extension or have a DBA pre-install it in a stable schema visible to the migration role. Catalog tables are still created only in the selected database schema.

[!WARNING] Omitting --database-schema uses public and emits a warning on every startup. Prefer a dedicated schema such as open_discogs.

Schema names must be 1–63 lowercase letters, digits, or underscores and begin with a letter or underscore.

Configuration

Precedence is CLI > ENV > default.

CLI	Environment	Type	Default	Required	Valid values / purpose
--database-url	OPEN_DISCOGS_BATCH_DATABASE_URL	URI	none	yes	postgres[ql]://user:password@host[:port]/database; percent-encode credentials
--database-schema	OPEN_DISCOGS_BATCH_DATABASE_SCHEMA	string	public	no	1–63 lowercase letters, digits, or underscores; starts with a letter or underscore
--entities, -e	OPEN_DISCOGS_BATCH_ENTITIES	string list	all four	no	Non-empty subset of artist,label,master,release
--dump-month, -m	OPEN_DISCOGS_BATCH_DUMP_MONTH	yyyy-MM	latest per entity	no	2008-03 through the current calendar month
--data-dir	OPEN_DISCOGS_BATCH_DATA_DIR	path	~/.cache/open-discogs-batch	no	Writable download directory
--chunk-size, -b	OPEN_DISCOGS_BATCH_CHUNK_SIZE	integer	5000	no	1..2,147,483,647; roots per transaction chunk
--max-workers	OPEN_DISCOGS_BATCH_MAX_WORKERS	integer	visible CPU count	no	1..2,147,483,647; concurrent import workers
--cleanup, -c	OPEN_DISCOGS_BATCH_CLEANUP	boolean	false	no	Remove only selected dumps after successful import
--force, -f	OPEN_DISCOGS_BATCH_FORCE	boolean	false	no	Reprocess an otherwise skippable successful manifest from zero
--allow-downgrade	OPEN_DISCOGS_BATCH_ALLOW_DOWNGRADE	boolean	false	no	Permit and audit an older dump than the entity checkpoint
--help, -h	—	action	false	no	Show help without connecting to PostgreSQL
--version, -v	—	action	false	no	Show version without connecting to PostgreSQL

Boolean ENV values accept true/false, 1/0, yes/no, and on/off case-insensitively. OPEN_DISCOGS_BATCH_ENTITIES is comma-separated. Inject the database URI through a secret mechanism instead of command history.

Operations

Progress output

Destination	Output
Interactive stderr	One visual byte-progress bar for download or source read
Non-terminal stdout	JSON progress records: byte_progress plus durable import_progress
Terminal stdout	Structured records are suppressed

Byte progress describes transfer/read activity. Import progress reports durably committed roots, not merely parsed rows; its percentage appears only after an exact entity total is stored. Progress records are one JSON object per line. Floating-point progress metrics are numeric values rounded to at most two decimal places. Ordinary lifecycle messages can also use stdout; do not treat the entire stream as a JSON-only document. Keep stderr separate from collectors.

Resources

--max-workers limits import concurrency, not CPU usage. The default is the Go runtime's visible CPU count; no percentage heuristic is applied. Use container or scheduler limits for a hard CPU quota.

Working memory grows with chunk-size × max-workers × relation fan-out. Release has the highest fan-out. The PostgreSQL pool allows at most max-workers + 1 open connections: one per worker plus one coordinator. Agree that budget with the DBA and reduce chunk size or workers under pressure.

Container

The non-root image supports linux/amd64 and linux/arm64.

docker run --rm --cpus=4 \
  -e OPEN_DISCOGS_BATCH_DATABASE_URL='postgresql://user:password@db:5432/open_discogs' \
  -e OPEN_DISCOGS_BATCH_DATABASE_SCHEMA=open_discogs \
  -e OPEN_DISCOGS_BATCH_ENTITIES=artist,label,master,release \
  -e OPEN_DISCOGS_BATCH_DATA_DIR=/data \
  -e OPEN_DISCOGS_BATCH_MAX_WORKERS=4 \
  -v open-discogs-data:/data \
  ghcr.io/echovisionlab/go-open-discogs-batch:latest

Mount writable storage only when downloads must survive container replacement.

Development

Source builds require Go 1.27.2. Integration tests label every owned Docker resource with a per-run identity and use tmpfs for PostgreSQL. In-process cleanup and CI's always-run teardown remove only those containers, networks, and volumes, then verify that residue is zero. Persistent test volumes and bind mounts are not allowed. Each test process shares one PostgreSQL container while every test gets an isolated database that is forcibly dropped during cleanup.

gofmt -w .
go mod tidy
go vet ./...
go test -race -coverprofile=coverage.out -covermode=atomic ./...

CI checks formatting, module consistency, vet, race detection, PostgreSQL and dump E2E behavior, cleanup residue, and 100% statement coverage.

See CI and contributions for which changes run these checks.

License

MIT. See LICENSE. Retain the state303 attribution required by the license.

Documentation

Overview

Copyright © 2022 state303 <state303@dsub.io>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.