import-registries

Documentation

Overview

Command import-registries helps grow data/brokers.yaml from public data-broker lists: the US state registries (California CPPA, Vermont, Oregon, Texas) and the IAB TCF Global Vendor List.

The US state registries are one-click CSV/XLSX downloads behind state websites, not stable APIs, so fetching those is a manual step - see docs/auditing.md for the URLs. The IAB GVL is a public JSON the tool fetches itself. Either way this tool does the tedious half: it normalises the rows, diffs them against the list already embedded in eraser, and writes two files:

candidates.yaml - rows with no obvious match in the current list, in
                  brokers.yaml shape, ready for you to add email/opt_out_url
                  and paste into data/brokers.yaml
review.md       - rows that fuzzy-match an existing broker (probably already
                  covered; skim to be sure)

Usage:

go run ./scripts/import-registries -csv registry.csv -name-col "Data Broker Name" \
    [-url-col "Website"] [-email-col "Email Address"] [-out .]

go run ./scripts/import-registries -gvl https://vendor-list.consensu.org/v3/vendor-list.json

Note: the GVL is ad-tech vendors, most of them cookie/RTB-based. It yields hundreds of candidates with no opt-out email and is best used as a periodic cross-check for missing big names, not a bulk import - see docs/auditing.md.