zonedb

README

Public Zone Database

The Public Zone Database (ZoneDB) is a free, open-source database containing a list and associated metadata of public DNS zones (domain name extensions). It attempts to be exhaustive, including current, retired, and withdrawn top-level domains and subdomains.

The intended use case is programs that interface with the public domain name system, including registry and registrar services. The data overlaps with and shares test cases with the Public Suffix List maintained by the Mozilla Foundation. The source data was originally derived from the internal zone database in use at Domainr.

Structure

The database consists of a list of zones (zones.txt) and associated metadata in JSON format.

CLI

The zonedb CLI is fully documented. Run go run cmd/zonedb/main.go -h to see its arguments.

zones.txt

The zones.txt file is a UTF-8 encoded text file containing a list of IDN & lower-case domain names, one per line, followed by a newline (\n) character. The domain names in zones.txt are sorted according to the following rules:

1. Number of labels (top-level domains, followed by second- and third-level domains)
2. Parent domain (e.g. uk)
3. Subdomain (e.g. co.uk)

Metadata

Each domain with associated metadata will have a JSON file in the metadata directory.

RDAP

When adding an RDAP server URL, only the base URL is needed. e.g. https://rdap.sidn.nl/

Updates

A GitHub Actions workflow updates ZoneDB each night.

If a new SLD or third-level domain needs to be added to zones.txt, follow these steps locally:

1. Add the new zone to the bottom of the zones.txt file.
2. Run make normalize to normalize the data changes.
3. Run go run cmd/zonedb/main.go -update -w -zones {new zone} to update its metadata.
4. Create a pull request for the changes, and confirm the tests are passing.

JSON files

If an update to one of the {zone}.json files is needed, do this locally:

1. make normalize # tidies the json files, and runs go generate
2. commit the changes

Implementations

Go

go get github.com/zonedb/zonedb

This repository contains a reference implementation of the database written in Go. Unlike the source data, which is in Unicode, the Go implementation represents domain names in ASCII IDNA form for interop with existing libraries.

Contributing

Fork this repository, make changes, and send a pull request. Before submitting a PR, run make normalize to normalize any changes. The data is structured to minimize diff size.

zonedb build tool

This database is generated and validated using the zonedb tool in this directory. To install, make sure you have a working Go installation (1.4+) and run this command:

go get -u github.com/zonedb/zonedb/cmd/zonedb

You can also run the zonedb tool directly: go run cmd/zonedb/main.go

Example commands

List all zones that have wildcarded DNS:

zonedb -list-wildcards

List all zones tagged geo:

zonedb -tags geo

List a given zone's tags:

zonedb -zones capetown -list-tags

Add a tag to multiple zones (and write the output):

zonedb -zones capetown,durban,joburg -add-tags city -w

Remove a tag from a zone (and write the output):

zonedb -zones la -remove-tags generic -w

Add a location to a zone (and write the output):

zonedb -zones alsace -add-locations fr-a -w

Add zh-Hans-HK (Hong Kong Simplified Chinese) as a language (and write the output):

zonedb -zones 香港 -add-languages zh-Hans-HK -w

List all zones with only one language:

zonedb -grep '"languages"\: \[\s+\S+\s+\]'

List all zones with the geo tag that support a Chinese language variant:

zonedb  -tags geo -grep zh-

List all IDN zones with the geo tag:

zonedb -grep '[^\x{0000}-\x{007F}]' -tags geo

Or, using the -idn tag, all IDN TLDs with the geo tag:

zonedb -tlds -idn -tags geo

License

The ZoneDB database is licensed under the Open Database License (ODbL) version 1.0. See LICENSE-DATA.md for more information.

The Go package derived from this database is licensed under the MIT license. See LICENSE for more information.

Copyright © the Public Zone Database authors.

Documentation

Index

• Constants
• Variables
• func IsTLD(domain string) bool
• func IsZone(domain string) bool
• type Tags
  • func (tags Tags) And(q Tags) bool
  • func (tags Tags) String() string
• type Zone
  • func PublicZone(domain string) *Zone
  • func (z *Zone) AllowsIDN() bool
  • func (z *Zone) AllowsRegistration() bool
  • func (z *Zone) IsDelegated() bool
  • func (z *Zone) IsInRootZone() bool
  • func (z *Zone) IsTLD() bool
  • func (z *Zone) Languages() []string
  • func (z *Zone) RDAPURLs() []string
  • func (z *Zone) WhoisServer() string
  • func (z *Zone) WhoisURL() string

Examples

• Tags.And
• Tags.String

Constants

const (
	TagAdult          = 1
	TagBrand          = 2
	TagCity           = 4
	TagClosed         = 8
	TagCommunity      = 16
	TagCountry        = 32
	TagGeneric        = 64
	TagGeo            = 128
	TagInfrastructure = 256
	TagPrivate        = 512
	TagRegion         = 1024
	TagReserved       = 2048
	TagRetired        = 4096
	TagSpecial        = 8192
	TagSponsored      = 16384
	TagTest           = 32768
	TagWithdrawn      = 65536
)

Tag values corresponding to bit shifts (1 << iota)

Variables

var List list

List implements the cookiejar.PublicSuffixList interface.

var TLDs = z[:1757]

TLDs is a slice of all top-level domain Zones.

var TagStrings = map[Tags]string{
	TagAdult:          "adult",
	TagBrand:          "brand",
	TagCity:           "city",
	TagClosed:         "closed",
	TagCommunity:      "community",
	TagCountry:        "country",
	TagGeneric:        "generic",
	TagGeo:            "geo",
	TagInfrastructure: "infrastructure",
	TagPrivate:        "private",
	TagRegion:         "region",
	TagReserved:       "reserved",
	TagRetired:        "retired",
	TagSpecial:        "special",
	TagSponsored:      "sponsored",
	TagTest:           "test",
	TagWithdrawn:      "withdrawn",
}

TagStrings maps integer tag values to strings.

var TagValues = map[string]Tags{
	"adult":          TagAdult,
	"brand":          TagBrand,
	"city":           TagCity,
	"closed":         TagClosed,
	"community":      TagCommunity,
	"country":        TagCountry,
	"generic":        TagGeneric,
	"geo":            TagGeo,
	"infrastructure": TagInfrastructure,
	"private":        TagPrivate,
	"region":         TagRegion,
	"reserved":       TagReserved,
	"retired":        TagRetired,
	"special":        TagSpecial,
	"sponsored":      TagSponsored,
	"test":           TagTest,
	"withdrawn":      TagWithdrawn,
}

TagValues maps tag names to integer values.

var ZoneMap map[string]*Zone

ZoneMap maps domain names to Zones.

var Zones = z[:]

Zones is a slice of all Zones in the database.

Functions

func IsTLD

func IsTLD(domain string) bool

IsTLD returns true if the input domain is a top-level domain.

func IsZone

func IsZone(domain string) bool

IsZone returns true if the input domain is a Zone.

Types

type Tags

type Tags uint32

Tags are stored in a single integer as a bit field.

func (Tags) And

func (tags Tags) And(q Tags) bool

And performs a bitwise AND between tags and q, comparing the result to zero. Returns true if any tags match q.

Example

z := ZoneMap["bananarepublic"]
fmt.Println(z.Tags.And(TagBrand | TagGeo))

Output:

true

func (Tags) String

func (tags Tags) String() string

String returns a space-delimited list of values for tags.

Example

z := ZoneMap["aero"]
fmt.Println(z.Tags.String())

Output:

generic sponsored

type Zone

type Zone struct {
	// Normalized (ASCII, punycode) fully-qualified domain name
	Domain string

	// Parent Zone (nil if Zone is a TLD)
	Parent *Zone

	// Slice of subdomain (child) Zones (nil if empty)
	Subdomains []Zone

	// Tags stored as an integer bit field
	Tags Tags

	// Registry operator for this Zone
	RegistryOperator string

	// Informational URL for this Zone
	InfoURL string

	// DNS name servers for this Zone
	NameServers []string

	// Wildcard addresses for unregistered subdomains
	Wildcards []string

	// Locations associated with this Zone
	Locations []string
	// contains filtered or unexported fields
}

Zone represents a single DNS zone (a public suffix), where subdomains may be registered or created.

func PublicZone

func PublicZone(domain string) *Zone

PublicZone returns the public zone for a given domain name or nil if none found. Input must be normalized by the client (lowercase, ASCII-encoded).

func (*Zone) AllowsIDN

func (z *Zone) AllowsIDN() bool

AllowsIDN returns true if the zone operator (registry) permits registration of non-ASCII labels under this Zone.

func (*Zone) AllowsRegistration

func (z *Zone) AllowsRegistration() bool

AllowsRegistration returns true if the Zone’s authority (registry) permits registration of subdomains of this Zone. Examples include zones that are closed, withdrawn, or retired zones. A closed zone is where the registry operator does not permit registration, typically at the second level, like .ck. A withdrawn zone is a gTLD that was withdrawn and removed from the root zone file by the RO. A retired zone was active, but has since been undelegated.

func (*Zone) IsDelegated

func (z *Zone) IsDelegated() bool

IsDelegated returns true if the Zone has name servers.

func (*Zone) IsInRootZone

func (z *Zone) IsInRootZone() bool

IsInRootZone returns true if the Zone is a top-level domain. present in the root DNS zone.

func (*Zone) IsTLD

func (z *Zone) IsTLD() bool

IsTLD returns true if the Zone is a top-level domain.

func (*Zone) Languages

func (z *Zone) Languages() []string

Languages returns a slice of BCP 47 language specifiers.

func (*Zone) RDAPURLs

func (z *Zone) RDAPURLs() []string

RDAPURLs returns the set of RDAP URL endpoints for a zone.

func (*Zone) WhoisServer

func (z *Zone) WhoisServer() string

WhoisServer returns the whois server that responds on port 43 for the zone. It first searches the specific zone, then the parent, returning an empty string if none found.

func (*Zone) WhoisURL

func (z *Zone) WhoisURL() string

WhoisURL returns a URL to retrieve whois data for a subdomain of the zone. It first searches the specific zone, then the parent, returning an empty string if none found.