zp

README

zp - zone files parser

Handy helper to parse zone files into some data store.

An example shows how to send records to a fast Clickhouse SQL-server.

Usage

No need to say that you should get zone files first (czds.icann.org, verisign etc).

Start Clickhouse

Start the server:

docker run -itd \
           --name some-clickhouse-server \
           --ulimit nofile=262144:262144 \
           -v your-persistent-storage:/var/lib/clickhouse/data \
           -p 127.0.0.1:8123:8123 \
           yandex/clickhouse-server

Get a Clickhouse SQL shell:

docker run -it \
           --rm \
           --link some-clickhouse-server:clickhouse-server \
           yandex/clickhouse-client --host clickhouse-server

Ingest zonefiles

You do not need to unpack .gz files, the example will do it for you.

go run cmd/ingest-clickhouse/ingester.go -h
  -c string
    	Clickhouse URL (default "http://127.0.0.1:8123/default")
  -f string
    	Directory with zone files with .gz extension (default ".")
  -workers int
    	Number of sending workers (default 4)

Documentation

Index

• func FetchZoneFile(path string, tld string, rc chan Record) error
• type DBRecord
• type Record
  • func NewRecord(line string, tld string) (*Record, error)

Functions

func FetchZoneFile

func FetchZoneFile(path string, tld string, rc chan Record) error

FetchZoneFile fetches gzipped zone file and push Record entries to a channel specified in the config

Types

type DBRecord

type DBRecord struct {
	Record
	Date time.Time `db:"date"` // actual data datetime
}

DBRecord is the Record with a date

type Record

type Record struct {
	RType  string `db:"rtype"`  // record type: A, AAAA or NS by now
	Domain string `db:"domain"` // domain name without tld
	Value  string `db:"value"`  // value of the record: IPv4, IPv6 or nameserver domain name
	TLD    string `db:"tld"`    // com, name, ru etc
}

Record represents a single record

func NewRecord

func NewRecord(line string, tld string) (*Record, error)

NewRecord parses a line to a zone file record