fetch

README

go-whosonfirst-fetch

Tools for fetching Who's On First records and their ancestors.

Documentation

Example

import (
	"context"

	_ "github.com/whosonfirst/go-reader-whosonfirst-data"
	
	"github.com/whosonfirst/go-reader"
	"github.com/whosonfirst/go-writer"
	"github.com/whosonfirst/go-whosonfirst-fetch"
)

func main() {

	ctx := context.Background()

	r, _ := reader.NewReader(ctx, "whosonfirst-data://")
	wr, _ := writer.NewWriter(ctx, "stdout://)

	fetcher_opts, _ := fetch.DefaultOptions()
	
	fetcher, _ := fetch.NewFetcher(ctx, r, wr, fetcher_opts)

	ids := []int{
		1360695651,
	}

	belongs_to := []string{
		"all",
	}
	
	fetcher.FetchIDs(ctx, ids, belongs_to...)
}

Error handling omitted for the sake of brevity.

The example above would result in the following Who's On First documents being retrieved:

• 136/069/565/1/1360695651.geojson
• 137/769/436/9/1377694369.geojson
• 102/063/913/102063913.geojson
• 102/191/581/102191581.geojson
• 101/756/499/101756499.geojson
• 856/331/11/85633111.geojson
• 856/825/53/85682553.geojson

Tools

To build binary versions of these tools run the cli Makefile target. For example:

$> make cli
go build -mod vendor -o bin/fetch cmd/fetch/main.go

fetch

$> ./bin/fetch -h
Fetch one or more Who's on First records and, optionally, their ancestors.

Usage:
  ./bin/fetch [options] [path1 path2 ... pathN]

Options:
  -belongs-to value
    	One or more placetypes that a given ID may belong to to also fetch. You may also pass 'all' as a short-hand to fetch the entire hierarchy for a place.
  -max-clients int
    	The maximum number of concurrent requests for multiple Who's On First records. (default 10)
  -reader-uri string
    	A valid whosonfirst/go-reader URI. (default "whosonfirst-data://")
  -retries int
    	The maximum number of attempts to try fetching a record. (default 3)
  -writer-uri string
    	A valid whosonfirst/go-writer URI. (default "null://")

Notes:

pathN may be any valid Who's On First ID or URI that can be parsed by the
go-whosonfirst-uri package.

Fetch one or more Who's on First records and, optionally, their ancestors. For example:

$> ./bin/fetch -belongs-to all -writer-uri stdout:// 1360695651 | grep 'wof:name'
    "wof:name":"Berlin Brandenburg Willy Brandt Airport",
    "wof:name":"Germany",
    "wof:name":"Sch\u00f6nefeld",
    "wof:name":"Europe",
    "wof:name":"Sch\u00f6nefeld",
    "wof:name":"Dahme-Spreewald",
    "wof:name":"Brandenburg",

Under the hood this tool uses a number of other packages for handling specific tasks:

• It uses the go-reader package to read Who's On First records that are fetched.
• It uses the go-writer package to write those records.

Readers

Readers exported by the following packages are available to the fetch tool:

• https://github.com/whosonfirst/go-reader
• https://github.com/whosonfirst/go-reader-whosonfirst-data
• https://github.com/whosonfirst/go-reader-http
• https://github.com/whosonfirst/go-reader-github

Writers

Writers exported by the following packages are available to the fetch tool:

• https://github.com/whosonfirst/go-writer

See also

• https://github.com/whosonfirst/go-reader
• https://github.com/whosonfirst/go-reader-whosonfirst-data
• https://github.com/whosonfirst/go-reader-http
• https://github.com/whosonfirst/go-writer
• https://github.com/whosonfirst/go-whosonfirst-findingaid

Documentation

Overview

package fetch provides methods for fetching Who's On First records and their ancestors.

Index

• type Fetcher
  • func NewFetcher(ctx context.Context, rdr reader.Reader, wr writer.Writer, opts *Options) (*Fetcher, error)
  • func (f *Fetcher) FetchID(ctx context.Context, id int64, fetch_belongsto []string, done_ch chan bool, ...)
  • func (f *Fetcher) FetchIDs(ctx context.Context, ids []int64, belongs_to ...string) ([]int64, error)
• type Options
  • func DefaultOptions() (*Options, error)

Types

type Fetcher

type Fetcher struct {
	// contains filtered or unexported fields
}

type Fetcher is a struct for retrieving Who's On First documents.

func NewFetcher

func NewFetcher(ctx context.Context, rdr reader.Reader, wr writer.Writer, opts *Options) (*Fetcher, error)

NewFetcher returns a new `Fecther` instance configured to read Who's On First documents using 'rdr' and to store them using 'wr'. Additional configuration options are defined by 'opts'

func (*Fetcher) FetchID

func (f *Fetcher) FetchID(ctx context.Context, id int64, fetch_belongsto []string, done_ch chan bool, err_ch chan error)

FetchID will retrieve the Who's On First record for 'id'. If 'belongs_to' is non-empty it is assumed to be a list of valid Who's On First placetypes and used to determine additional ancestor records listed in each record retrieved that will subsequently be fetched. This method is designed to be run in a Go routine and signals the 'done_ch' and 'err_ch' channels with it is complete or an error is triggered.

func (*Fetcher) FetchIDs

func (f *Fetcher) FetchIDs(ctx context.Context, ids []int64, belongs_to ...string) ([]int64, error)

FetchIDs retrieves Who's On First documents matching 'ids'. If 'belongs_to' is non-empty it is assumed to be a list of valid Who's On First placetypes and used to determine additional ancestor records listed in each record retrieved that will subsequently be fetched.

type Options

type Options struct {
	// Timings is a boolean flag to indicate whether timings should be recorded
	Timings bool
	// MaxClients is the number of simultaneous clients in use to fetch Who's On First records
	MaxClients int
	// Logger is a `log.Logger` instance for providing feedback
	Logger *log.Logger
	// Retries is the number of times to retry a failed attemp to fetch a Who's On First record
	Retries int
}

Options is a struct containing configuration options for fetching Who's On First record

func DefaultOptions

func DefaultOptions() (*Options, error)

DefaultOptions returns a `Options` instance with: timings and retries disabled, the maximum number of simultaneous clients set to 10 and a `log.Default` logging instance.