clone

package module
v0.0.0-...-8800bd2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 18, 2017 License: BSD-3-Clause Imports: 19 Imported by: 0

README

go-whosonfirst-clone

Tools and libraries for cloning (not syncing) Who's on First data to your local machine.

This is still very much a work in progress so you might want to wait before using it. For the adventurous...

Setup

Go

Install Go. There are package installers for Mac and Windows, and build from source options.

TIP: On Mac, verify your bash profile includes:

export PATH=$PATH:/usr/local/go/bin

Next clone this repo (assuming you haven't already done that :-)

$> git clone git@github.com:whosonfirst/go-whosonfirst-clone.git

Installation

The easiest way to install all the dependencies and compile all of the code and command line tools is to use the handy Makefile that is included with this repository, like this:

$> cd go-whosonfirst-clone
$> make deps
$> make bin

In addition to fetching all the necessary dependencies this will clone the go-whosonfirst-clone packages in to the src directory (along with all the dependencies) which is a thing you need to do because of the way Go expects code to organized. It's kind of weird and annoying but also shouting-at-the-sky territory so the Makefile is designed to hide the bother from you.

If you don't have make installed on your computer or just want to do things by hand then you should spend some time reading the Makefile itself. The revelant "targets" (which are the equivalent of commands in Makefile-speak) that you will need are deps for fetching dependencies, self for cloning files and bin for building the command line tools.

Usage

$> ./bin/wof-clone -h
Usage of ./bin/wof-clone:
  -dest string
    	Where to write files
  -loglevel string
    	    The level of detail for logging (default "info")
  -procs int
    	 The number of concurrent processes to clone data with (default 8)
  -skip-existing
	Skip existing files on disk (without checking for remote changes)
  -source string
    	  Where to look for files (default "https://whosonfirst.mapzen.com/data/")
Example
$>./bin/wof-clone -dest ../tmp/ -skip-existing /usr/local/mapzen/whosonfirst-data/meta/wof-microhood-latest.csv
[clone] 10:55:03.713219 [info] processed 35 files (ok: 0 error: 0 skipped: 35) in 0.000877 seconds

See also

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type WOFClone

type WOFClone struct {
	Source         string
	Dest           string
	Success        int64
	Error          int64
	Skipped        int64
	Scheduled      int64
	Completed      int64
	MaxFilehandles int64
	Filehandles    int64
	MaxRetries     float64 // max percentage of errors over scheduled
	Failed         []string
	Logger         *log.WOFLogger
	// contains filtered or unexported fields
}

func NewWOFClone

func NewWOFClone(source string, dest string, procs int, logger *log.WOFLogger) (*WOFClone, error)

func (*WOFClone) CloneMetaFile

func (c *WOFClone) CloneMetaFile(file string, skip_existing bool, force_updates bool) error

func (*WOFClone) ClonePath

func (c *WOFClone) ClonePath(rel_path string, ensure_changes bool) error

func (*WOFClone) EnsureFilehandles

func (c *WOFClone) EnsureFilehandles()

func (*WOFClone) Fetch

func (c *WOFClone) Fetch(method string, remote string) (*http.Response, error)

func (*WOFClone) HasChanged

func (c *WOFClone) HasChanged(local string, remote string) (bool, error)

func (*WOFClone) HasHashChanged

func (c *WOFClone) HasHashChanged(local_hash string, remote string) (bool, error)

func (*WOFClone) Process

func (c *WOFClone) Process(remote string, local string) error

func (*WOFClone) ProcessRetries

func (c *WOFClone) ProcessRetries() bool

func (*WOFClone) SetMaxFilehandles

func (c *WOFClone) SetMaxFilehandles()

func (*WOFClone) Status

func (c *WOFClone) Status()

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL