eve-sdeloader

README

eve-sdeloader

eve-sdeloader is a tool for importing an EVE static data export from EVE Developer Resources to a Postgresql database.

Installing

go get github.com/lflux/eve-sdeloader should install the tool. If this fails, check out the repository under your $GOPATH and use dep to fetch the dependencies:

$ mkdir -p $GOPATH/src/github.com/lflux
$ cd $GOPATH/src/github.com/lflux
$ git clone https://github.com/lflux/eve-sdeloader.git
$ cd eve-sdeloader
$ dep ensure
$ go install .

The tool will be in $GOPATH/bin/eve-sdeloader (by default GOPATH is ~/go as of Go 1.8)

Database preparation

Easiest way to create a database for this tool is to connect to the database as a superuser and create a role that can log in, along with a database owned by that user:

CREATE ROLE sdetest LOGIN;
CREATE DATABASE sdtest OWNER sdetest;

Usage

$ cd /tmp
$ unzip ~/Downloads/sde-20170818-TRANQUILITY.zip
$ curl -O https://raw.githubusercontent.com/fuzzysteve/yamlloader/master/invVolumes1.csv # or copy from this repo
$ curl -O https://raw.githubusercontent.com/fuzzysteve/yamlloader/master/invVolumes2.csv # or copy from this repo
# One entry needs fixing due to YAML vagaries
$ gsed -i -e 's/radius: 0.0059e18/radius: 0.0059e\+18/' sde/fsd/universe/wormhole/G-R00031/G-C00311/constellation.static
$ eve-sdeloader

By default eve-sdeloader will try to import SDE files from under ./sde and ./invVolumes{1,2}.csv to the database sdetest on localhost with the user sdetest. This is all overridable with CLI flags:

Usage of ./eve-sdeloader:
  -cpuprofile string
    	File to write cpu profile to
  -dbhost string
    	Database host (default "localhost")
  -dbname string
    	Database name (default "sdetest")
  -dbpassword string
    	Database password
  -dbport int
    	Database port (default 5432)
  -dbuser string
    	Database username (default "sdetest")
  -invvoldirectory string
    	Directory containing invVolumes{1,2}.csv (default ".")
  -memprofile string
    	File to write memory profile to
  -nobsd
    	Disable importing of BSD directory
  -nouniverse
    	Disable importing of universe data
  -sdedirectory string
    	Directory containing an unzipped EVE SDE YAML dump (default "./sde")
  -single-file string
    	Import only a single FSD file

Current status

Import works with Postgres and is 99% compatible with the importer from @fuzzysteve. A full import currently takes about 7 minutes on a Macbook Pro (to contrast with 48 minutes for the python importer). The TODO file attempts to keep track of project status.

Currently tested on Go 1.9 with Postgres 9.6 on OSX 10.12 (Sierra), but it probably works all the way back to Postgres 8.4 and should work fine on Linux.

Differences from the Python importer

• Negative 0 floats are imported as -0, whereas the python importer treats both as 0
• invtraits has different traitids due to insertion order. We attempt to keep these mostly the same by sorting types and skills, but the python importer doesn't sort at all and uses whatever order python map iteration gives you.

Profiling

eve-sdeloader accepts -cpuprofile and -memprofile flags to dump profiling output from runtime/pprof into pprof files that can be consumed by other tools.

Issues

Please raise issues on Github.

License

eve-sdeloader is copyright (c) 2017 Ian Delahorne and licensed under the MIT license, which can be found in the LICENSE file.