gosmparse

package module
Version: v1.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 17, 2021 License: MIT Imports: 9 Imported by: 2

README

OpenStreetMap PBF Parser in Go

GoDoc Build Status

gosmparse uses a callback driven API, which is stable (Documentation).

It has been designed with performance and maximum usage convenience in mind; on an Intel Core i7-6820HQ with NVMe flash it is able to process ~75 MB/s, so a planet file can be processed in under 10 minutes. If you find possible speed-ups or other improvements, let me know.

Characteristics

  • fast
  • tested with different files from different sources/generators
  • more than 85% test coverage and benchmarks for all hot spots
  • one dependency only: protobuf package (a few more are used by tests and are included in the module)
  • can read from any io.Reader (e.g. for parsing during download)
  • supports history files
Non-Features
  • Does not build geometries
  • No element cache

Install

go get -u github.com/thomersch/gosmparse

Example Usage

// Implement the gosmparser.OSMReader interface here.
// Streaming data will call those functions.
type dataHandler struct{}

func (d *dataHandler) ReadNode(n gosmparse.Node)         {}
func (d *dataHandler) ReadWay(w gosmparse.Way)           {}
func (d *dataHandler) ReadRelation(r gosmparse.Relation) {}

func ExampleNewDecoder() {
	r, err := os.Open("filename.pbf")
	if err != nil {
		panic(err)
	}
	dec := gosmparse.NewDecoder(r)
	// Parse will block until it is done or an error occurs.
	err = dec.Parse(&dataHandler{})
	if err != nil {
		panic(err)
	}
}

Download & Parse

It is possible to parse during download, so you don't have to wait for a download to finish to be able to start the parsing/processing. You can simply use the standard Go net/http package and pass resp.Body to the decoder.

resp, err := http.Get("http://download.geofabrik.de/europe/germany/bremen-latest.osm.pbf")
if err != nil {
	panic(err)
}
defer resp.Body.Close()
dec := gosmparse.NewDecoder(resp.Body)
err = dec.Parse(&dataHandler{})
if err != nil {
	panic(err)
}

Did it break?

If you found a case, where gosmparse broke, please report it and provide the file that caused the failure.

Documentation

Overview

Package gosmparse is a library for parsing OpenStreetMap binary PBF files.

It has been designed for very fast, flexible, streamed parsing of small and large files.

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Decoder

type Decoder struct {
	// QueueSize allows to tune the memory usage vs. parse speed.
	// A larger QueueSize will consume more memory, but may speed up the parsing process.
	QueueSize int
	Workers   int
	// contains filtered or unexported fields
}

A Decoder reads and decodes OSM data from an input stream.

func NewDecoder

func NewDecoder(r io.Reader) *Decoder

NewDecoder returns a new decoder that reads from r.

Example
package main

import (
	"os"

	"github.com/thomersch/gosmparse"
)

// Implement the gosmparser.OSMReader interface here.
// Streaming data will call those functions.
type dataHandler struct{}

func (d *dataHandler) ReadNode(n gosmparse.Node)         {}
func (d *dataHandler) ReadWay(w gosmparse.Way)           {}
func (d *dataHandler) ReadRelation(r gosmparse.Relation) {}

func main() {
	r, err := os.Open("filename.pbf")
	if err != nil {
		panic(err)
	}
	dec := gosmparse.NewDecoder(r)
	// Parse will block until it is done or an error occurs.
	err = dec.Parse(&dataHandler{})
	if err != nil {
		panic(err)
	}
}
Output:

func NewDecoderWithInfo

func NewDecoderWithInfo(r io.Reader) *Decoder

NewDecoderWithInfo returns a new decoder similar to NewDecoder, but will populate the Info field in the elements. Use this if you need meta data.

func (*Decoder) Parse

func (d *Decoder) Parse(o OSMReader) error

Parse starts the parsing process that will stream data into the given OSMReader.

type Element

type Element struct {
	ID   int64
	Tags map[string]string

	// Info is only populated if you use NewDecoderWithInfo.
	Info *Info
}

Element contains common attributes of an OSM element (node/way/relation).

type Info

type Info struct {
	Version   int
	Timestamp time.Time
	Changeset int64
	UID       int
	User      string
	Visible   bool
}

Info contains the metadata of an element.

type MemberType

type MemberType int

MemberType describes the type of a relation member (node/way/relation).

const (
	NodeType MemberType = iota
	WayType
	RelationType
)

type Node

type Node struct {
	Element
	Lat float64
	Lon float64
}

Node is an OSM data element with a position and tags (key/value pairs).

type OSMReader

type OSMReader interface {
	ReadNode(Node)
	ReadWay(Way)
	ReadRelation(Relation)
}

OSMReader is the interface that needs to be implemented in order to receive Elements from the parsing process.

type Relation

type Relation struct {
	Element
	Members []RelationMember
}

Relation is an OSM data element that contains multiple elements (RelationMember) and has tags (key/value pairs).

type RelationMember

type RelationMember struct {
	ID   int64
	Type MemberType
	Role string
}

RelationMember refers to an element in a relation. It contains the ID of the element (node/way/relation) and the role.

type Way

type Way struct {
	Element
	NodeIDs []int64
}

Way is an OSM data element that consists of Nodes and tags (key/value pairs). Ways can describe line strings or areas.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL