osmbr

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 27, 2026 License: MIT Imports: 8 Imported by: 0

README

osmbr

A low-level Go library for reading OpenStreetMap PBF files. Designed for minimal allocation and caller-controlled memory.

Design

  • Caller-managed buffers — Allocate buffer structs once and reuse them across blocks. After warm-up, reading a whole file makes zero heap allocations.
  • Scanner-style APIs — Sequential reads with sticky errors checked after iteration.
  • Raw values — Returns raw integers for coordinates and string-table indices for tags. The caller applies granularity/offset conversion.
  • No domain types — There are no Node, Way, or Relation structs. The caller reads fields from buffer structs and builds whatever representation it needs.

Protobuf decoding is done in-package against the PBF schema, so the only dependency is klauspost/compress for DEFLATE.

Install

go get github.com/invisiblefunnel/osmbr

Requires Go 1.24 or later.

Usage

f, err := os.Open("region.osm.pbf")
if err != nil {
    log.Fatal(err)
}
defer f.Close()

var (
    dec   osmbr.Decompressor
    pb    osmbr.PrimitiveBlock
    dnBuf osmbr.DenseNodesBuf
    wBuf  osmbr.WayBuf
    rBuf  osmbr.RelationBuf
)

br := osmbr.NewBlockReader(f)
for br.Next() {
    if br.Type() != "OSMData" {
        continue
    }
    data, err := dec.Decompress(br.Blob())
    if err != nil {
        log.Fatal(err)
    }
    if err := pb.DecodeFrom(data); err != nil {
        log.Fatal(err)
    }

    gs := pb.Groups()
    for gs.Next() {
        switch gs.Type() {
        case osmbr.GroupTypeDense:
            if err := gs.DecodeDenseNodes(&dnBuf, nil); err != nil {
                log.Fatal(err)
            }
            for i, id := range dnBuf.IDs {
                lat := dnBuf.Lats[i]*int64(pb.Granularity) + pb.LatOffset
                lon := dnBuf.Lons[i]*int64(pb.Granularity) + pb.LonOffset
                _ = id
                _ = lat
                _ = lon
            }

        case osmbr.GroupTypeWays:
            ws := gs.WayScanner()
            for id, ok := ws.Next(&wBuf, nil); ok; id, ok = ws.Next(&wBuf, nil) {
                _ = id       // way ID
                _ = wBuf.Refs // referenced node IDs (absolute)
            }
            if err := ws.Err(); err != nil {
                log.Fatal(err)
            }

        case osmbr.GroupTypeRelations:
            rs := gs.RelationScanner()
            for id, ok := rs.Next(&rBuf, nil); ok; id, ok = rs.Next(&rBuf, nil) {
                _ = id          // relation ID
                _ = rBuf.MemIDs // member IDs (absolute)
                _ = rBuf.Types  // member types (MemberTypeNode, MemberTypeWay, MemberTypeRelation)
            }
            if err := rs.Err(); err != nil {
                log.Fatal(err)
            }
        }
    }
    if err := gs.Err(); err != nil {
        log.Fatal(err)
    }
}
if err := br.Err(); err != nil {
    log.Fatal(err)
}

API overview

The reading pipeline flows top-down:

BlockReader

NewBlockReader(r io.Reader) reads PBF file blocks sequentially. Call Next() to advance, then Type() for the block type ("OSMHeader" or "OSMData") and Blob() for the raw Blob protobuf bytes. Blob() points into the reader's own storage and is invalidated by the next read. Use a Decompressor to decompress it.

Errors stop the walk for good: a failed read leaves the stream parked mid-block, where the following bytes are payload rather than a length prefix, so every later Next() reports false and Err() keeps returning that first failure until you Reset.

The zero value is ready to use after Reset(r), so a BlockReader can live inside a struct you already own:

type worker struct {
    br  osmbr.BlockReader
    dec osmbr.Decompressor
}

func (w *worker) run(r io.Reader) {
    w.br.Reset(r)
    for w.br.Next() {
        // ...
    }
}

Reset also lets one reader walk many files without reallocating.

Handing blocks to other goroutines

When a blob must outlive the next read — a producer feeding worker goroutines, say — use NextInto(dst []byte) ([]byte, bool) instead. It reads directly into caller-owned storage, allocating a larger slice only when cap(dst) is too small, so the returned slice may not share storage with dst; retain the returned one. On EOF or error it returns dst[:0], so a pooled buffer is never lost.

var pool sync.Pool
br := osmbr.NewBlockReader(f)
for {
    buf, _ := pool.Get().([]byte)
    blob, ok := br.NextInto(buf)
    if !ok {
        pool.Put(blob) // dst[:0] — nothing lost
        break
    }
    if br.Type() != "OSMData" {
        pool.Put(blob[:0])
        continue
    }
    jobs <- blob // worker returns blob[:0] to pool when done
}
if err := br.Err(); err != nil {
    log.Fatal(err)
}

See examples/count for the full producer/worker pattern.

Header

DecodeHeader(data []byte) decodes a decompressed OSMHeader block, returning a Header with the bounding box, required/optional features, writing program, source, and replication metadata.

Decompressor

Decompress(blob []byte) parses and decompresses a raw Blob message, returning the decompressed payload. Allocate one per goroutine and reuse across blocks. The returned slice points into the Decompressor's own storage and is valid until the next Decompress call — including for uncompressed (raw) blobs, which are copied rather than aliased so that advancing the BlockReader can never rewrite a payload you still hold.

osmbr reads the zlib wrapper itself and drives klauspost/compress's DEFLATE reader directly, reusing its state across blocks.

Every zlib blob's Adler-32 trailer is verified by default, which costs about 14% of decompression time (~10% of a whole-file read). Inflating rejects most corruption on its own and Decompress independently requires the output to end exactly at Blob.raw_size, but neither covers a stored (uncompressed) DEFLATE block: a flipped bit there changes neither the stream's structure nor its length, so the checksum is the only thing that catches it. Skip the check only for input whose integrity is already assured:

dec := osmbr.Decompressor{SkipChecksum: true}

Note the raw_size length check applies only when Blob.raw_size is present. Without it, output is bounded by the 32 MiB blob limit and nothing else.

Only raw and zlib_data blobs are supported; lzma, bzip2, lz4, and zstd blobs return an error.

PrimitiveBlock

DecodeFrom(data []byte) populates the block's string table, granularity, and coordinate offsets from decompressed block data. Call Groups() to get a GroupScanner. String table entries are zero-copy slices into the data — copy any strings you need to retain past the next Decompress call.

  • String(i int) []byte — look up a string table entry by index
  • Granularity / LatOffset / LonOffset / DateGranularity — coordinate and timestamp conversion parameters
GroupScanner

Iterates over PrimitiveGroup messages within a block. Call Type() to check the group kind, then use the appropriate decoder. Check Err() after iteration finishes to distinguish error from EOF.

GroupType Decoder
GroupTypeDense gs.DecodeDenseNodes(&buf, info)
GroupTypeWays gs.WayScanner()
GroupTypeRelations gs.RelationScanner()
GroupTypeNodes gs.NodeScanner()
Buffer types
Type Fields Notes
DenseNodesBuf IDs, Lats, Lons, KeysVals Parallel arrays; IDs/Lats/Lons are absolute (delta-decoded)
WayBuf Keys, Vals, Refs Refs are absolute node IDs (delta-decoded)
RelationBuf Keys, Vals, RolesSID, MemIDs, Types MemIDs absolute (delta-decoded)
NodeBuf Keys, Vals Individual nodes (rare in practice)
InfoBuf Version, Timestamp, Changeset, UID, UserSID, Visible Per-entity metadata
DenseInfoBuf Versions, Timestamps, Changesets, UIDs, UserSIDs, Visibles Per-node metadata arrays; Timestamps/Changesets/UIDs/UserSIDs absolute (delta-decoded)

Pass nil for the info parameter to skip metadata decoding.

Coordinate conversion

The library returns raw integers. Convert to nanodegrees using the block's parameters:

lat_nanodeg := dnBuf.Lats[i]*int64(pb.Granularity) + pb.LatOffset
lon_nanodeg := dnBuf.Lons[i]*int64(pb.Granularity) + pb.LonOffset

Default granularity is 100 nanodegrees. Default offsets are 0.

To get degrees, divide by 1e9:

latDeg := float64(lat_nanodeg) / 1e9
lonDeg := float64(lon_nanodeg) / 1e9

Tag decoding

Tags are pairs of string-table indices. For ways and relations, Keys and Vals are parallel arrays:

for i := range wBuf.Keys {
    key := pb.String(int(wBuf.Keys[i]))
    val := pb.String(int(wBuf.Vals[i]))
    fmt.Printf("%s = %s\n", key, val)
}

For dense nodes, tags are packed into a flat KeysVals array with 0 delimiters between nodes:

j := 0
for i := range dnBuf.IDs {
    // j+1 keeps a malformed trailing key with no value from reading past the end.
    for j+1 < len(dnBuf.KeysVals) && dnBuf.KeysVals[j] != 0 {
        key := pb.String(int(dnBuf.KeysVals[j]))
        val := pb.String(int(dnBuf.KeysVals[j+1]))
        j += 2
        fmt.Printf("node %d: %s = %s\n", dnBuf.IDs[i], key, val)
    }
    j++ // skip the 0 delimiter
}

KeysVals is not validated, and neither are the indices in it. String(i) panics on an out-of-range index, so check against NumStrings() when reading files you do not trust.

Metadata

Pass an InfoBuf or DenseInfoBuf to decode version, timestamp, changeset, and user metadata. Pass nil to skip it.

For ways and relations:

var iBuf osmbr.InfoBuf
for id, ok := ws.Next(&wBuf, &iBuf); ok; id, ok = ws.Next(&wBuf, &iBuf) {
    fmt.Printf("way %d: v%d changeset=%d user=%s\n",
        id, iBuf.Version, iBuf.Changeset, pb.String(int(iBuf.UserSID)))
}

For dense nodes:

var diBuf osmbr.DenseInfoBuf
gs.DecodeDenseNodes(&dnBuf, &diBuf)
for i, id := range dnBuf.IDs {
    ts := diBuf.Timestamps[i] * int64(pb.DateGranularity) // milliseconds since epoch
    fmt.Printf("node %d: v%d ts=%d\n", id, diBuf.Versions[i], ts)
}

DenseInfoBuf.UserSIDs is delta-decoded, while the single-entity InfoBuf.UserSID is not — the PBF schema declares DenseInfo.user_sid as a delta-coded sint32 and Info.user_sid as a plain uint32. Both index the block's string table; only the dense one needs unwinding, and osmbr does it for you.

DecodeDenseNodes guarantees each DenseInfoBuf array is either empty or exactly as long as IDs, rejecting files that disagree, so a non-empty array indexes by node position without a bounds check of its own. An array is empty when the file omits that field — Visibles outside full-history extracts, or all of them for a file stripped of metadata — so check len() once before the loop rather than per node.

Both buffers are cleared at the start of every call, so one of each can be reused for a whole file. Metadata is optional per entity and per group: an entity with no Info reads back as the zero InfoBuf, and a group with no DenseInfo leaves every DenseInfoBuf array empty, rather than either one retaining what the previous entity or group left behind.

Correctness

The decoders here are written for speed: unrolled varint loops, fused delta accumulation, buffers reused across blocks, zlib driven through flate rather than compress/zlib. None of that is safe to check against tests that assert what the same code produced, so it is checked against a second decoder instead.

reference_test.go is a complete PBF decoder written for obviousness — one value per loop iteration, no unrolling, no buffer reuse, compress/zlib for inflation, allocating freely. Where it is deliberately strict or deliberately lax, the rule is stated where it is applied and attributed to the protobuf spec, the PBF format, or a documented osmbr choice. Its scalar conversions and varint limits are pinned to google.golang.org/protobuf's by table tests, so the oracle cannot quietly drift into agreeing with a bug.

differential_test.go requires the two to agree on every entry point, in two senses: identical error behaviour, meaning both accept an input or both reject it, and identical output, meaning every decoded value matches exactly when both accept. Values are compared only when both accept, since a decoder that reports an error is free to leave anything in the caller's buffers.

Beyond value comparison, the oracle pins the invariants that only hold across calls: Next, NextInto, and Reset agree with each other and with a reader that returns one byte at a time; a Decompressor survives every blob, valid or not, and decodes the same bytes on reuse; SkipChecksum never rejects a blob the default accepts; and a whole file read the documented way — one BlockReader recycling one buffer, feeding one Decompressor — matches a fresh decoder per block. That last one is the only check that spans blocks, which is where a buffer handed out with the wrong lifetime would show.

fuzz_test.go has a native fuzz target per entry point plus one end-to-end over whole files, each wired to the oracle. Seeds combine synthetic messages that reach specific branches with the first few entities of the bundled extract re-encoded compactly. Inputs that once decoded wrongly live in testdata/fuzz/ and are replayed by plain go test; TestDiffOracleOnRealFile checks the whole extract, uncapped, on every run.

go test ./...                                   # includes the seed corpus and the real-file oracle
go test -run '^FuzzPBFFile$' -fuzz '^FuzzPBFFile$' -fuzztime=5m .

The fuzz targets bound their own cost — an input size cap, and limits on repeated and total inflation — because an oracle slow enough to stall a fuzzing worker reports nothing at all. Each bound says at its definition what it gives up. A pull request runs the seed corpus under the race detector plus a short fuzz run per target; a scheduled weekly job fuzzes each target for longer in its own job and uploads anything it finds.

Performance

Measured on the bundled 3.1 MB extract (Go 1.26, arm64), reading every block through decompression and full entity decode, with Adler-32 verification at its default setting:

time/op allocs/op
Whole file, no metadata ~23 ms 0
Whole file, with metadata ~25 ms 0

Timings drift a few percent between runs on the same machine, so treat them as a scale rather than a target; the allocation counts are exact.

Decompression accounts for roughly 79% of the total, and the rest is protobuf decode. Checksum verification is about 14% of the decompression figure, so SkipChecksum: true takes a whole-file read down to roughly 21 ms / 23 ms. Reproduce with:

go test -bench=. -benchmem -run=^$ .

The benchmark suite has two tiers: micro-benchmarks over synthetic inputs that isolate each hot path, and end-to-end benchmarks over the bundled extract. Use benchstat to compare runs.

Non-goals

This library intentionally does not provide:

  • Domain types — No Node/Way/Relation structs. Build your own from the buffer fields.
  • Filtering — All entities in a block are decoded. Skip what you don't need in your loop.
  • Concurrency — Single-threaded. Parallelize at the block level in your own code.
  • Semantic conversion — Coordinates stay as raw integers; timestamps stay as raw values. The caller applies the conversion.

Acknowledgments

Inspired by tidwall/osmfile.

Portions of the implementation, tests, and documentation were developed with the assistance of Claude Code.

Documentation

Index

Examples

Constants

View Source 
const (
	MemberTypeNode     = int32(0)
	MemberTypeWay      = int32(1)
	MemberTypeRelation = int32(2)
)

Member type constants for Relation members.

Variables

This section is empty.

Functions

func DecodeDenseNodes 

func DecodeDenseNodes(groupData []byte, buf *DenseNodesBuf, info *DenseInfoBuf) error

DecodeDenseNodes decodes a DenseNodes PrimitiveGroup into buf. groupData is the raw bytes of a PrimitiveGroup message (from GroupScanner.groupData). Resets all slices to [:0] then appends. Delta-decodes IDs, Lats, Lons. Pass a non-nil info to also decode DenseInfo metadata; nil skips it.

Types

type BlockReader 

type BlockReader struct {
	// contains filtered or unexported fields
}

BlockReader reads PBF FileBlocks from an io.Reader. The zero value is ready to use after Reset, so a BlockReader can be embedded in a caller-owned struct and reused without allocating.

Call Next to advance and Blob to get the current block's raw Blob protobuf message, or NextInto to read into caller-owned storage. Type and Offset describe the current block. Use a Decompressor to decompress the blob.

BlockReader is not safe for concurrent use.

Example 
package main

import (
	"fmt"
	"os"

	"github.com/invisiblefunnel/osmbr"
)

func main() {
	f, err := os.Open("testdata/us-virgin-islands-260414.osm.pbf")
	if err != nil {
		fmt.Println(err)
		return
	}
	defer f.Close()

	br := osmbr.NewBlockReader(f)
	for br.Next() {
		fmt.Printf("type=%s offset=%d len=%d\n", br.Type(), br.Offset(), len(br.Blob()))
		break // first block only for the example
	}
	if err := br.Err(); err != nil {
		fmt.Println(err)
	}
}

Output:
type=OSMHeader offset=0 len=193

func NewBlockReader 

func NewBlockReader(r io.Reader) *BlockReader

NewBlockReader returns a BlockReader that reads PBF blocks from r.

Equivalent to declaring a BlockReader and calling Reset; use that form to keep the reader inside a struct you already own.

func (*BlockReader) Blob 

func (br *BlockReader) Blob() []byte

Blob returns the raw Blob protobuf message bytes read by the most recent call to Next. It is invalidated by the next call to Next or NextInto, and is empty once Next has returned false.

func (*BlockReader) Err 

func (br *BlockReader) Err() error

Err returns the first non-EOF error encountered, which is also the only one: reading stops at the first failure. Reset clears it.

func (*BlockReader) Next 

func (br *BlockReader) Next() bool

Next reads the next FileBlock into br's own storage, which Blob then returns. It reports false on EOF or error; call Err to distinguish them. Errors are sticky, so a loop may check Err once after it ends.

Use NextInto instead when the blob must outlive the next read, such as when handing blocks to worker goroutines.

func (*BlockReader) NextInto added in v0.3.0 

func (br *BlockReader) NextInto(dst []byte) ([]byte, bool)

NextInto reads the next FileBlock into dst, overwriting it, and returns the slice holding the raw Blob protobuf message. When cap(dst) is too small it allocates a larger slice instead, so the returned slice may have a different backing array than dst.

It reports false on EOF or error, returning dst[:0] so a caller that pools buffers never loses one. Call Err to distinguish EOF from an error. After a successful call, Type and Offset describe the current block.

Errors are sticky. A failure leaves the underlying reader positioned mid-block, where the next four bytes are payload rather than a length prefix, so reading on would decode garbage as blocks; once a call fails, every later one reports false until Reset.

Example 
package main

import (
	"fmt"
	"os"
	"sync"

	"github.com/invisiblefunnel/osmbr"
)

func main() {
	f, err := os.Open("testdata/us-virgin-islands-260414.osm.pbf")
	if err != nil {
		fmt.Println(err)
		return
	}
	defer f.Close()

	// NextInto reads into caller-owned storage, so a blob stays valid after
	// the reader moves on — what a producer feeding worker goroutines needs.
	// On EOF or error it hands the buffer back, so nothing leaks out of the
	// pool.
	var pool sync.Pool
	br := osmbr.NewBlockReader(f)
	buf, _ := pool.Get().([]byte)
	blob, ok := br.NextInto(buf)
	if !ok {
		pool.Put(blob)
		fmt.Println(br.Err())
		return
	}
	fmt.Printf("type=%s len=%d\n", br.Type(), len(blob))
	pool.Put(blob[:0])
}

Output:
type=OSMHeader len=193

func (*BlockReader) Offset 

func (br *BlockReader) Offset() int64

Offset returns the byte offset where the current block starts in the underlying reader. Use with io.Seeker to re-read a specific block later.

func (*BlockReader) Reset added in v0.2.0 

func (br *BlockReader) Reset(r io.Reader)

Reset reuses br to read from r, preserving buffer capacities. Use this to walk many files with one BlockReader instead of allocating a new one per file.

func (*BlockReader) Type 

func (br *BlockReader) Type() string

Type returns the block type ("OSMHeader" or "OSMData").

type Decompressor 

type Decompressor struct {
	// SkipChecksum turns off validation of the Adler-32 trailer on zlib blobs.
	//
	// Verification is on by default and costs roughly 14% of decompression
	// time. Inflating catches most corruption on its own, and Decompress
	// independently requires the output to end exactly at Blob.raw_size, but
	// neither covers a stored (uncompressed) DEFLATE block: a flipped bit
	// there changes neither the stream's structure nor its length, so the
	// checksum is the only thing standing between it and the caller.
	//
	// Set it before the first call to Decompress, and only for input whose
	// integrity is already assured.
	SkipChecksum bool
	// contains filtered or unexported fields
}

Decompressor parses and decompresses raw PBF Blob messages. Allocate one per goroutine and reuse across blocks to avoid per-block allocations.

Decompressor is not safe for concurrent use.

Example 
package main

import (
	"fmt"
	"os"

	"github.com/invisiblefunnel/osmbr"
)

func main() {
	f, err := os.Open("testdata/us-virgin-islands-260414.osm.pbf")
	if err != nil {
		fmt.Println(err)
		return
	}
	defer f.Close()

	var dec osmbr.Decompressor
	br := osmbr.NewBlockReader(f)
	if !br.Next() {
		fmt.Println("no blocks")
		return
	}
	data, err := dec.Decompress(br.Blob())
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Printf("decompressed %d bytes\n", len(data))
}

Output:
decompressed 179 bytes

func (*Decompressor) Decompress 

func (d *Decompressor) Decompress(blob []byte) ([]byte, error)

Decompress parses a raw Blob protobuf message and returns the decompressed payload. The returned slice points into the Decompressor's own storage and is valid until the next call to Decompress, whatever compression the Blob used.

type DenseInfoBuf 

type DenseInfoBuf struct {
	Versions   []int32
	Timestamps []int64 // delta-decoded; milliseconds since Unix epoch
	Changesets []int64 // delta-decoded
	UIDs       []int32 // delta-decoded
	UserSIDs   []int32 // delta-decoded; indices into the block's string table
	Visibles   []bool
}

DenseInfoBuf holds optional per-node metadata arrays decoded from a DenseInfo message. All slices are grown as needed (capacity preserved across calls). Pass a non-nil *DenseInfoBuf to DecodeDenseNodes to populate; nil skips it.

DecodeDenseNodes guarantees every array here is either empty or exactly as long as DenseNodesBuf.IDs, so an array that is non-empty can be indexed by node position without a bounds check of its own. A file that violates this is rejected rather than decoded. An array is empty whenever the group omits that field — including when the group carries no DenseInfo at all, which clears anything a previous group left behind.

Delta-decoded fields: Timestamps, Changesets, UIDs, UserSIDs. Non-delta fields: Versions.

Note UserSIDs is delta-coded here while the single-entity InfoBuf.UserSID is not: the PBF schema declares DenseInfo.user_sid as a delta-coded sint32 and Info.user_sid as a plain uint32.

type DenseNodesBuf 

type DenseNodesBuf struct {
	IDs      []int64
	Lats     []int64
	Lons     []int64
	KeysVals []int32
}

DenseNodesBuf is caller-managed memory for decoding a DenseNodes group. Allocate once and reuse across blocks to avoid per-block allocations. After warm-up, all slices grow to accommodate the largest block seen and are then reused without further allocation.

IDs, Lats, and Lons contain delta-decoded absolute values. To convert Lats[i] and Lons[i] to nanodegrees: 

lat_nanodeg = Lats[i] * int64(pb.Granularity) + pb.LatOffset
lon_nanodeg = Lons[i] * int64(pb.Granularity) + pb.LonOffset

KeysVals encodes tags as a flat array of string-table indices: 

(keyIdx valIdx)* 0  per node, repeated

The 0 value delimits one node's tags from the next. KeysVals is not validated: a malformed file may end mid-pair, and the indices themselves are not checked against the string table, so bound the pair read and use NumStrings before calling String. Example iteration: 

j := 0
for i := range buf.IDs {
    // j+1 keeps a trailing key with no value from reading past the end.
    for j+1 < len(buf.KeysVals) && buf.KeysVals[j] != 0 {
        key := pb.String(int(buf.KeysVals[j]))
        val := pb.String(int(buf.KeysVals[j+1]))
        j += 2
    }
    j++ // skip the 0 delimiter
}
Example 
package main

import (
	"fmt"
	"os"

	"github.com/invisiblefunnel/osmbr"
)

func main() {
	f, err := os.Open("testdata/us-virgin-islands-260414.osm.pbf")
	if err != nil {
		fmt.Println(err)
		return
	}
	defer f.Close()

	var (
		dec   osmbr.Decompressor
		pb    osmbr.PrimitiveBlock
		dnBuf osmbr.DenseNodesBuf
	)
	br := osmbr.NewBlockReader(f)
	for br.Next() {
		if br.Type() != "OSMData" {
			continue
		}
		data, err := dec.Decompress(br.Blob())
		if err != nil {
			fmt.Println(err)
			return
		}
		if err := pb.DecodeFrom(data); err != nil {
			fmt.Println(err)
			return
		}
		gs := pb.Groups()
		for gs.Next() {
			if gs.Type() != osmbr.GroupTypeDense {
				continue
			}
			if err := gs.DecodeDenseNodes(&dnBuf, nil); err != nil {
				fmt.Println(err)
				return
			}
			// Convert the first node's raw lat/lon to nanodegrees.
			latNanodeg := dnBuf.Lats[0]*int64(pb.Granularity) + pb.LatOffset
			lonNanodeg := dnBuf.Lons[0]*int64(pb.Granularity) + pb.LonOffset
			fmt.Printf("first node id=%d lat=%d lon=%d\n",
				dnBuf.IDs[0], latNanodeg, lonNanodeg)
			return
		}
	}
}

Output:
first node id=38344686 lat=17757614800 lon=-64585070900

type GroupScanner 

type GroupScanner struct {
	// contains filtered or unexported fields
}

GroupScanner iterates over PrimitiveGroups within a PrimitiveBlock. Obtain one via PrimitiveBlock.Groups. GroupScanner is a value type.

func (*GroupScanner) DecodeDenseNodes 

func (gs *GroupScanner) DecodeDenseNodes(buf *DenseNodesBuf, info *DenseInfoBuf) error

DecodeDenseNodes decodes the current DenseNodes group into buf. Only valid when Type() == GroupTypeDense. Pass a non-nil info to also decode per-node metadata; nil skips it.

func (*GroupScanner) Err added in v0.1.1 

func (gs *GroupScanner) Err() error

Err returns the first error encountered during iteration.

func (*GroupScanner) Next 

func (gs *GroupScanner) Next() bool

Next advances to the next PrimitiveGroup. Returns false on EOF or error. Call Err to distinguish between them.

func (*GroupScanner) NodeScanner 

func (gs *GroupScanner) NodeScanner() NodeScanner

NodeScanner returns a NodeScanner for the current group. Only valid when Type() == GroupTypeNodes. Note: non-dense nodes are rare in practice; most OSM data uses DenseNodes.

func (*GroupScanner) RelationScanner 

func (gs *GroupScanner) RelationScanner() RelationScanner

RelationScanner returns a RelationScanner for the current group. Only valid when Type() == GroupTypeRelations.

func (*GroupScanner) Type 

func (gs *GroupScanner) Type() GroupType

Type returns the GroupType of the current group.

func (*GroupScanner) WayScanner 

func (gs *GroupScanner) WayScanner() WayScanner

WayScanner returns a WayScanner for the current group. Only valid when Type() == GroupTypeWays.

type GroupType 

type GroupType int8

GroupType identifies the kind of entities in a PrimitiveGroup. 

const (
	GroupTypeUnknown    GroupType = 0
	GroupTypeNodes      GroupType = 1
	GroupTypeDense      GroupType = 2
	GroupTypeWays       GroupType = 3
	GroupTypeRelations  GroupType = 4
	GroupTypeChangesets GroupType = 5
)
type Header struct {
	// BBox is the bounding box of the data in nanodegrees.
	// Left and Right are longitude; Top and Bottom are latitude.
	BBox HeaderBBox

	// RequiredFeatures lists features a parser must support (e.g. "OsmSchema-V0.6", "DenseNodes").
	RequiredFeatures []string

	// OptionalFeatures lists features a parser may optionally handle (e.g. "Sort.Type_then_ID").
	OptionalFeatures []string

	// WritingProgram identifies the tool that created the file (e.g. "osmium/1.16.0").
	WritingProgram string

	// Source identifies the data source.
	Source string

	// ReplicationTimestamp is seconds since Unix epoch for the replication state.
	ReplicationTimestamp int64

	// ReplicationSequenceNumber is the replication sequence number.
	ReplicationSequenceNumber int64

	// ReplicationBaseURL is the base URL for replication diff files.
	ReplicationBaseURL string
}

Header holds the decoded contents of an OSMHeader block.

func DecodeHeader 

func DecodeHeader(data []byte) (Header, error)

DecodeHeader decodes a decompressed OSMHeader block.

type HeaderBBox 

type HeaderBBox struct {
	Left, Right, Top, Bottom int64
}

HeaderBBox holds bounding box coordinates in nanodegrees.

type InfoBuf 

type InfoBuf struct {
	Version    int32
	Timestamp  int64 // milliseconds since Unix epoch
	Changeset  int64
	UID        int32
	UserSID    uint32 // index into the block's string table
	Visible    bool
	HasVisible bool // false if the visible field was absent
}

InfoBuf holds optional per-entity metadata decoded from an Info message. Pass a non-nil *InfoBuf to WayScanner.Next, RelationScanner.Next, or NodeScanner.Next to populate it; nil skips decoding entirely.

Each Next zeroes the buffer before decoding, so one InfoBuf can be reused across a whole file: an entity carrying no Info reads back as the zero value rather than as the previous entity's metadata.

type NodeBuf 

type NodeBuf struct {
	Keys []uint32 // string table indices for tag keys
	Vals []uint32 // string table indices for tag values
}

NodeBuf is caller-managed memory for decoding individual Node entities. Non-dense nodes are rare in practice; most OSM data uses DenseNodes. Reuse across calls to avoid per-node allocations.

type NodeScanner 

type NodeScanner struct {
	// contains filtered or unexported fields
}

NodeScanner iterates over individual Node messages in a PrimitiveGroup. Obtain one via GroupScanner.NodeScanner. NodeScanner is a value type. Note: in practice, OSM planet files and extracts use DenseNodes exclusively.

func (*NodeScanner) Err 

func (ns *NodeScanner) Err() error

Err returns the first error encountered during iteration.

func (*NodeScanner) Next 

func (ns *NodeScanner) Next(buf *NodeBuf, info *InfoBuf) (id, lat, lon int64, ok bool)

Next decodes the next Node into buf and returns its ID, lat, and lon. Resets buf slices to [:0] then appends (capacity preserved). Returns (0, 0, 0, false) when no more nodes remain. lat and lon are raw sint64 values. Convert to nanodegrees: 

lat_nanodeg = lat * int64(pb.Granularity) + pb.LatOffset

Pass a non-nil info to also decode the Node's Info; nil skips it.

type PrimitiveBlock 

type PrimitiveBlock struct {
	// Granularity is the coordinate granularity in nanodegrees (default 100).
	// To convert a raw lat/lon integer to nanodegrees:
	//
	//   lat_nanodeg = Lats[i] * int64(Granularity) + LatOffset
	//   lon_nanodeg = Lons[i] * int64(Granularity) + LonOffset
	Granularity int32
	// LatOffset is the latitude offset in nanodegrees (default 0).
	LatOffset int64
	// LonOffset is the longitude offset in nanodegrees (default 0).
	LonOffset int64
	// DateGranularity is the timestamp granularity in milliseconds (default 1000).
	DateGranularity int32
	// contains filtered or unexported fields
}

PrimitiveBlock holds the decoded metadata and string table for an OSMData block. Call DecodeFrom to populate from a Decompressor's output. Call Groups to iterate groups.

String table entries are zero-copy slices into the data passed to DecodeFrom. They are only valid until the next call to DecodeFrom on this block or to Decompressor.Decompress on the underlying decompressor (which reuses its buffer).

func (*PrimitiveBlock) DecodeFrom 

func (pb *PrimitiveBlock) DecodeFrom(data []byte) error

DecodeFrom populates the PrimitiveBlock from decompressed OSMData block bytes (typically the result of Decompressor.Decompress).

String table entries reference data's memory. Copy entries you need to retain past the next Decompressor.Decompress or DecodeFrom call.

func (*PrimitiveBlock) Groups 

func (pb *PrimitiveBlock) Groups() GroupScanner

Groups returns a GroupScanner for iterating over the PrimitiveGroups in this block. The scanner re-reads from the original block data.

Example 
package main

import (
	"fmt"
	"os"

	"github.com/invisiblefunnel/osmbr"
)

func main() {
	f, err := os.Open("testdata/us-virgin-islands-260414.osm.pbf")
	if err != nil {
		fmt.Println(err)
		return
	}
	defer f.Close()

	var (
		dec osmbr.Decompressor
		pb  osmbr.PrimitiveBlock
	)
	br := osmbr.NewBlockReader(f)
	for br.Next() {
		if br.Type() != "OSMData" {
			continue
		}
		data, err := dec.Decompress(br.Blob())
		if err != nil {
			fmt.Println(err)
			return
		}
		if err := pb.DecodeFrom(data); err != nil {
			fmt.Println(err)
			return
		}
		gs := pb.Groups()
		for gs.Next() {
			fmt.Printf("group type=%d\n", gs.Type())
		}
		break // one OSMData block is enough for the example
	}
}

Output:
group type=2

func (*PrimitiveBlock) NumStrings 

func (pb *PrimitiveBlock) NumStrings() int

NumStrings returns the number of entries in the string table.

func (*PrimitiveBlock) String 

func (pb *PrimitiveBlock) String(i int) []byte

String returns the string table entry at index i. The returned slice is a zero-copy reference into the block data and is only valid until the next call to DecodeFrom or Decompressor.Decompress. Panics if i is out of range.

type RelationBuf 

type RelationBuf struct {
	Keys     []uint32 // string table indices for tag keys
	Vals     []uint32 // string table indices for tag values
	RolesSID []int32  // string table indices for member roles
	MemIDs   []int64  // delta-decoded absolute member IDs
	Types    []int32  // member types: MemberTypeNode, MemberTypeWay, MemberTypeRelation
}

RelationBuf is caller-managed memory for decoding Relation entities. Reuse across calls to avoid per-relation allocations. Keys, Vals, RolesSID, MemIDs, and Types are parallel arrays. MemIDs contains delta-decoded absolute member IDs.

type RelationScanner 

type RelationScanner struct {
	// contains filtered or unexported fields
}

RelationScanner iterates over Relation messages in a PrimitiveGroup. Obtain one via GroupScanner.RelationScanner. RelationScanner is a value type.

func (*RelationScanner) Err 

func (rs *RelationScanner) Err() error

Err returns the first error encountered during iteration.

func (*RelationScanner) Next 

func (rs *RelationScanner) Next(buf *RelationBuf, info *InfoBuf) (id int64, ok bool)

Next decodes the next Relation into buf and returns its ID. Resets buf slices to [:0] then appends (capacity preserved). Returns (0, false) when no more relations remain. Pass a non-nil info to also decode the Relation's Info; nil skips it.

type WayBuf 

type WayBuf struct {
	Keys []uint32 // string table indices for tag keys
	Vals []uint32 // string table indices for tag values
	Refs []int64  // delta-decoded absolute referenced node IDs
}

WayBuf is caller-managed memory for decoding Way entities. Reuse across calls to avoid per-way allocations. After DecodeWay, Keys and Vals are parallel string-table index arrays. Refs contains delta-decoded absolute node IDs.

type WayScanner 

type WayScanner struct {
	// contains filtered or unexported fields
}

WayScanner iterates over Way messages in a PrimitiveGroup. Obtain one via GroupScanner.WayScanner. WayScanner is a value type.

func (*WayScanner) Err 

func (ws *WayScanner) Err() error

Err returns the first error encountered during iteration.

func (*WayScanner) Next 

func (ws *WayScanner) Next(buf *WayBuf, info *InfoBuf) (id int64, ok bool)

Next decodes the next Way into buf and returns its ID. Resets buf slices to [:0] then appends (capacity preserved). Returns (0, false) when no more ways remain. Pass a non-nil info to also decode the Way's Info; nil skips it.

Source Files

View all Source files

Directories

Path Synopsis
examples
count command
Example count reads a PBF file and counts blocks and entity versions.
 Example count reads a PBF file and counts blocks and entity versions.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.