osmbr

package module
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 16, 2026 License: MIT Imports: 6 Imported by: 0

README

osmbr

A low-level Go library for reading OpenStreetMap PBF files. Designed for minimal allocation and caller-controlled memory.

Design

  • Caller-managed buffers — Allocate buffer structs once and reuse them across blocks. After warm-up, the hot path makes zero heap allocations.
  • Scanner pattern — Standard for scanner.Next() { ... } idiom with sticky errors.
  • Raw values — Returns raw integers for coordinates and string-table indices for tags. The caller applies granularity/offset conversion.
  • No domain types — There are no Node, Way, or Relation structs. The caller reads fields from buffer structs and builds whatever representation it needs.

Install

go get github.com/invisiblefunnel/osmbr

Requires Go 1.24 or later.

Usage

f, err := os.Open("region.osm.pbf")
if err != nil {
    log.Fatal(err)
}
defer f.Close()

var (
    dec   osmbr.Decompressor
    pb    osmbr.PrimitiveBlock
    dnBuf osmbr.DenseNodesBuf
    wBuf  osmbr.WayBuf
    rBuf  osmbr.RelationBuf
)

br := osmbr.NewBlockReader(f)
for br.Next() {
    if br.Type() != "OSMData" {
        continue
    }
    data, err := dec.Decompress(br.Blob())
    if err != nil {
        log.Fatal(err)
    }
    if err := pb.DecodeFrom(data); err != nil {
        log.Fatal(err)
    }

    gs := pb.Groups()
    for gs.Next() {
        switch gs.Type() {
        case osmbr.GroupTypeDense:
            if err := gs.DecodeDenseNodes(&dnBuf, nil); err != nil {
                log.Fatal(err)
            }
            for i, id := range dnBuf.IDs {
                lat := dnBuf.Lats[i]*int64(pb.Granularity) + pb.LatOffset
                lon := dnBuf.Lons[i]*int64(pb.Granularity) + pb.LonOffset
                _ = id
                _ = lat
                _ = lon
            }

        case osmbr.GroupTypeWays:
            ws := gs.WayScanner()
            for id, ok := ws.Next(&wBuf, nil); ok; id, ok = ws.Next(&wBuf, nil) {
                _ = id       // way ID
                _ = wBuf.Refs // referenced node IDs (absolute)
            }
            if err := ws.Err(); err != nil {
                log.Fatal(err)
            }

        case osmbr.GroupTypeRelations:
            rs := gs.RelationScanner()
            for id, ok := rs.Next(&rBuf, nil); ok; id, ok = rs.Next(&rBuf, nil) {
                _ = id          // relation ID
                _ = rBuf.MemIDs // member IDs (absolute)
                _ = rBuf.Types  // member types (MemberTypeNode, MemberTypeWay, MemberTypeRelation)
            }
            if err := rs.Err(); err != nil {
                log.Fatal(err)
            }
        }
    }
    if err := gs.Err(); err != nil {
        log.Fatal(err)
    }
}
if err := br.Err(); err != nil {
    log.Fatal(err)
}

API overview

The reading pipeline flows top-down:

BlockReader

NewBlockReader(r io.Reader) reads PBF file blocks sequentially. Call Next() to advance, Type() for the block type ("OSMHeader" or "OSMData"), and Blob() for the raw Blob protobuf bytes. Use a Decompressor to decompress them.

Header

DecodeHeader(data []byte) decodes a decompressed OSMHeader block, returning a Header with the bounding box, required/optional features, writing program, source, and replication metadata.

Decompressor

Decompress(blob []byte) parses and decompresses a raw Blob message, returning the decompressed payload. Allocate one per goroutine and reuse across blocks. Uses klauspost/compress for zlib decompression with reusable decompressor state.

PrimitiveBlock

DecodeFrom(data []byte) populates the block's string table, granularity, and coordinate offsets from decompressed block data. Call Groups() to get a GroupScanner. String table entries are zero-copy slices into the data — copy any strings you need to retain past the next Decompress call.

  • String(i int) []byte — look up a string table entry by index
  • Granularity / LatOffset / LonOffset / DateGranularity — coordinate and timestamp conversion parameters
GroupScanner

Iterates over PrimitiveGroup messages within a block. Call Type() to check the group kind, then use the appropriate decoder. Check Err() after iteration finishes to distinguish error from EOF.

GroupType Decoder
GroupTypeDense gs.DecodeDenseNodes(&buf, info)
GroupTypeWays gs.WayScanner()
GroupTypeRelations gs.RelationScanner()
GroupTypeNodes gs.NodeScanner()
Buffer types
Type Fields Notes
DenseNodesBuf IDs, Lats, Lons, KeysVals Parallel arrays; delta-decoded in-place
WayBuf Keys, Vals, Refs Refs are absolute node IDs (delta-decoded)
RelationBuf Keys, Vals, RolesSID, MemIDs, Types MemIDs absolute (delta-decoded)
NodeBuf Keys, Vals Individual nodes (rare in practice)
InfoBuf Version, Timestamp, Changeset, UID, UserSID, Visible Per-entity metadata
DenseInfoBuf Versions, Timestamps, Changesets, UIDs, UserSIDs, Visibles Per-node metadata arrays

Pass nil for the info parameter to skip metadata decoding.

Coordinate conversion

The library returns raw integers. Convert to nanodegrees using the block's parameters:

lat_nanodeg := dnBuf.Lats[i]*int64(pb.Granularity) + pb.LatOffset
lon_nanodeg := dnBuf.Lons[i]*int64(pb.Granularity) + pb.LonOffset

Default granularity is 100 nanodegrees. Default offsets are 0.

To get degrees, divide by 1e9:

latDeg := float64(lat_nanodeg) / 1e9
lonDeg := float64(lon_nanodeg) / 1e9

Tag decoding

Tags are pairs of string-table indices. For ways and relations, Keys and Vals are parallel arrays:

for i := range wBuf.Keys {
    key := pb.String(int(wBuf.Keys[i]))
    val := pb.String(int(wBuf.Vals[i]))
    fmt.Printf("%s = %s\n", key, val)
}

For dense nodes, tags are packed into a flat KeysVals array with 0 delimiters between nodes:

j := 0
for i := range dnBuf.IDs {
    for j < len(dnBuf.KeysVals) && dnBuf.KeysVals[j] != 0 {
        key := pb.String(int(dnBuf.KeysVals[j]))
        val := pb.String(int(dnBuf.KeysVals[j+1]))
        j += 2
        fmt.Printf("node %d: %s = %s\n", dnBuf.IDs[i], key, val)
    }
    j++ // skip the 0 delimiter
}

Metadata

Pass an InfoBuf or DenseInfoBuf to decode version, timestamp, changeset, and user metadata. Pass nil to skip it.

For ways and relations:

var iBuf osmbr.InfoBuf
for id, ok := ws.Next(&wBuf, &iBuf); ok; id, ok = ws.Next(&wBuf, &iBuf) {
    fmt.Printf("way %d: v%d changeset=%d user=%s\n",
        id, iBuf.Version, iBuf.Changeset, pb.String(int(iBuf.UserSID)))
}

For dense nodes:

var diBuf osmbr.DenseInfoBuf
gs.DecodeDenseNodes(&dnBuf, &diBuf)
for i, id := range dnBuf.IDs {
    ts := diBuf.Timestamps[i] * int64(pb.DateGranularity) // milliseconds since epoch
    fmt.Printf("node %d: v%d ts=%d\n", id, diBuf.Versions[i], ts)
}

Non-goals

This library intentionally does not provide:

  • Domain types — No Node/Way/Relation structs. Build your own from the buffer fields.
  • Filtering — All entities in a block are decoded. Skip what you don't need in your loop.
  • Concurrency — Single-threaded. Parallelize at the block level in your own code.
  • Semantic conversion — Coordinates stay as raw integers; timestamps stay as raw values. The caller applies the conversion.

Acknowledgments

Inspired by tidwall/osmfile.

Portions of the implementation, tests, and documentation were developed with the assistance of Claude Code.

Documentation

Index

Constants

View Source 
const (
	MemberTypeNode     = int32(0)
	MemberTypeWay      = int32(1)
	MemberTypeRelation = int32(2)
)

Member type constants for Relation members.

Variables

This section is empty.

Functions

func DecodeDenseNodes 

func DecodeDenseNodes(groupData []byte, buf *DenseNodesBuf, info *DenseInfoBuf) error

DecodeDenseNodes decodes a DenseNodes PrimitiveGroup into buf. groupData is the raw bytes of a PrimitiveGroup message (from GroupScanner.groupData). Resets all slices to [:0] then appends. Delta-decodes IDs, Lats, Lons in-place. Pass a non-nil info to also decode DenseInfo metadata; nil skips it.

Types

type BlockReader 

type BlockReader struct {
	// contains filtered or unexported fields
}

BlockReader reads PBF FileBlocks from an io.Reader. Call Next to advance, then Type and Blob to access the current block. Blob returns the raw Blob protobuf message; use a Decompressor to decompress it.

BlockReader is not safe for concurrent use.

func NewBlockReader 

func NewBlockReader(r io.Reader) *BlockReader

NewBlockReader returns a BlockReader that reads PBF blocks from r.

func (*BlockReader) Blob 

func (br *BlockReader) Blob() []byte

Blob returns the raw Blob protobuf message bytes for the current block. Use a Decompressor to decompress them. Valid only until the next call to Next.

func (*BlockReader) Err 

func (br *BlockReader) Err() error

Err returns the first non-EOF error encountered.

func (*BlockReader) Next 

func (br *BlockReader) Next() bool

Next reads the next FileBlock. Returns false on EOF or error. Call Err to distinguish between them.

func (*BlockReader) Offset 

func (br *BlockReader) Offset() int64

Offset returns the byte offset where the current block starts in the underlying reader. Use with io.Seeker to re-read a specific block later.

func (*BlockReader) Type 

func (br *BlockReader) Type() string

Type returns the block type ("OSMHeader" or "OSMData").

type Decompressor 

type Decompressor struct {
	// contains filtered or unexported fields
}

Decompressor parses and decompresses raw PBF Blob messages. Allocate one per goroutine and reuse across blocks to avoid per-block allocations.

Decompressor is not safe for concurrent use.

func (*Decompressor) Decompress 

func (d *Decompressor) Decompress(blob []byte) ([]byte, error)

Decompress parses a raw Blob protobuf message and returns the decompressed payload. The returned slice is valid until the next call to Decompress.

type DenseInfoBuf 

type DenseInfoBuf struct {
	Versions   []int32
	Timestamps []int64  // delta-decoded; milliseconds since Unix epoch
	Changesets []int64  // delta-decoded
	UIDs       []int32  // delta-decoded
	UserSIDs   []uint32 // indices into the block's string table
	Visibles   []bool
}

DenseInfoBuf holds optional per-node metadata arrays decoded from a DenseInfo message. All slices are grown as needed (capacity preserved across calls). Pass a non-nil *DenseInfoBuf to DecodeDenseNodes to populate; nil skips it.

Delta-decoded fields: Timestamps, Changesets, UIDs. Non-delta fields: Versions, UserSIDs.

type DenseNodesBuf 

type DenseNodesBuf struct {
	IDs      []int64
	Lats     []int64
	Lons     []int64
	KeysVals []int32
}

DenseNodesBuf is caller-managed memory for decoding a DenseNodes group. Allocate once and reuse across blocks to avoid per-block allocations. After warm-up, all slices grow to accommodate the largest block seen and are then reused without further allocation.

IDs, Lats, and Lons contain delta-decoded absolute values. To convert Lats[i] and Lons[i] to nanodegrees: 

lat_nanodeg = Lats[i] * int64(pb.Granularity) + pb.LatOffset
lon_nanodeg = Lons[i] * int64(pb.Granularity) + pb.LonOffset

KeysVals encodes tags as a flat array of string-table indices: 

(keyIdx valIdx)* 0  per node, repeated

The 0 value delimits one node's tags from the next. Example iteration: 

j := 0
for i := range buf.IDs {
    for j < len(buf.KeysVals) && buf.KeysVals[j] != 0 {
        key := pb.String(int(buf.KeysVals[j]))
        val := pb.String(int(buf.KeysVals[j+1]))
        j += 2
    }
    j++ // skip the 0 delimiter
}

type GroupScanner 

type GroupScanner struct {
	// contains filtered or unexported fields
}

GroupScanner iterates over PrimitiveGroups within a PrimitiveBlock. Obtain one via PrimitiveBlock.Groups. GroupScanner is a value type.

func (*GroupScanner) DecodeDenseNodes 

func (gs *GroupScanner) DecodeDenseNodes(buf *DenseNodesBuf, info *DenseInfoBuf) error

DecodeDenseNodes decodes the current DenseNodes group into buf. Only valid when Type() == GroupTypeDense. Pass a non-nil info to also decode per-node metadata; nil skips it.

func (*GroupScanner) Err added in v0.1.1 

func (gs *GroupScanner) Err() error

Err returns the first error encountered during iteration.

func (*GroupScanner) Next 

func (gs *GroupScanner) Next() bool

Next advances to the next PrimitiveGroup. Returns false on EOF or error. Call Err to distinguish between them.

func (*GroupScanner) NodeScanner 

func (gs *GroupScanner) NodeScanner() NodeScanner

NodeScanner returns a NodeScanner for the current group. Only valid when Type() == GroupTypeNodes. Note: non-dense nodes are rare in practice; most OSM data uses DenseNodes.

func (*GroupScanner) RelationScanner 

func (gs *GroupScanner) RelationScanner() RelationScanner

RelationScanner returns a RelationScanner for the current group. Only valid when Type() == GroupTypeRelations.

func (*GroupScanner) Type 

func (gs *GroupScanner) Type() GroupType

Type returns the GroupType of the current group.

func (*GroupScanner) WayScanner 

func (gs *GroupScanner) WayScanner() WayScanner

WayScanner returns a WayScanner for the current group. Only valid when Type() == GroupTypeWays.

type GroupType 

type GroupType int8

GroupType identifies the kind of entities in a PrimitiveGroup. 

const (
	GroupTypeUnknown    GroupType = 0
	GroupTypeNodes      GroupType = 1
	GroupTypeDense      GroupType = 2
	GroupTypeWays       GroupType = 3
	GroupTypeRelations  GroupType = 4
	GroupTypeChangesets GroupType = 5
)
type Header struct {
	// BBox is the bounding box of the data in nanodegrees.
	// Left and Right are longitude; Top and Bottom are latitude.
	BBox HeaderBBox

	// RequiredFeatures lists features a parser must support (e.g. "OsmSchema-V0.6", "DenseNodes").
	RequiredFeatures []string

	// OptionalFeatures lists features a parser may optionally handle (e.g. "Sort.Type_then_ID").
	OptionalFeatures []string

	// WritingProgram identifies the tool that created the file (e.g. "osmium/1.16.0").
	WritingProgram string

	// Source identifies the data source.
	Source string

	// ReplicationTimestamp is seconds since Unix epoch for the replication state.
	ReplicationTimestamp int64

	// ReplicationSequenceNumber is the replication sequence number.
	ReplicationSequenceNumber int64

	// ReplicationBaseURL is the base URL for replication diff files.
	ReplicationBaseURL string
}

Header holds the decoded contents of an OSMHeader block.

func DecodeHeader 

func DecodeHeader(data []byte) (Header, error)

DecodeHeader decodes a decompressed OSMHeader block.

type HeaderBBox 

type HeaderBBox struct {
	Left, Right, Top, Bottom int64
}

HeaderBBox holds bounding box coordinates in nanodegrees.

type InfoBuf 

type InfoBuf struct {
	Version    int32
	Timestamp  int64 // milliseconds since Unix epoch
	Changeset  int64
	UID        int32
	UserSID    uint32 // index into the block's string table
	Visible    bool
	HasVisible bool // false if the visible field was absent
}

InfoBuf holds optional per-entity metadata decoded from an Info message. Pass a non-nil *InfoBuf to WayScanner.Next, RelationScanner.Next, or NodeScanner.Next to populate it; nil skips decoding entirely.

type NodeBuf 

type NodeBuf struct {
	Keys []uint32 // string table indices for tag keys
	Vals []uint32 // string table indices for tag values
}

NodeBuf is caller-managed memory for decoding individual Node entities. Non-dense nodes are rare in practice; most OSM data uses DenseNodes. Reuse across calls to avoid per-node allocations.

type NodeScanner 

type NodeScanner struct {
	// contains filtered or unexported fields
}

NodeScanner iterates over individual Node messages in a PrimitiveGroup. Obtain one via GroupScanner.NodeScanner. NodeScanner is a value type. Note: in practice, OSM planet files and extracts use DenseNodes exclusively.

func (*NodeScanner) Err 

func (ns *NodeScanner) Err() error

Err returns the first error encountered during iteration.

func (*NodeScanner) Next 

func (ns *NodeScanner) Next(buf *NodeBuf, info *InfoBuf) (id, lat, lon int64, ok bool)

Next decodes the next Node into buf and returns its ID, lat, and lon. Resets buf slices to [:0] then appends (capacity preserved). Returns (0, 0, 0, false) when no more nodes remain. lat and lon are raw sint64 values. Convert to nanodegrees: 

lat_nanodeg = lat * int64(pb.Granularity) + pb.LatOffset

Pass a non-nil info to also decode the Node's Info; nil skips it.

type PrimitiveBlock 

type PrimitiveBlock struct {
	// Granularity is the coordinate granularity in nanodegrees (default 100).
	// To convert a raw lat/lon integer to nanodegrees:
	//
	//   lat_nanodeg = Lats[i] * int64(Granularity) + LatOffset
	//   lon_nanodeg = Lons[i] * int64(Granularity) + LonOffset
	Granularity int32
	// LatOffset is the latitude offset in nanodegrees (default 0).
	LatOffset int64
	// LonOffset is the longitude offset in nanodegrees (default 0).
	LonOffset int64
	// DateGranularity is the timestamp granularity in milliseconds (default 1000).
	DateGranularity int32
	// contains filtered or unexported fields
}

PrimitiveBlock holds the decoded metadata and string table for an OSMData block. Call DecodeFrom to populate from a Decompressor's output. Call Groups to iterate groups.

String table entries are zero-copy slices into the data passed to DecodeFrom. They are only valid until the next call to DecodeFrom on this block or to Decompressor.Decompress on the underlying decompressor (which reuses its buffer).

func (*PrimitiveBlock) DecodeFrom 

func (pb *PrimitiveBlock) DecodeFrom(data []byte) error

DecodeFrom populates the PrimitiveBlock from decompressed OSMData block bytes (typically the result of Decompressor.Decompress).

String table entries reference data's memory. Copy entries you need to retain past the next Decompressor.Decompress or DecodeFrom call.

func (*PrimitiveBlock) Groups 

func (pb *PrimitiveBlock) Groups() GroupScanner

Groups returns a GroupScanner for iterating over the PrimitiveGroups in this block. The scanner re-reads from the original block data.

func (*PrimitiveBlock) NumStrings 

func (pb *PrimitiveBlock) NumStrings() int

NumStrings returns the number of entries in the string table.

func (*PrimitiveBlock) String 

func (pb *PrimitiveBlock) String(i int) []byte

String returns the string table entry at index i. The returned slice is a zero-copy reference into the block data and is only valid until the next call to DecodeFrom or Decompressor.Decompress. Panics if i is out of range.

type RelationBuf 

type RelationBuf struct {
	Keys     []uint32 // string table indices for tag keys
	Vals     []uint32 // string table indices for tag values
	RolesSID []int32  // string table indices for member roles
	MemIDs   []int64  // delta-decoded absolute member IDs
	Types    []int32  // member types: MemberTypeNode, MemberTypeWay, MemberTypeRelation
}

RelationBuf is caller-managed memory for decoding Relation entities. Reuse across calls to avoid per-relation allocations. Keys, Vals, RolesSID, MemIDs, and Types are parallel arrays. MemIDs contains delta-decoded absolute member IDs.

type RelationScanner 

type RelationScanner struct {
	// contains filtered or unexported fields
}

RelationScanner iterates over Relation messages in a PrimitiveGroup. Obtain one via GroupScanner.RelationScanner. RelationScanner is a value type.

func (*RelationScanner) Err 

func (rs *RelationScanner) Err() error

Err returns the first error encountered during iteration.

func (*RelationScanner) Next 

func (rs *RelationScanner) Next(buf *RelationBuf, info *InfoBuf) (id int64, ok bool)

Next decodes the next Relation into buf and returns its ID. Resets buf slices to [:0] then appends (capacity preserved). Returns (0, false) when no more relations remain. Pass a non-nil info to also decode the Relation's Info; nil skips it.

type WayBuf 

type WayBuf struct {
	Keys []uint32 // string table indices for tag keys
	Vals []uint32 // string table indices for tag values
	Refs []int64  // delta-decoded absolute referenced node IDs
}

WayBuf is caller-managed memory for decoding Way entities. Reuse across calls to avoid per-way allocations. After DecodeWay, Keys and Vals are parallel string-table index arrays. Refs contains delta-decoded absolute node IDs.

type WayScanner 

type WayScanner struct {
	// contains filtered or unexported fields
}

WayScanner iterates over Way messages in a PrimitiveGroup. Obtain one via GroupScanner.WayScanner. WayScanner is a value type.

func (*WayScanner) Err 

func (ws *WayScanner) Err() error

Err returns the first error encountered during iteration.

func (*WayScanner) Next 

func (ws *WayScanner) Next(buf *WayBuf, info *InfoBuf) (id int64, ok bool)

Next decodes the next Way into buf and returns its ID. Resets buf slices to [:0] then appends (capacity preserved). Returns (0, false) when no more ways remain. Pass a non-nil info to also decode the Way's Info; nil skips it.

Source Files

View all Source files

Directories

Path Synopsis
examples
count command
Example count reads a PBF file and counts blocks and entity versions.
 Example count reads a PBF file and counts blocks and entity versions.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.