flatread

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 19, 2026 License: MIT Imports: 4 Imported by: 1

README

flatread

CI Go Reference

Read FlatBuffers buffers when you don't have the schema.

No dependencies. Every accessor is bounds-checked, so malformed input returns a zero value instead of panicking.

Why

The normal FlatBuffers workflow compiles a .fbs schema into generated accessors, and the field names live in that generated code. If you're handed a binary payload without its schema there's nothing to generate from, and the official library can't help, because it needs the types you're trying to work out.

That comes up more often than the docs suggest:

  • reverse engineering a protocol from a capture
  • inspecting a payload from a service that doesn't publish its schema
  • triaging a file that decodes on one machine but not another
  • checking what an SDK actually sends, rather than what its docs claim

flatread reads a buffer positionally. Fields are addressed by vtable slot number instead of by name. That's enough to walk an unknown buffer, and enough to write a real decoder once you've worked out which slot means what.

Install

go get github.com/Mootjelh/flatread

go install github.com/Mootjelh/flatread/cmd/flatdump@latest

The CLI

Point flatdump at a payload and it describes every populated field, guessing each one's type from its bytes:

$ flatdump payload.bin
root @0x40 (168 bytes)
  slot 4   scalar   u32=4242 u16=4242 u8=146
  slot 6   string   "hello"
  slot 8   bytes    3
  slot 10  table
    slot 4   scalar   u32=7 u16=7 u8=7
  slot 12  vector   2 tables
    [0]
      slot 4   scalar   u32=11 u16=11 u8=11
    [1]
      slot 4   scalar   u32=22 u16=22 u8=22

It reads stdin when given no file, so it drops into a pipeline:

curl -s https://example.com/payload | flatdump -depth 8

With -json the same walk comes out machine-readable, which is what you want once you are grepping across a directory of captures:

flatdump -json payload.bin | jq '.fields[] | select(.kind == "string")'

Buffers that are not plain

Two things are worth checking before you conclude a payload is empty.

Size-prefixed buffers. gRPC and any format that stores several messages back to back put a uint32 byte count in front of the buffer. flatdump detects that and says so; in code it is RootSizePrefixed, with IsSizePrefixed for the guess.

This matters because feeding such a buffer to plain Root does not fail loudly. It reads the size as the root offset, lands somewhere arbitrary, and usually reports a table with no fields at all:

$ flatdump -prefix no sized.bin
root @0xa8 (172 bytes)

No error, no fields, and identical to a buffer that genuinely has nothing in it.

File identifiers. A schema can attach four bytes of identifier directly after the root offset, and when it is there it is usually the fastest way to find out what you are holding, because it is picked to be human-readable:

$ flatdump payload.bin
file identifier "FLTR"
root @0x40 (168 bytes)
...

Nothing in the buffer records whether that field is present, so FileIdentifier returns the bytes plus whether they look like an identifier at all. A false there means "these are not plausible", not "this schema has none".

The library

root, err := flatread.Root(buf)
if err != nil {
    return err
}

for _, slot := range root.Slots() {
    fmt.Println(slot, root.Guess(slot))
}

name := root.String(6)
ids  := root.Int32Vector(8)

if pricing, ok := root.Table(10); ok {
    total := pricing.Uint32(4)
}

for i := 0; i < root.VectorLen(12); i++ {
    if offer, ok := root.TableAt(12, i); ok {
        _ = offer.String(4)
    }
}

Scalars (Byte, Bool, Int8 through Int64, Uint16 through Uint64, Float32, Float64), strings, byte vectors, numeric and string vectors, nested tables and vectors of tables. Has distinguishes an absent field from one that's present and zero.

Entry points: Root for an ordinary buffer, RootSizePrefixed for one carrying a length, TableAtOffset when you already know where a table starts.

Vectors of structs

Structs are not tables. They're fixed-size records stored inline in the vector, with no vtable and no offsets, so TableAt cannot read them: it treats each element as an offset, adds a number that was never one, and returns a table pointing at whatever sits there. No error, no panic, just plausible nonsense.

StructAt reads them, given the element size from the schema:

n := root.StructVectorLen(6, 8)          // 8-byte records
for i := 0; i < n; i++ {
    rec := root.StructAt(6, i, 8)
    x := binary.LittleEndian.Uint32(rec)
    y := binary.LittleEndian.Uint32(rec[4:])
}

StructVectorLen also catches a wrong size, because the declared count multiplied by that size stops fitting the buffer. Plain VectorLen can't: it doesn't know how wide an element is.

Unions

A union becomes two fields in consecutive slots: the type tag at n, the value at n+2.

switch kind, value, ok := root.Union(6); {
case !ok:
    // absent, or the tag is UnionNone
case kind == 1:
    _ = value.String(4)
}

for i := 0; i < root.UnionVectorLen(10); i++ {
    kind, value, ok := root.UnionAt(10, i)
    ...
}

A tag of UnionNone in a union vector is a hole, not the end. Stopping there skips everything after it.

How a buffer is laid out

Worth having to hand while reading a dump. Everything is little-endian.

buf[0:4]                 uoffset to the root table

table[-soffset]          the table's vtable. soffset is a SIGNED int32
                         stored at the table's own position, so the
                         vtable normally sits before the table
vtable[0:2]              vtable size in bytes
vtable[2:4]              inline table size in bytes
vtable[4+2i : 6+2i]      byte offset of field i inside the table,
                         0 meaning absent

Field i lives at slot number 4+2i, so the first field is slot 4, the second slot 6, the third slot 8. Slot numbers are what you pass to every accessor. Without a schema they're the only stable handle you have.

Offsets to strings, vectors and nested tables are uoffsets: unsigned, and relative to the position of the offset itself.

Limitations

flatread recovers structure, not types. Guess works from a length prefix that fits, a run of printable bytes, a known magic number, or an offset that resolves to a readable vtable. That's genuinely all the information the buffer holds.

The clearest consequence, which has a test pinning it so it doesn't get "fixed" later: a vector of uint32{1, 2, 3} and the twelve bytes that encode it are the same twelve bytes, so Guess reports bytes. Nothing in a FlatBuffers buffer records a vector's element width.

The same applies to vectors of structs, which is why they need StructAt and an element size you supply.

So use Guess to find the slots you care about, confirm each one against real data, then read them with the typed accessors. Don't build a decoder on it.

flatread also doesn't write buffers. Use the official library for that. Once you know the schema you should be generating code from it anyway.

Safety

Every accessor is bounds-checked and returns a zero value rather than panicking. This package is by definition pointed at bytes nobody has validated, so malformed input is the ordinary case here, not an exceptional one.

That property is enforced rather than claimed:

go test -fuzz=FuzzReader

The fuzz target walks every accessor over arbitrary input, including offset cycles: a buffer whose field points back at its own table, which an encoder would never emit but a corrupt or hostile file can. CI runs a 60-second pass on every push.

The trade-off is that a zero return is ambiguous. It means either "the field holds zero" or "there is no such field". Use Has when the difference matters.

Roaring bitmaps

Byte vectors carrying a RoaringBitmap serial cookie are recognised and labelled, since Roaring turns up often in search and analytics payloads. Decoding one needs a dependency this package doesn't take, so there's a hook for it:

flatread.DumpWith(buf, flatread.DumpOptions{
    Annotate: func(b []byte) string {
        if !flatread.LooksRoaring(b) {
            return ""
        }
        bm := roaring.New()
        if _, err := bm.ReadFrom(bytes.NewReader(b)); err != nil {
            return "roaring: " + err.Error()
        }
        return fmt.Sprintf("%d bits set", bm.GetCardinality())
    },
})

License

MIT

Documentation

Overview

Package flatread reads FlatBuffers buffers for which you have no schema.

The normal FlatBuffers workflow compiles a .fbs schema into generated accessors, and field names live in that generated code. When you are handed a binary payload without its schema (reverse engineering a protocol, inspecting a capture, triaging a corrupt file) there is nothing to generate from, and the standard library is of no help.

This package reads a buffer positionally instead: fields are addressed by their vtable slot number rather than by name. That is enough to walk an unknown buffer, and enough to write a decoder once you have worked out which slot means what.

Buffer layout

Everything is little-endian. Reading the code below is much easier with this to hand: 

buf[0:4]                 uoffset to the root table

table[-soffset]          the table's vtable (soffset is a SIGNED int32
                         stored at the table's own position)
vtable[0:2]              vtable size in bytes
vtable[2:4]              inline table size in bytes
vtable[4+2*i : 6+2*i]    byte offset of field i inside the table,
                         0 meaning the field is absent

So field i lives at slot number 4+2*i: the first field is slot 4, the second slot 6, and so on. Slot numbers, not indices, are what you pass here.

Offsets to strings, vectors and nested tables are uoffsets: unsigned, and relative to the position of the offset itself.

Safety

Every accessor is bounds-checked and returns a zero value rather than panicking. That is deliberate: this package exists to be pointed at bytes nobody has validated, so malformed input is the expected case and not an exceptional one. The fuzz test in this package asserts it.

The cost of that choice is that a zero value is ambiguous. It means either "the field holds zero" or "there is no such field". Use Table.Has when the difference matters.

Index

Examples

Constants

View Source 
const UnionNone byte = 0

UnionNone is the type tag a union carries when it holds nothing. FlatBuffers reserves 0 for it in every union enum.

Variables

This section is empty.

Functions

func Dump 

func Dump(buf []byte) string

Dump walks a buffer from its root and describes every populated field, guessing each one's type from its bytes.

This is the tool you reach for first when handed an unknown payload: it turns an opaque blob into a list of slot numbers with plausible types, which is what you need in order to start matching them against whatever documentation, generated code or client bundle you do have.

A malformed buffer produces a description of the error rather than a panic.

Example

Dump is the first thing to run against an unfamiliar payload: it turns an opaque blob into a list of slot numbers with plausible types.

Note slot 8, which really is a vector of uint32 {1, 2, 3} and is reported as three bytes. Nothing in a FlatBuffers buffer records a vector's element width, so that is the honest answer rather than a bug. 

fmt.Print(Dump(sample()))

Output:
root @0x40 (168 bytes)
  slot 4   scalar   u32=4242 u16=4242 u8=146
  slot 6   string   "hello"
  slot 8   bytes    3
  slot 10  table
    slot 4   scalar   u32=7 u16=7 u8=7
  slot 12  vector   2 tables
    [0]
      slot 4   scalar   u32=11 u16=11 u8=11
    [1]
      slot 4   scalar   u32=22 u16=22 u8=22

func DumpWith 

func DumpWith(buf []byte, opts DumpOptions) string

DumpWith is Dump with control over depth, breadth and annotation.

func FileIdentifier 

func FileIdentifier(buf []byte) (string, bool)

FileIdentifier reads the optional four-byte identifier a schema can attach to its buffers, which sits directly after the root offset.

The bool reports whether those four bytes look like an identifier at all, meaning printable ASCII with no spaces. Nothing in the buffer records whether the field is present: a schema without a file_identifier puts padding or the start of a vtable in the same place. So a false here means "these bytes are not a plausible identifier", not "this schema has none".

When it is present it is usually the fastest way to work out what you have been handed, because it is chosen to be human-readable and often appears in the schema, the docs or the code that produced the buffer.

func IsSizePrefixed 

func IsSizePrefixed(buf []byte) bool

IsSizePrefixed guesses whether buf carries a size prefix, by checking whether its first uint32 accounts for the rest of the buffer exactly.

Like Table.Guess this reads structure rather than a recorded fact, and it can be fooled: a plain buffer whose root offset happens to equal len(buf)-4 looks identical. It is a starting point for an unknown payload, not something to branch on in production once you know the format.

func LooksRoaring 

func LooksRoaring(b []byte) bool

LooksRoaring reports whether b starts with a RoaringBitmap serial cookie: 12346 for a plain bitmap, or 12347 in the low 16 bits when run containers are present.

This only inspects the header. To actually decode one, hand the bytes to a Roaring library. This package deliberately has no dependencies.

Types

type DumpOptions 

type DumpOptions struct {
	// MaxDepth limits how far nested tables are followed. 0 means the default
	// of 5, which is deep enough to see the shape of most payloads without
	// printing a megabyte.
	MaxDepth int

	// MaxVectorElems limits how many elements of a table vector are expanded.
	// 0 means the default of 3. The remainder is summarised as "... N more".
	MaxVectorElems int

	// Annotate, when set, is called for every byte vector and its result is
	// appended to that line. This is the hook for anything that needs a
	// dependency: counting bits in a RoaringBitmap, sniffing a compression
	// header, decoding a nested buffer. Return "" to add nothing.
	Annotate func(b []byte) string
}

DumpOptions controls DumpWith. The zero value is what Dump uses.

type Kind 

type Kind int

Kind is a guess at what a field holds.

FlatBuffers does not record field types in the buffer. The schema carries them, and a schema is exactly what you do not have here. So Table.Guess leans entirely on structure: a length prefix that fits, bytes that are all printable, a known magic number, an offset that resolves to a readable vtable.

It is right often enough to be the fastest way into an unknown buffer, and it is wrong often enough that you should never build a decoder on it. Use it to find the slots you care about, confirm each one against real data, then read them with the typed accessors. 

const (
	// KindScalar is the fallback: an inline number, or a field whose offset
	// resolves to nothing recognisable. Note that KindScalar is what an unknown
	// field degrades to, so it means "no better guess" rather than "definitely
	// a number".
	KindScalar Kind = iota

	// KindString is a length-prefixed run of printable ASCII.
	KindString

	// KindBytes is a length-prefixed run that is not printable and not
	// recognised. A vector of any fixed-width numeric type also lands here, and
	// so does a vector of structs: nothing in the buffer distinguishes a byte
	// vector from an int32 vector or from packed fixed-size records. Read those
	// with [Table.StructAt] once the schema tells you the element size.
	KindBytes

	// KindRoaring is a byte vector carrying a RoaringBitmap serial cookie.
	// Roaring is common in search and analytics payloads, compact enough to be
	// worth recognising, and its cookie is distinctive.
	KindRoaring

	// KindTable is an offset resolving to a plausible vtable.
	KindTable

	// KindVectorOfTables is a vector whose elements all resolve to plausible
	// vtables.
	KindVectorOfTables
)

func (Kind) String 

func (k Kind) String() string

String renders the Kind for display.

type Table 

type Table struct {
	// contains filtered or unexported fields
}

Table is a FlatBuffers table at a position inside a buffer. The zero Table is not usable; get one from Root.

A Table is a view, not a copy: it holds a reference to the underlying buffer and no accessor allocates unless it must (Table.String and the vector accessors do; the scalar ones do not).

Example

Once Dump has told you which slots exist, read them with the typed accessors. Slot numbers are the schema you did not get. 

root, err := Root(sample())
if err != nil {
	panic(err)
}

fmt.Println("slots: ", root.Slots())
fmt.Println("string:", root.String(6))
fmt.Println("vector:", root.Int32Vector(8))

if nested, ok := root.Table(10); ok {
	fmt.Println("nested:", nested.Uint32(4))
}
for i := 0; i < root.VectorLen(12); i++ {
	if sub, ok := root.TableAt(12, i); ok {
		fmt.Printf("elem %d: %d\n", i, sub.Uint32(4))
	}
}

Output:
slots:  [4 6 8 10 12]
string: hello
vector: [1 2 3]
nested: 7
elem 0: 11
elem 1: 22

func Root 

func Root(buf []byte) (Table, error)

Root returns the root table of a FlatBuffers buffer.

It fails only when the buffer is too small to hold a root offset, or when that offset points outside the buffer. It does NOT validate the rest of the buffer. That is what the accessors' bounds checks are for.

func RootSizePrefixed 

func RootSizePrefixed(buf []byte) (Table, error)

RootSizePrefixed returns the root table of a size-prefixed buffer.

The size-prefixed layout puts a uint32 byte count in front of an otherwise ordinary buffer, so that a reader taking bytes off a stream knows where one message ends. gRPC uses it, and so does anything that stores several buffers back to back in one file.

Passing such a buffer to Root does not fail loudly. Root reads the size as the root offset, and what happens next depends on the number: usually an error, sometimes a table full of nonsense. If you do not know which layout you have, IsSizePrefixed guesses.

Trailing bytes past the declared size are ignored, so this can be called on a stream buffer that holds more than one message.

func TableAtOffset 

func TableAtOffset(buf []byte, pos uint32) (Table, error)

TableAtOffset returns the table at an absolute byte position in buf.

Use it when you already know where a table starts, for instance when a size-prefixed buffer puts the root somewhere other than offset 0, or when you are re-entering a buffer at a position Table.Pos gave you earlier.

func (Table) Bool 

func (t Table) Bool(slot uint16) bool

Bool reads a bool field. FlatBuffers stores these as a single byte.

func (Table) Buffer 

func (t Table) Buffer() []byte

Buffer returns the underlying buffer. It is not copied.

func (Table) Byte 

func (t Table) Byte(slot uint16) uint8

Byte reads a uint8 field.

func (Table) Bytes 

func (t Table) Bytes(slot uint16) []byte

Bytes reads a vector of bytes, returned as a sub-slice of the buffer without copying. Do not retain it past the lifetime of the buffer, and do not write to it.

func (Table) BytesAt 

func (t Table) BytesAt(slot uint16, i int) []byte

BytesAt reads element i of a vector of strings or byte vectors, without copying.

func (Table) Float32 

func (t Table) Float32(slot uint16) float32

Float32 reads a float32 field.

func (Table) Float32Vector added in v0.3.0 

func (t Table) Float32Vector(slot uint16) []float32

Float32Vector reads a vector of float32.

func (Table) Float64 

func (t Table) Float64(slot uint16) float64

Float64 reads a float64 field.

func (Table) Float64Vector added in v0.3.0 

func (t Table) Float64Vector(slot uint16) []float64

Float64Vector reads a vector of float64.

func (Table) Guess 

func (t Table) Guess(slot uint16) Kind

Guess reports what a field most likely holds. See Kind for how much to trust it.

func (Table) Has 

func (t Table) Has(slot uint16) bool

Has reports whether the field is present, as distinct from present and zero.

FlatBuffers omits fields equal to their default, so a writer that stored 0 may well have written nothing at all. Has cannot recover that distinction. It reports what is in the buffer, which is the most anyone can know.

func (Table) Int8 

func (t Table) Int8(slot uint16) int8

Int8 reads an int8 field.

func (Table) Int16 

func (t Table) Int16(slot uint16) int16

Int16 reads an int16 field.

func (Table) Int32 

func (t Table) Int32(slot uint16) int32

Int32 reads an int32 field.

func (Table) Int32Vector 

func (t Table) Int32Vector(slot uint16) []int32

Int32Vector reads a vector of int32.

func (Table) Int64 

func (t Table) Int64(slot uint16) int64

Int64 reads an int64 field.

func (Table) Int64Vector 

func (t Table) Int64Vector(slot uint16) []int64

Int64Vector reads a vector of int64.

func (Table) Pos 

func (t Table) Pos() uint32

Pos is the table's absolute byte position in the buffer.

func (Table) Slots 

func (t Table) Slots() []uint16

Slots lists the vtable slots this table actually populates, ascending.

This is the entry point for exploring an unknown buffer: it tells you which fields are present without knowing what any of them mean.

func (Table) String 

func (t Table) String(slot uint16) string

String reads a string field. FlatBuffers strings are length-prefixed and NUL-terminated; the terminator is not included.

The bytes are not validated as UTF-8, because a schema-less buffer is exactly where a "string" often turns out to be something else.

func (Table) StringAt 

func (t Table) StringAt(slot uint16, i int) string

StringAt reads element i of a vector of strings.

func (Table) StringVector 

func (t Table) StringVector(slot uint16) []string

StringVector reads a vector of strings.

An element that is out of range comes back as "", the same as Table.StringAt would give, so the length of the result always matches the vector's declared length.

func (Table) StructAt added in v0.2.0 

func (t Table) StructAt(slot uint16, i, size int) []byte

StructAt returns the bytes of element i in a vector of structs, without copying.

FlatBuffers structs are not tables. They are fixed-size records stored inline in the vector, one after another, with no vtable and no offsets. That makes them unreadable through Table.TableAt, which treats every element as an offset: on a struct vector it adds a number that was never an offset to the element position and hands back a Table pointing at whatever sits there.

The element size comes from the schema. Nothing in the buffer records it, which is also why Table.Guess cannot tell a struct vector from a byte vector and reports KindBytes for both.

Fields inside the returned bytes are at fixed offsets, again from the schema, and are read with encoding/binary directly.

func (Table) StructVectorLen added in v0.2.0 

func (t Table) StructVectorLen(slot uint16, size int) int

StructVectorLen reports how many structs of the given size a vector holds, or 0 when the declared count does not fit the buffer.

This differs from Table.VectorLen, which trusts the count without knowing the element width. A truncated buffer shows up here and not there.

func (Table) Table 

func (t Table) Table(slot uint16) (Table, bool)

Table reads a nested table field. The bool reports whether it was present and in range.

func (Table) TableAt 

func (t Table) TableAt(slot uint16, i int) (Table, bool)

TableAt reads element i of a vector of tables.

func (Table) Uint16 

func (t Table) Uint16(slot uint16) uint16

Uint16 reads a uint16 field.

func (Table) Uint32 

func (t Table) Uint32(slot uint16) uint32

Uint32 reads a uint32 field.

func (Table) Uint32Vector 

func (t Table) Uint32Vector(slot uint16) []uint32

Uint32Vector reads a vector of uint32.

func (Table) Uint64 

func (t Table) Uint64(slot uint16) uint64

Uint64 reads a uint64 field.

func (Table) Union added in v0.3.0 

func (t Table) Union(typeSlot uint16) (byte, Table, bool)

Union reads a union: its type tag and the table it points at. The bool is false when the tag is UnionNone or the value is missing.

Union members are usually tables, which is what this returns. A schema can also put a string or a struct in a union; for those, read typeSlot+2 directly with Table.String or Table.StructAt, since the value slot is an ordinary slot.

func (Table) UnionAt added in v0.3.0 

func (t Table) UnionAt(typeSlot uint16, i int) (byte, Table, bool)

UnionAt reads element i of a vector of unions.

An element whose tag is UnionNone reports false. That is not an error and not the end of the vector: a union vector can hold empty slots between full ones, so keep walking to Table.UnionVectorLen.

func (Table) UnionType added in v0.3.0 

func (t Table) UnionType(typeSlot uint16) byte

UnionType reads the type tag of a union without resolving its value.

A union in a schema becomes two fields in the table, in consecutive slots: the tag at typeSlot, saying which member is present, and the value at typeSlot+2. Without the schema the tag is just a number, but it is the number the schema's enum is keyed on, so it is what you match against once you know what the members are.

func (Table) UnionVectorLen added in v0.3.0 

func (t Table) UnionVectorLen(typeSlot uint16) int

UnionVectorLen reports how many elements a vector of unions holds.

A union vector is two parallel vectors: the tags, as a vector of bytes at typeSlot, and the values at typeSlot+2. An encoder keeps them the same length. This reports the shorter of the two, so any index it accepts is in range for both, which matters on a truncated buffer where they are not.

func (Table) VectorLen 

func (t Table) VectorLen(slot uint16) int

VectorLen reports the element count of a vector field, of any element type.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.