fdbgo/

README

Pure Go FoundationDB Client

A native Go FDB client that speaks the FDB wire protocol directly — no cgo, no libfdb_c. Connects to FDB clusters over TCP, handles the full transaction lifecycle, and is wire-compatible with FDB 7.3.x.

Why

The official Go binding (github.com/apple/foundationdb/bindings/go) wraps libfdb_c via cgo. This creates deployment friction (must ship the C library), cross-compilation pain, and CGo overhead. This package eliminates all of that.

Status

Feature-complete. Full Apple C binding API parity — zero stubs remaining (except RebootWorker, admin-only). Beats CGo on reads.

Working and tested against real FDB 7.3.77:

• Get, GetKey, GetRange (multi-shard, all streaming modes), GetEstimatedRangeSizeBytes, GetRangeSplitPoints
• Set, Clear, ClearRange, all 14 atomic mutation types
• Transact with automatic retry (all retryable FDB error codes including tag_throttled, cluster_version_changed)
• MVCC conflict detection, OnError, exponential backoff with configurable MaxRetryDelay
• Coordinator bootstrap, GRV batching + caching (100ms TTL), storage server location discovery
• Snapshot reads, Watch (long-poll), Versionstamp, Tenants (CRUD via system keys)
• Transaction options: RYW disable, snapshot RYW disable, size limit, timeout, retry limit, lock-aware
• GetPipelined for true request pipelining (no goroutine per Get)
• TLS support (mutual auth + CA cert), QueueModel load balancing (C++ Smoother + server penalty), connection keep-warm
• Read-your-writes cache with full atomic op merging (all 14 types mirror C++ Atomic.h)
• LocalityGetAddressesForKey, LocalityGetBoundaryKeys, OpenWithConnectionString, GetClientStatus

Architecture

pkg/fdbgo/
├── client/          Transaction lifecycle, retry logic, read/commit paths
├── transport/       TCP framing, ConnectPacket handshake, connection multiplexing
├── wire/            FDB FlatBuffers framework: Writer, Reader, VTable computation
│   └── types/       One file per FDB message type + vtables_generated.go
└── cmd/             (at repo root: cmd/fdb-schema-extract/)

wire/ — Serialization framework

FDB uses a custom FlatBuffers-like binary format (NOT Google FlatBuffers). Each serialized message has:

• A vtable — array of uint16 field offsets, determines where each field lives in the object body
• An object body — fixed-size struct with fields at vtable-specified offsets
• Out-of-line data — variable-length fields (strings, vectors, nested structs) referenced via RelativeOffsets

The wire package implements this format:

• Writer / ObjectWriter — serialize messages. Handles vtable emission, soffset computation, RelativeOffset patching, nested struct allocation, OOL data.
• Reader — deserialize messages. Navigates FakeRoot wrapper, reads fields by vtable slot index. Self-describing: reads the vtable from the wire data, so forward/backward compatible.
• VTable — the type alias []uint16. Generated constants in vtables_generated.go.
• ReadErrorOr — unwraps FDB's ErrorOr<T> union responses.
• MarshalStructBlob / PackVectorOfStructBlobs — serialize vector elements for VectorRef<serialize_member>.
• WriteRootObject — for union_like_traits types (ErrorOr) where the root object IS the union (no FakeRoot wrapper).

wire/types/ — Message types

Each FDB message type (request or reply) has a Go file with:

• A struct with typed fields
• MarshalFDB() for requests — constructs the full wire message using vtable-derived offsets
• UnmarshalFDB() / UnmarshalFrom() for replies — reads fields via wire.Reader

All field offsets are derived from vtable constants in vtables_generated.go. No hardcoded byte offsets. If FDB changes a struct layout, regenerating the vtables is sufficient.

Shared write helpers (WriteReplyPromise, WriteTenantInfo, writeKeySelectorRef) encapsulate common nested struct patterns.

vtables_generated.go — The bridge between C++ and Go

This is the single source of truth for wire layout. Generated by cmd/fdb-schema-extract/, a C++ binary that compiles against real FDB headers and extracts:

• VTable constants — field byte offsets, computed by C++ template metaprogramming (get_vtable<Fields...>())
• VTable closures — all vtables transitively reachable from a message type
• File identifiers — per-message type hashes
• Slot constants — Reader slot indices with field names (e.g., ClientDBInfoSlotGrvProxies = 0)

28 types extracted. Slot indices computed mechanically from C++ traits (union_like = 2 slots, everything else = 1). Field names parsed from C++ source via name_capture.cpp.

transport/ — TCP layer

• Conn — multiplexed FDB connection. Multiple concurrent requests share one TCP connection, matched by endpoint tokens (128-bit UIDs).
• ConnectPacket — FDB handshake (protocol version negotiation, IPv4/IPv6).
• Frame format: [packetLen(4)][xxh3Checksum(8)][destToken(16)][body].
• PING keepalive: responds to server PINGs, sends outbound PINGs via connectionMonitor (matches C++ FlowTransport). Detects dead connections in ~3.5s.

client/ — Transaction lifecycle

• Database — connection manager. Transact() with automatic retry.
• Transaction — buffered mutations, read conflict tracking, GRV, commit.
• GRVBatcher — batches concurrent GetReadVersion calls into single RPC.
• LocationCache — maps keys to storage server addresses via GetKeyServerLocationsRequest.
• Connection pool with port-match reuse (handles Docker address translation).

FDB version compatibility

This client targets FDB 7.3.x. The wire format is defined by C++ code, not an IDL. Compatibility with other versions requires:

1. Matching the protocol version in the ConnectPacket handshake
2. Matching the vtable layouts for all message types we send
3. Handling any new fields in responses (the Reader is forward-compatible — unknown vtable entries are ignored)

Upgrading to a new FDB version

# 1. Checkout the FDB source at the target version
cd /path/to/foundationdb
git checkout 7.4.0  # or whatever

# 2. Rebuild the C++ vtable extractor
cd /path/to/this-repo
bash cmd/fdb-schema-extract/build.sh /path/to/foundationdb pkg/fdbgo/wire/types/vtables_generated.go

# 3. Diff the generated output
git diff pkg/fdbgo/wire/types/vtables_generated.go

# 4. For each changed type:
#    - If only vtable VALUES changed (field offsets shifted): no Go code changes needed,
#      the MarshalFDB methods use int(vt[N]) which adapts automatically.
#    - If SLOT COUNT changed (field added/removed): update the MarshalFDB method to
#      write/skip the new field. Update the slot constant references.
#    - If a NEW message type was added: create a new Go file in wire/types/.

# 5. Update the protocol version constant in transport/
#    (ProtocolVersion73 → ProtocolVersion74)

# 6. Run tests against a testcontainer at the new version

The key insight: vtable constants are the only C++ build-time artifact. Everything else is Go code that references these constants. The C++ extractor bridges C++'s compile-time template metaprogramming (which Go can't do) into static Go constants.

What changes between FDB versions

What	Frequency	Impact
Field offsets within a type	Rare (layout algorithm is stable)	Automatic — vtable constants absorb it
New fields added to existing types	Occasional	Add the field to MarshalFDB, or leave absent (zero = Optional not present)
New message types	Rare	New Go file in wire/types/
Protocol version number	Every major version	Update ProtocolVersion constant
Endpoint indices (method ordering in interfaces)	Very rare	Update Endpoint* constants in transaction.go
Serialization logic changes (conditional branches)	Very rare	Update MarshalFDB method logic

What does NOT change

• The FlatBuffers wire format itself (vtable + soffset + RelativeOffset)
• The FakeRoot wrapping pattern
• The ErrorOr union flattening
• UID layout (always 16 bytes)
• ReplyPromise structure
• TCP framing and checksum format

Testing

# All tests (uses Bazel, includes testcontainer tests against real FDB)
just test

# Specific test
bazelisk test //pkg/fdbgo/client:client_test --test_arg="-test.run=TestSetGet" \
  --test_arg="-test.v" --test_output=streamed --strategy=TestRunner=local

Client tests run against real FDB 7.3.77 via testcontainers-go (Docker required). 78 C binding port tests (96% of C test suite) + 30 correctness tests + 6 fault injection tests + 15 interop tests (Go↔CGo) + benchmarks. Line coverage: 72.4% (client), 78.8% (record layer). Binding stress: 200+ seeds × 1000 ops validated (0 failures). just coverage generates HTML report with per-package coverage.

Benchmarks

# Pure Go vs CGo (libfdb_c) — full Get path comparison
bazelisk run //pkg/fdbgo/client:client_test -- \
  -test.run='^$' \
  -test.bench='BenchmarkGetValue' \
  -test.benchtime=10s \
  -test.benchmem \
  -test.count=3

Both benchmarks read the same 100-byte key from FDB testcontainers. Measures the full path: GRV + locate + read + parse.

Baseline (Ryzen 9 3900X, FDB 7.3.77 testcontainer, 2026-04-12):

Benchmark	ns/op	B/op	allocs/op
BenchmarkGet/Go/100B	58,000	1,785	18
BenchmarkGet/CGo/100B	205,000	~400	14
BenchmarkSet/Go/100B	1,005,000	1,919	20
BenchmarkSet/CGo/100B	1,007,000	180	9

Reads: Go beats CGo by 3.5x. Writes: parity (1.00x). The 18 allocs/op on Get is the structural floor — top allocators: ReadFrame payload (9%), PrepareReply channel (13% with pool miss), conflict range buffer (6%), transaction struct (5%). All already pooled or minimal. Frame buffer pooling investigated (dayshift-6c) — no improvement because body shares backing array with payload, pooling requires an extra copy.

Fault injection

The client supports a custom DialFunc for testing failure scenarios against real FDB. Same pattern as http.Transport.DialContext — no mocks, no artificial interfaces.

// faultConn wraps net.Conn to inject failures at the TCP level.
type faultConn struct {
    net.Conn
    killReads atomic.Bool
}

func (f *faultConn) Read(b []byte) (int, error) {
    if f.killReads.Load() {
        return 0, io.EOF  // simulate network failure
    }
    return f.Conn.Read(b)
}

// Inject the custom dialer before connecting.
cluster := client.NewClusterFromConfig(cf)
cluster.SetDialFunc(func(ctx context.Context, network, addr string) (net.Conn, error) {
    conn, err := net.DialTimeout(network, addr, 5*time.Second)
    if err != nil {
        return nil, err
    }
    fc := &faultConn{Conn: conn}
    // Store fc somewhere so the test can arm/disarm it later
    return fc, nil
})
cluster.Connect(ctx)

// Later, arm the fault:
fc.killReads.Store(true)   // next Read → EOF → connection dies
// ... commit happens, server processes it, but reply is lost ...

// Disarm and reconnect:
fc.killReads.Store(false)

What you can test

Scenario	How
commit_unknown_result (1021)	Kill reads after commit frame sent — server commits but client sees EOF
Network partition	Kill reads indefinitely — all RPCs timeout
Slow network	Add time.Sleep in Read — triggers context deadline
Connection reset	Return net.ErrClosed from Read/Write

How self-conflicting works (commit_unknown_result)

When OnError receives error 1021, the transaction MAY have committed on the server. To prevent double-apply on retry:

1. OnError(1021) copies all write conflict ranges into the read conflict set
2. Transaction is reset, Transact() retries the user function
3. On the retry's commit, the commit proxy checks: "did any read conflict ranges change since read version?"
4. Since the ORIGINAL commit wrote to those ranges, the check fails → not_committed (1020)
5. The retry does NOT apply mutations — no double-apply

This achieves the same safety as C++ NativeAPI::makeSelfConflicting(). Additionally, commitDummyTransaction runs a synchronization barrier before returning commit_unknown_result — a separate transaction that conflicts with the original, confirming it's no longer in-flight at the commit proxy. Both mechanisms combined match C++ exactly. Verified by TestCommitUnknownResult_NoDoubleApply: atomic ADD 5 to a counter, kill the reply, verify counter=15 (not 20).

Known divergences from C++ (audited 2026-04-12)

Systematic audit against foundationdb/fdbclient/NativeAPI.actor.cpp, ReadYourWritesTransaction.actor.cpp, and Atomic.h. All correctness bugs were fixed; these are intentional or architectural differences:

Area	C++ behavior	Go behavior	Impact
Self-conflicting (1021)	commitDummyTransaction + makeSelfConflicting random range at \xFF/SC/	commitDummyTransaction (sync barrier) + copy write→read conflicts in OnError	Matching C++: dummy confirms original is out of system, self-conflicting prevents double-apply
Auto-reset after commit	No auto-reset at API >= 410	postCommitReset() clears state for reuse	Design choice: Go API expects tx reuse after commit
onProxiesChanged	Wakes commit/GRV/location on proxy topology change	proxiesChanged broadcast wakes commit reply + GRV/location backoff loops	Matching C++: immediate wake-up on proxy failover
FLAG_FIRST_IN_BATCH	Commit flag for priority ordering	Not exposed	Missing API surface, no behavioral gap
getRange RYW merge	Segment-tree RYWIterator with demand-fetch + SnapshotCache	Iterative fetch+merge with SnapshotCache (sorted interval map)	Matching C++: server reads cached and reused within a transaction. SnapshotCache + iterative merge loop.
getKey boundary short-circuit	Returns "" or \xFF\xFF without network	Same (implemented dayshift-6b)	Matching C++
tag_throttled custom delay	Uses cx->throttledTags + TAG_THROTTLE_RECHECK_INTERVAL	tagThrottles.maxDuration with same capping	Matching C++: max(backoff, min(7s, tagDuration))
proxy_tag_throttled accumulated delay	Tracks proxyTagThrottledDuration, sends back to proxy	Tracks duration but not yet sent back to proxy in GRV request	Rate feedback incomplete (LOW); throttle still works via standard backoff
QueueModel key	endpoint.token.first() (uint64)	Address string (host:port)	Cosmetic; same server identity in practice
Load balance secondDelay	Speculative second request after delay to hedge slow servers	sendFrameWithHedge() — race best + second-best server with max(10ms, 2x latency) delay	Matching C++: all 3 read paths (getValue, getKey, getRange) hedge

Adding a new request/response type

1. Add the C++ type to cmd/fdb-schema-extract/main.cpp (include header, add extractType<T>() call)
2. If the type has field names in its serialize() method, add the source file + type name to name_capture.cpp
3. Rebuild: bash cmd/fdb-schema-extract/build.sh /path/to/fdb pkg/fdbgo/wire/types/vtables_generated.go
4. Create wire/types/my_type.go with struct + MarshalFDB() / UnmarshalFrom() using the generated vtable and slot constants
5. Wire it into the client code
6. Test against real FDB