fdb-client

README

fdb-client

Public Go smart client for FrogoDB.

Scope

• ./pkg/client: smart client (routing, pooling, policies, batch/pipeline).
• ./pkg/queries: reusable query helpers (timeseries, previous, window, lsh).
• ./pkg/protocol, ./pkg/ripemd160: client-side wire/hash support.

Install

go get github.com/FrogoAI/fdb-client

Connect

Use one or more seed nodes. The client connects to the first available seed, discovers the cluster topology in the background, and routes keys directly to the node that owns the partition.

package main

import (
    "context"
    "log"
    "time"

    "github.com/FrogoAI/fdb-client/pkg/client"
)

func main() {
    c, err := client.New("node1:3000", "node2:3000", "node3:3000")
    if err != nil {
        log.Fatal(err)
    }
    defer c.Close()

    ctx := context.Background()

    pingCtx, cancel := context.WithTimeout(ctx, 2*time.Second)
    defer cancel()

    if err := c.Ping(pingCtx); err != nil {
        log.Fatalf("FrogoDB connection is not live: %v", err)
    }
}

For explicit connection policy:

c, err := client.NewWithConfig(client.Config{
    Seeds:             []string{"node1:3000", "node2:3000"},
    PoolSizePerNode:   64,
    ConnectionTimeout: 5 * time.Second,
    IdleTimeout:       55 * time.Second,
    TendInterval:      10 * time.Millisecond,
    MaxErrorRate:      100,
    ErrorRateWindow:   time.Second,
})

For Twelve-Factor deployments, build the same Config from environment variables and pass it to NewWithConfig:

export FDB_SEEDS=node1:3000,node2:3000
export FDB_CONNECTION_TIMEOUT=5s
export FDB_IDLE_TIMEOUT=55s
export FDB_TEND_INTERVAL=10ms
export FDB_POOL_SIZE_PER_NODE=64
export FDB_MAX_ERROR_RATE=100
export FDB_ERROR_RATE_WINDOW=1s
export FDB_MULTIPLEXING=false

cfg, err := client.GetConfigFromEnv()
if err != nil {
    log.Fatal(err)
}

c, err := client.NewWithConfig(cfg)

Unset values use the DefaultConfig defaults, with FDB_SEEDS defaulting to localhost:3000. Invalid set values return an error instead of falling back.

Topology Inspection

The client exposes the discovered topology through nameable public readers:

nodes := c.Nodes()
version := c.PartitionMapVersion()
log.Printf("partition map version=%d nodes=%v", version, nodes)

for _, n := range c.Cluster().Nodes() {
    log.Printf("node=%s active=%t", n.Address(), n.Active())
}

Records

Put writes bins, Get reads records, and Delete removes records. Write options make bin semantics, existence checks, and TTL behavior explicit.

err := c.Put(ctx, "myns", "users", "user-123", map[string]any{
    "name":  "Alice",
    "age":   int64(30),
    "score": 95.5,
}, client.WithMergeBins(), client.WithPreserveTTL())
if err != nil {
    log.Fatal(err)
}

rec, err := c.Get(ctx, "myns", "users", "user-123")
if err != nil {
    log.Fatal(err)
}
log.Printf("name=%s age=%d", rec.Bins["name"], rec.Bins["age"])

existed, err := c.Delete(ctx, "myns", "users", "user-123")
if err != nil {
    log.Fatal(err)
}
log.Printf("deleted=%t", existed)

Common write options:

• client.WithMergeBins(): update supplied bins and preserve untouched bins.
• client.WithReplaceBins(): rebuild the record from only supplied bins.
• client.WithPreserveTTL(): keep the current TTL.
• client.WithTTL(seconds): set a new TTL in seconds.
• client.WithClearTTL(): clear expiration.
• client.WithCreateOnly(): fail if the key already exists.
• client.WithReplace(): require that the key already exists.
• client.WithGeneration(n): optimistic locking.

Default read, write, and batch policies can be updated with SetReadPolicy, SetWritePolicy, and SetBatchPolicy. The setters are safe to call while other goroutines use the client; each operation uses one snapshot of the applicable default policy before applying per-call options and routing.

Query Helpers

The query packages sit on top of the client and accept small operation interfaces defined by each helper package. *client.Client satisfies all helper contracts, so the same client value can be passed directly. queries.Client remains available only as a deprecated aggregate compatibility contract for code that names it explicitly, but new tests can mock only the operations used by the helper under test.

import (
    "time"

    "github.com/FrogoAI/fdb-client/pkg/queries"
    "github.com/FrogoAI/fdb-client/pkg/queries/lsh"
    "github.com/FrogoAI/fdb-client/pkg/queries/previous"
    "github.com/FrogoAI/fdb-client/pkg/queries/timeseries"
    "github.com/FrogoAI/fdb-client/pkg/queries/window"
)

src := queries.NewMapSource(map[string]any{
    "event_id":             "evt-789",
    "standard.user_id":     "user-42",
    "standard.email":       "alice@example.com",
    "standard.merchant_id": "shop-456",
    "amount":               42.5,
    "frequency":            int64(10),
    "recency":              3.14,
    "event_time":           time.Now(),
})

Timeseries

Use timeseries queries for time-bucketed aggregations such as counts, averages, standard deviation, min/max, distinct counts, and percentiles.

tsReq := timeseries.Request{
    Name:           "transaction_stats",
    Namespace:      "scoring",
    GroupBy:        []string{"standard.user_id"},
    Range:          24 * time.Hour,
    Fields:         queries.FieldCount | queries.FieldAvg | queries.FieldSTD | queries.FieldMin | queries.FieldMax,
    Value:          "amount",
    TTL:            48 * time.Hour,
    IncludeCurrent: true,
}

tsResult, err := timeseries.Execute(ctx, c, tsReq, src)
if err != nil {
    log.Fatal(err)
}
log.Printf("count=%d avg=%.2f std=%.2f", tsResult.Count, tsResult.Average(), tsResult.STD())

Previous

Use previous queries to retrieve a value from the previous event for the same entity. Exclude can require the previous event to differ on another field.

prevReq := previous.Request{
    Name:           "prev_amount",
    Namespace:      "scoring",
    Ref:            "standard.user_id",
    Retrieve:       "amount",
    Exclude:        "standard.merchant_id",
    EventID:        src.String("event_id"),
    TTL:            24 * time.Hour,
    IncludeCurrent: false,
}

prevResult, err := previous.Execute(ctx, c, prevReq, src)
if err != nil {
    log.Fatal(err)
}
if prevResult.Found {
    log.Printf("previous amount=%v", prevResult.Value)
}

Window

Use window queries for exact sliding-window aggregation over the last N events.

winReq := window.Request{
    Name:           "last_10_amounts",
    Namespace:      "scoring",
    Ref:            "standard.user_id",
    Value:          "amount",
    WindowSize:     10,
    Fields:         queries.FieldCount | queries.FieldAvg | queries.FieldSTD | queries.FieldPercentile,
    PercentileP:    0.90,
    EventID:        src.String("event_id"),
    TTL:            24 * time.Hour,
    IncludeCurrent: true,
}

winResult, err := window.Execute(ctx, c, winReq, src)
if err != nil {
    log.Fatal(err)
}
log.Printf("count=%d avg=%.2f p90=%.2f", winResult.Count, winResult.Avg, winResult.Percentile)

LSH

Use LSH queries for near-duplicate string buckets and vector-based behavioural clustering. The server computes the LSH bucket and stores entries with the requested TTL. Embedded LSH storage IDs are internal; dedup callers only see the bucket ID, kept as lowercase hex from the first eight bytes of SHA-1 over <dedup-scope>:<input>.

dedupResult, err := lsh.Dedup(ctx, c, lsh.DedupRequest{
    Namespace: "scoring",
    Reference: "standard.email",
    TTL:       24 * time.Hour,
}, src)
if err != nil {
    log.Fatal(err)
}
log.Printf("email dedup bucket=%s", dedupResult.BucketID)

vectorResult, err := lsh.Vector(ctx, c, lsh.VectorRequest{
    Namespace:  "scoring",
    Attributes: []string{"amount", "frequency", "recency"},
    TTL:        24 * time.Hour,
}, src)
if err != nil {
    log.Fatal(err)
}
log.Printf("behavioral cluster=%s", vectorResult.BehavioralID)

For direct client-level LSH calls without the query helper source mapping:

bucketID, err := c.LSHDedup(ctx, "scoring", "email", "alice@example.com", client.WithTTL(86400))
if err != nil {
    log.Fatal(err)
}
log.Printf("email bucket=%s", bucketID)

behavioralID, err := c.LSHVector(ctx, "scoring", "behavior", []float64{42.5, 10, 3.14}, client.WithTTL(86400))
if err != nil {
    log.Fatal(err)
}
log.Printf("behavioral id=%s", behavioralID)

For lsh.Dedup, the helper sends group lsh_dedup; the server scopes that group by DedupRequest.Namespace, so the same reference value can produce different bucket IDs in different namespaces. Direct LSHDedup calls with custom groups use the group as the dedup scope.

Bloom Filter

Bloom filters are exposed through atomic Operate calls. They are useful for fast membership checks where false positives are acceptable.

_, err := c.Operate(ctx, "myns", "filters", "seen-users", []client.Operation{
    client.BloomInitOp("bloom", 10000, 0.01),
})
if err != nil {
    log.Fatal(err)
}

_, err = c.Operate(ctx, "myns", "filters", "seen-users", []client.Operation{
    client.BloomAddOp("bloom", []byte("user-42")),
})
if err != nil {
    log.Fatal(err)
}

rec, err := c.Operate(ctx, "myns", "filters", "seen-users", []client.Operation{
    client.BloomTestOp("bloom", []byte("user-42")),
})
if err != nil {
    log.Fatal(err)
}
log.Printf("probably seen=%v", rec.Bins["bloom"])

Other bloom helpers include client.BloomRemoveOp and client.BloomResetOp.

HyperLogLog

HyperLogLog estimates cardinality, for example unique users or unique devices.

_, err := c.Operate(ctx, "myns", "stats", "page-visitors", []client.Operation{
    client.HLLInitOp("visitors", 14, 6),
})
if err != nil {
    log.Fatal(err)
}

_, err = c.Operate(ctx, "myns", "stats", "page-visitors", []client.Operation{
    client.HLLAddOp("visitors", []byte("user-1"), []byte("user-2"), []byte("user-3")),
})
if err != nil {
    log.Fatal(err)
}

rec, err := c.Operate(ctx, "myns", "stats", "page-visitors", []client.Operation{
    client.HLLCountOp("visitors"),
})
if err != nil {
    log.Fatal(err)
}
log.Printf("unique visitors=%v", rec.Bins["visitors"])

Use client.HLLUnionOp, client.HLLUnionCountOp, and client.HLLIntersectCountOp to combine estimates across records.

TDigest

TDigest estimates quantiles and distribution statistics, for example p95 or p99 latency.

_, err := c.Operate(ctx, "myns", "stats", "latency", []client.Operation{
    client.TDigestAddOp("tdigest", 42.0, 1.0),
    client.TDigestAddOp("tdigest", 125.0, 1.0),
    client.TDigestAddOp("tdigest", 300.0, 1.0),
})
if err != nil {
    log.Fatal(err)
}

rec, err := c.Operate(ctx, "myns", "stats", "latency", []client.Operation{
    client.TDigestQuantileOp("tdigest", 0.99),
})
if err != nil {
    log.Fatal(err)
}
log.Printf("p99 latency=%v", rec.Bins["tdigest"])

Other TDigest helpers include client.TDigestCountOp, client.TDigestMinOp, client.TDigestMaxOp, client.TDigestCDFOp, and client.TDigestMergeOp.

More Documentation

• ./docs/client.md: full client API, query helpers, CRUD, scans, batch operations, policies, and topology.
• ./docs/protocol.md: binary wire protocol and command authoring guide.

Contributing and Security

• ./CONTRIBUTING.md: development workflow and contribution guidelines.
• ./SECURITY.md: private vulnerability reporting policy.

License

MIT. See ./LICENSE.

Contracts

• This repository must not import private server modules.
• FrogoDB server can depend on this client module.

Build

go test ./...
make lint
make test-race
make bench

Integration tests are opt-in because they need a live FrogoDB server:

FDB_INTEGRATION_SEEDS=localhost:3000 make test-integration

The default go test ./... path does not include the integration build tag.

Benchmark results are tracked with a committed benchmarks/baseline.txt. Use make bench for a local current run and make bench-compare to compare it against the baseline.