go-warp

README

Warp

Warp is a data orchestration and synchronization layer for distributed Go backends.

It sits between your application and your primary storage (Database, API), providing:

• L1 Caching (In-Memory)
• Distributed Invalidation (via NATS, Kafka, etc.)
• Consistency Modes (Strong Local, Eventual Distributed, Strong Distributed)
• Distributed Locking & Leases
• Conflict Resolution (Merge Engine)

Features

• Request Coalescing: Built-in singleflight protection against Thundering Herd.
• Adaptive Batching: Automatically groups and compresses invalidation events under high load.
• Zero-Copy Serialization: Optional ByteCodec for maximum throughput.
• Polyglot Sidecar: Connect standard Redis clients (Node.js, Python, etc.) via warp-proxy.

  Note: The Proxy is optimized for Consistency and Interoperability, providing access to Warp's unified state from any language. For raw millisecond-level throughput in Go, use the Core library directly.

Why Warp?

Most caching libraries are just... caches. You Set a key, and it sits there until it expires. In a distributed system, this leads to stale data.

Warp is different. It treats data as a living entity that needs to be synchronized across your cluster.

Benchmarks

Results from cmd/bench/ suite running on a standard developer machine (Docker/Podman):

System	Mode	Ops/sec	Avg Latency	P99 Latency
Warp	StrongLocal	~2,700,000	370 ns	0.6 ms
Warp	Eventual (Mesh)	~3,000,000	330 ns	0.6 ms
Warp	Eventual (L2=Redis)	~2,400,000	400 ns	0.8 ms
Ristretto	Local Cache	~12,000,000	80 ns	0.004 ms
Redis	Remote	~55,000	18 µs	1.6 ms
DragonFly	Remote	~118,000	8.5 µs	1.1 ms
Warp Proxy	Sidecar (Node.js)	~170,000	-	Pipelined (Batch=50)
Warp Proxy	Sidecar (Python)	~80,000	-	Pipelined (Batch=50)

Takeaway: Warp allows you to access data ~10-20x faster than standard Redis by serving reads from memory (L1) while maintaining eventual consistency across the cluster.

• Need speed? Use ModeStrongLocal.
• Need consistency? Use ModeEventualDistributed and Warp will automatically notify other nodes when you update data.
• Need coordination? Use ModeStrongDistributed to ensure writes are acknowledged by a quorum of nodes.
• Need to group data? Use Leases to invalidate a user's entire session cache with one call.

Documentation

• Getting Started: Build a User Session Store in 5 minutes.
• Presets: Production-ready configurations (NewRedisEventual, etc.).
• Core Concepts: Understand Consistency Modes and Architecture.
• Best Practices: Recommended patterns for common scenarios.
• Modules:
  • Cache: Adaptive TTL, Pluggable Backends.
  • Sync Bus: NATS, Kafka, Warp Mesh (UDP/Gossip), Custom Adapters.
  • Lock: Distributed Locking.
  • Leases: Group Invalidation.
  • Merge: Conflict Resolution.
  • Validator: Background Consistency Checks.
  • Federation: Multi-region synchronization.
  • Sidecar: Polyglot proxy (RESP protocol).
• Glossary: Terminology definition.

Client Libraries

• Warp Store (JS): A high-performance, dual-layer (Memory + IndexedDB) caching store for frontend applications. Supports offline mode, session restoration, and is designed to work seamlessly with Warp's backend concepts.
• Warp Sidecar: A polyglot proxy that allows you to access Warp's unified state from any language.

Installation

go get github.com/mirkobrombin/go-warp/v1

Quick Example

package main

import (
    "context"
    "fmt"
    "time"
    "github.com/mirkobrombin/go-warp/v1/core"
    "github.com/mirkobrombin/go-warp/v1/presets"
)

type User struct {
    Name string
}

func main() {
    // 1. Setup with Preset (Production-ready)
    // Automatically wires In-Memory L1 + Redis L2 + Redis Bus
    w := presets.NewRedisEventual[User](presets.RedisOptions{
        Addr: "localhost:6379",
    })

    // 2. Register Key
    w.Register("user:*", core.ModeEventualDistributed, 10*time.Minute)

    // 3. Use
    ctx := context.Background()
    w.Set(ctx, "user:123", User{Name: "Alice"})
    
    val, _ := w.Get(ctx, "user:123")
    fmt.Println(val.Name)
}

Warp Mesh (Zero-Infra)

Warp Mesh allows you to cluster your Go backends without Redis, NATS, or Kafka. It uses UDP Multicast for local discovery and a Unicast Gossip protocol for cloud environments.

w := presets.NewMeshEventual[User](mesh.MeshOptions{
    Port: 7946,
    Peers: []string{"10.0.0.1:7946"}, // Static seeds for cloud
})

License

MIT License. See LICENSE for details.

IA

I am using Gemini, to help me write the documentation and debug issues, since the cocebase is really large. I want to make sure I am not missing anything important as this library aims to be used in critical production systems.

I am just honest and transparent. Peace and love.