little_bigtable

README

Little Bigtable

A local emulator for Cloud Bigtable with persistence to a SQLite or PostgreSQL backend.

The Cloud SDK provided cbtemulator is in-memory and does not support persistence which limits it's applicability. This project is a fork of cbtemulator from google-cloud-go/bigtable/bttest

	cbtemulator	"little" Bigtable	Bigtable
Storage	In-Memory	sqlite3 or postgres	Distributed GFS
Type	Emulator	Emulator	Managed Production Datastore
Scaling	Single process	Single process	Scalable multi-node backend
GC	async GC	per-row GC at read time

Features

Data Plane (gRPC)

• ReadRows — full support with row keys, row ranges, reversed scans, row limits
• MutateRow / MutateRows — atomic row mutations (SetCell, DeleteFromColumn, DeleteFromFamily, DeleteFromRow)
• CheckAndMutateRow — conditional mutations with predicate filters
• ReadModifyWriteRow — atomic append and increment operations
• SampleRowKeys — row key sampling for map-reduce partitioning
• PingAndWarm — returns Unimplemented (safe, no crash)

Admin (gRPC)

• Instance CRUD — Create, Get, List, Update, Delete instances
• Table CRUD — Create, List, Get, Delete tables
• Column Families — ModifyColumnFamilies (add/drop)
• Row Ranges — DropRowRange (by prefix or delete all)
• Consistency — GenerateConsistencyToken, CheckConsistency
• Materialized Views (CMV) — Create, Get, List, Update, Delete with write-time sync and delete propagation

Filters

Supported row filters: Chain, Interleave, Condition, Sink, PassAll, BlockAll, RowKeyRegex, RowSample, FamilyNameRegex, ColumnQualifierRegex, ColumnRange, TimestampRange, ValueRange, ValueRegex, CellsPerColumnLimit, CellsPerRowLimit, CellsPerRowOffset, StripValueTransformer, ApplyLabelTransformer.

Persistence

All state is persisted (to SQLite or PostgreSQL, per -database-driver) and survives emulator restarts:

• instances_t / clusters_t / app_profiles_t — instance, cluster, and app profile metadata
• tables_t — table definitions and column families
• rows_t — row keys and cell data
• materialized_views_t — CMV registrations
• authorized_views_t / logical_views_t / backups_t — authorized views, logical views, and backups
• change_log_t — change stream mutation log

Forward Compatibility

New gRPC methods added by Google to the Bigtable proto (ExecuteQuery, OpenTable, ReadChangeStream, etc.) safely return codes.Unimplemented without crashing.

Usage

Usage of ./little_bigtable:
  -database-driver string
      database/sql driver name: postgres or sqlite3 (default "postgres")
  -database-url string
      database/sql connection string
  -db-file string
      legacy sqlite3 data file path (default "little_bigtable.db")
  -host string
      the address to bind to on the local machine (default "localhost")
  -port int
      the port number to bind to on the local machine (default 9000)
  -strict-admin
      require instances to exist before table/data APIs are used (default true)
  -version
      show version

With -database-driver sqlite3, -db-file is used to build the connection string automatically. With -database-driver postgres (the default), pass a connection string via -database-url.

In the environment for your application, set the BIGTABLE_EMULATOR_HOST environment variable to the host and port where little_bigtable is running. This environment variable is automatically detected by the Bigtable SDK or the cbt CLI. For example:

export BIGTABLE_EMULATOR_HOST="127.0.0.1:9000"
./run_my_app

Running with Docker (Persistent Storage)

little_bigtable stores all metadata, tables, column families, and row data in a single SQLite database file. When running in a container, mount a volume for the database path so data persists across container restarts, updates, and recreations.

Note: Always pass -host 0.0.0.0 inside a container so the gRPC server binds to all interfaces rather than container loopback (localhost/127.0.0.1).

Using docker run

Mount a named volume to /data and set -db-file accordingly:

# Create a named volume
docker volume create little_bigtable_data

# Run the container with persistent storage
docker run -d \
  --name little_bigtable \
  -p 9000:9000 \
  -v little_bigtable_data:/data \
  <image-name> \
  -host 0.0.0.0 \
  -port 9000 \
  -db-file /data/little_bigtable.db

Or using a host directory bind mount:

mkdir -p ./data
docker run -d \
  --name little_bigtable \
  -p 9000:9000 \
  -v "$(pwd)/data:/data" \
  <image-name> \
  -host 0.0.0.0 \
  -port 9000 \
  -db-file /data/little_bigtable.db

Using docker-compose.yml

services:
  little_bigtable:
    image: <image-name>
    container_name: little_bigtable
    ports:
      - "9000:9000"
    volumes:
      - little_bigtable_data:/data
    command:
      - "-host"
      - "0.0.0.0"
      - "-port"
      - "9000"
      - "-db-file"
      - "/data/little_bigtable.db"
    restart: unless-stopped

volumes:
  little_bigtable_data:

Using with cbt CLI

cbt -project my-project createinstance my-instance "My Instance" my-cluster
cbt -project my-project -instance my-instance createtable my-table families=cf1
cbt -project my-project -instance my-instance set my-table row1 cf1:col1=value1
cbt -project my-project -instance my-instance read my-table

REST API (localcloud console)

When running under localcloud, row-level operations are available via REST:

# Browse rows
curl http://localhost:8080/bigtable/admin/v2/projects/my-project/instances/my-instance/tables/my-table/rows?limit=50

# Write cells
curl -X POST http://localhost:8080/bigtable/admin/v2/projects/my-project/instances/my-instance/tables/my-table/rows \
  -H "Content-Type: application/json" \
  -d '{"rowKey": "row1", "cells": {"cf1:col1": "value1"}}'

# Delete a row
curl -X DELETE http://localhost:8080/bigtable/admin/v2/projects/my-project/instances/my-instance/tables/my-table/rows/row1

Syncing with Upstream

To fetch updates from the upstream repo (bitly/little_bigtable) and merge them into your branch:

# 1. Fetch all updates from upstream
git fetch upstream

# 2. Merge upstream/master into the current branch
git merge upstream/master --no-edit

# 3. Push the merge commit to origin
git push origin <your-branch>

Limitations

• Non-production features (snapshots, restores, GoogleSQL queries) return codes.Unimplemented.
• GoogleSQL queries (ExecuteQuery/PrepareQuery) are not supported.
• Session protocol (OpenTable/OpenAuthorizedView/OpenMaterializedView) is not implemented — not needed for correctness with standard SDK usage.
• Single-node emulator by design; clusters, multi-region, replication not supported.
• Some GC rule types (Intersection) are not fully supported.
• CMV shadow tables do not auto-update when source table column families change after CMV creation.
• Some filters are not implemented or have partial support. See cbtemulator docs

Documentation

Overview

bigtable-emulator-extended launches a PostgreSQL-backed Bigtable emulator on the given address.