README
¶
coldkeep (POC)
Status: Research prototype / proof‑of‑concept.
Not production‑ready. Do not use for real or sensitive data.
coldkeep is an experimental local-first content‑addressed file storage
prototype written in Go.
It stores files as chunks inside container files on disk, and
tracks metadata in Postgres.
This repo is meant for learning, experimentation, and discussion --- not for operational backup/storage use.
What it does (today)
- Store a file (or folder) by splitting it into chunks.
- Deduplicate chunks using SHA‑256.
- Pack chunks into container files up to a max size.
- Restore a file by reconstructing it from chunks.
- Remove a logical file (decrements chunk ref counts).
- Run GC to delete unreferenced chunks/containers.
- Basic stats.
Design sketch
- logical_file: user-facing file entry (name, size, file_hash)
- chunk: content-addressed chunk (hash, size, ref_count, container_id, offset)
- file_chunk: mapping from logical_file → ordered chunk list
- container: physical file on disk that stores chunk data
Containers are plain files under:
storage/containers/
Quickstart (Using the included samples/ folder)
This repository includes a small samples/ directory for testing.
🐳 With Docker
Start services:
docker compose up -d --build
Store a sample file:
docker compose run --rm -v "$PWD/samples:/samples" app store /samples/hello.txt
Store the sample folder:
docker compose run --rm -v "$PWD/samples:/samples" app store-folder /samples
List stored files:
docker compose run --rm app list
Restore a file (replace ID as needed):
docker compose run --rm app restore 1 _out.bin
Show stats:
docker compose run --rm app stats
Run garbage collection:
docker compose run --rm app gc
💻 Local (without Docker)
Start Postgres (example via Docker):
docker compose up -d postgres
Build:
cd app
go build -o ../coldkeep .
cd ..
Store the sample folder:
./coldkeep store-folder samples
List files:
./coldkeep list
Restore a file:
./coldkeep restore 1 ./restored.bin
Show stats:
./coldkeep stats
Run GC:
./coldkeep gc
Configuration
The app reads DB connection info from environment variables
(see docker-compose.yml for defaults).
Storage goes to:
./storage/
You can safely delete the entire storage/ directory during testing.
Known limitations (important)
Crash consistency
coldkeep is not crash-consistent. Some operations combine filesystem writes with DB transactions, but filesystem changes cannot be rolled back if a DB transaction fails.
Possible effects:
- Orphan container files on disk
- Temporary disagreement between DB and disk
- Partially applied container sealing/compression
Use only with disposable test data.
Whole-container compression
If container compression is enabled (gzip/zstd), restores may become very slow because compressed streams are not seekable.
Default for this POC is no container compression.
Concurrency & integrity
- Concurrency guarantees are minimal.
- Heavy concurrent store/remove/gc is not a goal for v0.
- Concurrent operations may leave orphan bytes inside containers.
Security
See SECURITY.md.
This project is a prototype and should not be used to protect sensitive information.
Development
Tests
- Unit tests:
go test ./... - Integration tests require Postgres (see tests for details).
Smoke script
scripts/smoke.sh provides a quick end‑to‑end test using the samples/
folder.
Local (without Docker)
docker compose up -d postgres go build -o coldkeep ./app bash scripts/smoke.sh
Docker
docker compose up -d postgres
docker compose run --rm
-e COLDKEEP_SAMPLES_DIR=/samples
-e COLDKEEP_STORAGE_DIR=/tmp/coldkeep-storage
-v "$PWD/samples:/samples"
--entrypoint bash
app scripts/smoke.sh
Roadmap ideas
- Framed container format for random access with compression.
- Stronger crash consistency.
- Safer concurrent operations.
- Cloud backends experiments.
- CLI improvements and richer stats.
Contributing
Contributions and discussion are welcome.
See CONTRIBUTING.md.
License
Apache-2.0. See LICENSE.
Directories