mlstore-savings

README

mlstore-savings

A small, local, read-only command-line tool that estimates how much deduplication and compression would shrink a given set of files or directories — so you can size the savings MemoryLayer Storage would give you on your data, without uploading anything.

Privacy guarantee: everything runs fully locally. Files are opened read-only, the scanned data is never modified, and the tool makes no network calls — nothing leaves your machine.

It depends only on upstream open-source libraries (the Kopia content-defined splitter and the klauspost zstd codec, both Apache-2.0 / BSD) plus the Go standard library. It contains no proprietary code.

What it measures

Point it at real files/dirs and it:

1. Walks the paths recursively and splits each file into content-defined chunks, so redundancy is found by content rather than by filename or version.
2. Identifies duplicate chunks across all scanned files, so identical or near-identical content collapses to a single copy.
3. Compresses what is worth compressing: zstd for text/JSON/code, stored raw for already-compressed media (jpg/png/mp4/zip/…) and for tensor formats (.safetensors/.pt/.npy/…), which stay byte-for-byte on disk so they can still be mmap'd or loaded zero-copy.

It then reports the dedup ratio, compression ratio, effective footprint, overall reduction, and a projected storage / egress cost saving.

Install

go install github.com/scitrera/mlstore-savings@latest

Or build from source:

git clone https://github.com/scitrera/mlstore-savings
cd mlstore-savings
go build -o mlstore-savings .

Usage

mlstore-savings [flags] <path>...

Flag	Default	Meaning
-storage-price	0.023	object storage price in $/GB-month (~S3 Standard)
-egress-price	0.09	egress price in $/GB served
-egress-monthly	0	GB/month served (enables egress projection when >0)
-splitter	DYNAMIC-1M-BUZHASH	content-defined chunking splitter
-json	false	emit JSON instead of the human-readable summary

Run mlstore-savings -help for the full list (it also states that nothing leaves your machine).

Examples

Analyze a data directory with a custom price and an egress projection:

mlstore-savings -storage-price 0.023 -egress-monthly 1000 ./data

Machine-readable output:

mlstore-savings -json ./data

Run via Docker against a directory mounted read-only:

docker run --rm -v /data:/in:ro ghcr.io/scitrera/mlstore-savings /in
docker run --rm -v /data:/in:ro ghcr.io/scitrera/mlstore-savings -json /in

Sample output

Storage savings analysis (local, read-only — nothing left this machine)
=======================================================================

Files scanned:        25
Chunks:               30 total, 30 unique

Logical size:         3.88 MiB
After dedup:          3.88 MiB   (dedup ratio 1.00x)
After compression:    2.14 MiB   (compression ratio 1.82x)
Effective footprint:  2.14 MiB

Overall reduction:    45.0%   (1.82x smaller)

Cost projection
---------------
Storage price:        $0.0230 /GB-month
Logical -> stored:    0.004 GB -> 0.002 GB  (0.002 GB saved)
Storage savings:      $0.00 /month   $0.00 /year
Egress price:         $0.0900 /GB   (1000 GB/mo served)
Egress savings:       $40.48 /month   $485.78 /year

How the estimate relates to MemoryLayer Storage

This tool is an estimator. It models the same two levers MemoryLayer Storage applies — content-defined dedup across objects, and compression chosen by content type — so the ratios it reports are a good guide to what the platform would store for the same data. It deliberately stops there: it does not implement the platform's pack format, global dedup index, garbage collection, or server integration, none of which are needed to project savings.

Treat the output as a well-grounded estimate, not a guarantee. Real-world results vary with how your data is written and re-written over time.

License

Apache License 2.0. See LICENSE and NOTICE.

Documentation

Overview

Command mlstore-savings is a local, read-only dedup + compression "savings analyzer". Point it at real files/dirs and it estimates the dedup ratio, compression ratio, effective footprint, and projected $ savings you'd get from MemoryLayer Storage — by modelling the same two levers the platform applies: content-defined dedup across objects, and compression chosen by content type.

Privacy: everything runs locally. The tool opens files read-only, never writes to the scanned data, and makes NO network calls — nothing leaves the machine.

It depends only on upstream open-source libraries (the Kopia content-defined splitter and the klauspost zstd codec, both Apache-2.0) plus the Go standard library. It contains no proprietary MemoryLayer code.

Usage:

mlstore-savings [flags] <path>...

Example:

mlstore-savings -storage-price 0.023 -egress-monthly 1000 ./data
mlstore-savings -json ./data