sumdb

README

sumdb

A custom implementation of the Go sumdb server.

This package implements the standard sumdb protocol via golang.org/x/mod/sumdb. When a client requests a module checksum, the server checks the local Store first. If no record exists, it fetches the module from the upstream proxy (default: proxy.golang.org), computes the h1 hashes, stores the record with its Merkle tree hashes, and returns the result.

Usage

go get github.com/pseudomuto/sumdb

Implement the Store interface to provide persistence:

• RecordID / Records / AddRecord - module record storage
• ReadHashes / WriteHashes - Merkle tree hash storage
• TreeSize / SetTreeSize - tree state management

See godoc for the full interface and examples/db/ for a complete SQLite implementation.

Data Model

The sumdb maintains three types of data:

Records are module checksum entries. Each record contains the module path, version, and the h1: hash lines (one for the module zip, one for go.mod). Records are assigned sequential IDs starting from 0.

Hashes form a Merkle tree that provides cryptographic proof of the record history. When a record is added, its content is hashed and incorporated into the tree. The tree structure allows clients to verify that records haven't been tampered with and that the server is append-only.

Tree size tracks the current number of records. This is used to compute the tree's root hash and to determine where new records are inserted.

When a new module is looked up:

1. The module is fetched from the upstream proxy and its h1: hashes are computed
2. A record is created with the module's checksums
3. The Merkle tree hashes are computed for the new record's position
4. The tree size is incremented

The signed tree head (returned by Signed()) contains the current tree size and root hash, signed with the server's private key. Clients use this to verify the integrity of records they receive.

Concurrency

The SumDB type is safe for concurrent use. Module lookups use a three-tier concurrency model:

1. Fast path (concurrent): Existing records are looked up via RecordID without any locking. Multiple goroutines can read simultaneously.

2. Singleflight deduplication: When a module isn't found, concurrent requests for the same module are deduplicated. Only one goroutine fetches from the upstream proxy; others wait and receive the same result. This prevents redundant network calls.

3. Serialized writes: Record creation is protected by a mutex because each record's position in the Merkle tree depends on the current tree size. Concurrent inserts of different modules are serialized to maintain tree consistency.

If your Store implementation supports transactions (by implementing TxStore), the record insert and tree hash updates are wrapped in a transaction for atomicity. This ensures that a failure during tree hash computation won't leave an orphaned record.

Important: A Store instance should only be used by a single SumDB. Sharing a Store across multiple SumDB instances is not supported and may corrupt the Merkle tree.

Documentation

Index

• Variables
• func GenerateKeys(name string) (string, string, error)
• type Option
  • func WithHTTPClient(c *http.Client) Option
  • func WithStore(s Store) Option
  • func WithUpstream(u *url.URL) Option
• type Record
• type Store
• type SumDB
  • func New(name string, skey string, opts ...Option) (*SumDB, error)
  • func (s *SumDB) Handler() http.Handler
  • func (s *SumDB) Lookup(ctx context.Context, mod module.Version) (int64, error)
  • func (s *SumDB) ReadRecords(ctx context.Context, id, n int64) ([][]byte, error)
  • func (s *SumDB) ReadTileData(ctx context.Context, t tlog.Tile) ([]byte, error)
  • func (s *SumDB) Signed(ctx context.Context) ([]byte, error)
• type TxStore

Variables

var ErrNotFound = errors.New("record not found")

ErrNotFound is returned when a requested record does not exist in the store.

Functions

func GenerateKeys

func GenerateKeys(name string) (string, string, error)

GenerateKeys creates a new keypair and returns the encoded signer key, and verifier key.

The name identifies the key (e.g., "sum.golang.org").

The signer key is secret and must be stored securely. The verifier key can be shared publicly for clients to verify signatures.

Types

type Option

type Option func(*SumDB)

Option configures a SumDB instance.

func WithHTTPClient

func WithHTTPClient(c *http.Client) Option

WithHTTPClient sets the client used to communicate with the proxy.

func WithStore

func WithStore(s Store) Option

WithStore sets the Store for handling persistence of the tree.

func WithUpstream

func WithUpstream(u *url.URL) Option

WithUpstream sets the upstream proxy to query when no records are found.

type Record

type Record struct {
	ID      int64
	Path    string
	Version string
	Data    []byte
}

Record represents a module checksum entry in the sumdb.

type Store

type Store interface {
	// RecordID returns the ID of the record for the given module path and version.
	// Returns ErrNotFound if no record exists.
	RecordID(ctx context.Context, path, version string) (int64, error)

	// Records returns records with IDs in the interval [id, id+n).
	// The returned slice may have fewer than n records if the range extends
	// beyond the current tree size.
	Records(ctx context.Context, id, n int64) ([]*Record, error)

	// AddRecord adds a new entry for the specified module.
	// The record's ID field is ignored; the store assigns the next sequential ID.
	// Returns the assigned ID.
	AddRecord(ctx context.Context, r *Record) (int64, error)

	// ReadHashes returns the hashes at the given storage indexes.
	// Indexes are computed using tlog.StoredHashIndex(level, n).
	// The returned slice must have the same length as indexes.
	ReadHashes(ctx context.Context, indexes []int64) ([]tlog.Hash, error)

	// WriteHashes stores hashes at the given storage indexes.
	// indexes and hashes must have the same length.
	WriteHashes(ctx context.Context, indexes []int64, hashes []tlog.Hash) error

	// TreeSize returns the current number of records in the tree.
	TreeSize(ctx context.Context) (int64, error)

	// SetTreeSize updates the tree size.
	// This should be called after successfully adding a record and its hashes.
	SetTreeSize(ctx context.Context, size int64) error
}

Store defines the persistence interface for sumdb data. Implementations must be safe for concurrent use.

A Store instance should only be used by a single SumDB. Sharing a Store across multiple SumDB instances is not supported and may corrupt the Merkle tree, as write serialization is handled at the SumDB level.

type SumDB

type SumDB struct {
	// contains filtered or unexported fields
}

SumDB is a checksum database server that implements the Go sumdb protocol.

It implements the ServerOpts interface defined in https://pkg.go.dev/golang.org/x/mod@v0.31.0/sumdb#ServerOps.

func New

func New(name string, skey string, opts ...Option) (*SumDB, error)

New creates a new SumDB instance with the given server name and signing key. The name identifies this sumdb (e.g., "sum.example.com"). The skey must be in note signer format: "PRIVATE+KEY+<name>+<hash>+<keydata>".

NB: You can use GenerateKeys to create a valid signing key.

func (*SumDB) Handler

func (s *SumDB) Handler() http.Handler

Handler returns an HTTP handler for serving the sumdb over HTTP.

func (*SumDB) Lookup

func (s *SumDB) Lookup(ctx context.Context, mod module.Version) (int64, error)

Lookup finds or creates a record for the given module version. If the record doesn't exist, it fetches the module from the upstream proxy, computes the checksums, and stores the new record with its tree hashes. Concurrent lookups for the same module are deduplicated via singleflight.

func (*SumDB) ReadRecords

func (s *SumDB) ReadRecords(ctx context.Context, id, n int64) ([][]byte, error)

ReadRecords returns the raw data for records with IDs in [id, id+n).

func (*SumDB) ReadTileData

func (s *SumDB) ReadTileData(ctx context.Context, t tlog.Tile) ([]byte, error)

ReadTileData returns the raw record data for a data tile. Data tiles (L=-1) contain concatenated record data rather than hashes.

func (*SumDB) Signed

func (s *SumDB) Signed(ctx context.Context) ([]byte, error)

Signed returns the signed tree head for the current tree state.

type TxStore

type TxStore interface {
	Store

	// WithTx executes fn within a database transaction.
	// If fn returns nil, the transaction is committed.
	// If fn returns an error or panics, the transaction is rolled back.
	//
	// The Store passed to fn represents the transactional view and must be used
	// for all operations within the callback.
	WithTx(ctx context.Context, fn func(Store) error) error
}

TxStore is an optional extension of Store that provides transaction support. When a Store implements TxStore, atomic operations will use transactions.

Implementations that do not support transactions can simply implement Store. The SumDB will detect TxStore support at runtime and use transactions when available.