mapstore

package module
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 13, 2026 License: MIT Imports: 18 Imported by: 0

README

MapStore for Go

Go Report Card lint test

MapStore is a local, filesystem‑backed map database with pluggable codecs (JSON or custom), optional per‑key encryption via the OS keyring, and optional full‑text search via SQLite FTS5.

Features

  • File store

    • It keeps a map[string]any in sync with files on disk, the file can be encoded as JSON (inbuilt), or any format using a custom file encoder/decoder.
    • It is a thread-safe map store with atomic file writes and optimistic concurrency.
    • Pluggable codecs for both keys and values inside the map, including an encrypted string encoder backed by github.com/zalando/go-keyring.
    • Listener hooks so callers can observe every mutation written to disk.
    • Optional SQLite FTS5 integration for fast search, with helpers for incremental sync.

  • Directory store: A convenience manager that partitions data across subdirectories and paginates listings.

  • Pure Go (no cgo), cross-platform (Linux/macOS/Windows), Go 1.25+.

Capabilities and Extensibility

  • File encoders

    • Supply your own IOEncoderDecoder via WithFileEncoderDecoder.
    • JSON file encode/decode - use the inbuilt jsonencdec.JSONEncoderDecoder to encode/decode files as JSON.

  • Encode key or value at sub-path

    • Override encoding of specific keys or values with WithKeyEncDecGetter or WithValueEncDecGetter.
    • Value encryption - use the inbuilt keyringencdec.EncryptedStringValueEncoderDecoder to transparently store sensitive string values through the OS keyring.

  • Directory Partitioning

    • Swap in your own PartitionProvider to control directory layout.
    • Month based partitioning - use the inbuilt dirpartition.MonthPartitionProvider to split files across month based directories.

  • File naming

    • The file store is opaque to filenames, allowing for any naming scheme.
    • The directory store uses a FileKey based design to allow for control of encoding and decoding of data inside file names for efficient traversal.
    • UUIDv7 based filename provider - use the inbuilt UUIDv7 based provider to derive and use, collision free and semantic data based filenames.

  • File change events

    • Custom listeners can be plugged into the file store to observe file events.
    • Pluggable Full text search
      • Inbuilt, pure go, sqlite backed (via glebarez driver + modernc sqlite), fts engine.
      • Pluggable iterator utility ftsengine.SyncIterToFTS for efficient, incremental index updates.

Installation

go get github.com/ppipada/mapstore-go

Quick Start

Packages

  • github.com/ppipada/mapstore-go — file and directory stores, options, events, and types.
  • github.com/ppipada/mapstore-go/jsonencdec — JSON file encoder/decoder.
  • github.com/ppipada/mapstore-go/keyringencdec — per-value encryption (AES‑256‑GCM; key in OS keyring).
  • github.com/ppipada/mapstore-go/dirpartition — partitioning strategies (month/no-op).
  • github.com/ppipada/mapstore-go/uuidv7filename — UUIDv7‑backed filename helpers.
  • github.com/ppipada/mapstore-go/ftsengine — FTS5 engine and sync helpers.

Concurrency Model

MapStore uses optimistic concurrency when writing files:

  • Writes create a deep copy of the in‑memory map, encode it (value encoding first, then key encoding), and atomically rename a temp file into place.
  • Before writing, the store compares current file stat to a remembered snapshot. If it changed, SetAll and DeleteFile return a conflict error. SetAll retries a few times automatically; key‑level mutations (SetKey/DeleteKey) honor the AutoFlush setting and propagate conflict errors from the internal flush.
  • This is best‑effort across processes. Two writers racing between the pre‑write CAS check and rename may still result in last‑writer‑wins. If you need stronger cross‑process guarantees, coordinate at the application level.

Keyring Notes

  • keyringencdec.EncryptedStringValueEncoderDecoder uses the OS keyring to store the AES‑256 key.
  • In headless or container environments, ensure a compatible keyring backend is available, or avoid using the keyring‑based encoder.

Development

  • Formatting follows gofumpt and golines via golangci-lint, which is also used for linting. All rules are in .golangci.yml.
  • Useful scripts are defined in taskfile.yml; requires Task.

License

Copyright (c) 2026 - Present - Pankaj Pipada

All source code in this repository, unless otherwise noted, is licensed under the MIT License. See LICENSE for details.

Documentation

Index

Constants

View Source 
const (
	SortOrderAscending  = "asc"
	SortOrderDescending = "desc"
)

Variables

View Source 
var ErrFileConflict = errors.New("concurrent modification detected for a file")

ErrFileConflict is when flush/delete detects that somebody modified the file since we last read/wrote it.

Functions

This section is empty.

Types

type DirOption 

type DirOption func(*MapDirectoryStore)

DirOption is a functional option for configuring the MapDirectoryStore.

func WithDirFileListeners 

func WithDirFileListeners(ls ...FileListener) DirOption

WithDirFileListeners registers one or more listeners when the directory store is created.

func WithDirLogger added in v0.0.3 

func WithDirLogger(logger *slog.Logger) DirOption

WithDirLogger adds a logger to use for the map dir store.

func WithDirPageSize 

func WithDirPageSize(size int) DirOption

WithDirPageSize sets the default page size for pagination.

type FileEntry 

type FileEntry struct {
	BaseRelativePath string
	PartitionName    string
	FileInfo         os.FileInfo
}

type FileEvent 

type FileEvent struct {
	Op Operation
	// Absolute path of the backing JSON file.
	File string
	// Nil for file-level ops.
	Keys []string
	// Nil for OpSetFile / OpResetFile.
	OldValue any
	// Nil for delete.
	NewValue any
	// Deep-copy of the entire map after the change.
	Data      map[string]any
	Timestamp time.Time
}

FileEvent is delivered *after* a mutation has been written to disk.

type FileKey 

type FileKey struct {
	FileName string
	XAttr    any
}

type FileKeyEncDecGetter 

type FileKeyEncDecGetter func(pathSoFar []string) StringEncoderDecoder

FileKeyEncDecGetter: given the path so far, if applicable, returns a StringEncoderDecoder It encodes decodes: The key at the path i.e last part of the path array. Key encoders are applied after values. If a key decoder fails (e.g., invalid base64), loading fails with a descriptive error for visibility.

type FileListener 

type FileListener func(FileEvent)

FileListener is a callback that observes mutations.

type FileOption 

type FileOption func(*MapFileStore)

FileOption defines a function type that applies a configuration option to the MapFileStore.

func WithCreateIfNotExists 

func WithCreateIfNotExists(createIfNotExists bool) FileOption

WithCreateIfNotExists sets the option to create the file if it does not exist.

func WithFileAutoFlush 

func WithFileAutoFlush(autoFlush bool) FileOption

WithFileAutoFlush sets the AutoFlush option.

func WithFileEncoderDecoder 

func WithFileEncoderDecoder(encoder IOEncoderDecoder) FileOption

WithFileEncoderDecoder sets a custom encoder/decoder for the store.

func WithFileListeners 

func WithFileListeners(ls ...FileListener) FileOption

WithFileListeners registers one or more listeners during store creation.

func WithFileLogger added in v0.0.3 

func WithFileLogger(logger *slog.Logger) FileOption

WithFileLogger adds a logger to use for the map file store.

func WithKeyEncDecGetter 

func WithKeyEncDecGetter(getter FileKeyEncDecGetter) FileOption

WithKeyEncDecGetter registers the user’s key encoding decoding handler callback.

func WithValueEncDecGetter 

func WithValueEncDecGetter(valueEncDecGetter FileValueEncDecGetter) FileOption

WithValueEncDecGetter registers the user’s value encoding decoding handler callback.

type FileValueEncDecGetter 

type FileValueEncDecGetter func(pathSoFar []string) IOEncoderDecoder

FileValueEncDecGetter: given the path so far, if applicable, returns a EncoderDecoder. It encodes decodes: Value at the key i.e value at last part of the path array. Value encoders are applied to values first; when encoding, their output is base64‑encoded for storage. On decode, if the stored value is not a string, it is left unchanged (tolerant mode). Provide your `WithValueEncDecGetter` to control which paths get encoded.

type IOEncoderDecoder 

type IOEncoderDecoder interface {
	Encode(w io.Writer, value any) error
	Decode(r io.Reader, value any) error
}

IOEncoderDecoder is an interface that defines methods for encoding and decoding data.

type ListingConfig 

type ListingConfig struct {
	SortOrder        string
	PageSize         int
	FilterPartitions []string // If empty, list all partitions.
	FilenamePrefix   string   // If non-empty, only return files with this prefix.
}

ListingConfig holds all options for listing files.

type MapDirectoryStore 

type MapDirectoryStore struct {
	// contains filtered or unexported fields
}

MapDirectoryStore manages multiple MapFileStores within a directory.

func NewMapDirectoryStore 

func NewMapDirectoryStore(
	baseDir string,
	createIfNotExists bool,
	partitionProvider PartitionProvider,
	fileEncoderDecoder IOEncoderDecoder,
	opts ...DirOption,
) (*MapDirectoryStore, error)

NewMapDirectoryStore initializes a new MapDirectoryStore with the given base directory and options.

func (*MapDirectoryStore) CloseAll 

func (mds *MapDirectoryStore) CloseAll() error

CloseAll closes every cached MapFileStore in this directory instance and clears the cache.

func (*MapDirectoryStore) CloseFile 

func (mds *MapDirectoryStore) CloseFile(fileKey FileKey) error

CloseFile closes the MapFileStore for the given FileKey (if it was opened) and removes it from the cache.

func (*MapDirectoryStore) DeleteFile 

func (mds *MapDirectoryStore) DeleteFile(fileKey FileKey) error

DeleteFile removes the file with the given filename from the base directory. It is a thin wrapper around Open and DeleteFile.

func (*MapDirectoryStore) GetFileData 

func (mds *MapDirectoryStore) GetFileData(
	fileKey FileKey,
	forceFetch bool,
) (map[string]any, error)

GetFileData returns the data from the specified file in the store. It is a thin wrapper around Open and GetAll.

func (*MapDirectoryStore) ListFiles 

func (mds *MapDirectoryStore) ListFiles(
	config ListingConfig,
	pageToken string,
) (fileEntries []FileEntry, nextPageToken string, err error)

ListFiles lists files according to the config and page token. - When filtering partitions via `ListingConfig.FilterPartitions`, unreadable or missing partitions are skipped. - When not filtering (listing all), an unreadable partition is a hard error from the provider read. - `FilenamePrefix` filters files by basename prefix within each partition, after sorting.

func (*MapDirectoryStore) ListPartitions 

func (mds *MapDirectoryStore) ListPartitions(
	baseDir, sortOrder, pageToken string,
	pageSize int,
) (partitions []string, nextPageToken string, err error)

func (*MapDirectoryStore) OpenFile 

func (mds *MapDirectoryStore) OpenFile(
	fileKey FileKey,
	createIfNotExists bool,
	defaultData map[string]any,
) (*MapFileStore, error)

OpenFile returns a cached or newly created MapFileStore for the given FileKey. It is concurrency-safe and ensures only one instance per file path.

func (*MapDirectoryStore) SetFileData 

func (mds *MapDirectoryStore) SetFileData(fileKey FileKey, data map[string]any) error

SetFileData sets the provided data for the given file. It is a thin wrapper around Open and SetAll.

type MapFileStore 

type MapFileStore struct {
	// contains filtered or unexported fields
}

MapFileStore is a file-backed implementation of a thread-safe key-value store. Concurrency:

  • Uses optimistic CAS (os.Stat snapshot) plus atomic rename for writes.
  • Across processes this is best-effort; if the file changes between the CAS check and rename, last-writer-wins may occur.
  • SetAll retries on conflict; DeleteFile returns ErrFileConflict on change.

func NewMapFileStore 

func NewMapFileStore(
	filename string,
	defaultData map[string]any,
	fileEncoderDecoder IOEncoderDecoder,
	opts ...FileOption,
) (*MapFileStore, error)

NewMapFileStore initializes a new MapFileStore. If the file does not exist and createIfNotExists is false, it returns an error.

func (*MapFileStore) Close 

func (store *MapFileStore) Close() error

func (*MapFileStore) DeleteFile 

func (store *MapFileStore) DeleteFile() error

DeleteFile removes the backing file atomically, emits an OpDeleteFile event and clears lastStat. Returns ErrFileConflict if the file changed since we last observed it.

func (*MapFileStore) DeleteKey 

func (store *MapFileStore) DeleteKey(keys []string) error

DeleteKey deletes the value associated with the given key. The key can be a dot-separated path to a nested value.

func (*MapFileStore) Flush 

func (store *MapFileStore) Flush() error

Flush writes the current data to the file. No event is emitted for flush.

func (*MapFileStore) GetAll 

func (store *MapFileStore) GetAll(forceFetch bool) (map[string]any, error)

GetAll returns a copy of all data in the store, refreshing from the file first.

func (*MapFileStore) GetKey 

func (store *MapFileStore) GetKey(keys []string) (any, error)

GetKey retrieves the value associated with the given key. The key can be a dot-separated path to a nested value.

func (*MapFileStore) Reset 

func (store *MapFileStore) Reset() error

Reset removes all data from the store.

func (*MapFileStore) SetAll 

func (store *MapFileStore) SetAll(data map[string]any) error

SetAll overwrites all data in the store with the provided data. It retries automatically if another writer wins the race and flushUnlocked returns ErrFileConflict.

func (*MapFileStore) SetKey 

func (store *MapFileStore) SetKey(keys []string, value any) error

SetKey sets the value for the given key. The key can be a dot-separated path to a nested value.

type Operation 

type Operation string

Operation is the kind of mutation that happened on a file or a key. 

const (
	OpSetFile    Operation = "setFile"
	OpResetFile  Operation = "resetFile"
	OpDeleteFile Operation = "deleteFile"
	OpSetKey     Operation = "setKey"
	OpDeleteKey  Operation = "deleteKey"
)

type PartitionProvider 

type PartitionProvider interface {
	GetPartitionDir(key FileKey) (string, error)
	ListPartitions(baseDir, sortOrder, pageToken string,
		pageSize int) (partitions []string, nextPageToken string, err error)
}

PartitionProvider defines an interface for determining the partition directory for a file.

type StringEncoderDecoder 

type StringEncoderDecoder interface {
	Encode(plain string) string
	Decode(encoded string) (string, error)
}

StringEncoderDecoder is an interface that defines methods for encoding and decoding a string to another string.

Source Files

View all Source files

Directories

Path Synopsis
internal
encdecutil
fileutil
integration
maputil
Package keyringencdec provides a value encoder/decoder backed by the OS keyring.
 Package keyringencdec provides a value encoder/decoder backed by the OS keyring.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.