restinpieces

README

REST in pieces

RestInPieces is a Go framework for building secure, high-performance servers. It is designed to be extended and customized, providing a solid foundation for your own applications while remaining lightweight and focused. The framework uses SQLite as its default database and minimizes reliance on third-party packages, emphasizing simplicity and performance.

To get started, follow the Bootstrapping Guide, which walks you through the initial setup of a new application.

Core Philosophy: One Process Application

This framework is built on the philosophy of One Process Application architectural approach that consolidates an entire application and its core dependencies into a single, self-contained binary.

The central idea is to "absorb" the typical components of a modern web service—such as the database, cache, and job queue—into one application binary. Instead of managing a complex stack of separate services (e.g., a database server, a Redis instance, a reverse proxy), you deploy a single binary. This dramatically simplifies development, deployment, and maintenance.

By running a single Go binary with an embedded SQLite database on one VM, it allows developers to focus on building features rather than managing distributed systems, providing a high-performance foundation that can serve a growing business for years. When the time comes to scale beyond what a single large server can offer, the business will have the resources and clarity to do so effectively.

This approach follows One Process Programming Notes.

Content

Framework Key Features

• Data Durability
• Database Drivers
• Router
• Cache
• Authentication
• Security
• Core Infrastructure
• Configuration Management
• Deployment & Operations
• Frontend Integration
• Job Framework
• Performance
• Metrics
• Logger
• Notifications
• Mailer
• Middleware
• Extensibility

Building on the Framework

• Layout Best Practices
• Examples
• Building the Project
• TODO

Key Features

Data Durability

The "one process" paradigm intentionally avoids external dependencies like separate database servers, as they would violate the architectural principle of maintaining a single process per virtual machine. Consequently, the framework relies on an embedded SQLite database for data persistence. This design choice places critical importance on the durability of the single database file.

To address this, the framework provides pure Go implementations of all available SQLite sync protocols, in the restinpieces-backup repository, and continuous real-time replication with point-in-time recovery via restinpieces-litestream:

Method	Use	Implementation
Online Backup API	local backup	cmd/onlineapi
VACUUM INTO	local backup	cmd/vacuum
sqlite3_rsync	remote backup, delta-based	cmd/sqlite-rsync
rsync pull client	remote backup, incremental	cmd/rsync
sftp pull client	remote backup	cmd/sftp
litestream	continuous replication, point-in-time	restinpieces-litestream

For the framework-side backup configuration, see doc/backup.md.

Database Drivers

The framework uses pure-Go modernc.org/sqlite; zombiezen.com/go/sqlite as the alternative.

Router

The framework uses Go's standard http.ServeMux as the default router. Since Go 1.22 it supports path parameters. The router is swappable; an alternative based on julienschmidt/httprouter is at restinpieces-httprouter.

Cache

For in-memory caching, the framework ships its own preallocated LRU cache (package cache), built entirely on the Go standard library. The cache.Cache interface lets you swap in your own implementation via WithCache; a ristretto-based implementation is available at restinpieces-cache.

Authentication

The framework authenticates with JSON Web Tokens (JWT) sent as bearer tokens in the Authorization header. JWT signing keys derive from user credentials (email and password hash) plus a server secret, so changing the password invalidates existing tokens.

The system supports multiple authentication and account management workflows through a set of API endpoints:

• Password-based: User registration (/register-with-password), login (/auth-with-password), and token refresh (/auth-refresh).
• OAuth2: (/auth-with-oauth2) exchanges the provider token, fetches user info, and creates or links the local user account. (/list-oauth2-providers) lists configured providers.
• Account Management: Email verification, password reset, and email change run as multi-step flows. Each flow sends a unique, short-lived JWT to the user's email via the job queue; the user submits it back to a confirmation endpoint.

Security

The "one process" paradigm simplifies deployment by running a single binary on a single VM, but it also means the application is directly exposed to the internet without a reverse proxy like Nginx acting as a first line of defense. This necessitates a defensive approach to security. The framework addresses this with a suite of built-in middleware designed to protect the server from common threats. These include dynamic IP blocking (BlockIp) to mitigate traffic spikes, hostname validation against a whitelist (BlockHost), request body size limitation (BlockRequestBody), and User-Agent filtering (BlockUaList). The framework also helps secure client communications by automatically setting security headers like Strict-Transport-Security.

No CORS support is provided as it contradicts the One Process philosophy. If you need cross-origin requests, you'll need to implement CORS middleware yourself.

Core Infrastructure

The framework is built on standard Go patterns, utilizing middleware and handlers to provide a familiar and robust development experience. It features a set of discoverable API endpoints for essential services, such as token refreshing (/api/refresh-auth) and OAuth2 authentication (/api/auth-with-oauth2), facilitating easy integration and exploration.

Configuration Management

Configuration lives in the SQLite database as encrypted TOML in the app_config table (schema: migrations/schema/app/app_config.sql). ripc manages it — a server-side tool that reads and writes the local database and age key files on the production machine. It supports versioning, diffing, and rollbacks. ripc also handles custom configuration scopes for your modules. See the ripc documentation.

ripc runs on the server and edits local state; ripdep (source) runs on your machine and calls ripc over SSH.

A key feature is support for dynamic updates. The server listens for the SIGHUP signal to trigger a hot-reload of the configuration, allowing most settings to be changed in real-time without service interruption. While the majority of parameters can be updated on-the-fly, critical changes like modifications to TLS certificates require a full server reload to be applied.

Deployment & Operations

The framework provides ripdep, a comprehensive CLI tool designed to manage the full lifecycle of your application. It acts as a high-level wrapper around the ripc binary, orchestrating complex DevOps tasks and remote operations via SSH directly from your local developer machine (control plane). In contrast, ripc is the server-side companion — it runs on the production machine itself, operating on the local filesystem.

• Remote DevOps: Wraps low-level ripc commands to handle configuration, maintenance modes, and log monitoring without needing manual server access.
• Disaster Recovery: Simplifies the process of bootstrapping new servers and recovering from backups (including Litestream integration) through dedicated commands like build-bootstrap and build-recovery.

This tool encourages a workflow where most configuration and operational decisions are made locally, then securely applied to the remote environment. For detailed usage, see the Deployment Guide.

Frontend Integration

The framework includes a JavaScript SDK for frontend-backend interaction. The SDK covers password-based and OAuth2 flows, plus error handling, local storage, and request helpers. See example usage at restinpieces-js-sdk.

Job Framework

The framework includes a job queue for asynchronous tasks, one-time and recurrent. It moves work such as sending emails off the request-response cycle.

The system is composed of a scheduler that claims jobs from the job_queue table and an executor that runs the corresponding handler. The framework provides built-in handlers for core functionalities like sending password reset emails, email verifications, and performing local database backups.

You can easily extend the system to run your own custom tasks. This involves two main steps:

1. Write a Job Handler: Create a new handler that implements the JobHandler interface. This is where you define the logic for your task.
2. Insert a Job: Add a new record to the job_queue table in the database. The scheduler will automatically pick it up and execute it using your custom handler.

This design allows for a clean separation of concerns and makes it straightforward to add new background processing capabilities to your application.

Performance

Engineered for high throughput, the framework is capable of handling thousands of requests per second while maintaining a minimal footprint by avoiding unnecessary external dependencies. Production-ready builds are further optimized for size and efficiency, ensuring rapid deployment and execution in resource-constrained environments.

Metrics

The framework provides built-in metrics collection using the prometheus/client_golang library. It includes a middleware that tracks the total number of HTTP requests (http_server_requests_total), a counter labeled by HTTP status code, allowing for detailed monitoring of server responses. Metrics collection can be toggled on or off via configuration without a server restart and is exposed on a configurable endpoint (e.g., /metrics) for a Prometheus server to scrape.

Logger

The framework's logging is built upon the standard slog library for structured logging. It includes a high-performance batching handler that writes logs to the SQLite database, with configurable flush intervals and log levels. For incoming requests, a dedicated middleware logs request details but truncates overly long URI, User-Agent, Referer, and IP values to maintain clean logs. The entire logging implementation can be replaced with a user-defined logger to accommodate custom requirements.

Notifications

The framework's notification system is designed around a Notifier interface, which standardizes how notifications are sent. The primary data structure, Notification, carries a Type (e.g., Alarm, Metric), Source, Message, and a map of Fields for additional structured data.

An official implementation for Discord is included, which sends formatted messages to a configured webhook URL. This notifier operates asynchronously, using goroutines for non-blocking Send calls. It incorporates a rate limiter to prevent API abuse and automatically truncates messages that exceed Discord's 2000-character limit. Developers can create custom notifiers for other services (like Slack or email) by providing their own implementation of the Notifier interface.

Mailer

The framework includes a Mailer component for sending transactional emails over SMTP. It is designed to be flexible and resilient, handling common account management workflows.

• Configuration: The mailer is configured through the application's central configuration provider, allowing for dynamic updates to SMTP settings (host, port, credentials, TLS) without a server restart.
• Protocol Support: It supports standard SMTP authentication methods (PLAIN, CRAM-MD5) and connection security (explicit TLS and STARTTLS).
• Transactional Emails: Pre-built methods are included for common user actions:
  • Email address verification
  • Password reset requests
  • Email change notifications
• Asynchronous Sending: Emails are sent in a non-blocking manner using goroutines, with context-based timeouts to prevent long-running operations from impacting application performance.
• Templating: It uses simple, embedded HTML templates for emails, which can be easily customized.

Middleware

The framework provides a collection of built-in middleware to handle common cross-cutting concerns like security, logging, and metrics.

• ResponseRecorder: A utility middleware that wraps the standard http.ResponseWriter to capture the status code, response size, and timing information. This is used internally by other middleware like Metrics and RequestLog and should typically be the first middleware in the chain.
• RequestLog: Provides structured logging for every incoming HTTP request. It captures details like method, URI, status, duration, remote IP, and user agent, with configurable length limits to keep logs concise.
• Metrics: Collects Prometheus-compatible metrics for HTTP requests, labeled by status code. When activated, metrics are exposed on a configurable endpoint (e.g., /metrics) for scraping.
• BlockIp: Acts as a dynamic IP blocking mechanism to protect the server from traffic spikes and potential denial-of-service attacks. It uses a Top-K sketch algorithm to identify and temporarily block IP addresses that are responsible for a disproportionate amount of traffic, a circuit breaker under heavy load.
• BlockHost: Enforces security by validating the Host header of incoming requests against a configurable whitelist of allowed hostnames. It supports exact matches and wildcard subdomains (e.g., *.example.com).
• BlockRequestBody: Limits the size of incoming request bodies to a configurable maximum. This helps prevent resource exhaustion from excessively large payloads and can be configured to exclude specific URL paths.
• BlockUaList: Filters requests by matching the User-Agent string against a configurable regular expression. This can be used to block scrapers, bots, or other unwanted clients.
• TLSHeaderSTS: Sets the Strict-Transport-Security (HSTS) header for all responses served over a TLS connection, instructing browsers to communicate with the server only over HTTPS.
• Maintenance: When activated via configuration, this middleware puts the server into maintenance mode. It responds to all requests with a 503 Service Unavailable status code, allowing for system updates without shutting down the server.
• Gzip: Serves pre-compressed static assets (.gz files) from a given file system (fs.FS) to clients that support gzip encoding. This reduces bandwidth and improves load times. If a compressed file is not found, it seamlessly falls back to the next handler.

Examples

Detailed examples and integration guides are available to help you build with the framework. You can explore a complete JavaScript SDK Integration at restinpieces-js-sdk to see how to connect your frontend, or review implementations of Custom Routers and DB Drivers at restinpieces-non-default for advanced customization scenarios.

Extensibility

Beyond its core features, the framework is designed to be easily extended to meet diverse application needs. It includes a built-in file server with gzip compression for efficient delivery of static assets and a dedicated asset pipeline for minification and bundling of HTML, CSS, and JavaScript, leveraging scripts available at restinpieces-js-sdk/gen.

Layout Best Practices

Applications built on restinpieces follow this structure:

myapp/
├── cmd/myapp/main.go    # entry point: flags, wiring, daemons, jobs, srv.Run()
├── app.go               # your App wrapper — embeds *core.App, adds your state
├── handlers/            # HTTP handlers as methods on *App
├── middleware/          # custom middleware (App-aware via closure, or plain funcs)
├── routes.go            # register framework + application routes
├── jobs/                # job handler implementations
├── daemons/             # daemon constructors
└── web/src/, web/dist/  # frontend assets, embedded via go:embed

App wrapper. Define your own *App struct that embeds *core.App and adds project-specific state (extra DB pools, third-party clients). Handlers are methods on *App — no global variables, trivial to test.

Handlers and middleware. Handlers use standard http.HandlerFunc signatures. Middleware closes over *App when it needs framework services; stateless middleware stays a plain func(http.Handler) http.Handler.

Routes. All route registration lives in routes.go. Call app.Router().Register() with your chains — the framework's built-in routes are registered internally by restinpieces.New(), yours go on top.

Jobs and daemons. Job handlers implement the framework's handler interface; daemons are long-running background processes. Both are constructed with the state they need and registered in main.go via srv.AddJobHandler / srv.AddDaemon.

Configuration. Store secrets encrypted via age in the shared SQLite database. Use your own config scope (never "application", which the framework reserves). Generations are immutable — save always creates a new record, giving you a full audit trail.

For full detail and code examples, see Layout Best Practices.

Building the Project

Build Server

Builds the example server application.

go build -ldflags="-s -w" -trimpath -o restinpieces_server ./cmd/example/

Build CLI

Builds the ripc command-line tool.

go build -ldflags="-s -w" -trimpath -o ripc ./cmd/ripc/

TODO

Todos.

Documentation

Index

• Variables
• func New(opts ...Option) (*core.App, *server.Server, error)
• func NewModerncConn(dbPath string, pragmas ...map[string]string) (*sql.DB, error)
• func NewModerncPool(dbPath string, pragmas ...map[string]string) (*sql.DB, error)
• type Option
  • func WithAgeKeyPath(path string) Option
  • func WithCache(c cache.Cache[string, interface{}]) Option
  • func WithDbApp(dbApp db.DbApp) Option
  • func WithLogger(l *slog.Logger) Option
  • func WithModerncPool(pool *sql.DB) Option
  • func WithNotifier(n notify.Notifier) Option
  • func WithRouter(r router.Router) Option

Variables

var DefaultLoggerOptions = &slog.HandlerOptions{
	Level: slog.LevelDebug,
	ReplaceAttr: func(groups []string, a slog.Attr) slog.Attr {
		if a.Key == slog.TimeKey {
			return slog.Attr{}
		}
		return a
	},
}

Functions

func New

func New(opts ...Option) (*core.App, *server.Server, error)

New creates a new App instance and Server with the provided options. It initializes the core application components like database, router, cache first, then loads configuration from the database.

func NewModerncConn

func NewModerncConn(dbPath string, pragmas ...map[string]string) (*sql.DB, error)

NewModerncConn creates a new single SQLite connection with the shared performance pragmas. The database file must already exist. Caller pragmas may be passed as a map of pragma name to value; a pragma on the same key as a default replaces it.

func NewModerncPool

func NewModerncPool(dbPath string, pragmas ...map[string]string) (*sql.DB, error)

NewModerncPool creates a new Modernc SQLite connection pool with reasonable defaults compatible with restinpieces (e.g., WAL mode enabled, busy_timeout set). Caller pragmas may be passed as a map of pragma name to value, e.g. map[string]string{"cache_size": "10000"}; a pragma on the same key as a default replaces it.

Types

type Option

type Option func(*initializer)

func WithAgeKeyPath

func WithAgeKeyPath(path string) Option

WithAgeKeyPath sets the path to the age identity file in the initializer

func WithCache

func WithCache(c cache.Cache[string, interface{}]) Option

WithCache sets the cache implementation

func WithDbApp

func WithDbApp(dbApp db.DbApp) Option

WithDbApp sets the application's database implementation. It expects a single concrete type (like *crawshaw.Db) that implements db.DbApp.

func WithLogger

func WithLogger(l *slog.Logger) Option

WithLogger sets the logger implementation

func WithModerncPool

func WithModerncPool(pool *sql.DB) Option

WithModerncPool configures the App to use the Modernc SQLite implementation with an existing pool. The user is responsible for creating and managing the lifecycle of the provided pool.

func WithNotifier

func WithNotifier(n notify.Notifier) Option

WithNotifier sets the notifier implementation

func WithRouter

func WithRouter(r router.Router) Option

WithRouter sets the router implementation