README
¶
filefs - Fixed-Size File-Backed Filesystem
The filefs package implements the absfs.FileSystem interface using a single
pre-allocated file as backing storage.
filefs is designed for critical systems where storage availability must be guaranteed regardless of host disk pressure. It pre-allocates a fixed-size file on the host filesystem, then manages that space internally with block allocation. Even if other processes fill the host disk completely, filefs continues to operate within its reserved space.
This is analogous to how operating systems use disk partitions to isolate critical storage from user activity, but without requiring kernel-level partition management.
Features
- Guaranteed Storage: Pre-allocates a fixed-size backing file, immune to host disk-full conditions
- POSIX-like File Operations: Create, Open, Read, Write, Seek, Truncate, Close
- Directory Operations: Mkdir, MkdirAll, Remove, RemoveAll, Chdir, Getwd, Readdir
- Symbolic Links: Full symlink support with cycle detection
- File Permissions: Chmod, Chown, Lchown with standard Unix-style permissions
- File Metadata: Stat, Lstat, modification times, ownership, and size tracking
- Persistence: Data writes go directly to disk; metadata is persisted on Sync/Close
- Capacity Tracking: Free() and Capacity() report available and total space
- absfs Interface: Full implementation of absfs.FileSystem and absfs.SymlinkFileSystem
- Thread Safety: All operations are safe for concurrent use
Install
go get github.com/absfs/filefs
Example Usage
package main
import (
"fmt"
"log"
"github.com/absfs/filefs"
)
func main() {
// Create a 64 MB filesystem backed by a single file
fs, err := filefs.New("/var/lib/myapp/storage.fs", 64*1024*1024)
if err != nil {
log.Fatal(err)
}
defer fs.Close()
// Use like any absfs.FileSystem
fs.MkdirAll("/config", 0755)
f, _ := fs.Create("/config/app.json")
f.Write([]byte(`{"key": "value"}`))
f.Close()
// Read it back
data, _ := fs.ReadFile("/config/app.json")
fmt.Println(string(data)) // {"key": "value"}
// Check available space
fmt.Printf("Free: %d MB / %d MB\n",
fs.Free()/1024/1024,
fs.Capacity()/1024/1024)
}
When to Use filefs
Use filefs when:
- Your application must continue writing data even if the host disk fills up
- You need partition-like storage isolation without root privileges or OS support
- You're building a daemon, agent, or embedded system that requires guaranteed log/state storage
- You want a self-contained filesystem image that can be copied or backed up as a single file
Use memfs instead when:
- You don't need persistence (testing, scratch space)
- You don't need protection from host disk pressure
Use osfs instead when:
- You want to work with the real host filesystem directly
- You don't need isolation from disk-full conditions
How It Works
On-Disk Layout
The backing file is divided into fixed-size blocks (4096 bytes by default):
+-------------+----------------+------------------+------------------+
| Superblock | Block Bitmap | Metadata Region | Data Blocks |
| (1 block) | (variable) | (256 blocks) | (remaining) |
+-------------+----------------+------------------+------------------+
- Superblock: Filesystem parameters, layout offsets, free block count
- Block Bitmap: One bit per data block (0 = free, 1 = allocated)
- Metadata Region: Serialized directory tree, block mappings, and symlinks
- Data Blocks: File contents, allocated on demand from the free pool
Persistence Model
- File data is written directly to the backing file on every
Write()call. No buffering, no data loss window. - Metadata (directory structure, file sizes, block allocation) is persisted when you call
Sync()orClose(). After a crash, reopening recovers the last synced state.
// Data is immediately on disk after Write
f, _ := fs.Create("/important.dat")
f.Write(criticalData)
f.Close()
// Persist the metadata so the file survives a restart
fs.Sync()
Space Guarantees
When filefs creates the backing file, it pre-allocates the full size on disk. This means:
- The OS reserves the disk blocks immediately
- Other processes filling the disk cannot reclaim this space
- filefs operations that need more blocks check the internal bitmap, not the host disk
- If the filefs is full, writes return
ENOSPC— but this is independent of host disk state
This is the same principle behind dedicated partitions, loop-mounted filesystem
images, and pre-allocated database files like SQLite in WAL mode with
journal_size_limit.
Thread Safety
filefs is thread-safe. The FileByteStore uses a mutex to protect block
allocation and file I/O. The inode package provides lock-free atomic
timestamps and mutex-protected directory operations. Multiple goroutines
can safely read and write different files concurrently.
For maximum throughput with heavy concurrent workloads, consider wrapping
with lockfs for reader-writer lock granularity at the filesystem level.
Limitations
- Fixed size: The backing file size cannot be changed after creation. Choose a size that accommodates your expected usage with headroom.
- Metadata region: The metadata region is fixed at 1 MB (256 blocks). This is sufficient for tens of thousands of files/directories. Extremely large directory trees may exceed this limit.
- Block size: Fixed at 4096 bytes. Files smaller than one block still consume one full block. Very small files waste space; very large files work fine but require more block map entries.
- No journaling: Metadata is only persisted on
Sync()/Close(). A crash between writes and sync loses metadata changes (file data already on disk is preserved, but the directory tree reverts to the last sync point). - Single-host: The backing file should only be opened by one process at a time. No file locking is enforced between processes.
Related
- absfs - The absfs filesystem interface
- memfs - In-memory filesystem (no persistence)
- osfs - Host OS filesystem wrapper
- lockfs - Thread-safe filesystem wrapper
- inode - Inode and directory tree primitives
License
This project is governed by the MIT License. See LICENSE.
Documentation
¶
Overview ¶
Package filefs provides a fixed-size file-backed filesystem implementation that conforms to the absfs.FileSystem interface.
filefs is designed for critical systems where storage availability must be guaranteed regardless of host disk pressure. It pre-allocates a single file of fixed size on the host filesystem, then manages that space internally using block allocation. Once created, the backing file's size never changes, so even if other processes fill the host disk completely, filefs continues to operate within its pre-allocated space.
This is analogous to how operating systems use disk partitions to isolate critical storage from user activity, but without requiring kernel-level partition management.
Usage ¶
// Create a 64 MB filesystem backed by a single file
fs, err := filefs.New("/var/lib/myapp/storage.fs", 64*1024*1024)
if err != nil {
log.Fatal(err)
}
defer fs.Close()
// Use like any absfs.FileSystem
f, _ := fs.Create("/config/app.json")
f.Write([]byte(`{"key": "value"}`))
f.Close()
// Sync persists metadata (data writes go directly to the backing file)
fs.Sync()
Persistence ¶
File data is written directly to the backing file on every Write call. Metadata (directory tree, file sizes, block mappings) is persisted on Sync() and Close(). After a crash, reopening the file recovers the last synced state.
Thread Safety ¶
FileSystem is safe for concurrent use by multiple goroutines.
Index ¶
- type File
- func (f *File) Close() error
- func (f *File) Name() string
- func (f *File) Read(p []byte) (int, error)
- func (f *File) ReadAt(b []byte, off int64) (int, error)
- func (f *File) ReadDir(n int) ([]fs.DirEntry, error)
- func (f *File) Readdir(n int) ([]os.FileInfo, error)
- func (f *File) Readdirnames(n int) ([]string, error)
- func (f *File) Seek(offset int64, whence int) (int64, error)
- func (f *File) Stat() (os.FileInfo, error)
- func (f *File) Sync() error
- func (f *File) Truncate(size int64) error
- func (f *File) Write(p []byte) (int, error)
- func (f *File) WriteAt(b []byte, off int64) (int, error)
- func (f *File) WriteString(s string) (int, error)
- type FileByteStore
- func (s *FileByteStore) ReadAt(ino uint64, p []byte, off int64) (int, error)
- func (s *FileByteStore) Remove(ino uint64) error
- func (s *FileByteStore) Stat(ino uint64) (int64, error)
- func (s *FileByteStore) Truncate(ino uint64, size int64) error
- func (s *FileByteStore) WriteAt(ino uint64, p []byte, off int64) (int, error)
- type FileSystem
- func (fs *FileSystem) Capacity() int64
- func (fs *FileSystem) Chdir(name string) error
- func (fs *FileSystem) Chmod(name string, mode os.FileMode) error
- func (fs *FileSystem) Chown(name string, uid, gid int) error
- func (fs *FileSystem) Chtimes(name string, atime time.Time, mtime time.Time) error
- func (fs *FileSystem) Close() error
- func (fs *FileSystem) Create(name string) (absfs.File, error)
- func (fs *FileSystem) Free() int64
- func (fs *FileSystem) Getwd() (string, error)
- func (fs *FileSystem) Lchown(name string, uid, gid int) error
- func (fs *FileSystem) Lstat(name string) (os.FileInfo, error)
- func (fs *FileSystem) Mkdir(name string, perm os.FileMode) error
- func (fs *FileSystem) MkdirAll(name string, perm os.FileMode) error
- func (fs *FileSystem) Open(name string) (absfs.File, error)
- func (fs *FileSystem) OpenFile(name string, flag int, perm os.FileMode) (absfs.File, error)
- func (fs *FileSystem) ReadDir(name string) ([]fs.DirEntry, error)
- func (fs *FileSystem) ReadFile(name string) ([]byte, error)
- func (fs *FileSystem) Readlink(name string) (string, error)
- func (fs *FileSystem) Remove(name string) error
- func (fs *FileSystem) RemoveAll(name string) error
- func (fs *FileSystem) Rename(oldpath, newpath string) error
- func (fs *FileSystem) Stat(name string) (os.FileInfo, error)
- func (fs *FileSystem) Sub(dir string) (fs.FS, error)
- func (fs *FileSystem) Symlink(oldname, newname string) error
- func (fs *FileSystem) Sync() error
- func (fs *FileSystem) TempDir() string
- func (fs *FileSystem) Truncate(name string, size int64) error
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type File ¶
type File struct {
// contains filtered or unexported fields
}
File represents an open file in the file-backed filesystem.
type FileByteStore ¶
type FileByteStore struct {
// contains filtered or unexported fields
}
FileByteStore implements inode.ByteStore using a fixed-size backing file.
File data is stored in fixed-size blocks within the backing file. Each inode's data is mapped to an ordered list of block indices. Blocks are allocated from a bitmap-managed free pool.
Thread Safety: All methods are safe for concurrent use. A mutex protects all state mutations and file I/O to ensure consistency.
func (*FileByteStore) Remove ¶
func (s *FileByteStore) Remove(ino uint64) error
Remove deletes all data associated with the inode.
func (*FileByteStore) Stat ¶
func (s *FileByteStore) Stat(ino uint64) (int64, error)
Stat returns the current logical size of the inode's data.
type FileSystem ¶
type FileSystem struct {
Umask os.FileMode
Tempdir string
// contains filtered or unexported fields
}
FileSystem represents a fixed-size file-backed filesystem.
func New ¶
func New(filepath string, size int64) (*FileSystem, error)
New creates or opens a filefs filesystem at the given path.
If the file does not exist, it is created and formatted with the given size. If the file already exists, it is opened and the existing filesystem is loaded; the size parameter is ignored in this case.
The size must be at least 1 MB.
func (*FileSystem) Capacity ¶
func (fs *FileSystem) Capacity() int64
Capacity returns the total data capacity in bytes.
func (*FileSystem) Chdir ¶
func (fs *FileSystem) Chdir(name string) error
func (*FileSystem) Close ¶
func (fs *FileSystem) Close() error
Close syncs metadata and closes the backing file.
func (*FileSystem) Free ¶
func (fs *FileSystem) Free() int64
Free returns the number of free bytes available in the filesystem.
func (*FileSystem) Getwd ¶
func (fs *FileSystem) Getwd() (string, error)
func (*FileSystem) Remove ¶
func (fs *FileSystem) Remove(name string) error
func (*FileSystem) RemoveAll ¶
func (fs *FileSystem) RemoveAll(name string) error
func (*FileSystem) Rename ¶
func (fs *FileSystem) Rename(oldpath, newpath string) error
func (*FileSystem) Symlink ¶
func (fs *FileSystem) Symlink(oldname, newname string) error
func (*FileSystem) Sync ¶
func (fs *FileSystem) Sync() error
Sync persists the filesystem metadata to the backing file. File data is already written on each Write call; Sync saves the directory tree, block mappings, and allocation bitmap.
func (*FileSystem) TempDir ¶
func (fs *FileSystem) TempDir() string
Source Files