ufs

README

ufs

Unified File System (UFS) is a library that allows go apps to use the fs.FS interface to access file systems across many storage layers.

Features

• Read-only ReadFS: Each file system implements Go's fs.FS interface with some extensions such as Close() error to manage the lifecycle of the access to the file system.
• Read-write FS: Provides read-write support for file systems that allow it.

File Systems

Storage	URI	Implementation	Description
Null	null://	nullfs.go	Acts as /dev/null. Any files written to this location are forgotten. Reads always resolve to empty files. All directories are read with no contents.
Memory	memory://	memfs.go	Stores files with virtual memory of the application. Forgotten when the application stops.
Local	file://	localfs.go	Local files on the hard drive. The URI specifies the mount point of the local file system. Access to files outside of the mounted directory are not allowed. This is backed by Go's os.RootFS.

Commands

# Build
go build ./...

# Test
go test ./...

# Lint
go fmt ./...
go vet ./...

Use ollama with Claude Code

$Env:ANTHROPIC_AUTH_TOKEN="ollama"
$Env:ANTHROPIC_API_KEY=""
$Env:ANTHROPIC_BASE_URL="http://local:11434"

claude --model qwen3.6:27b

export ANTHROPIC_AUTH_TOKEN="ollama"
export ANTHROPIC_API_KEY=""
export ANTHROPIC_BASE_URL="http://mega:11434"

claude --model qwen3.6:27b

Documentation

Overview

Package ufs provides a unified virtual file system abstraction for Go. It allows applications to treat diverse storage backends — local disk, memory, archives, Google Cloud Storage, remote Git repositories, and more — through a single consistent interface.

Creating a file system

Use New with a URI to open a file system:

fsys, err := ufs.New("memory://")       // in-memory, volatile
fsys, err := ufs.New("null://")         // /dev/null semantics
fsys, err := ufs.New("/path/to/dir")    // local directory
fsys, err := ufs.New("file:///abs/dir") // same, explicit scheme
fsys, err := ufs.New("gs://bucket/prefix") // Google Cloud Storage
fsys, err := ufs.New("https://host/file.zip") // remote archive (downloaded to temp dir)

Every FS must be closed when no longer needed.

Nested mounts

CreateURI builds a URI that layers multiple file systems at different paths. Pass the resulting URI to New:

uri, _ := ufs.CreateURI("file:///data", map[string]string{
    "cache": "memory://",
})
fsys, _ := ufs.New(uri)

Path conventions

All path arguments follow fs.ValidPath: forward-slash separated, no leading slash, no "." or ".." components. The root directory is always ".".

Index

• func Copy(srcFS fs.FS, srcFilename string, destFS FS, destFilename string) error
• func CreateURI(name string, nested map[string]string) (string, error)
• func ForEachFileInfo(fsys fs.FS, dir string, f func(fs.FileInfo) error) error
• func ForEachFilename(fsys fs.FS, dir string, f func(string) error) error
• func List(fsys fs.FS, dir string) ([]string, error)
• func ListFiles(fsys fs.FS, dir string) ([]string, error)
• func Rsync(srcFS fs.FS, destFS FS, dir string) error
• type FS
  • func New(ctx context.Context, name string) (FS, error)
• type File
• type FileInfo
• type ForEachFileInfoIter
• type ForEachFilenameIter
• type ListFilenames
• type ReadFS
• type ReadFile

Examples

• Copy
• CreateURI
• ForEachFilename
• List
• ListFiles
• New (Memory)
• New (Null)
• Rsync

Functions

func Copy

func Copy(srcFS fs.FS, srcFilename string, destFS FS, destFilename string) error

Copy copies the single file at srcFilename in srcFS to destFilename in destFS. The parent directory of destFilename must already exist. The destination file is created (or truncated) via FS.Create.

Example

ExampleCopy shows copying a single file between two file systems.

ctx := context.Background()
src, _ := New(ctx, "memory://")
dst, _ := New(ctx, "memory://")
defer src.Close()
defer dst.Close()

f, _ := src.Create("hello.txt")
f.WriteString("hello")
f.Close()

if err := Copy(src, "hello.txt", dst, "copy.txt"); err != nil {
	log.Fatal(err)
}

data, _ := dst.ReadFile("copy.txt")
fmt.Println(string(data))

Output:
hello

func CreateURI

func CreateURI(name string, nested map[string]string) (string, error)

CreateURI constructs a URI understood by New that layers additional file systems at specific mount paths inside the base file system. The nested map maps mount-point paths (e.g. "cache", "data/scratch") to the URI of the file system to mount there. Paths follow fs.ValidPath conventions.

Pass the returned URI directly to New:

 ctx := context.Background()
	uri, _ := ufs.CreateURI("file:///srv/data", map[string]string{
	    "tmp": "memory://",
	})
	fsys, _ := ufs.New(ctx, uri)

Returns an error if name or any nested URI cannot be parsed.

Example

ExampleCreateURI shows building a URI for a file system with a nested mount.

ctx := context.Background()
// A memory FS with no nested mounts.
uri, err := CreateURI("memory://", nil)
if err != nil {
	log.Fatal(err)
}
fmt.Println(uri)

// Open it — New accepts URIs produced by CreateURI.
fsys, err := New(ctx, uri)
if err != nil {
	log.Fatal(err)
}
defer fsys.Close()
fmt.Println(fsys.String())

Output:
memory:
nestFS(memory://)

func ForEachFileInfo

func ForEachFileInfo(fsys fs.FS, dir string, f func(fs.FileInfo) error) error

ForEachFileInfo calls f for each file (not directory) under dir, providing its fs.FileInfo. It is the typed companion to ForEachFilename and prefers a native ForEachFileInfoIter implementation when available, falling back to fs.WalkDir. The walk stops and returns the first non-nil error from f.

func ForEachFilename

func ForEachFilename(fsys fs.FS, dir string, f func(string) error) error

ForEachFilename calls f for each file path (not directory) under dir, streaming results without building an intermediate slice. If fsys implements ForEachFilenameIter, its native implementation is used directly; otherwise the paths are collected via ListFiles and iterated. f receives paths relative to dir. The walk stops and returns the first non-nil error from f.

Example

ExampleForEachFilename shows streaming file names without building a slice, which saves memory for large trees.

ctx := context.Background()
fsys, _ := New(ctx, "memory://")
defer fsys.Close()

for _, name := range []string{"a.txt", "b.txt"} {
	f, _ := fsys.Create(name)
	f.Close()
}

ForEachFilename(fsys, ".", func(name string) error {
	fmt.Println(name)
	return nil
})

Output:
a.txt
b.txt

func List

func List(fsys fs.FS, dir string) ([]string, error)

List returns all paths (both files and directories) under dir in lexical order. The root directory "." is never included in the result. For files-only, prefer ListFiles.

Example

ExampleList shows listing all entries including directories.

ctx := context.Background()
fsys, _ := New(ctx, "memory://")
defer fsys.Close()

fsys.MkdirAll("subdir", fs.ModePerm)
f, _ := fsys.Create("subdir/c.txt")
f.Close()

entries, _ := List(fsys, ".")
for _, p := range entries {
	fmt.Println(p)
}

Output:
subdir
subdir/c.txt

func ListFiles

func ListFiles(fsys fs.FS, dir string) ([]string, error)

ListFiles returns the paths of all files (excluding directories) under dir in lexical order. If fsys implements ListFilenames, its native implementation is used to avoid building intermediate fs.FileInfo values.

Example

ExampleListFiles shows listing only files (no directories) under a path.

ctx := context.Background()
fsys, _ := New(ctx, "memory://")
defer fsys.Close()

fsys.MkdirAll("subdir", fs.ModePerm)
for _, name := range []string{"a.txt", "b.txt", "subdir/c.txt"} {
	f, _ := fsys.Create(name)
	f.Close()
}

files, _ := ListFiles(fsys, ".")
for _, p := range files {
	fmt.Println(p)
}

Output:
a.txt
b.txt
subdir/c.txt

func Rsync

func Rsync(srcFS fs.FS, destFS FS, dir string) error

Rsync copies all files under dir from srcFS into destFS, preserving the relative path structure. Parent directories in destFS are created with fs.ModePerm as needed. Existing files in destFS are overwritten. The copy is not atomic: if an error occurs mid-walk, destFS may be partially written.

dir must satisfy fs.ValidPath; use "." to copy the entire file system.

Example

ExampleRsync shows recursively mirroring all files from one FS into another.

ctx := context.Background()
src, _ := New(ctx, "memory://")
dst, _ := New(ctx, "memory://")
defer src.Close()
defer dst.Close()

src.MkdirAll("subdir", fs.ModePerm)
for _, name := range []string{"a.txt", "subdir/b.txt"} {
	f, _ := src.Create(name)
	f.WriteString("content")
	f.Close()
}

if err := Rsync(src, dst, "."); err != nil {
	log.Fatal(err)
}

files, _ := ListFiles(dst, ".")
for _, p := range files {
	fmt.Println(p)
}

Output:
a.txt
subdir/b.txt

Types

type FS

type FS interface {
	ReadFS

	// Create opens a new writable file at name, replacing any existing file at
	// that path. Parent directories are not created automatically; call
	// MkdirAll first if needed.
	Create(name string) (File, error)

	// MkdirAll creates the directory at name and any missing parent directories,
	// using perm for newly created nodes. It is a no-op if the directory already
	// exists. Backends that do not have a real directory concept (e.g. GCS) treat
	// this as a no-op.
	MkdirAll(name string, perm fs.FileMode) error

	// String returns a human-readable description of the file system, typically
	// the URI or absolute path that was used to open it.
	String() string
}

FS is a read-write file system. It extends ReadFS with file creation and directory creation.

func New

func New(ctx context.Context, name string) (FS, error)

New opens a file system identified by name. The returned FS wraps the backend in a nestFS layer that automatically mounts archives found inside the tree (see below). Always call Close on the returned FS when done.

URI schemes

• memory:// — volatile in-memory file system; all data is lost when the FS is closed or the process exits. Safe for concurrent use.
• null:// — /dev/null semantics: Create and MkdirAll always succeed, writes are accepted but discarded, reads return empty content, Stat reports everything as a directory. Useful in tests.
• angry:// — always returns fs.ErrInvalid; used to exercise error-handling paths in tests.
• file://path or a bare path — local directory, mounted read-write via os.OpenRoot (Go 1.24+). Access outside the mount root is rejected by the OS. On Windows, directory Stat always reports size 0 (unlike the raw os package which may report 4096).
• gs://bucket/prefix — Google Cloud Storage bucket, optionally scoped to a prefix. Credentials are resolved via ADC; unauthenticated access is tried as a fallback.
• https:// or http:// URL ending in a recognised archive extension — the archive is downloaded to a temporary directory, mounted read-only, and the temporary directory is removed when Close is called.
• A path ending in .git — the repository is shallow-cloned into a temporary directory (not available on AIX).
• A local path pointing to a recognised archive (.zip, .tar, .tar.gz, etc.) is mounted read-only through the archive's contents.

Nested mounts and archive auto-mounting

The returned FS wraps all backends in a nestFS layer. When a directory entry named foo.zip (or any recognised archive extension) exists, the virtual path foo.zip.d is automatically exposed as a read-only mount of that archive's contents. No explicit configuration is required.

Use CreateURI to pre-configure additional mount points before calling New.

Example (Memory)

ExampleNew_memory demonstrates a volatile in-memory file system. All data is lost when the FS is closed or the process exits.

ctx := context.Background()
fsys, err := New(ctx, "memory://")
if err != nil {
	log.Fatal(err)
}
defer fsys.Close()

f, err := fsys.Create("hello.txt")
if err != nil {
	log.Fatal(err)
}
f.WriteString("hello, world")
f.Close()

data, err := fsys.ReadFile("hello.txt")
if err != nil {
	log.Fatal(err)
}
fmt.Println(string(data))

Output:
hello, world

Example (Null)

ExampleNew_null demonstrates the null file system. It accepts all writes and Create calls without error, but data is immediately discarded. Reads always return empty content. Useful as a write sink in tests.

ctx := context.Background()
fsys, err := New(ctx, "null://")
if err != nil {
	log.Fatal(err)
}
defer fsys.Close()

f, err := fsys.Create("discard.txt")
if err != nil {
	log.Fatal(err)
}
n, writeErr := f.WriteString("this data is discarded")
f.Close()
fmt.Printf("wrote %d bytes, err=%v\n", n, writeErr)

// ReadFile always returns an empty byte slice, not an error.
data, _ := fsys.ReadFile("discard.txt")
fmt.Printf("read %d bytes\n", len(data))

Output:
wrote 22 bytes, err=<nil>
read 0 bytes

type File

type File interface {
	ReadFile
	io.ReadWriteSeeker
	io.ReaderAt
	io.StringWriter
}

File is a read-write file handle. It extends ReadFile with write, seek, and random-read operations. All methods are safe to call at any time after the file is opened; unlike bare fs.File, no method returns a "not supported" error.

type FileInfo

type FileInfo interface {
	fs.FileInfo
}

FileInfo provides file metadata. It currently mirrors fs.FileInfo and is defined as a separate interface to allow future extensions without breaking callers.

type ForEachFileInfoIter

type ForEachFileInfoIter interface {
	// ForEachFileInfo calls f for each file (not directory) under dir. If f
	// returns a non-nil error the walk stops and that error is returned.
	ForEachFileInfo(dir string, f func(fs.FileInfo) error) error
}

ForEachFileInfoIter is an optional interface for streaming fs.FileInfo values without building a full slice. ForEachFileInfo uses this when the file system implements it.

type ForEachFilenameIter

type ForEachFilenameIter interface {
	// ForEachFilename calls f for each file path (not directory) under dir. If f
	// returns a non-nil error the walk stops and that error is returned.
	ForEachFilename(dir string, f func(string) error) error
}

ForEachFilenameIter is an optional interface for streaming file names without building a full slice. ForEachFilename uses this when the file system implements it.

type ListFilenames

type ListFilenames interface {
	// ListFilenames returns the paths of all files (not directories) under dir,
	// in unspecified order, with reduced allocations.
	ListFilenames(string) ([]string, error)
}

ListFilenames is an optional interface that a file system may implement to return all file paths under a directory without building an intermediate fs.FileInfo slice, reducing memory usage for large trees. ListFiles will use this interface when available.

type ReadFS

type ReadFS interface {
	fs.FS
	io.Closer
	fs.ReadDirFS
	fs.ReadFileFS
	fs.ReadLinkFS
	fs.StatFS
}

ReadFS is a read-only file system. In addition to the standard fs.FS interface it requires io.Closer for lifecycle management and the four extended read interfaces from the standard library. Callers should always Close a ReadFS when they are done with it.

type ReadFile

type ReadFile interface {
	fs.File
}

ReadFile is a read-only file handle. It satisfies fs.File.