fsimpl

README

hairyhenderson/go-fsimpl

This module contains a collection of Go filesystem implementations that can be discovered dynamically by URL scheme.

These filesystems implement the fs.FS interface introduced in Go 1.16. This means that currently all implementations are read-only, however this may change in the future (see golang/go#45757 for progress).

Most implementations implement the fs.ReadDirFS interface, though the httpfs filesystem does not.

Some extensions are available to help add specific functionality to certain filesystems:

• WithContextFS - injects a context into a filesystem, for propagating cancellation in filesystems that support it.
• WithHeaderFS - sets the http.Header for all HTTP requests used by the filesystem. This can be useful for authentication, or for requesting specific content types.
• WithHTTPClientFS - sets the *http.Client for all HTTP requests to be made with.

Many of the filesystem packages also have their own extensions.

This module also provides ContentType, an extension to the fs.FileInfo type to help identify an appropriate MIME content type for a given file. For filesystems that support it, the HTTP Content-Type header is used for this. Otherwise, the type is guessed from the file extension.

History & Project Status

This module is in active development, and the API is still subject to breaking changes.

The filesystem packages should operate correctly, based on the tests, but there may be edge cases that are not covered. Please open an issue if you find one!

Most of these filesystems are based on code from gomplate, which supports all of these as datasources. This module is intended to eventually be used within gomplate.

Supported Filesystems

Here's the list of filesystems & URL schemes supported by this module:

Package	Scheme(s)	Description
awsimdsfs	aws+imds	AWS IMDS
awssmfs	aws+sm	AWS Secrets Manager
awssmpfs	aws+smp	AWS Systems Manager Parameter Store
blobfs	azblob	Azure Blob Storage
blobfs	gs	Google Cloud Storage
blobfs	s3	Amazon S3
consulfs	consul, consul+http, consul+https	HashiCorp Consul
filefs	file	local filesystem
gitfs	git, git+file, git+http, git+https, git+ssh	local/remote git repository
httpfs	http, https	HTTP server
tracefs	n/a	a filesystem that instruments other filesystems for tracing with OpenTelemetry
vaultfs	vault, vault+http, vault+https	HashiCorp Vault

See the individual package documentation for more details.

Installation

Use go get to install the latest version of go-fsimpl:

$ go get -u github.com/hairyhenderson/go-fsimpl

Usage

If you know that you want an HTTP filesystem, for example:

import (
	"net/url"

	"github.com/hairyhenderson/go-fsimpl/httpfs"
)

func main() {
	base, _ := url.Parse("https://example.com")
	fsys, _ := httpfs.New(base)

	f, _ := fsys.Open("hello.txt")
	defer f.Close()

	// now do what you like with the file...
}

If you're not sure what filesystem type you need (for example, if you're dealing with a user-provided URL), you can use a filesystem mux:

import (
	"github.com/hairyhenderson/go-fsimpl"
	"github.com/hairyhenderson/go-fsimpl/blobfs"
	"github.com/hairyhenderson/go-fsimpl/filefs"
	"github.com/hairyhenderson/go-fsimpl/gitfs"
	"github.com/hairyhenderson/go-fsimpl/httpfs"
)

func main() {
	mux := fsimpl.NewMux()
	mux.Add(filefs.FS)
	mux.Add(httpfs.FS)
	mux.Add(blobfs.FS)
	mux.Add(gitfs.FS)

	// for example, a URL that points to a subdirectory at a specific tag in a
	// given git repo, hosted on GitHub and authenticated with SSH...
	fsys, err := mux.Lookup("git+ssh://git@github.com/foo/bar.git//baz#refs/tags/v1.0.0")
	if err != nil {
		log.Fatal(err)
	}

	f, _ := fsys.Open("hello.txt")
	defer f.Close()

	// now do what you like with the file...
}

Developing

You will require git including git daemon and consul executables on your path for running the tests.

License

The MIT License

Copyright (c) 2021-2024 Dave Henderson

Documentation

Overview

Package fsimpl contains read-only fs.FS implementations for various different local and remote back-ends, along with a lookup function that allows for mapping URL schemes to matching filesystems.

Index

• func ContentType(fi fs.FileInfo) string
• func WithContextFS(ctx context.Context, fsys fs.FS) fs.FS
• func WithHTTPClientFS(client *http.Client, fsys fs.FS) fs.FS
• func WithHeaderFS(headers http.Header, fsys fs.FS) fs.FS
• type FSMux
  • func NewMux() FSMux
  • func (m FSMux) Add(fs FSProvider)
  • func (m FSMux) Lookup(u string) (fs.FS, error)
  • func (m FSMux) New(u *url.URL) (fs.FS, error)
  • func (m FSMux) Schemes() []string
• type FSProvider
  • func FSProviderFunc(f func(*url.URL) (fs.FS, error), schemes ...string) FSProvider
  • func WrappedFSProvider(fsys fs.FS, schemes ...string) FSProvider

Functions

func ContentType

func ContentType(fi fs.FileInfo) string

ContentType returns the MIME content type for the given io/fs.FileInfo. If fi has a ContentType method, it will be used first, otherwise the filename's extension will be used. See the docs for mime.TypeByExtension for details on how extension lookup works.

The returned value may have parameters (e.g. "application/json; charset=utf-8") which can be parsed with mime.ParseMediaType.

func WithContextFS

func WithContextFS(ctx context.Context, fsys fs.FS) fs.FS

WithContextFS injects a context into the filesystem fs, if the filesystem supports it (i.e. has a WithContext method). This can be used to propagate cancellation.

func WithHTTPClientFS

func WithHTTPClientFS(client *http.Client, fsys fs.FS) fs.FS

WithHTTPClientFS injects an HTTP client into the filesystem fs, if the filesystem supports it (i.e. has a WithHTTPClient method).

func WithHeaderFS

func WithHeaderFS(headers http.Header, fsys fs.FS) fs.FS

WithHeaderFS injects the given HTTP header into the filesystem fs, if the filesystem supports it (i.e. has a WithHeader method).

Types

type FSMux

type FSMux map[string]func(*url.URL) (fs.FS, error)

FSMux allows you to dynamically look up a registered filesystem for a given URL. All filesystems provided in this module can be registered, and additional filesystems can be registered given an implementation of FSProvider. FSMux is itself an FSProvider, which provides the superset of all registered filesystems.

func NewMux

func NewMux() FSMux

NewMux returns an FSMux ready for use.

func (FSMux) Add

func (m FSMux) Add(fs FSProvider)

Add registers the given filesystem provider for its supported URL schemes. If any of its schemes are already registered, they will be overridden.

func (FSMux) Lookup

func (m FSMux) Lookup(u string) (fs.FS, error)

Lookup returns an appropriate filesystem for the given URL. Use Add to register providers.

func (FSMux) New

func (m FSMux) New(u *url.URL) (fs.FS, error)

New - implements FSProvider

func (FSMux) Schemes

func (m FSMux) Schemes() []string

Schemes - implements FSProvider

type FSProvider

type FSProvider interface {
	// Schemes returns the valid URL schemes for this filesystem
	Schemes() []string

	// New returns a filesystem from the given URL
	New(u *url.URL) (fs.FS, error)
}

FSProvider provides a filesystem for a set of defined schemes

func FSProviderFunc

func FSProviderFunc(f func(*url.URL) (fs.FS, error), schemes ...string) FSProvider

FSProviderFunc -

func WrappedFSProvider

func WrappedFSProvider(fsys fs.FS, schemes ...string) FSProvider

WrappedFSProvider is an FSProvider that returns the given fs.FS. When given a URL with a non-root path (i.e. not '/'), fs.Sub will be used to return a filesystem appropriate for the URL.