fsutil

package module
v0.57.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 18, 2025 License: AGPL-3.0 Imports: 22 Imported by: 0

Documentation

Overview

Package fsutil provides various implementations of the fs.FS interface, primarily for working with remote files.

To support URI schemes, the package defines the Protocol interface, which takes a URI and returns an appropriate fs.FS implementation and the path to the file within the filesystem.

To support multiple URI schemes, the package provides the "Mux" protocol, which delegates the URI to the appropriate protocol based on the scheme.

Example: 

mux := NewMux(map[string]ProtoFunc{
	"file":  func(uri *url.URL) (Protocol, error) { return NewFileProto(), nil },
	"http":  func(uri *url.URL) (Protocol, error) { return NewHTTPProto(context.Background()), nil },
	"https": func(uri *url.URL) (Protocol, error) { return NewHTTPProto(context.Background()), nil },
})

// Get the appropriate filesystem and path for the URI.
fs, path, err := mux.FileSystem(errutil.Must(http.Parse("http://example.com/file")))
if err != nil {
	log.Fatal(err)
}

b, err := fs.ReadFile(path)
if err != nil {
	log.Fatal(err)
}

fmt.Println(string(b))

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func IPFSPathResolution 

func IPFSPathResolution(cid string) func(f *httpFS, name string) (*netURL.URL, error)

func IPFSSubdomainResolution 

func IPFSSubdomainResolution(cid string) func(f *httpFS, name string) (*netURL.URL, error)

func NewCacheFS 

func NewCacheFS(fs fs.FS, opts ...CacheFSOption) (fs.FS, error)

NewCacheFS creates a new cache filesystem.

The cache filesystem caches the contents of the files in the cache directory. If the file is not found in the cache, it will be read from the underlying file system and cached.

func NewChainFS 

func NewChainFS(opts ...ChainFSOption) fs.FS

NewChainFS creates a new chain filesystem.

The chain filesystem chains multiple file systems together. It will try to open a file in the first file system. If it fails, it will try the next one, and so on. If all file systems fail, it will return an error.

func NewChecksumFS 

func NewChecksumFS(fs fs.FS, opts ...ChecksumFSOption) (fs.FS, error)

NewChecksumFS creates a new checksum file system.

The file system wraps an existing file system, computes the checksum of the file contents, and compares it with the provided checksum. The checksum must be provided in the file name as a query parameter, e.g., "file?checksum=0x1234...".

If the checksum does not match, the file system returns an error when reading the file.

func NewGzipFS 

func NewGzipFS(fs fs.FS, opts ...GzipFSOption) fs.FS

NewGzipFS creates a new gzip filesystem.

The gzip filesystem will wrap the given filesystem and add gzip decompression functionality.

func NewHTTPFS 

func NewHTTPFS(ctx context.Context, baseURI *netURL.URL, opts ...HTTPFSOption) (fs.FS, error)

NewHTTPFS creates a new HTTP file system.

func NewIPFSFS 

func NewIPFSFS(ctx context.Context, cid string, opts ...IPFSOption) (fs.FS, error)

NewIPFSFS creates a new IPFS filesystem.

The IPFS filesystem uses IPFS gateways to resolve IPFS paths. To verify the integrity of the file contents and ensure that returned data is valid, an optional checksum hash can be provided as a "checksum" parameter in the URL.

It is important to provide a checksum, as there is no guarantee that the data returned from IPFS gateways is valid. A misconfigured or malicious gateway could return a different or corrupted file.

func NewRetryFS 

func NewRetryFS(ctx context.Context, fs fs.FS, attempts int, delay time.Duration) fs.FS

NewRetryFS wraps the given FS to add retry functionality.

func ParseURI 

func ParseURI(p Protocol, uri string) (fs.FS, string, error)

ParseURI is a helper function that parses a URI for a given protocol and returns the appropriate filesystem and path.

As a special case, if the URI does not contain a scheme, it is assumed to be a file URI and is prefixed with "file:///".

Types

type CacheFSOption 

type CacheFSOption func(*cacheFS)

func WithCacheDir 

func WithCacheDir(dir string) CacheFSOption

WithCacheDir sets the cache directory.

func WithCacheNamespace added in v0.57.1 

func WithCacheNamespace(ns string) CacheFSOption

WithCacheNamespace sets the cache namespace, that is added to the hash of the file name to create a unique cache file name.

type ChainFSOption 

type ChainFSOption func(*chainFS)

func WithChainFilesystems 

func WithChainFilesystems(fs ...fs.FS) ChainFSOption

WithChainFilesystems sets the file systems to chain.

func WithChainRandOrder 

func WithChainRandOrder() ChainFSOption

WithChainRandOrder sets the file systems to chain in random order.

type ChecksumFSOption 

type ChecksumFSOption func(*checksumFS)

func WithChecksumHash 

func WithChecksumHash(hash func() hash.Hash) ChecksumFSOption

WithChecksumHash sets the hash function used to compute the checksum. The default hash function is LegacyKeccak256.

func WithChecksumParamName 

func WithChecksumParamName(name string) ChecksumFSOption

WithChecksumParamName sets the name of the URL query parameter that contains the checksum value. The default parameter name is "checksum".

func WithChecksumVerifyMode 

func WithChecksumVerifyMode(mode ChecksumFSVerifyMode) ChecksumFSOption

WithChecksumVerifyMode sets the mode of the checksum verification. The default mode is ChecksumFSVerifyAfterRead.

type ChecksumFSVerifyMode 

type ChecksumFSVerifyMode int
const (
	// ChecksumFSVerifyAfterRead verifies the checksum after reading the file
	// contents.
	ChecksumFSVerifyAfterRead ChecksumFSVerifyMode = iota

	// ChecksumFSVerifyAfterOpen verifies the checksum immediately after
	// opening the file.
	ChecksumFSVerifyAfterOpen
)

type FileOption 

type FileOption func(*fileProto)

func WithFileWorkingDir 

func WithFileWorkingDir(wd string) FileOption

WithFileWorkingDir sets the working directory for the file protocol.

type GzipFSOption 

type GzipFSOption func(*gzipFS)

func WithGzipCheckExtension 

func WithGzipCheckExtension(check bool) GzipFSOption

WithGzipCheckExtension enables or disables checking the file extension to determine whether to decompress the file. If enabled, only files with the specified extensions will be decompressed.

func WithGzipExtensions 

func WithGzipExtensions(exts ...string) GzipFSOption

WithGzipExtensions sets the list of file extensions that will be decompressed. The default extension is "gz". Ignored if WithGzipCheckExtension is set to false.

func WithGzipReadLimit 

func WithGzipReadLimit(limit int64) GzipFSOption

WithGzipReadLimit sets the maximum size of the decompressed data. If the decompressed data exceeds the limit, the io.ErrUnexpectedEOF error will be returned. The default limit is 128MiB.

type HTTPFSOption 

type HTTPFSOption func(*httpFS)

func WithHTTPClient 

func WithHTTPClient(client *http.Client) HTTPFSOption

WithHTTPClient sets the HTTP client used to perform HTTP requests.

type IPFSGateway 

type IPFSGateway struct {
	Scheme    string
	Host      string
	ResolveFn func(cid string) func(f *httpFS, name string) (*netURL.URL, error)
}

type IPFSOption 

type IPFSOption func(*ipfsFS)

func WithIPFSChecksumHash 

func WithIPFSChecksumHash(hash func() hash.Hash) IPFSOption

WithIPFSChecksumHash sets the hash function used to compute the checksum.

func WithIPFSGateways 

func WithIPFSGateways(gateways ...*IPFSGateway) IPFSOption

WithIPFSGateways sets the IPFS gateways used to resolve IPFS paths.

func WithIPFSHTTPClient 

func WithIPFSHTTPClient(client *http.Client) IPFSOption

WithIPFSHTTPClient sets the HTTP client used to perform HTTP requests.

type ProtoFunc 

type ProtoFunc func(*netURL.URL) (Protocol, error)

ProtoFunc is a function that creates a Protocol from a URL.

type Protocol 

type Protocol interface {
	FileSystem(uri *netURL.URL) (fs fs.FS, path string, err error)
}

Protocol defines a file system protocol. It provides a file system instance and a path within that file system for a given URI.

The returned file system always points to the highest possible level directory.

func NewCacheProto 

func NewCacheProto(proto Protocol, opts ...CacheFSOption) Protocol

NewCacheProto creates a new cache protocol.

The cache protocol will wrap the filesystem returned by a given protocol with a cache filesystem.

func NewChainProto 

func NewChainProto(opts ...ChainFSOption) Protocol

NewChainProto creates a new chain protocol.

func NewChecksumProto 

func NewChecksumProto(proto Protocol, opts ...ChecksumFSOption) Protocol

NewChecksumProto creates a new checksum protocol.

func NewFSProto added in v0.57.1 

func NewFSProto(f fs.FS) Protocol

NewFSProto creates a new file system protocol that uses the provided file system.

func NewFileProto 

func NewFileProto(opts ...FileOption) Protocol

NewFileProto creates a new file protocol that uses the local filesystem. The URI scheme must be "file" and the host must be empty or "localhost". The working directory is set to "/" by default.

func NewGzipProto 

func NewGzipProto(proto Protocol, opts ...GzipFSOption) Protocol

NewGzipProto creates a new gzip protocol.

The gzip protocol will wrap the filesystem returned by a given protocol with a gzip filesystem.

func NewHTTPProto 

func NewHTTPProto(ctx context.Context, opts ...HTTPFSOption) Protocol

The HTTP protocol is used to create an HTTP file system.

func NewIPFSProto 

func NewIPFSProto(ctx context.Context, opts ...IPFSOption) Protocol

NewIPFSProto creates a new IPFS protocol.

The IPFS protocol is used to create an IPFS file system.

func NewMux 

func NewMux(ps map[string]ProtoFunc) Protocol

NewMux creates a new protocol multiplexer that routes URIs to registered protocols based on their scheme.

func NewRetryProto 

func NewRetryProto(ctx context.Context, proto Protocol, attempts int, delay time.Duration) Protocol

NewRetryProto creates a new retry protocol.

The retry protocol will wrap the filesystem returned by a given protocol with a retry filesystem.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.