vfs

README

Overview

vfs provides a unified interface for working with different storage backends (local OS, Azure, S3, GCS, SFTP, etc.) in a consistent way. Rather than handling each backend independently, you interact with a core set of interfaces that abstract backend details. This allows you to:

• Seamlessly read, write, and manage files across multiple storages.
• Treat remote or cloud-based paths as if they were local (e.g. using standard io.Reader/io.Writer interfaces).
• Simplify environment-agnostic development by keeping most logic backend-agnostic.
• Plug in or mock storage functionality to facilitate testing and modular application design.

By focusing on FileSystem, Location, and File interfaces, you can build reusable flows for file operations without needing to deeply understand each backend's specific APIs. Users can add or swap backends as needed, providing flexibility for hybrid or evolving storage requirements.

Installation

go get -u github.com/c2fo/vfs/v7

Quick Start

package main

import (
    "fmt"
    "strings"
    "io"
    "log"

    "github.com/c2fo/vfs/v7"
    "github.com/c2fo/vfs/v7/vfssimple"
)

func main() {
    // Create local OS file from a URI
    osFile, err := vfssimple.NewFile("file:///tmp/example.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer osFile.Close()

    // Write to the file
    _, err = io.Copy(osFile, strings.NewReader("Hello from vfs!"))
    if err != nil {
        log.Fatal(err)
    }

    if err := osFile.Close(); err != nil {
        log.Fatal(err)
    }
    fmt.Println("File created and written:", osFile.URI())
}

This snippet shows the basic setup: an osFile is created from a URI and written to using standard libraries.

Advanced Usage

• See the GoDoc for deeper technical details, backend-specific options, and additional APIs.

• Specialized operations like copying between S3 locations or advanced configs (e.g. Azure credentials) are documented in each backend doc.

• vfscp

• vfssimple

• backend

  • os backend
  • gs backend
  • s3 backend
  • in-memory backend
  • sftp backend
  • ftp backend
  • azure backend

• utils

Additional Tools

• lockfile: Provides distributed locking mechanisms using vfs backends.
• vfsevents: Production-ready file event monitoring with support for real-time notifications, retry logic, and multiple watcher types (VFS polling, S3 events, GCS events).

FAQ

Q: Why am I seeing an empty file when using io.Copy on some backends if my source is empty? A: An empty Reader often means the backend doesn't write a file until data is actually written. Use utils.TouchCopy if you need to ensure a zero-byte file is created.

Q: Will vfs v6 still be supported? A: Yes and no. We will continue to provide security patches and bug fixes for v6, but new features and enhancements will be added to v7.

Q: How long will v6 be supported? A: We will support v6 until the next major release, v8, is released. Then v7 will be supported until v9 is released.

Upgrading

Upgrading from v6 to v7

Please review these changes and update your code accordingly to ensure compatibility with v7.

S3 Backend

The project now uses the aws-sdk-go-v2 library instead of the deprecated, EOL aws-sdk-go. This update necessitated these changes to the S3 backend:

• The S3 backend's s3fs.Client() function now returns an s3.Client which is a subset of AWS's sdk v2 functionality. This change may require updates to your code if you were relying on client functionality not directly required by the s3 vfs backend.
• The Option.Retry field is now an aws.Retryer instead of a request.Retry. Ensure that your Option logic is compatible with the new type.

Azure Backend

• Scheme for Azure has been updated from https to az. Update your code to use the new scheme.
• Authority for Azure has been updated from blob.core.windows.net to my-container-name, such that the full URI is az://my-blob-name/path/to/file.txt rather than https://my-storage-account-name.core.windows.net/my-container-name/path/to/file.txt.

GS Backend

• The Options.Retry field, with the now deprecated vfs.Retry type, has been moved to gs.FileSystem as the new gs.Retyer type. It is now set via gs.NewFileSystem function using functional option gs.WithRetryer. All previous vfs.Retry usage has been replaced with gs.WithRetryer. Update your code to use the new gs.WithRetryer functional option.

All Backends

Some methods in the Location and FileSystem interfaces have been deprecated because they use terminology that doesn't apply to all backends. They will be removed in a future release. Update your code to use the new methods. See #235.

• location.Volume() method which returns the authority as a string has been deprecated in favor of the location.Authority() method which returns an authority.Authority struct. Update your code to use the Authority().String() method instead of Volume().
• location.ChangeDir() method ash been deprecated in favor of the existing location.NewLocation() method. Update your code to use the NewLocation() method instead of ChangeDir().
• vfs.Options struct has been deprecated in favor of using backend-specific structs.
• filesystem.Retry() method has been deprecated in favor of using backend-specific functional options, such as gs.WithRetryer.

Additionally, we have added functional option interface, FileSystemOption, to allow for more flexible configuration of backends. This interface allows for more complex configuration options to be passed to the via the NewFileSystem function. This will replace backend-specific chainable functions that require casting the filesystem to the backend type first. See #238.

Upgrading from v5 to v6

With v6, sftp.Options struct changed to accept an array of Key Exchange algorithms rather than a string. To update, change the syntax of the auth commands.

  "keyExchanges":"diffie-hellman-group-a256"

becomes

  "keyExchanges":["diffie-hellman-group-a256"]

Contributing

1. Fork it (git clone https://github.com/c2fo/vfs.git)
2. Create your feature branch (git checkout -b feature/fooBar)
3. Commit your changes (git commit -am "Add some fooBar")
4. Push to the branch (git push origin feature/fooBar)
5. Create a new Pull Request

License

Distributed under the MIT License. See the LICENSE.md file for details.

Documentation

Overview

Package vfs provides a pluggable, extensible, and opinionated set of file system functionality for Go across a number of file system types such as os, S3, and GCS.

Philosophy

When building our platform, initially we wrote a library that was something to the effect of

  if config.DISK == "S3" {
	  // do some s3 file system operation
  } else if config.DISK == "mock" {
      // fake something
  } else {
      // do some native os.xxx operation
  }

Not only was ugly but because the behaviors of each "file system" were different and we had to constantly alter the file locations and pass a bucket string (even if the fs didn't know what a bucket was).

We found a handful of third-party libraries that were interesting but none of them had everything we needed/wanted. Of particular inspiration was https://github.com/spf13/afero in its composition of the super-powerful stdlib io.* interfaces. Unfortunately, it didn't support Google Cloud Storage and there was still a lot of passing around of strings and structs. Few, if any, of the vfs-like libraries provided interfaces to easily and confidently create new file system backends.

What we needed/wanted was the following(and more):

• self-contained set of structs that could be passed around like a file/dir handle
• the struct would represent an existing or nonexistent file/dir
• provide common (and only common) functionality across all file system so that after initialization, we don't care what the underlying file system is and can therefore write our code agnostically/portably
• use io.* interfaces such as io.Reader and io.Writer without needing to call a separate function
• extensibility to easily add other needed file systems like Microsoft Azure Cloud File Storage or SFTP
• prefer native atomic functions when possible (ie S3 to S3 moving would use the native move api call rather than copy-delete)
• a uniform way of addressing files regardless of file system. This is why we use complete URI's in vfssimple
• fmt.Stringer interface so that the file struct passed to a log message (or other Stringer use) would show the URI
• mockable file system
• pluggability so that third-party implementations of our interfaces could be used

Install

Pre 1.17:

go get -u github.com/c2fo/vfs/v7

Post 1.17:

go install -u github.com/c2fo/vfs/v7

Upgrading

Upgrading from v6 to v7

Please review these changes and update your code accordingly to ensure compatibility with v7.

S3 Backend

The project now uses the aws-sdk-go-v2 library instead of the deprecated, EOL aws-sdk-go. This update necessitated a these to the S3 backend:

• The S3 backend's filesystem.Client() function now returns an `s3.Client` which is a subset of AWS's sdk v2 functionality. This change may require updates to your code if you were relying client functionality not directly required by the s3 vfs backend.

• The `Option.Retry` field is now an `aws.Retryer` instead of a `request.Retry`. Ensure that your Option logic is compatible with the new type.

Azure Backend

• Scheme for Azure has been updated from `https` to `az`. Update your code to use the new scheme.

• Authority for Azure has been updated from `blob.core.windows.net` to `<blob-container-name>`, such that the full URI is `az://<blob-container-name>/path/to/file.txt` rather than `https://<storage-account-name>.core.windows.net/<blob-container-name>/path/to/file.txt`.

GS Backend

• The Options.Retry field, with the now deprecated vfs.Retry type, has been moved to gs.FileSystem as the new gs.Retryer type. It is now set via the gs.NewFileSystem() using functional option gs.WithRetryer. All previous Retry usage has been replaced with gs.WithRetryer. Update your code to use the new gs.WithRetryer functional option.

All Backends

Some methods in the Location and FileSystem interfaces have been deprecated because they use terminology that doesn't apply to all backends. They will be removed in a future release. Update your code to use the new methods. See https://github.com/C2FO/vfs/issues/235.

• location.Volume() method which returns the authority as a string has been deprecated in favor of the location.Authority() method which returns an authority.Authority struct. Update your code to use the Authority().String() method instead of Volume().

• location.ChangeDir() method ash been deprecated in favor of the existing location.NewLocation() method. Update your code to use the NewLocation() method instead of ChangeDir().

• vfs.Options struct has been deprecated in favor of using backend-specific structs.

• FileSystem.Retry() method has been deprecated in favor of using backend-specific functional options.

Additionally, we have added functional option interface, `NewFileSystemOption`, to allow for more flexible configuration of backends. This interface allows for more complex configuration options to be passed to the via the `NewFileSystem` function. This will replace backend-specific chainable functions that require casting the filesystem to the backend type first. See https://github.com/C2FO/vfs/issues/238.

Upgrading from v5 to v6

With v6.0.0, sftp.Options struct changed to accept an array of Key Exchange algorithms rather than a string. To update, change the syntax of the auth commands.

"keyExchanges":"diffie-hellman-group-a256"

becomes

"keyExchanges":["diffie-hellman-group-a256"]

Usage

We provide vfssimple as basic way of initializing file system backends (see each implementation's docs about authentication). vfssimple pulls in every c2fo/vfs backend. If you need to reduce the backend requirements (and app memory footprint) or add a third party backend, you'll need to implement your own "factory". See backend doc for more info.

You can then use those file systems to initialize locations which you'll be referencing frequently, or initialize files directly

osFile, err := vfssimple.NewFile("file:///path/to/file.txt")
s3File, err := vfssimple.NewFile("s3://bucket/prefix/file.txt")

osLocation, err := vfssimple.NewLocation("file:///tmp/")
s3Location, err := vfssimple.NewLocation("s3://bucket/")

osTmpFile, err := osLocation.NewFile("anotherFile.txt") // file at /tmp/anotherFile.txt

You can perform a number of actions without any consideration for the underlying system's api or implementation details.

osFileExists, err := osFile.Exists() // true, nil
s3FileExists, err := s3File.Exists() // false, nil
err = osFile.CopyToFile(s3File) // nil
s3FileExists, err = s3File.Exists() // true, nil

movedOsFile, err := osFile.MoveToLocation(osLocation)
osFileExists, err = osFile.Exists() // false, nil (move actions delete the original file)
movedOsFileExists, err := movedOsFile.Exists() // true, nil

s3FileUri := s3File.URI() // s3://bucket/prefix/file.txt
s3FileName := s3File.Name() // file.txt
s3FilePath := s3File.Path() // /prefix/file.txt

File's io.* interfaces may be used directly:

reader := strings.NewReader("Clear is better than clever")
gsFile, err := vfssimple.NewFile("gs://somebucket/path/to/file.txt")

byteCount, err := io.Copy(gsFile, reader)
err := gsFile.Close()

Third-party Backends

* none so far

Feel free to send a pull request if you want to add your backend to the list.

# Ideas See https://github.com/C2FO/vfs/discussions

• Provide better List() functionality with more abstracted filtering and paging (iterator?) Return File structs vs URIs?
• Add better/any context.Context() support
• Standardize errors (sentinel errors?) and utilize error wrapping
• Implement (functional options)[https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis] vs empty interface vfs.Options

Contributing

1. Fork it (<https://github.com/c2fo/vfs/fork>)
2. Create your feature branch (`git checkout -b feature/fooBar`)
3. Commit your changes (`git commit -am 'Add some fooBar'`)
4. Push to the branch (`git push origin feature/fooBar`)
5. Create a new Pull Request

License

Distributed under the MIT license. See `http://github.com/c2fo/vfs/License.md for more information.

Definitions

* absolute path - A path is said to be absolute if it provides the entire context need to find a file, including the file system root. An absolute path must begin with a slash and may include . and .. directories.

* file path - A file path ends with a filename and therefore may not end with a slash. It may be relative or absolute.

* location path - A location/dir path must end with a slash. It may be relative or absolute.

* relative path - A relative path is a way to locate a dir or file relative to another directory. A relative path may not begin with a slash but may include . and .. directories.

* URI - A Uniform Resource Identifier (URI) is a string of characters that unambiguously identifies a particular resource. To guarantee uniformity, all URIs follow a predefined set of syntax rules, but also maintain extensibility through a separately defined hierarchical naming scheme (e.g. http://).

* authority - The authority section of a URI is used to specify the authentication information for the server or service being accessed. It is separated from the rest of the URI by a double slash (//). The authority section is further broken down into userinfo, host, and port.

* userinfo - The userinfo section may contain a username and password separated by a colon. The username and password are separated by a colon and followed by an @ symbol. The password may be omitted.

Index

• Constants
• type Error
  • func (e Error) Error() string
• type File
• type FileSystem
• type Location
• type Options
• type Retry
  • func DefaultRetryer() Retry

Constants

const (
	// CopyToNotPossible - CopyTo/MoveTo operations are only possible when seek position is 0,0
	CopyToNotPossible = Error("current cursor offset is not 0 as required for this operation")

	// ErrNotExist - File does not exist
	ErrNotExist = Error("file does not exist")

	// ErrSeekInvalidOffset - Offset is invalid. Must be greater than or equal to 0
	ErrSeekInvalidOffset = Error("seek: invalid offset")

	// ErrSeekInvalidWhence - Whence is invalid.  Must be one of the following: 0 (io.SeekStart), 1 (io.SeekCurrent), or 2 (io.SeekEnd)
	ErrSeekInvalidWhence = Error("seek: invalid whence")
)

Types

type Error

type Error string

Error is a type that allows for error constants below

func (Error) Error

func (e Error) Error() string

Error returns a string representation of the error

type File

type File interface {
	io.Closer
	io.Reader
	io.Seeker
	io.Writer
	fmt.Stringer

	// Exists returns boolean if the file exists on the file system.  Returns an error, if any.
	Exists() (bool, error)

	// Location returns the vfs.Location for the File.
	Location() Location

	// CopyToLocation will copy the current file to the provided location.
	//
	//   * Upon success, a vfs.File, representing the file at the new location, will be returned.
	//   * In the case of an error, nil is returned for the file.
	//   * CopyToLocation should use native functions when possible within the same scheme.
	//   * If the file already exists at the location, the contents will be overwritten with the current file's contents.
	//   * Unless Seek position is at 0,0 a vfs.CopyToNotPossible will be returned
	//   * CopyToLocation will Close both the source and target Files which therefore can't be appended to without first
	//     calling Seek() to move the cursor to the end of the file.
	CopyToLocation(location Location) (File, error)

	// CopyToFile will copy the current file to the provided file instance.
	//
	//   * In the case of an error, nil is returned for the file.
	//   * CopyToLocation should use native functions when possible within the same scheme.
	//   * If the file already exists, the contents will be overwritten with the current file's contents.
	//   * Unless Seek position is at 0,0 a vfs.CopyToNotPossible will be returned
	//   * CopyToFile will Close both the source and target Files which therefore can't be appended to without first
	//     calling Seek() to move the cursor to the end of the file.
	CopyToFile(file File) error

	// MoveToLocation will move the current file to the provided location.
	//
	//   * If the file already exists at the location, the contents will be overwritten with the current file's contents.
	//   * If the location does not exist, an attempt will be made to create it.
	//   * Upon success, a vfs.File, representing the file at the new location, will be returned.
	//   * In the case of an error, nil is returned for the file.
	//   * When moving within the same Scheme, native move/rename should be used where possible.
	//   * If the file already exists, the contents will be overwritten with the current file's contents.
	//   * Unless Seek position is at 0,0 a vfs.CopyToNotPossible will be returned
	//   * MoveToLocation will Close both the source and target Files which therefore can't be appended to without first
	//     calling Seek() to move the cursor to the end of the file.
	MoveToLocation(location Location) (File, error)

	// MoveToFile will move the current file to the provided file instance.
	//
	//   * If the file already exists, the contents will be overwritten with the current file's contents.
	//   * Unless Seek position is at 0,0 a vfs.CopyToNotPossible will be returned
	//   * The current instance of the file will be removed.
	//   * MoveToFile will Close both the source and target Files which therefore can't be appended to without first
	//     calling Seek() to move the cursor to the end of the file.
	MoveToFile(file File) error

	// Delete unlinks the File on the file system.
	Delete(opts ...options.DeleteOption) error

	// LastModified returns the timestamp the file was last modified (as *time.Time).
	LastModified() (*time.Time, error)

	// Size returns the size of the file in bytes.
	Size() (uint64, error)

	// Path returns absolute path, including filename, ie /some/path/to/file.txt
	//
	// If the directory portion of a file is desired, call
	//   someFile.Location().Path()
	Path() string

	// Name returns the base name of the file path.
	//
	// For file:///some/path/to/file.txt, it would return file.txt
	Name() string

	// Touch creates a zero-length file on the vfs.File if no File exists.  Update File's last modified timestamp.
	// Returns error if unable to touch File.
	Touch() error

	// URI returns the fully qualified absolute URI for the File.  IE, s3://bucket/some/path/to/file.txt
	URI() string
}

File represents a file on a file system. A File may or may not actually exist on the file system.

type FileSystem

type FileSystem interface {
	// NewFile initializes a File on the specified URI authority at path 'absFilePath'.
	//
	//   * Accepts authority and an absolute file path.
	//   * Upon success, a vfs.File, representing the file's new path (location path + file relative path), will be returned.
	//   * On error, nil is returned for the file.
	//   * Note that not all file systems will have an "authority" and will therefore be "":
	//       file:///path/to/file has an authority of "" and name /path/to/file
	//     whereas
	//       s3://mybucket/path/to/file has an authority of "mybucket and name /path/to/file
	//     results in /tmp/dir1/newerdir/file.txt for the final vfs.File path.
	//   * The file may or may not already exist.
	NewFile(authority string, absFilePath string, opts ...options.NewFileOption) (File, error)

	// NewLocation initializes a Location on the specified URI authority with the given path.
	//
	//   * Accepts authority and an absolute location path.
	//   * The file may or may not already exist. Note that on key-store file systems like S3 or GCS, paths never truly exist.
	//   * On error, nil is returned for the location.
	//
	// See NewFile for note on authority.
	NewLocation(authority string, absLocPath string) (Location, error)

	// Name returns the name of the FileSystem ie: Amazon S3, os, Google Cloud Storage, etc.
	Name() string

	// Scheme returns the uri scheme used by the FileSystem: s3, file, gs, etc.
	Scheme() string

	// Retry will return the retry function to be used by any file system.
	//
	// Deprecated: This method is deprecated and will be removed in a future release.
	Retry() Retry
}

FileSystem represents a file system with any authentication accounted for.

type Location

type Location interface {
	// String returns the fully qualified absolute URI for the Location.  IE, file://bucket/some/path/
	fmt.Stringer

	// List returns a slice of strings representing the base names of the files found at the Location.
	//
	//   * All implementations are expected to return ([]string{}, nil) in the case of a non-existent directory/prefix/location.
	//   * If the user cares about the distinction between an empty location and a non-existent one, Location.Exists() should
	//     be checked first.
	List() ([]string, error)

	// ListByPrefix returns a slice of strings representing the base names of the files found in Location whose filenames
	// match the given prefix.
	//
	//   * All implementations are expected to return ([]string{}, nil) in the case of a non-existent directory/prefix/location.
	//   * "relative" prefixes are allowed, ie, listByPrefix from "/some/path/" with prefix "to/somepattern" is the same as
	//     location "/some/path/to/" with prefix of "somepattern"
	//   * If the user cares about the distinction between an empty location and a non-existent one, Location.Exists() should
	//     be checked first.
	ListByPrefix(prefix string) ([]string, error)

	// ListByRegex returns a slice of strings representing the base names of the files found in Location that matched the
	// given regular expression.
	//
	//   * All implementations are expected to return ([]string{}, nil) in the case of a non-existent directory/prefix/location.
	//   * If the user cares about the distinction between an empty location and a non-existent one, Location.Exists() should
	//     be checked first.
	ListByRegex(regex *regexp.Regexp) ([]string, error)

	// Volume returns the volume as string. In URI parlance, volume equates to authority.
	// For example s3://mybucket/path/to/file.txt, volume would return "mybucket".
	//
	// Note: Some file systems may not have a volume and will return "".
	//
	// Deprecated: Use Authority instead.
	//   authStr := loc.Authority().String()
	Volume() string

	// Authority returns the Authority for the Location.
	//
	// For example:
	// 	 sftp//bob@acme.com/path/to/file.txt, Authority.String() would return "bob@acme.com".
	//   s3://my-bucket/path/to/file.txt,     Authority.String() would return "my-bucket".
	//   file://C/path/to/file.txt,           Authority.String() would return "C".
	//   mem://my-namespace/path/to/file.txt, Authority.String() would return "my-namespace".
	Authority() authority.Authority

	// Path returns absolute location path, ie /some/path/to/.  An absolute path must be resolved to its shortest path:
	// see path.Clean
	Path() string

	// Exists returns boolean if the location exists on the file system. Returns an error if any.
	Exists() (bool, error)

	// NewLocation is an initializer for a new Location relative to the existing one.
	//
	// Given location:
	//     loc := fs.NewLocation(:s3://mybucket/some/path/to/")
	// calling:
	//     newLoc, err := loc.NewLocation("../../")
	// would return a new vfs.Location representing:
	//     s3://mybucket/some/
	//
	//   * Accepts a relative location path.
	NewLocation(relLocPath string) (Location, error)

	// ChangeDir updates the existing Location's path to the provided relative location path.
	//
	// Given location:
	// 	   loc := fs.NewLocation("file:///some/path/to/")
	// calling:
	//     loc.ChangeDir("../../")
	// would update the current location instance to
	// file:///some/.
	//
	//   * ChangeDir accepts a relative location path.
	//
	// Deprecated: Use NewLocation instead:
	//     loc, err := loc.NewLocation("../../")
	ChangeDir(relLocPath string) error

	// FileSystem returns the underlying vfs.FileSystem struct for Location.
	FileSystem() FileSystem

	// NewFile will instantiate a vfs.File instance at or relative to the current location's path.
	//
	//   * Accepts a relative file path.
	//   * In the case of an error, nil is returned for the file.
	//   * Resultant File path will be the shortest path name equivalent of combining the Location path and relative path, if any.
	//       ie, /tmp/dir1/ as location and relFilePath "newdir/./../newerdir/file.txt"
	//       results in /tmp/dir1/newerdir/file.txt for the final vfs.File path.
	//   * Upon success, a vfs.File, representing the file's new path (location path + file relative path), will be returned.
	//   * The file may or may not already exist.
	NewFile(relFilePath string, opts ...options.NewFileOption) (File, error)

	// DeleteFile deletes the file of the given name at the location.
	//
	// This is meant to be a short cut for instantiating a new file and calling delete on that, with all the necessary
	// error handling overhead.
	//
	// * Accepts relative file path.
	DeleteFile(relFilePath string, opts ...options.DeleteOption) error

	// URI returns the fully qualified absolute URI for the Location.  IE, s3://bucket/some/path/
	//
	// URI's for locations must always end with a slash.
	URI() string
}

Location represents a file system path which serves as a start point for directory-like functionality. A location may or may not actually exist on the file system.

type Options

type Options interface{}

Options are structs that contain various options specific to the file system

Deprecated: This type is deprecated and will be removed in a future release. Use the specific options struct for the file system with the NewFileSystem function:

fs := s3.NewFileSystem(s3.WithOptions(s3Opts))

type Retry

type Retry func(wrapped func() error) error

Retry is a function that can be used to wrap any operation into a definable retry operation. The wrapped argument is called by the underlying VFS implementation.

Ex:

var retrier Retry = func(wrapped func() error) error {
  var ret error
  for i := 0; i < 5; i++ {
     if err := wrapped(); err != nil { ret = err; continue }
  }
  return ret
}

Deprecated: This type is deprecated and will be removed in a future release.

func DefaultRetryer

func DefaultRetryer() Retry

DefaultRetryer returns a no-op retryer which simply calls the wrapped command without looping.

Deprecated: This function is deprecated and will be removed in a future release.