filesystem

package
v2.48.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 22, 2023 License: Apache-2.0, BSD-3-Clause, MIT Imports: 8 Imported by: 37

Documentation

Overview

Package filesystem contains an extensible file system abstraction. It allows various kinds of storage systems to be used uniformly, notably through textio.

Registered file systems at minimum implement the Interface abstraction, and can then optionally implement Remover, Renamer, and Copier to support rename operations. Filesystems are only expected to handle their own IO, and not cross file system IO. Should cross file system IO be required, additional utility methods should be added to this package to support them.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Copy added in v2.36.0

func Copy(ctx context.Context, fs Interface, oldpath, newpath string) error

Copy replicates the file at oldpath to newpath. Requires the paths to be on the same filesystem.

If the file system implements Copier, it uses that, otherwise it does so manually.

func Match added in v2.47.0

func Match(pattern, name string) (bool, error)

Match is a platform agnostic version of filepath.Match where \ is treated as / on Windows.

func Read

func Read(ctx context.Context, fs Interface, filename string) ([]byte, error)

Read fully reads the given file from the file system.

func Register

func Register(scheme string, fs func(context.Context) Interface)

Register registers a file system backend under the given scheme. For example, "hdfs" would be registered a HFDS file system and HDFS paths used transparently.

func Rename added in v2.36.0

func Rename(ctx context.Context, fs Interface, oldpath, newpath string) error

Rename moves the file at oldpath to newpath. Requires the paths to be on the same filesystem.

Rename will use Renamer, Remover, and Copier interfaces if implemented. Renamer is tried first, and used if available. Otherwise, Rename requires Remover to be implemented, and calls Copy.

func ValidateScheme

func ValidateScheme(path string)

ValidateScheme panics if the given path's scheme does not have a corresponding file system registered.

func Write

func Write(ctx context.Context, fs Interface, filename string, data []byte) error

Write writes the given content to the file system.

Types

type Copier added in v2.36.0

type Copier interface {
	Copy(ctx context.Context, oldpath, newpath string) error
}

Copier is an interface for copying files in the filesystem.

type Interface

type Interface interface {
	io.Closer

	// List expands a pattern to a list of filenames.
	// Returns nil if there are no matching files.
	List(ctx context.Context, glob string) ([]string, error)

	// OpenRead opens a file for reading.
	OpenRead(ctx context.Context, filename string) (io.ReadCloser, error)
	// OpenWrite opens a file for writing. If the file already exist, it will be
	// overwritten. The returned io.WriteCloser should be closed to commit the write.
	OpenWrite(ctx context.Context, filename string) (io.WriteCloser, error)
	// Size returns the size of a file in bytes.
	Size(ctx context.Context, filename string) (int64, error)
}

Interface is a filesystem abstraction that allows beam io sources and sinks to use various underlying storage systems transparently.

func New

func New(ctx context.Context, path string) (Interface, error)

New returns a new Interface for the given file path's scheme.

type LastModifiedGetter added in v2.48.0

type LastModifiedGetter interface {
	LastModified(ctx context.Context, filename string) (time.Time, error)
}

LastModifiedGetter is an interface for getting the last modified time of a file.

type Remover added in v2.36.0

type Remover interface {
	Remove(ctx context.Context, filename string) error
}

Remover is an interface for removing files from the filesystem. To be considered for promotion to Interface.

type Renamer added in v2.36.0

type Renamer interface {
	Rename(ctx context.Context, oldpath, newpath string) error
}

Renamer is an interface for renaming or moving files in the filesystem.

Directories

Path Synopsis
Package gcs contains a Google Cloud Storage (GCS) implementation of the Beam file system.
Package gcs contains a Google Cloud Storage (GCS) implementation of the Beam file system.
Package local contains a local file implementation of the Beam file system.
Package local contains a local file implementation of the Beam file system.
Package memfs contains a in-memory Beam filesystem.
Package memfs contains a in-memory Beam filesystem.
Package s3 contains an AWS S3 implementation of the Beam file system.
Package s3 contains an AWS S3 implementation of the Beam file system.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL