fs

Documentation

Index

• Constants
• func IsDirOrSymlink(de os.DirEntry) bool
• func IsPartiallyRemovedDir(dirPath string) bool
• func IsPathExist(path string) bool
• func IsTemporaryFileName(fn string) bool
• func MustClose(f *os.File)
• func MustCloseParallel(cs []MustCloser)
• func MustCopyDirectory(srcPath, dstPath string)
• func MustCopyFile(srcPath, dstPath string)
• func MustCreateFlockFile(dir string) *os.File
• func MustFileSize(path string) uint64
• func MustGetFreeSpace(path string) uint64
• func MustGetTotalSpace(path string) uint64
• func MustHardLinkFiles(srcDir, dstDir string)
• func MustMkdirFailIfExist(path string)
• func MustMkdirIfNotExist(path string)
• func MustReadData(r filestream.ReadCloser, data []byte)
• func MustReadDir(dir string) []os.DirEntry
• func MustRemoveDir(dirPath string)
• func MustRemoveDirContents(dir string)
• func MustRemovePath(path string)
• func MustSymlinkRelative(srcPath, dstPath string)
• func MustSyncPath(path string)
• func MustSyncPathAndParentDir(path string)
• func MustWriteAtomic(path string, data []byte, canOverwrite bool)
• func MustWriteData(w filestream.WriteCloser, data []byte)
• func MustWriteSync(path string, data []byte)
• type MustCloser
• type MustReadAtCloser
• type ParallelReaderAtOpener
  • func (pro *ParallelReaderAtOpener) Add(path string, rc *MustReadAtCloser, fileSize *uint64)
  • func (pro *ParallelReaderAtOpener) Run()
• type ReaderAt
  • func MustOpenReaderAt(path string) *ReaderAt
  • func NewReaderAt(f *os.File) *ReaderAt
  • func (r *ReaderAt) MustClose()
  • func (r *ReaderAt) MustFadviseSequentialRead(prefetch bool)
  • func (r *ReaderAt) MustReadAt(p []byte, off int64)
  • func (r *ReaderAt) Path() string
  • func (r *ReaderAt) SetUseLocalStats()

Constants

const FlockFilename = "flock.lock"

FlockFilename is the filename for the file created by MustCreateFlockFile().

Functions

func IsDirOrSymlink

func IsDirOrSymlink(de os.DirEntry) bool

IsDirOrSymlink returns true if de is directory or symlink.

func IsPartiallyRemovedDir

func IsPartiallyRemovedDir(dirPath string) bool

IsPartiallyRemovedDir returns true if dirPath is partially removed because of unclean shutdown during the MustRemoveDir() call.

The caller must call MustRemoveDir(dirPath) on partially removed dirPath.

func IsPathExist

func IsPathExist(path string) bool

IsPathExist returns whether the given path exists.

func IsTemporaryFileName

func IsTemporaryFileName(fn string) bool

IsTemporaryFileName returns true if fn matches temporary file name pattern from MustWriteAtomic.

func MustClose

func MustClose(f *os.File)

MustClose must close the given file f.

func MustCloseParallel

func MustCloseParallel(cs []MustCloser)

MustCloseParallel closes all the cs in parallel.

Parallel closing reduces the time needed to flush the data to the underlying files on close on high-latency storage systems such as NFS or Ceph.

func MustCopyDirectory

func MustCopyDirectory(srcPath, dstPath string)

MustCopyDirectory creates dstPath and copies all the files in srcPath to dstPath.

The caller is responsible for calling MustSyncPath() for the parent directory of dstPath.

func MustCopyFile

func MustCopyFile(srcPath, dstPath string)

MustCopyFile copies the file from srcPath to dstPath.

func MustCreateFlockFile

func MustCreateFlockFile(dir string) *os.File

MustCreateFlockFile creates FlockFilename file in the directory dir and returns the handler to the file.

func MustFileSize

func MustFileSize(path string) uint64

MustFileSize returns file size for the given path.

func MustGetFreeSpace

func MustGetFreeSpace(path string) uint64

MustGetFreeSpace returns free space for the given directory path.

func MustGetTotalSpace

func MustGetTotalSpace(path string) uint64

MustGetTotalSpace returns the total disk space for the given directory path.

func MustHardLinkFiles

func MustHardLinkFiles(srcDir, dstDir string)

MustHardLinkFiles creates dstDir and makes hard links for all the files from srcDir in dstDir.

The caller is responsible for calling MustSyncPath for the parent directory of dstDir.

func MustMkdirFailIfExist

func MustMkdirFailIfExist(path string)

MustMkdirFailIfExist creates the given path dir if it isn't exist.

If the directory at the given path already exists, then the function logs the fatal error and exits the process.

The caller is responsible for MustSyncPath() call for the parent directory for the path.

func MustMkdirIfNotExist

func MustMkdirIfNotExist(path string)

MustMkdirIfNotExist creates the given path dir if it isn't exist.

The caller is responsible for MustSyncPath() call for the parent directory for the path.

func MustReadData

func MustReadData(r filestream.ReadCloser, data []byte)

MustReadData reads len(data) bytes from r.

func MustReadDir

func MustReadDir(dir string) []os.DirEntry

MustReadDir reads directory entries at the given dir.

func MustRemoveDir

func MustRemoveDir(dirPath string)

MustRemoveDir removes the dirPath with all its contents.

The dirPath contents may be partially deleted if unclean shutdown happens during the removal. The caller must verify whether the given directory is partially removed via IsPartiallyRemovedDir() call on the startup before using it. If the directory is partially removed, it must be removed again via MustRemoveDir() call.

func MustRemoveDirContents

func MustRemoveDirContents(dir string)

MustRemoveDirContents removes all the contents of the given dir if it exists.

It doesn't remove the dir itself, so the dir may be mounted to a separate partition.

func MustRemovePath

func MustRemovePath(path string)

MustRemovePath removes the given path. It must be either a file or an empty directory.

Use MustRemoveDir for removing non-empty directories.

func MustSymlinkRelative

func MustSymlinkRelative(srcPath, dstPath string)

MustSymlinkRelative creates relative symlink for srcPath in dstPath.

The caller is responsible for calling MustSyncPath() for the parent directory of dstPath.

func MustSyncPath

func MustSyncPath(path string)

MustSyncPath syncs contents of the given path.

func MustSyncPathAndParentDir

func MustSyncPathAndParentDir(path string)

MustSyncPathAndParentDir fsyncs the path and the parent dir.

This guarantees that the path is visible and readable after unclean shutdown.

func MustWriteAtomic

func MustWriteAtomic(path string, data []byte, canOverwrite bool)

MustWriteAtomic atomically writes data to the given file path.

This function returns only after the file is fully written and synced to the underlying storage.

This function guarantees that the file at path either fully written or not written at all on app crash in the middle of the write.

If the file at path already exists, then the file is overwritten atomically if canOverwrite is true. Otherwise, error is returned.

func MustWriteData

func MustWriteData(w filestream.WriteCloser, data []byte)

MustWriteData writes data to w.

func MustWriteSync

func MustWriteSync(path string, data []byte)

MustWriteSync writes data to the file at path and then calls fsync on the created file.

The fsync guarantees that the written data survives hardware reset after successful call.

This function may leave the file at the path in inconsistent state on app crash in the middle of the write. Use MustWriteAtomic if the file at the path must be either written in full or not written at all on app crash in the middle of the write.

Types

type MustCloser

type MustCloser interface {
	MustClose()
}

MustCloser must implement MustClose() function.

type MustReadAtCloser

type MustReadAtCloser interface {
	// Path must return path for the reader (e.g. file path, url or in-memory reference)
	Path() string

	// MustReadAt must read len(p) bytes from offset off to p.
	MustReadAt(p []byte, off int64)

	// MustClose must close the reader.
	MustClose()
}

MustReadAtCloser is rand-access read interface.

type ParallelReaderAtOpener

type ParallelReaderAtOpener struct {
	// contains filtered or unexported fields
}

ParallelReaderAtOpener opens ReaderAt files in parallel.

ParallelReaderAtOpener speeds up opening multiple ReaderAt files on high-latency storage systems such as NFS or Ceph.

func (*ParallelReaderAtOpener) Add

func (pro *ParallelReaderAtOpener) Add(path string, rc *MustReadAtCloser, fileSize *uint64)

Add adds a task for opening the file at the given path and storing it to *r, while storing the file size into *fileSize.

Call Run() for running all the registered tasks in parallel.

func (*ParallelReaderAtOpener) Run

func (pro *ParallelReaderAtOpener) Run()

Run executes all the registered tasks in parallel.

type ReaderAt

type ReaderAt struct {
	// contains filtered or unexported fields
}

ReaderAt implements rand-access reader.

func MustOpenReaderAt

func MustOpenReaderAt(path string) *ReaderAt

MustOpenReaderAt opens ReaderAt for reading from the file located at path.

MustClose must be called on the returned ReaderAt when it is no longer needed.

func NewReaderAt

func NewReaderAt(f *os.File) *ReaderAt

NewReaderAt returns ReaderAt for reading from f.

NewReaderAt takes ownership for f, so it shouldn't be closed by the caller.

MustClose must be called on the returned ReaderAt when it is no longer needed.

func (*ReaderAt) MustClose

func (r *ReaderAt) MustClose()

MustClose closes r.

func (*ReaderAt) MustFadviseSequentialRead

func (r *ReaderAt) MustFadviseSequentialRead(prefetch bool)

MustFadviseSequentialRead hints the OS that f is read mostly sequentially.

if prefetch is set, then the OS is hinted to prefetch f data.

func (*ReaderAt) MustReadAt

func (r *ReaderAt) MustReadAt(p []byte, off int64)

MustReadAt reads len(p) bytes at off from r.

func (*ReaderAt) Path

func (r *ReaderAt) Path() string

Path returns path to r.

func (*ReaderAt) SetUseLocalStats

func (r *ReaderAt) SetUseLocalStats()

SetUseLocalStats switches to local stats collection instead of global stats collection.

This function must be called before the first call to MustReadAt().

Collecting local stats may improve performance on systems with big number of CPU cores, since the locally collected stats is pushed to global stats only at MustClose() call instead of pushing it at every MustReadAt call.