fileop

README

fileop

Collection of file operation helpers.

Use Cases

This repository contains interface abstraction implementations for several commonly used file systems. It makes it easy to quickly adapt various file system operations in your business code.

Basic Interfaces

interface.go

Read-only Operations

type Reader interface {
	Open(name string) (io.ReadCloser, error)
}

type DirReader interface {
	Readdir(dirname string, n int) ([]fs.FileInfo, error)
	Readdirnames(dirname string, n int) ([]string, error)
}

type Stater interface {
	Stat(name string) (fs.FileInfo, error)
}

type Walker interface {
	Walk(root string, walkFn filepath.WalkFunc) error
}

Create/Modify Operations

type Writer interface {
	DirCreator
	Create(name string) (io.WriteCloser, error)
	Rename(oldPath, newPath string) error
}

type DirCreator interface {
	Mkdir(dirname string, perm fs.FileMode) error
	MkdirAll(dirname string, perm fs.FileMode) error
}

type Cleaner interface {
	Remove(name string) error
	RemoveAll(name string) error
}

File system operation interface needs to implement all the above methods

type FileSystem interface {
	Reader
	DirReader
	Writer
	Cleaner
	Stater
	Walker
}

File System

Instantiation

filesystem.New(c Config) (fileop.FileSystemWithCloser, error)

Supported file systems:

• disk (local)
• hdfs (HDFS)

Simple File System

interface_simple.go

Instantiation

simplefs.New(c Config) (fileop.FileSystemSimpleBucket, error) 

Supported file systems:

• disk (local)
• hdfs (HDFS)
• obs (Huawei OBS)
• minio (minio, S3 compatible)

Helper Functions

Compression

Supported compression file types:

• NONE
• GZIP
• ZLIB
• SNAPPY

NewCompressReader(file io.Reader, ct CompressType) (io.ReadCloser, error)
NewCompressWriter(buf io.Writer, ct CompressType) (io.WriteCloser, error)

Global Configuration for CompressWriter

• fileop.UsePGZIP=[true|false]
  • Use "github.com/klauspost/pgzip" library for writing gzip files.
  • Default is "github.com/klauspost/compress/gzip" library.
• fileop.PGZIPBlocks=[number]
  • Set the number of PGZIP blocks.
  • Default is 4.

File

File read/write

• example/file_writer_reader
• example/fileutil_write_read

How to Extend

• Extend a new data source example/extend_filesource

Documentation

Overview

Package fileop provides a collection of file operation helpers with abstraction interfaces for various file systems.

Index

• Variables
• func IsUnhandledFileReaderError(err error) bool
• func NewCompressReader(file io.Reader, ct CompressType) (io.ReadCloser, error)
• func NewCompressWriter(buf io.Writer, ct CompressType) (io.WriteCloser, error)
• type Cleaner
• type CompressType
  • func (ct CompressType) String() string
• type DirCreator
• type DirReader
• type FileReader
  • func NewFileReader(fri FileReaderInterface, srcPath string, ct CompressType) (*FileReader, error)
  • func (fr *FileReader) Close() error
  • func (fr *FileReader) Read(p []byte) (int, error)
  • func (fr *FileReader) ReadLine() ([]byte, error)
• type FileReaderInterface
• type FileSystem
• type FileSystemSimple
• type FileSystemSimpleBucket
• type FileSystemWithCloser
• type FileWriter
  • func NewFileWriter(fwi FileWriterInterface, dstPath string, bufSize int, ct CompressType) (*FileWriter, error)
  • func (fw *FileWriter) Close() error
  • func (fw *FileWriter) Write(p []byte) (int, error)
  • func (fw *FileWriter) WriteLine(line []byte) (int, error)
• type FileWriterInterface
• type ISourceLister
• type ISourceReader
• type ITargetUploader
• type NewReaderFunc
• type NewWriterFunc
• type Reader
• type Stater
• type Walker
• type Writer

Variables

var (
	UsePGZIP    bool
	PGZIPBlocks = 4
)

Functions

func IsUnhandledFileReaderError

func IsUnhandledFileReaderError(err error) bool

func NewCompressReader

func NewCompressReader(file io.Reader, ct CompressType) (io.ReadCloser, error)

func NewCompressWriter

func NewCompressWriter(buf io.Writer, ct CompressType) (io.WriteCloser, error)

Types

type Cleaner

type Cleaner interface {
	Remove(name string) error
	RemoveAll(name string) error
}

Cleaner provides file and directory removal operations.

type CompressType

type CompressType int

CompressType represents the type of compression to be used.

const (
	NONE   CompressType = iota // No compression
	GZIP                       // GZIP compression
	ZLIB                       // ZLIB compression
	SNAPPY                     // SNAPPY compression
)

Supported compression types.

func (CompressType) String

func (ct CompressType) String() string

String returns the string representation of the compression type.

type DirCreator

type DirCreator interface {
	Mkdir(dirname string, perm fs.FileMode) error
	MkdirAll(dirname string, perm fs.FileMode) error
}

DirCreator provides directory creation operations.

type DirReader

type DirReader interface {
	Readdir(dirname string, n int) ([]fs.FileInfo, error)
	Readdirnames(dirname string, n int) ([]string, error)
}

DirReader provides directory reading operations.

type FileReader

type FileReader struct {
	Path string

	EOF bool
	// contains filtered or unexported fields
}

func NewFileReader

func NewFileReader(fri FileReaderInterface, srcPath string, ct CompressType) (*FileReader, error)

NewFileReader create *FileReader on any FileReaderInterface. NOTE: you can call IsUnhandledFileReaderError judge errors can not solve by retry.

func (*FileReader) Close

func (fr *FileReader) Close() error

func (*FileReader) Read

func (fr *FileReader) Read(p []byte) (int, error)

func (*FileReader) ReadLine

func (fr *FileReader) ReadLine() ([]byte, error)

type FileReaderInterface

type FileReaderInterface Reader

FileReaderInterface is an alias for Reader interface.

type FileSystem

type FileSystem interface {
	Reader
	DirReader
	Writer
	Cleaner
	Stater
	Walker
}

FileSystem combines all file system operations.

type FileSystemSimple

type FileSystemSimple interface {
	ISourceReader
	ITargetUploader
}

FileSystemSimple combines basic source reader and target uploader interfaces.

type FileSystemSimpleBucket

type FileSystemSimpleBucket interface {
	FileSystemSimple
	Bucket(name string) FileSystemSimpleBucket
	PutStreamWithContentType(reader io.Reader, remote string, contentType string) error
}

FileSystemSimpleBucket extends FileSystemSimple with bucket operations and content type support for object storage systems.

type FileSystemWithCloser

type FileSystemWithCloser interface {
	FileSystem
	io.Closer
}

FileSystemWithCloser extends FileSystem with a Close method.

type FileWriter

type FileWriter struct {
	Path string
	// contains filtered or unexported fields
}

func NewFileWriter

func NewFileWriter(fwi FileWriterInterface, dstPath string, bufSize int, ct CompressType) (*FileWriter, error)

func (*FileWriter) Close

func (fw *FileWriter) Close() error

func (*FileWriter) Write

func (fw *FileWriter) Write(p []byte) (int, error)

func (*FileWriter) WriteLine

func (fw *FileWriter) WriteLine(line []byte) (int, error)

type FileWriterInterface

type FileWriterInterface interface {
	MkdirAll(dirname string, perm fs.FileMode) error
	Create(name string) (io.WriteCloser, error)
}

FileWriterInterface provides basic file writing operations.

type ISourceLister

type ISourceLister interface {
	io.Closer
	DirReader
}

ISourceLister provides directory listing operations for source file systems.

type ISourceReader

type ISourceReader interface {
	ISourceLister
	Reader
}

ISourceReader combines source listing and reading operations.

type ITargetUploader

type ITargetUploader interface {
	io.Closer
	Put(local, remote string) error
	PutStream(reader io.Reader, remote string) error
	PutEmpty(remote string) error
	Exist(remote string) bool
}

ITargetUploader provides upload operations for target file systems.

type NewReaderFunc

type NewReaderFunc func(fri FileReaderInterface, path string, ct CompressType) (io.ReadCloser, error)

NewReaderFunc is a function type for creating new readers with compression support.

type NewWriterFunc

type NewWriterFunc func(fwi FileWriterInterface, path string, bufSize int, ct CompressType) (io.WriteCloser, error)

NewWriterFunc is a function type for creating new writers with compression and buffer support.

type Reader

type Reader interface {
	Open(name string) (io.ReadCloser, error)
}

Reader provides read operations for files.

type Stater

type Stater interface {
	Stat(name string) (fs.FileInfo, error)
}

Stater provides file stat operations.

type Walker

type Walker interface {
	Walk(root string, walkFn filepath.WalkFunc) error
}

Walker provides file tree walking operations.

type Writer

type Writer interface {
	DirCreator
	Create(name string) (io.WriteCloser, error)
	Rename(oldPath, newPath string) error
}

Writer provides write operations for files and directories.