ioutil

Documentation

Overview

Package ioutil implements some I/O utility functions which are not covered by the standard library.

Index

• Constants
• Variables
• func AppendFile(dst string, src string, osync bool) error
• func Copy(dst io.Writer, src io.Reader) (written int64, err error)
• func CopyAligned(w io.Writer, r io.Reader, alignedBuf []byte, totalSize int64, file *os.File) (int64, error)
• func NewDeadlineWriter(w io.WriteCloser, timeout time.Duration) io.WriteCloser
• func NewSkipReader(r io.Reader, n int64) io.Reader
• func NopCloser(w io.Writer) io.WriteCloser
• func ReadFile(name string) ([]byte, error)
• func ReadFileWithFileInfo(name string) ([]byte, fs.FileInfo, error)
• func SameFile(fi1, fi2 os.FileInfo) bool
• func WaitPipe() (*PipeReader, *PipeWriter)
• type DeadlineWriter
  • func (w *DeadlineWriter) Close() error
  • func (w *DeadlineWriter) Write(buf []byte) (int, error)
• type LimitWriter
  • func LimitedWriter(w io.Writer, skipBytes int64, limit int64) *LimitWriter
  • func (w *LimitWriter) Close() error
  • func (w *LimitWriter) Write(p []byte) (n int, err error)
• type ODirectReader
  • func (o *ODirectReader) Close() error
  • func (o *ODirectReader) Read(buf []byte) (n int, err error)
• type PipeReader
  • func (r *PipeReader) CloseWithError(err error) error
• type PipeWriter
  • func (w *PipeWriter) CloseWithError(err error) error
• type SkipReader
  • func (s *SkipReader) Read(p []byte) (int, error)
• type WriteOnCloser
  • func WriteOnClose(w io.Writer) *WriteOnCloser
  • func (w *WriteOnCloser) Close() error
  • func (w *WriteOnCloser) HasWritten() bool
  • func (w *WriteOnCloser) Write(p []byte) (int, error)

Constants

const (
	BlockSizeSmall       = 32 * humanize.KiByte // Default r/w block size for smaller objects.
	BlockSizeLarge       = 2 * humanize.MiByte  // Default r/w block size for larger objects.
	BlockSizeReallyLarge = 4 * humanize.MiByte  // Default write block size for objects per shard >= 64MiB
)

Block sizes constant.

const DirectioAlignSize = 4096

DirectioAlignSize - DirectIO alignment needs to be 4K. Defined here as directio.AlignSize is defined as 0 in MacOS causing divide by 0 error.

Variables

var (
	ODirectPoolXLarge = sync.Pool{
		New: func() interface{} {
			b := disk.AlignedBlock(BlockSizeReallyLarge)
			return &b
		},
	}
	ODirectPoolLarge = sync.Pool{
		New: func() interface{} {
			b := disk.AlignedBlock(BlockSizeLarge)
			return &b
		},
	}
	ODirectPoolSmall = sync.Pool{
		New: func() interface{} {
			b := disk.AlignedBlock(BlockSizeSmall)
			return &b
		},
	}
)

O_DIRECT aligned sync.Pool's

var (
	// OpenFileDirectIO allows overriding default function.
	OpenFileDirectIO = disk.OpenFileDirectIO
	// OsOpen allows overriding default function.
	OsOpen = os.Open
	// OsOpenFile allows overriding default function.
	OsOpenFile = os.OpenFile
)

Functions

func AppendFile

func AppendFile(dst string, src string, osync bool) error

AppendFile - appends the file "src" to the file "dst"

func Copy

func Copy(dst io.Writer, src io.Reader) (written int64, err error)

Copy is exactly like io.Copy but with re-usable buffers.

func CopyAligned

func CopyAligned(w io.Writer, r io.Reader, alignedBuf []byte, totalSize int64, file *os.File) (int64, error)

CopyAligned - copies from reader to writer using the aligned input buffer, it is expected that input buffer is page aligned to 4K page boundaries. Without passing aligned buffer may cause this function to return error.

This code is similar in spirit to io.Copy but it is only to be used with DIRECT I/O based file descriptor and it is expected that input writer *os.File not a generic io.Writer. Make sure to have the file opened for writes with syscall.O_DIRECT flag.

func NewDeadlineWriter

func NewDeadlineWriter(w io.WriteCloser, timeout time.Duration) io.WriteCloser

NewDeadlineWriter wraps a writer to make it respect given deadline value per Write(). If there is a blocking write, the returned Writer will return whenever the timer hits (the return values are n=0 and err=context.Canceled.)

func NewSkipReader

func NewSkipReader(r io.Reader, n int64) io.Reader

NewSkipReader - creates a SkipReader

func NopCloser

func NopCloser(w io.Writer) io.WriteCloser

NopCloser returns a WriteCloser with a no-op Close method wrapping the provided Writer w.

func ReadFile

func ReadFile(name string) ([]byte, error)

ReadFile reads the named file and returns the contents. A successful call returns err == nil, not err == EOF. Because ReadFile reads the whole file, it does not treat an EOF from Read as an error to be reported.

passes NOATIME flag for reads on Unix systems to avoid atime updates.

func ReadFileWithFileInfo

func ReadFileWithFileInfo(name string) ([]byte, fs.FileInfo, error)

ReadFileWithFileInfo reads the named file and returns the contents. A successful call returns err == nil, not err == EOF. Because ReadFile reads the whole file, it does not treat an EOF from Read as an error to be reported, additionall returns os.FileInfo

func SameFile

func SameFile(fi1, fi2 os.FileInfo) bool

SameFile returns if the files are same.

func WaitPipe

func WaitPipe() (*PipeReader, *PipeWriter)

WaitPipe implements wait-group backend io.Pipe to provide synchronization between read() end with write() end.

Types

type DeadlineWriter

type DeadlineWriter struct {
	io.WriteCloser
	// contains filtered or unexported fields
}

DeadlineWriter deadline writer with context

func (*DeadlineWriter) Close

func (w *DeadlineWriter) Close() error

Close closer interface to close the underlying closer

func (*DeadlineWriter) Write

func (w *DeadlineWriter) Write(buf []byte) (int, error)

type LimitWriter

type LimitWriter struct {
	io.Writer
	// contains filtered or unexported fields
}

LimitWriter implements io.WriteCloser.

This is implemented such that we want to restrict an enscapsulated writer upto a certain length and skip a certain number of bytes.

func LimitedWriter

func LimitedWriter(w io.Writer, skipBytes int64, limit int64) *LimitWriter

LimitedWriter takes an io.Writer and returns an ioutil.LimitWriter.

func (*LimitWriter) Close

func (w *LimitWriter) Close() error

Close closes the LimitWriter. It behaves like io.Closer.

func (*LimitWriter) Write

func (w *LimitWriter) Write(p []byte) (n int, err error)

Write implements the io.Writer interface limiting upto configured length, also skips the first N bytes.

type ODirectReader

type ODirectReader struct {
	File      *os.File
	SmallFile bool
	// contains filtered or unexported fields
}

ODirectReader - to support O_DIRECT reads for erasure backends.

func (*ODirectReader) Close

func (o *ODirectReader) Close() error

Close - Release the buffer and close the file.

func (*ODirectReader) Read

func (o *ODirectReader) Read(buf []byte) (n int, err error)

Read - Implements Reader interface.

type PipeReader

type PipeReader struct {
	*io.PipeReader
	// contains filtered or unexported fields
}

PipeReader is similar to io.PipeReader with wait group

func (*PipeReader) CloseWithError

func (r *PipeReader) CloseWithError(err error) error

CloseWithError close with supplied error the reader end

type PipeWriter

type PipeWriter struct {
	*io.PipeWriter
	// contains filtered or unexported fields
}

PipeWriter is similar to io.PipeWriter with wait group

func (*PipeWriter) CloseWithError

func (w *PipeWriter) CloseWithError(err error) error

CloseWithError close with supplied error the writer end.

type SkipReader

type SkipReader struct {
	io.Reader
	// contains filtered or unexported fields
}

SkipReader skips a given number of bytes and then returns all remaining data.

func (*SkipReader) Read

func (s *SkipReader) Read(p []byte) (int, error)

type WriteOnCloser

type WriteOnCloser struct {
	io.Writer
	// contains filtered or unexported fields
}

WriteOnCloser implements io.WriteCloser and always executes at least one write operation if it is closed.

This can be useful within the context of HTTP. At least one write operation must happen to send the HTTP headers to the peer.

func WriteOnClose

func WriteOnClose(w io.Writer) *WriteOnCloser

WriteOnClose takes an io.Writer and returns an ioutil.WriteOnCloser.

func (*WriteOnCloser) Close

func (w *WriteOnCloser) Close() error

Close closes the WriteOnCloser. It behaves like io.Closer.

func (*WriteOnCloser) HasWritten

func (w *WriteOnCloser) HasWritten() bool

HasWritten returns true if at least one write operation was performed.

func (*WriteOnCloser) Write

func (w *WriteOnCloser) Write(p []byte) (int, error)