hddreader

Documentation

Overview

Package hddreader wraps an os.File object with a algorithm which tries to optimise parallel reading of multiple files from a hdd. It does this by limiting number of open files and reading files in order of disk offsets. Main components: File: file object, semi-interchangeable with os.File. Implements: Reader, Closer, WriterTo Disk: disk object, schedules file reads bufferpool: pool of reusable same-sized []byte buffers

Index

• type Disk
  • func NewDisk(maxread int, maxwindow int, ahead int64, behind int64, maxopen int, bufkB int) *Disk
  • func (d *Disk) Close()
  • func (self *Disk) Start(wait int)
• type File
  • func Open(name string, disk *Disk) (*File, error)
  • func OpenFile(name string, flag int, perm os.FileMode, disk *Disk, keepOpen bool) (*File, error)
  • func (f *File) Close() (err error)
  • func (f *File) Name() string
  • func (f *File) Read(b []byte) (n int, err error)
  • func (f *File) WriteTo(w io.Writer) (n int64, err error)

Types

type Disk

type Disk struct {
	// contains filtered or unexported fields
}

A Disk schedules read operations for files. It shoulds be created using NewDisk. A call to Disk.Start() is required before the disk will allow associated files to start reading data.

Example usage:

d := hddreader.NewDisk(1, 0, 0, 0, 100, 64)
for _, p := range(paths) {
        wg.Add(1)
        go read_stuff_from(p, &wg)
        // calls hddreader.OpenFile() and file.Read() but will block until d.Start() called
}
d.Start()  // enables reading and will unblock pending Read() calls in disk offset order
wg.Wait()
d.Close()

func NewDisk

func NewDisk(maxread int, maxwindow int, ahead int64, behind int64, maxopen int, bufkB int) *Disk

NewDisk creates a new disk object to schedule read operations in order of increasing physical offset on the disk.

Up to maxread files will be read concurrently. An additional up to maxwindow files may be opened for reading if they are within (-behind,+ahead) bytes of the current head position. An additional up to maxopen files may be open for the sole purpose of reading disk offset.

If bufkB > 0 then an internal buffer pool is created to buffer WriteTo() calls. Note that while read order is preserved, buffering may change the order in which WriteTo(w) calls complete, depending on speed at which buffers are writted to w.

func (*Disk) Close

func (d *Disk) Close()

Closes closes the disk scheduler and frees buffer memory

func (*Disk) Start

func (self *Disk) Start(wait int)

Start needs to be called once in order to enable file data to start reading. If wait > 0 then will wait until that many read requests have been registered before starting

type File

type File struct {
	Offset uint64 // file location relative to start of disk
	// contains filtered or unexported fields
}

a File struct wraps an os.File stuct with the necessary field and methods for hdd-optimised reads.

func Open

func Open(name string, disk *Disk) (*File, error)

Open creates a new File object associated with a Disk, and register the file's physical position on the disk. It may block until the disk's total open file count falls below the disk's open file limit.

func OpenFile

func OpenFile(name string, flag int, perm os.FileMode, disk *Disk, keepOpen bool) (*File, error)

func (*File) Close

func (f *File) Close() (err error)

Close closes f, notifying the disk scheduler accordingly

func (*File) Name

func (f *File) Name() string

Name returns file name as provided to Open

func (*File) Read

func (f *File) Read(b []byte) (n int, err error)

Read reads up to len(b) bytes from the File. It returns the number of bytes read and an error, if any. EOF is signaled by a zero count with err set to io.EOF.

func (*File) WriteTo

func (f *File) WriteTo(w io.Writer) (n int64, err error)

WriteTo implements io.WriterTo interface.