README
¶
MemDiskBuffer
A hybrid buffer which preferrs to use the memory and then uses disk past a defined threshold. The value of using a buffer like this is to be able to buffer a stream without knowing the final size.
Documentation: https://pkg.go.dev/github.com/pschou/go-memdiskbuf
Documentation
¶
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ( // Toggle debug Debug = false )
Functions ¶
Types ¶
type Buffer ¶
type Buffer struct {
// contains filtered or unexported fields
}
func NewBuffer ¶
When calling a new path, provide a file name which could potentially be used for a buffer once the minimum threshold has been met. Any writes below the threshold will not touch the disk, while any writes over the threshold will.
Please note that DiskBuf should be a power of 2 multiplied by 4KB, so a good value would be 16K. This is because the sector size of a disk or NAND memory is usually 4k.
Example ¶
buf := memdiskbuf.NewBuffer("/nowhere/unused_file", 10<<10, 8<<10)
{
str := "This is a test\n"
n, err := buf.Write([]byte(str))
fmt.Printf("Wrote %q (%d) err: %s\n", str, n, err)
}
{
n, err := io.Copy(os.Stdout, buf)
fmt.Println("Copied", n, "err:", err)
}
Output: Wrote "This is a test\n" (15) err: %!s(<nil>) This is a test Copied 15 err: <nil>
func (*Buffer) Read ¶
Read reads from the buffer. The first read will switch the buffer from writing mode to reading mode to prevent further writes. One can use Reset() to clear out the buffer and return to writing mode.
func (*Buffer) ReadAt ¶
ReadAt reads from the buffer. The first read will switch the buffer from writing mode to reading mode to prevent further writes. One can use Reset() to clear out the buffer and return to writing mode.
func (*Buffer) Rewind ¶
func (b *Buffer) Rewind()
Until a Buffer is reset, the buffer can be Rewound to restart from the beginning.
Example ¶
buf := memdiskbuf.NewBuffer("/nowhere/unused_file", 10<<10, 8<<10)
{
str := "This is a test\n"
n, err := buf.Write([]byte(str))
fmt.Printf("Wrote %q (%d) err: %s\n", str, n, err)
}
{
n, err := io.Copy(os.Stdout, buf)
fmt.Println("Copied", n, "err:", err)
}
buf.Rewind()
{
n, err := io.Copy(os.Stdout, buf)
fmt.Println("Copied", n, "err:", err)
}
Output: Wrote "This is a test\n" (15) err: %!s(<nil>) This is a test Copied 15 err: <nil> This is a test Copied 15 err: <nil>
func (*Buffer) Write ¶
Write to the Buffer, first filling up the memory buffer and then to the disk.
Example (ToDisk) ¶
buf := memdiskbuf.NewBuffer("./data.tmp", 5, 10)
str := "This is a test of a long test data string as a dump to file.\n"
{
n, err := buf.Write([]byte(str))
fmt.Printf("Wrote %q (%d) err: %s\n", str, n, err)
}
var out strings.Builder
{
n, err := io.Copy(&out, buf)
fmt.Println("Copied", n, "err:", err)
}
if out.String() == str {
fmt.Println("Strings match!")
}
buf.Reset()
Output: Wrote "This is a test of a long test data string as a dump to file.\n" (61) err: %!s(<nil>) Copied 61 err: <nil> Strings match!
Example (TwiceError) ¶
buf := memdiskbuf.NewBuffer("/nowhere/unused_file", 10<<10, 8<<10)
str := "This is a test\n"
n, err := buf.Write([]byte(str))
fmt.Printf("Wrote %q (%d) err: %s\n", str, n, err)
io.Copy(io.Discard, buf)
n, err = buf.Write([]byte(str))
fmt.Printf("Second write %q (%d) err: %s\n", str, n, err)
Output: Wrote "This is a test\n" (15) err: %!s(<nil>) Second write "This is a test\n" (0) err: Already in read mode
Example (WithReset) ¶
buf := memdiskbuf.NewBuffer("./abe.tmp", 5, 10)
str := `Four score and seven years ago our fathers brought forth on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal.
Now we are engaged in a great civil war, testing whether that nation, or any nation so conceived and so dedicated, can long endure. We are met on a great battle-field of that war. We have come to dedicate a portion of that field, as a final resting place for those who here gave their lives that that nation might live. It is altogether fitting and proper that we should do this.
But, in a larger sense, we can not dedicate—we can not consecrate—we can not hallow—this ground. The brave men, living and dead, who struggled here, have consecrated it, far above our poor power to add or detract. The world will little note, nor long remember what we say here, but it can never forget what they did here. It is for us the living, rather, to be dedicated here to the unfinished work which they who fought here have thus far so nobly advanced. It is rather for us to be here dedicated to the great task remaining before us—that from these honored dead we take increased devotion to that cause for which they gave the last full measure of devotion—that we here highly resolve that these dead shall not have died in vain—that this nation, under God, shall have a new birth of freedom—and that government of the people, by the people, for the people, shall not perish from the earth.
—Abraham Lincoln`
{
n, err := buf.Write([]byte(str))
fmt.Println("Wrote", n, "err:", err)
}
var out strings.Builder
{
n, err := io.Copy(&out, buf)
fmt.Println("Copied", n, "err:", err)
}
if out.String() == str {
fmt.Println("Strings match!")
}
{ // Quick mini read of 11 bytes
short := make([]byte, 11)
buf.ReadAt(short, 15)
fmt.Println("short:", string(short))
}
out.Reset()
buf.Reset()
{
n, err := buf.Write([]byte("Hello"))
fmt.Println("Wrote", n, "err:", err)
}
{
n, err := io.Copy(&out, buf)
fmt.Println("Copied", n, "err:", err)
}
if out.String() == "Hello" {
fmt.Println("Strings match!")
}
Output: Wrote 1488 err: <nil> Copied 1488 err: <nil> Strings match! short: seven years Wrote 5 err: <nil> Copied 5 err: <nil> Strings match!
type WriterAtBuf ¶
type WriterAtBuf struct {
// User provided function for in-memory inspection of the data before it is
// written to disk. A useful way to use this is for calling a Hash function
// over the data so it doesn't have to be re-read from the disk.
StreamFunc func([]byte)
// contains filtered or unexported fields
}
A buffer for WriteAt calls. The practical use for this module is to reconstruct an original file when only specific fragments are available at any given moment, such as from a semi-ordered UDP stream. Note: A larger buffer size will prevent buffer overruns when UDP traffic has more sever unordering in packets.
This WriterAtBuf assumes the payload being written is immutable, meaning the bytes written and then re-written will never change.
An example of a UDP buffer for file transfers:
fh, _ := os.Create("my.dat")
wab := NewWriterAtBuf(fh, 20<<10) // A generous 20k data ordering buffer
// Listen for UDP packets with offset positioning
wab.WriteAt(UDPdata, offset)
func NewWriterAtBuf ¶
func NewWriterAtBuf(fh io.WriterAt, bufSize int) *WriterAtBuf
NewWriterAtBuf creates a new WriterAtBuf for buffering WriteAt calls with a specified bufSize. For optimal performance, the BufSize should be at least 8k and preferably around 32k.
func NewWriterAtBufWithBlockSize ¶
func NewWriterAtBufWithBlockSize(fh io.WriterAt, bufSize, blockSize int) *WriterAtBuf
NewWriterAtBufWithBlockSize creates a new WriterAtBuf for buffering WriteAt calls with a specified bufSize. For optimal performance, the BufSize should be at least 8k and preferably around 32k. The block size should be a 2^n multiple of 4k, like 4k, 8k, ... to ensure the best performance for sector size matching.
The bufSize must be greater than twice the blockSize to ensure that the block can properly be blocked off and written to the destination.
Example ¶
fh, err := os.OpenFile("example_file", os.O_RDWR|os.O_CREATE, 0600)
if err != nil {
log.Fatal(err)
}
fh.Truncate(100)
wab := memdiskbuf.NewWriterAtBufWithBlockSize(fh, 40, 10)
wab.WriteAt([]byte("world,"), 6)
wab.WriteAt([]byte("hello "), 0)
wab.WriteAt([]byte(" will "), 12)
wab.WriteAt([]byte("you be"), 18)
wab.WriteAt([]byte("iend?\n"), 30)
wab.WriteAt([]byte(" my fr"), 24)
err = wab.Flush()
if err != nil {
fmt.Println("err:", err)
}
dat := make([]byte, 36)
fh.ReadAt(dat, 0)
fmt.Printf("%q\n", dat)
fmt.Println("written:", wab.Written())
fh.Close()
os.Remove("example_file")
Output: "hello world, will you be my friend?\n" written: 36
func (*WriterAtBuf) Flush ¶
func (w *WriterAtBuf) Flush() (err error)
Flush - flushes what is in the buffer to disk, but don't advance the buffer to prevent blocks getting out of sync. An error will be returned if there are gaps in the buffer.
func (*WriterAtBuf) FlushAll ¶
func (w *WriterAtBuf) FlushAll() (err error)
FlushAll - flushes all the buffer to disk, gaps and all. Doesn't advance the buffer to prevent blocks getting out of sync. An error will be returned if there was an error writing to the target file.
Note: FlushAll does not call the StreamFunc as Flush() is the proper way to ensure all the segments are written correctly. FlushAll is a sort of "give up" function when an error in writing happens and one wants to get the current state of a file, gaps and all.
func (*WriterAtBuf) Flushable ¶
func (w *WriterAtBuf) Flushable() (n int64, err error)
Flushable returns the flushable size of a file if a Flush is called. An error is returned should there be any gaps.
func (*WriterAtBuf) Reset ¶
func (w *WriterAtBuf) Reset(fh io.WriterAt)
Reset will reset the counters and reuse the underlying buffer for another WriterAt interface.
Source Files