Documentation
¶
Overview ¶
The blocks package is responsible for reading and writing potentially compressed binary data in blocks or chunks.
It allows O(1) access to the content of any block if you know the offset of the block in a sea of bytes.
The reader and writer objects implement many of the common `io` package interfaces which should make them drop in replacements for many use cases where you'd use `byte.Buffer`, `byte.Reader`, `bufio.Reader`, `bufio.Writer`, etc.
Index ¶
Examples ¶
Constants ¶
const ( NO_COMPRESSION = iota SNAPPY_COMPRESSION = iota )
Variables ¶
var ( DefaultHeaderLen = 3 DefaultBlockSize = 65535 )
Any FastReader that uses the defaults will use this allocation pool.
var BadSeek = errors.New("block reader can only seek relative to beginning of file.")
Functions ¶
This section is empty.
Types ¶
type FastReader ¶
type FastReader struct {
// contains filtered or unexported fields
}
Reader has the ability to uncompress and read any potentially compressed data written via blocks.Writer. Reads ahead to decompress future blocks before you need them to improve performance.
Example ¶
reader := NewFastReader(context.Background(), bytes.NewReader(
[]byte("\x05\x00\x00hello\x05\x00\x00 worl\x01\x00\x00d"),
), 5)
result := make([]byte, 11)
n, _ := reader.Read(result)
fmt.Printf("%q", result[:n])
Output: "hello world"
func NewFastReader ¶
Version of blocks.Reader which reads ahead and decompresses future blocks before you need them to improve performance.
func (*FastReader) Peek ¶
func (r *FastReader) Peek(n int) []byte
Returns the next n bytes in the buffer, without advancing. A following call to Read will return the same bytes.
func (*FastReader) Read ¶
func (r *FastReader) Read(p []byte) (n int, err error)
Implements io.Reader interface.
func (*FastReader) ReadByte ¶
func (r *FastReader) ReadByte() (c byte, err error)
Implements io.ByteReader interface.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader has the ability to uncompress and read any potentially compressed data written via blocks.Writer.
Example ¶
reader := NewReader(bytes.NewReader(
[]byte("\x05\x00\x00hello\x05\x00\x00 worl\x01\x00\x00d"),
), 5)
result := make([]byte, 11)
n, _ := reader.Read(result)
fmt.Printf("%q", result[:n])
Output: "hello world"
func NewByteReader ¶
Transforms a bytestring into a block reader. blockSize must be the same size used to originally write the blocks you're attempting to read. Otherwise you'll definitely get incorrect results.
func NewReader ¶
Transforms any object which implements io.Reader and io.Seeker into a block reader. blockSize must be the same used to originally write the blocks you're attempting to read. the same size used to write the blocks you're attempting to read. Otherwise you'll definitely get incorrect results.
func (*Reader) Peek ¶
Returns the next n bytes in the buffer, without advancing. A following call to Read will return the same bytes.
type Writer ¶
Writer implements the io.Writer interface and is meant to be used in place of a io.Writer or bufio.Writer when writing data.
Data is written in blocks or chunks, and optionally compressed with snappy compression.
When data is written, if the buffered amount then exceeds the configured blockSize, the block is encoded and compressed and written to the underlying io.Writer.
Each block can be read using the blocks.Reader object, and has the following format:
[int16/int32/int64:blockLength][int8:encoding][bytes(blockLength):data]
blockLength's type is the smallest fixed length integer size that can contain the max configured blockSize. For instance, if blockSize is 4096 bytes, we'll used an uint16. If the blockSize is 128 KB, we'll use a uint32, etc.
Example ¶
buffer := new(bytes.Buffer)
writer := NewWriter(buffer, 5)
writer.Write([]byte("hello world"))
writer.Flush()
fmt.Printf("%q", buffer.Bytes())
Output: "\x05\x00\x00hello\x05\x00\x00 worl\x01\x00\x00d"
func (*Writer) Buffered ¶
Returns the number of bytes that have been written to the blocks.Writer, but haven't yet been encoded and written to the underlying io.Writer. This should never exceed the configured max blockSize.
Source Files