Documentation ¶
Overview ¶
Package recordio implements a file format for a sequence of variable-length data.
The layout of each record is
+-----------------------------------------------------------| | 4-bytes Magic number (including compression mode) | +-----------------------------------------------------------| | 4-byte Uncompressed record size (uint32 in little-endian) | +-----------------------------------------------------------| | 4-byte Compressed record size (uint32 in little-endian) | +-----------------------------------------------------------| | Record data (maximum 1024MB) | +-----------------------------------------------------------|
Example:
// Writing records. f, _ := os.Create("data.rec") defer f.Close() w, _ := recordio.NewWriter(f, recordio.Snappy) w.Write([]byte("this is a record")) w.Write([]byte("this is a second record")) ... // Reading records. f, _ := os.Open("data.rec") defer f.Close() r, _ := recordio.NewReader(f) for { rec, err := r.Read() if err == io.EOF { break } ... }
Index ¶
Examples ¶
Constants ¶
View Source
const ( // NoCompression does not compress records. NoCompression = modeBase + iota // Snappy compresses records with Snappy. Snappy // Zlib compresses records with zlib. Zlib // EndOfMode is the end of mode. EndOfMode )
Variables ¶
View Source
var ( // ErrInvalidHeader means that a record header is not valid. ErrInvalidHeader = errors.New("invalid record header") // ErrInvalidMode means that a (compression) mode is not valid. ErrInvalidMode = errors.New("invalid mode") // ErrRecordCorrupted means that a record is corrupted. ErrRecordCorrupted = errors.New("record corrupted") // ErrRecordTooLarge means that a record is too large. ErrRecordTooLarge = errors.New("record too large") )
Functions ¶
This section is empty.
Types ¶
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader implements sequential reading of variable-length data.
func NewReader ¶
NewReader returns a new reader that reads records from r.
Example ¶
package main import ( "bytes" "fmt" "io" "github.com/jhahn21/recordio" ) func main() { b := bytes.NewReader([]byte{210, 159, 49, 224, 5, 0, 0, 0, 23, 0, 0, 0, 255, 6, 0, 0, 115, 78, 97, 80, 112, 89, 1, 9, 0, 0, 85, 255, 35, 229, 102, 105, 114, 115, 116, 210, 159, 49, 224, 6, 0, 0, 0, 24, 0, 0, 0, 255, 6, 0, 0, 115, 78, 97, 80, 112, 89, 1, 10, 0, 0, 211, 224, 211, 202, 115, 101, 99, 111, 110, 100}) r, err := recordio.NewReader(b) if err != nil { panic(err) } for { d, err := r.Read() if err == io.EOF { break } fmt.Println(string(d)) } }
Output: first second
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer implements sequential writing of variable-length data.
func NewWriter ¶
NewWriter returns a new writer that writes records to w.
Example ¶
package main import ( "bytes" "fmt" "github.com/jhahn21/recordio" ) func main() { var b bytes.Buffer w, err := recordio.NewWriter(&b, recordio.Snappy) if err != nil { panic(err) } w.Write([]byte("first")) w.Write([]byte("second")) fmt.Println(b.Bytes()) }
Output: [210 159 49 224 5 0 0 0 23 0 0 0 255 6 0 0 115 78 97 80 112 89 1 9 0 0 85 255 35 229 102 105 114 115 116 210 159 49 224 6 0 0 0 24 0 0 0 255 6 0 0 115 78 97 80 112 89 1 10 0 0 211 224 211 202 115 101 99 111 110 100]
Click to show internal directories.
Click to hide internal directories.