Version: v0.0.0-...-7a1a713 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Jul 1, 2013 License: Apache-2.0, BSD-3-Clause Imports: 4 Imported by: 0



Package record reads and writes sequences of records. Each record is a stream of bytes that completes before the next record starts.

When reading, call Next to obtain an io.Reader for the next record. Next will return io.EOF when there are no more records. It is valid to call Next without reading the current record to exhaustion.

When writing, call Next to obtain an io.Writer for the next record. Calling Next finishes the current record. Call Close to finish the final record.

Optionally, call Flush to finish the current record and flush the underlying writer without starting a new record. To start a new record after flushing, call Next.

Neither Readers or Writers are safe to use concurrently.

Example code:

func read(r io.Reader) ([]string, error) {
	var ss []string
	sr := record.NewReader(r)
	for {
		err := sr.Next()
		if err == io.EOF {
		if err != nil {
			return nil, err
		s, err := ioutil.ReadAll(sr)
		if err != nil {
			return nil, err
		ss = append(ss, string(s))
	return ss, nil

func write(w io.Writer, ss []string) error {
	sw := record.NewWriter(w)
	for _, s := range ss {
		if _, err := sw.Write([]byte(s)), err != nil {
			return err
	return sw.Close()

The wire format is that the stream is divided into 32KiB blocks, and each block contains a number of tightly packed chunks. Chunks cannot cross block boundaries. The last block may be shorter than 32 KiB. Any unused bytes in a block must be zero.

A record maps to one or more chunks. Each chunk has a 7 byte header (a 4 byte checksum, a 2 byte little-endian uint16 length, and a 1 byte chunk type) followed by a payload. The checksum is over the chunk type and the payload.

There are four chunk types: whether the chunk is the full record, or the first, middle or last chunk of a multi-chunk record. A multi-chunk record has one first chunk, zero or more middle chunks, and one last chunk.

The wire format allows for limited recovery in the face of data corruption: on a format error (such as a checksum mismatch), the reader moves to the next block and looks for the next full or first chunk.



This section is empty.


This section is empty.


This section is empty.


type Reader

type Reader struct {
	// contains filtered or unexported fields

Reader reads records from an underlying io.Reader.

func NewReader

func NewReader(r io.Reader) *Reader

NewReader returns a new reader.

func (*Reader) Next

func (r *Reader) Next() (io.Reader, error)

Next returns a reader for the next record. It returns io.EOF if there are no more records. The reader returned becomes stale after the next Next call, and should no longer be used.

type Writer

type Writer struct {
	// contains filtered or unexported fields

Writer writes records to an underlying io.Writer.

func NewWriter

func NewWriter(w io.Writer) *Writer

NewWriter returns a new Writer.

func (*Writer) Close

func (w *Writer) Close() error

Close finishes the current record and closes the writer.

func (*Writer) Flush

func (w *Writer) Flush() error

Flush finishes the current record, writes to the underlying writer, and flushes it if that writer implements interface{ Flush() error }.

func (*Writer) Next

func (w *Writer) Next() (io.Writer, error)

Next returns a writer for the next record. The writer returned becomes stale after the next Close, Flush or Next call, and should no longer be used.

Source Files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL