linereader

README

linereader

Simple buffered line reader for Go: read a stream line by line — or by raw bytes, or as a plain io.Reader — over one chain of reusable blocks, without ever losing a byte between styles.

lr := linereader.NewLineReader(file, 0, 0) // 0 = defaults (4096)

for {
	line, err := lr.ReadLine()
	if err == io.EOF {
		break
	}
	if err != nil {
		return err
	}
	fmt.Printf("%s (ended by %#x)\n", line, int(lr.LastEolType))
}

What it does

• ReadLine() ([]byte, error) — the next line, terminator consumed and never included. A line ends at whatever happens first: \r, \n or EOF. The pair \r\n counts as one terminator — even when it rides a block boundary (a \r as the last buffered byte makes the reader pull another block before deciding; unless EOF got there first: a lone CR then). What ended the line lands in LastEolType: EolLf, EolCr, EolCrLf, or EolEof for a final unterminated line.
• ReadBytes(n uint64) ([]byte, error) — the next n raw bytes, buffer first: whatever already sits in the chain is served before touching the stream. Short only at the end.
• PeekBytes(n uint64) ([]byte, error) — look without taking: the same buffer-first pull, but the offset stays put — the next read serves these same bytes again.
• SkipBytes(n uint64) (uint64, error) — exactly ReadBytes, only returning how many bytes were skipped instead of a buffer (literally: that is the implementation). Peek and skip together make the classic sniff: peek a BOM (or any magic bytes), decide, skip it.
• Read(p []byte) (int, error) — the canonical io.Reader contract: read the header of something with ReadLine, then hand the reader itself to any io.Reader consumer — it continues right where you stopped, buffered bytes first.
• IoLineReader — the interface for line consumers: interface { ReadLine() ([]byte, error) }. Sign your functions against it and any line source can stand in; LineReader is the reference implementation.

How it works

Blocks are grabbed from the stream (BlockSize, default 4096) and chained; lines are found across the chain. Consumed blocks are discarded whole — bytes never move. A line contained in one block is served as a sub-slice, no copy; one crossing blocks gets assembled. MaxLineLength (default 4096) caps how far a line may grow without an EOL — an error instead of unbounded memory.

Errors, EOF and sources that revive

Buffered data always comes out first: errors (io.EOF included) surface only once the chain is drained. A final unterminated line is data, not an error — it comes back with err == nil and LastEolType == EolEof; only the next call returns io.EOF.

The stream's error is kept in CachedError and served once, then cleared: the next attempt asks the stream again. A source that comes back to life after an EOF — a growing file, tail-follow style — revives with a plain poll loop, no rearming:

for {
	line, err := lr.ReadLine()
	if err == io.EOF {
		time.Sleep(interval) // the source may grow
		continue
	}
	...
}

Install

go get github.com/ot4go/linereader

License

MIT — see LICENSE.

Documentation

Index

• type ByteBuffer
• type EolType
• type IoLineReader
• type LineReader
  • func NewLineReader(reader io.Reader, blockSize uint64, maxLineSize uint64) *LineReader
  • func (lr *LineReader) PeekBytes(nBytes uint64) ([]byte, error)
  • func (lr *LineReader) Read(p []byte) (int, error)
  • func (lr *LineReader) ReadBytes(nBytes uint64) ([]byte, error)
  • func (lr *LineReader) ReadLine() ([]byte, error)
  • func (lr *LineReader) SkipBytes(nBytes uint64) (uint64, error)

Types

type ByteBuffer

type ByteBuffer []byte

type EolType

type EolType int

EolType tells what ended a line: which terminator happened first.

const (
	EolNone EolType = 0
	EolLf   EolType = 0x000A
	EolCr   EolType = 0x000D
	EolCrLf EolType = 0x0D0A
	EolEof  EolType = 1
)

type IoLineReader

type IoLineReader interface {
	ReadLine() ([]byte, error)
}

IoLineReader is what a line consumer needs from a reader: just the line. LineReader is the reference implementation, anything else may stand in.

type LineReader

type LineReader struct {
	BlockSize     uint64
	Buffers       []*ByteBuffer
	Offset        uint64
	Reader        io.Reader
	MaxLineLength uint64
	LastEolType   EolType // what ended the last line ReadLine served
	CachedError   error   // the stream's last error (io.EOF included) — served once, then cleared
}

func NewLineReader

func NewLineReader(reader io.Reader, blockSize uint64, maxLineSize uint64) *LineReader

func (*LineReader) PeekBytes

func (lr *LineReader) PeekBytes(nBytes uint64) ([]byte, error)

PeekBytes looks at the next nBytes without consuming them: the same buffer-first pull as ReadBytes, but the offset stays put — the next read serves these same bytes again. Fewer only at the end (nil error while something is there); nothing at all: nil and the stream's error.

func (*LineReader) Read

func (lr *LineReader) Read(p []byte) (int, error)

Read implements io.Reader: fills p from the buffered chain first, then from the stream. Short reads at the end, io.EOF when nothing is left — the canonical contract.

func (*LineReader) ReadBytes

func (lr *LineReader) ReadBytes(nBytes uint64) ([]byte, error)

ReadBytes hands back the next nBytes as raw bytes — buffer first: whatever already sits in the chain is served before touching the stream, then blocks are pulled until the count is met or the stream ends. At the end it may return fewer bytes (with a nil error while something came); nothing left at all: nil and the stream's error.

func (*LineReader) ReadLine

func (lr *LineReader) ReadLine() ([]byte, error)

ReadLine returns the next line without its terminator, consuming it. A line ends at whatever happens first: '\r', '\n' or EOF; the pair \r\n counts as one terminator — even when it rides a block boundary: with the '\r' as the last buffered byte, another block is read before deciding (unless EOF got there first: a lone CR then). What ended the line lands in LastEolType: EolLf, EolCr, EolCrLf, or EolEof for a final unterminated line. At the very end: nil, io.EOF (and EolNone). A line growing past MaxLineLength without an EOL is an error.

func (*LineReader) SkipBytes

func (lr *LineReader) SkipBytes(nBytes uint64) (uint64, error)

SkipBytes consumes the next nBytes without handing them back — the companion of PeekBytes: peek, decide, skip. It IS ReadBytes with the buffer discarded (literally: that is the implementation, so the semantics can never drift), returning how many bytes went instead.