record_reader

package
Version: v0.32.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 13, 2021 License: MIT Imports: 5 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var ErrIncompleteRecord = errors.New("stream finished in the middle of a record")

Functions

This section is empty.

Types

type Options

type Options struct {
	// The max number of records in a RecordGroup.
	// Higher numbers will use more memory.
	GroupSize int
}

type RecordGroup

type RecordGroup struct {
	// The bytes of all the records in the group.
	Bytes []byte
	// The number of records present in the bytes.
	Count int
}

RecordGroup is a complete chunk of records read from the reader. Its bytes can be safely written.

func ReadAll

func ReadAll(rr RecordReader) (RecordGroup, error)

ReadAll returns a record group that has read every record from the given reader, stopping after the first error.

Note that unlike RecordReader.Next, if an error occurs, the RecordGroup would have non-empty bytes (if there are any records in the reader stream).

type RecordReader

type RecordReader interface {
	// Next returns the bytes for the next chunk of records,
	// or an error. error will never be non-nil
	// if the group bytes are > 0.
	Next() (RecordGroup, error)
}

func NewCsv

func NewCsv(in io.Reader, opts *Options) RecordReader

NewCsv returns a record reader that reports each CSV row as a record. Note that CSV rows are not just delimited via '\n'- there is in fact no way to know whether a character is inline a record without having parsed the file (consider that newlines can be within cells, and quotes are escaped via doubling-up, so even something like `","` may refer to the string `","` within a cell, or `"",""` in the CSV).

func NewJson

func NewJson(in io.Reader, opts *Options) RecordReader

NewJson returns a record reader that parses Apex-flavor JSON and converts it into bytes of JSONLines format in its records.

Note this is not a generic JSON parser- we cannot parse the JSON both because it'd be slower, but especially because it's not really possible to parse JSON like `[{}, {}]` incrementally. So we read lines and figure out when we are between objects.

Becuase of this, only Apex-flavor pretty JSON is supported. Like:

[ {

"x": 1

}, {

"x: 2

} ]

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL