zerocsv

package module
v1.2.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 19, 2026 License: MIT Imports: 10 Imported by: 0

README

go-zerocsv

Go Reference CI License

Zero-allocation, flat-memory CSV I/O for Go. go-zerocsv reads CSV faster than encoding/csv and writes with zero heap allocations per record. It reuses a single compacting buffer, so a 5M-row file maintains a constant ~5 KB memory footprint—compared to ~540 MB and 10 million heap allocations with standard encoding/csv (and 140 MB with ReuseRecord = true). Same parsing semantics, conformance-fuzzed against encoding/csv.

Features

  • Zero-allocation hot paths: Write() and lazy Read() parse without heap allocations via buffer reuse.
  • Flat-memory streaming: Reusable compacting buffer maintains a ~5 KB footprint regardless of input size.
  • Typed scanning: Record.Scan(&id, &name, &score) parses fields directly from byte slices without string allocations.
  • Custom types (Scanner & Valuer): Implement FieldScanner and FieldValuer for zero-allocation domain parsing/formatting.
  • Value-typed columns: Compact 48-byte Column struct passes fields by value without interface boxing.
  • Fuzz-tested parity: Conformance-fuzzed against encoding/csv (strict, lazy-quotes, and field counts). Zero dependencies.

Installation

Requires Go 1.18 or higher (for any and native fuzzing).

go get github.com/fikrimohammad/go-zerocsv

Quick Start

Writing
package main

import (
	"bytes"
	"fmt"

	zerocsv "github.com/fikrimohammad/go-zerocsv"
)

func main() {
	var buf bytes.Buffer
	w := zerocsv.NewWriter(&buf)

	w.Write(
		zerocsv.ColumnString("name"),
		zerocsv.ColumnInt(30),
		zerocsv.ColumnBool(true),
	)
	w.Flush()

	fmt.Print(buf.String())
	// name,30,true
}
Reading (Typed & Zero-Allocation)
package main

import (
	"fmt"
	"io"
	"log"
	"strings"

	zerocsv "github.com/fikrimohammad/go-zerocsv"
)

func main() {
	input := "id,name,score,active\n1,Alice,3.14,true\n2,Bob,2.50,false\n"
	r := zerocsv.NewReader(strings.NewReader(input))

	for {
		rec, err := r.Read()
		if err == io.EOF {
			break
		}
		if err != nil {
			log.Fatal(err)
		}
		if rec.IsFirst() {
			continue // skip header
		}

		var (
			id     int
			name   string
			score  float64
			active bool
		)
		if err := rec.Scan(&id, &name, &score, &active); err != nil {
			log.Fatal(err)
		}
		fmt.Printf("id=%d name=%s score=%.2f active=%t\n", id, name, score, active)
	}
}

Advanced Usage

Eager Reading with ReadAll()

To eagerly load all remaining CSV records into safe, owned Records:

records, err := r.ReadAll()
if err != nil {
	log.Fatal(err)
}
for _, rec := range records {
	fmt.Println(rec.Strings())
}
Custom options

Options are shared by the writer and reader; each applies only to the side it concerns.

// Tab-separated values.
w := zerocsv.NewWriter(&buf, zerocsv.WithDelimiter('\t'))

// CRLF line endings instead of LF.
w := zerocsv.NewWriter(&buf, zerocsv.WithCRLF())

// Tolerate malformed quoting instead of returning a parse error.
r := zerocsv.NewReader(f, zerocsv.WithLazyQuotes())

// Enforce 3 fields per record, or disable the auto-detected check:
r := zerocsv.NewReader(f, zerocsv.WithFieldsPerRecord(3))
r := zerocsv.NewReader(f, zerocsv.WithFieldsPerRecord(-1)) // variable widths

// Cap the reader's internal buffer so a single oversized record fails with
// ErrRecordTooLarge instead of growing without bound.
r := zerocsv.NewReader(f, zerocsv.WithMaxBuffer(1 << 20))

// The auto-detected count is observable on both reader and writer:
w := zerocsv.NewWriter(&buf)
fmt.Println(w.FieldsPerRecord()) // 0 until the first record is written
Typed columns (Writer)
w.Write(
	zerocsv.ColumnInt64(42),
	zerocsv.ColumnFloat64(3.14),
	zerocsv.ColumnBool(true),
	zerocsv.ColumnString(time.Now().Format(time.RFC3339)),
	zerocsv.ColumnBytes([]byte("raw-bytes")),
)
Zero-allocation writing loop

Reuse one []Column slice across writes so nothing is allocated per record.

row := make([]zerocsv.Column, 3)
for _, v := range values {
	row[0] = zerocsv.ColumnString(v.Name)
	row[1] = zerocsv.ColumnInt(v.Age)
	row[2] = zerocsv.ColumnBool(v.Active)
	_ = w.Write(row...)
}
Custom Types (Scanner & Valuer)

Custom domain types can control their CSV parsing and formatting with zero allocations by implementing FieldScanner and FieldValuer:

type Date time.Time

// Reader: zero-allocation parsing from raw field bytes
func (d *Date) ScanCSV(field []byte) error {
	t, err := time.Parse("2006-01-02", string(field))
	if err != nil {
		return err
	}
	*d = Date(t)
	return nil
}

// Writer: zero-allocation formatting into writer scratch buffer
func (d Date) AppendCSV(dst []byte) ([]byte, error) {
	return time.Time(d).AppendFormat(dst, "2006-01-02"), nil
}

Usage:

// Scanning into custom type
var d Date
err := rec.Scan(&id, &name, &d)

// Writing custom type
err := w.Write(zerocsv.ColumnString("alice"), zerocsv.ColumnValuer(d))

Benchmarks

Measured on an AMD Ryzen 5 8400F (12 threads), linux/amd64, Go 1.26.5. Run them yourself with:

go test -bench=. -benchmem ./benchmark

B/op is the total bytes allocated per operation — the real memory cost. Allocation counts alone are misleading: zerocsv's few allocations are one reused buffer, while encoding/csv's many allocations are small objects that accumulate into hundreds of megabytes.

Reading — full pass over a whole file

Each iteration parses all n rows with a fresh reader; MB/s is throughput and B/op is the cumulative allocation for the whole pass.

Rows zerocsv ns/op zerocsv MB/s zerocsv B/op zerocsv allocs stdlib (default) ns/op stdlib B/op stdlib allocs stdlib (ReuseRecord) ns/op stdlib (ReuseRecord) B/op stdlib (ReuseRecord) allocs
100K 6.45ms 445.5 5.0 KB 12 11.33ms 10.8 MB 200,013 8.75ms 2.8 MB 100,014
500K 32.6ms 440.4 5.0 KB 12 54.54ms 54.0 MB 1,000,013 43.29ms 14.0 MB 500,014
1M 66.6ms 431.5 5.0 KB 12 109.2ms 108 MB 2,000,013 82.20ms 28.0 MB 1,000,014
5M 334.7ms 429.5 5.0 KB 12 530.4ms 540 MB 10,000,013 415.9ms 140 MB 5,000,014

zerocsv allocates a constant ~5.0 KB and 12 objects no matter how many rows are read: it reuses one small buffer that is compacted between records, so its B/op stays flat while encoding/csv's grows linearly to 540 MB at 5M rows (even with ReuseRecord = true, which reuses the outer slice header but still allocates 140 MB across 5 million field strings). A record larger than the buffer grows it on demand to fit that single record, and the buffer is trimmed back to ~4 KB once the record has been consumed, so memory never stays pinned at the peak record size. Buffers up to 256 KB are kept as-is to avoid grow/trim churn for records in that size band. For hostile or untrusted input, WithMaxBuffer caps the buffer so a single oversized record fails with ErrRecordTooLarge instead of growing without bound.

Writing — full pass over a whole file

Each iteration writes all n rows (6 fields each, including int, float64 and RFC3339 time) to io.Discard with a fresh writer. B/op is the cumulative allocation for the whole pass.

Rows zerocsv ns/op zerocsv B/op zerocsv allocs stdlib ns/op stdlib B/op stdlib allocs
100K 13.6ms 4.3 KB 6 20.1ms 4.7 MB 399,890
500K 67.3ms 4.3 KB 6 101.0ms 24.0 MB 1,999,892
1M 137.8ms 4.3 KB 6 204.4ms 47.9 MB 3,999,895
5M 708.3ms 4.3 KB 6 1,119ms 272 MB 19,999,917

zerocsv writes ~1.5x faster than encoding/csv and allocates a constant 4.3 KB (one 4 KB buffer plus a small numeric scratch) regardless of row count, whereas encoding/csv allocates ~4 objects per record for its strconv conversions — 272 MB of cumulative allocation for 5M rows.

Reading — per record
Benchmark ns/op B/op allocs/op
zerocsv Read 50.9 19 0
encoding/csv Read 115.2 114 2
encoding/csv Read (ReuseRecord = true) 84.6 35 1
Writing — per record
Benchmark ns/op B/op allocs/op
zerocsv Write (strings) 63.8 0 0
encoding/csv Write (strings) 58.2 0 0
zerocsv Write (mixed types) 95.7 0 0
encoding/csv Write (mixed types) 142.5 31 2
zerocsv Write (with time) 142.4 0 0
encoding/csv Write (with time) 200.5 54 3

For pre-formatted strings both writers are allocation-free and comparable. When values need formatting, zerocsv formats into its own scratch buffer without allocating, so the mixed and time cases stay at 0 B/op and 0 allocs/op while encoding/csv allocates for every strconv/Format call.

Real-World Impact: GC Pressure & Memory Limits (GOMEMLIMIT)

In containerized environments (Kubernetes, AWS ECS, Lambda), memory allocations trigger garbage collection (GC) cycles and CPU throttling. Below is the performance of streaming 3M rows under a 150 MiB memory ceiling (GOMEMLIMIT=150MiB):

1. Reading (Streaming Ingestion)
Reader Ingestion Time Cumulative Heap Alloc GC Cycles Triggered Behavior
go-zerocsv 267ms ~5 KB 0 Stable, flat memory
encoding/csv (ReuseRecord=true) 2,392ms 138 MB 14,594 GC Thrashing (9x slowdown)
encoding/csv (default) 1,941ms 366 MB 10,517 GC Thrashing (7x slowdown)

Because standard encoding/csv allocates millions of transient heap strings, the Go runtime repeatedly halts execution to collect dead strings to stay under the memory ceiling. go-zerocsv remains allocation-free and runs at full speed.

2. Writing (Batch Exporting with Mixed Types)
Writer Export Time Cumulative Heap Alloc GC Cycles Triggered Behavior
go-zerocsv 414ms ~4 KB 0 Direct buffer formatting
encoding/csv (with strconv/Format) 604ms 153 MB 43 GC overhead from conversions

go-zerocsv.Writer formats typed primitives (int, float, time.Time, bool) directly into its internal scratch buffer via value-typed Columns, avoiding the heap allocations required by manual strconv and time.Format calls.

Documentation

Full API documentation and runnable examples are available on pkg.go.dev.

Concurrency

Writer and Reader are stateful and not safe for concurrent use. Use a separate instance per goroutine.

Limitations & Memory Model

go-zerocsv intentionally makes trade-offs to achieve zero allocations:

  • Buffer aliasing on lazy Read(): Field byte slices in a lazy Record point directly into the reader's internal buffer. Data must be consumed (via Scan(), Bytes(), or String()) before the next call to Read(). Use ReadAll() if you need records that safely own their storage.
  • Single-byte delimiters: The delimiter is a single byte (e.g. ,, \t, |); multi-rune delimiters are not supported.
  • No comments or whitespace trimming: Comment lines (#) are not recognized, and leading whitespace is not automatically stripped.
  • Non-resumable parse errors: Once a fatal parse error occurs, Read and ReadAll continue returning it. (Field-count mismatches are non-fatal, matching encoding/csv).

Roadmap

  • Multi-byte & string delimiter support: Support for Unicode runes ('§', '·') and multi-character string delimiters ("||", "~|~") while retaining zero-allocation streaming.

Contributing

  1. Fork the repository.
  2. Create your feature branch (git checkout -b feature/amazing-feature).
  3. Commit your changes (git commit -m 'Add some amazing feature').
  4. Push to the branch (git push origin feature/amazing-feature).
  5. Open a Pull Request.
Local Quality Checks

Make sure to run the linters and tests locally before submitting your code:

# Run golangci-lint locally
golangci-lint run ./...

# Run tests with the race detector enabled
go test -race -count=1 ./...

# Smoke-test the fuzzers (anchor the pattern; run in the package root)
go test -run='^$' -fuzz='^FuzzReaderConformance$' -fuzztime=30s .

License

Distributed under the MIT License. See LICENSE for more information.

Documentation

Index

Constants

This section is empty.

Variables

View Source
var ErrBareQuote = errors.New("bare \" in non-quoted field")

ErrBareQuote is returned when a bare '"' appears in a non-quoted field.

View Source
var ErrEmptyRecord = errors.New("zerocsv: empty record")

ErrEmptyRecord is returned by Write when no columns are provided.

View Source
var ErrFieldCount = errors.New("wrong number of fields")

ErrFieldCount is returned by Read or Write when a record's field count does not match the expected number of fields (see WithFieldsPerRecord). It is non-fatal, like encoding/csv: the record is still returned or written and reading or writing can continue.

View Source
var ErrInvalidDelim = errors.New("zerocsv: invalid field delimiter")

ErrInvalidDelim is returned when a delimiter that would corrupt the CSV structure is configured on a Writer or Reader.

View Source
var ErrQuote = errors.New("extraneous or missing \" in quoted-field")

ErrQuote is returned for an extraneous or missing '"' in a quoted field.

View Source
var ErrRecordTooLarge = errors.New("zerocsv: record larger than the maximum buffer size")

ErrRecordTooLarge is returned by Read when a record is larger than the maximum buffer size configured with WithMaxBuffer and therefore cannot be parsed in memory. Reading cannot continue past the record.

Functions

This section is empty.

Types

type Column

type Column struct {
	// contains filtered or unexported fields
}

Column is a tagged, value-typed CSV field. Pass it to Write by value; building and writing columns performs no heap allocation for all constructors.

func ColumnBool

func ColumnBool(v bool) Column

ColumnBool returns a Column containing v, written as "true" or "false".

func ColumnBytes

func ColumnBytes(b []byte) Column

ColumnBytes returns a Column containing b. The slice is written as-is, with no copy and no heap allocation.

func ColumnFloat32

func ColumnFloat32(v float32) Column

ColumnFloat32 returns a Column containing v.

func ColumnFloat64

func ColumnFloat64(v float64) Column

ColumnFloat64 returns a Column containing v.

func ColumnInt

func ColumnInt(v int) Column

ColumnInt returns a Column containing v.

func ColumnInt8

func ColumnInt8(v int8) Column

ColumnInt8 returns a Column containing v.

func ColumnInt16

func ColumnInt16(v int16) Column

ColumnInt16 returns a Column containing v.

func ColumnInt32

func ColumnInt32(v int32) Column

ColumnInt32 returns a Column containing v.

func ColumnInt64

func ColumnInt64(v int64) Column

ColumnInt64 returns a Column containing v.

func ColumnString

func ColumnString(s string) Column

ColumnString returns a Column containing s.

func ColumnUint

func ColumnUint(v uint) Column

ColumnUint returns a Column containing v.

func ColumnUint8

func ColumnUint8(v uint8) Column

ColumnUint8 returns a Column containing v.

func ColumnUint16

func ColumnUint16(v uint16) Column

ColumnUint16 returns a Column containing v.

func ColumnUint32

func ColumnUint32(v uint32) Column

ColumnUint32 returns a Column containing v.

func ColumnUint64

func ColumnUint64(v uint64) Column

ColumnUint64 returns a Column containing v.

func ColumnUintptr

func ColumnUintptr(v uintptr) Column

ColumnUintptr returns a Column containing v.

func ColumnValuer added in v1.2.0

func ColumnValuer(v FieldValuer) Column

ColumnValuer returns a Column containing v, which appends its CSV representation with zero heap allocations.

func (Column) Kind added in v1.2.0

func (c Column) Kind() ColumnKind

Kind returns the ColumnKind of c.

type ColumnKind

type ColumnKind uint8

ColumnKind identifies the payload type stored in a Column.

const (
	ColumnKindString ColumnKind = iota
	ColumnKindBytes
	ColumnKindInt
	ColumnKindUint
	ColumnKindFloat
	ColumnKindFloat32
	ColumnKindBool
	ColumnKindValuer
)

type FieldScanner added in v1.2.0

type FieldScanner interface {
	ScanCSV(field []byte) error
}

FieldScanner is implemented by custom types that can scan their value directly from a raw CSV field byte slice.

type FieldValuer added in v1.2.0

type FieldValuer interface {
	AppendCSV(dst []byte) ([]byte, error)
}

FieldValuer is implemented by custom types that can append their CSV field representation directly into a scratch buffer with zero heap allocations.

type Option

type Option func(*options)

Option configures a Writer or Reader at construction time.

func WithCRLF

func WithCRLF() Option

WithCRLF makes the Writer end each record with "\r\n" instead of "\n".

func WithDelimiter

func WithDelimiter(d byte) Option

WithDelimiter sets the field delimiter, for example ',' for CSV, '\t' for TSV, or ';' for semicolon-separated values. Only single ASCII bytes are supported. The NUL byte, '"', '\r', '\n' and any byte above '\x7f' are rejected: an invalid delimiter marks a Writer or Reader as failed, and Read, ReadAll, Write, WriteAll, Flush and Error report the error.

func WithFieldsPerRecord added in v1.1.0

func WithFieldsPerRecord(n int) Option

WithFieldsPerRecord sets the expected number of fields per record, applying to both the Reader and the Writer.

If n is positive, Read, ReadAll and Write require every record to have exactly n fields and return ErrFieldCount otherwise. If n is 0, the count is taken from the first record and enforced on all subsequent ones, like encoding/csv's default. If n is negative, no check is made and records may have a variable number of fields. Blank lines read by the Reader never take part in the check.

Like encoding/csv, ErrFieldCount is non-fatal: the mismatched record is still returned (Reader) or written (Writer), and reading or writing may continue.

func WithLazyQuotes

func WithLazyQuotes() Option

WithLazyQuotes makes the Reader tolerate malformed quoting: a bare '"' in an unquoted field, or a non-doubled '"' in a quoted field, is treated as a literal character instead of returning a parse error.

func WithMaxBuffer added in v1.1.0

func WithMaxBuffer(n int) Option

WithMaxBuffer caps the Reader's internal buffer at n bytes. A record larger than n cannot be parsed in memory, so Read returns ErrRecordTooLarge rather than letting the buffer grow without bound. A non-positive n means no limit (the default).

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader reads CSV records with zero allocations per record.

The input buffer and the field slice are allocated once in NewReader and reused for the lifetime of the Reader. Read returns records lazily one at a time with zero heap allocations, while ReadAll eagerly reads all remaining records into a slice of owned Records.

func NewReader

func NewReader(r io.Reader, opts ...Option) *Reader

NewReader returns a Reader that parses CSV records from r, applying opts. An invalid delimiter marks the Reader as failed; Read, ReadAll and Error report the error.

func (*Reader) Error added in v0.2.0

func (r *Reader) Error() error

Error returns the first error encountered while reading, or nil if none has occurred. io.EOF is normal termination and is not treated as an error, so Error returns nil after a record stream has been read to completion. A Reader configured with an invalid delimiter is failed from the start.

func (*Reader) FieldsPerRecord added in v1.1.0

func (r *Reader) FieldsPerRecord() int

FieldsPerRecord returns the expected number of fields per record. It reflects the value configured with WithFieldsPerRecord: with auto-detection (the default) it is 0 until the first record is read, after which it is the field count learned from that record; a negative value means no check is in effect.

func (*Reader) Read added in v1.2.0

func (r *Reader) Read() (Record, error)

Read reads one record from r. The returned Record provides zero-allocation access to fields via Scan, or safe access via String, Bytes, and Strings.

If the record has an unexpected number of fields (see WithFieldsPerRecord), Read returns the Record along with ErrFieldCount. Like encoding/csv, this error is non-fatal: the record is usable and subsequent calls to Read continue reading the stream.

If no more records remain, Read returns a zero Record with io.EOF.

func (*Reader) ReadAll added in v1.2.0

func (r *Reader) ReadAll() ([]Record, error)

ReadAll reads all remaining records from r into a slice of Records. Each Record in the returned slice owns its field data and remains valid indefinitely. A successful call returns err == nil (io.EOF is treated as normal completion).

If an error (such as ErrFieldCount or a parse error) is encountered, ReadAll stops immediately and returns the records read so far along with the error, matching encoding/csv.ReadAll semantics.

type Record

type Record struct {
	// contains filtered or unexported fields
}

Record is a single CSV record parsed by Reader.

A Record obtained from Read() provides safe, encapsulated access to the parsed fields. To achieve zero heap allocations on the hot path, field data is accessed via Scan (for typed values or reusable []byte buffers), String, Bytes, or Strings.

func (Record) Bytes added in v1.2.0

func (rec Record) Bytes(idx int, dst []byte) []byte

Bytes copies the field at index idx into dst and returns the resulting slice. If cap(dst) is large enough, Bytes performs zero heap allocations. It panics if idx is out of range [0, Len()).

func (Record) Error added in v1.2.0

func (rec Record) Error() error

Error returns the non-fatal error associated with this record (e.g. ErrFieldCount), or nil if the record had no errors.

func (Record) IsFirst added in v1.2.0

func (rec Record) IsFirst() bool

IsFirst reports whether this record is the first non-blank record read from the stream, useful for header detection.

func (Record) Len

func (rec Record) Len() int

Len returns the number of fields in the record.

func (Record) Scan added in v1.2.0

func (rec Record) Scan(dst ...any) error

Scan parses the record's fields into the destination pointers, one per field, in order.

Supported destination types:

  • *string: copies field as a string
  • *[]byte: copies field into caller's slice capacity (0 allocs if capacity suffices)
  • *bool: parses boolean ("true", "false", "1", "0", ...) in-place (0 allocs)
  • *int, *int8, *int16, *int32, *int64: parses integer in-place (0 allocs)
  • *uint, *uint8, *uint16, *uint32, *uint64, *uintptr: parses unsigned integer in-place (0 allocs)
  • *float32, *float64: parses float in-place (0 allocs)
  • FieldScanner: delegates parsing to custom ScanCSV method (0 allocs)

Scan returns an error if the number of destinations does not match Len(), if any destination pointer is nil, or if parsing fails.

func (Record) String added in v1.2.0

func (rec Record) String(idx int) string

String returns the field at index idx as a string. It panics if idx is out of range [0, Len()).

func (Record) Strings added in v1.2.0

func (rec Record) Strings() []string

Strings returns the record's fields as a new slice of strings.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer writes CSV records with zero allocations per write.

The bufio.Writer and the numeric scratch buffer are allocated once in NewWriter and reused for the lifetime of the Writer. Write/WriteAll perform no heap allocations on the hot path as long as the caller reuses a []Column slice (e.g. Write(row...)) rather than passing freshly constructed variadic args.

func NewWriter added in v0.2.0

func NewWriter(w io.Writer, opts ...Option) *Writer

NewWriter returns a Writer that writes CSV records to w, applying opts. If w is already a *bufio.Writer with a buffer at least as large as the default (4096 bytes), it is reused directly rather than being wrapped again.

func (*Writer) Error

func (w *Writer) Error() error

Error returns the first error encountered during Write, WriteAll or Flush, or nil if none has occurred.

func (*Writer) FieldsPerRecord added in v1.1.0

func (w *Writer) FieldsPerRecord() int

FieldsPerRecord returns the expected number of fields per record, as configured with WithFieldsPerRecord. With auto-detection (the default) it is 0 until the first record is written, after which it is the field count learned from that record; a negative value means no check is in effect.

func (*Writer) Flush

func (w *Writer) Flush() error

Flush writes any buffered data to the underlying writer and returns the first error encountered during Write, WriteAll or Flush, if any.

func (*Writer) Write

func (w *Writer) Write(cols ...Column) error

Write writes cols as a single CSV record to the underlying writer. It returns ErrEmptyRecord if cols is empty, or the first error encountered while writing the record.

If a field count is in effect (see WithFieldsPerRecord) and cols has a different number of fields, Write returns ErrFieldCount. Like encoding/csv, the error is non-fatal: the record is still written and writing may continue.

func (*Writer) WriteAll

func (w *Writer) WriteAll(rows [][]Column) error

WriteAll writes each row of rows as a CSV record to the underlying writer, flushes any buffered data, and returns the first error encountered, if any.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL