README
¶
go-zerocsv
CSV I/O that doesn't make you pay. go-zerocsv reads CSV more than twice as
fast as encoding/csv and writes with zero heap allocations per record. It
reuses a single small buffer, so a 5M-row file costs the same few kilobytes as
a 100-row one — while the standard library burns 540 MB and 10 million
allocations on the read alone. Same parsing semantics, conformance-fuzzed
against encoding/csv.
Features
- Zero allocations on the hot path: each
Writeand eachNextparses without heap allocation; the buffer and field slices are allocated once and reused. - Field-count validation: like
encoding/csv, records are checked to have a consistent number of fields (auto-detected from the first record) on both read and write, unless disabled withWithFieldsPerRecord(-1). - Zero dependencies: relies exclusively on the Go standard library.
- Fuzz-tested parity: reader output is conformance-fuzzed against
encoding/csvin strict, lazy-quote, and field-count modes. - Flat memory: the reader holds one reusable buffer regardless of input
size, so a 5M-row file costs roughly as much memory as a 100-row one. The
buffer grows only to fit the largest single record, shrinks back afterward,
and can be capped with
WithMaxBuffer.
Installation
Requires Go 1.18 or higher (for any and native fuzzing).
go get github.com/fikrimohammad/go-zerocsv
Quick Start
Writing
package main
import (
"bytes"
"fmt"
zerocsv "github.com/fikrimohammad/go-zerocsv"
)
func main() {
var buf bytes.Buffer
w := zerocsv.NewWriter(&buf)
w.Write(
zerocsv.ColumnString("name"),
zerocsv.ColumnInt(30),
zerocsv.ColumnBool(true),
)
w.Flush()
fmt.Print(buf.String())
// name,30,true
}
Reading
package main
import (
"fmt"
"io"
"log"
"strings"
zerocsv "github.com/fikrimohammad/go-zerocsv"
)
func main() {
input := "name,age\nAlice,30\nBob,40\n"
r := zerocsv.NewReader(strings.NewReader(input))
for {
rec, err := r.Next()
if err == io.EOF {
break
}
if err != nil {
log.Fatal(err)
}
for i := 0; i < rec.Len(); i++ {
fmt.Printf("%d:%s ", i, rec.ValueAt(i))
}
fmt.Println()
}
}
Advanced Usage
Custom options
Options are shared by the writer and reader; each applies only to the side it concerns.
// Tab-separated values.
w := zerocsv.NewWriter(&buf, zerocsv.WithDelimiter('\t'))
// CRLF line endings instead of LF.
w := zerocsv.NewWriter(&buf, zerocsv.WithCRLF())
// Tolerate malformed quoting instead of returning a parse error.
r := zerocsv.NewReader(f, zerocsv.WithLazyQuotes())
// Enforce 3 fields per record, or disable the auto-detected check:
r := zerocsv.NewReader(f, zerocsv.WithFieldsPerRecord(3))
r := zerocsv.NewReader(f, zerocsv.WithFieldsPerRecord(-1)) // variable widths
// Cap the reader's internal buffer so a single oversized record fails with
// ErrRecordTooLarge instead of growing without bound.
r := zerocsv.NewReader(f, zerocsv.WithMaxBuffer(1 << 20))
// The auto-detected count is observable on both reader and writer:
w := zerocsv.NewWriter(buf)
fmt.Println(w.FieldsPerRecord()) // 0 until the first record is written
Typed columns
w.Write(
zerocsv.ColumnInt64(42),
zerocsv.ColumnFloat64(3.14),
zerocsv.ColumnTime(time.Now(), time.RFC3339),
zerocsv.ColumnBytes([]byte("raw")),
)
Zero-allocation writing loop
Reuse one []Column slice across writes so nothing is allocated per record.
row := make([]zerocsv.Column, 3)
for _, v := range values {
row[0] = zerocsv.ColumnString(v.Name)
row[1] = zerocsv.ColumnInt(v.Age)
row[2] = zerocsv.ColumnBool(v.Active)
_ = w.Write(row...)
}
Zero-copy reading
ValueAt returns a zero-copy view into the reader's reused buffer. The value
is valid only until the next Next call. Convert it to a string to own the
data, since string([]byte) copies:
for {
rec, err := r.Next()
if err == io.EOF {
break
}
if err != nil {
log.Fatal(err)
}
name := string(rec.ValueAt(0)) // owned copy, safe to retain
_ = name
}
Benchmarks
Measured on an AMD Ryzen 5 8400F (12 threads), linux/amd64, Go 1.26.5. Run them yourself with:
go test -bench=. -benchmem ./benchmark
B/op is the total bytes allocated per operation — the real memory cost.
Allocation counts alone are misleading: zerocsv's few allocations are one
reused buffer, while encoding/csv's many allocations are small objects that
accumulate into hundreds of megabytes.
Reading — full pass over a whole file
Each iteration parses all n rows with a fresh reader; MB/s is throughput
and B/op is the cumulative allocation for the whole pass.
| Rows | zerocsv ns/op | zerocsv MB/s | zerocsv B/op | zerocsv allocs | stdlib ns/op | stdlib MB/s | stdlib B/op | stdlib allocs |
|---|---|---|---|---|---|---|---|---|
| 100K | 6.54ms | 439.3 | 4.9 KB | 12 | 10.78ms | 266.8 | 10.3 MB | 200,013 |
| 500K | 33.1ms | 434.0 | 4.9 KB | 12 | 51.0ms | 282.1 | 51.5 MB | 1,000,013 |
| 1M | 66.3ms | 433.8 | 4.9 KB | 12 | 101.6ms | 283.0 | 103 MB | 2,000,013 |
| 5M | 330.5ms | 435.0 | 4.9 KB | 12 | 510.6ms | 281.6 | 515 MB | 10,000,013 |
zerocsv allocates a constant ~4.9 KB and 12 objects no matter how many rows
are read: it reuses one small buffer that is compacted between records, so its
B/op stays flat while encoding/csv's grows linearly to 515 MB at 5M rows.
A record larger than the buffer grows it on demand to fit that single record,
and the buffer is trimmed back to ~4 KB once the record has been consumed, so
memory never stays pinned at the peak record size. Buffers up to 256 KB are
kept as-is to avoid grow/trim churn for records in that size band. For hostile
or untrusted input, WithMaxBuffer caps the buffer so a single oversized
record fails with ErrRecordTooLarge instead of growing without bound.
Writing — full pass over a whole file
Each iteration writes all n rows (6 fields each, including int, float64
and RFC3339 time) to io.Discard with a fresh writer. B/op is the
cumulative allocation for the whole pass.
| Rows | zerocsv ns/op | zerocsv B/op | zerocsv allocs | stdlib ns/op | stdlib B/op | stdlib allocs |
|---|---|---|---|---|---|---|
| 100K | 14.1ms | 4.2 KB | 5 | 18.7ms | 4.5 MB | 399,890 |
| 500K | 71.8ms | 4.2 KB | 5 | 95.2ms | 22.8 MB | 1,999,890 |
| 1M | 143.5ms | 4.2 KB | 5 | 191.1ms | 45.7 MB | 3,999,890 |
| 5M | 755.1ms | 4.2 KB | 5 | 1,019ms | 259 MB | 19,999,894 |
zerocsv writes ~1.4x faster than encoding/csv and allocates a constant
4.2 KB (one 4 KB buffer plus a small numeric scratch) regardless of row count,
whereas encoding/csv allocates ~4 objects per record for its strconv
conversions — 259 MB of cumulative allocation for 5M rows.
Reading — per record
| Benchmark | ns/op | B/op | allocs/op |
|---|---|---|---|
zerocsv Next |
54.6 | 19 | 0 |
encoding/csv Read |
116.4 | 114 | 2 |
Writing — per record
| Benchmark | ns/op | B/op | allocs/op |
|---|---|---|---|
zerocsv Write (strings) |
64.7 | 0 | 0 |
encoding/csv Write (strings) |
58.6 | 0 | 0 |
zerocsv Write (mixed types) |
100.7 | 0 | 0 |
encoding/csv Write (mixed types) |
144.3 | 31 | 2 |
zerocsv Write (with time) |
148.8 | 0 | 0 |
encoding/csv Write (with time) |
200.1 | 54 | 3 |
For pre-formatted strings both writers are allocation-free and comparable.
When values need formatting, zerocsv formats into its own scratch buffer
without allocating, so the mixed and time cases stay at 0 B/op and 0 allocs/op
while encoding/csv allocates for every strconv/Format call.
Documentation
Full API documentation and runnable examples are available on pkg.go.dev.
Concurrency
Writer and Reader are stateful and not safe for concurrent use. Use a
separate instance per goroutine.
Limitations
go-zerocsv intentionally does not implement every encoding/csv feature:
- The delimiter is a single byte; multi-rune delimiters are not supported.
- Comment lines are not recognized, and leading whitespace is never trimmed.
- The parser cannot resume after a malformed record: once a parse error
occurs,
Nextkeeps returning it. (Field-count mismatches are the exception and are non-fatal, matchingencoding/csv.)
Contributing
- Fork the repository.
- Create your feature branch (
git checkout -b feature/amazing-feature). - Commit your changes (
git commit -m 'Add some amazing feature'). - Push to the branch (
git push origin feature/amazing-feature). - Open a Pull Request.
Local Quality Checks
Make sure to run the linters and tests locally before submitting your code:
# Run golangci-lint locally
golangci-lint run ./...
# Run tests with the race detector enabled
go test -race -count=1 ./...
# Smoke-test the fuzzers (anchor the pattern; run in the package root)
go test -run='^$' -fuzz='^FuzzReaderConformance$' -fuzztime=30s .
License
Distributed under the MIT License. See LICENSE for more information.
Documentation
¶
Index ¶
- Variables
- type Column
- func ColumnAny(v any) Column
- func ColumnBool(v bool) Column
- func ColumnBytes(b []byte) Column
- func ColumnFloat32(v float32) Column
- func ColumnFloat64(v float64) Column
- func ColumnInt(v int) Column
- func ColumnInt8(v int8) Column
- func ColumnInt16(v int16) Column
- func ColumnInt32(v int32) Column
- func ColumnInt64(v int64) Column
- func ColumnString(s string) Column
- func ColumnTime(t time.Time, layout string) Column
- func ColumnUint(v uint) Column
- func ColumnUint8(v uint8) Column
- func ColumnUint16(v uint16) Column
- func ColumnUint32(v uint32) Column
- func ColumnUint64(v uint64) Column
- func ColumnUintptr(v uintptr) Column
- type ColumnKind
- type Option
- type Reader
- type Record
- type Writer
Constants ¶
This section is empty.
Variables ¶
var ErrBareQuote = errors.New("bare \" in non-quoted field")
ErrBareQuote is returned when a bare '"' appears in a non-quoted field.
var ErrEmptyRecord = errors.New("zerocsv: empty record")
ErrEmptyRecord is returned by Write when no columns are provided.
var ErrFieldCount = errors.New("wrong number of fields")
ErrFieldCount is returned by Next or Write when a record's field count does not match the expected number of fields (see WithFieldsPerRecord). It is non-fatal, like encoding/csv: the record is still returned or written and reading or writing can continue.
var ErrInvalidDelim = errors.New("zerocsv: invalid field delimiter")
ErrInvalidDelim is returned when a delimiter that would corrupt the CSV structure is configured on a Writer or Reader.
var ErrQuote = errors.New("extraneous or missing \" in quoted-field")
ErrQuote is returned for an extraneous or missing '"' in a quoted field.
var ErrRecordTooLarge = errors.New("zerocsv: record larger than the maximum buffer size")
ErrRecordTooLarge is returned by Next when a record is larger than the maximum buffer size configured with WithMaxBuffer and therefore cannot be parsed in memory. Reading cannot continue past the record.
Functions ¶
This section is empty.
Types ¶
type Column ¶
type Column struct {
// contains filtered or unexported fields
}
Column is a tagged, value-typed CSV field. Pass it to Write by value; building and writing columns performs no heap allocation for the typed constructors. ColumnAny is the exception and may allocate.
func ColumnAny ¶
ColumnAny returns a Column containing v, rendered with fmt.Sprint when the record is written. Unlike the typed constructors, this may allocate.
func ColumnBool ¶
ColumnBool returns a Column containing v, written as "true" or "false".
func ColumnBytes ¶
ColumnBytes returns a Column containing b. The slice is written as-is, with no copy.
func ColumnFloat32 ¶
ColumnFloat32 returns a Column containing v.
func ColumnFloat64 ¶
ColumnFloat64 returns a Column containing v.
func ColumnTime ¶
ColumnTime returns a Column containing t, formatted with layout when the record is written.
func ColumnUintptr ¶
ColumnUintptr returns a Column containing v.
type Option ¶
type Option func(*options)
Option configures a Writer or Reader at construction time.
func WithCRLF ¶
func WithCRLF() Option
WithCRLF makes the Writer end each record with "\r\n" instead of "\n".
func WithDelimiter ¶
WithDelimiter sets the field delimiter, for example ',' for CSV, '\t' for TSV, or ';' for semicolon-separated values. The NUL byte, '"', '\r' and '\n' are rejected: an invalid delimiter marks a Writer or Reader as failed, and Next, Write, WriteAll, Flush and Error report the error.
func WithFieldsPerRecord ¶ added in v1.1.0
WithFieldsPerRecord sets the expected number of fields per record, applying to both the Reader and the Writer.
If n is positive, Next and Write require every record to have exactly n fields and return ErrFieldCount otherwise. If n is 0, the count is taken from the first record and enforced on all subsequent ones, like encoding/csv's default. If n is negative, no check is made and records may have a variable number of fields. Blank lines read by the Reader never take part in the check.
Like encoding/csv, ErrFieldCount is non-fatal: the mismatched record is still returned (Reader) or written (Writer), and reading or writing may continue.
func WithLazyQuotes ¶
func WithLazyQuotes() Option
WithLazyQuotes makes the Reader tolerate malformed quoting: a bare '"' in an unquoted field, or a non-doubled '"' in a quoted field, is treated as a literal character instead of returning a parse error.
func WithMaxBuffer ¶ added in v1.1.0
WithMaxBuffer caps the Reader's internal buffer at n bytes. A record larger than n cannot be parsed in memory, so Next returns ErrRecordTooLarge rather than letting the buffer grow without bound. A non-positive n means no limit (the default).
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader reads CSV records with zero allocations per record.
The input buffer and the field slice are allocated once in NewReader and reused for the lifetime of the Reader. The fields of a Record returned by Next are zero-copy views into that buffer, so they are valid only until the next call to Next.
func NewReader ¶
NewReader returns a Reader that parses CSV records from r, applying opts. An invalid delimiter marks the Reader as failed; Next and Error report the error.
func (*Reader) Error ¶ added in v0.2.0
Error returns the first error encountered while reading, or nil if none has occurred. io.EOF is normal termination and is not treated as an error, so Error returns nil after a record stream has been read to completion. A Reader configured with an invalid delimiter is failed from the start.
func (*Reader) FieldsPerRecord ¶ added in v1.1.0
FieldsPerRecord returns the expected number of fields per record. It reflects the value configured with WithFieldsPerRecord: with auto-detection (the default) it is 0 until the first record is read, after which it is the field count learned from that record; a negative value means no check is in effect.
func (*Reader) Next ¶
Next parses and returns the next record, or io.EOF when no records remain. It returns ErrBareQuote or ErrQuote for malformed quoting, unless WithLazyQuotes is in effect, and propagates any error from the underlying reader.
If a field count is in effect (see WithFieldsPerRecord) and the record has a different number of fields, Next returns the record along with ErrFieldCount. Like encoding/csv, this error is non-fatal: the returned Record is valid and reading may continue.
The returned Record is reused by the Reader and overwritten by each call to Next. Its fields are zero-copy views into the Reader's internal buffer and are valid only until the next call to Next, like encoding/csv's ReuseRecord mode. Copy a field with Record.Copy, or convert it to a string, if it must outlive the read loop.
type Record ¶
type Record struct {
// contains filtered or unexported fields
}
Record is a parsed CSV record. The Reader reuses a single Record and overwrites it on every call to Next, so a Record is valid only until the Reader is advanced again. Its fields are zero-copy views into the Reader's internal buffer; use Copy to retain the values beyond the next call to Next.
func (*Record) Copy ¶ added in v0.2.0
Copy returns the record's fields as owned strings. The returned slice and its strings remain valid after subsequent calls to Next.
func (*Record) ValueAt ¶
ValueAt returns the field at index idx as a zero-copy view into the Reader's internal buffer. The slice is valid only until the next call to Next on the Reader that produced the Record. Convert it to a string to get an owned copy, since string(v) always copies. It panics if idx is negative or not less than Len.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer writes CSV records with zero allocations per write.
The bufio.Writer and the numeric scratch buffer are allocated once in NewWriter and reused for the lifetime of the Writer. Write/WriteAll perform no heap allocations on the hot path as long as the caller reuses a []Column slice (e.g. Write(row...)) rather than passing freshly constructed variadic args.
func NewWriter ¶ added in v0.2.0
NewWriter returns a Writer that writes CSV records to w, applying opts. If w is already a *bufio.Writer with a buffer at least as large as the default (4096 bytes), it is reused directly rather than being wrapped again.
func (*Writer) Error ¶
Error returns the first error encountered during Write, WriteAll or Flush, or nil if none has occurred.
func (*Writer) FieldsPerRecord ¶ added in v1.1.0
FieldsPerRecord returns the expected number of fields per record, as configured with WithFieldsPerRecord. With auto-detection (the default) it is 0 until the first record is written, after which it is the field count learned from that record; a negative value means no check is in effect.
func (*Writer) Flush ¶
Flush writes any buffered data to the underlying writer and returns the first error encountered during Write, WriteAll or Flush, if any.
func (*Writer) Write ¶
Write writes cols as a single CSV record to the underlying writer. It returns ErrEmptyRecord if cols is empty, or the first error encountered while writing the record.
If a field count is in effect (see WithFieldsPerRecord) and cols has a different number of fields, Write returns ErrFieldCount. Like encoding/csv, the error is non-fatal: the record is still written and writing may continue.
Source Files
Directories