csv

Documentation

Overview

Package csv reads and writes comma separated values.

The reader returns a Table, which is a schema and a column for each field, rather than a frame. That is what keeps this package underneath kuma in the import graph, so kuma.ReadCSV can be a few lines over the top of it, and so anything that wants Arrow columns out of a text file can use this on its own without pulling in a query engine.

Schema inference

With no types given, the reader looks at the first Options.InferRows rows and works out what each column holds. The types it will infer are int64, float64, bool and string, and nothing else. A value that reads as an integer is an integer, one that reads as a number but not an integer is a float, and the words true and false are a boolean. A column that holds integers in some rows and floats in others is a float column. Anything else mixed together is a string column, which is the type that can always hold what turned up.

Inference is deliberately narrower than what a cast will accept. The single letters t and f are a boolean to strconv.ParseBool and are a string here, because a column of chemical symbols should not become a boolean column on the strength of the row that says F. Naming the type in Options.Types gets the permissive reading, since a caller who writes the type has said what the column is.

Dates and timestamps are not inferred. Text to timestamp parsing is a milestone 8 job, so for now a date column reads as a string and is cast afterwards.

Missing values

An empty field is a missing value in every column, including a string column, because a CSV file cannot tell an empty string apart from an absent one. Options.NullValues replaces that rule with a list of the caller's own, so a file that writes NA can say so, and a file that really does mean the empty string can pass a list that does not contain it.

Reading part of a file

Options.Columns names the columns to read and drops the rest. A column that is not named is not inferred, not parsed and not held, which on a wide file is most of the work, and the columns come out in the order they were asked for rather than in the order the file has them.

Writing

Write takes a table back out to a file. Values go out as they are stored, so a column of numbers is formatted into the output buffer and never becomes a column of strings along the way, and a field is quoted when it holds the delimiter, a quote or a line ending and left alone when it does not.

Reading back what was written gives what went in, with one thing that cannot work: an empty string comes back as a missing value, because the file has no way to say which one it meant. WriteOptions.NullValue is the way out when the difference matters.

Speed

This is the reference reader and it is not the fast one. It is encoding/csv, which is correct about quotes and embedded newlines and both line endings, with one value parsed per field. The vectorized reader from document 05 is checked against what is here, and where the two disagree this one is right by definition.

The writer is not encoding/csv's. That one takes a []string, which would mean a string allocated for every value in the table on the way past, so this one formats into a buffer of its own and hands whole blocks to the writer underneath.

Stability: tier 1, stable.

Index

• Constants
• Variables
• func Write(w io.Writer, t *Table, opts *WriteOptions) error
• func WriteFile(path string, t *Table, opts *WriteOptions) error
• type Options
• type Table
  • func Read(r io.Reader, opts *Options) (*Table, error)
  • func ReadFile(path string, opts *Options) (*Table, error)
• type ValueError
  • func (e *ValueError) Error() string
  • func (e *ValueError) Unwrap() []error
• type WriteOptions

Examples

• Read
• Read (Columns)
• Read (NoHeader)
• Read (Types)
• ValueError
• Write
• Write (Options)

Constants

const (
	// DefaultInferRows is how many rows the reader looks at before deciding
	// what the columns hold. It is large enough that a column of integers with
	// a float somewhere down the file is usually caught, and small enough that
	// deciding costs nothing next to reading.
	DefaultInferRows = 1024

	// DefaultChunkSize is how many rows go into one chunk of a column. A column
	// arrives as a list of chunks rather than one buffer, so reading a file
	// larger than memory allows never asks for one allocation the size of the
	// file.
	DefaultChunkSize = 65536
)

Default values for the options that have one.

Variables

var (
	// ErrNoData is returned when there is nothing to read at all, not even a
	// header. An input with a header and no rows is a table with no rows and
	// is not an error.
	ErrNoData = errors.New("no data")

	// ErrNames is returned when Options.Names has a different number of names
	// from the number of fields in the file, or when two columns end up with
	// the same name.
	ErrNames = errors.New("bad column names")

	// ErrNoColumn is returned when Options.Types names a column the file does
	// not have.
	ErrNoColumn = errors.New("no such column")

	// ErrUnsupportedType is returned when Options.Types names a type this
	// reader cannot parse text into, such as a list or a struct, and when a
	// column being written holds a type that has no text form at all.
	ErrUnsupportedType = errors.New("unsupported column type")

	// ErrDelimiter is returned when the delimiter or the comment character
	// cannot do the job, which means a quote, a line ending, a rune that is
	// not one, or the same character for both.
	ErrDelimiter = errors.New("invalid delimiter")

	// ErrTable is returned when a table cannot be written because it does not
	// hold together: a schema and a list of columns of different lengths, or
	// columns with different numbers of rows.
	ErrTable = errors.New("malformed table")

	// ErrValue is what a value that will not parse unwraps to. The error
	// itself is a *ValueError, which says which line and which column.
	ErrValue = errors.New("bad value")

	// ErrFieldCount is returned when a row has a different number of fields
	// from the one before it. ErrQuote is returned for a quote where the
	// format does not allow one, which Options.LazyQuotes turns off.
	//
	// Both come from [encoding/csv] and are named here so that a caller has
	// one package to ask about an error rather than two. The error itself is
	// an *encoding/csv.ParseError, which says which line.
	ErrFieldCount = stdcsv.ErrFieldCount
	ErrQuote      = stdcsv.ErrQuote
)

The errors this package returns, all comparable with errors.Is.

Functions

func Write

func Write(w io.Writer, t *Table, opts *WriteOptions) error

Write writes the table as a comma separated file.

The columns go out as they are stored, so a column of numbers is formatted into the output buffer and never becomes a column of strings on the way. A value is quoted when it holds the delimiter, a quote or a line ending, and left alone when it does not, which is the rule encoding/csv writes by and the rule this package reads back.

A type with no text of its own, which today means the timestamps and the dates, is cast to a string first and written from that. That costs a copy of the column and it is the only case that does.

Example

package main

import (
	"fmt"
	"os"
	"strings"

	"github.com/tamnd/kuma/csv"
)

func main() {
	in := "sym,qty,px\nAAPL,100,182.5\nMSFT,,411.2\n"

	t, err := csv.Read(strings.NewReader(in), nil)
	if err != nil {
		fmt.Println(err)
		return
	}

	// The table goes back out as it came in. A missing quantity is an empty
	// field again, since that is what the file said in the first place.
	if err := csv.Write(os.Stdout, t, nil); err != nil {
		fmt.Println(err)
	}
}

Output:
sym,qty,px
AAPL,100,182.5
MSFT,,411.2

Example (Options)

package main

import (
	"fmt"
	"os"
	"strings"

	"github.com/tamnd/kuma/csv"
)

func main() {
	in := "sym,qty,px\nAAPL,100,182.5\nMSFT,,411.2\n"

	t, err := csv.Read(strings.NewReader(in), nil)
	if err != nil {
		fmt.Println(err)
		return
	}

	err = csv.Write(os.Stdout, t, &csv.WriteOptions{
		Delimiter: '\t',
		NullValue: "NA",
		Precision: 2,
	})
	if err != nil {
		fmt.Println(err)
	}
}

Output:
sym	qty	px
AAPL	100	182.50
MSFT	NA	411.20

func WriteFile

func WriteFile(path string, t *Table, opts *WriteOptions) error

WriteFile writes the table to the file at path, creating it if it is not there and truncating it if it is.

Types

type Options

type Options struct {
	// Delimiter is what separates fields. Zero means a comma.
	Delimiter rune

	// Comment, if not zero, starts a comment line. A line whose first
	// non-space character is this is skipped entirely. It cannot be the same
	// as Delimiter.
	Comment rune

	// NoHeader says the first line is data rather than names. The names then
	// come from Names, or are generated as column_1, column_2 and so on.
	NoHeader bool

	// Names, if given, is the column names to use. It overrides a header line,
	// which is still read and thrown away, so a file with names that are not
	// valid identifiers can be renamed on the way in. It must have one name
	// for every field.
	Names []string

	// Columns, if given, is the columns to read, named as they are after Names
	// has been applied. A name that is not a column in the file is an error, and
	// so is the same name twice.
	//
	// A column that is not named has no type inferred, no value parsed and no
	// memory held. Its fields are still pulled out of each line, because a
	// delimited file cannot be read past a field without reading it, but that is
	// the cheap half of the work.
	//
	// The columns come out in the order they are named here, which need not be
	// the order the file has them.
	Columns []string

	// Types names the type of a column instead of inferring it. A column that
	// is not in here is inferred as usual. A name that is not a column in the
	// file is an error, since it is almost always a typo.
	//
	// The types that can be named are bool, the signed and unsigned integers,
	// the floats, string and binary. The parse is the same one [kernel.Cast]
	// does from text, which is more permissive than inference.
	Types map[string]dtype.DataType

	// InferRows is how many rows to look at before deciding what the columns
	// hold. Zero means [DefaultInferRows]. A negative value reads the whole
	// input before deciding, which is exact and holds the file in memory
	// twice.
	InferRows int

	// NullValues is the list of field values that mean nothing is there. A nil
	// list means the empty field and nothing else. A list that is not nil
	// replaces that rule rather than adding to it, so an empty but not nil
	// list means no field is missing and an empty field is an empty string.
	NullValues []string

	// ChunkSize is how many rows go into one chunk of a column. Zero means
	// [DefaultChunkSize].
	ChunkSize int

	// Skip is how many lines to throw away before reading anything, for the
	// files that start with a banner. The header, if there is one, is the
	// first line after these.
	Skip int

	// TrimLeadingSpace drops the space at the start of a field, even inside
	// quotes.
	TrimLeadingSpace bool

	// LazyQuotes accepts a quote in an unquoted field and an unescaped quote
	// in a quoted one, rather than reporting them.
	LazyQuotes bool

	// IgnoreParseErrors turns a value that will not parse into a missing value
	// instead of stopping the read. It is for the file that is nearly clean
	// and has to be loaded today.
	IgnoreParseErrors bool
}

Options controls how a file is read. The zero Options is the useful default: comma separated, a header row, types inferred, and an empty field meaning a missing value.

type Table

type Table = array.Table

Table is what the reader returns and what the writer takes, which is a schema and the columns that go with it.

It is array.Table because a table out of a CSV file and a table out of a parquet file are the same thing, and a caller holding one should not have to convert it to hand it to the other. A caller who wants rows and names and a query engine wants kuma.ReadCSV, which turns one of these into a frame.

func Read

func Read(r io.Reader, opts *Options) (*Table, error)

Read reads a whole file into columns.

The reader looks at the first rows to work out what each column holds, then reads the rest into that. What it decides and how to override it is described on Options and in the package documentation.

Everything is read. A file too large to hold in memory wants the lazy kuma.ScanCSV, which reads a chunk at a time and never has more than one of them alive. What this returns is a column in chunks either way, so nothing asks for one allocation the size of the file.

Example

package main

import (
	"fmt"
	"strings"

	"github.com/tamnd/kuma/csv"
)

func main() {
	in := `sym,qty,px,live
AAPL,100,182.5,true
MSFT,,411.2,false
GOOG,300,,true
`

	t, err := csv.Read(strings.NewReader(in), nil)
	if err != nil {
		fmt.Println(err)
		return
	}

	fmt.Println(t.Schema)
	fmt.Println(t.NumRows(), "rows")

	// Nothing is missing from sym and something is missing from qty, which is
	// what the schema says above and what the column carries below.
	fmt.Println(t.Columns[1])
}

Output:
schema<sym: string not null, qty: int64, px: float64, live: bool not null>
3 rows
array.Chunked{int64, len 3, nulls 1, chunks 1}

Example (Columns)

package main

import (
	"fmt"
	"strings"

	"github.com/tamnd/kuma/csv"
)

func main() {
	// A file of trades with a lot in it, where the question being asked is
	// about two of the columns. The rest are never parsed and never stored.
	in := `ts,sym,qty,px,venue,broker,note
2026-01-02T09:30:00Z,AAPL,100,182.5,XNAS,GS,opening
2026-01-02T09:30:01Z,MSFT,200,411.2,XNAS,MS,
`

	t, err := csv.Read(strings.NewReader(in), &csv.Options{
		Columns: []string{"sym", "qty"},
	})
	if err != nil {
		fmt.Println(err)
		return
	}

	fmt.Println(t.Schema)
	fmt.Println(t.NumCols(), "columns of", 7)
}

Output:
schema<sym: string not null, qty: int64 not null>
2 columns of 7

Example (NoHeader)

package main

import (
	"fmt"
	"strings"

	"github.com/tamnd/kuma/csv"
)

func main() {
	in := "AAPL,100\nMSFT,200\n"

	t, err := csv.Read(strings.NewReader(in), &csv.Options{
		NoHeader: true,
		Names:    []string{"sym", "qty"},
	})
	if err != nil {
		fmt.Println(err)
		return
	}

	fmt.Println(t.Schema)
}

Output:
schema<sym: string not null, qty: int64 not null>

Example (Types)

package main

import (
	"fmt"
	"strings"

	"github.com/tamnd/kuma/csv"
	"github.com/tamnd/kuma/dtype"
)

func main() {
	// The file writes a zip code, which is a name for a place rather than a
	// number, and the leading zero is part of it. Naming the type is how a
	// column stops being guessed at.
	in := `zip,pop
02134,12345
90210,21345
`

	t, err := csv.Read(strings.NewReader(in), &csv.Options{
		Types: map[string]dtype.DataType{"zip": dtype.String},
	})
	if err != nil {
		fmt.Println(err)
		return
	}

	fmt.Println(t.Schema)
	for i := range t.NumRows() {
		fmt.Printf("%s ", t.Columns[0].Bytes(i))
	}
	fmt.Println()
}

Output:
schema<zip: string not null, pop: int64 not null>
02134 90210

func ReadFile

func ReadFile(path string, opts *Options) (*Table, error)

ReadFile reads the file at path. It is Read over an open file, with the name of the file in any error the read returns.

type ValueError

type ValueError struct {
	// Line is the line in the input the value came from, counting from one and
	// counting the header.
	Line int

	// Column is the name of the column.
	Column string

	// Type is the type the value was being read as.
	Type string

	// Value is the field as it appeared in the file.
	Value string

	// Err is what the parse said, usually [strconv.ErrSyntax] or
	// [strconv.ErrRange].
	Err error
}

ValueError says that one field could not be read as the type of its column.

It carries the line, the column name and the value, because that is what it takes to go and look at the file. A parse error with only the message says the file is wrong somewhere, which is not much help in a million rows.

csv: line 4823, column "qty": cannot read "n/a" as int64: invalid syntax

Example

package main

import (
	"errors"
	"fmt"
	"strings"

	"github.com/tamnd/kuma/csv"
	"github.com/tamnd/kuma/dtype"
)

func main() {
	in := `qty
100
lots
`

	_, err := csv.Read(strings.NewReader(in), &csv.Options{
		Types: map[string]dtype.DataType{"qty": dtype.Int64},
	})
	fmt.Println(err)
	fmt.Println(errors.Is(err, csv.ErrValue))

	var ve *csv.ValueError
	if errors.As(err, &ve) {
		fmt.Println("line", ve.Line, "of column", ve.Column)
	}
}

Output:
csv: line 3, column "qty": cannot read "lots" as int64: invalid syntax
true
line 3 of column qty

func (*ValueError) Error

func (e *ValueError) Error() string

Error returns the message described on ValueError.

func (*ValueError) Unwrap

func (e *ValueError) Unwrap() []error

Unwrap returns the errors this wraps, so that both errors.Is(err, csv.ErrValue) and errors.Is(err, strconv.ErrRange) answer about the same error.

type WriteOptions

type WriteOptions struct {
	// Delimiter is what goes between fields. Zero means a comma.
	Delimiter rune

	// NoHeader leaves out the line of column names.
	NoHeader bool

	// Names, if given, is what to write on the header line instead of the
	// names in the schema. It must have one name for every column.
	Names []string

	// NullValue is what a missing value is written as. The default is an empty
	// field, which is what the reader reads back as missing.
	NullValue string

	// Precision is how many digits a float is written with. Zero means the
	// shortest text that reads back as the same value, which is what a file
	// that will be read again wants. A number above zero is that many digits
	// after the point, which is what a file that will be looked at wants. A
	// file that wants no digits after the point wants an integer column.
	Precision int

	// QuoteAll puts quotes around every field rather than around the ones that
	// need them. Some readers outside Go expect it and no reader minds it.
	QuoteAll bool

	// CRLF ends each line with a carriage return and a newline. The default is
	// a newline on every platform, because a file is data rather than text on
	// a screen.
	CRLF bool
}

WriteOptions controls how a table is written. The zero WriteOptions is the useful default: comma separated, a header row, an empty field for a missing value, and floats written with the fewest digits that read back as the same value.