csvcopy

package module

v0.1.0 Latest Latest Go to latest Published: Aug 10, 2026 License: MIT Imports: 9 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/AMUDENN/go-csv-copy

Links

Open Source Insights

README ¶

go-csv-copy

Streaming CSV reader for Go, shaped for bulk-loading into PostgreSQL. No dependencies outside the standard library.

⸻

📋 Overview

The package reads CSV one row at a time: a row is read, handed over and forgotten, so memory does not depend on the size of the file.

What it is built for is COPY FROM — the row sources satisfy pgx.CopyFromSource, so a file of any size loads into Postgres without being buffered. That is the primary use case, not the only one: Reader is a plain CSV reader and Typed decodes into your own types, both usable with no database in sight. See Without a database.

Two ways of fixing a file's shape are supported, because both turn up in practice:

Raw — the shape comes from the file's own header. Whatever columns arrived, in their order, every value as text. This is the staging-table case, where the input's structure is not known ahead of time and a SQL script types the data afterwards.
Typed[S, D] — the shape comes from csv struct tags. Columns are matched by name, so their order in the file does not matter, extra columns are ignored, and a column a tag asks for and the file lacks is an error. Each row passes through your convert, which types and validates the values.

Why pgx is not imported

The package does not depend on pgx and is compatible with it anyway. Go interfaces are structural, and pgx.CopyFromSource is nothing more than a method set:

type CopyFromSource interface {
    Next() bool
    Values() ([]any, error)
    Err() error
}

Any type with those methods satisfies it without declaring a dependency. So the application picks the version of pgx, not the library, and upgrading pgx never requires upgrading csvcopy.

⸻

📦 Install

go get github.com/AMUDENN/go-csv-copy

import "github.com/AMUDENN/go-csv-copy"   // package csvcopy

Go 1.24 or newer. The go directive names a minor version on purpose: pinning a patch would force every consumer to fetch a toolchain, which is a strange thing for a package whose whole point is bringing nothing with it.

⸻

🚀 Quick start

Shape from the file header (staging table)

src, err := csvcopy.NewRaw(file)
if err != nil {
    return fmt.Errorf("parse input: %w", err)
}

columns := src.Columns()
if len(columns) == 0 {
    return nil // an empty file is not an error
}

// The names came out of the file. This one builds DDL from them, so they are
// untrusted input: a column called `x" ); DROP TABLE clients; --` is just a text
// file someone wrote. Quote them, or refuse the header outright.
if err := csvcopy.ValidateColumns(columns); err != nil {
    return err
}

if err := createStagingTable(ctx, tx, table, columns); err != nil {
    return err
}

n, err := tx.CopyFrom(ctx, pgx.Identifier{table}, columns, src)
if srcErr := src.Err(); srcErr != nil {
    return fmt.Errorf("read input at line %d: %w", src.Line(), srcErr)
}
if err != nil {
    return fmt.Errorf("copy from: %w", err)
}

Check src.Err() before err: when the source fails on a row, pgx returns its own, less informative error, while the cause sits in the source together with the line number.

Shape from struct tags (typed load)

// The shape of the file: all text, tags name the columns in the header.
type clientRow struct {
    ID        string `csv:"client_id"`
    Email     string `csv:"email"`
    Balance   string `csv:"balance"`
    CreatedAt string `csv:"created_at"`
}

// Your typing and validation - the package stays out of it.
func (r *clientRow) toClient() (*Client, error) { /* ... */ }

var clientColumns = []string{"id", "email", "balance", "created_at"}

rows, err := csvcopy.NewTyped(file, (*clientRow).toClient)
if err != nil {
    return err
}

source, err := csvcopy.NewCopy(rows, len(clientColumns),
    func(dst []any, c *Client) []any {
        return append(dst, c.ID, c.Email, c.Balance, c.CreatedAt)
    })
if err != nil {
    return err
}

_, err = tx.CopyFrom(ctx, pgx.Identifier{"clients_tmp"}, clientColumns, source)
if srcErr := source.Err(); srcErr != nil {
    return fmt.Errorf("read input at line %d: %w", source.Line(), srcErr)
}
if err != nil {
    return fmt.Errorf("copy from: %w", err)
}

if source.Rows() == 0 {
    // the file was empty - nothing to load
}

Typed satisfies RowSource[D], so decoding and the layout of a COPY row can live in different layers: one hands out a RowSource, and the layer that owns the database decides which columns it lands in.

Without a database

Reader and Typed stand on their own — no COPY, no pgx, no []any. Ranging cannot carry an error out, so the loop stops at the first bad row and Err() reports it afterwards, the same shape as bufio.Scanner:

rows, err := csvcopy.NewTyped(file, (*clientRow).toClient)
if err != nil {
    return err
}

for client := range rows.All() {
    if err := send(ctx, client); err != nil {
        return err
    }
}
if err := rows.Err(); err != nil {
    return err
}

When no decoding is wanted either, Reader yields raw records:

reader, err := csvcopy.NewReader(file)
if err != nil {
    return err
}

fmt.Println(reader.Columns())

for record := range reader.All() {
    fmt.Println(record) // reused between iterations - copy it to keep it
}
return reader.Err()

⸻

🏗 API

Layers

Constructor	Returns	Use when
`NewReader(r, opts...)`	`*Reader` — header + row-by-row reading	you only need CSV parsing, no COPY
`NewRaw(r, opts...)`	`*Raw` — `pgx.CopyFromSource` + `Columns()`	the shape comes from the file header
`NewTyped[S, D](r, convert, opts...)`	`*Typed[S, D]` — `RowSource[D]`	the shape comes from `csv` tags
`NewCopy[T](src, columns, encode)`	`*Copy[T]` — `pgx.CopyFromSource`	adapting any `RowSource[T]` to COPY

Every constructor returns an error rather than panicking. Nil wiring — a nil reader, convert, src or encode — is ErrSchema.

Interfaces

// RowSource is anything that yields values one at a time. Typed satisfies it,
// and so can a source of your own over XLSX, an API or a generator.
type RowSource[T any] interface {
    Next() bool
    Value() T
    Err() error
}

Options

Option	Default	Effect
`WithComma(r rune)`	`';'`	field delimiter
`WithLazyQuotes(bool)`	`false`	tolerate a bare quote, and a quoted field that never closes, instead of failing
`WithTrimLeadingSpace(bool)`	`true`	drop white space at the start of a field
`WithTrimValues(bool)`	`true`	`TrimSpace` every value
`WithHeaderRow(n uint)`	`1`	which row holds the header; rows above it are dropped
`WithNormalizeHeader(fn)`	collapse whitespace	normalize a name before matching — applied to both the file's column names and the `csv` tag values, so it must be pure and idempotent
`WithTag(string)`	`"csv"`	which struct tag `Typed` reads column names from
`WithVariableColumns(bool)`	`false`	accept rows of a different width: extra values dropped, missing trailing ones → `nil` in `Raw`, `""` in `Typed`
`WithAllowMissingColumns(bool)`	`false`	do not fail when a tagged column is absent from the header
`WithPointerValues(bool)`	`true`	`Raw` yields `*string`, which costs no allocation per cell. `false` gives plain `string`
`WithMaxRecordBytes(int64)`	`64 MiB`	cap on one record; zero removes it. Exceeding it is `ErrRecordTooLarge`
`WithComment(rune)`	`0` (off)	a rune that starts a comment line, skipped wherever it appears

The `csv` tag

The whole tag is the column name. There are no comma-separated options: a field tagged csv:"name,omitempty" asks for a column literally called name,omitempty, which no header will have, and ErrMissingColumns says so. - is the one value with a meaning of its own — it drops the field.

Two rules exist because breaking them loses data silently, and the package refuses both with ErrSchema:

type row struct {
    base                       // ✗ tags inside an embedded struct are not matched
    A string `csv:"id"`
    B string `csv:"id"`        // ✗ two fields, one column
}

Embedded fields are not walked into. Left to pass, a tag one level down would bind to nothing, the field would read as empty on every row, and the column it named would reach the database as NULL — the same damage ErrMissingColumns exists to prevent, only without the error. List the columns on the struct itself.

Why `;` and not `,`

encoding/csv defaults to ,, so this looks like gratuitous disagreement. It follows the target case. These files come out of spreadsheet exports on machines whose locale uses , as the decimal separator, where Excel and LibreOffice write ; — and where a comma-delimited file with a single 1,5 in it is silently one column wider than its header. Anything reading such files hits ; overwhelmingly more often than ,.

Pass WithComma(',') for RFC 4180 files. The default is one option away either direction; what it should not be is a surprise, hence this paragraph.

Diagnostics

Method	Gives
`All()`	a `range`-able iterator: `[]string` on `Reader`, `[]any` on `Raw`, `D` on `Typed`
`Err()`	the error that stopped the stream; end of file is not one
`Columns()` / `Header()`	the header as read and normalized
`Record()`	the raw record last read, for error messages
`Unused()`	header columns no tag bound to
`Line()`	the 1-based physical line of the file the current row starts on
`Rows()`	how many rows have been handed out

Copy forwards Line() and Record() to whatever it wraps, so a source can go straight into NewCopy without keeping a second reference to it just to ask where a failure came from. A source with no lines — over an API, or a generator — answers 0 and nil.

`nil` in `Raw`, `""` in `Typed`

Under WithVariableColumns, a value the record never reached becomes SQL NULL in Raw and the empty string in Typed. The asymmetry is forced, not chosen: Raw hands pgx an []any and can put nil in it, while a tagged field is declared string and has no nil to hold. In Postgres NULL and '' are different values, so the difference matters.

Typed.Truncated() tells the two cases apart, read after Next():

for source.Next() {
    if source.Truncated() {
        log.Warn("short record", "line", source.Line())
    }
}

convert cannot see it — it is called inside Next() with the struct as its only argument, and widening that signature would change every caller's code. Reject a short row in the loop instead.

Unused() is worth logging as a warning: when an export renames a column, the tag simply matches nothing, no error is raised, and the wrong data reaches the database. The unbound column is the only visible trace.

⚠️ Columns() returns untrusted input. The names come out of the file, and the staging-table pattern puts them on the path to CREATE TABLE — a column called x" ); DROP TABLE clients; -- is just a text file someone wrote. Quote every name that reaches a statement with pgx.Identifier{name}.Sanitize(), or refuse the header up front with csvcopy.ValidateColumns(columns), which rejects empty names, duplicates, names over 63 bytes (Postgres truncates at NAMEDATALEN and two columns then collide) and anything outside [A-Za-z0-9_], listing every violation at once. CopyFrom itself quotes them.

⚠️ WithAllowMissingColumns(true) is dangerous. The absent field reads as the empty string on every row and reaches the database as NULL, quietly wiping whatever that column held. Only use it where that is acceptable.

Errors

Errors are split by who can fix them.

Sentinel	Wraps `ErrParse`	Cause
`ErrParse`	—	the file's content is wrong: a malformed record, a failing `convert`
`ErrMissingColumns`	yes	the header lacks a column a tag asks for
`ErrInvalidColumns`	yes	`ValidateColumns` refused a header name
`ErrRecordTooLarge`	yes	one record outgrew `WithMaxRecordBytes`
`ErrIO`	no	the stream failed, not the file: a dropped connection, a cancelled context, a bad disk
`ErrSchema`	no	a bug in the calling code: not a struct, a tag on a non-string or unexported field, a tag inside an embedded struct, two fields asking for one column, a nil reader/convert/src/encode, an unusable delimiter

The split exists because the three answers differ: ErrSchema means fix the code, ErrIO means retry, ErrParse means the file is bad — quarantine it. Getting this wrong is not theoretical. Until ErrIO existed, a dropped TCP connection was reported as ErrParse, so anyone following the advice above would quarantine a perfectly good file forever because the network blinked once.

The classification is not guesswork: encoding/csv reports everything the parser objects to as a *csv.ParseError and passes anything else back from the underlying reader unchanged, so the shape of the error is the evidence. The original cause stays in the chain either way, so errors.Is(err, context.Canceled) still answers.

Everything a file can cause wraps ErrParse and carries the line number:

csv parse: record on line 4213: wrong number of fields
csv parse: line 4213: balance: strconv.ParseFloat: parsing "n/a": invalid syntax

The line is the physical line of the file, not a count of records, so it stays right across blank lines (encoding/csv skips them) and quoted fields spanning several lines. It is what Line() returns, and the number to open the file at.

If your application already has a sentinel for a bad file, alias it and every existing errors.Is check keeps working:

var ErrBadFile = csvcopy.ErrParse

ErrSchema deliberately does not wrap ErrParse: no input file will ever fix it, so quarantine-the-file logic must not fire on it. It is a code bug, and it surfaces on the very first run — including a run on an empty file.

⸻

📐 Semantics and guarantees

Empty input (zero bytes, or only a BOM) is not an error: Columns() == nil, Next() == false, Err() == nil, Rows() == 0.
The BOM (EF BB BF) is stripped, read with io.ReadFull so that a short read from a slow io.Reader — which the contract allows — cannot leave a in the first column name.
The first bad row stops the stream, for good. Rows are never skipped: a partly loaded table is worse than a failed load. The error is available from Err(), and it stays there — a source that has failed yields nothing more, so ranging All() a second time cannot resume past the bad row. Breaking out of a loop is different: that is not an error, and the next pull carries on.
Record() names the row that failed, as far as encoding/csv got with it. Line() names the physical line it starts on.
Memory is constant and independent of the row count — bounded by the largest single record, and that bound is WithMaxRecordBytes (64 MiB by default). encoding/csv assembles a record in one buffer and has no limit of its own, so a field that opens a quote and never closes it is read to the end of the file and the whole file becomes one value; the cap is what makes the guarantee hold on input nobody checked. Values() reuses one slice, which is safe under pgx.CopyFrom because it encodes a row before asking for the next. If you drive a source by hand, do not retain the result of Values() between iterations.
Not safe for concurrent use. One source, one goroutine — the same as pgx.CopyFrom.
Reader, Raw and Typed do not close the io.Reader you give them.

⸻

⚡ Performance

Live memory is constant, but allocations over a pass grow linearly, and it is worth knowing where from. 100k rows of 5 columns, go1.26.1, Ryzen 5 7500F:

	ns/op	B/op	allocs/op	per row
`Raw`	12.6 ms	3.2 MB	100 034	1
`Raw` via `All()`	12.5 ms	3.2 MB	100 034	1
`Typed`	14.6 ms	3.2 MB	100 039	1
`Typed` via `All()`	15.0 ms	3.2 MB	100 039	1
`Raw` + `WithPointerValues(false)`	22.1 ms	11.2 MB	600 033	6

Ranging costs nothing: an iter.Seq returned from a method closes over the source once per pass, not once per row, so All() sits on the same allocation count as the Next() loop it replaces.

The one allocation per row is encoding/csv: it creates one string per record even with ReuseRecord, and that is the floor short of unsafe.

The extra five in the last row are the columns. Boxing a string into an any always allocates 16 bytes for its header, so plain strings cost one allocation per cell — that is the shape of Values() ([]any, error). A *string out of an array allocated once per file is pointer-shaped: the interface holds it directly and boxing is free.

That is why *string is the default, and pgx cannot tell the difference — measured, not argued. The integration tests load the same file both ways into an all-TEXT table and compare an md5 of the rows, then load both ways into a table of bigint, numeric and date. Both agree.

Pass WithPointerValues(false) if you drive the source yourself and would rather type-switch over string than *string.

It matters more the wider the file

20k rows of 30 columns:

	ns/op	B/op	allocs/op	per row
`Raw`	13.2 ms	5.3 MB	20 066	1
`Typed`	15.7 ms	5.3 MB	20 099	1
`Raw` + `WithPointerValues(false)`	21.8 ms	14.9 MB	620 065	31

Six times the columns, thirty-one times the allocations once the values are strings. The default and Typed both stay flat at one per row — apply is linear in the number of bound fields but writes into a struct and allocates nothing. What the default saves grows with the width of the file: 31× here against 6× on five columns.

The `encode` mistake that undoes it

In Copy the boxing happens inside your own encode, so WithPointerValues does not affect it. What does affect it is whether encode appends into dst or returns a fresh slice. The second reads perfectly naturally and nothing stops you writing it:

// Correct: appends into the buffer Copy keeps.
func(dst []any, c *Client) []any { return append(dst, c.ID, c.Email) }

// Costs one allocation per row, forever.
func(_ []any, c *Client) []any { return []any{c.ID, c.Email} }

	ns/op	B/op	allocs/op
appending into `dst`	23.5 ms	11.2 MB	600 042
returning a new slice	26.4 ms	19.2 MB	700 043

⸻

🔧 Fitting it into existing code

Both layers rely on structural typing, so the package usually drops in without reshaping the caller:

If you already have your own row-source interface with Next/Value/Err, it can be replaced by csvcopy.RowSource[T] — the method sets match and the implementations do not change.
If another format is read alongside CSV (XLSX, a stream from an API), it is enough for its source to implement the same Next/Values/Err. Both branches stay pgx.CopyFromSource, and choosing a format comes down to returning different values of one interface.
Your own sentinel for a bad file becomes an alias of csvcopy.ErrParse, and errors.Is keeps working.

⸻

🔍 Alternatives

Package	License	Streaming	Shape from header	`CopyFromSource`
`csvcopy`	MIT	yes, pull	yes	yes
`jszwec/csvutil`	MIT	yes, pull	no	no
`gocarina/gocsv`	MIT	yes, push	no	no
`artonge/go-csv-tag`	GPL-3.0	no, `ReadAll`	no	no

The difference between pull and push matters here. pgx.CopyFrom pulls: it calls Next(), encodes the row into its buffer, and only then asks for the next one. Packages like gocsv push — UnmarshalToChan(in io.Reader, c interface{}) and UnmarshalToCallback(in io.Reader, f interface{}) drive the loop themselves. Connecting such a source to CopyFrom needs a goroutine and a channel between them, which means concurrency, backpressure and carrying an error across a goroutine boundary, all inside an open transaction. A pull interface of three methods needs none of that.

If you only need CSV decoded into structs and no COPY FROM, use csvutil: it is more mature and does more (inline, omitempty, custom unmarshalers). csvcopy solves the narrower problem of loading into PostgreSQL, which is why it has a layer that takes its shape from the file header and adapters for CopyFrom, and why it deliberately does no type conversion — that belongs to the caller, who needs to tell "empty → NULL" from "zero → 0".

⸻

📄 License

MIT.

Documentation ¶

Overview ¶

Package csvcopy streams CSV one row at a time.

A row is read, handed over and forgotten, so memory does not depend on the size of the file. There are no dependencies outside the standard library.

Bulk-loading into PostgreSQL is what it is shaped for, and the reason the row sources expose Next/Values/Err: that method set is pgx.CopyFromSource, satisfied structurally rather than by importing pgx, so the version of pgx stays the application's choice. Nothing here requires a database, though - Reader is a plain CSV reader, and Typed decodes into your own types. Both have All for ranging.

Layers ¶

Two ways to fix the shape of a file, because both turn up in practice.

Raw takes the shape from the file's own header - whatever columns arrived, in their order, all as text. This is the staging table case, where a SQL script types the data afterwards. Values come out as *string, which pgx dereferences and which costs no allocation to box; WithPointerValues(false) gives plain string instead:

src, err := csvcopy.NewRaw(file)
if err != nil {
	return err
}
if len(src.Columns()) == 0 {
	return nil // empty file, nothing to load
}
// The names came out of the file, so they are untrusted on the way to DDL.
if err := csvcopy.ValidateColumns(src.Columns()); err != nil {
	return err
}

n, err := tx.CopyFrom(ctx, pgx.Identifier{table}, src.Columns(), src)
// The source's error first: it names the line, pgx's does not.
if srcErr := src.Err(); srcErr != nil {
	return fmt.Errorf("read input at line %d: %w", src.Line(), srcErr)
}
if err != nil {
	return err
}

Typed takes the shape from struct tags. Columns are matched by name, so the file may reorder them or add new ones; a column a tag asks for and the file lacks is an error. Every tagged field is a string, and convert turns the row into whatever the program actually works with - only the caller can tell an empty cell from a zero:

type row struct {
	ID   string `csv:"id"`
	Name string `csv:"name"`
}

rows, err := csvcopy.NewTyped(file, (*row).toEntity)
if err != nil {
	return err
}

source, err := csvcopy.NewCopy(rows, len(columns), func(dst []any, e *entity) []any {
	return append(dst, e.ID, e.Name)
})
if err != nil {
	return err
}
n, err := tx.CopyFrom(ctx, pgx.Identifier{table}, columns, source)
if srcErr := source.Err(); srcErr != nil {
	return fmt.Errorf("read input at line %d: %w", source.Line(), srcErr)
}
if err != nil {
	return err
}

NewCopy adapts any RowSource - Typed, or one of your own over XLSX or an API - to pgx.CopyFromSource.

Both examples check the source before pgx, and that order is not stylistic. When a row fails, pgx reports that its source stopped; the source reports what went wrong and on which line. Reading pgx's error first throws the useful one away.

Without a database ¶

Reader and Typed stand on their own. Ranging stops at the first bad row and Err reports it afterwards, the same shape as bufio.Scanner:

rows, err := csvcopy.NewTyped(file, (*row).toEntity)
if err != nil {
	return err
}

for entity := range rows.All() {
	send(entity)
}
if err := rows.Err(); err != nil {
	return err
}

Reader.All yields raw records as []string, for when no decoding is wanted either.

Guarantees ¶

An empty input is not an error: no columns, no rows, no error. The caller decides what that means.

Memory is bounded by the largest single record rather than by the file, and that bound is WithMaxRecordBytes - 64 MiB by default. encoding/csv assembles a record in one buffer and caps nothing, so a field that opens a quote and never closes it is read to the end of the file and the whole file becomes one value. Exceeding the cap is ErrRecordTooLarge.

A UTF-8 BOM is stripped, read with io.ReadFull so a slow reader cannot leave it in place.

The first bad row stops the stream for good. Rows are not skipped: a partly loaded table is worse than a failed load. The error is available from Err, names the line, and stays there - a source that has failed yields nothing more, so ranging All again cannot resume past the row that broke. Breaking out of a loop is not an error, and the next pull carries on from where it stopped.

Record and Line name the row that failed, so an error message can carry it. Line is the physical line of the file, taken from encoding/csv, so it stays right across blank lines and quoted fields spanning several lines.

Errors are split by who can fix them, because the three answers differ. Errors the file's content causes wrap ErrParse, so an application can alias its own sentinel to it and quarantine the file. Errors the stream causes - a dropped connection, a cancelled context - wrap ErrIO instead and deserve a retry, not a quarantine: the file is fine. Errors the calling code causes wrap ErrSchema, because no file will ever fix them.

Values reuses one slice between rows. That is safe under pgx.CopyFrom, which encodes a row before asking for the next; a caller driving a source by hand must not retain it.

Nothing here is safe for concurrent use, and neither is pgx.CopyFrom.

Index ¶

Variables
func NormalizeSpace(s string) string
func ValidateColumns(columns []string) error
type Copy
- func NewCopy[T any](src RowSource[T], columns int, encode func(dst []any, item T) []any) (*Copy[T], error)
type Option
type Raw
- func NewRaw(r io.Reader, opts ...Option) (*Raw, error)
type Reader
- func NewReader(r io.Reader, opts ...Option) (*Reader, error)
type RowSource
type Typed
- func NewTyped[S any, D any](r io.Reader, convert func(*S) (D, error), opts ...Option) (*Typed[S, D], error)

Examples ¶

NewTyped

Constants ¶

This section is empty.

Variables ¶

View Source

var ErrIO = errors.New("csv read")

ErrIO is wrapped by every error the input stream causes rather than the file's content: a connection dropped mid-read, a cancelled context, a disk that failed. The cause is left in the chain, so errors.Is(err, context.Canceled) still answers.

It deliberately does not wrap ErrParse. A caller that quarantines a file on ErrParse must not quarantine a good file because the network blinked - the file is fine, the read is not, and the right response is to try again rather than to give up on the input.

View Source

var ErrInvalidColumns = fmt.Errorf("%w: header has unusable column names", ErrParse)

ErrInvalidColumns reports a header that must not reach a DDL statement.

Wraps ErrParse: it is a property of the file, and the caller who quarantines bad files wants this one quarantined too.

View Source

var ErrMissingColumns = fmt.Errorf("%w: header is missing columns", ErrParse)

ErrMissingColumns reports columns that a csv tag asked for and the header does not have.

This is fatal rather than a warning: a field bound to nothing would read as the empty string on every row and reach the database as NULL, quietly wiping the column it was supposed to fill. WithAllowMissingColumns lifts the check for callers who accept that risk.

View Source

var ErrParse = errors.New("csv parse")

ErrParse is wrapped by every error that a file can cause: a malformed record, a header that does not match the struct, a failing convert. An application can therefore alias its own sentinel to it and keep its existing errors.Is checks working:

var ErrBadFile = csvcopy.ErrParse

View Source

var ErrRecordTooLarge = fmt.Errorf("%w: %w", ErrParse, errRecordTooLarge)

ErrRecordTooLarge reports a single record longer than WithMaxRecordBytes allows.

Almost always an unclosed quote: encoding/csv then reads to the end of the file looking for the closing one, and the whole file becomes a single field. The limit is what keeps memory bounded on input the caller did not write.

View Source

var ErrSchema = errors.New("csv schema")

ErrSchema reports wiring that cannot work whatever the file holds: a struct that cannot be decoded into at all - not a struct, a tagged field that is not a string or not exported, two fields asking for one column, a tag inside an embedded struct where it would bind to nothing - a nil reader, convert, src or encode, or a delimiter encoding/csv will not accept.

It deliberately does not wrap ErrParse. No input file will ever fix it, so a caller that retries or quarantines files on ErrParse should not treat it as a bad file - it is a bug in the calling code.

Functions ¶

func NormalizeSpace ¶

func NormalizeSpace(s string) string

NormalizeSpace is the default header normalizer: it collapses every run of whitespace into a single space and trims the ends.

Exported column names arrive wrapped across lines or padded for alignment, so "date of\n birth" and "date of birth" have to resolve to the same column.

func ValidateColumns ¶ added in v0.1.0

func ValidateColumns(columns []string) error

ValidateColumns rejects a header whose names cannot safely be used to build a statement.

A staging table is created from the file's own column names, which makes those names untrusted input on the path to CREATE TABLE. A file with a column called

x" ); DROP TABLE clients; --

is not a hypothetical; it is a text file, and anyone can write one. This refuses the four shapes that either break a statement or change its meaning: an empty name, a duplicate, a name over 63 bytes, and anything outside [A-Za-z0-9_].

Every violation is reported at once rather than the first one, so one run tells the whole story of the file instead of one name per attempt.

Deliberately strict, and it rejects more than injection: "date of birth" and any non-ASCII name are refused too. A file whose column names are prose is a file whose names should be mapped explicitly, not quoted and hoped for. Passing this is not a licence to skip quoting - build identifiers with pgx.Identifier{name}.Sanitize() regardless. For CopyFrom, pgx quotes them itself.

An empty header - the empty-file case - passes: there is nothing to build from and nothing to be unsafe with.

Types ¶

type Copy ¶

type Copy[T any] struct {
	// contains filtered or unexported fields
}

Copy adapts a RowSource to pgx.CopyFromSource.

Only the mapping of a value onto its columns lives here; where the values came from is none of this layer's business.

func NewCopy ¶

func NewCopy[T any](src RowSource[T], columns int, encode func(dst []any, item T) []any) (*Copy[T], error)

NewCopy wraps src, laying each value out with encode.

columns is how many values encode appends; it only sizes the backing slice, so being wrong costs an allocation rather than correctness. encode must append in the same order as the column list passed to pgx.CopyFrom - the two are a pair, and Postgres cannot notice when they disagree if the types are compatible.

A nil src or encode is ErrSchema, the same as a nil reader or convert elsewhere in the package: it is wiring the calling code got wrong, and no input file will fix it.

func (*Copy[T]) Err ¶

func (c *Copy[T]) Err() error

Err reports the underlying source's error.

func (*Copy[T]) Line ¶ added in v0.1.0

func (c *Copy[T]) Line() int

Line is the line of the file the current row came from, or zero when the source does not have lines - one over an API or a generator does not.

Asked of the source through an interface rather than required by RowSource, so a source that cannot answer does not have to declare a method returning nothing useful.

It exists because the advice for a failed CopyFrom is to read the source's error first, since that is the one naming the line. Without this, following that advice meant keeping the Typed value in a second variable purely to ask it - and the obvious code, which passes the source straight into NewCopy, could not.

func (*Copy[T]) Next ¶

func (c *Copy[T]) Next() bool

Next advances the underlying source.

func (*Copy[T]) Record ¶ added in v0.1.0

func (c *Copy[T]) Record() []string

Record is the raw record behind the current row, or nil when the source does not keep one. The counterpart to Line, and asked for the same way.

func (*Copy[T]) Rows ¶

func (c *Copy[T]) Rows() int64

Rows is the number of rows handed out so far. Counted here rather than asked of the source, which need not track it.

func (*Copy[T]) Values ¶

func (c *Copy[T]) Values() ([]any, error)

Values lays out the current value.

The result is stored back, so a row wider than columns grows the slice once instead of reallocating on every row. Safe to reuse because pgx encodes the row before pulling the next one.

type Option ¶

type Option func(*settings)

Option configures a Reader and everything built on top of it.

func WithAllowMissingColumns ¶

func WithAllowMissingColumns(allow bool) Option

WithAllowMissingColumns downgrades a missing tagged column from an error to silence.

Dangerous: the field will read as empty on every row and reach the database as NULL, wiping whatever that column held. Only use it where that is acceptable.

func WithComma ¶

func WithComma(comma rune) Option

WithComma sets the field delimiter. Defaults to ';'.

A quote, a carriage return, a newline and an invalid rune cannot delimit anything encoding/csv is willing to read, so the constructor rejects them with ErrSchema rather than letting the first row fail with ErrParse.

func WithComment ¶ added in v0.1.0

func WithComment(comment rune) Option

WithComment sets a rune that starts a comment line. Zero, the default, means the file has no comments.

A line whose first rune is this one is skipped entirely, wherever it appears - which is what WithHeaderRow cannot do, since that only drops a fixed number of lines at the top.

Validated like the delimiter, and for the same reason: a comment rune equal to the delimiter, or one encoding/csv will not accept, is ErrSchema from the constructor rather than ErrParse on the first row. Skipped lines do not shift the line numbers in errors - those come from encoding/csv, which counts the physical file.

func WithHeaderRow ¶

func WithHeaderRow(row uint) Option

WithHeaderRow sets which row holds the header, counting from 1. Rows before it are read and dropped, so a file may carry a title or a note above its table. Defaults to 1.

A header past the end of the file is the empty-file case, not an error.

func WithLazyQuotes ¶

func WithLazyQuotes(lazy bool) Option

WithLazyQuotes tolerates quoting encoding/csv would otherwise reject: a bare quote inside an unquoted field, and a quoted field that never closes. Defaults to false.

Turning it on trades a precise error for silent damage. `1;"2"3;4` becomes the field `2"3` with the right number of fields, so nothing objects. An unclosed quote is worse: the parser reads to EOF looking for the closing one and the entire rest of the file arrives as a single value. What surfaces then is a field-count error naming the line the file ended on rather than the line the quote opened on - and with WithVariableColumns there is no error at all.

Off, the same input is a parse error naming the line and the column of the quote.

func WithMaxRecordBytes ¶ added in v0.1.0

func WithMaxRecordBytes(n int64) Option

WithMaxRecordBytes caps how large one record may be. Zero, or anything negative, removes the cap. Defaults to 64 MiB.

The cap is what makes "memory does not depend on the size of the file" true for input nobody checked. encoding/csv assembles a record in one buffer and has no limit of its own, so a field that opens a quote and never closes it is read to the end of the file and the whole file becomes one value. Exceeding the cap is ErrRecordTooLarge.

The bound is approximate: csv.Reader buffers ahead, so the accounting is off by up to one buffer. It is an upper bound on memory, not a byte count to assert against.

func WithNormalizeHeader ¶

func WithNormalizeHeader(fn func(string) string) Option

WithNormalizeHeader replaces the header normalizer. Passing nil restores identity. Defaults to NormalizeSpace.

It runs on both sides of the match: on every column name read from the file, and on every csv tag value read from a struct. That is what keeps a normalizer like strings.ToLower working - lowering only the file's names would stop them matching tags written in any other case.

So it has to be pure and idempotent. It is called once per column and once per tagged field when a plan is built, never on the row path, and a function that returns different answers for the same input turns column matching into a coin toss.

func WithPointerValues ¶

func WithPointerValues(pointers bool) Option

WithPointerValues controls whether Raw hands out *string or string. Defaults to true, which is *string.

Putting a string into an []any boxes it, and boxing a string always allocates 16 bytes for its header, so plain strings cost one allocation per cell. A pointer is pointer-shaped: the interface holds it directly and boxing is free. The strings live in one array allocated per file, and a value the row stopped short of is a nil interface, still NULL. On five columns that is 6 allocations per row against 1; on thirty it is 31 against 1.

The default is *string because pgx cannot tell the difference, and that is a measured result rather than an argument: the integration tests load the same file both ways into an all-TEXT table and compare an md5 of the rows, and load both ways into a table of bigint, numeric and date. Both pass.

Pass false if you drive the source yourself and want plain strings - a type switch over []any is easier to write against string than *string. Nothing else in the package is affected: Typed decodes into your struct fields, and in Copy the boxing happens inside your own encode.

func WithTag ¶

func WithTag(tag string) Option

WithTag sets the struct tag Typed reads column names from. Defaults to "csv".

The whole tag is the column name. There are no comma-separated options - a field tagged `csv:"name,omitempty"` asks for a column literally called "name,omitempty", which no header will have. The only value with a meaning of its own is "-", which drops the field.

func WithTrimLeadingSpace ¶

func WithTrimLeadingSpace(trim bool) Option

WithTrimLeadingSpace drops white space at the start of a field. Defaults to true.

func WithTrimValues ¶

func WithTrimValues(trim bool) Option

WithTrimValues applies strings.TrimSpace to every value. Defaults to true.

func WithVariableColumns ¶

func WithVariableColumns(variable bool) Option

WithVariableColumns accepts rows whose field count differs from the header's. Extra values are dropped; missing trailing ones become NULL in Raw and "" in Typed.

That difference is real, not a wording slip. Raw hands pgx an []any and can put nil there, which is SQL NULL. A tagged field in Typed is declared string, so there is no nil to assign and an absent value is indistinguishable from an empty one - and in Postgres a NULL and an empty string are different values. Typed.Truncated reports which case the current row is, since convert cannot see it.

Off by default: a row that is the wrong width usually means the delimiter or the quoting is misread, and failing beats loading shifted data.

type Raw ¶

type Raw struct {
	// contains filtered or unexported fields
}

Raw streams a file whose shape is only known from its own header.

Every column arrives as text, in the file's order, under the file's names - the shape of a staging table that a SQL script types afterwards. Nothing here knows or cares what the columns mean.

It satisfies pgx.CopyFromSource. That interface is not imported: Go interfaces are structural, so the method set alone is enough and the version of pgx stays the application's choice.

func NewRaw ¶

func NewRaw(r io.Reader, opts ...Option) (*Raw, error)

NewRaw reads the header and prepares a source over the rows after it.

func (*Raw) All ¶

func (s *Raw) All() iter.Seq[[]any]

All iterates the rows, for driving the source by hand rather than handing it to pgx.CopyFrom.

Stops at the first bad row; Err reports it afterwards. The yielded slice is reused by the next iteration.

If you want the values as text rather than as []any, range Reader.All instead - there is no reason to go through the boxing.

func (*Raw) Columns ¶

func (s *Raw) Columns() []string

Columns is the column list to pass to pgx.CopyFrom, taken from the header.

Empty when the file was empty - check it before creating a table, there is nothing to load.

The names come out of the file, so they are untrusted input, and the staging-table pattern puts them on the path to CREATE TABLE. A column called

x" ); DROP TABLE clients; --

is just a text file someone wrote. Quote every name that reaches a statement with pgx.Identifier{name}.Sanitize(), or reject the header up front with ValidateColumns. For CopyFrom itself, pgx quotes them.

func (*Raw) Err ¶

func (s *Raw) Err() error

Err returns the error that stopped the stream, if any. Check it before the error pgx.CopyFrom returns: this one names the line.

func (*Raw) Line ¶

func (s *Raw) Line() int

Line is the 1-based line number of the current row.

func (*Raw) Next ¶

func (s *Raw) Next() bool

Next advances to the next row, stopping at the end of the file or at the first malformed record.

func (*Raw) Record ¶

func (s *Raw) Record() []string

Record returns the raw record behind the current row, for error messages.

func (*Raw) Rows ¶

func (s *Raw) Rows() int64

Rows is the number of rows handed out so far.

func (*Raw) Values ¶

func (s *Raw) Values() ([]any, error)

Values returns the current row.

The slice is reused between rows, which is safe for pgx.CopyFrom because it encodes each row before asking for the next. A caller driving the source by hand must not hold on to it.

type Reader ¶

type Reader struct {
	// contains filtered or unexported fields
}

Reader is the bottom layer both sources are built on: it strips a UTF-8 BOM, configures encoding/csv, and reads the header.

It reads one record at a time and never holds more than the current one, which is what keeps the whole pipeline to a constant amount of memory no matter how large the file is. It does not close the underlying io.Reader.

func NewReader ¶

func NewReader(r io.Reader, opts ...Option) (*Reader, error)

NewReader reads the header and prepares the reader for row-by-row use.

An empty input is not an error: Columns returns nil and Read returns io.EOF at once. What an empty file means is the caller's decision, not this package's.

func (*Reader) All ¶

func (r *Reader) All() iter.Seq[[]string]

All iterates the records left in the file.

Ranging cannot carry an error out, so the loop stops at the first bad record and Err reports it afterwards - the same shape as bufio.Scanner:

for record := range reader.All() {
	...
}
if err := reader.Err(); err != nil {
	return err
}

Breaking out of the loop leaves the reader usable, and the next pull picks up where the range left off. An error does not: the reader is done, and ranging it again yields nothing rather than resuming past the record that failed.

The yielded slice is reused by the next iteration. Copy it if it has to outlive one turn of the loop.

func (*Reader) Columns ¶

func (r *Reader) Columns() []string

Columns returns the header as it was read and normalized, or nil if the input was empty. The slice is shared, not copied - it is handed straight to pgx.CopyFrom, which does not modify it.

These names come out of the file and are untrusted input. Normalizing collapses whitespace; it does not make a name safe to paste into a statement. Before one reaches DDL, quote it - pgx.Identifier{name}.Sanitize() - or put the header through ValidateColumns. For CopyFrom, pgx quotes them itself.

func (*Reader) Err ¶

func (r *Reader) Err() error

Err returns the error that stopped reading, if any. End of file is not one.

func (*Reader) Line ¶

func (r *Reader) Line() int

Line is the 1-based line of the file the current record starts on.

It is the physical line, taken from encoding/csv rather than counted here, so it stays right across the two things that make a record count drift from it: blank lines, which encoding/csv skips, and a quoted field spanning several lines. That number is the one an error message must carry - the caller opens the file at it.

After a failed Read it names the record that failed.

func (*Reader) Read ¶

func (r *Reader) Read() ([]string, error)

Read returns the next record, or io.EOF when there are none left.

The first bad record ends the reader: the error is returned again by every later call and stays in Err. Rows are never skipped, so a caller cannot resume past a bad record and mistake a truncated file for a whole one.

The returned slice is reused by the next call. Every error other than io.EOF carries the line number and wraps one of three sentinels: ErrRecordTooLarge if the record outgrew WithMaxRecordBytes, ErrParse if the content is malformed, or ErrIO if the stream underneath failed.

func (*Reader) Record ¶

func (r *Reader) Record() []string

Record returns the record last read, for error messages that need the offending row rather than just its number.

After a failed Read it holds that record as far as encoding/csv got with it, or nil where it could not produce one at all.

Only valid until the next Read: the backing slice is reused.

type RowSource ¶

type RowSource[T any] interface {
	Next() bool
	Value() T
	Err() error
}

RowSource is anything that yields values one at a time: Typed, or a source of your own over XLSX, an API or a generator.

The interface is a pull, not a push, and that is the point. pgx.CopyFrom drives the loop itself - it asks for a row, encodes it into its send buffer, and only then asks for the next one. A source that pushes rows into a channel or a callback would need a goroutine between it and CopyFrom, and with it error handling across a goroutine boundary inside an open transaction.

type Typed ¶

type Typed[S any, D any] struct {
	// contains filtered or unexported fields
}

Typed streams a file whose shape is fixed by a struct.

S is the row as it appears in the file - every field a string, tagged with the column name it comes from. D is what the rest of the program works with, produced by convert. The split is deliberate: only the caller can decide whether an empty cell is a NULL or a zero, and whether a bad value fails the file or the field, so this package never converts anything itself.

Typed satisfies RowSource[D], so decoding and the layout of a COPY row can live in different layers: one hands out a RowSource, and the layer that owns the database decides which columns it lands in.

func NewTyped ¶

func NewTyped[S any, D any](r io.Reader, convert func(*S) (D, error), opts ...Option) (*Typed[S, D], error)

NewTyped reads the header, binds it to S's tags and prepares a source over the rows after it.

convert is called once per row with a pointer to a struct that is reused for the whole file, so it must not keep that pointer. Returning an error from it stops the stream, and the error is reported with the line it came from.

Only the fields a column bound to are written before each call. A field no tag asked for - untagged, or tagged "-" - is never touched, so whatever convert leaves in one it will see again on the next row.

Example ¶

const file = "name;id\nAlice;1\nBob;2\n"

rows, err := NewTyped(strings.NewReader(file), toEntity)
if err != nil {
	panic(err)
}

source, err := NewCopy(rows, 2, func(dst []any, e *entity) []any {
	return append(dst, e.ID, e.Name)
})
if err != nil {
	panic(err)
}

// tx.CopyFrom(ctx, pgx.Identifier{"people"}, []string{"id", "name"}, source)
for source.Next() {
	values, _ := source.Values()
	fmt.Println(values...)
}
if err = source.Err(); err != nil {
	panic(err)
}

Output:
1 Alice
2 Bob

func (*Typed[S, D]) All ¶

func (s *Typed[S, D]) All() iter.Seq[D]

All iterates the decoded rows.

This is the layer that is useful with no database in sight: CSV in, your own type out, one row at a time. Stops at the first row that fails to parse or convert, and Err reports it afterwards:

for client := range rows.All() {
	...
}
if err := rows.Err(); err != nil {
	return err
}

Unlike the other All methods, what is yielded here is whatever convert returned, so nothing is reused behind your back.

func (*Typed[S, D]) Err ¶

func (s *Typed[S, D]) Err() error

Err returns the error that stopped the stream, if any.

func (s *Typed[S, D]) Header() []string

Header returns the header as read and normalized, or nil if the file was empty.

func (*Typed[S, D]) Line ¶

func (s *Typed[S, D]) Line() int

Line is the 1-based line number of the current row.

func (*Typed[S, D]) Next ¶

func (s *Typed[S, D]) Next() bool

Next advances to the next row, stopping at the end of the file, at the first malformed record, or at the first row convert rejects.

func (*Typed[S, D]) Record ¶

func (s *Typed[S, D]) Record() []string

Record returns the raw record behind the current row, for error messages.

func (*Typed[S, D]) Rows ¶

func (s *Typed[S, D]) Rows() int64

Rows is the number of rows handed out so far. Zero after a full pass means the file held nothing, which is not in itself an error.

func (*Typed[S, D]) Truncated ¶ added in v0.1.0

func (s *Typed[S, D]) Truncated() bool

Truncated reports whether the record behind the current row ran out before a bound column, so that field holds "" because the value was absent rather than empty.

Only meaningful after Next returned true, and only ever true under WithVariableColumns - without it a short record is an error instead.

It exists because the distinction is invisible where it matters most. Raw can put nil in an []any and get SQL NULL; a tagged field is declared string, so an absent value and an empty one both arrive as "", and in Postgres a NULL and an empty string are different values. Read it after Next if that difference matters:

for source.Next() {
	if source.Truncated() {
		log.Warn("short record", "line", source.Line())
	}
}

convert cannot see this. It is called inside Next, with the struct as its only argument, and adding a second one would change the signature every caller writes. If a row needs to be rejected for being short, reject it here.

func (*Typed[S, D]) Unused ¶

func (s *Typed[S, D]) Unused() []string

Unused returns the header columns no tag bound to.

Worth logging as a warning: when an export renames a column, the tag stops matching it, no error is raised, and the column that was renamed away is the only trace left.

The slice is shared, not copied. It is built once per file and never written again, so reading it is safe; do not modify it.

func (*Typed[S, D]) Value ¶

func (s *Typed[S, D]) Value() D

Value returns the row Next produced. Only meaningful after Next returned true.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

README ¶

go-csv-copy

📋 Overview

Why pgx is not imported

📦 Install

🚀 Quick start

Shape from the file header (staging table)

Shape from struct tags (typed load)

Without a database

🏗 API

Layers

Interfaces

Options

The csv tag

Why ; and not ,

Diagnostics

nil in Raw, "" in Typed

Errors

📐 Semantics and guarantees

⚡ Performance

It matters more the wider the file

The encode mistake that undoes it

🔧 Fitting it into existing code

🔍 Alternatives

📄 License

Documentation ¶

Overview ¶

Layers ¶

Without a database ¶

Guarantees ¶

Index ¶

Examples ¶

Constants ¶

Variables ¶

Functions ¶

func NormalizeSpace ¶

func ValidateColumns ¶ added in v0.1.0

Types ¶

type Copy ¶

func NewCopy ¶

func (*Copy[T]) Err ¶

func (*Copy[T]) Line ¶ added in v0.1.0

func (*Copy[T]) Next ¶

func (*Copy[T]) Record ¶ added in v0.1.0

func (*Copy[T]) Rows ¶

func (*Copy[T]) Values ¶

type Option ¶

func WithAllowMissingColumns ¶

func WithComma ¶

func WithComment ¶ added in v0.1.0

func WithHeaderRow ¶

func WithLazyQuotes ¶

func WithMaxRecordBytes ¶ added in v0.1.0

func WithNormalizeHeader ¶

func WithPointerValues ¶

func WithTag ¶

func WithTrimLeadingSpace ¶

func WithTrimValues ¶

func WithVariableColumns ¶

type Raw ¶

func NewRaw ¶

func (*Raw) All ¶

func (*Raw) Columns ¶

func (*Raw) Err ¶

func (*Raw) Line ¶

func (*Raw) Next ¶

func (*Raw) Record ¶

func (*Raw) Rows ¶

func (*Raw) Values ¶

type Reader ¶

func NewReader ¶

func (*Reader) All ¶

func (*Reader) Columns ¶

func (*Reader) Err ¶

func (*Reader) Line ¶

func (*Reader) Read ¶

func (*Reader) Record ¶

type RowSource ¶

type Typed ¶

func NewTyped ¶

The `csv` tag

Why `;` and not `,`

`nil` in `Raw`, `""` in `Typed`

The `encode` mistake that undoes it