csvcopy

README

go-csv-copy

Streaming CSV reader for Go, shaped for bulk-loading into PostgreSQL. No dependencies outside the standard library.

⸻

📋 Overview

The package reads CSV one row at a time: a row is read, handed over and forgotten, so memory does not depend on the size of the file.

What it is built for is COPY FROM — the row sources satisfy pgx.CopyFromSource, so a file of any size loads into Postgres without being buffered. That is the primary use case, not the only one: Reader is a plain CSV reader and Typed decodes into your own types, both usable with no database in sight. See Without a database.

Two ways of fixing a file's shape are supported, because both turn up in practice:

• Raw — the shape comes from the file's own header. Whatever columns arrived, in their order, every value as text. This is the staging-table case, where the input's structure is not known ahead of time and a SQL script types the data afterwards.
• Typed[S, D] — the shape comes from csv struct tags. Columns are matched by name, so their order in the file does not matter, extra columns are ignored, and a column a tag asks for and the file lacks is an error. Each row passes through your convert, which types and validates the values.

Why pgx is not imported

The package does not depend on pgx and is compatible with it anyway. Go interfaces are structural, and pgx.CopyFromSource is nothing more than a method set:

type CopyFromSource interface {
    Next() bool
    Values() ([]any, error)
    Err() error
}

Any type with those methods satisfies it without declaring a dependency. So the application picks the version of pgx, not the library, and upgrading pgx never requires upgrading csvcopy.

⸻

📦 Install

go get github.com/AMUDENN/go-csv-copy

import "github.com/AMUDENN/go-csv-copy"   // package csvcopy

Go 1.24 or newer. The go directive names a minor version on purpose: pinning a patch would force every consumer to fetch a toolchain, which is a strange thing for a package whose whole point is bringing nothing with it.

⸻

🚀 Quick start

Shape from the file header (staging table)

src, err := csvcopy.NewRaw(file)
if err != nil {
    return fmt.Errorf("parse input: %w", err)
}

columns := src.Columns()
if len(columns) == 0 {
    return nil // an empty file is not an error
}

if err := createStagingTable(ctx, tx, table, columns); err != nil {
    return err
}

n, err := tx.CopyFrom(ctx, pgx.Identifier{table}, columns, src)
if srcErr := src.Err(); srcErr != nil {
    return fmt.Errorf("parse input: %w", srcErr)
}
if err != nil {
    return fmt.Errorf("copy from: %w", err)
}

Check src.Err() before err: when the source fails on a row, pgx returns its own, less informative error, while the cause sits in the source together with the line number.

Shape from struct tags (typed load)

// The shape of the file: all text, tags name the columns in the header.
type clientRow struct {
    ID        string `csv:"client_id"`
    Email     string `csv:"email"`
    Balance   string `csv:"balance"`
    CreatedAt string `csv:"created_at"`
}

// Your typing and validation - the package stays out of it.
func (r *clientRow) toClient() (*Client, error) { /* ... */ }

var clientColumns = []string{"id", "email", "balance", "created_at"}

rows, err := csvcopy.NewTyped(file, (*clientRow).toClient)
if err != nil {
    return err
}

source, err := csvcopy.NewCopy(rows, len(clientColumns),
    func(dst []any, c *Client) []any {
        return append(dst, c.ID, c.Email, c.Balance, c.CreatedAt)
    })
if err != nil {
    return err
}

_, err = tx.CopyFrom(ctx, pgx.Identifier{"clients_tmp"}, clientColumns, source)
if err != nil {
    if srcErr := source.Err(); srcErr != nil {
        err = srcErr
    }
    return fmt.Errorf("copy from: %w", err)
}

if source.Rows() == 0 {
    // the file was empty - nothing to load
}

Typed satisfies RowSource[D], so decoding and the layout of a COPY row can live in different layers: one hands out a RowSource, and the layer that owns the database decides which columns it lands in.

Without a database

Reader and Typed stand on their own — no COPY, no pgx, no []any. Ranging cannot carry an error out, so the loop stops at the first bad row and Err() reports it afterwards, the same shape as bufio.Scanner:

rows, err := csvcopy.NewTyped(file, (*clientRow).toClient)
if err != nil {
    return err
}

for client := range rows.All() {
    if err := send(ctx, client); err != nil {
        return err
    }
}
if err := rows.Err(); err != nil {
    return err
}

When no decoding is wanted either, Reader yields raw records:

reader, err := csvcopy.NewReader(file)
if err != nil {
    return err
}

fmt.Println(reader.Columns())

for record := range reader.All() {
    fmt.Println(record) // reused between iterations - copy it to keep it
}
return reader.Err()

⸻

🏗 API

Layers

Constructor	Returns	Use when
NewReader(r, opts...)	*Reader — header + row-by-row reading	you only need CSV parsing, no COPY
NewRaw(r, opts...)	*Raw — pgx.CopyFromSource + Columns()	the shape comes from the file header
NewTyped[S, D](r, convert, opts...)	*Typed[S, D] — RowSource[D]	the shape comes from csv tags
NewCopy[T](src, columns, encode)	*Copy[T] — pgx.CopyFromSource	adapting any RowSource[T] to COPY

Every constructor returns an error rather than panicking. Nil wiring — a nil reader, convert, src or encode — is ErrSchema.

Interfaces

// RowSource is anything that yields values one at a time. Typed satisfies it,
// and so can a source of your own over XLSX, an API or a generator.
type RowSource[T any] interface {
    Next() bool
    Value() T
    Err() error
}

Options

Option	Default	Effect
WithComma(r rune)	';'	field delimiter
WithLazyQuotes(bool)	true	tolerate a bare quote inside a field instead of failing
WithTrimLeadingSpace(bool)	true	drop white space at the start of a field
WithTrimValues(bool)	true	TrimSpace every value
WithHeaderRow(n uint)	1	which row holds the header; rows above it are dropped
WithNormalizeHeader(fn)	collapse whitespace	normalize a column name before matching
WithTag(string)	"csv"	which struct tag Typed reads column names from
WithVariableColumns(bool)	false	accept rows of a different width: missing trailing values → nil, extra ones dropped
WithAllowMissingColumns(bool)	false	do not fail when a tagged column is absent from the header
WithPointerValues(bool)	false	Raw yields *string instead of string, removing one allocation per cell

The csv tag

The whole tag is the column name. There are no comma-separated options: a field tagged csv:"name,omitempty" asks for a column literally called name,omitempty, which no header will have, and ErrMissingColumns says so. - is the one value with a meaning of its own — it drops the field.

Two rules exist because breaking them loses data silently, and the package refuses both with ErrSchema:

type row struct {
    base                       // ✗ tags inside an embedded struct are not matched
    A string `csv:"id"`
    B string `csv:"id"`        // ✗ two fields, one column
}

Embedded fields are not walked into. Left to pass, a tag one level down would bind to nothing, the field would read as empty on every row, and the column it named would reach the database as NULL — the same damage ErrMissingColumns exists to prevent, only without the error. List the columns on the struct itself.

Diagnostics

Method	Gives
All()	a range-able iterator: []string on Reader, []any on Raw, D on Typed
Err()	the error that stopped the stream; end of file is not one
Columns() / Header()	the header as read and normalized
Record()	the raw record last read, for error messages
Unused()	header columns no tag bound to
Line()	the 1-based physical line of the file the current row starts on
Rows()	how many rows have been handed out

Unused() is worth logging as a warning: when an export renames a column, the tag simply matches nothing, no error is raised, and the wrong data reaches the database. The unbound column is the only visible trace.

⚠️ WithAllowMissingColumns(true) is dangerous. The absent field reads as the empty string on every row and reaches the database as NULL, quietly wiping whatever that column held. Only use it where that is acceptable.

Errors

Errors are split by who can fix them.

Sentinel	Wraps ErrParse	Cause
ErrParse	—	a malformed record, a failing convert, a read failure
ErrMissingColumns	yes	the header lacks a column a tag asks for
ErrSchema	no	a bug in the calling code: not a struct, a tag on a non-string or unexported field, a tag inside an embedded struct, two fields asking for one column, a nil reader/convert/src/encode, an unusable delimiter

Everything a file can cause wraps ErrParse and carries the line number:

csv parse: record on line 4213: wrong number of fields
csv parse: line 4213: balance: strconv.ParseFloat: parsing "n/a": invalid syntax

The line is the physical line of the file, not a count of records, so it stays right across blank lines (encoding/csv skips them) and quoted fields spanning several lines. It is what Line() returns, and the number to open the file at.

If your application already has a sentinel for a bad file, alias it and every existing errors.Is check keeps working:

var ErrBadFile = csvcopy.ErrParse

ErrSchema deliberately does not wrap ErrParse: no input file will ever fix it, so quarantine-the-file logic must not fire on it. It is a code bug, and it surfaces on the very first run — including a run on an empty file.

⸻

📐 Semantics and guarantees

• Empty input (zero bytes, or only a BOM) is not an error: Columns() == nil, Next() == false, Err() == nil, Rows() == 0.
• The BOM (EF BB BF) is stripped, read with io.ReadFull so that a short read from a slow io.Reader — which the contract allows — cannot leave a  in the first column name.
• The first bad row stops the stream, for good. Rows are never skipped: a partly loaded table is worse than a failed load. The error is available from Err(), and it stays there — a source that has failed yields nothing more, so ranging All() a second time cannot resume past the bad row. Breaking out of a loop is different: that is not an error, and the next pull carries on.
• Record() names the row that failed, as far as encoding/csv got with it. Line() names the physical line it starts on.
• Memory is constant and independent of the row count. Values() reuses one slice, which is safe under pgx.CopyFrom because it encodes a row before asking for the next. If you drive a source by hand, do not retain the result of Values() between iterations.
• Not safe for concurrent use. One source, one goroutine — the same as pgx.CopyFrom.
• Reader, Raw and Typed do not close the io.Reader you give them.

⸻

⚡ Performance

Live memory is constant, but allocations over a pass grow linearly, and it is worth knowing where from. 100k rows of 5 columns, go1.26.1, Ryzen 5 7500F:

	ns/op	B/op	allocs/op	per row
Typed	13.6 ms	3.2 MB	100 038	1
Typed via All()	14.4 ms	3.2 MB	100 038	1
Raw	18.7 ms	11.2 MB	600 033	6
Raw via All()	20.6 ms	11.2 MB	600 032	6
Raw + WithPointerValues	11.2 ms	3.2 MB	100 033	1

Ranging costs nothing: an iter.Seq returned from a method closes over the source once per pass, not once per row, so All() sits on the same allocation count as the Next() loop it replaces.

The one allocation per row is encoding/csv: it creates one string per record even with ReuseRecord, and that is the floor short of unsafe.

The other five are the columns. Boxing a string into an any always allocates 16 bytes for its header, so Raw pays one allocation per cell by default — that is the shape of Values() ([]any, error). WithPointerValues(true) yields *string out of an array allocated once per file: a pointer is pointer-shaped, the interface holds it directly, and boxing is free. On a 30-column file the difference is thirtyfold.

The option is off by default: pgx dereferences *T through its pointer encode plan and *string is the ordinary way to pass a nullable text value, but this has not been verified against a real Postgres. Turn it on deliberately and check the loaded result.

In Copy the boxing happens inside your own encode, so the option does not affect it.

⸻

🔧 Fitting it into existing code

Both layers rely on structural typing, so the package usually drops in without reshaping the caller:

• If you already have your own row-source interface with Next/Value/Err, it can be replaced by csvcopy.RowSource[T] — the method sets match and the implementations do not change.
• If another format is read alongside CSV (XLSX, a stream from an API), it is enough for its source to implement the same Next/Values/Err. Both branches stay pgx.CopyFromSource, and choosing a format comes down to returning different values of one interface.
• Your own sentinel for a bad file becomes an alias of csvcopy.ErrParse, and errors.Is keeps working.

⸻

🔍 Alternatives

Package	License	Streaming	Shape from header	CopyFromSource
csvcopy	MIT	yes, pull	yes	yes
jszwec/csvutil	MIT	yes, pull	no	no
gocarina/gocsv	MIT	yes, push	no	no
artonge/go-csv-tag	GPL-3.0	no, ReadAll	no	no

The difference between pull and push matters here. pgx.CopyFrom pulls: it calls Next(), encodes the row into its buffer, and only then asks for the next one. Packages like gocsv push — UnmarshalToChan(in io.Reader, c interface{}) and UnmarshalToCallback(in io.Reader, f interface{}) drive the loop themselves. Connecting such a source to CopyFrom needs a goroutine and a channel between them, which means concurrency, backpressure and carrying an error across a goroutine boundary, all inside an open transaction. A pull interface of three methods needs none of that.

If you only need CSV decoded into structs and no COPY FROM, use csvutil: it is more mature and does more (inline, omitempty, custom unmarshalers). csvcopy solves the narrower problem of loading into PostgreSQL, which is why it has a layer that takes its shape from the file header and adapters for CopyFrom, and why it deliberately does no type conversion — that belongs to the caller, who needs to tell "empty → NULL" from "zero → 0".

⸻

📄 License

MIT.

Documentation

Overview

Package csvcopy streams CSV one row at a time.

A row is read, handed over and forgotten, so memory does not depend on the size of the file. There are no dependencies outside the standard library.

Bulk-loading into PostgreSQL is what it is shaped for, and the reason the row sources expose Next/Values/Err: that method set is pgx.CopyFromSource, satisfied structurally rather than by importing pgx, so the version of pgx stays the application's choice. Nothing here requires a database, though - Reader is a plain CSV reader, and Typed decodes into your own types. Both have All for ranging.

Layers

Two ways to fix the shape of a file, because both turn up in practice.

Raw takes the shape from the file's own header - whatever columns arrived, in their order, all as text. This is the staging table case, where a SQL script types the data afterwards:

src, err := csvcopy.NewRaw(file)
if err != nil {
	return err
}
if len(src.Columns()) == 0 {
	return nil // empty file, nothing to load
}
n, err := tx.CopyFrom(ctx, pgx.Identifier{table}, src.Columns(), src)

Typed takes the shape from struct tags. Columns are matched by name, so the file may reorder them or add new ones; a column a tag asks for and the file lacks is an error. Every tagged field is a string, and convert turns the row into whatever the program actually works with - only the caller can tell an empty cell from a zero:

type row struct {
	ID   string `csv:"id"`
	Name string `csv:"name"`
}

rows, err := csvcopy.NewTyped(file, (*row).toEntity)
if err != nil {
	return err
}

source, err := csvcopy.NewCopy(rows, len(columns), func(dst []any, e *entity) []any {
	return append(dst, e.ID, e.Name)
})
if err != nil {
	return err
}
n, err := tx.CopyFrom(ctx, pgx.Identifier{table}, columns, source)

NewCopy adapts any RowSource - Typed, or one of your own over XLSX or an API - to pgx.CopyFromSource.

Without a database

Reader and Typed stand on their own. Ranging stops at the first bad row and Err reports it afterwards, the same shape as bufio.Scanner:

rows, err := csvcopy.NewTyped(file, (*row).toEntity)
if err != nil {
	return err
}

for entity := range rows.All() {
	send(entity)
}
if err := rows.Err(); err != nil {
	return err
}

Reader.All yields raw records as []string, for when no decoding is wanted either.

Guarantees

An empty input is not an error: no columns, no rows, no error. The caller decides what that means.

A UTF-8 BOM is stripped, read with io.ReadFull so a slow reader cannot leave it in place.

The first bad row stops the stream for good. Rows are not skipped: a partly loaded table is worse than a failed load. The error is available from Err, names the line, and stays there - a source that has failed yields nothing more, so ranging All again cannot resume past the row that broke. Breaking out of a loop is not an error, and the next pull carries on from where it stopped.

Record and Line name the row that failed, so an error message can carry it. Line is the physical line of the file, taken from encoding/csv, so it stays right across blank lines and quoted fields spanning several lines.

Errors a file can cause wrap ErrParse, so an application can alias its own sentinel to it. Errors the calling code causes wrap ErrSchema instead, because no file will fix them.

Values reuses one slice between rows. That is safe under pgx.CopyFrom, which encodes a row before asking for the next; a caller driving a source by hand must not retain it.

Nothing here is safe for concurrent use, and neither is pgx.CopyFrom.

Index

• Variables
• func NormalizeSpace(s string) string
• type Copy
  • func NewCopy[T any](src RowSource[T], columns int, encode func(dst []any, item T) []any) (*Copy[T], error)
  • func (c *Copy[T]) Err() error
  • func (c *Copy[T]) Next() bool
  • func (c *Copy[T]) Rows() int64
  • func (c *Copy[T]) Values() ([]any, error)
• type Option
  • func WithAllowMissingColumns(allow bool) Option
  • func WithComma(comma rune) Option
  • func WithHeaderRow(row uint) Option
  • func WithLazyQuotes(lazy bool) Option
  • func WithNormalizeHeader(fn func(string) string) Option
  • func WithPointerValues(pointers bool) Option
  • func WithTag(tag string) Option
  • func WithTrimLeadingSpace(trim bool) Option
  • func WithTrimValues(trim bool) Option
  • func WithVariableColumns(variable bool) Option
• type Raw
  • func NewRaw(r io.Reader, opts ...Option) (*Raw, error)
  • func (s *Raw) All() iter.Seq[[]any]
  • func (s *Raw) Columns() []string
  • func (s *Raw) Err() error
  • func (s *Raw) Line() int
  • func (s *Raw) Next() bool
  • func (s *Raw) Record() []string
  • func (s *Raw) Rows() int64
  • func (s *Raw) Values() ([]any, error)
• type Reader
  • func NewReader(r io.Reader, opts ...Option) (*Reader, error)
  • func (r *Reader) All() iter.Seq[[]string]
  • func (r *Reader) Columns() []string
  • func (r *Reader) Err() error
  • func (r *Reader) Line() int
  • func (r *Reader) Read() ([]string, error)
  • func (r *Reader) Record() []string
• type RowSource
• type Typed
  • func NewTyped[S any, D any](r io.Reader, convert func(*S) (D, error), opts ...Option) (*Typed[S, D], error)
  • func (s *Typed[S, D]) All() iter.Seq[D]
  • func (s *Typed[S, D]) Err() error
  • func (s *Typed[S, D]) Header() []string
  • func (s *Typed[S, D]) Line() int
  • func (s *Typed[S, D]) Next() bool
  • func (s *Typed[S, D]) Record() []string
  • func (s *Typed[S, D]) Rows() int64
  • func (s *Typed[S, D]) Unused() []string
  • func (s *Typed[S, D]) Value() D

Examples

• NewTyped

Variables

var ErrMissingColumns = fmt.Errorf("%w: header is missing columns", ErrParse)

ErrMissingColumns reports columns that a csv tag asked for and the header does not have.

This is fatal rather than a warning: a field bound to nothing would read as the empty string on every row and reach the database as NULL, quietly wiping the column it was supposed to fill. WithAllowMissingColumns lifts the check for callers who accept that risk.

var ErrParse = errors.New("csv parse")

ErrParse is wrapped by every error that a file can cause: a malformed record, a header that does not match the struct, a failing convert. An application can therefore alias its own sentinel to it and keep its existing errors.Is checks working:

var ErrBadFile = csvcopy.ErrParse

var ErrSchema = errors.New("csv schema")

ErrSchema reports wiring that cannot work whatever the file holds: a struct that cannot be decoded into at all - not a struct, a tagged field that is not a string or not exported, two fields asking for one column, a tag inside an embedded struct where it would bind to nothing - a nil reader, convert, src or encode, or a delimiter encoding/csv will not accept.

It deliberately does not wrap ErrParse. No input file will ever fix it, so a caller that retries or quarantines files on ErrParse should not treat it as a bad file - it is a bug in the calling code.

Functions

func NormalizeSpace

func NormalizeSpace(s string) string

NormalizeSpace is the default header normalizer: it collapses every run of whitespace into a single space and trims the ends.

Exported column names arrive wrapped across lines or padded for alignment, so "date of\n birth" and "date of birth" have to resolve to the same column.

Types

type Copy

type Copy[T any] struct {
	// contains filtered or unexported fields
}

Copy adapts a RowSource to pgx.CopyFromSource.

Only the mapping of a value onto its columns lives here; where the values came from is none of this layer's business.

func NewCopy

func NewCopy[T any](src RowSource[T], columns int, encode func(dst []any, item T) []any) (*Copy[T], error)

NewCopy wraps src, laying each value out with encode.

columns is how many values encode appends; it only sizes the backing slice, so being wrong costs an allocation rather than correctness. encode must append in the same order as the column list passed to pgx.CopyFrom - the two are a pair, and Postgres cannot notice when they disagree if the types are compatible.

A nil src or encode is ErrSchema, the same as a nil reader or convert elsewhere in the package: it is wiring the calling code got wrong, and no input file will fix it.

func (*Copy[T]) Err

func (c *Copy[T]) Err() error

Err reports the underlying source's error.

func (*Copy[T]) Next

func (c *Copy[T]) Next() bool

Next advances the underlying source.

func (*Copy[T]) Rows

func (c *Copy[T]) Rows() int64

Rows is the number of rows handed out so far. Counted here rather than asked of the source, which need not track it.

func (*Copy[T]) Values

func (c *Copy[T]) Values() ([]any, error)

Values lays out the current value.

The result is stored back, so a row wider than columns grows the slice once instead of reallocating on every row. Safe to reuse because pgx encodes the row before pulling the next one.

type Option

type Option func(*settings)

Option configures a Reader and everything built on top of it.

func WithAllowMissingColumns

func WithAllowMissingColumns(allow bool) Option

WithAllowMissingColumns downgrades a missing tagged column from an error to silence.

Dangerous: the field will read as empty on every row and reach the database as NULL, wiping whatever that column held. Only use it where that is acceptable.

func WithComma

func WithComma(comma rune) Option

WithComma sets the field delimiter. Defaults to ';'.

A quote, a carriage return, a newline and an invalid rune cannot delimit anything encoding/csv is willing to read, so the constructor rejects them with ErrSchema rather than letting the first row fail with ErrParse.

func WithHeaderRow

func WithHeaderRow(row uint) Option

WithHeaderRow sets which row holds the header, counting from 1. Rows before it are read and dropped, so a file may carry a title or a note above its table. Defaults to 1.

A header past the end of the file is the empty-file case, not an error.

func WithLazyQuotes

func WithLazyQuotes(lazy bool) Option

WithLazyQuotes allows a bare quote inside an unquoted field instead of failing the record. Defaults to true.

func WithNormalizeHeader

func WithNormalizeHeader(fn func(string) string) Option

WithNormalizeHeader replaces the header normalizer. Passing nil restores identity. Defaults to NormalizeSpace.

func WithPointerValues

func WithPointerValues(pointers bool) Option

WithPointerValues makes Raw hand out *string instead of string, which removes one allocation per column per row.

Putting a string into an []any boxes it, and boxing a string always allocates 16 bytes for its header - so the default costs one allocation per cell, which on a wide file dwarfs everything else. A pointer is pointer-shaped: the interface holds it directly and boxing is free. The strings live in one array allocated per file, and a value the row stopped short of is a nil interface, still NULL.

Off by default on purpose. pgx dereferences *T through its pointer encode plan and *string is the ordinary way to pass a nullable text value, so this should be transparent - but "should" is not "measured against a real database". Turn it on, load a real file, compare the result, then make it the default.

Affects Raw only. In Copy the boxing happens inside your own encode func.

func WithTag

func WithTag(tag string) Option

WithTag sets the struct tag Typed reads column names from. Defaults to "csv".

The whole tag is the column name. There are no comma-separated options - a field tagged `csv:"name,omitempty"` asks for a column literally called "name,omitempty", which no header will have. The only value with a meaning of its own is "-", which drops the field.

func WithTrimLeadingSpace

func WithTrimLeadingSpace(trim bool) Option

WithTrimLeadingSpace drops white space at the start of a field. Defaults to true.

func WithTrimValues

func WithTrimValues(trim bool) Option

WithTrimValues applies strings.TrimSpace to every value. Defaults to true.

func WithVariableColumns

func WithVariableColumns(variable bool) Option

WithVariableColumns accepts rows whose field count differs from the header's. Missing trailing values become NULL and extra ones are dropped.

Off by default: a row that is the wrong width usually means the delimiter or the quoting is misread, and failing beats loading shifted data.

type Raw

type Raw struct {
	// contains filtered or unexported fields
}

Raw streams a file whose shape is only known from its own header.

Every column arrives as text, in the file's order, under the file's names - the shape of a staging table that a SQL script types afterwards. Nothing here knows or cares what the columns mean.

It satisfies pgx.CopyFromSource. That interface is not imported: Go interfaces are structural, so the method set alone is enough and the version of pgx stays the application's choice.

func NewRaw

func NewRaw(r io.Reader, opts ...Option) (*Raw, error)

NewRaw reads the header and prepares a source over the rows after it.

func (*Raw) All

func (s *Raw) All() iter.Seq[[]any]

All iterates the rows, for driving the source by hand rather than handing it to pgx.CopyFrom.

Stops at the first bad row; Err reports it afterwards. The yielded slice is reused by the next iteration.

If you want the values as text rather than as []any, range Reader.All instead - there is no reason to go through the boxing.

func (*Raw) Columns

func (s *Raw) Columns() []string

Columns is the column list to pass to pgx.CopyFrom, taken from the header.

Empty when the file was empty - check it before creating a table, there is nothing to load.

func (*Raw) Err

func (s *Raw) Err() error

Err returns the error that stopped the stream, if any. Check it before the error pgx.CopyFrom returns: this one names the line.

func (*Raw) Line

func (s *Raw) Line() int

Line is the 1-based line number of the current row.

func (*Raw) Next

func (s *Raw) Next() bool

Next advances to the next row, stopping at the end of the file or at the first malformed record.

func (*Raw) Record

func (s *Raw) Record() []string

Record returns the raw record behind the current row, for error messages.

func (*Raw) Rows

func (s *Raw) Rows() int64

Rows is the number of rows handed out so far.

func (*Raw) Values

func (s *Raw) Values() ([]any, error)

Values returns the current row.

The slice is reused between rows, which is safe for pgx.CopyFrom because it encodes each row before asking for the next. A caller driving the source by hand must not hold on to it.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader is the bottom layer both sources are built on: it strips a UTF-8 BOM, configures encoding/csv, and reads the header.

It reads one record at a time and never holds more than the current one, which is what keeps the whole pipeline to a constant amount of memory no matter how large the file is. It does not close the underlying io.Reader.

func NewReader

func NewReader(r io.Reader, opts ...Option) (*Reader, error)

NewReader reads the header and prepares the reader for row-by-row use.

An empty input is not an error: Columns returns nil and Read returns io.EOF at once. What an empty file means is the caller's decision, not this package's.

func (*Reader) All

func (r *Reader) All() iter.Seq[[]string]

All iterates the records left in the file.

Ranging cannot carry an error out, so the loop stops at the first bad record and Err reports it afterwards - the same shape as bufio.Scanner:

for record := range reader.All() {
	...
}
if err := reader.Err(); err != nil {
	return err
}

Breaking out of the loop leaves the reader usable, and the next pull picks up where the range left off. An error does not: the reader is done, and ranging it again yields nothing rather than resuming past the record that failed.

The yielded slice is reused by the next iteration. Copy it if it has to outlive one turn of the loop.

func (*Reader) Columns

func (r *Reader) Columns() []string

Columns returns the header as it was read and normalized, or nil if the input was empty. The slice is shared, not copied - it is handed straight to pgx.CopyFrom, which does not modify it.

func (*Reader) Err

func (r *Reader) Err() error

Err returns the error that stopped reading, if any. End of file is not one.

func (*Reader) Line

func (r *Reader) Line() int

Line is the 1-based line of the file the current record starts on.

It is the physical line, taken from encoding/csv rather than counted here, so it stays right across the two things that make a record count drift from it: blank lines, which encoding/csv skips, and a quoted field spanning several lines. That number is the one an error message must carry - the caller opens the file at it.

After a failed Read it names the record that failed.

func (*Reader) Read

func (r *Reader) Read() ([]string, error)

Read returns the next record, or io.EOF when there are none left.

The first bad record ends the reader: the error is returned again by every later call and stays in Err. Rows are never skipped, so a caller cannot resume past a bad record and mistake a truncated file for a whole one.

The returned slice is reused by the next call. Every error other than io.EOF wraps ErrParse and carries the line number.

func (*Reader) Record

func (r *Reader) Record() []string

Record returns the record last read, for error messages that need the offending row rather than just its number.

After a failed Read it holds that record as far as encoding/csv got with it, or nil where it could not produce one at all.

Only valid until the next Read: the backing slice is reused.

type RowSource

type RowSource[T any] interface {
	Next() bool
	Value() T
	Err() error
}

RowSource is anything that yields values one at a time: Typed, or a source of your own over XLSX, an API or a generator.

The interface is a pull, not a push, and that is the point. pgx.CopyFrom drives the loop itself - it asks for a row, encodes it into its send buffer, and only then asks for the next one. A source that pushes rows into a channel or a callback would need a goroutine between it and CopyFrom, and with it error handling across a goroutine boundary inside an open transaction.

type Typed

type Typed[S any, D any] struct {
	// contains filtered or unexported fields
}

Typed streams a file whose shape is fixed by a struct.

S is the row as it appears in the file - every field a string, tagged with the column name it comes from. D is what the rest of the program works with, produced by convert. The split is deliberate: only the caller can decide whether an empty cell is a NULL or a zero, and whether a bad value fails the file or the field, so this package never converts anything itself.

Typed satisfies RowSource[D], so decoding and the layout of a COPY row can live in different layers: one hands out a RowSource, and the layer that owns the database decides which columns it lands in.

func NewTyped

func NewTyped[S any, D any](r io.Reader, convert func(*S) (D, error), opts ...Option) (*Typed[S, D], error)

NewTyped reads the header, binds it to S's tags and prepares a source over the rows after it.

convert is called once per row with a pointer to a struct that is reused for the whole file, so it must not keep that pointer. Returning an error from it stops the stream, and the error is reported with the line it came from.

Only the fields a column bound to are written before each call. A field no tag asked for - untagged, or tagged "-" - is never touched, so whatever convert leaves in one it will see again on the next row.

Example

const file = "name;id\nAlice;1\nBob;2\n"

rows, err := NewTyped(strings.NewReader(file), toEntity)
if err != nil {
	panic(err)
}

source, err := NewCopy(rows, 2, func(dst []any, e *entity) []any {
	return append(dst, e.ID, e.Name)
})
if err != nil {
	panic(err)
}

// tx.CopyFrom(ctx, pgx.Identifier{"people"}, []string{"id", "name"}, source)
for source.Next() {
	values, _ := source.Values()
	fmt.Println(values...)
}
if err = source.Err(); err != nil {
	panic(err)
}

Output:
1 Alice
2 Bob

func (*Typed[S, D]) All

func (s *Typed[S, D]) All() iter.Seq[D]

All iterates the decoded rows.

This is the layer that is useful with no database in sight: CSV in, your own type out, one row at a time. Stops at the first row that fails to parse or convert, and Err reports it afterwards:

for client := range rows.All() {
	...
}
if err := rows.Err(); err != nil {
	return err
}

Unlike the other All methods, what is yielded here is whatever convert returned, so nothing is reused behind your back.

func (*Typed[S, D]) Err

func (s *Typed[S, D]) Err() error

Err returns the error that stopped the stream, if any.

func (*Typed[S, D]) Header

func (s *Typed[S, D]) Header() []string

Header returns the header as read and normalized, or nil if the file was empty.

func (*Typed[S, D]) Line

func (s *Typed[S, D]) Line() int

Line is the 1-based line number of the current row.

func (*Typed[S, D]) Next

func (s *Typed[S, D]) Next() bool

Next advances to the next row, stopping at the end of the file, at the first malformed record, or at the first row convert rejects.

func (*Typed[S, D]) Record

func (s *Typed[S, D]) Record() []string

Record returns the raw record behind the current row, for error messages.

func (*Typed[S, D]) Rows

func (s *Typed[S, D]) Rows() int64

Rows is the number of rows handed out so far. Zero after a full pass means the file held nothing, which is not in itself an error.

func (*Typed[S, D]) Unused

func (s *Typed[S, D]) Unused() []string

Unused returns the header columns no tag bound to.

Worth logging as a warning: when an export renames a column, the tag stops matching it, no error is raised, and the column that was renamed away is the only trace left.

The slice is shared, not copied. It is built once per file and never written again, so reading it is safe; do not modify it.

func (*Typed[S, D]) Value

func (s *Typed[S, D]) Value() D

Value returns the row Next produced. Only meaningful after Next returned true.