Documentation
¶
Overview ¶
Package decode reads CSV one row at a time and turns it into your values.
A row is read, handed over and forgotten, so memory does not depend on the size of the file. Nothing here knows about a database: the sources satisfy pgx.CopyFromSource by having the right method set, and pgx is not imported. Laying a value out across the columns of a COPY is copyfrom's job, and this package does not know that package exists.
Three layers, from the file up:
Reader records as []string, straight from encoding/csv. Raw one value per column of the file's own header, all text. Typed a struct fixed by csv tags, converted by a func you supply.
Raw is the staging-table case: whatever columns arrived, in their order, typed later by a SQL script. Typed is the case where the program knows what a row means - columns are matched by name, so an export may reorder them or add new ones, and a column a tag asks for and the file lacks is an error rather than a silent NULL.
Every tagged field is a string on purpose. Only the caller can tell an empty cell from a zero value, or decide whether a bad one fails the file or the field, so this package converts nothing itself.
All three have All for ranging, and stop at the first bad row with the reason in Err - the same shape as bufio.Scanner:
rows, err := decode.NewTyped(file, (*row).toEntity)
if err != nil {
return err
}
for entity := range rows.All() {
send(entity)
}
if err := rows.Err(); err != nil {
return err
}
Errors wrap csvcopy.ErrParse, csvcopy.ErrIO or csvcopy.ErrSchema, by who can fix them. See the csvcopy package doc for the guarantees this one is held to.
Index ¶
- Variables
- func NormalizeSpace(s string) string
- type Option
- func WithAllowMissingColumns(allow bool) Option
- func WithComma(comma rune) Option
- func WithComment(comment rune) Option
- func WithHeaderRow(row uint) Option
- func WithLazyQuotes(lazy bool) Option
- func WithMaxRecordBytes(n int64) Option
- func WithNormalizeHeader(fn func(string) string) Option
- func WithPointerValues(pointers bool) Option
- func WithTag(tag string) Option
- func WithTrimLeadingSpace(trim bool) Option
- func WithTrimValues(trim bool) Option
- func WithVariableColumns(variable bool) Option
- type Raw
- type Reader
- type Typed
- func (s *Typed[S, D]) All() iter.Seq[D]
- func (s *Typed[S, D]) Err() error
- func (s *Typed[S, D]) Extra() int
- func (s *Typed[S, D]) Header() []string
- func (s *Typed[S, D]) Line() int
- func (s *Typed[S, D]) Next() bool
- func (s *Typed[S, D]) Record() []string
- func (s *Typed[S, D]) Rows() int64
- func (s *Typed[S, D]) Truncated() bool
- func (s *Typed[S, D]) Unused() []string
- func (s *Typed[S, D]) Value() D
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrDuplicateColumns = fmt.Errorf("%w: header has duplicate columns", csvcopy.ErrParse)
ErrDuplicateColumns reports a column a csv tag asked for that the header carries more than once.
The mirror image of two fields asking for one column, which is csvcopy.ErrSchema: there the struct is ambiguous, here the file is, and neither can be resolved by picking one. Binding to the first occurrence would let the file's column order decide which values are loaded, silently.
Normalizing makes this reachable without a literally duplicated header: the default NormalizeSpace collapses "a b" and "a b" into one name. A duplicate no tag asks for is not an error - it stays in Unused, as it always did.
var ErrMissingColumns = fmt.Errorf("%w: header is missing columns", csvcopy.ErrParse)
ErrMissingColumns reports columns that a csv tag asked for and the header does not have.
This is fatal rather than a warning: a field bound to nothing would read as the empty string on every row and reach the database as NULL, quietly wiping the column it was supposed to fill. WithAllowMissingColumns lifts the check for callers who accept that risk.
var ErrRecordTooLarge = fmt.Errorf("%w: %w", csvcopy.ErrParse, errRecordTooLarge)
ErrRecordTooLarge reports a single record longer than WithMaxRecordBytes allows.
Almost always an unclosed quote: encoding/csv then reads to the end of the file looking for the closing one, and the whole file becomes a single field. The limit is what keeps memory bounded on input the caller did not write.
It wraps csvcopy.ErrParse, and there is one case where that is generous to itself: the budget covers a whole Read, and encoding/csv skips blank and comment lines inside one, so a run of them longer than the cap arrives here although no record in the file is oversized. Unreachable at the 64 MiB default and reachable if you tighten the cap - see WithMaxRecordBytes, which explains why the obvious fix would reopen the hole the budget closes. If you quarantine files on ErrParse and run a small cap, this is the one error to think about before doing so.
Functions ¶
func NormalizeSpace ¶
NormalizeSpace is the default header normalizer: it collapses every run of whitespace into a single space and trims the ends.
Exported column names arrive wrapped across lines or padded for alignment, so "date of\n birth" and "date of birth" have to resolve to the same column.
Types ¶
type Option ¶
type Option func(*settings)
Option configures a Reader and everything built on top of it.
func WithAllowMissingColumns ¶
WithAllowMissingColumns downgrades a missing tagged column from an error to silence.
Dangerous: the field will read as empty on every row and reach the database as NULL, wiping whatever that column held. Only use it where that is acceptable.
func WithComma ¶
WithComma sets the field delimiter. Defaults to ';'.
A quote, a carriage return, a newline and an invalid rune cannot delimit anything encoding/csv is willing to read, so the constructor rejects them with csvcopy.ErrSchema rather than letting the first row fail with csvcopy.ErrParse.
func WithComment ¶
WithComment sets a rune that starts a comment line. Zero, the default, means the file has no comments.
A line whose first rune is this one is skipped entirely, wherever it appears - which is what WithHeaderRow cannot do, since that only drops a fixed number of lines at the top.
Validated like the delimiter, and for the same reason: a comment rune equal to the delimiter, or one encoding/csv will not accept, is csvcopy.ErrSchema from the constructor rather than csvcopy.ErrParse on the first row. Skipped lines do not shift the line numbers in errors - those come from encoding/csv, which counts the physical file.
func WithHeaderRow ¶
WithHeaderRow sets which row holds the header, counting from 1. Rows before it are read and dropped, so a file may carry a title or a note above its table. Defaults to 1.
A header past the end of the file is the empty-file case, not an error.
func WithLazyQuotes ¶
WithLazyQuotes tolerates quoting encoding/csv would otherwise reject: a bare quote inside an unquoted field, and a quoted field that never closes. Defaults to false.
Turning it on trades a precise error for silent damage. `1;"2"3;4` becomes the field `2"3` with the right number of fields, so nothing objects. An unclosed quote is worse: the parser reads to EOF looking for the closing one and the entire rest of the file arrives as a single value. What surfaces then is a field-count error naming the line the file ended on rather than the line the quote opened on - and with WithVariableColumns there is no error at all.
Off, the same input is a parse error naming the line and the column of the quote.
func WithMaxRecordBytes ¶
WithMaxRecordBytes caps how large one record may be. Zero removes the cap. Defaults to 64 MiB.
A negative value is csvcopy.ErrSchema from the constructor, not a second way of spelling zero. It is almost always arithmetic on a config gone wrong - a byte count computed from an unset field, a subtraction, an overflow - and reading that as "remove the only bound on this package's memory" is not a safe thing to do silently. Pass 0 to mean it.
The cap is what makes "memory does not depend on the size of the file" true for input nobody checked. encoding/csv assembles a record in one buffer and has no limit of its own, so a field that opens a quote and never closes it is read to the end of the file and the whole file becomes one value. Exceeding the cap is ErrRecordTooLarge.
The cap is on the record, not on the process: peak memory while one oversized record is being read is up to about 4x it - 263 MiB measured against the 64 MiB default. encoding/csv holds the physical line in one buffer and the assembled record in another, both grown by doubling, and a doubling has the old array and the new one live at the same time. Size the cap against the memory you can afford divided by four, not against the memory you can afford.
The count itself is approximate too, in the other direction: csv.Reader buffers ahead, so bytes drawn from the input and bytes that ended up in the record differ by up to one buffer. It is a bound on memory, not a byte count to assert against.
A small cap also caps the read sizes underneath it - a Read that would overrun the remaining budget is trimmed to what is left. Irrelevant at 64 MiB; at something like 64 KiB it means the reader below sees short reads, which is worth knowing if it is a network connection.
The budget is per Read rather than per line, and encoding/csv skips blank lines and comment lines inside one - so a run of them longer than the cap is reported as ErrRecordTooLarge even though no single record is oversized. Since that error wraps csvcopy.ErrParse, a caller that quarantines files on ErrParse would quarantine a good one. Invisible at 64 MiB; worth knowing before tightening the cap on a file that carries comment blocks or long runs of blank lines. See budgetReader for why resetting the budget per line would be a worse trade than this is.
func WithNormalizeHeader ¶
WithNormalizeHeader replaces the header normalizer. Passing nil restores identity. Defaults to NormalizeSpace.
It runs on both sides of the match: on every column name read from the file, and on every csv tag value read from a struct. That is what keeps a normalizer like strings.ToLower working - lowering only the file's names would stop them matching tags written in any other case.
So it has to be pure and idempotent. It is called once per column and once per tagged field when a plan is built, never on the row path, and a function that returns different answers for the same input turns column matching into a coin toss.
func WithPointerValues ¶
WithPointerValues controls whether Raw hands out *string or string. Defaults to true, which is *string.
Putting a string into an []any boxes it, and boxing a string always allocates 16 bytes for its header, so plain strings cost one allocation per cell. A pointer is pointer-shaped: the interface holds it directly and boxing is free. The strings live in one array allocated per file, and a value the row stopped short of is a nil interface, still NULL. On five columns that is 6 allocations per row against 1; on thirty it is 31 against 1.
The default is *string because pgx cannot tell the difference, and that is a measured result rather than an argument: the integration tests load the same file both ways into an all-TEXT table and compare an md5 of the rows, and load both ways into a table of bigint, numeric and date. Both pass.
Pass false if you drive the source yourself and want plain strings - a type switch over []any is easier to write against string than *string. Nothing else in the package is affected: Typed decodes into your struct fields, and in copyfrom.Copy the boxing happens inside your own encode.
func WithTag ¶
WithTag sets the struct tag Typed reads column names from. Defaults to "csv".
The whole tag is the column name. There are no comma-separated options - a field tagged `csv:"name,omitempty"` asks for a column literally called "name,omitempty", which no header will have. The only value with a meaning of its own is "-", which drops the field.
An empty name is csvcopy.ErrSchema from the constructor: reflect.StructTag.Get("") answers "" for every field, so nothing would bind and every row would decode as empty. A struct with no field carrying this tag is csvcopy.ErrSchema too - the same failure, from the other side, and typically `json:"..."` where `csv:"..."` was meant.
func WithTrimLeadingSpace ¶
WithTrimLeadingSpace drops white space at the start of a field. Defaults to true.
func WithTrimValues ¶
WithTrimValues applies strings.TrimSpace to every value. Defaults to true.
It does not exempt quoted fields, and that is the one surprise in it. Quoting is how a CSV says "these spaces are data", so `" x "` arrives as "x" and a field of three deliberate spaces arrives as "" - which, in a column the caller treats as nullable, is a different value from what the file held.
The default follows the target case: these files come out of spreadsheet exports where padding is alignment rather than content. Turn it off where the padding is data - a fixed-width export, a column of codes - and values come through byte for byte.
func WithVariableColumns ¶
WithVariableColumns accepts rows whose field count differs from the header's. Extra values are dropped; missing trailing ones become NULL in Raw and "" in Typed.
That difference is real, not a wording slip. Raw hands pgx an []any and can put nil there, which is SQL NULL. A tagged field in Typed is declared string, so there is no nil to assign and an absent value is indistinguishable from an empty one - and in Postgres a NULL and an empty string are different values. Typed.Truncated reports which case the current row is, since convert cannot see it.
Off by default: a row that is the wrong width usually means the delimiter or the quoting is misread, and failing beats loading shifted data.
type Raw ¶
type Raw struct {
// contains filtered or unexported fields
}
Raw streams a file whose shape is only known from its own header.
Every column arrives as text, in the file's order, under the file's names - the shape of a staging table that a SQL script types afterwards. Nothing here knows or cares what the columns mean.
It satisfies pgx.CopyFromSource. That interface is not imported: Go interfaces are structural, so the method set alone is enough and the version of pgx stays the application's choice.
func NewRaw ¶
NewRaw reads the header and prepares a source over the rows after it.
Use the pointer it returns. Copying the value would give two sources sharing one Reader and one row buffer, which is not a shape anything here is written for.
func (*Raw) All ¶
All iterates the rows, for driving the source by hand rather than handing it to pgx.CopyFrom.
Stops at the first bad row; Err reports it afterwards. The yielded slice is reused by the next iteration.
If you want the values as text rather than as []any, range Reader.All instead - there is no reason to go through the boxing.
func (*Raw) Columns ¶
Columns is the column list to pass to pgx.CopyFrom, taken from the header.
Empty when the file was empty - check it before creating a table, there is nothing to load.
The names come out of the file, so they are untrusted input, and the staging-table pattern puts them on the path to CREATE TABLE. A column called
x" ); DROP TABLE clients; --
is just a text file someone wrote. Quote every name that reaches a statement with pgx.Identifier{name}.Sanitize(), or reject the header up front with copyfrom.ValidateColumns. For CopyFrom itself, pgx quotes them.
func (*Raw) Err ¶
Err returns the error that stopped the stream, if any. Check it before the error pgx.CopyFrom returns: this one names the line.
func (*Raw) Extra ¶
Extra is how many values the current record had beyond the header's columns, all of which were dropped.
This one cannot be seen any other way. A record wider than the header loses its tail silently: Values is sized by the header, the extra cells never reach it, and nothing in the data hints that they existed. And a row that is too wide almost always means the delimiter or the quoting is being misread - the signal WithVariableColumns is off by default to preserve, handed back for callers who turned it on and still want to know:
for src.Next() {
if n := src.Extra(); n > 0 {
log.Warn("dropped values", "line", src.Line(), "count", n)
}
}
Zero without WithVariableColumns, where a wide record is an error instead.
func (*Raw) Next ¶
Next advances to the next row, stopping at the end of the file or at the first malformed record.
func (*Raw) Truncated ¶
Truncated reports whether the record behind the current row ran out before the header's last column, so the trailing values are NULL because they were absent rather than because the file left them blank.
Only meaningful after Next returned true, and only ever true under WithVariableColumns - without it a short record is an error instead.
Unlike Typed.Truncated, this one is not the only way to see it: a NULL in the data says the same thing. It is here so a caller does not have to go looking through []any to find out, and so the two sources answer the same question the same way.
func (*Raw) Values ¶
Values returns the current row.
The slice is reused between rows, which is safe for pgx.CopyFrom because it encodes each row before asking for the next. A caller driving the source by hand must not hold on to it.
Calling it before the first Next is csvcopy.ErrSchema: there is no row yet, and the slice at that point is one empty value per column - a row that would load without complaint. pgx.CopyFrom always calls Next first; a caller driving the source by hand is the one that can get here.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader is the bottom layer both sources are built on: it strips a UTF-8 BOM, configures encoding/csv, and reads the header.
It reads one record at a time and never holds more than the current one, which is what keeps the whole pipeline to a constant amount of memory no matter how large the file is. It does not close the underlying io.Reader.
func NewReader ¶
NewReader reads the header and prepares the reader for row-by-row use.
An empty input is not an error: Columns returns nil and Read returns io.EOF at once. What an empty file means is the caller's decision, not this package's.
Every option is validated here, including the ones only a Typed will read: a Reader has no use for WithTag, but one settings type serves all three layers, and a caller who mistyped an option should learn it from the constructor they called rather than from the one they call next week.
func (*Reader) All ¶
All iterates the records left in the file.
Ranging cannot carry an error out, so the loop stops at the first bad record and Err reports it afterwards - the same shape as bufio.Scanner:
for record := range reader.All() {
...
}
if err := reader.Err(); err != nil {
return err
}
Breaking out of the loop leaves the reader usable, and the next pull picks up where the range left off. An error does not: the reader is done, and ranging it again yields nothing rather than resuming past the record that failed.
The yielded slice is reused by the next iteration. Copy it if it has to outlive one turn of the loop.
func (*Reader) Columns ¶
Columns returns the header as it was read and normalized, or nil if the input was empty. The slice is shared, not copied - it is handed straight to pgx.CopyFrom, which does not modify it.
These names come out of the file and are untrusted input. Normalizing collapses whitespace; it does not make a name safe to paste into a statement. Before one reaches DDL, quote it - pgx.Identifier{name}.Sanitize() - or put the header through copyfrom.ValidateColumns. For CopyFrom, pgx quotes them itself.
func (*Reader) Line ¶
Line is the 1-based line of the file the current record starts on.
It is the physical line, taken from encoding/csv rather than counted here, so it stays right across the two things that make a record count drift from it: blank lines, which encoding/csv skips, and a quoted field spanning several lines. That number is the one an error message must carry - the caller opens the file at it.
After a failed Read it names the record that failed, when encoding/csv could say which one that was - that is, for csvcopy.ErrParse. It cannot for csvcopy.ErrIO or ErrRecordTooLarge, since neither arrives as a *csv.ParseError, and there this stays on the last record read in full: the record that failed starts somewhere after it. The error message says so in as many words, "after line N" rather than "line N", so the two cases are told apart in the text and not only here.
func (*Reader) Read ¶
Read returns the next record, or io.EOF when there are none left.
The first bad record ends the reader: the error is returned again by every later call and stays in Err. Rows are never skipped, so a caller cannot resume past a bad record and mistake a truncated file for a whole one.
The returned slice is reused by the next call. Every error other than io.EOF says where it happened - see Line - and wraps one of three sentinels: ErrRecordTooLarge if the record outgrew WithMaxRecordBytes, csvcopy.ErrParse if the content is malformed, or csvcopy.ErrIO if the stream underneath failed.
func (*Reader) Record ¶
Record returns the record last read, for error messages that need the offending row rather than just its number.
After a failed Read it holds that record as far as encoding/csv got with it, or nil where it could not produce one at all.
Only valid until the next Read: the backing slice is reused.
type Typed ¶
Typed streams a file whose shape is fixed by a struct.
S is the row as it appears in the file - every field a string, tagged with the column name it comes from. D is what the rest of the program works with, produced by convert. The split is deliberate: only the caller can decide whether an empty cell is a NULL or a zero, and whether a bad value fails the file or the field, so this package never converts anything itself.
Typed satisfies copyfrom.RowSource[D], so decoding and the layout of a COPY row can live in different layers: one hands out a RowSource, and the layer that owns the database decides which columns it lands in. This package never learns which.
func NewTyped ¶
func NewTyped[S any, D any](r io.Reader, convert func(*S) (D, error), opts ...Option) (*Typed[S, D], error)
NewTyped reads the header, binds it to S's tags and prepares a source over the rows after it.
convert is called once per row with a pointer to a struct that is reused for the whole file, so it must not keep that pointer. Returning an error from it stops the stream, and the error is reported with the line it came from.
That error is wrapped in csvcopy.ErrParse, since the usual reason a row fails to convert is the row. Where that is wrong - convert reached a lookup table that was down, and the file is fine - return an error wrapping csvcopy.ErrIO and it stays csvcopy.ErrIO, so a caller that quarantines files on csvcopy.ErrParse does not quarantine this one. csvcopy.ErrSchema is kept the same way. A cancelled context needs no wrapping: an error carrying context.Canceled or context.DeadlineExceeded is reported as csvcopy.ErrIO.
Only the fields a column bound to are written before each call. A field no tag asked for - untagged, or tagged "-" - is never touched, so whatever convert leaves in one it will see again on the next row.
Use the pointer this returns; the value behind it must not be copied. The decode plan holds the addresses of that struct's own fields, so a copy would decode into the original while convert read the copy - every field empty, on every row, with nothing reporting it. go vet refuses the copy, which is why noCopy is embedded.
Example ¶
Columns are matched by name, so the file may hold them in any order. Handing the result to pgx is copyfrom's business, not this package's - see ExampleNewCopy.
const file = "name;id\nAlice;1\nBob;2\n"
rows, err := NewTyped(strings.NewReader(file), toEntity)
if err != nil {
panic(err)
}
for e := range rows.All() {
fmt.Println(e.ID, e.Name)
}
if err = rows.Err(); err != nil {
panic(err)
}
Output: 1 Alice 2 Bob
func (*Typed[S, D]) All ¶
All iterates the decoded rows.
This is the layer that is useful with no database in sight: CSV in, your own type out, one row at a time. Stops at the first row that fails to parse or convert, and Err reports it afterwards:
for client := range rows.All() {
...
}
if err := rows.Err(); err != nil {
return err
}
Unlike the other All methods, what is yielded here is whatever convert returned, so nothing is reused behind your back.
func (*Typed[S, D]) Extra ¶
Extra is how many values the current record had beyond the header's columns, all of which were dropped.
The counterpart to Truncated, for the other end of the record, and the one that cannot be seen any other way here. A tag can only ask for a column the header names, so values past the last one bind to nothing and never reach the struct - Record shows the raw record, but nothing in the decoded row says it was longer than the file said it would be.
That matters because a record wider than its header almost always means the delimiter or the quoting is being misread, which is the signal WithVariableColumns turns off. Read it in the loop if you want it back:
for source.Next() {
if n := source.Extra(); n > 0 {
log.Warn("dropped values", "line", source.Line(), "count", n)
}
}
Only meaningful after Next returned true, and zero without WithVariableColumns, where a wide record is an error instead.
func (*Typed[S, D]) Header ¶
Header returns the header as read and normalized, or nil if the file was empty.
func (*Typed[S, D]) Next ¶
Next advances to the next row, stopping at the end of the file, at the first malformed record, or at the first row convert rejects.
func (*Typed[S, D]) Record ¶
Record returns the raw record behind the current row, for error messages.
func (*Typed[S, D]) Rows ¶
Rows is the number of rows handed out so far. Zero after a full pass means the file held nothing, which is not in itself an error.
func (*Typed[S, D]) Truncated ¶
Truncated reports whether the record behind the current row ran out before a bound column, so that field holds "" because the value was absent rather than empty.
Only meaningful after Next returned true, and only ever true under WithVariableColumns - without it a short record is an error instead.
It exists because the distinction is invisible where it matters most. Raw can put nil in an []any and get SQL NULL; a tagged field is declared string, so an absent value and an empty one both arrive as "", and in Postgres a NULL and an empty string are different values. Read it after Next if that difference matters:
for source.Next() {
if source.Truncated() {
log.Warn("short record", "line", source.Line())
}
}
convert cannot see this. It is called inside Next, with the struct as its only argument, and adding a second one would change the signature every caller writes. If a row needs to be rejected for being short, reject it here.
func (*Typed[S, D]) Unused ¶
Unused returns the header columns no tag bound to.
Worth logging as a warning: when an export renames a column, the tag stops matching it, no error is raised, and the column that was renamed away is the only trace left.
The slice is shared, not copied. It is built once per file and never written again, so reading it is safe; do not modify it.
Source Files