csvdecoder

README

csvdecoder

csvdecoder is a Go library for parsing and deserializing csv files into Go objects. It relies on encoding/csv for the actual parsing of the CSV file and follows a similar usage pattern as the database/sql package for scanning rows.

csvdecoder allows to iterate through the CSV records (using 'Next') and scan the fields into target variables or fields of variables (using 'Scan').

The scanning method is not thread safe; Next and Scan are not expected to be called concurrently.

Installation

go get github.com/stefantds/csvdecoder

Supported formats

Csvdecoder supports converting columns read from the source file into the following types:

• *string
• *int, *int8, *int16, *int32, *int64
• *uint, *uint8, *uint16, *uint32, *uint64
• *bool
• *float32, *float64
• a slice of values. Note that the CSV field must be a valid JSON array. If not a JSON array, a custom decoder implementing the csvdecoder.Interface interface must be implemented.
• an array of values. Note that the CSV field must be a valid JSON array. If not a JSON array, a custom decoder implementing the csvdecoder.Interface interface must be implemented.
• a pointer to any type implementing the csvdecoder.Interface interface

Usage

import (
	"fmt"
	"os"

	"github.com/stefantds/csvdecoder"
)

type User struct {
	Name   string
	Active bool
	Age    int
}

func Example_simple() {
	// the csv file contains the values:
	//john,44,true
	//lucy,48,false
	//mr hyde,34,true
	file, err := os.Open("./data/simple.csv")
	if err != nil {
		// handle error
		return
	}
	defer file.Close()

	// create a new decoder that will read from the given file
	decoder, err := csvdecoder.New(file)
	if err != nil {
		// handle error
		return
	}

	// iterate over the rows in the file
	for decoder.Next() {
		var u User

		// scan the first three values in the name, age and active fields respectively
		if err := decoder.Scan(&u.Name, &u.Age, &u.Active); err != nil {
			// handle error
			return
		}
		fmt.Println(u)
	}

	// check if the loop stopped prematurely because of an error
	if err = decoder.Err(); err != nil {
		// handle error
		return
	}

	// Output: {john true 44}
	// {lucy false 48}
	// {mr hyde true 34}
}

See also the example files for more usage examples.

Configuration

The behavior of the decoder can be configured by passing one of following options when creating the decoder:

• Comma: the character that separates values. Default value is comma.
• IgnoreHeaders: if set to true, the first line will be ignored. This is useful when the CSV file contains a header line.
• IgnoreUnmatchingFields: if set to true, the number of fields and scan targets are allowed to be different. By default, if they don't match exactly it will cause an error.

	decoder, err := csvdecoder.NewWithConfig(file, csvdecoder.Config{Comma: ';', IgnoreHeaders: true})

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as needed.

License

MIT

Documentation

Overview

csvdecoder is a tool for parsing and deserializing CSV values into Go objects. It follows the same usage pattern as the Rows scanning using database/sql package. It relies on encoding/csv for the actual csv parsing.

csvdecoder allows to iterate through the CSV records (using 'Next') and scan the fields into target variables or fields of variables (using 'Scan'). The methods 'Next' and 'Scan' are not thread-safe and are not expected to be called concurrently.

csvdecoder supports converting CSV fields into any of the following types:

*string
*int, *int8, *int16, *int32, *int64
*uint, *uint8, *uint16, *uint32, *uint64
*bool
*float32, *float64
a slice of values. Note that the CSV field must be a valid JSON array. If not a JSON array, a custom decoder implementing the csvdecoder.Interface interface must be implemented.
an array of values. Note that the CSV field must be a valid JSON array. If not a JSON array, a custom decoder implementing the csvdecoder.Interface interface must be implemented.
a pointer to any type implementing the csvdecoder.Interface interface

csvdecoder uses the same terminology as package encoding/csv: A csv file contains zero or more records. Each record contains one or more fields separated by the fields separator (the "comma"). The fields separator character can be configured to be another character than comma. Each record is separated by the newline character. The final record may optionally be followed by a newline character.

The behavior of the decoder can be configured by passing one of following options when creating the decoder:

Comma: the character that separates values. The default value is comma.
IgnoreHeaders: if set to true, the first line will be ignored. This is useful when the CSV file contains a header line.
IgnoreUnmatchingFields: if set to true, the number of fields and scan targets are allowed to be different. By default, if they don't match exactly it will cause an error.

See README.md for more info.

Example (Custom_decoder)

package main

import (
	"encoding/json"
	"fmt"
	"strings"

	"github.com/stefantds/csvdecoder"
)

type Point struct {
	X int
	Y int
}

// DecodeField implements the csvdecoder.Interface type
func (p *Point) DecodeField(field string) error {
	// the decode code is specific to the way the object is serialized.
	// in this example the point is encoded as a JSON array with two values

	data := make([]int, 2)
	if err := json.NewDecoder(strings.NewReader(field)).Decode(&data); err != nil {
		return fmt.Errorf("could not parse %s as JSON array: %w", field, err)
	}

	(*p).X = data[0]
	(*p).Y = data[1]

	return nil
}

func main() {
	// the csv separator is a semicolon in this example
	exampleData := strings.NewReader(
		`[0, 0];[0, 2];[1, 2]
[-1, 2];[0, -2];[1, 0]
`)

	// create a new decoder that will read from the given file
	decoder, err := csvdecoder.NewWithConfig(exampleData, csvdecoder.Config{Comma: ';'})
	if err != nil {
		// handle error
		return
	}

	// iterate over the rows in the file
	for decoder.Next() {
		var a, b, c Point

		// scan the first values to the types
		if err := decoder.Scan(&a, &b, &c); err != nil {
			// handle error
			return
		}
		fmt.Printf("a: %v, b: %v, c: %v\n", a, b, c)
	}

	// check if the loop stopped prematurely because of an error
	if err = decoder.Err(); err != nil {
		// handle error
		return
	}

}

Output:

a: {0 0}, b: {0 2}, c: {1 2}
a: {-1 2}, b: {0 -2}, c: {1 0}

Example (Simple)

package main

import (
	"fmt"
	"os"

	"github.com/stefantds/csvdecoder"
)

type User struct {
	Name   string
	Active bool
	Age    int
}

func main() {

	// the csv file contains the values:
	//john,44,true
	//lucy,48,false
	//mr hyde,34,true
	file, err := os.Open("./example_data/simple.csv")
	if err != nil {
		// handle error
		return
	}
	defer file.Close()

	// create a new decoder that will read from the given file
	decoder, err := csvdecoder.New(file)
	if err != nil {
		// handle error
		return
	}

	// iterate over the rows in the file
	for decoder.Next() {
		var u User

		// scan the first three values in the name, age and active fields respectively
		if err := decoder.Scan(&u.Name, &u.Age, &u.Active); err != nil {
			// handle error
			return
		}
		fmt.Println(u)
	}

	// check if the loop stopped prematurely because of an error
	if err = decoder.Err(); err != nil {
		// handle error
		return
	}

}

Output:

{john true 44}
{lucy false 48}
{mr hyde true 34}

Example (Slices)

package main

import (
	"fmt"
	"strings"

	"github.com/stefantds/csvdecoder"
)

type MyStringCollection []string

// DecodeField implements the csvdecoder.Interface type
func (c *MyStringCollection) DecodeField(field string) error {
	// the decode code is specific to the way the value is serialized.
	// in this example the array is represented as int values separated by space
	*c = MyStringCollection(strings.Split(field, " "))

	return nil
}

func main() {
	// the csv separator is a semicolon in this example
	// the values are arrays serialized in two different ways.
	exampleData := strings.NewReader(
		`jon;elvis boris ahmed jane;["jo", "j"]
jane;lucas george;["j", "jay"]
`)

	// create a new decoder that will read from the given file
	decoder, err := csvdecoder.NewWithConfig(exampleData, csvdecoder.Config{Comma: ';'})
	if err != nil {
		// handle error
		return
	}

	type Person struct {
		Name      string
		Friends   MyStringCollection
		Nicknames []string
	}

	// iterate over the rows in the file
	for decoder.Next() {
		var p Person

		// scan the first values to the types
		if err := decoder.Scan(&p.Name, &p.Friends, &p.Nicknames); err != nil {
			// handle error
			return
		}
		fmt.Printf("%v\n", p)
	}

	// check if the loop stopped prematurely because of an error
	if err = decoder.Err(); err != nil {
		// handle error
		return
	}

}

Output:

{jon [elvis boris ahmed jane] [jo j]}
{jane [lucas george] [j jay]}

Index

• Variables
• type Config
• type Decoder
  • func New(r io.Reader) (*Decoder, error)
  • func NewWithConfig(r io.Reader, config Config) (*Decoder, error)
  • func (p *Decoder) Err() error
  • func (p *Decoder) Next() bool
  • func (p *Decoder) Scan(dest ...interface{}) error
• type Interface

Examples

• Package (Custom_decoder)
• Package (Simple)
• Package (Slices)

Variables

var (
	ErrEOF                 = errors.New("end of file reached") // ErrEOF is thrown if the EOF is reached by the Next method.
	ErrScanTargetsNotMatch = errors.New("the number of scan targets does not match the number of csv fields")
	ErrReadingOccurred     = errors.New("can't continue after a reading error")
	ErrNextNotCalled       = errors.New("scan called without calling Next")
)

Types

type Config

type Config struct {
	Comma                  rune // the character that separates values. Default value is comma.
	IgnoreHeaders          bool // if set to true, the first line will be ignored
	IgnoreUnmatchingFields bool // if set to true, the number of fields and scan targets are allowed to be different
}

Config is a type that can be used to configure a decoder.

type Decoder

type Decoder struct {
	// contains filtered or unexported fields
}

func New

func New(r io.Reader) (*Decoder, error)

New returns a new CSV decoder that reads from r

func NewWithConfig

func NewWithConfig(r io.Reader, config Config) (*Decoder, error)

New returns a new CSV decoder that reads from r. The decoder can be given a custom configuration.

func (*Decoder) Err

func (p *Decoder) Err() error

Err returns the reading error, if any, that was encountered during iteration.

func (*Decoder) Next

func (p *Decoder) Next() bool

Next prepares the next result row for reading with the Scan method. It returns nil on success, or false if there is no next result row or an error happened while preparing it. Err should be consulted to distinguish between the two cases.

Every call to Scan, even the first one, must be preceded by a call to Next. Next must not be called concurrently.

func (*Decoder) Scan

func (p *Decoder) Scan(dest ...interface{}) error

Scan copies the values in the current row into the values pointed at by dest. With the default behavior, it will throw an error if the number of values in dest is different from the number of values. If the `IgnoreUnmatchingFields` flag is set, it will ignore the fields and the arguments that have no match.

Scan converts columns read from the source into the following types:

*string
*int, *int8, *int16, *int32, *int64
*uint, *uint8, *uint16, *uint32, *uint64
*bool
*float32, *float64
a pointer to any type implementing Decoder interface
a slice of values that can be decoded from a JSON array by the JSON Decoder
an array of values that can be decoded from a JSON array by the JSON Decoder

Scan must not be called concurrently.

type Interface

type Interface interface {
	DecodeField(s string) error
}

The Interface type describes the requirements for a type that can be decoded into a Go value by the csvdecoder. Any type that implements it may be used as a target in the Scan method.

The Decode method allows to implement a custom decoding logic. If it returns an error, the parsing and decoding is stopped and the error is returned to the caller of Scan.