csvdecoder

README

csvdecoder

csvdecoder is a Go library for parsing and decoding csv files into Go objects. It uses encoding/csv for the parsing and follows a usage pattern inspired by the database/sql package.

The scanning method is not multi-thread safe; Next and Scan are not expected to be called concurrently.

Installation

go get github.com/stefantds/csvdecoder

Supported formats

Csvdecoder supports converting columns read from the source file into the following types:

• *string
• *int, *int8, *int16, *int32, *int64
• *uint, *uint8, *uint16, *uint32, *uint64
• *bool
• *float32, *float64
• a slice of values. Note that the CSV record must be a valid JSON array. If not a JSON array, a custom decoder implementing the csvdecoder.Interface interface must be implemented.
• an array of values. Note that the CSV record must be a valid JSON array. If not a JSON array, a custom decoder implementing the csvdecoder.Interface interface must be implemented.
• a pointer to any type implementing the csvdecoder.Interface interface

Usage

import (
	"fmt"
	"os"

	"github.com/stefantds/csvdecoder"
)

type User struct {
	Name   string
	Active bool
	Age    int
}

func Example_simple() {
	// the csv file contains the values:
	//john,44,true
	//lucy,48,false
	//mr hyde,34,true
	file, err := os.Open("./data/simple.csv")
	if err != nil {
		// handle error
		return
	}
	defer file.Close()

	// create a new decoder that will read from the given file
	decoder, err := csvdecoder.New(file)
	if err != nil {
		// handle error
		return
	}

	// iterate over the rows in the file
	for decoder.Next() {
		var u User

		// scan the first three values in the name, age and active fields respectively
		if err := decoder.Scan(&u.Name, &u.Age, &u.Active); err != nil {
			// handle error
			return
		}
		fmt.Println(u)
	}

	// check if the loop stopped prematurely because of an error
	if err = decoder.Err(); err != nil {
		// handle error
		return
	}

	// Output: {john true 44}
	// {lucy false 48}
	// {mr hyde true 34}
}

See also the example files for more usage examples.

Configuration

The behaviour of the decoder can be configured by passing one of following options when creating the decoder:

• Comma: the character that separates values. Default value is comma.
• IgnoreHeaders: if set to true, the first line will be ignored
• IgnoreUnmatchingFields: if set to true, the number of records and scan targets are allowed to be different. By default, if they don't match exactly it will cause an error.

	decoder, err := csvdecoder.NewWithConfig(file, csvdecoder.Config{Comma: ';', IgnoreHeaders: true})

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as needed.

License

MIT

Documentation

Overview

Example (Custom_decoder)

package main

import (
	"encoding/json"
	"fmt"
	"strings"

	"github.com/stefantds/csvdecoder"
)

type Point struct {
	X int
	Y int
}

// DecodeRecord implements the csvdecoder.Interface type
func (p *Point) DecodeRecord(record string) error {
	// the decode code is specific to the way the object is serialized.
	// in this example the point is encoded as a JSON array with two values

	data := make([]int, 2)
	if err := json.NewDecoder(strings.NewReader(record)).Decode(&data); err != nil {
		return fmt.Errorf("could not parse %s as JSON array: %w", record, err)
	}

	(*p).X = data[0]
	(*p).Y = data[1]

	return nil
}

func main() {
	// the csv separator is a semicolon in this example
	exampleData := strings.NewReader(
		`[0, 0];[0, 2];[1, 2]
[-1, 2];[0, -2];[1, 0]
`)

	// create a new decoder that will read from the given file
	decoder, err := csvdecoder.NewWithConfig(exampleData, csvdecoder.Config{Comma: ';'})
	if err != nil {
		// handle error
		return
	}

	// iterate over the rows in the file
	for decoder.Next() {
		var a, b, c Point

		// scan the first values to the types
		if err := decoder.Scan(&a, &b, &c); err != nil {
			// handle error
			return
		}
		fmt.Printf("a: %v, b: %v, c: %v\n", a, b, c)
	}

	// check if the loop stopped prematurely because of an error
	if err = decoder.Err(); err != nil {
		// handle error
		return
	}

}

Output:

a: {0 0}, b: {0 2}, c: {1 2}
a: {-1 2}, b: {0 -2}, c: {1 0}

Example (Simple)

package main

import (
	"fmt"
	"os"

	"github.com/stefantds/csvdecoder"
)

type User struct {
	Name   string
	Active bool
	Age    int
}

func main() {

	// the csv file contains the values:
	//john,44,true
	//lucy,48,false
	//mr hyde,34,true
	file, err := os.Open("./data/simple.csv")
	if err != nil {
		// handle error
		return
	}
	defer file.Close()

	// create a new decoder that will read from the given file
	decoder, err := csvdecoder.New(file)
	if err != nil {
		// handle error
		return
	}

	// iterate over the rows in the file
	for decoder.Next() {
		var u User

		// scan the first three values in the name, age and active fields respectively
		if err := decoder.Scan(&u.Name, &u.Age, &u.Active); err != nil {
			// handle error
			return
		}
		fmt.Println(u)
	}

	// check if the loop stopped prematurely because of an error
	if err = decoder.Err(); err != nil {
		// handle error
		return
	}

}

Output:

{john true 44}
{lucy false 48}
{mr hyde true 34}

Example (Slices)

package main

import (
	"fmt"
	"strings"

	"github.com/stefantds/csvdecoder"
)

type MyStringCollection []string

// DecodeRecord implements the csvdecoder.Interface type
func (c *MyStringCollection) DecodeRecord(record string) error {
	// the decode code is specific to the way the value is serialized.
	// in this example the array is represented as int values separated by space
	*c = MyStringCollection(strings.Split(record, " "))

	return nil
}

func main() {
	// the csv separator is a semicolon in this example
	// the values are arrays serialized in two different ways.
	exampleData := strings.NewReader(
		`jon;elvis boris ahmed jane;["jo", "j"]
jane;lucas george;["j", "jay"]
`)

	// create a new decoder that will read from the given file
	decoder, err := csvdecoder.NewWithConfig(exampleData, csvdecoder.Config{Comma: ';'})
	if err != nil {
		// handle error
		return
	}

	type Person struct {
		Name      string
		Friends   MyStringCollection
		Nicknames []string
	}

	// iterate over the rows in the file
	for decoder.Next() {
		var p Person

		// scan the first values to the types
		if err := decoder.Scan(&p.Name, &p.Friends, &p.Nicknames); err != nil {
			// handle error
			return
		}
		fmt.Printf("%v\n", p)
	}

	// check if the loop stopped prematurely because of an error
	if err = decoder.Err(); err != nil {
		// handle error
		return
	}

}

Output:

{jon [elvis boris ahmed jane] [jo j]}
{jane [lucas george] [j jay]}

Index

• Variables
• type Config
• type Decoder
  • func New(r io.Reader) (*Decoder, error)
  • func NewWithConfig(r io.Reader, config Config) (*Decoder, error)
  • func (p *Decoder) Err() error
  • func (p *Decoder) Next() bool
  • func (p *Decoder) Scan(dest ...interface{}) error
• type Interface

Examples

• Package (Custom_decoder)
• Package (Simple)
• Package (Slices)

Variables

var (
	ErrEOF                 = errors.New("end of file reached") // ErrEOF is thrown if the EOF is reached by the Next method.
	ErrScanTargetsNotMatch = errors.New("the number of scan targets does not match the number of csv records")
	ErrReadingOccurred     = errors.New("can't continue after a reading error")
	ErrNextNotCalled       = errors.New("scan called without calling Next")
)

Types

type Config

type Config struct {
	Comma                  rune // the character that separates values. Default value is comma.
	IgnoreHeaders          bool // if set to true, the first line will be ignored
	IgnoreUnmatchingFields bool // if set to true, the number of records and scan targets are allowed to be different
}

Config is a type that can be used to configure a decoder.

type Decoder

type Decoder struct {
	// contains filtered or unexported fields
}

func New

func New(r io.Reader) (*Decoder, error)

New returns a new CSV decoder that reads from r

func NewWithConfig

func NewWithConfig(r io.Reader, config Config) (*Decoder, error)

New returns a new CSV decoder that reads from r. The decoder can be given a custom configuration.

func (*Decoder) Err

func (p *Decoder) Err() error

Err returns the reading error, if any, that was encountered during iteration.

func (*Decoder) Next

func (p *Decoder) Next() bool

Next prepares the next result row for reading with the Scan method. It returns nil on success, or false if there is no next result row or an error happened while preparing it. Err should be consulted to distinguish between the two cases.

Every call to Scan, even the first one, must be preceded by a call to Next. Next must not be called concurrently.

func (*Decoder) Scan

func (p *Decoder) Scan(dest ...interface{}) error

Scan copies the values in the current row into the values pointed at by dest. With the default behavior, it will throw an error if the number of values in dest is different from the number of values. If the `IgnoreUnmatchingFields` flag is set, it will ignore the records and the arguments that have no match.

Scan converts columns read from the source into the following types:

*string
*int, *int8, *int16, *int32, *int64
*uint, *uint8, *uint16, *uint32, *uint64
*bool
*float32, *float64
a pointer to any type implementing Decoder interface
a slice of values that can be decoded from a JSON array by the JSON Decoder
an array of values that can be decoded from a JSON array by the JSON Decoder

Scan must not be called concurrently.

type Interface

type Interface interface {
	DecodeRecord(s string) error
}

The Interface type describes the requirements for a type that can be decoded into a Go value by the csvdecoder. Any type that implements it may be used as a target in the Scan method.

The Decode method allows to implement a custom decoding logic. If it returns an error, the parsing and decoding is stopped and the error is returned to the caller of Scan.