README
¶
CSV Importer
Imports a CSV file as List<T> where T is a struct with fields corresponding to the CSV's column headers. The struct spec can also be set manually with the -header flag.
Usage
$ cd csv-import
$ go build
$ ./csv-import <PATH> http://localhost:8000::foo
Some places for CSV files
- https://data.cityofnewyork.us/api/views/kku6-nxdu/rows.csv?accessType=DOWNLOAD
- http://www.opendatacache.com/
CSV Exporter
Export a dataset in CSV format to stdout with column headers.
Usage
$ cd csv-export
$ go build
$ ./csv-export http://localhost:8000:foo
Documentation
¶
Index ¶
- Variables
- func EscapeStructFieldFromCSV(input string) string
- func FindPrimaryKeys(r *csv.Reader, numSamples, maxLenPrimaryKeyList, numFields int) [][]int
- func GetFieldNamesFromIndices(headers []string, indices []int) []string
- func GetListElemDesc(l types.List, vr types.ValueReader) types.StructDesc
- func GetMapElemDesc(m types.Map, vr types.ValueReader) types.StructDesc
- func KindsToStrings(kinds KindSlice) []string
- func MakeStructTemplateFromHeaders(headers []string, structName string, kinds KindSlice) (temp types.StructTemplate, fieldOrder []int, kindMap []types.NomsKind)
- func NewCSVReader(res io.Reader, comma rune) *csv.Reader
- func ReadToColumnar(r *csv.Reader, structName string, headers []string, kinds KindSlice, ...) (s types.Struct)
- func ReadToList(r *csv.Reader, structName string, headers []string, kinds KindSlice, ...) (l types.List)
- func ReadToMap(r *csv.Reader, structName string, headersRaw []string, primaryKeys []string, ...) types.Map
- func SkipRecords(r *csv.Reader, n uint) error
- func StringToRune(delimiter string) (rune, error)
- func StringToValue(s string, k types.NomsKind) (types.Value, error)
- func WriteList(l types.List, sd types.StructDesc, comma rune, output io.Writer)
- func WriteMap(m types.Map, sd types.StructDesc, comma rune, output io.Writer)
- type KindSlice
Constants ¶
This section is empty.
Variables ¶
var StringToKind = func(kindMap map[types.NomsKind]string) map[string]types.NomsKind { m := map[string]types.NomsKind{} for k, v := range kindMap { m[v] = k } return m }(types.KindToString)
StringToKind maps names of valid NomsKinds (e.g. Bool, Number, etc) to their associated types.NomsKind
Functions ¶
func EscapeStructFieldFromCSV ¶
EscapeStructFieldFromCSV removes special characters and replaces spaces with camelCasing (camel case turns to camelCase)
func FindPrimaryKeys ¶
FindPrimaryKeys reads numSamples from r, using the first numFields and returns slices of []int indices that are primary keys for those samples
func GetListElemDesc ¶
func GetListElemDesc(l types.List, vr types.ValueReader) types.StructDesc
GetListElemDesc ensures that l is a types.List of structs, pulls the types.StructDesc that describes the elements of l out of vr, and returns the StructDesc.
func GetMapElemDesc ¶
func GetMapElemDesc(m types.Map, vr types.ValueReader) types.StructDesc
GetMapElemDesc ensures that m is a types.Map of structs, pulls the types.StructDesc that describes the elements of m out of vr, and returns the StructDesc. If m is a nested types.Map of types.Map, then GetMapElemDesc will descend the levels of the enclosed types.Maps to get to a types.Struct
func KindsToStrings ¶
KindsToStrings looks up each element of kinds in the types.KindToString map and returns a slice of answers
func MakeStructTemplateFromHeaders ¶
func MakeStructTemplateFromHeaders(headers []string, structName string, kinds KindSlice) (temp types.StructTemplate, fieldOrder []int, kindMap []types.NomsKind)
MakeStructTemplateFromHeaders creates a struct type from the headers using |kinds| as the type of each field. If |kinds| is empty, default to strings.
func NewCSVReader ¶
NewCSVReader returns a new csv.Reader that splits on comma
func ReadToColumnar ¶
func ReadToColumnar(r *csv.Reader, structName string, headers []string, kinds KindSlice, vrw types.ValueReadWriter, limit uint64) (s types.Struct)
ReadToColumnar takes a CSV reader and reads data from each column into a separate list. Values from columns in each successive row are appended to the column-specific lists whose type is described by headers. Finally, a new "Columnar" struct is created that consists of one field for each column and each field contains a list of values. If the original data contained headers it is expected that the input reader has already read those and are pointing at the first data row. If kinds is non-empty, it will be used to type the fields in the generated structs; otherwise, they will be left as string-fields. In addition to the list, ReadToList returns the typeDef of the structs in the list.
func ReadToList ¶
func ReadToList(r *csv.Reader, structName string, headers []string, kinds KindSlice, vrw types.ValueReadWriter, limit uint64) (l types.List)
ReadToList takes a CSV reader and reads data into a typed List of structs. Each row gets read into a struct named structName, described by headers. If the original data contained headers it is expected that the input reader has already read those and are pointing at the first data row. If kinds is non-empty, it will be used to type the fields in the generated structs; otherwise, they will be left as string-fields. In addition to the list, ReadToList returns the typeDef of the structs in the list.
func ReadToMap ¶
func ReadToMap(r *csv.Reader, structName string, headersRaw []string, primaryKeys []string, kinds KindSlice, vrw types.ValueReadWriter, limit uint64) types.Map
ReadToMap takes a CSV reader and reads data into a typed Map of structs. Each row gets read into a struct named structName, described by headers. If the original data contained headers it is expected that the input reader has already read those and are pointing at the first data row. If kinds is non-empty, it will be used to type the fields in the generated structs; otherwise, they will be left as string-fields.
func StringToRune ¶
StringToRune returns the rune contained in delimiter or an error.
func StringToValue ¶
StringToValue takes a piece of data as a string and attempts to convert it to a types.Value of the appropriate types.NomsKind.
Types ¶
type KindSlice ¶
KindSlice is an alias for []types.NomsKind. It's needed because types.NomsKind are really just 8 bit unsigned ints, which are what Go uses to represent 'byte', and this confuses the Go JSON marshal/unmarshal code -- it treats them as byte arrays and base64 encodes them!
func StringsToKinds ¶
StringsToKinds looks up each element of strs in the StringToKind map and returns a slice of answers
Source Files
Directories