csvio

package
v0.0.0-...-3c68a24 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 16, 2018 License: Apache-2.0 Imports: 9 Imported by: 0

Documentation

Overview

Package csvio contains Apache Beam transforms for reading CSV files with https://github.com/gocarina/gocsv.

TODO: * Create a CSV Writer

Example
// Example of using the csvio package.
package main

import (
	"context"
	"reflect"

	"bramp.net/morebeam/csvio"

	"github.com/apache/beam/sdks/go/pkg/beam"
	"github.com/apache/beam/sdks/go/pkg/beam/log"
	"github.com/apache/beam/sdks/go/pkg/beam/transforms/stats"
	"github.com/apache/beam/sdks/go/pkg/beam/x/beamx"
	"github.com/apache/beam/sdks/go/pkg/beam/x/debug"
)

// Painting represents a single record in the csv file.
type Painting struct {
	Artist  string `csv:"artist"`
	Title   string `csv:"title"`
	Year    int    `csv:"year"`
	NotUsed string `csv:"-"` // Ignored field
}

func extractFn(painting Painting) string {
	return painting.Artist
}

func main() {
	beam.Init()

	p, s := beam.NewPipelineWithRoot()

	// Read the CSV file.
	paintings := csvio.Read(s, "testdata/paintings.csv", reflect.TypeOf(Painting{}))

	// Extract just the artist's name.
	artists := beam.ParDo(s, extractFn, paintings)

	// Count the number of paintings by each artist.
	counts := stats.Count(s, artists)
	debug.Print(s, counts)

	ctx := context.Background()
	if err := beamx.Run(ctx, p); err != nil {
		log.Fatalf(ctx, "Failed to execute job: %v", err)
	}

}
Output:

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

func Read

func Read(s beam.Scope, glob string, t reflect.Type) beam.PCollection

Read reads a set of CSV files and returns the lines as a PCollection<T>. T is defined by the reflect.TypeOf( YourType{} ) with csv tags as descripted by https://github.com/gocarina/gocsv

Types

This section is empty.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL