parquetio

Documentation

Overview

Package parquetio contains transforms for reading and writing parquet files

Index

• func Read(s beam.Scope, glob string, t reflect.Type) beam.PCollection
• func Write(s beam.Scope, filename string, col beam.PCollection)

Functions

func Read

func Read(s beam.Scope, glob string, t reflect.Type) beam.PCollection

Read reads a set of files and returns lines as a PCollection<elem> based on type of a parquetStruct (struct with parquet tags). For example:

type Student struct {
  Name    string  `parquet:"name=name, type=BYTE_ARRAY, convertedtype=UTF8, encoding=PLAIN_DICTIONARY"`
  Age     int32   `parquet:"name=age, type=INT32, encoding=PLAIN"`
  Id      int64   `parquet:"name=id, type=INT64"`
  Weight  float32 `parquet:"name=weight, type=FLOAT"`
  Sex     bool    `parquet:"name=sex, type=BOOLEAN"`
  Day     int32   `parquet:"name=day, type=INT32, convertedtype=DATE"`
  Ignored int32   //without parquet tag and won't write
}

func Write

func Write(s beam.Scope, filename string, col beam.PCollection)

Write writes a PCollection<parquetStruct> to .parquet file. Write expects elements of a struct type with parquet tags For example:

type Student struct {
  Name    string  `parquet:"name=name, type=BYTE_ARRAY, convertedtype=UTF8, encoding=PLAIN_DICTIONARY"`
  Age     int32   `parquet:"name=age, type=INT32, encoding=PLAIN"`
  Id      int64   `parquet:"name=id, type=INT64"`
  Weight  float32 `parquet:"name=weight, type=FLOAT"`
  Sex     bool    `parquet:"name=sex, type=BOOLEAN"`
  Day     int32   `parquet:"name=day, type=INT32, convertedtype=DATE"`
  Ignored int32   //without parquet tag and won't write
}