transform

Documentation

Overview

Package transform includes these cast functions to allow the project to accomplish two things:

• convert strings from the CSV files to other formats (e.g. int, float32, time.Time);
• differentiate empty values (such as 0 for int) from missing values.

This is achieved using pointers, so we have nil as a marker for missing value.

Since our use case involves serving this data in JSON format, this is crucial so we can use `null` when there is no value, and "0" when the value of an integer, for example, is 0.

Index

• Constants
• func Transform(dir string, db database, maxParallelDBQueries, batchSize int, ...) error

Constants

const BatchSize = 8192

BatchSize determines the size of the batches used to create the initial JSON data in the database.

const MaxParallelDBQueries = 8

MaxParallelDBQueries is the default for maximum number of parallels save queries sent to the database

const NationalTreasureFileName = "TABMUN.CSV"

NationalTreasureFileName is the name of the CSV containing city names and their codes.

Functions

func Transform

func Transform(dir string, db database, maxParallelDBQueries, batchSize int, privacy, mem bool) error

Transform the downloaded files for company venues creating a database record per CNPJ