converter

Documentation

Index

• Constants
• func Convert(collector string, r io.Reader, dst io.Writer)
• func ObjExists(ctx context.Context, gcsCli *storage.Client, object, bucket string) (bool, error)
• func ProcessMRTArchive(ctx context.Context, gcsCli *storage.Client, cfg *Config) error
• type Config

Constants

const ProjectMetadataKey = "routingDataProject"

ProjectMetadataKey maps to the project source in an archive's GCS metadata.

Functions

func Convert

func Convert(collector string, r io.Reader, dst io.Writer)

Convert translates the bzip'ed MRT raw bytes into a BigQuery compatible format and write to the destination.

func ObjExists

func ObjExists(ctx context.Context, gcsCli *storage.Client, object, bucket string) (bool, error)

ObjExists checks if a converted archive already exists at the destination.

func ProcessMRTArchive

func ProcessMRTArchive(ctx context.Context, gcsCli *storage.Client, cfg *Config) error

ProcessMRTArchive converts an MRT dump into updates on GCS, which will later be picked up by BigQuery automatically. ProcessMRTDump converts on a best- effort basis as it will convert as much as it can from every archive, and it only supports archives of updates.

Types

type Config

type Config struct {
	SrcBucket string
	DstBucket string
	SrcObject string
}