utils

Documentation

Overview

Package utils contains common helper functions used across multiple other packages. Utils should not import any Spanner migration tool packages.

Package utils contains common helper functions used across multiple other packages. Utils should not import any Spanner migration tool packages.

Index

• func AnalyzeError(err error, URI string) error
• func CheckEqualSets(a, b []string) bool
• func Close(f *os.File)
• func CompareSchema(conv1, conv2 *internal.Conv) error
• func ContainsAny(s string, l []string) bool
• func CreateGCSBucket(bucketName, projectID, location string) error
• func DownloadFromGCS(bucketName, filePath, tmpFile string) (*os.File, error)
• func GenerateName(prefix string) (string, error)
• func GetBanner(now time.Time, db string) string
• func GetClient(ctx context.Context, db string) (*sp.Client, error)
• func GetDatabaseName(driver string, now time.Time) (string, error)
• func GetFileSize(f *os.File) (int64, error)
• func GetGcloudDataflowCommand(req *dataflowpb.LaunchFlexTemplateRequest) string
• func GetInstance(ctx context.Context, project string, out *os.File) (string, error)
• func GetLegacyModeSupportedDrivers() []string
• func GetPassword() string
• func GetProject() (string, error)
• func GetValidDrivers() []string
• func IsLegacyModeSupportedDriver(driver string) bool
• func IsValidDriver(driver string) bool
• func NewDatabaseAdminClient(ctx context.Context) (*database.DatabaseAdminClient, error)
• func NewInstanceAdminClient(ctx context.Context) (*instance.InstanceAdminClient, error)
• func NewSpannerClient(ctx context.Context, db string) (*sp.Client, error)
• func ParseDbURI(dbURI string) (project, instance, dbName string)
• func ParseGCSFilePath(filePath string) (*url.URL, error)
• func PrintPermissionsWarning(driver string, out *os.File)
• func PrintSeekError(driver string, err error, out *os.File)
• func ReadSpannerSchema(ctx context.Context, conv *internal.Conv, client *sp.Client) error
• func SetupLogFile() (*os.File, error)
• func SumMapValues(m map[string]int64) int64
• func TargetDbToDialect(targetDb string) string
• func WriteToGCS(filePath, fileName, data string) error
• type IOStreams
  • func NewIOStreams(driver string, dumpFile string) IOStreams
• type ManifestTable
  • func PreloadGCSFiles(tables []ManifestTable) ([]ManifestTable, error)

Functions

func AnalyzeError

func AnalyzeError(err error, URI string) error

AnalyzeError inspects an error returned from Cloud Spanner and adds information about potential root causes e.g. authentication issues.

func CheckEqualSets

func CheckEqualSets(a, b []string) bool

CheckEqualSets checks if the set of values in a and b are equal.

func Close

func Close(f *os.File)

Close closes file.

func CompareSchema

func CompareSchema(conv1, conv2 *internal.Conv) error

CompareSchema compares the spanner schema of two conv objects and returns specific error if they don't match

func ContainsAny

func ContainsAny(s string, l []string) bool

func CreateGCSBucket

func CreateGCSBucket(bucketName, projectID, location string) error

func DownloadFromGCS

func DownloadFromGCS(bucketName, filePath, tmpFile string) (*os.File, error)

DownloadFromGCS returns the dump file that is downloaded from GCS.

func GenerateName

func GenerateName(prefix string) (string, error)

func GetBanner

func GetBanner(now time.Time, db string) string

GetBanner prints banner message after command line process is finished.

func GetClient

func GetClient(ctx context.Context, db string) (*sp.Client, error)

GetClient returns a new Spanner client. It uses the background context.

func GetDatabaseName

func GetDatabaseName(driver string, now time.Time) (string, error)

GetDatabaseName generates database name with driver_date prefix.

func GetFileSize

func GetFileSize(f *os.File) (int64, error)

func GetGcloudDataflowCommand

func GetGcloudDataflowCommand(req *dataflowpb.LaunchFlexTemplateRequest) string

Generate the equivalent gCloud CLI command to launch a dataflow job with the same parameters and environment flags as the input body.

func GetInstance

func GetInstance(ctx context.Context, project string, out *os.File) (string, error)

GetInstance returns the Spanner instance we should use for creating DBs. If the user specified instance (via flag 'instance') then use that. Otherwise try to deduce the instance using gcloud.

func GetLegacyModeSupportedDrivers

func GetLegacyModeSupportedDrivers() []string

func GetPassword

func GetPassword() string

func GetProject

func GetProject() (string, error)

GetProject returns the cloud project we should use for accessing Spanner. Use environment variable GCLOUD_PROJECT if it is set. Otherwise, use the default project returned from gcloud.

func GetValidDrivers

func GetValidDrivers() []string

func IsLegacyModeSupportedDriver

func IsLegacyModeSupportedDriver(driver string) bool

func IsValidDriver

func IsValidDriver(driver string) bool

func NewDatabaseAdminClient

func NewDatabaseAdminClient(ctx context.Context) (*database.DatabaseAdminClient, error)

NewDatabaseAdminClient returns a new db-admin client. It respects SPANNER_API_ENDPOINT.

func NewInstanceAdminClient

func NewInstanceAdminClient(ctx context.Context) (*instance.InstanceAdminClient, error)

NewInstanceAdminClient returns a new instance-admin client. It respects SPANNER_API_ENDPOINT.

func NewSpannerClient

func NewSpannerClient(ctx context.Context, db string) (*sp.Client, error)

NewSpannerClient returns a new Spanner client. It respects SPANNER_API_ENDPOINT.

func ParseDbURI

func ParseDbURI(dbURI string) (project, instance, dbName string)

func ParseGCSFilePath

func ParseGCSFilePath(filePath string) (*url.URL, error)

func PrintPermissionsWarning

func PrintPermissionsWarning(driver string, out *os.File)

PrintPermissionsWarning prints permission warning.

func PrintSeekError

func PrintSeekError(driver string, err error, out *os.File)

func ReadSpannerSchema

func ReadSpannerSchema(ctx context.Context, conv *internal.Conv, client *sp.Client) error

ReadSpannerSchema fills conv by querying Spanner infoschema treating Spanner as both the source and dest.

func SetupLogFile

func SetupLogFile() (*os.File, error)

SetupLogFile configures the file used for logs. By default we just drop logs on the floor. To enable them (e.g. to debug Cloud Spanner client library issues), set logfile to a non-empty filename. Note: this tool itself doesn't generate logs, but some of the libraries it uses do. If we don't set the log file, we see a number of unhelpful and unactionable logs spamming stdout, which is annoying and confusing.

func SumMapValues

func SumMapValues(m map[string]int64) int64

func TargetDbToDialect

func TargetDbToDialect(targetDb string) string

func WriteToGCS

func WriteToGCS(filePath, fileName, data string) error

Types

type IOStreams

type IOStreams struct {
	In, SeekableIn, Out *os.File
	BytesRead           int64
}

IOStreams is a struct that contains the file descriptor for dumpFile.

func NewIOStreams

func NewIOStreams(driver string, dumpFile string) IOStreams

NewIOStreams returns a new IOStreams struct such that input stream is set to open file descriptor for dumpFile if driver is PGDUMP or MYSQLDUMP. Input stream defaults to stdin. Output stream is always set to stdout.

type ManifestTable

type ManifestTable struct {
	Table_name    string   `json:"table_name"`
	File_patterns []string `json:"file_patterns"`
}

Spanner migration tool accepts a manifest file in the form of a json which unmarshalls into the ManifestTables struct.

func PreloadGCSFiles

func PreloadGCSFiles(tables []ManifestTable) ([]ManifestTable, error)

PreloadGCSFiles downloads gcs files to tmp and updates the file paths in manifest with the local path.