Documentation ¶
Overview ¶
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Copyright (c) YugabyteDB, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Index ¶
- Constants
- Variables
- func BoolVar(flagSet *pflag.FlagSet, p *utils.BoolStr, name string, value bool, ...)
- func CleanupChildProcesses()
- func ClearMigrationAssessmentDone() error
- func CreateMigrationProjectIfNotExists(dbType string, exportDir string)
- func Execute()
- func ExitIfAlreadyCutover(importerOrExporterRole string)
- func GetAllLeafPartitions(table sqlname.NameTuple) []sqlname.NameTuple
- func GetCommandID(c *cobra.Command) string
- func GetDefaultPGSchema(schema string, separator string) (string, bool)
- func GetRootTableOfPartition(table sqlname.NameTuple) (sqlname.NameTuple, error)
- func GetSourceDBTypeFromMSR() string
- func GetTableRowCount(filePath string) map[string]int64
- func InitLogging(logDir string, disableLogging bool, cmdName string)
- func InitNameRegistry(exportDir string, role string, sconf *srcdb.Source, sdb srcdb.SourceDB, ...) error
- func InitiateCutover(dbRole string, prepareforFallback bool) error
- func IsMigrationAssessmentDone() (bool, error)
- func PrintElapsedDuration()
- func SetMigrationAssessmentDoneInMSR() error
- func ShutdownProcess(pid int, forceShutdownAfterSeconds int) error
- func UpdateTableApproxRowCount(source *srcdb.Source, exportDir string, ...)
- type AssessmentReport
- type Batch
- func (batch *Batch) Delete() error
- func (batch *Batch) GetFilePath() string
- func (batch *Batch) GetQueryIsBatchAlreadyImported() string
- func (batch *Batch) GetQueryToRecordEntryInDB(rowsAffected int64) string
- func (batch *Batch) GetTableName() sqlname.NameTuple
- func (batch *Batch) IsDone() bool
- func (batch *Batch) IsInterrupted() bool
- func (batch *Batch) IsNotStarted() bool
- func (batch *Batch) MarkDone() error
- func (batch *Batch) MarkPending() error
- func (batch *Batch) Open() (*os.File, error)
- type BatchWriter
- type ConflictDetectionCache
- type EventChannelMetaInfo
- type EventQueue
- type EventQueueSegment
- type EventSegmentCopier
- type EventSegmentDeleter
- type ExportSnapshotStatus
- type FileImportState
- type ImportDataProgressReporter
- type ImportDataState
- func (s *ImportDataState) Clean(filePath string, tableNameTup sqlname.NameTuple) error
- func (s *ImportDataState) DiscoverTableToFilesMapping() (map[string][]string, error)
- func (s *ImportDataState) GetAllBatches(filePath string, tableNameTup sqlname.NameTuple) ([]*Batch, error)
- func (s *ImportDataState) GetCompletedBatches(filePath string, tableNameTup sqlname.NameTuple) ([]*Batch, error)
- func (s *ImportDataState) GetEventChannelsMetaInfo(migrationUUID uuid.UUID) (map[int]EventChannelMetaInfo, error)
- func (s *ImportDataState) GetFileImportState(filePath string, tableNameTup sqlname.NameTuple) (FileImportState, error)
- func (s *ImportDataState) GetImportedByteCount(filePath string, tableNameTup sqlname.NameTuple) (int64, error)
- func (s *ImportDataState) GetImportedEventsStatsForTable(tableNameTup sqlname.NameTuple, migrationUUID uuid.UUID) (*tgtdb.EventCounter, error)
- func (s *ImportDataState) GetImportedRowCount(filePath string, tableNameTup sqlname.NameTuple) (int64, error)
- func (s *ImportDataState) GetImportedSnapshotRowCountForTable(tableNameTup sqlname.NameTuple) (int64, error)
- func (s *ImportDataState) GetPendingBatches(filePath string, tableNameTup sqlname.NameTuple) ([]*Batch, error)
- func (s *ImportDataState) GetTotalNumOfEventsImportedByType(migrationUUID uuid.UUID) (int64, int64, int64, error)
- func (s *ImportDataState) InitLiveMigrationState(migrationUUID uuid.UUID, numChans int, startClean bool, ...) error
- func (s *ImportDataState) IsEventBatchAlreadyImported(batch *tgtdb.EventBatch, migrationUUID uuid.UUID) (bool, error)
- func (s *ImportDataState) NewBatchWriter(filePath string, tableNameTup sqlname.NameTuple, batchNumber int64) *BatchWriter
- func (s *ImportDataState) PrepareForFileImport(filePath string, tableNameTup sqlname.NameTuple) error
- func (s *ImportDataState) Recover(filePath string, tableNameTup sqlname.NameTuple) ([]*Batch, int64, int64, bool, error)
- type ImportFileTask
- type MyFormatter
- type ProgressTracker
- type TableExportStatus
- type UnsupportedFeature
Constants ¶
const ( INHERITANCE_ISSUE_REASON = "TABLE INHERITANCE not supported in YugabyteDB" CONSTRAINT_TRIGGER_ISSUE_REASON = "CONSTRAINT TRIGGER not supported yet." COMPOUND_TRIGGER_ISSUE_REASON = "COMPOUND TRIGGER not supported in YugabyteDB." STORED_GENERATED_COLUMN_ISSUE_REASON = "Stored generated column not supported." GIST_INDEX_ISSUE_REASON = "Schema contains GIST index which is not supported." )
const ( CODE_BLOCK_NOT_STARTED = 0 CODE_BLOCK_STARTED = 1 CODE_BLOCK_COMPLETED = 2 )
const ( KB = 1024 MB = 1024 * 1024 META_INFO_DIR_NAME = "metainfo" NEWLINE = '\n' ORACLE_DEFAULT_PORT = 1521 MYSQL_DEFAULT_PORT = 3306 POSTGRES_DEFAULT_PORT = 5432 YUGABYTEDB_YSQL_DEFAULT_PORT = 5433 YUGABYTEDB_DEFAULT_DATABASE = "yugabyte" YUGABYTEDB_DEFAULT_SCHEMA = "public" ORACLE = "oracle" MYSQL = "mysql" POSTGRESQL = "postgresql" YUGABYTEDB = "yugabytedb" LAST_SPLIT_NUM = 0 SPLIT_INFO_PATTERN = "[0-9]*.[0-9]*.[0-9]*.[0-9]*" LAST_SPLIT_PATTERN = "0.[0-9]*.[0-9]*.[0-9]*" COPY_MAX_RETRY_COUNT = 10 MAX_SLEEP_SECOND = 60 DEFAULT_BATCH_SIZE_ORACLE = 10000000 DEFAULT_BATCH_SIZE_YUGABYTEDB = 20000 DEFAULT_BATCH_SIZE_POSTGRESQL = 100000 INDEX_RETRY_COUNT = 5 DDL_MAX_RETRY_COUNT = 5 SCHEMA_VERSION_MISMATCH_ERR = "Query error: schema version mismatch for table" SNAPSHOT_ONLY = "snapshot-only" SNAPSHOT_AND_CHANGES = "snapshot-and-changes" CHANGES_ONLY = "changes-only" TARGET_DB = "target" FF_DB = "ff" SOURCE_REPLICA_DB_IMPORTER_ROLE = "source_replica_db_importer" SOURCE_DB_IMPORTER_ROLE = "source_db_importer" TARGET_DB_IMPORTER_ROLE = "target_db_importer" SOURCE_DB_EXPORTER_ROLE = "source_db_exporter" TARGET_DB_EXPORTER_FF_ROLE = "target_db_exporter_ff" TARGET_DB_EXPORTER_FB_ROLE = "target_db_exporter_fb" IMPORT_FILE_ROLE = "import_file" ROW_UPDATE_STATUS_NOT_STARTED = 0 ROW_UPDATE_STATUS_IN_PROGRESS = 1 ROW_UPDATE_STATUS_COMPLETED = 3 COLOCATION_CLAUSE = "colocation" )
const ( NOT_INITIATED = "NOT_INITIATED" INITIATED = "INITIATED" COMPLETED = "COMPLETED" )
const ( QUEUE_DIR_NAME = "queue" QUEUE_SEGMENT_FILE_NAME = "segment" QUEUE_SEGMENT_FILE_EXTENSION = "ndjson" )
const ( // The _v2 is appended in the table name so that the import code doesn't // try to use the similar table created by the voyager 1.3 and earlier. // Voyager 1.4 uses import data state format that is incompatible from // the earlier versions. BATCH_METADATA_TABLE_SCHEMA = "ybvoyager_metadata" BATCH_METADATA_TABLE_NAME = BATCH_METADATA_TABLE_SCHEMA + "." + "ybvoyager_import_data_batches_metainfo_v3" EVENT_CHANNELS_METADATA_TABLE_NAME = BATCH_METADATA_TABLE_SCHEMA + "." + "ybvoyager_import_data_event_channels_metainfo" EVENTS_PER_TABLE_METADATA_TABLE_NAME = BATCH_METADATA_TABLE_SCHEMA + "." + "ybvoyager_imported_event_count_by_table" )
Variables ¶
var END_OF_QUEUE_SEGMENT_EVENT = &tgtdb.Event{Op: "end_of_source_queue_segment"}
var EOFMarker = []byte(`\.`)
var EVENT_BATCH_MAX_RETRY_COUNT = 50
var EVENT_CHANNEL_SIZE int // has to be > MAX_EVENTS_PER_BATCH
var FLUSH_BATCH_EVENT = &tgtdb.Event{Op: "flush_batch"}
var InProgressTableSno int
var MAX_EVENTS_PER_BATCH int
var MAX_INTERVAL_BETWEEN_BATCHES int //ms
var NUM_EVENT_CHANNELS int
var (
PARENT_COMMAND_USAGE = "Parent command. Refer to the sub-commands for usage help."
)
var (
ProcessShutdownRequested bool
)
var StopArchiverSignal bool
var TableToColumnNames = utils.NewStructMap[sqlname.NameTuple, []string]() // map of table name to columnNames
Functions ¶
func CleanupChildProcesses ¶ added in v1.6.5
func CleanupChildProcesses()
func ClearMigrationAssessmentDone ¶ added in v1.7.0
func ClearMigrationAssessmentDone() error
func CreateMigrationProjectIfNotExists ¶
setup a project having subdirs for various database objects IF NOT EXISTS
func Execute ¶
func Execute()
Execute adds all child commands to the root command and sets flags appropriately. This is called by main.main(). It only needs to happen once to the rootCmd.
func ExitIfAlreadyCutover ¶ added in v1.6.2
func ExitIfAlreadyCutover(importerOrExporterRole string)
func GetAllLeafPartitions ¶ added in v1.6.0
func GetCommandID ¶ added in v1.5.0
func GetDefaultPGSchema ¶ added in v1.7.0
func GetRootTableOfPartition ¶ added in v1.6.0
func GetSourceDBTypeFromMSR ¶ added in v1.6.0
func GetSourceDBTypeFromMSR() string
func GetTableRowCount ¶
func InitLogging ¶
func InitNameRegistry ¶ added in v1.7.0
func InitiateCutover ¶ added in v1.6.2
func IsMigrationAssessmentDone ¶ added in v1.7.0
func PrintElapsedDuration ¶ added in v1.7.0
func PrintElapsedDuration()
func SetMigrationAssessmentDoneInMSR ¶ added in v1.7.0
func SetMigrationAssessmentDoneInMSR() error
func ShutdownProcess ¶ added in v1.6.5
this function wait for process to exit after signalling it to stop
Types ¶
type AssessmentReport ¶ added in v1.7.0
type AssessmentReport struct { SchemaSummary utils.SchemaSummary `json:"SchemaSummary"` Sizing *migassessment.SizingAssessmentReport `json:"Sizing"` UnsupportedDataTypes []utils.TableColumnsDataTypes `json:"UnsupportedDataTypes"` UnsupportedFeatures []UnsupportedFeature `json:"UnsupportedFeatures"` TableIndexStats *[]migassessment.TableIndexStats `json:"TableIndexStats"` }
func ParseJSONToAssessmentReport ¶ added in v1.7.0
func ParseJSONToAssessmentReport(reportPath string) (*AssessmentReport, error)
func (*AssessmentReport) GetClusterSizingRecommendation ¶ added in v1.7.0
func (ar *AssessmentReport) GetClusterSizingRecommendation() string
func (*AssessmentReport) GetColocatedTablesRecommendation ¶ added in v1.7.0
func (ar *AssessmentReport) GetColocatedTablesRecommendation() ([]string, error)
func (*AssessmentReport) GetShardedTablesRecommendation ¶ added in v1.7.0
func (ar *AssessmentReport) GetShardedTablesRecommendation() ([]string, error)
type Batch ¶ added in v1.4.0
type Batch struct { Number int64 TableNameTup sqlname.NameTuple SchemaName string FilePath string // Path of the batch file. BaseFilePath string // Path of the original data file. OffsetStart int64 OffsetEnd int64 RecordCount int64 ByteCount int64 TmpConnectionString string Interrupted bool }
func (*Batch) GetFilePath ¶ added in v1.5.0
func (*Batch) GetQueryIsBatchAlreadyImported ¶ added in v1.5.0
func (*Batch) GetQueryToRecordEntryInDB ¶ added in v1.5.0
func (*Batch) GetTableName ¶ added in v1.5.0
func (*Batch) IsInterrupted ¶ added in v1.4.0
func (*Batch) IsNotStarted ¶ added in v1.4.0
func (*Batch) MarkPending ¶ added in v1.4.0
type BatchWriter ¶ added in v1.4.0
type BatchWriter struct { NumRecordsWritten int64 // contains filtered or unexported fields }
func (*BatchWriter) Init ¶ added in v1.4.0
func (bw *BatchWriter) Init() error
func (*BatchWriter) WriteHeader ¶ added in v1.4.0
func (bw *BatchWriter) WriteHeader(header string) error
func (*BatchWriter) WriteRecord ¶ added in v1.4.0
func (bw *BatchWriter) WriteRecord(record string) error
type ConflictDetectionCache ¶ added in v1.6.5
ConflictDetectionCache is a thread-safe class used to store and manage conflicting events during migration's streaming phase. Conflict occurs when two events have the same unique key column value. For example, if we have a table with a unique key column "email" with a existing row: {id: 1, email: 'abc@example.com'}, and two new events comes in: event1: DELETE FROM users WHERE id = 1; event2: INSERT INTO users (id, email) VALUES (2, 'abc@example.com');
In this case, event1 and event2 are considered as a conflicting events, because they have the same unique key column value.
In a concurrent environment we can't just apply the second event because both the events can be part of different parallel batches and we can't guarantee the order of the events in the batches.
So, this cache stores events like event1 and wait for them to be processed before processing event2.
There can be total 4 types of conflicts: 1. DELETE-INSERT 2. DELETE-UPDATE 3. UPDATE-INSERT 4. UPDATE-UPDATE
Case: UPDATE-INSERT conflict:
example_table (id PK, email UNIQUE)
// Insert initial rows INSERT INTO example_table VALUES (1, 'user21@example.com'); INSERT INTO example_table VALUES (2, 'user22@example.com'); INSERT INTO example_table VALUES (3, 'user23@example.com'); INSERT INTO example_table VALUES (4, 'user24@example.com');
UPDATE example_table SET email = 'user224@example.com' WHERE id = 4;
-- Insert a new row with the conflicting email INSERT INTO example_table VALUES (5, 'user24@example.com');
Case: UPDATE-UPDATE conflict:
example_table (id PK, email UNIQUE)
// Insert initial rows INSERT INTO example_table VALUES (1, 'user31@example.com'); INSERT INTO example_table VALUES (2, 'user32@example.com'); INSERT INTO example_table VALUES (3, 'user33@example.com'); INSERT INTO example_table VALUES (4, 'user34@example.com');
UPDATE example_table SET email = 'updated_user2@example.com' WHERE id = 2;
-- Another conflicting update for id = 3, setting it to previous value of id = 2 UPDATE example_table SET email = 'user32@example.com' WHERE id = 3;
Case: DELETE-UPDATE conflict:
example_table (id PK, email UNIQUE)
// Insert initial rows INSERT INTO example_table VALUES (1, 'user41@example.com'); INSERT INTO example_table VALUES (2, 'user42@example.com'); INSERT INTO example_table VALUES (3, 'user43@example.com'); INSERT INTO example_table VALUES (4, 'user44@example.com');
DELETE FROM example_table WHERE id = 2;
-- Another conflicting update for id = 3, setting it to previous value of id = 2 UPDATE example_table SET email = 'user42@example.com' WHERE id = 3;
func NewConflictDetectionCache ¶ added in v1.6.5
func (*ConflictDetectionCache) Put ¶ added in v1.6.5
func (c *ConflictDetectionCache) Put(event *tgtdb.Event)
func (*ConflictDetectionCache) RemoveEvents ¶ added in v1.6.5
func (c *ConflictDetectionCache) RemoveEvents(batch *tgtdb.EventBatch)
func (*ConflictDetectionCache) WaitUntilNoConflict ¶ added in v1.6.5
func (c *ConflictDetectionCache) WaitUntilNoConflict(incomingEvent *tgtdb.Event)
type EventChannelMetaInfo ¶ added in v1.6.0
type EventQueue ¶ added in v1.5.0
func NewEventQueue ¶ added in v1.5.0
func NewEventQueue(exportDir string) *EventQueue
func (*EventQueue) GetNextSegment ¶ added in v1.5.0
func (eq *EventQueue) GetNextSegment() (*EventQueueSegment, error)
GetNextSegment returns the next segment to process
type EventQueueSegment ¶ added in v1.5.0
type EventQueueSegment struct { FilePath string SegmentNum int64 // 0-based // contains filtered or unexported fields }
func NewEventQueueSegment ¶ added in v1.5.0
func NewEventQueueSegment(filePath string, segmentNum int64) *EventQueueSegment
func (*EventQueueSegment) Close ¶ added in v1.5.0
func (eqs *EventQueueSegment) Close() error
func (*EventQueueSegment) IsProcessed ¶ added in v1.5.0
func (eqs *EventQueueSegment) IsProcessed() bool
func (*EventQueueSegment) MarkProcessed ¶ added in v1.5.0
func (eqs *EventQueueSegment) MarkProcessed()
func (*EventQueueSegment) NextEvent ¶ added in v1.5.0
func (eqs *EventQueueSegment) NextEvent() (*tgtdb.Event, error)
ReadEvent reads an event from the segment file. Waits until an event is available.
func (*EventQueueSegment) Open ¶ added in v1.5.0
func (eqs *EventQueueSegment) Open() error
type EventSegmentCopier ¶ added in v1.5.0
type EventSegmentCopier struct {
Dest string
}
func NewEventSegmentCopier ¶ added in v1.5.0
func NewEventSegmentCopier(dest string) *EventSegmentCopier
func (*EventSegmentCopier) Run ¶ added in v1.5.0
func (m *EventSegmentCopier) Run() error
type EventSegmentDeleter ¶ added in v1.5.0
type EventSegmentDeleter struct {
FSUtilisationThreshold int
}
func NewEventSegmentDeleter ¶ added in v1.5.0
func NewEventSegmentDeleter(fsUtilisationThreshold int) *EventSegmentDeleter
func (*EventSegmentDeleter) Run ¶ added in v1.5.0
func (d *EventSegmentDeleter) Run() error
type ExportSnapshotStatus ¶ added in v1.6.5
type ExportSnapshotStatus struct {
Tables map[string]*TableExportStatus `json:"tables"`
}
func NewExportSnapshotStatus ¶ added in v1.6.5
func NewExportSnapshotStatus() *ExportSnapshotStatus
func (*ExportSnapshotStatus) GetTableStatusByTableName ¶ added in v1.7.0
func (e *ExportSnapshotStatus) GetTableStatusByTableName(tableName string) []*TableExportStatus
type FileImportState ¶ added in v1.4.0
type FileImportState string
const ( FILE_IMPORT_STATE_UNKNOWN FileImportState = "FILE_IMPORT_STATE_UNKNOWN" FILE_IMPORT_NOT_STARTED FileImportState = "FILE_IMPORT_NOT_STARTED" FILE_IMPORT_IN_PROGRESS FileImportState = "FILE_IMPORT_IN_PROGRESS" FILE_IMPORT_COMPLETED FileImportState = "FILE_IMPORT_COMPLETED" )
type ImportDataProgressReporter ¶ added in v1.4.0
func NewImportDataProgressReporter ¶ added in v1.4.0
func NewImportDataProgressReporter(disablePb bool) *ImportDataProgressReporter
func (*ImportDataProgressReporter) AddProgressAmount ¶ added in v1.4.0
func (pr *ImportDataProgressReporter) AddProgressAmount(task *ImportFileTask, progressAmount int64)
func (*ImportDataProgressReporter) FileImportDone ¶ added in v1.4.0
func (pr *ImportDataProgressReporter) FileImportDone(task *ImportFileTask)
func (*ImportDataProgressReporter) ImportFileStarted ¶ added in v1.4.0
func (pr *ImportDataProgressReporter) ImportFileStarted(task *ImportFileTask, totalProgressAmount int64)
type ImportDataState ¶ added in v1.4.0
type ImportDataState struct {
// contains filtered or unexported fields
}
metainfo/import_data_state/table::<table_name>/file::<base_name>:<path_hash>/
link -> dataFile batch::<batch_num>.<offset_end>.<record_count>.<byte_count>.<state>
func NewImportDataState ¶ added in v1.4.0
func NewImportDataState(exportDir string) *ImportDataState
func (*ImportDataState) Clean ¶ added in v1.4.0
func (s *ImportDataState) Clean(filePath string, tableNameTup sqlname.NameTuple) error
func (*ImportDataState) DiscoverTableToFilesMapping ¶ added in v1.4.0
func (s *ImportDataState) DiscoverTableToFilesMapping() (map[string][]string, error)
TODO:TABLENAME: revisit??
func (*ImportDataState) GetAllBatches ¶ added in v1.4.0
func (*ImportDataState) GetCompletedBatches ¶ added in v1.4.0
func (*ImportDataState) GetEventChannelsMetaInfo ¶ added in v1.6.0
func (s *ImportDataState) GetEventChannelsMetaInfo(migrationUUID uuid.UUID) (map[int]EventChannelMetaInfo, error)
func (*ImportDataState) GetFileImportState ¶ added in v1.4.0
func (s *ImportDataState) GetFileImportState(filePath string, tableNameTup sqlname.NameTuple) (FileImportState, error)
func (*ImportDataState) GetImportedByteCount ¶ added in v1.4.0
func (*ImportDataState) GetImportedEventsStatsForTable ¶ added in v1.6.0
func (s *ImportDataState) GetImportedEventsStatsForTable(tableNameTup sqlname.NameTuple, migrationUUID uuid.UUID) (*tgtdb.EventCounter, error)
func (*ImportDataState) GetImportedRowCount ¶ added in v1.4.0
func (*ImportDataState) GetImportedSnapshotRowCountForTable ¶ added in v1.6.0
func (s *ImportDataState) GetImportedSnapshotRowCountForTable(tableNameTup sqlname.NameTuple) (int64, error)
func (*ImportDataState) GetPendingBatches ¶ added in v1.4.0
func (*ImportDataState) GetTotalNumOfEventsImportedByType ¶ added in v1.6.0
func (*ImportDataState) InitLiveMigrationState ¶ added in v1.6.0
func (*ImportDataState) IsEventBatchAlreadyImported ¶ added in v1.7.0
func (s *ImportDataState) IsEventBatchAlreadyImported(batch *tgtdb.EventBatch, migrationUUID uuid.UUID) (bool, error)
func (*ImportDataState) NewBatchWriter ¶ added in v1.4.0
func (s *ImportDataState) NewBatchWriter(filePath string, tableNameTup sqlname.NameTuple, batchNumber int64) *BatchWriter
func (*ImportDataState) PrepareForFileImport ¶ added in v1.4.0
func (s *ImportDataState) PrepareForFileImport(filePath string, tableNameTup sqlname.NameTuple) error
type ImportFileTask ¶ added in v1.4.0
type MyFormatter ¶
type MyFormatter struct{}
type ProgressTracker ¶ added in v1.3.0
type ProgressTracker struct {
// contains filtered or unexported fields
}
func NewProgressTracker ¶ added in v1.3.0
func NewProgressTracker(totalRowCount map[string]int64) *ProgressTracker
func (*ProgressTracker) Done ¶ added in v1.3.0
func (pt *ProgressTracker) Done(status *dbzm.ExportStatus)
func (*ProgressTracker) UpdateProgress ¶ added in v1.3.0
func (pt *ProgressTracker) UpdateProgress(status *dbzm.ExportStatus)
type TableExportStatus ¶ added in v1.6.5
type TableExportStatus struct { TableName string `json:"table_name"` // table.Qualified.MinQuoted FileName string `json:"file_name"` Status string `json:"status"` ExportedRowCountSnapshot int64 `json:"exported_row_count_snapshot"` }
For Non-debezium cases
type UnsupportedFeature ¶ added in v1.7.0
Source Files ¶
- analyzeSchema.go
- archiveChangesCommand.go
- archiveCommand.go
- assessMigrationCommand.go
- common.go
- conflictDetectionCache.go
- constants.go
- cutover.go
- cutoverStatus.go
- cutoverToSource.go
- cutoverToSourceReplica.go
- cutoverToTargetCmd.go
- endCommand.go
- endMigrationCommand.go
- eventQueue.go
- export.go
- exportData.go
- exportDataDebezium.go
- exportDataFromTarget.go
- exportDataStatus.go
- exportDataStatusCommand.go
- exportSchema.go
- getCommand.go
- getDataMigrationReportCommand.go
- import.go
- importData.go
- importDataFileCommand.go
- importDataProgressReporter.go
- importDataState.go
- importDataStatusCommand.go
- importDataToSource.go
- importDataToSourceReplica.go
- importSchema.go
- importSchemaYugabyteDB.go
- initiate.go
- live_migration.go
- logging.go
- progressTracker.go
- root.go
- version.go