gloader

README

GLoader

Migrate your data across any source and destination with a single command!

---

GLoader is a powerful and flexible CLI tool for data migration between different databases. It provides a seamless way to migrate your data from any source database to any destination database, Whether you are upgrading your database or moving data between different systems; GLoader makes the process efficient and reliable.

List of contents

• List of supported databases
• Getting started
  • Installation
  • Usage
• Glossary
• Contributing
• Security
• Credits
• License

List of supported databases

Database \ As	Source	Destination
MySQL	✅	❌ (Soon)
CockroachDB	❌ (Soon)	✅
PostgreSQL	❌ (Soon)	❌ (Soon)
MongoDB	❌	❌
SQLite	❌	❌
SQL Server	❌	❌
Oracle	❌	❌
Redis	❌	❌
Cassandra	❌	❌
Elasticsearch	❌	❌
Kafka	❌	❌
RabbitMQ	❌	❌
DynamoDB	❌	❌

Note: The database that is marked with ❌ will be supported soon. However, if you have time and want to contribute, you can help us to support them faster by contributing to the project.

Getting started

---

Installation

Debian based Linux distributions

curl -fsSL https://repo.gloader.tech/apt/gpg.key | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/gloader.gpg
echo "deb https://repo.gloader.tech/apt * *" > /etc/apt/sources.list.d/gloader.list
sudo apt update && sudo apt install gloader

Fedora / RHEL / CentOS

echo '[gloader]
name=Gloader
baseurl=https://repo.gloader.tech/yum
enabled=1
gpgcheck=1
gpgkey=https://repo.gloader.tech/yum/gpg.key' | sudo tee /etc/yum.repos.d/gloader.repo
sudo yum install gloader

Ubuntu

--- OR ---

sudo snap install gloader

Homebrew

brew tap mohammadv184/gloader
brew install gloader

Go install (Not recommended)

go install github.com/mohammadv184/gloader@latest

Binary builds

You can download the binary builds from the releases

Deb and RPM packages

You can download the deb and rpm packages from the releases

Usage

gloader run <source-dsn> <destination-dsn> [flags]
flags:
      --end-offset stringToInt64           end offset for each table (default [])
  -e, --exclude strings                    exclude tables from migration
  -f, --filter stringToStringSlice         filter data to migrate
      --filter-all strings                 filter data to migrate (all tables)
  -h, --help                               help for run
  -r, --rows-per-batch uint                number of rows per batch (default 100)
  -s, --sort stringToStringSlice           sort data to migrate in ascending order
      --sort-all strings                   sort data to migrate in ascending order (all tables)
  -S, --sort-reverse stringToStringSlice   sort data to migrate in descending order
      --sort-reverse-all strings           sort data to migrate in descending order (all tables)
      --start-offset stringToInt64         start offset for each table (default [])
  -t, --table strings                      migrate only these tables
  -w, --workers uint                       number of workers (default 3)

Arguments

• source-dsn: The Data Source Name (DSN) used to connect to the source database.
• destination-dsn: The DSN used to connect to the destination database.

Flags

• --start-offset: The initial row offset for each table. This sets the starting point for migrating rows from the source to the destination.
• --end-offset: The final row offset for each table, limiting the number of rows migrated.
• --exclude: Exclude specific tables from the migration process.
• --table: Selectively migrate specific tables.
• --filter: Apply data filters to rows being migrated. Use operators such as =, !=, >, >=, <, and <=.
• --filter-all: Apply a universal data filter for all tables.
• --sort: Sort data in ascending order before migration.
• --sort-all: Apply ascending sorting for all tables.
• --sort-reverse: Sort data in descending order before migration.
• --sort-reverse-all: Apply descending sorting for all tables.
• --rows-per-batch: Set the number of rows migrated per batch.
• --workers: Specify the number of parallel migration workers.

Examples

Migrate all tables from the source to the destination

gloader run "mysql://root:root@tcp(localhost:3306)/source" "mysql://root:root@tcp(localhost:3306)/destination"

Migrate only the users table from the source to the destination

gloader run "mysql://root:root@tcp(localhost:3306)/source" "mysql://root:root@tcp(localhost:3306)/destination" --table users

Migrate all tables except the users table from the source to the destination

gloader run "mysql://root:root@tcp(localhost:3306)/source" "mysql://root:root@tcp(localhost:3306)/destination" --exclude users

Migrate all tables from the source to the destination with a filter

gloader run "mysql://root:root@tcp(localhost:3306)/source" "mysql://root:root@tcp(localhost:3306)/destination" --filter "id > 100"

Migrate all tables from the source to the destination with a filter for all tables

gloader run "mysql://root:root@tcp(localhost:3306)/source" "mysql://root:root@tcp(localhost:3306)/destination" --filter-all "id > 100"

Migrate all tables from the source to the destination with a sort

gloader run "mysql://root:root@tcp(localhost:3306)/source" "mysql://root:root@tcp(localhost:3306)/destination" --sort "id"

Migrate all tables from the source to the destination with a sort for all tables

gloader run "mysql://root:root@tcp(localhost:3306)/source" "mysql://root:root@tcp(localhost:3306)/destination" --sort-all "id"

Migrate all tables from the source to the destination with a sort-reverse

gloader run "mysql://root:root@tcp(localhost:3306)/source" "mysql://root:root@tcp(localhost:3306)/destination" --sort-reverse "id"

Migrate all tables from the source to the destination with a sort-reverse for all tables

gloader run "mysql://root:root@tcp(localhost:3306)/source" "mysql://root:root@tcp(localhost:3306)/destination" --sort-reverse-all "id"

Migrate all tables from the source to the destination with a start offset for each table

gloader run "mysql://root:root@tcp(localhost:3306)/source" "mysql://root:root@tcp(localhost:3306)/destination" --start-offset "users=100"

Migrate all tables from the source to the destination with a end offset for each table

gloader run "mysql://root:root@tcp(localhost:3306)/source" "mysql://root:root@tcp(localhost:3306)/destination" --end-offset "users=100"

Migrate all tables from the source to the destination with a rows per batch and workers

gloader run "mysql://root:root@tcp(localhost:3306)/source" "mysql://root:root@tcp(localhost:3306)/destination" --sort-all "id" --rows-per-batch 1000 --workers 10

Glossary

• Data: A fundamental unit of information comprising a key, value, and data type. Data is the smallest entity in GLoader and represents the content being migrated.
• DataType: Denotes the type of data, such as strings, numbers, booleans, dates, JSON, etc., used in migration.
• DataSet: : A collection of data items representing a single row in a relational database.
• DataBatch: A group of DataSets migrated together in a single operation.
• DataBuffer: A storage space containing DataBatches fetched from the source database.
• DataCollection: An assembly of DataSets representing a table in a relational database.
• DataMap: A map associating DataCollections with their respective attributes, is equivalent to a table schema in a relational database.
• Database: A collection of DataCollections representing a database in a relational context.
• Migration: The process of relocating data from a source database to a target destination.

Security

If you discover any security-related issues, please email mohammadv184@gmail.com instead of using the issue tracker.

Credits

• Mohammad Abbasi
• All Contributors

License

The MIT License (MIT). Please see License File for more information.

Documentation

Index

• Constants
• Variables
• func NewStats() *stats.Stats
• type BufferObserverAdapter
  • func NewBufferObserverAdapter(s *stats.Stats, dcName string) *BufferObserverAdapter
  • func (b *BufferObserverAdapter) LengthChanged(l uint64)
  • func (b *BufferObserverAdapter) Read(n int)
  • func (b *BufferObserverAdapter) SizeChanged(size uint64)
  • func (b *BufferObserverAdapter) Write(n int)
• type GLoader
  • func NewGLoader() *GLoader
  • func (g *GLoader) Dest(name, dsn string) error
  • func (g *GLoader) Exclude(dataCollections ...string) *GLoader
  • func (g *GLoader) Filter(dataCollection, key string, condition driver.Condition, value string) *GLoader
  • func (g *GLoader) FilterAll(key string, condition driver.Condition, value string) *GLoader
  • func (g *GLoader) GetDestDetails(ctx context.Context) (driver.DatabaseDetail, error)
  • func (g *GLoader) GetSrcDetails(ctx context.Context) (driver.DatabaseDetail, error)
  • func (g *GLoader) Include(dataCollections ...string) *GLoader
  • func (g *GLoader) OrderBy(dataCollection, key string, direction driver.Direction) *GLoader
  • func (g *GLoader) OrderByAll(key string, direction driver.Direction) *GLoader
  • func (g *GLoader) SetEndOffset(dataCollection string, offset uint64) *GLoader
  • func (g *GLoader) SetRowsPerBatch(rowsPerBatch uint64) *GLoader
  • func (g *GLoader) SetStartOffset(dataCollection string, offset uint64) *GLoader
  • func (g *GLoader) SetWorkers(workers uint) *GLoader
  • func (g *GLoader) Src(name, dsn string) error
  • func (g *GLoader) Start() error
  • func (g *GLoader) StartWithContext(ctx context.Context) error
  • func (g *GLoader) Stats() *stats.Stats
  • func (g *GLoader) Stop()
• type MetricKey
  • func (m MetricKey) String() string
• type Reader
  • func NewReader(ctx context.Context, dataCollection string, buffer *data.Buffer, ...) *Reader
  • func (r *Reader) RunWorker(startOffset, endOffset, rowPerBatch uint64)
  • func (r *Reader) SetEndOffset(endOffset uint64)
  • func (r *Reader) SetRowsPerBatch(rowsPerBatch uint64)
  • func (r *Reader) SetStartOffset(startOffset uint64)
  • func (r *Reader) SetWorkers(workers uint)
  • func (r *Reader) Start() error
• type Writer
  • func NewWriter(ctx context.Context, dataCollection string, buffer *data.Buffer, ...) *Writer
  • func (w *Writer) RunWorker()
  • func (w *Writer) SetRowsPerBatch(rowsPerBatch uint64)
  • func (w *Writer) SetWorkers(workers uint)
  • func (w *Writer) Start() error

Constants

const (
	DefaultRowsPerBatch = 100
	DefaultWorkers      = 3
)

Variables

var (
	ErrBufferNotSet                 = errors.New("buffer not set")
	ErrConnectionPoolNotSet         = errors.New("connection pool not set")
	ErrDataMapNotSet                = errors.New("data map not set")
	ErrEndOffsetLessThanStartOffset = errors.New("end offset less than start offset")
	ErrEndOffsetRequired            = errors.New("end offset required")
	ErrSrcConnectionIsRequired      = errors.New("source connection is required")
	ErrDestConnectionIsRequired     = errors.New("destination connection is required")
)

var ErrCCCauseStopFuncCalled = errors.New("stop func called")

Functions

func NewStats

func NewStats() *stats.Stats

Types

type BufferObserverAdapter

type BufferObserverAdapter struct {
	// contains filtered or unexported fields
}

func NewBufferObserverAdapter

func NewBufferObserverAdapter(s *stats.Stats, dcName string) *BufferObserverAdapter

func (*BufferObserverAdapter) LengthChanged

func (b *BufferObserverAdapter) LengthChanged(l uint64)

func (*BufferObserverAdapter) Read

func (b *BufferObserverAdapter) Read(n int)

func (*BufferObserverAdapter) SizeChanged

func (b *BufferObserverAdapter) SizeChanged(size uint64)

func (*BufferObserverAdapter) Write

func (b *BufferObserverAdapter) Write(n int)

type GLoader

type GLoader struct {
	// contains filtered or unexported fields
}

func NewGLoader

func NewGLoader() *GLoader

func (*GLoader) Dest

func (g *GLoader) Dest(name, dsn string) error

func (*GLoader) Exclude

func (g *GLoader) Exclude(dataCollections ...string) *GLoader

func (*GLoader) Filter

func (g *GLoader) Filter(dataCollection, key string, condition driver.Condition, value string) *GLoader

func (*GLoader) FilterAll

func (g *GLoader) FilterAll(key string, condition driver.Condition, value string) *GLoader

func (*GLoader) GetDestDetails

func (g *GLoader) GetDestDetails(ctx context.Context) (driver.DatabaseDetail, error)

func (*GLoader) GetSrcDetails

func (g *GLoader) GetSrcDetails(ctx context.Context) (driver.DatabaseDetail, error)

func (*GLoader) Include

func (g *GLoader) Include(dataCollections ...string) *GLoader

func (*GLoader) OrderBy

func (g *GLoader) OrderBy(dataCollection, key string, direction driver.Direction) *GLoader

func (*GLoader) OrderByAll

func (g *GLoader) OrderByAll(key string, direction driver.Direction) *GLoader

func (*GLoader) SetEndOffset

func (g *GLoader) SetEndOffset(dataCollection string, offset uint64) *GLoader

func (*GLoader) SetRowsPerBatch

func (g *GLoader) SetRowsPerBatch(rowsPerBatch uint64) *GLoader

func (*GLoader) SetStartOffset

func (g *GLoader) SetStartOffset(dataCollection string, offset uint64) *GLoader

func (*GLoader) SetWorkers

func (g *GLoader) SetWorkers(workers uint) *GLoader

func (*GLoader) Src

func (g *GLoader) Src(name, dsn string) error

func (*GLoader) Start

func (g *GLoader) Start() error

func (*GLoader) StartWithContext

func (g *GLoader) StartWithContext(ctx context.Context) error

func (*GLoader) Stats

func (g *GLoader) Stats() *stats.Stats

func (*GLoader) Stop

func (g *GLoader) Stop()

type MetricKey

type MetricKey string

const (
	MetricBufferSizeBytes            MetricKey = "buffer.size.bytes"
	MetricBufferLengthRows           MetricKey = "buffer.length.rows"
	MetricBufferTotalWriteLengthRows MetricKey = "buffer.totalWriteLength.rows"
	MetricBufferTotalReadLengthRows  MetricKey = "buffer.totalReadLength.rows"
)

func (MetricKey) String

func (m MetricKey) String() string

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

func NewReader

func NewReader(ctx context.Context, dataCollection string, buffer *data.Buffer, dataMap *data.Map, connectionP *driver.ConnectionPool) *Reader

func (*Reader) RunWorker

func (r *Reader) RunWorker(startOffset, endOffset, rowPerBatch uint64)

func (*Reader) SetEndOffset

func (r *Reader) SetEndOffset(endOffset uint64)

func (*Reader) SetRowsPerBatch

func (r *Reader) SetRowsPerBatch(rowsPerBatch uint64)

func (*Reader) SetStartOffset

func (r *Reader) SetStartOffset(startOffset uint64)

func (*Reader) SetWorkers

func (r *Reader) SetWorkers(workers uint)

func (*Reader) Start

func (r *Reader) Start() error

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

func NewWriter

func NewWriter(ctx context.Context, dataCollection string, buffer *data.Buffer, connectionP *driver.ConnectionPool) *Writer

func (*Writer) RunWorker

func (w *Writer) RunWorker()

func (*Writer) SetRowsPerBatch

func (w *Writer) SetRowsPerBatch(rowsPerBatch uint64)

func (*Writer) SetWorkers

func (w *Writer) SetWorkers(workers uint)

func (*Writer) Start

func (w *Writer) Start() error