extractor

Documentation

Index

• type Checkpoint
• type Extractor
  • func NewExtractor(db database.Database, table, outputFile, format string, batchSize int64, ...) *Extractor
  • func (e *Extractor) Extract(ctx context.Context) error
  • func (e *Extractor) LoadCheckpoint() error
  • func (e *Extractor) SaveCheckpoint() error
• type ParquetWriter
  • func NewParquetWriter(file *os.File, columns []database.Column) ParquetWriter

Types

type Checkpoint

type Checkpoint struct {
	Table      string `json:"table"`
	LastOffset int64  `json:"last_offset"`
}

Checkpoint represents the extraction progress

type Extractor

type Extractor struct {
	// contains filtered or unexported fields
}

Extractor handles data extraction from a database table

func NewExtractor

func NewExtractor(db database.Database, table, outputFile, format string, batchSize int64, concurrency int) *Extractor

NewExtractor creates a new Extractor instance

func (*Extractor) Extract

func (e *Extractor) Extract(ctx context.Context) error

Extract extracts data from the database table

func (*Extractor) LoadCheckpoint

func (e *Extractor) LoadCheckpoint() error

LoadCheckpoint loads the last saved extraction progress

func (*Extractor) SaveCheckpoint

func (e *Extractor) SaveCheckpoint() error

SaveCheckpoint saves the current extraction progress

type ParquetWriter

type ParquetWriter interface {
	Write(values []interface{}) error
	WriteStop() error
}

ParquetWriter defines the interface for writing Parquet files

func NewParquetWriter

func NewParquetWriter(file *os.File, columns []database.Column) ParquetWriter

NewParquetWriter creates a new ParquetWriter instance