cdc

README

SQLite-CDC

A change-data-capture engine for SQLite.

Overview

This project implements change-data-capture, or CDC, for SQLite databases. The current implementation works by installing triggers on target tables that record the before and after states of a row modification to a change log table. The engine then watches the SQLite files for changes, reads batches of change records, and then sends them to a configurable destination for processing.

Each entry in the log contains the following information:

• Timestamp of change
• Type of change (INSERT, UPDATE, or DELETE)
• Name of source table
• Before (when type is UPDATE or DELETE)
• After (when type is INSERT or UPDATE)

The before and after fields contain the values of all columns from the source table before and after the row was modified.

The engine can read up to a configurable number of records as a batch and deliver these changes to some processing logic or destination. These change records can then be used to build higher level replication systems such as read-only replicas of SQLite data in a networked database such as MySQL, client-side caches of data with automated invalidation, real-time ETL and data warehousing, or even more advanced systems such as bidirectional replication between independent databases.

Project Status

Consider this project in alpha or a work-in-progress state.

My original plan was to put this in production by the end of 2024 but I've missed that goal. As of writing, I've only tested the project on small scale, non-production workloads. I've also added some benchmarks to help set expectations or estimates around performance impacts but they aren't guaranteed to be accurate for production workloads. Use at your own risk.

If you end up doing your own testing or experimentation with this project then please let me know your results. I'd be grateful for any success or failure you can share.

Adding CDC To A Database

The included cmd/sqlite-cdc-setup command can be used to add or remove CDC support from a database. This command attaches triggers to a target set of tables that automatically populate a change log table named __cdc_log by default.

go run cmd/sqlite-cdc-setup/main.go --db test.sqlite --table my_table --setup
go run cmd/sqlite-cdc-setup/main.go --db test.sqlite --table my_table --teardown

Use the --help flag to get a summary of the options. For convenience:

Usage of sqlite-cdc-setup:
  -blobs
        Enable support for blobs
  -db string
        SQLite file path
  -db-params string
        SQLite connection parameters. See <https://pkg.go.dev/modernc.org/sqlite#Driver.Open> for parameter syntax (default "_pragma=journal_mode(wal)&_pragma=busy_timeout(5000)")
  -disable-subsec
        Disable subsecond time resolution to support old clients
  -log-table string
        Name of the table to store CDC log entries (default "__cdc_log")
  -setup
        Perform initial setup of the database for CDC before starting in any mode
  -table value
        A table name to monitor. Can be specified multiple times
  -teardown
        Perform teardown of the CDC tables and triggers. Setting the teardown flag prevents any other action. The process will perform the teardown and then exit
  -version
        Print version and exit

Running The Example Processor

The easiest way to test the changelog table and process the contents is to use the cmd/sqlite-cdc command. To see CDC events streamed to the console use something like:

go run cmd/sqlite-cdc/main.go --output json --db test.sqlite --table my_table --bootstrap --cdc

This will begin streaming JSON encoded change events to the console starting with an event for every existing row followed by events from the changelog table:

{
    "table":"my_table",
    "timestamp":"2024-05-05T19:03:11.5681173-05:00",
    "operation":"INSERT",
    "before":null,
    "after":{"email":"noreply@example.com","favorite_color":"blue","id":1,"username":"example"}
}

{
    "table":"my_table",
    "timestamp":"2024-05-06T00:03:34Z",
    "operation":"UPDATE",
    "before":{"email":"noreply@example.com","favorite_color":"blue","id":1,"username":"example"},
    "after":{"email":"noreply@example.com","favorite_color":"green","id":1,"username":"example"}
}

{
    "table":"my_table",
    "timestamp":"2024-05-06T00:04:07Z",
    "operation":"DELETE",
    "before":{"email":"noreply@example.com","favorite_color":"green","id":1,"username":"example"},
    "after":null
}

Use the --help flag to see all the available options. Here's a snapshot of what's available:

Usage of sqlite-cdc:
  -batch-size int
        The max number of log entries to collect in each batch (default 256)
  -blobs
        Enable support for blobs
  -bootstrap
        Read all existing records as if they are inserts and then exit. If this flag is set in addition to the cdc flag the cdc mode will begin after the bootstrap is complete
  -cdc
        Run a continuous extraction of the CDC log.
  -db string
        SQLite file path
  -db-params string
        SQLite connection parameters. See <https://pkg.go.dev/modernc.org/sqlite#Driver.Open> for parameter syntax (default "_pragma=journal_mode(wal)&_pragma=busy_timeout(5000)")
  -disable-subsec
        Disable subsecond time resolution to support old clients
  -log-table string
        Name of the table to store CDC log entries (default "__cdc_log")
  -output string
        Write destination for log entries. Valid options are - for simplified stdout, json for full JSON stdout, or an HTTP URL that will receive POST requests containing batches of log entries. See <pkg.go.dev/github.com/kevinconway/sqlite-cdc/handlers#HTTP> for more. (default "-")
  -table value
        A table name to monitor. Can be specified multiple times
  -version
        Print version and exit

Limits Of The Provided Processor

The example processor exists primarily for demonstrations and for quickly performing an initial test of the system. In most cases, I expect that you will use the example as a template for creating your own custom build that integrates your own event handling logic and better integrates with your runtime environment by, for example, adding instrumentation.

As is, the example processor can perform all the CDC responsibilities but supports only three outputs:

• Simplified logging to STDOUT (ex: <timestamp>: <table> <operation>)
• JSON logging to STDOUT (shown in the previous section)
• HTTP POST

If you absolutely must use the example as-is then the HTTP POST output enables you to redirect batches of CDC events to a specified URL. That receiver can then implement any logic you need. Note that the current version of the project does not include any form of built-in retries so your POST endpoint must implement those internally. See https://pkg.go.dev/github.com/kevinconway/sqlite-cdc/handlers#HTTP for details on the API contract.

Extending The Engine

Custom builds need to provide two components: an event handler and a database client.

The database client can be any SQLite driver for Go that implements the standard library databases/sql driver interface. See https://github.com/cvilsmeier/go-sqlite-bench for benchmarks covering most of the popular drivers available for Go.

The event handler is anything that implements:

type ChangesHandler interface {
	HandleChanges(ctx context.Context, changes Changes) error
}

Each call to HandleChanges receives a batch of change records. Each batch of changes given to your event handler will be in change order. The engine makes only one call to the handler at a time so that batches are also strictly processed in change order. If the handler returns an error then the engine considers this a critical fault and shuts down. If the handler returns success then the entire batch is considered successful and the relevant changes are removed from the log. Your handler is responsible for durability or reliability behaviors such as retries with backoff.

API Documentation

https://pkg.go.dev/github.com/kevinconway/sqlite-cdc

Limitations

BLOB Support

By default, the engine ignores all columns with a BLOB type. BLOB support can be enabled by using the WithBlobSupport(true) option when constructing the engine or the --blobs flag when using the example executable. However, BLOB support is limited even when enabled.

When enabled, all values in a BLOB typed column are converted to an upper-case, hexadecimal representation of the BLOB's value using the SQLite hex function. This is done to make the values representable in JSON because JSON has no native expression of raw bytes.

Note, however, that SQLite distinguishes between column type and data type. By default, column types are not enforced and SQLite is capable of storing any data type within any column. For example, a column may be defined with type INTEGER but that does not prevent a client from inserting values that are of type TEXT or BLOB.

This project cannot handle BLOB type data in a non-BLOB type column. I suggest using STRICT tables to avoid this situation.

Note also that larger blob values take longer to convert to hex and can result in poorer performance.

Very Wide Tables

The engine uses SQLite functions to produce JSON representations of a record. SQLite has a hard limit of 127 arguments per function call. This equates to a limit of 63 columns convertible per function call.

All rows from tables with less than 64 columns are converted to JSON in a single function call. Tables with 64 or more columns engage in an alternative conversion process. The wide table conversion process groups columns into sets of 63, generates a JSON object from each set, and performs a series of JSON patches to merge each column set with the previously generated JSON object.

Wide table conversion is limited to 1000 columns because this is the default stack depth limit for SQLite. Wider tables result in poorer performance.

WAL Mode Required

Running sqlite-cdc on your database necessarily adds reads and writes that are concurrent with your existing usage. Practically, enabling WAL mode is the only way for this to work.

Clients With Version Less Than 3.42.0

The log table contains a timestamp that represents the time of change. SQLite datetime resolution is second precision by default. Version 3.42.0 added a subsecond modifier to time functions that enables millisecond precision.

If any of your clients are older than 3.42.0 then you must use the WithoutSubsecondTime option when constructing an engine or the --disable-subsec flag in the example executable.

It's important to note that the version is associated with the client and not the database file. It is possible to have both old and new clients operating on the same database file. The lowest versioned client determines the limitations.

Clients With Version Less Than 3.38.0

Any clients older than 3.38.0 must be compiled with JSON support enabled. This is enabled by default on 3.38.0 and newer clients.

Why Not Use The Session API?

SQLite can be built with an extension called session that provides an API for recording database changes and formatting them into change sets. The resulting change sets contain very similar information to the current design of the change log that this project uses.

On the surface, the sessions extension appears to be a great fit for CDC. Starting a session begins a capture of all changes to a select set of tables. Each change record describes modifications to a table row. Those change records accumulate in a log that can be fetched at any time. This is conceptually the same as what this project does.

However, the sessions extension has three notable behaviors that make it the wrong choice for general purpose CDC. The first is that session change records are only materialized when reading the log and only one change is generated per row that was modified. As a result, sessions are a poor way to receive real-time change notifications and are only capable of producing the final state of a record rather than a series of interim changes. The second behavior is that the session log can only be fetched as a whole but never resets. As a result, the session log grows unbounded and there is no mechanism to filter out already handled changes. The third behavior is that a session is not necessarily persisted beyond the lifetime of the database connection it's associated with. As a result, there is no way to restore CDC from the last known position if the system exited for any reason.

The sessions extension appears to be designed to operate in a specific kind of environment where SQLite is being used as a data structure rather than a typical SQL database. Sessions work best when more tightly integrated into application logic and aren't well suited for this project's "bolt-on" or sidecar model.

Performance Impacts Of The CDC Triggers

I don't yet have any production performance metrics to share. Until then, the code includes benchmarks that attempt to measure the performance impacts of the change capture triggers and the different ways they manage or encode data. The benchmarks are run with the following pragmas:

• journal_mode(wal)
• busy_timeout(5000)
• synchronous(normal)
• foreign_keys(on)

You can run the suite of benchmarks using either standard Go tooling or the included make benchmarks rule. The Makefile rule generates a set of benchmark outputs in .build/.bench/ that contain 20 runs of each benchmark. The make benchmark-reports rule will generate a comparative analysis of the contents of each output file as a CSV. Note that the full suite of benchmarks can take potentially more than an hour to complete.

The results below are based on running the benchmarks on a local workstation. The workstation was not dedicated to the benchmarks and subject to jitter. I've included the percent differences but not the absolute values because the absolute values will differ based on hardware but the percent differences should be somewhat consistent across different hardware configurations. For a sense of scale, the absolute values for individual operations without triggers, in my test runs, were in the magnitude of 10s of microseconds.

My personal interpretation of the overall results is that the CDC triggers add marginal write performance overhead for the majority of use cases.

Simple Tables

The "simple" table benchmarks are run without concurrency and compare the cost of insert, update, and delete statements. The tables used in these benchmarks are defined with all integer columns, an integer primary key, and without ROWID. The number of columns never exceeds 63 which is the maximum count that can be converted to a CDC record by the triggers in a single step.

Columns	Insert (% Difference)	Update (% Difference)	Delete (% Difference)
2	97%	100%	113%
4	96%	96%	105%
8	93%	99%	111%
16	99%	111%	127%
32	106%	158%	153%
64	105%	179%	203%

For most table sizes the average overhead appears to be around 100% compared to a single operation which matches the expectations of the triggers adding an additional insert for each modifying operation. The overhead of the JSON encoding of the data appears minimal for small table but grows with column count.

Wide Tables

The wide table benchmarks are the same as the simple table benchmarks but the tables are always larger than 63 columns. This engages a workaround for the max SQLite function parameter limit which is 127. To generate JSON objects with more than 63 key/value pairs the system computes objects in batches of up to 63 columns and then merges the results using the json_patch function. The performance impacts grow relative to the number of objects that must be merged. Here's a table and graph illustrating the performance impacts of wide tables:

Columns	Insert (% Difference)	Update (% Difference)	Delete (% Difference)
64	119%	225%	251%
128	195%	335%	556%
256	412%	696%	1,434%
512	1,011%	1,558%	4,948%
1000	3,263%	3,872%	33,814%

The overhead for wide tables appears to grow linearly with the number of objects that must be merged with the exception of deletes with 1000 columns. I don't have an explanation for that discontinuity yet.

BLOB Data

All BLOB data must be encoded to be JSON compatible. The impact of the encoding process grows with the size of the BLOB. Here's a table and graph showing the growth of encoding time as the byte size increases:

Blob Size	Percent Slower Than 16 Bytes
16	0%
64	67%
256	64%
1024	69%
4096	74%
16384	85%
32768	102%
65536	126%
131072	187%
262144	287%
524288	510%
1048576	937%

For a sense of scale, the encoding of 16 bytes took an average of about 46 microseconds on my particular workstation.

Compatibility With Other Replication Tools

I have not yet tested with any of the SQLite replication tools I know about:

• https://github.com/superfly/litefs
• https://github.com/benbjohnson/litestream
• https://github.com/maxpert/marmot
• https://github.com/rqlite/rqlite

This tool should be compatible with any SQLite implementation or modification that supports using a standard SQLite client. This means that rqlite is not compatible because it requires all database writes to happen through an HTTP API rather than a SQLite client. SQLite derivatives that work through a custom filesystem, such as litefs may work but likely cannot rely on the integrated filesystem watcher to detect SQLite file changes. The workaround is to use the time interval based polling for changes that is included.

The litestream project should be compatible because it uses a standard filesystem and supports arbitrary SQLite clients. I also suspect marmot is compatible but redundant because it implements a very similar trigger based system to this project.

Developing

This project only requires a Go installation to work on and is compatible with standard Go tooling. For example, you can run go test ./... to run the tests.

For convenience, I've included a devcontainer configuration and a Makefile with the following rules:

• build - Create executables for both CLIs
• test - Run all test suites
• test/lint - Run static analysis
• test/unit - Run unit tests
• test/coverage - Generate a coverage report
• benchmarks - Run all benchmarks (NOTE: This can take a long time)
• benchmark-reports - Generate comparison reports from benchmarks
• tools - Download any Go tooling used for project automation
• tools/update - Update all tools to the latest version
• fmt - Run the project's auto formatter
• clean - Remove all build and test artifacts
• clean/test - Remove test artifacts and coverage reports
• clean/tools - Remove any downloaded tools

License

The code for this project is licensed under the Apache 2.0 terms.

See LICENSE for full details.

Prior Art

This project is inspired by https://github.com/JayJamieson/sqlite-cdc/ and https://github.com/maxpert/marmot.

Documentation

Index

• type CDC
  • func NewTriggerEngine(db *sql.DB, handler ChangesHandler, tables []string, options ...Option) (CDC, error)
• type Change
  • func (c Change) String() string
• type Changes
• type ChangesHandler
• type ChangesHandlerFunc
  • func (fn ChangesHandlerFunc) HandleChanges(ctx context.Context, changes Changes) error
• type Engine
• type Installer
• type Operation
• type Option
  • func WithBlobSupport(v bool) Option
  • func WithLogTableName(name string) Option
  • func WithMaxBatchSize(size int) Option
  • func WithSignal(signal Signal) Option
  • func WithoutSubsecondTime(v bool) Option
• type Signal
  • func NewChannelSignal(input <-chan SignalEvent) (Signal, error)
  • func NewFSNotifySignal(db *sql.DB) (Signal, error)
  • func NewMultiSignal(signals ...Signal) (Signal, error)
  • func NewTimeSignal(interval time.Duration) (Signal, error)
• type SignalEvent
• type TriggerEngine
  • func (c *TriggerEngine) Bootstrap(ctx context.Context) error
  • func (c *TriggerEngine) BootstrapAndCDC(ctx context.Context) error
  • func (c *TriggerEngine) CDC(ctx context.Context) error
  • func (c *TriggerEngine) Close(ctx context.Context) error
  • func (c *TriggerEngine) Setup(ctx context.Context) error
  • func (c *TriggerEngine) Teardown(ctx context.Context) error

Types

type CDC

type CDC interface {
	Installer
	Engine
}

CDC represents a complete implementation of a CDC for SQLite.

Installer methods should not be called if any Engine methods are running. Only one Engine method may be called per instance of CDC with the exception of Close which may be called at any time to end a running operation. The instance is no longer valid after an Engine method completes.

func NewTriggerEngine

func NewTriggerEngine(db *sql.DB, handler ChangesHandler, tables []string, options ...Option) (CDC, error)

NewTriggerEngine returns a CDC implementation based on table triggers.

This implementation works with any SQLite driver and uses only SQL operations to implement CDC. For each specified table to monitor, the implementation creates triggers for AFTER INSERT, AFTER UPDATE, and AFTER DELETE. These triggers populate a log table, named __cdc_log by default. The log table entries contain effectively identical information as the Change struct.

The before and after images are stored as JSON objects in the log table. The JSON objects are generated from the column names and values in the table.

See the TriggerEngine documentation for more details.

type Change

type Change struct {
	Table     string          `json:"table"`
	Timestamp time.Time       `json:"timestamp"`
	Operation Operation       `json:"operation"`
	Before    json.RawMessage `json:"before"`
	After     json.RawMessage `json:"after"`
}

func (Change) String

func (c Change) String() string

type Changes

type Changes []Change

type ChangesHandler

type ChangesHandler interface {
	HandleChanges(ctx context.Context, changes Changes) error
}

ChangesHandler implementations are given batches of database changes. Each batch of changes is in the same order as the CDC log table. Batches may contain changes from multiple tables.

If a handler returns an error then the entire batch is considered failed and retried. If a handler returns nil then the entire batch is considered successful and the relevant entries are removed from the CDC log table.

type ChangesHandlerFunc

type ChangesHandlerFunc func(ctx context.Context, changes Changes) error

ChangesHandlerFunc is an adaptor to allow the use of ordinary functions as ChangesHandler implementations. Note that you should not use this type directly and should instead always target the ChangesHandler type. For example, the appropriate use of this adaptor is:

var handler ChangesHandler = ChangesHandlerFunc(func(changes Changes) error {
	// handle changes
})

func (ChangesHandlerFunc) HandleChanges

func (fn ChangesHandlerFunc) HandleChanges(ctx context.Context, changes Changes) error

type Engine

type Engine interface {
	// CDC-only mode processes changes made to the database.
	//
	// This mode only operates on changes that have been captured by the CDC
	// implementation and does not process any unchanged records. This means
	// that CDC-only mode does not process existing, unmodified records like
	// Bootstrap does.
	//
	// This mode runs until stopped or it encounters an error.
	CDC(ctx context.Context) error
	// Bootstrap-only mode processes all existing records in the database.
	//
	// This mode only operates on the current state of existing records and does
	// not process any captured changes. All existing records are processed as
	// though they are INSERT operations. For convenience, the first change
	// handled by bootstrap mode for any table is always a BOOTSTRAP operation
	// with an empty before and after image. This signal may be used in systems
	// that manage state based on the stream of captured changes to indicate
	// that the previously accumulated state is likely invalid and must be
	// re-built from the bootstrap data.
	//
	// This mode runs until it completes a scan of each table and then exits.
	Bootstrap(ctx context.Context) error
	// Bootstrap-and-CDC mode is a combination of the bootstrap and CDC modes.
	//
	// This mode starts with a bootstrap and then enters CDC mode once it is
	// complete. Any changes to data during hte bootstrap are captured and
	// emitted once the engine enters CDC mode.
	//
	// This mode runs until stopped or it encounters an error.
	BootstrapAndCDC(ctx context.Context) error
	// Stop any ongoing CDC operations and shut down.
	Close(ctx context.Context) error
}

Engine represents the change handling portion of a CDC implementation.

Only one of these methods may be called per instance of CDC with the exception of Close which may be called at any time to end a running operation. Once any of these methods returns then the instance is no longer valid.

type Installer

type Installer interface {
	// Perform any setup necessary to support CDC.
	Setup(ctx context.Context) error
	// Perform any teardown necessary to remove CDC.
	Teardown(ctx context.Context) error
}

Installer is an optional set of methods that a CDC implementation may offer.

These methods must be present but may be no-ops if an implementation has no setup or teardown requirements. Generally, the setup method should be called before any Engine methods.

type Operation

type Operation string

const (
	Insert    Operation = "INSERT"
	Update    Operation = "UPDATE"
	Delete    Operation = "DELETE"
	Bootstrap Operation = "BOOTSTRAP"
)

type Option

type Option func(*TriggerEngine) error

func WithBlobSupport

func WithBlobSupport(v bool) Option

WithBlobSupport can enable or disable the storage of BLOB columns in the log table. This defaults to false because of the performance impacts of encoding BLOB type data.

func WithLogTableName

func WithLogTableName(name string) Option

WithLogTableName specifies the name of the log table. This defaults to __cdc_log but may be customized if needed.

func WithMaxBatchSize

func WithMaxBatchSize(size int) Option

WithMaxBatchSize specifies the maximum number of changes to process in a single batch. This defaults to 50.

func WithSignal

func WithSignal(signal Signal) Option

WithSignal installs a custom awakening signal that triggers the inspection of the log table when in CDC mode. The default signal is a combination of a filesystem watcher that signals when the SQLite files have changed and a 250ms timer used as a backstop for any missed filesystem events.

func WithoutSubsecondTime

func WithoutSubsecondTime(v bool) Option

WithoutSubsecondTime can disable the use of subsecond timestamps in the log table. This is only needed for old versions of SQLite and should be avoided otherwise.

type Signal

type Signal interface {
	// Waker returns a channel that will receive wake signals. If the channel
	// is closed by the Signal then the Signal has entered a terminal state.
	Waker() <-chan SignalEvent
	// Start implementations establish any state required to generate wake
	// signals. Long running tasks, such as continuous monitoring, should be
	// started in goroutines so that this method does not block.
	Start(ctx context.Context) error
	// Close releases any resources used by the Signal including any goroutines
	// started. Once a Signal is closed then it is in a terminal state and
	// cannot be re-used.
	Close() error
}

Signal implementations tell a CDC engine when to awaken while in CDC mode in order to process new changes.

func NewChannelSignal

func NewChannelSignal(input <-chan SignalEvent) (Signal, error)

NewChannelSignal returns a Signal implementation that proxies events from the given channel to the waker channel. This is useful for awakening based on external triggers.

func NewFSNotifySignal

func NewFSNotifySignal(db *sql.DB) (Signal, error)

NewFSNotifySignal creates a new Signal implementation that uses filesystem notifications to detect changes. This implementation uses the given database client to determine the path of the main SQLite database as well as any supplemental files such as the WAL when in WAL mode.

func NewMultiSignal

func NewMultiSignal(signals ...Signal) (Signal, error)

NewMultiSignal returns a Signal implementation that combines multiple signals into a single channel.

func NewTimeSignal

func NewTimeSignal(interval time.Duration) (Signal, error)

NewTimeSignal returns a new time-based Signal that emits a wake event on an interval. This can be used in environments where the file watching implementation is unsupported or unreliable.

type SignalEvent

type SignalEvent struct {
	Wake bool
	Err  error
}

SignalEvent indicates that a Signal has been triggered.

An engine should only check for changes if the Wake field is true. A non-nil error represents a terminal error for the Signal.

type TriggerEngine

type TriggerEngine struct {
	// contains filtered or unexported fields
}

TriggerEngine implements CDC using table triggers.

This implementation targets a specified set of tables and captures changes by using AFTER triggers to populate a change log table. The setup and teardown methods manage both the triggers and the log table. Currently, all target tables must be set up and torn down together and cannot be targeted individually.

The bootstrap mode is implemented by selecting batches of records from target tables. These are passed through to the bound ChangesHandler as they are selected. Each table bootstrap begins with the specified BOOTSTRAP operation event. Because this implementation of CDC uses table triggers and a persistent chang log table, a bootstrap is usually only needed once after running setup. If your system encounters a critical fault and needs to rebuild state from a bootstrap then you can safely run bootstrap again. However, subsequent runs of bootstrap mode do not clear the change log table.

The cdc mode is implemented by selecting batches of records from the change log table. The order of the log selection matches the natural sort order of the table which, itself, matches the order in which changes were made to the data. The frequency with which cdc mode checks for changes is determined by the bound Signal implementation. The default signal is a combination of a filesystem watcher and a time based interval. The filesystem watcher detects changes to the underlying SQLite files with the intent to handle changes as quickly as possible once they are persisted. However, the filesystem watcher is not infallible so a time based interval signal is included as a backstop. Generally, you are recommended to have some form of time based interval signal to augment any other signal choices.

By default, all change log entries are recorded with a millisecond precision timestamp. This precision is only available in SQLite 3.42.0 and later. If any system accessing the SQLite database is older than 3.42.0 then you must disable the subsecond timestamp with the WithoutSubsecondTime option.

By default, support for BLOB data is disabled and BLOB type columns are not included in change log records due to the performance impacts of encoding BLOB type data. If you need to handle BLOB type data then you must enable BLOB support with the WithBlobSupport option. Note, however, that this implementations identification of BLOB data is based on the declared column type and not the underlying data type. Any BLOB data in a non-BLOB column will cause a fault in this implementation. You are strongly recommended to use STRICT tables to avoid accidental BLOB data in a non-BLOB column.

func (*TriggerEngine) Bootstrap

func (c *TriggerEngine) Bootstrap(ctx context.Context) error

func (*TriggerEngine) BootstrapAndCDC

func (c *TriggerEngine) BootstrapAndCDC(ctx context.Context) error

func (*TriggerEngine) CDC

func (c *TriggerEngine) CDC(ctx context.Context) error

func (*TriggerEngine) Close

func (c *TriggerEngine) Close(ctx context.Context) error

func (*TriggerEngine) Setup

func (c *TriggerEngine) Setup(ctx context.Context) error

func (*TriggerEngine) Teardown

func (c *TriggerEngine) Teardown(ctx context.Context) error