README
¶
cdc-sink
cdc-sink is a toolkit for consuming a variety of logical replication feeds and inserting the data into a destination CockroachDB cluster.
This is a prototype and is not supported by Cockroach Labs except by prior arrangement.
Use of this tool is entirely at your own risk and Cockroach Labs makes no guarantees or warranties about its operation.
Documentation
¶
There is no documentation for this package.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
internal
|
|
|
cmd/dumphelp
Package dumphelp contains a hidden command to write the help strings for all files to an output directory.
|
Package dumphelp contains a hidden command to write the help strings for all files to an output directory. |
|
cmd/dumptemplates
Package dumptemplates contains a hidden command that will write the apply templates to disk.
|
Package dumptemplates contains a hidden command that will write the apply templates to disk. |
|
cmd/kafka
Package kafka contains a command to perform logical replication from a kafka feed.
|
Package kafka contains a command to perform logical replication from a kafka feed. |
|
cmd/licenses
Package licenses bundles the licence files for those go modules which are reachable from the main entry point.
|
Package licenses bundles the licence files for those go modules which are reachable from the main entry point. |
|
cmd/mkjwt
Package mkjwt contains a command to generate a signed JWT token from a user-provided private key.
|
Package mkjwt contains a command to generate a signed JWT token from a user-provided private key. |
|
cmd/mylogical
Package mylogical contains a command to perform logical replication from a mysql source server.
|
Package mylogical contains a command to perform logical replication from a mysql source server. |
|
cmd/pglogical
Package pglogical contains a command to perform logical replication from a PostgreSQL source server.
|
Package pglogical contains a command to perform logical replication from a PostgreSQL source server. |
|
cmd/preflight
Package preflight contains a command to assist with testing database connections.
|
Package preflight contains a command to assist with testing database connections. |
|
cmd/start
Package start contains the command to start the server.
|
Package start contains the command to start the server. |
|
cmd/version
Package version contains a command to print the build's bill-of-materials.
|
Package version contains a command to print the build's bill-of-materials. |
|
script
Package script contains support for loading configuration scripts built as JavaScript programs.
|
Package script contains support for loading configuration scripts built as JavaScript programs. |
|
sequencer
Package sequencer contains a number of sub-packages that implement various strategies for applying staged mutations.
|
Package sequencer contains a number of sub-packages that implement various strategies for applying staged mutations. |
|
sequencer/besteffort
Package besteffort contains a best-effort implementation of [types.MultiAcceptor].
|
Package besteffort contains a best-effort implementation of [types.MultiAcceptor]. |
|
sequencer/chaos
Package chaos allows errors to be introduced in a sequencer stack.
|
Package chaos allows errors to be introduced in a sequencer stack. |
|
sequencer/core
Package core contains a sequencer implementation that preserves source transactions and relative timestamps.
|
Package core contains a sequencer implementation that preserves source transactions and relative timestamps. |
|
sequencer/immediate
Package immediate contains a trivial [sequencer.Sequencer] implementation which writes data directly to the configured acceptor.
|
Package immediate contains a trivial [sequencer.Sequencer] implementation which writes data directly to the configured acceptor. |
|
sequencer/retire
Package retire contains a utility for removing old mutations.
|
Package retire contains a utility for removing old mutations. |
|
sequencer/scheduler
Package scheduler contains Sequencer-specific utilities for ensuring ordered access to rows.
|
Package scheduler contains Sequencer-specific utilities for ensuring ordered access to rows. |
|
sequencer/script
Package script provides an API shim that integrates the userscript into the [sequencer.Sequencer] API.
|
Package script provides an API shim that integrates the userscript into the [sequencer.Sequencer] API. |
|
sequencer/seqtest
Package seqtest provides a test fixture for instantiating sequencers and other general-purpose test helpers.
|
Package seqtest provides a test fixture for instantiating sequencers and other general-purpose test helpers. |
|
sequencer/sequtil
Package sequtil contains sequencer utility methods.
|
Package sequtil contains sequencer utility methods. |
|
sequencer/switcher
Package switcher contains a meta-Sequencer that switches between various modes of operation.
|
Package switcher contains a meta-Sequencer that switches between various modes of operation. |
|
sinkprod
Package sinkprod contains configuration and providers for connecting to production database(s).
|
Package sinkprod contains configuration and providers for connecting to production database(s). |
|
sinktest
Package sinktest contains utility types for writing cdc-sink tests.
|
Package sinktest contains utility types for writing cdc-sink tests. |
|
sinktest/all
Package all contains a test rig for all services.
|
Package all contains a test rig for all services. |
|
sinktest/base
Package base provides enough functionality to connect to a database, but does not provide any other services.
|
Package base provides enough functionality to connect to a database, but does not provide any other services. |
|
sinktest/mutations
Package mutations contains a helper to generate mutations.
|
Package mutations contains a helper to generate mutations. |
|
sinktest/recorder
Package recorder contains an acceptor implementation that records the method calls it sees.
|
Package recorder contains an acceptor implementation that records the method calls it sees. |
|
sinktest/scripttest
Package scripttest contains reusable test helpers for logical-replication tests.
|
Package scripttest contains reusable test helpers for logical-replication tests. |
|
source/cdc
Package cdc contains a http.Handler which can receive webhook events from a CockroachDB CDC changefeed.
|
Package cdc contains a http.Handler which can receive webhook events from a CockroachDB CDC changefeed. |
|
source/cdc/server
Package server contains an HTTP server that installs the CDC listener.
|
Package server contains an HTTP server that installs the CDC listener. |
|
source/kafka
Package kafka contains receives CockroachDB CDC changefeed events that are routed via a kafka cluster.
|
Package kafka contains receives CockroachDB CDC changefeed events that are routed via a kafka cluster. |
|
source/mylogical
Package mylogical contains support for reading a mySQL logical replication feed.
|
Package mylogical contains support for reading a mySQL logical replication feed. |
|
source/pglogical
Package pglogical contains support for reading a PostgreSQL logical replication feed.
|
Package pglogical contains support for reading a PostgreSQL logical replication feed. |
|
staging
Package staging contains all services which interact with the staging database.
|
Package staging contains all services which interact with the staging database. |
|
staging/checkpoint
Package checkpoint contains a utility for persisting checkpoint (fka resolved) timestamps.
|
Package checkpoint contains a utility for persisting checkpoint (fka resolved) timestamps. |
|
staging/leases
Package leases coordinates global, singleton activities.
|
Package leases coordinates global, singleton activities. |
|
staging/memo
Package memo implements a simple kv store with string keys.
|
Package memo implements a simple kv store with string keys. |
|
staging/stage
Package stage defines a means of storing and retrieving mutations to be applied to a table.
|
Package stage defines a means of storing and retrieving mutations to be applied to a table. |
|
staging/version
Package version is used to track incompatible schema changes within the staging tables.
|
Package version is used to track incompatible schema changes within the staging tables. |
|
target
Package target contains various services for performing operations on the target database cluster.
|
Package target contains various services for performing operations on the target database cluster. |
|
target/apply
Package apply contains code for applying mutations to tables.
|
Package apply contains code for applying mutations to tables. |
|
target/dlq
Package dlq writes unhandled mutations to dead-letter queues in the target database.
|
Package dlq writes unhandled mutations to dead-letter queues in the target database. |
|
target/schemawatch
Package schemawatch contains code to allow the schema of a target database to be queried and monitored.
|
Package schemawatch contains code to allow the schema of a target database to be queried and monitored. |
|
types
Package types contains data types and interfaces that define the major functional blocks of code within cdc-sink.
|
Package types contains data types and interfaces that define the major functional blocks of code within cdc-sink. |
|
util/applycfg
Package applycfg contains code for persisting applier configurations.
|
Package applycfg contains code for persisting applier configurations. |
|
util/auth/broken
Package broken contains an Authenticator that always fails.
|
Package broken contains an Authenticator that always fails. |
|
util/auth/jwt
Package jwt contains support for using JWT tokens to authenticate and authorize incoming connections.
|
Package jwt contains support for using JWT tokens to authenticate and authorize incoming connections. |
|
util/auth/reject
Package reject contains a types.Authenticator which always returns false.
|
Package reject contains a types.Authenticator which always returns false. |
|
util/auth/trust
Package trust contains a types.Authenticator which always returns true.
|
Package trust contains a types.Authenticator which always returns true. |
|
util/batches
Package batches contains support code for working with and testing batches of data.
|
Package batches contains support code for working with and testing batches of data. |
|
util/cmap
Package cmap contains an implementation of a canonicalizing map.
|
Package cmap contains an implementation of a canonicalizing map. |
|
util/crep
Package crep ("see-rep") contains a utility for producing a Canonical REPresentation of a value type and for comparing objects using a fuzzy equivalence relationship.
|
Package crep ("see-rep") contains a utility for producing a Canonical REPresentation of a value type and for comparing objects using a fuzzy equivalence relationship. |
|
util/diag
Package diag contains a mechanism for cdc-sink components to report structured diagnostic information.
|
Package diag contains a mechanism for cdc-sink components to report structured diagnostic information. |
|
util/fscopy
Package fscopy contains a utility for copying the contents of an [fs.FS] into the OS filesystem.
|
Package fscopy contains a utility for copying the contents of an [fs.FS] into the OS filesystem. |
|
util/hlc
Package hlc contains a trivial representation of CockroachDB's hybrid logical clock timestamp.
|
Package hlc contains a trivial representation of CockroachDB's hybrid logical clock timestamp. |
|
util/httpauth
Package httpauth contains a common function for extracting credentials from an HTTP request.
|
Package httpauth contains a common function for extracting credentials from an HTTP request. |
|
util/ident
Package ident contains types for safely representing SQL identifiers.
|
Package ident contains types for safely representing SQL identifiers. |
|
util/lockset
Package lockset contains a utility type that orders access to multiple resources.
|
Package lockset contains a utility type that orders access to multiple resources. |
|
util/logfmt
Package logfmt adds additional details to log messages with errors.
|
Package logfmt adds additional details to log messages with errors. |
|
util/merge
Package merge provides support for three-way merge operations.
|
Package merge provides support for three-way merge operations. |
|
util/metrics
Package metrics contains some common utility functions for constructing performance-monitoring metrics.
|
Package metrics contains some common utility functions for constructing performance-monitoring metrics. |
|
util/msort
Package msort contains utility functions for sorting and de-duplicating batches of mutations.
|
Package msort contains utility functions for sorting and de-duplicating batches of mutations. |
|
util/notify
Package notify contains utility code for notification behaviors.
|
Package notify contains utility code for notification behaviors. |
|
util/pjson
Package pjson implements a parallelized wrapper for encoding and decoding json objects.
|
Package pjson implements a parallelized wrapper for encoding and decoding json objects. |
|
util/retry
Package retry contains utility code for retrying database transactions.
|
Package retry contains utility code for retrying database transactions. |
|
util/secure
Package secure provides utilities to configure secure transport.
|
Package secure provides utilities to configure secure transport. |
|
util/serial
Package serial allows an otherwise-concurrent use of a database pool to be transparently deoptimized into serial use of a single transaction.
|
Package serial allows an otherwise-concurrent use of a database pool to be transparently deoptimized into serial use of a single transaction. |
|
util/stamp
Package stamp contains a utility for maintaining certain ordering invariants when queuing (time-)stamped values.
|
Package stamp contains a utility for maintaining certain ordering invariants when queuing (time-)stamped values. |
|
util/stdlogical
Package stdlogical contains a template for building a standard logical-replication CLI command.
|
Package stdlogical contains a template for building a standard logical-replication CLI command. |
|
util/stdpool
Package stdpool creates standardized database connection pools.
|
Package stdpool creates standardized database connection pools. |
|
util/stdserver
Package stdserver contains a generic HTTP server that can be used by sources that receive http requests.
|
Package stdserver contains a generic HTTP server that can be used by sources that receive http requests. |
|
util/stmtcache
Package stmtcache provides a cache for prepared statements.
|
Package stmtcache provides a cache for prepared statements. |
|
util/stopper
Package stopper contains a utility class for gracefully terminating long-running processes.
|
Package stopper contains a utility class for gracefully terminating long-running processes. |
|
util/stopvar
Package stopvar contains helpers that build on both the stopper and notify packages.
|
Package stopvar contains helpers that build on both the stopper and notify packages. |
|
util/subfs
Package subfs contains as [fs.FS] that performs string substitutions.
|
Package subfs contains as [fs.FS] that performs string substitutions. |
|
util/txguard
Package txguard provides a utility class that keeps a database transaction active on a periodic basis.
|
Package txguard provides a utility class that keeps a database transaction active on a periodic basis. |
|
util/workgroup
Package workgroup contains a concurrency-control utility.
|
Package workgroup contains a concurrency-control utility. |
Click to show internal directories.
Click to hide internal directories.