Package debeziumio contains cross-language functionality for using Debezium (http://kafka.apache.org/). These transforms only work on runners that support cross-language transforms.
Transforms specified here are cross-language transforms implemented in a different SDK (listed below). During pipeline construction, the Go SDK will need to connect to an expansion service containing information on these transforms in their native SDK.
To use an expansion service, it must be run as a separate process accessible during pipeline construction. The address of that process must be passed to the transforms in this package.
The version of the expansion service should match the version of the Beam SDK being used. For numbered releases of Beam, these expansions services are released to the Maven repository as modules. For development versions of Beam, it is recommended to build and run it from source using Gradle.
Current supported SDKs including expansion service modules * Java
- Vendored Module: beam-sdks-java-io-debezium-expansion-service
- Run via Gradle: ./gradlew :sdks:java:io:debezium:expansion-service:shadowJar java -jar <path-to-debezium-jar> <port>
- Reference Class: org.apache.beam.io.debezium.DebeziumIO
This section is empty.
This section is empty.
func ConnectionProperties ¶
func ConnectionProperties(cp string) readOption
ConnectionProperties specifies properties of the debezium connection passed as a string with format [propertyName=property;]*
func ExpansionAddr ¶
func ExpansionAddr(expansionAddr string) readOption
ExpansionAddr sets the expansion service address to use for DebeziumIO cross-langauage transform.
func MaxRecord(r int64) readOption
MaxRecord specifies maximum number of records to be fetched before stop.
func Read(s beam.Scope, username, password, host, port string, connectorClass DriverClassName, t reflect.Type, opts ...readOption) beam.PCollection
Read is an external PTransform which reads from Debezium and returns a JSON string. It requires the address of an expansion service for Debezium IO. If both the host and port address are provided as "", an appropriate expansion service will be automatically started; however this is slower than having a persistent expansion service running.
username := "debezium" password := "dbz" host := "localhost" port := "5432" connectorClass := debeziumIO.POSTGRESQL maxrecords := 1 debeziumio.Read(s.Scope("Read from debezium"), expansionAddr, username, password, host, port, connectorClass, reflectx.String, debeziumio.MaxRecord(maxrecords), debeziumio.ExpansionAddr("localhost:9000"))