conduit-connector-protocol

README

Conduit Connector Protocol

Warning
If you want to implement a Conduit connector in Go, you should use the Connector SDK.

This repository contains the definition of the Conduit connector protocol in gRPC. It also contains a thin Go layer that hides the gRPC implementation details without adding any functionality on top.

This repository is the only connection point between Conduit and a connector connector.

Implementing a connector in Go

We provide a Connector SDK for writing connectors in Go. In this case you won't directly use the contents of this repository, instead the SDK hides implementation details and provides utilities to make developing a connector as simple as possible.

If you want to implement a connector in any other language you will need to generate the protocol code yourself, this is explained in the next chapter.

Implementing a connector in other languages

You can use buf to generate code for building a Conduit connector in virtually any major language. To do that you need to create a buf.gen.yaml file and configure the connectors for the language you want to use.

For example here is a buf.gen.yaml file that is configured to generate C++ and Java code:

version: v1
plugins:
  - name: cpp
    out: gen/proto/cpp
  - name: java
    out: gen/proto/java

Then you can run this command to generate the code:

buf generate buf.build/conduitio/conduit-connector-protocol --template buf.gen.yaml

At this point you should have everything you need to start developing a connector. Make sure to implement all gRPC services according to the documentation in the proto definition and to follow the go-plugin instructions about writing a plugin in a language other than Go.

Once the connector is ready you need to create an entrypoint file which Conduit can run to start the connector. In case of compiled languages that is the compiled binary, in case of scripted languages you can create a simple shell script that starts the connector. Here is an example for python:

#!/usr/bin/env python my-connector.py

To run your connector as part of a Conduit pipeline you can create it using the connectors API and specify the path to the compiled connector binary in the field plugin.

Here is an example request to POST /v1/connectors (find more about the Conduit API):

{
  "type": "TYPE_SOURCE",
  "plugin": "/path/to/compiled/connector/binary",
  "pipelineId": "...",
  "config": {
    "name": "my-connector",
    "settings": {
      "my-key": "my-value"
    }
  }
}

Local development

We are using buf remote generation of protobuf code. When developing locally we don't want to push a new version of the proto files every time we make a change, that's why in that case we can switch to locally generated protobuf code.

To switch to locally generated protobuf code follow the following steps:

• run cd proto && buf generate
• cd into the newly generated folder internal in the root of the project
• create a go.mod file by running go mod init go.buf.build/grpc/go/conduitio/conduit-connector-protocol
• cd into the root of the project and run go mod edit -replace go.buf.build/grpc/go/conduitio/conduit-connector-protocol=./internal

Don't forget to revert the replace directive in the go.mod file before pushing your changes!

Acknowledgments

We took inspiration for our connector implementation from hashicorp/terraform-plugin-go .