example-buf-rpc-service

README

Go gRPC/ConnectRPC Service using Buf (v2)

Annotated example of gRPC/ConnectRPC service implementations in Go, using Buf (v2 config) for code generation and .proto dependency management

• buf.yaml contains single-module Buf workspace configuration, it's also where you specify BSR module dependencies for your modules (Buf shares deps list for all modules in a single workspace)

• buf.gen.yaml configures code generation. This example uses Buf Managed Mode feature. Managed Mode allows removal of language-specific annotations from .proto files (for example: go_package, java_package) and supply it through configuration instead

  • Initially, Managed Mode was a source of confusion for me when trying to use Buf for Go code generation. I hope I can provide more clarifications for anyone that might feel the same. Buf Managed Mode for Go

• proto folder contains Protobuf sources, written .proto with Managed Mode configuration in mind, it lacks option go_package that would be expected to generate Protobuf source in Go

• api folder contains generated Go code from Protobuf compilations

• pkg ConnectRPC and gRPC implementations

Buf Managed Mode for Go (go_package_prefix)

I wish there is simpler and shorter way to describe this problem better, but I can only explain it thoroughly and using example

To compile .proto into Go code, ALL .proto file MUST provide Go package's import. Traditionally, you specify this by using option go_package annotation inside .proto file.

Here is an example taken from google/type/datetime.proto that is authored WITH language-specific annotations.

// from: google/type/datetime.proto

// ...

package google.type;
import "google/protobuf/duration.proto";

// ...
option go_package = "google.golang.org/genproto/googleapis/type/datetime;datetime";
option java_multiple_files = true;
option java_outer_classname = "DateTimeProto";
option java_package = "com.google.type";
option objc_class_prefix = "GTP";

message DateTime {
    //...
}

We can observe how this go_package annotation being used in event of DateTime reuse (as dependency)

// file: proto/task/v1/task.proto

import "google/type/datetime.proto";

message Task {
    string id = 1;
    string title = 2;
    bool completed = 3;
    google.type.DateTime created_at = 4;
    google.type.DateTime updated_at = 5;
    google.type.DateTime deleted_at = 6;
}

If we look at the generated Go code, it includes import directive of that go_package specified in google/protobuf/datetime.photo to use DateTime struct as dependency.

// file: api/task/v1/task.pb.go
import (
  // This import match `go_package`
	datetime "google.golang.org/genproto/googleapis/type/datetime"
  // the rest of import
)

This approach is relatively okay from external .proto consumer side. Dependency resolution for Go becomes simple because it isolates transitive dependencies by simply using that Go package. Once you generate task.pb.go you just need to call go mod tidy after to resolve such package.

However, if you're the public .proto author, you have to maintain a public Go module/package if you don't want to force your .proto consumers to complicate their dependency management.

Buf Managed Mode aims to promote "consumer/language-agnostic" .proto definitions by removing language-specific annotations such as go_package from Protobuf sources to make it more reusable any consumer (e.g. someone outside org) and let consumers set their own go_package value depending on their project.

The task.proto definition in this project is authored WITHOUT any go_package or other language-specific annotations. But remember, go_package value MUST be provided for all .proto files for compiler to generate Go code, so we turn to buf.gen.yaml to tell the compiler how to determine go_package value for all .proto files.

My confusion with Managed Mode initially came from the assumption that Buf would have set default behavior to generate Go code by simply turning Managed Mode on.

# Wrong assumption: this should work...
managed:
  enabled: true

For Go specifically, Buf doesn't have enough default behavior preconfigured that would allow us to start generating Go code. You MUST still provide additional configuration in buf.gen.yaml. Confusingly, this configuration must be provided via override option

managed:
  enabled: true
  override:
    - file_option: go_package_prefix
      value: github.com/nadhifikbarw/example-buf-rpc-service/api

When you set go_package_prefix "override". go_package value for all .proto files would follow the rules of:

<prefix>/<proto_package_parts>

For example, my prefix is github.com/nadhifikbarw/example-buf-rpc-service/api so task.proto with package value task.v1 would be equivalent to setting:

option go_package = "github.com/nadhifikbarw/example-buf-rpc-service/api/task/v1";

You can disable Managed Mode behavior for certain .proto files (check config reference for granularity). For some cases, you sometimes SHOULD disable.

For example, I want to use Google generated protobuf types (therefore respecting go_package annotation in google/type/datetime.proto). This way, my generated task.pb.go keeps google.golang.org/genproto/googleapis/type/datetime import.

managed:
  enabled: true
  override:
    - file_option: go_package_prefix
      value: github.com/nadhifikbarw/example-buf-rpc-service/api
    # Disable Managed Mode behavior for `.proto` sources from this BSR module.
    # This will make compiler respect explicit `go_package` annotation
  disable:
    - file_option: go_package_prefix
      module: buf.build/googleapis/googleapis

Picking ConnectRPC or gRPC

In case you only learned about ConnectRPC. ConnectRPC is an alternative RPC framework created by Buf. While it should be considered as seperate RPC protocol, it maintains gRPC-compatibility. ConnectRPC is now a Cloud Native Computing Foundation (CNCF) sandbox project.

In actual project you obviously don't need to implement both. I implemented both here for exploration purposes. ConnectRPC provides much more convenient mechanism to handle web-based RPC client. If your requirements involve allowing RPC call from browser consider Connect Protocol

Buf only supports pushed Buf module (BSR module) for remote dependencies

Dependencies are shared between all modules in the workspace. The value must be a valid path to a BSR module (Buf module that has been pushed to BSR). This means that if you have a module you want to use as a dependency, it must also be pushed to the BSR.

Service Implementations

The service implementations are not production-grade, it's trivial to showcase differences between gRPC/ConnectRPC implementations style.

ConnectRPC allows RPC via HTTP easily

curl --header "Content-Type: application/json" --data "{\"title\": \"Task Title\"}" http://localhost:8080/task.v1.TaskService/ListTasks

gRPC-server uses grpcurl (I enabled reflection)

grpcurl -plaintext localhost:3000 task.v1.TaskService/ListTasks