go-service

README

Go Service

A framework to build services in go. This came out of out building services over the years and what I have considered good practices in building services. Hence it is highly subjective and opinionated.

This framework stands on the shoulder of giants so we don't reinvent the wheel!

Dependency Injection

This framework heavily relies on DI. We have chosen to use Uber FX. So there is great information online to get you up to speed.

Commands

A service has commands that are configured using Cobra. Each service has the following commands (you can add more):

• Server - This will host your API.
• Worker - This will host your background processing.
• Client - This will have a command that starts and finishes.

These are configured in the main function.

Configuration

The supported configuration kinds are as follows:

• YAML
• TOML

The configuration can be read from multiple sources by specifying a flag called --input. As per the following:

• env:CONFIG_FILE - Read from an env variable called CONFIG_FILE. This is the default if nothing is passed. The env variable can be file path or the configuration. If it is the config, we expect the format of extension:ENV_VARIABLE, where extension is the supported kinds and ENV_VARIABLE contains the contents of the config that are base64 encoded.
• file:path - Read from the path.

The reason for this is that we want to be able to separate how configuration is retrieved. This way we can use and application configuration system.

This is the configuration. We will outline the config required in each section.

Environment

You can specify the environment of the service.

Configuration

To configure, please specify the following:

environment: development

environment = "development"

Caching

The framework currently supports the following caching solutions:

• Redis Cache
• Ristretto

We also support the following compressions to optimize cache size:

• Snappy

Configuration

To configure, please specify the following:

cache:
  redis:
    addresses:
      server: localhost:6379
    username: test
    password: test
    db: 0
  ristretto:
    num_counters: 10000000
    max_cost: 100000000
    buffer_items: 64

[cache.redis]
username = "test"
password = "test"
db = 0

[cache.redis.addresses]
server = "localhost:6379"

[cache.ristretto]
num_counters = 10_000_000
max_cost = 100_000_000
buffer_items = 64

Feature

The framework supports OpenFeature.

Configuration

To configure, please specify the following:

feature:
  kind: flipt
  host: localhost:9000

[feature]
kind = "flipt"
host = "localhost:9000"

Runtime

We enhance the runtime with the following:

• Automaxprocs
• Automemlimit

SQL

For SQL databases we support the following:

• Postgres

We also support master, slave combinations with the awesome mssqlx.

Configuration

To configure, please specify the following:

sql:
  pg:
    masters:
      -
        url: postgres://test:test@localhost:5432/test?sslmode=disable
    slaves:
      -
        url: postgres://test:test@localhost:5432/test?sslmode=disable
    max_open_conns: 5
    max_idle_conns: 5
    conn_max_lifetime: 1h

[sql.pg]
max_open_conns = 5
max_idle_conns = 5
conn_max_lifetime = "1h"

[[sql.pg.masters]]
url = "postgres://test:test@localhost:5432/test?sslmode=disable"

[[sql.pg.slaves]]
url = "postgres://test:test@localhost:5432/test?sslmode=disable"

Health

The health package is based on go-health. This package allows us to create all sorts of ways to check external and internal systems.

We also provide ways to integrate into container integration systems. So we provide the following endpoints:

• /healthz - This allows us to check any external dependency and provide a breakdown of what is not functioning. This should only be used for verification.
• /livez: Can be used for k8s liveness.
• /readyz: Can be used for k8s readiness.

This is modelled around Kubernetes API health endpoints.

Telemetry

Telemetry is broken down in the following sections:

Logging

For logging we use Uber Zap.

Configuration

To configure, please specify the following:

telemetry:
  logger:
    enabled: true
    level: info

[telemetry.logger]
enabled = true
level = "info"

Metrics

For metrics we use Prometheus.

Trace

For distributed tracing we support the following:

• OpenTelemetry
• Baselime

Configuration

Below is the configuration for each system.

Baselime

To configure, please specify the following:

telemetry:
  tracer:
    enabled: true
    kind: baseline
    key: api-key

[telemetry.tracer]
enabled = true
kind = "baseline"
key = "api-key"

Default

To configure, please specify the following:

telemetry:
  tracer:
    enabled: true
    host: localhost:4318
    secure: false

[telemetry.tracer]
enabled = true
host = "localhost:4318"
secure = false

Token

The framework allows you to define different token generators and verifiers. We provide the config, how these are defined are up to you.

Configuration

To configure, please specify the following:

token:
  Kind: none

[token]
kind = "none"

Transport

The transport layer provides ways to abstract communication for in/out of the service. So we have the following integrations:

• gRPC - The author truly believes in IDLs.
• REST - This is achieved with gRPC-Gateway.
• NSQ

gRPC

Below is list of the provided interceptors:

• Limiter

REST

Below is list of the provided handlers:

• CORS
• Limiter

Configuration

To configure, please specify the following:

transport:
  http:
    port: 8000
    retry:
      timeout: 1s
      attempts: 3
  grpc:
    enabled: true
    port: 9000
    retry:
      timeout: 1s
      attempts: 3
  nsq:
    lookup_host: localhost:4161
    host: localhost:4150
    retry:
      timeout: 1s
      attempts: 3

[transport.http]
port = "8000"

[transport.http.retry]
timeout = "1s"
attempts = 3

[transport.grpc]
enabled = true
port = "9000"

[transport.grpc.retry]
timeout = "1s"
attempts = 3

[transport.nsq]
lookup_host = "localhost:4161"
host = "localhost:4150"

[transport.nsq.retry]
timeout = "1s"
attempts = 3

If you would like to enable TLS, do the following:

transport:
  http:
    security:
      enabled: true
      cert_file: certs/cert.pem
      key_file: certs/key.pem
  grpc:
    security:
      enabled: true
      cert_file: certs/cert.pem
      key_file: certs/key.pem

[transport.http.security]
enabled = true
cert_file = "certs/cert.pem"
key_file = "certs/key.pem"

[transport.grpc.security]
enabled = true
cert_file = "certs/cert.pem"
key_file = "certs/key.pem"

Debug

This section outlines all utilities added for you troubleshooting abilities.

statsviz

GET http://localhost:6060/debug/statsviz

Check out statsviz.

pprof

GET http://localhost:6060/debug/pprof/
GET http://localhost:6060/debug/pprof/cmdline
GET http://localhost:6060/debug/pprof/profile
GET http://localhost:6060/debug/pprof/symbol
GET http://localhost:6060/debug/pprof/trace

Check out pprof.

gopsutil

GET http://localhost:6060/debug/psutil

Check out gopsutil.

Configuration

To configure, please specify the following:

debug:
  port: 6060

[debug]
port = "6060"

Development

This section describes how to run and contribute to the project, if you are interested.

Style

We favour what is defined in the Uber Go Style Guide.

Dependencies

Please setup the following:

• https://github.com/FiloSottile/mkcert

Setup

To get yourself setup, please run:

git submodule sync
git submodule update --init

mkcert -install
make create-certs

make dep

Environment

As we rely on external services these need to be configured:

Starting

Please run:

make start

Stopping

Please run:

make stop

Testing

To be able to test locally, please run:

make specs