svc

README

SVC - Worker life-cycle manager

SVC is a framework that creates a long-running service process, managing the live-cycle of workers. It comes with "batteries-included" for convenience and uniformity across services.

Life-cycle

SVC takes care of starting a main service and shutting it down cleanly. An application comprises of one or more Worker. If zero worker are added, SVC shuts down immediately.

The life-cycle is:

1. Initialization phase (svc.New). Each service needs a name, a version. SVC tries to create a Zap logger that workers can make use of. The ideas is to have a consistent structure-logging experience throughout the service.

2. Adding workers (svc.AddWorker): Each worker needs a name; names have to be unique, otherwise SVC shuts down immediately. Workers can optionally implement the Healther interface, in which case SVC can report when all workers are ready or shutdown the service if a worker reports to be unhealthy. Adding a worker does not initialize nor run the worker, yet. Creating new workers should not block!

3. Run phase (svc.Run): Initialized and runs all added workers. Worker get synchronously initialized in the order they were added (worker.Init). If a worker fails to initialize itself, already initialized workers get terminated and then entire service is shut down. Initializing a worker should not block the service and should be quick as no deadline is given. After all workers have been initialized, the workers get asynchronously run (worker.Run). Worker's Run should block!

4. Shutdown phase (svc.Shutdown): SVC now waits until either: (i) it got a SigInt, SigTerm, or SigHup, (ii) an error from a running worker, or (iii) that all workers have finished successfully. Then it asynchronously terminates all initialized workers (worker.Terminate). Failing to terminate a worker only logs that error, termination of other workers continues. This phase has a deadline of 15s by default, thus workers should terminate as quickly and gracefully as possible.

Worker

A worker is a component representing some long-running process. It usually is a server itself, such as a HTTP or gRPC server. Workers often have a Controller.

The life-cycle is:

1. Instantiation phase: A worker should get instantiated and then added to the service via svc.AddWorker(name, worker). Each worker needs to have a unique name.

2. Initialization phase (worker.Init): A worker gets initialized and passed a named logger that it can keep to log throughout its life-time. Initialization must not block. If a worker fails to get initialized, SVC starts the shutdown routine.

3. Run phase (worker.Run): A worker should now execute a long-running task. When the task ends with an error, SVC immediately shuts down.

4. Termination phase (worker.Terminate): A worker is asked to terminate within a given grace period.

Controller

A controller is the core of a worker, usually containing business logic, in case of a server worker, usually the router handlers.

Batteries-included

All added router endpoints are served over HTTP using WithHTTPServer option.

Health checks (WithHealthz)

GET /live is always returning 200 from the time the service started. This is to have a point from which it is easy to know that the process is live in the container.

GET /ready is returning 200 if all the ready checks are looking good the workers. Otherwise it will return 503 with a JSON body of a list of the errors. This should ideally not be exported since the errors might contain sensitive information to debug from.

Metrics (WithMetrics & WithMetricsHandler)

GET /metrics serves all registered Prometheus metrics.

See Prometheus' http handler.

Dynamic log level (WithLogLevelHandlers)

GET /loglevel gets the current log level.

PUT /loglevel sets a new log level. This can be useful to temporarily change the service's log level to debug to allow for better troubleshooting.

See Zap's http_handler.go.

Pprof (Performance profiler) (WithPProfHandlers)

GET /debug/pprof serves an index page to allow dynamic profiling while the service is running.

See net/http/pprof.

Usage

package main

import (
	"github.com/voi-oss/svc"
	"go.uber.org/zap"
)

var _ svc.Worker = (*dummyWorker)(nil)

type dummyWorker struct{}

func (d *dummyWorker) Init(*zap.Logger) error { return nil }
func (d *dummyWorker) Terminate() error       { return nil }
func (d *dummyWorker) Run() error             { select {} }

func main() {
	s, err := svc.New("minimal-service", "1.0.0")
	svc.MustInit(s, err)

	w := &dummyWorker{}
	s.AddWorker("dummy-worker", w)

	s.Run()
}

For more details, see the examples.

Examples

• minimal: go run ./examples/minimal

Configuration

Customization

The framework supports customization by using the options pattern. All customization options should be defined in options.go

Logging

The log format can be configured by providing an Option on initialization. The supported formats are:

• JSON WithDevelopmentLogger() (default) or WithProductionLogger()
• Stackdriver WithStackdriverLogger() (prefered if running in GCP)
• Console WithConsoleLogger() (use when running locally)
• Customized WithLogger() (bring your own format)

Service Termination

Service termination must consider a variety of aspects. These aspects can be managed by SVC as follows:

• A wait period can be provided to delay the termination of workers whilst an external system is refreshing their service target list. In the case of gRPC in Kubernetes this should be 35 seconds to cover the 30 second DNS TTL of kuberentes headless services. For example WithTerminationWaitPeriod(35 * time.Second)
• A grace period can be provided to allow in flight requests to be processed by the service. This period should be the max timeout of the client making the request (excluding retries) plus the wait period. For example WithTerminationGracePeriod(55 * time.Second) where the wait period is 35 seconds and the grace period is 20 seconds.
• When running in Kubernetes you should also set a terminationGracePeriodSeconds on your kubernetes deployment. This period should be longer than your grace period. For example terminationGracePeriodSeconds: 60 would be a good value when your wait period is 35 seconds and your grace period is 55 seconds.

Contributions

We encourage and support an active, healthy community of contributors — including you! Details are in the contribution guide and the code of conduct. The svc maintainers keep an eye on issues and pull requests, but you can also report any negative conduct to opensource@voiapp.io.

Contributors

• @djui
• @drpytho
• @cvik
• @K-Phoen
• @ronanbarrett
• @zatte

I am missing?

If you feel you should be on this list, create a PR to add yourself.

License

Apache 2.0, see LICENSE.md.

Documentation

Index

• func LoadFromEnv(config interface{}) error
• type Gatherer
• type Healther
• type Option
  • func WithConsoleLogger(level zapcore.Level, opts ...zap.Option) Option
  • func WithDevelopmentLogger(opts ...zap.Option) Option
  • func WithHTTPServer(port string) Option
  • func WithHealthz() Option
  • func WithLogLevelHandlers() Option
  • func WithLogger(logger *zap.Logger, atom zap.AtomicLevel) Option
  • func WithMetrics() Option
  • func WithMetricsHandler() Option
  • func WithPProfHandlers() Option
  • func WithProductionLogger(opts ...zap.Option) Option
  • func WithRouter(router *http.ServeMux) Option
  • func WithStackdriverLogger(level zapcore.Level, opts ...zap.Option) Option
  • func WithTerminationGracePeriod(d time.Duration) Option
  • func WithTerminationWaitPeriod(d time.Duration) Option
  • func WithZapMetrics() Option
• type SVC
  • func MustInit(s *SVC, err error) *SVC
  • func New(name, version string, opts ...Option) (*SVC, error)
  • func (s *SVC) AddGatherer(gatherer prometheus.Gatherer)
  • func (s *SVC) AddWorker(name string, w Worker)
  • func (s *SVC) Logger() *zap.Logger
  • func (s *SVC) Run()
  • func (s *SVC) Shutdown()
• type Worker

Functions

func LoadFromEnv

func LoadFromEnv(config interface{}) error

LoadFromEnv parses environment variables into a given struct and validates its fields' values.

Types

type Gatherer

type Gatherer interface {
	Gatherer() prometheus.Gatherer
}

Gatherer is a place for workers to return a prometheus.Gatherer for SVC to serve on the metrics endpoint

type Healther

type Healther interface {
	Healthy() error
}

Healther defines a worker that can report his healthz status.

type Option

type Option func(*SVC) error

Option defines SVC's option type.

func WithConsoleLogger

func WithConsoleLogger(level zapcore.Level, opts ...zap.Option) Option

WithConsoleLogger is an option that uses a zap Logger with configurations set meant to be used for debugging in the console.

func WithDevelopmentLogger

func WithDevelopmentLogger(opts ...zap.Option) Option

WithDevelopmentLogger is an option that uses a zap Logger with configurations set meant to be used for development.

func WithHTTPServer

func WithHTTPServer(port string) Option

WithHTTPServer is an option that adds an internal HTTP server exposing observability routes.

func WithHealthz

func WithHealthz() Option

WithHealthz is an option that exposes Kubernetes conform Healthz HTTP routes.

func WithLogLevelHandlers

func WithLogLevelHandlers() Option

WithLogLevelHandlers is an option that sets up HTTP routes to read write the log level.

func WithLogger

func WithLogger(logger *zap.Logger, atom zap.AtomicLevel) Option

WithLogger is an option that allows you to provide your own customized logger.

func WithMetrics

func WithMetrics() Option

WithMetrics is an option that exports metrics via prometheus.

func WithMetricsHandler

func WithMetricsHandler() Option

WithMetricsHandler is an option that exposes Prometheus metrics for a Prometheus scraper.

func WithPProfHandlers

func WithPProfHandlers() Option

WithPProfHandlers is an option that exposes Go's Performance Profiler via HTTP routes.

func WithProductionLogger

func WithProductionLogger(opts ...zap.Option) Option

WithProductionLogger is an option that uses a zap Logger with configurations set meant to be used for production.

func WithRouter

func WithRouter(router *http.ServeMux) Option

WithRouter is an option that replaces the HTTP router with the given http router.

func WithStackdriverLogger

func WithStackdriverLogger(level zapcore.Level, opts ...zap.Option) Option

WithStackdriverLogger is an option that uses a zap Logger with configurations set meant to be used for production and is compliant with the GCP/Stackdriver format.

func WithTerminationGracePeriod

func WithTerminationGracePeriod(d time.Duration) Option

WithTerminationGracePeriod is an option that sets the termination grace period.

func WithTerminationWaitPeriod

func WithTerminationWaitPeriod(d time.Duration) Option

WithTerminationWaitPeriod is an option that sets the termination wait period.

func WithZapMetrics

func WithZapMetrics() Option

WithZapMetrics will add a hook to the zap logger and emit metrics to prometheus based on log level and log name.

type SVC

type SVC struct {
	Name    string
	Version string

	Router *http.ServeMux

	TerminationGracePeriod time.Duration
	TerminationWaitPeriod  time.Duration
	// contains filtered or unexported fields
}

SVC defines the worker life-cycle manager. It holds service metadata, router, logger, and the workers.

func MustInit

func MustInit(s *SVC, err error) *SVC

MustInit is a convenience function to check for and halt on errors.

func New

func New(name, version string, opts ...Option) (*SVC, error)

New instantiates a new service by parsing configuration and initializing a logger.

func (*SVC) AddGatherer

func (s *SVC) AddGatherer(gatherer prometheus.Gatherer)

func (*SVC) AddWorker

func (s *SVC) AddWorker(name string, w Worker)

AddWorker adds a named worker to the service. Added workers order is maintained.

func (*SVC) Logger

func (s *SVC) Logger() *zap.Logger

Logger returns the service's logger. Logger might be nil if New() fails.

func (*SVC) Run

func (s *SVC) Run()

Run runs the service until either receiving an interrupt or a worker terminates.

func (*SVC) Shutdown

func (s *SVC) Shutdown()

Shutdown signals the framework to terminate any already started workers and shutdown the service. The call is non-blocking. Terminating the workers comes with the guarantees as the `Run` method: All workers are given a total terminate grace-period until the service goes ahead completes the shutdown phase.

type Worker

type Worker interface {
	Init(*zap.Logger) error
	Run() error
	Terminate() error
}

Worker defines a SVC worker.