Documentation
¶
Overview ¶
Package lifecycle runs the services of a Dioad daemon: it starts them in order, rolling back on a failure; fans their failures into one channel; reports readiness; drains; and stops them in stages within a deadline.
A daemon registers each service with an App, then calls Run:
app := lifecycle.New(lifecycle.WithDrainDelay(5 * time.Second))
if err := app.Add("store", store); err != nil {
return err
}
if err := app.Add("api", apiServer); err != nil {
return err
}
return app.Run(ctx)
Services ¶
A Service has Start and Shutdown. A service that can also fail after it has started implements Failer, and one that can report that it has stopped working implements Liveness. A service that implements Ready adds a check of that name to the App's Readiness. All three are discovered by interface assertion, so a service implements only what it has.
The method set matches the one dioad/net's http.Server has (Start, Shutdown, Failed and Live), so a *http.Server is a Service with no adapter. It is Shutdown, not Stop, for that reason. A service's name is given when it is registered, not by the service.
Starting ¶
Start starts services in registration order. If one fails, the services that had already started are stopped, in the same way Stop would stop them, and Start returns the start error joined with any stop errors. A service whose Start fails is not stopped, so Start must clean up after itself.
Stopping ¶
Stop stops services in stages. A stage is a set of services stopped concurrently, optionally within a budget; the stages run one after another, all within one overall deadline: the drain delay plus the stop budget (WithStopBudget, 30 seconds by default). A stage marked Optional may name services that were not added. With no stages configured every service is its own stage, stopped in reverse registration order. A service that no stage names is stopped after the stages, in reverse registration order. Errors are combined with errors.Join.
A service's Shutdown must return when its context is done. The App cannot stop a Shutdown that ignores it.
Failure and drain ¶
A failure of a service that implements Failer is reported on App.Failed, wrapped in a ServiceError. Run, on a failure, stops the services at once; when its context is done instead, it first drains: Readiness reports not ready, and Run waits for the drain delay (WithDrainDelay) so that a load balancer stops sending requests before the listeners close.
Readiness ¶
Readiness is a health source: it has the Ready method that dioad/net's http.Server.AddHealthSource looks for, so it is reported by /health/ready without this package importing dioad/net. It is not ready until Start has returned (the start gate) and not ready once draining. In between it runs its named checks. A check can be cached for a time to live, and can report itself degraded, in which case the process stays ready and the check is reported (or WithDegraded). A check runs under a timeout (WithCheckTimeout, 5 seconds by default), so a wedged dependency fails the probe instead of hanging it.
AddDependencyCheck adds a check of a dependency the process can run without, such as a broker or another service. It is degraded rather than failing, because an outage of a dependency is shared by every instance and failing readiness would take the whole service out of rotation without moving traffic anywhere better. It runs under a 2 second timeout, below the 5 seconds a platform such as Fly allows a probe, and is cached for 30 seconds; options passed after the name and function replace those defaults.
Helpers ¶
StopFunc adapts a function to a Service that only has something to release on stop, such as a database. Workers runs background goroutines for the life of the App: they keep running while it drains and stop when their stage does.
Index ¶
- Constants
- Variables
- func Degraded(err error) error
- type App
- func (a *App) Add(name string, service Service) error
- func (a *App) Drain(ctx context.Context)
- func (a *App) Failed() <-chan error
- func (a *App) Live(ctx context.Context) error
- func (a *App) Readiness() *Readiness
- func (a *App) Ready(ctx context.Context) (any, error)
- func (a *App) Run(ctx context.Context) error
- func (a *App) Start(ctx context.Context) error
- func (a *App) Stop(ctx context.Context) error
- type CheckFunc
- type CheckOption
- type Failer
- type Liveness
- type Option
- type Readiness
- type ReadinessOption
- type Service
- type ServiceError
- type Stage
- type StopFunc
- type Workers
Examples ¶
Constants ¶
const ( // DefaultDependencyCheckTimeout is how long a check added with // AddDependencyCheck may run. It is below DefaultCheckTimeout on purpose: // 5 seconds is also what a platform commonly allows a readiness probe // (Fly's default), and a check that takes as long as the probe makes the // probe time out instead of the check report degraded. DefaultDependencyCheckTimeout = 2 * time.Second // DefaultDependencyCacheTTL is how long the result of a check added with // AddDependencyCheck is reused, so that frequent probes make at most one // request to the dependency per interval per instance. DefaultDependencyCacheTTL = 30 * time.Second )
const DefaultCheckTimeout = 5 * time.Second
DefaultCheckTimeout is how long a check may run before its context is cancelled, unless WithCheckTimeout says otherwise.
const DefaultStopBudget = 30 * time.Second
DefaultStopBudget is how long stopping the services may take, after the drain delay, when neither WithStopBudget nor WithStopTimeout is given.
Variables ¶
var ( // ErrAlreadyStarted is returned by Start, and by Add, once the App has // been started. ErrAlreadyStarted = errors.New("lifecycle: app already started") // ErrStarting is the reason Readiness reports before the App has started. ErrStarting = errors.New("starting") // ErrDraining is the reason Readiness reports once the App is draining or // stopping. ErrDraining = errors.New("draining") )
Functions ¶
Types ¶
type App ¶
type App struct {
// contains filtered or unexported fields
}
App starts, watches and stops a set of services. See the package documentation.
Example ¶
Services start in the order they are added. When one fails to start, the services that had already started are stopped, newest first, and Start returns the failure.
package main
import (
"context"
"errors"
"fmt"
"github.com/dioad/lifecycle"
)
// printer is a Service that prints what it is asked to do.
type printer struct {
name string
startErr error
}
func (p printer) Start(context.Context) error {
if p.startErr != nil {
fmt.Println("start", p.name, "failed")
return p.startErr
}
fmt.Println("start", p.name)
return nil
}
func (p printer) Shutdown(context.Context) error {
fmt.Println("stop", p.name)
return nil
}
func main() {
app := lifecycle.New()
for _, service := range []printer{
{name: "store"},
{name: "queue"},
{name: "api", startErr: errors.New("address in use")},
{name: "metrics"},
} {
if err := app.Add(service.name, service); err != nil {
fmt.Println(err)
return
}
}
fmt.Println("result:", app.Start(context.Background()))
}
Output: start store start queue start api failed stop queue stop store result: start api: address in use
func (*App) Add ¶
Add registers a service under name. Services start in the order they are added. It returns an error for an empty or repeated name, a nil service, or once the App has been started.
func (*App) Drain ¶
Drain makes Readiness report not ready, then waits for the drain delay or for ctx to be done, so that traffic stops before the services do.
func (*App) Failed ¶
Failed returns a channel that receives the failure of any service that implements Failer, as a *ServiceError, once Start has returned. It is nil before then, and never closes.
func (*App) Live ¶
Live returns nil while every started service that implements Liveness reports live, and otherwise the errors of those that do not, each naming its service.
func (*App) Readiness ¶
Readiness returns the App's readiness, for adding checks and for reporting as a health source.
func (*App) Ready ¶ added in v0.3.0
Ready is Readiness().Ready. Together with Live it lets an App be passed as a single health source, for example to dioad/net's Server.AddHealthSource, which then serves /health/live from Live and /health/ready from Ready.
func (*App) Run ¶
Run starts the services and blocks until ctx is done or a service fails, then stops them. When ctx is done it drains first and returns nil if the stop was clean. When a service fails it stops at once, without draining, and returns the *ServiceError joined with any stop errors. The drain and the stop share one stop timeout, and neither is cut short by ctx being done.
func (*App) Start ¶
Start starts the services in the order they were added, then opens Readiness. If a service fails to start, the ones already started are stopped and Start returns the failure joined with any stop errors; the App cannot then be started again. An App can be started once.
func (*App) Stop ¶
Stop stops the started services in stages, within the stop timeout and ctx's deadline, whichever is sooner, and returns the errors joined. It does not drain; see Drain. It does nothing for an App that was never started, that failed to start, or that has already stopped.
Example ¶
Stop stops the services in stages. The api stops first, so that nothing is sending the worker work; the store, which no stage names, stops last.
package main
import (
"context"
"fmt"
"time"
"github.com/dioad/lifecycle"
)
// printer is a Service that prints what it is asked to do.
type printer struct {
name string
startErr error
}
func (p printer) Start(context.Context) error {
if p.startErr != nil {
fmt.Println("start", p.name, "failed")
return p.startErr
}
fmt.Println("start", p.name)
return nil
}
func (p printer) Shutdown(context.Context) error {
fmt.Println("stop", p.name)
return nil
}
func main() {
app := lifecycle.New(lifecycle.WithStopStages(
lifecycle.Stage{Names: []string{"api"}, Budget: time.Second},
lifecycle.Stage{Names: []string{"worker"}},
))
for _, name := range []string{"store", "worker", "api"} {
if err := app.Add(name, printer{name: name}); err != nil {
fmt.Println(err)
return
}
}
if err := app.Start(context.Background()); err != nil {
fmt.Println(err)
return
}
fmt.Println("result:", app.Stop(context.Background()))
}
Output: start store start worker start api stop api stop worker stop store result: <nil>
type CheckFunc ¶
CheckFunc reports whether one thing a process depends on is working. It returns nil when it is, and an error saying why not when it is not. Wrap the error in Degraded to report a problem that does not stop the process taking traffic.
type CheckOption ¶
type CheckOption func(*check)
CheckOption configures a check added with AddCheck.
func WithCacheTTL ¶
func WithCacheTTL(ttl time.Duration) CheckOption
WithCacheTTL caches the check's result for ttl, so that a health probe that arrives every second does not query a database every second. A result is not cached when the probe's context is done, since that says nothing about the thing checked. Without it every Ready runs the check.
func WithCheckTimeout ¶ added in v0.2.0
func WithCheckTimeout(d time.Duration) CheckOption
WithCheckTimeout bounds how long the check may run. When it expires the check's context is cancelled, so a wedged dependency fails the probe quickly instead of hanging the load balancer's request. The default is DefaultCheckTimeout; zero or less removes the limit. The check must return when its context is done.
func WithDegraded ¶ added in v0.2.0
func WithDegraded() CheckOption
WithDegraded reports the check's failures as degradations: the check is reported, but the process stays ready. Use it for a dependency the process can run without. It is the same as the check returning Degraded(err).
type Failer ¶
type Failer interface {
Failed() <-chan error
}
Failer is implemented by a Service that can stop working after Start has returned. Failed returns a channel that receives, at most once, the error that ended the service. It need not close it.
type Liveness ¶
Liveness is implemented by a Service that can report that it has stopped working. Live returns nil while the service is working.
type Option ¶
type Option func(*App)
Option configures an App.
func WithDrainDelay ¶
WithDrainDelay sets how long Drain waits after reporting not ready, before the services are stopped. The default is no wait.
func WithStopBudget ¶ added in v0.2.0
WithStopBudget sets how long stopping the services may take, after the drain delay: the deadline for draining and stopping together is the drain delay plus this. The default is DefaultStopBudget (30 seconds). Zero or less means no deadline other than the context given to Stop. WithStopTimeout, if also given, takes precedence.
func WithStopStages ¶
WithStopStages sets the stages Stop stops services in. See the package documentation for what stops a service no stage names.
func WithStopTimeout ¶
WithStopTimeout sets the deadline for draining and stopping together, drain delay included. It takes precedence over WithStopBudget. Zero or less means no deadline other than the context given to Stop. Prefer WithStopBudget, which does not have to be changed when the drain delay is.
type Readiness ¶
type Readiness struct {
// contains filtered or unexported fields
}
Readiness reports whether a process should be sent traffic. It is not ready until MarkStarted, not ready once BeginDrain, and in between ready unless a check fails.
It has the Ready method that dioad/net's http.Server.AddHealthSource looks for. The zero value is not usable; call NewReadiness.
Example ¶
A process is not ready until it has started, stays ready while a check is degraded, is not ready when a check fails, and is not ready once it is draining, so that traffic stops before its listeners close.
package main
import (
"context"
"errors"
"fmt"
"github.com/dioad/lifecycle"
)
func main() {
readiness := lifecycle.NewReadiness()
var searchErr, dbErr error
readiness.AddCheck("db", func(context.Context) error { return dbErr })
readiness.AddCheck("search", func(context.Context) error { return lifecycle.Degraded(searchErr) })
report := func(when string) {
status, err := readiness.Ready(context.Background())
fmt.Printf("%s: status=%v err=%v\n", when, status, err)
}
report("before start")
readiness.MarkStarted()
report("started")
searchErr = errors.New("slow")
report("search degraded")
dbErr = errors.New("connection refused")
report("db down")
dbErr = nil
readiness.BeginDrain()
report("draining")
}
Output: before start: status=<nil> err=starting started: status=map[db:ok search:ok] err=<nil> search degraded: status=map[db:ok search:degraded: slow] err=<nil> db down: status=map[db:failed: connection refused search:degraded: slow] err=db: connection refused draining: status=<nil> err=draining
func NewReadiness ¶
func NewReadiness(opts ...ReadinessOption) *Readiness
NewReadiness returns a Readiness that is not ready until MarkStarted.
func (*Readiness) AddCheck ¶
func (r *Readiness) AddCheck(name string, fn CheckFunc, opts ...CheckOption)
AddCheck adds a named check. Checks run in the order they were added.
func (*Readiness) AddDependencyCheck ¶ added in v0.4.0
func (r *Readiness) AddDependencyCheck(name string, fn CheckFunc, opts ...CheckOption)
AddDependencyCheck adds a named check of a dependency the process can run without, such as a message broker, another service or an identity provider.
Its failure is reported as degraded and the process stays ready. An outage of a dependency is shared by every instance, so failing readiness would take the whole service out of rotation without moving traffic anywhere better.
The check runs under DefaultDependencyCheckTimeout and its result is cached for DefaultDependencyCacheTTL. opts are applied after these defaults, so one that sets the same thing replaces it; keep a timeout below the platform's probe timeout. Nothing undoes the degraded reporting: use AddCheck for a dependency the process cannot run without.
Example ¶
A dependency the process can run without, such as a message broker, is added with AddDependencyCheck: when it is down the process stays ready and the check is reported as degraded.
package main
import (
"context"
"errors"
"fmt"
"github.com/dioad/lifecycle"
)
func main() {
readiness := lifecycle.NewReadiness()
readiness.AddDependencyCheck("broker", func(context.Context) error {
return errors.New("connection refused")
})
readiness.MarkStarted()
status, err := readiness.Ready(context.Background())
fmt.Printf("status=%v err=%v\n", status, err)
}
Output: status=map[broker:degraded: connection refused] err=<nil>
func (*Readiness) BeginDrain ¶
func (r *Readiness) BeginDrain()
BeginDrain makes Ready report not ready from now on, so that traffic stops before listeners close. App does this when it drains or stops.
func (*Readiness) MarkStarted ¶
func (r *Readiness) MarkStarted()
MarkStarted opens the start gate: from now on Ready runs the checks. App does this when every service has started.
func (*Readiness) Ready ¶
Ready reports whether the process should be sent traffic. The first result is a status for each check, by name: "ok", "degraded: <reason>" or "failed: <reason>". The error is non-nil before MarkStarted (ErrStarting), after BeginDrain (ErrDraining), and when a check has failed; a degraded check does not make it non-nil.
type ReadinessOption ¶
type ReadinessOption func(*Readiness)
ReadinessOption configures a Readiness.
func WithClock ¶
func WithClock(now func() time.Time) ReadinessOption
WithClock sets the clock that check results are cached against. It is for tests.
type Service ¶
type Service interface {
// Start brings the service up and returns once it is serving. It must not
// block for the life of the service.
Start(ctx context.Context) error
// Shutdown stops the service. It must return when ctx is done.
Shutdown(ctx context.Context) error
}
Service is something an App starts and stops: an HTTP server, a worker, a publisher, a store.
type ServiceError ¶
type ServiceError struct {
// Service is the name the service was registered under.
Service string
// Err is the error the service reported.
Err error
}
ServiceError is a failure of a service, as reported on App.Failed.
func (*ServiceError) Unwrap ¶
func (f *ServiceError) Unwrap() error
Unwrap returns the error the service reported.
type Stage ¶
type Stage struct {
// Names are the names the services were registered under.
Names []string
// Budget, if positive, is how long the stage may take. A service that has
// not returned by then has its Shutdown context cancelled, and the next
// stage starts. It is inside the overall stop deadline.
Budget time.Duration
// Optional lets Names include services that were not added, which the
// stage skips. Use it for a service that exists only in some
// configurations, such as a management listener. Without it a name that is
// not a service is an error, so a typo is caught when the App starts.
Optional bool
}
Stage is a set of services that Stop stops concurrently.
Example (Optional) ¶
A stage that is marked Optional may name a service that exists only in some configurations, here a management listener that was not configured.
package main
import (
"context"
"fmt"
"github.com/dioad/lifecycle"
)
// printer is a Service that prints what it is asked to do.
type printer struct {
name string
startErr error
}
func (p printer) Start(context.Context) error {
if p.startErr != nil {
fmt.Println("start", p.name, "failed")
return p.startErr
}
fmt.Println("start", p.name)
return nil
}
func (p printer) Shutdown(context.Context) error {
fmt.Println("stop", p.name)
return nil
}
func main() {
app := lifecycle.New(lifecycle.WithStopStages(
lifecycle.Stage{Names: []string{"api"}},
lifecycle.Stage{Names: []string{"management"}, Optional: true},
))
_ = app.Add("api", printer{name: "api"})
_ = app.Start(context.Background())
_ = app.Stop(context.Background())
}
Output: start api stop api
type StopFunc ¶ added in v0.2.0
StopFunc is a Service that does nothing on Start and calls the function on Shutdown, for a resource that only has to be released, such as a database. Services stop in reverse registration order unless a stage says otherwise, so adding it before the services that use it makes it the last to stop.
Example ¶
StopFunc releases a resource that has nothing to start. Added first, the store is the last thing to stop; Workers keeps background goroutines running through the drain and stops them when their turn comes.
package main
import (
"context"
"fmt"
"sync"
"github.com/dioad/lifecycle"
)
func main() {
app := lifecycle.New()
_ = app.Add("store", lifecycle.StopFunc(func(context.Context) error {
fmt.Println("close store")
return nil
}))
_ = app.Add("worker", lifecycle.NewWorkers(func(ctx context.Context, wg *sync.WaitGroup) error {
wg.Go(func() {
<-ctx.Done()
fmt.Println("worker stopped")
})
return nil
}))
_ = app.Start(context.Background())
_ = app.Stop(context.Background())
}
Output: worker stopped close store
type Workers ¶ added in v0.2.0
type Workers struct {
// contains filtered or unexported fields
}
Workers is a Service for background goroutines. Start calls the launch function, which starts them; Shutdown cancels their context and waits for them to return.
The workers' context outlives the one passed to Start, which an App cancels when it begins to drain, so the workers keep running while the load balancer stops sending traffic and stop only when their stage does.
func NewWorkers ¶ added in v0.2.0
NewWorkers returns a Workers that calls launch on Start. launch starts its goroutines on wg and returns without waiting for them; they must return when the context is done.