cmd

package module
Version: v1.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 14, 2018 License: MIT Imports: 27 Imported by: 0

README

GitHub release GoDoc CircleCI Go Report Card

Go Command Framework

This is a framework to create well-behaving commands.

Features

  • Context-based goroutine management.
  • Signal handlers.
  • Graceful stop/restart for any kind of network servers.
  • Logging options.
  • Enhanced http.Server.
  • Ultra fast UUID-like ID generator.
  • Activity tracking.
  • Support for systemd socket activation.

Requirements

Go 1.7 or better.

Specifications

Commands using this framework implement these external specifications:

Command-line options

  • -logfile FILE

    Output logs to FILE instead of standard error.

  • -loglevel LEVEL

    Change logging threshold to LEVEL. Default is info.
    LEVEL is one of critical, error, warning, info, or debug.

  • -logformat FORMAT

    Change log formatter. Default is plain.
    FORMAT is one of plain, logfmt, or json.

Signal Handlers

  • SIGUSR1

    If -logfile is specified, this signal make the program reopen the log file to cooperate with an external log rotation program.

    On Windows, this is not implemented.

  • SIGINT and SIGTERM

    These signals cancel the context of the global environment, and hence goroutines registered with the environment. Usually this will result in graceful stop of network servers, if any.

    On Windows, only SIGINT is handled.

  • SIGHUP

    This signal is used to restart network servers gracefully. Internally, the main (master) process restarts its child process. The PID of the master process thus will not change.

    There is one limitation: the location of log file cannot be changed by graceful restart. To change log file location, the server need to be (gracefully) stopped and started.

    On Windows, this is not implemented.

  • SIGPIPE

    The framework changes the way Go handles SIGPIPE slightly. If a program using this framework receives SIGPIPE when writing to stdout or stderr, the program exits with status code 2. See #15 for details.

Environment variables

  • REQUEST_ID_HEADER

    The value of this variable is used as HTTP header name. The HTTP header is used to track activities across services. The default header name is "X-Cybozu-Request-ID".

  • CYBOZU_LISTEN_FDS

    This is used internally for graceful restart.

Usage

Read Tutorial, the design notes and godoc.

Real world examples

Pull requests are welcome to add your project to this list!

License

MIT

Documentation

Overview

Package cmd provides a framework that helps implementation of commands having these features:

Better logging:

By using github.com/cybozu-go/log package, logs can be structured in JSON or logfmt format. HTTP servers log accesses automatically.

Graceful exit:

The framework provides functions to manage goroutines and network server implementation that can be shutdown gracefully.

Signal handlers:

The framework installs SIGINT/SIGTERM signal handlers for graceful exit, and SIGUSR1 signal handler to reopen log files.

Environment

Environment is the heart of the framework. It provides a base context.Context that will be canceled before program stops, and methods to manage goroutines.

To use the framework easily, the framework provides an instance of Environment as the default, and functions to work with it.

Example (Basic)

The most basic usage of the framework. 

package main

import (
	"context"
	"flag"

	"github.com/cybozu-go/cmd"
	"github.com/cybozu-go/log"
)

func doSomething() error {
	return nil
}

func main() {
	flag.Parse()
	cmd.LogConfig{}.Apply()

	cmd.Go(func(ctx context.Context) error {
		err := doSomething()

		if err != nil {
			// non-nil error will be passed to Cancel
			// by the framework.
			return err
		}

		// on success, nil should be returned.
		return nil
	})

	// some more Go
	//cmd.Go(func(ctx context.Context) error {})

	// Stop declares no Go calls will be made from this point.
	// Calling Stop is optional if Cancel is guaranteed to be called
	// at some point.
	cmd.Stop()

	// Wait waits for all goroutines started by Go to complete,
	// or one of such goroutine returns non-nil error.
	err := cmd.Wait()
	if err != nil {
		log.ErrorExit(err)
	}
}

Output:
Example (Http)

HTTP server that stops gracefully. 

package main

import (
	"flag"
	"net/http"

	"github.com/cybozu-go/cmd"
	"github.com/cybozu-go/log"
)

func main() {
	flag.Parse() // must precedes LogConfig.Apply
	cmd.LogConfig{}.Apply()

	// log accesses in JSON Lines format.
	accessLog := log.NewLogger()
	accessLog.SetFormatter(log.JSONFormat{})

	// HTTP server.
	serv := &cmd.HTTPServer{
		Server: &http.Server{
			Handler: http.FileServer(http.Dir("/path/to/directory")),
		},
		AccessLog: accessLog,
	}

	// ListenAndServe is overridden to start a goroutine by cmd.Go.
	err := serv.ListenAndServe()
	if err != nil {
		log.ErrorExit(err)
	}

	// Wait waits for SIGINT or SIGTERM.
	// In this case, cmd.Stop can be omitted.
	err = cmd.Wait()

	// Use IsSignaled to determine err is the result of a signal.
	if err != nil && !cmd.IsSignaled(err) {
		log.ErrorExit(err)
	}
}

Output:

Index

Examples

Constants

View Source 
const (
	// RequestIDContextKey is a context key for request ID.
	RequestIDContextKey contextKey = "request_id"
)

Variables

This section is empty.

Functions

func BackgroundWithID added in v1.4.0 

func BackgroundWithID(ctx context.Context) context.Context

BackgroundWithID returns a new background context with an existing request ID in ctx, if any.

func Cancel 

func Cancel(err error) bool

Cancel cancels the base context of the global environment.

Passed err will be returned by Wait(). Once canceled, Go() will not start new goroutines.

Note that calling Cancel(nil) is perfectly valid. Unlike Stop(), Cancel(nil) cancels the base context and can gracefully stop goroutines started by Server.Serve or HTTPServer.ListenAndServe.

This returns true if the caller is the first that calls Cancel. For second and later calls, Cancel does nothing and returns false.

func FieldsFromContext added in v1.1.0 

func FieldsFromContext(ctx context.Context) map[string]interface{}

FieldsFromContext returns a map of fields containing context information. Currently, request ID field is included, if any.

func GenerateID added in v1.1.0 

func GenerateID() string

GenerateID genereates an ID using the default generator. Multiple goroutines can safely call this.

func Go 

func Go(f func(ctx context.Context) error)

Go starts a goroutine that executes f in the global environment.

f takes a drived context from the base context. The context will be canceled when f returns.

Goroutines started by this function will be waited for by Wait until all such goroutines return.

If f returns non-nil error, Cancel is called immediately with that error.

f should watch ctx.Done() channel and return quickly when the channel is closed.

func GoWithID added in v1.3.0 

func GoWithID(f func(ctx context.Context) error)

GoWithID calls Go with a context having a new request tracking ID.

func IsSignaled 

func IsSignaled(err error) bool

IsSignaled returns true if err returned by Wait indicates that the program has received SIGINT or SIGTERM.

func IsSystemdService added in v1.5.0 

func IsSystemdService() bool

IsSystemdService returns true if the program runs as a systemd service.

func Stop 

func Stop()

Stop just declares no further Go will be called.

Calling Stop is optional if and only if Cancel is guaranteed to be called at some point. For instance, if the program runs until SIGINT or SIGTERM, Stop is optional.

func SystemdListeners added in v1.2.0 

func SystemdListeners() ([]net.Listener, error)

SystemdListeners returns listeners from systemd socket activation.

func UTF8StringFromBytes added in v1.1.0 

func UTF8StringFromBytes(b []byte) string

UTF8StringFromBytes returns a valid UTF-8 string from maybe invalid slice of bytes.

func Wait 

func Wait() error

Wait waits for Stop or Cancel, and for all goroutines started by Go to finish.

The returned err is the one passed to Cancel, or nil. err can be tested by IsSignaled to determine whether the program got SIGINT or SIGTERM.

func WithRequestID added in v1.1.0 

func WithRequestID(ctx context.Context, reqid string) context.Context

WithRequestID returns a new context with a request ID as a value.

Types

type AccessLog 

type AccessLog struct {
	Topic    string    `json:"topic"`
	LoggedAt time.Time `json:"logged_at"`
	Severity string    `json:"severity"`
	Utsname  string    `json:"utsname"`
	Message  string    `json:"message"`

	Type           string  `json:"type"`             // "access"
	Elapsed        float64 `json:"response_time"`    // floating point number of seconds.
	Protocol       string  `json:"protocol"`         // "HTTP/1.1" or alike
	StatusCode     int     `json:"http_status_code"` // 200, 404, ...
	Method         string  `json:"http_method"`
	RequestURI     string  `json:"url"`
	Host           string  `json:"http_host"`
	RequestLength  int64   `json:"request_size"`
	ResponseLength int64   `json:"response_size"`
	RemoteAddr     string  `json:"remote_ipaddr"`
	UserAgent      string  `json:"http_user_agent"`
	RequestID      string  `json:"request_id"`
}

AccessLog is to decode access log records from HTTPServer. The struct is tagged for JSON format.

type Environment 

type Environment struct {
	// contains filtered or unexported fields
}

Environment implements context-based goroutine management.

func NewEnvironment 

func NewEnvironment(ctx context.Context) *Environment

NewEnvironment creates a new Environment.

This does *not* install signal handlers for SIGINT/SIGTERM for new environments. Only the global environment will be canceled on these signals.

Example

Barrier wait for gorutines. 

package main

import (
	"context"

	"github.com/cybozu-go/cmd"
	"github.com/cybozu-go/log"
)

func main() {
	// An independent environment.
	env := cmd.NewEnvironment(context.Background())

	env.Go(func(ctx context.Context) error {
		// do something
		return nil
	})
	// some more env.Go()

	// wait all goroutines started by env.Go().
	// These goroutines may also be canceled when the global
	// environment is canceled.
	env.Stop()
	err := env.Wait()
	if err != nil {
		log.ErrorExit(err)
	}

	// another environment for another barrier.
	env = cmd.NewEnvironment(context.Background())

	// env.Go, env.Stop, and env.Wait again.
}

Output:

func (*Environment) Cancel 

func (e *Environment) Cancel(err error) bool

Cancel cancels the base context.

Passed err will be returned by Wait(). Once canceled, Go() will not start new goroutines.

Note that calling Cancel(nil) is perfectly valid. Unlike Stop(), Cancel(nil) cancels the base context and can gracefully stop goroutines started by Server.Serve or HTTPServer.ListenAndServe.

This returns true if the caller is the first that calls Cancel. For second and later calls, Cancel does nothing and returns false.

func (*Environment) Go 

func (e *Environment) Go(f func(ctx context.Context) error)

Go starts a goroutine that executes f.

f takes a drived context from the base context. The context will be canceled when f returns.

Goroutines started by this function will be waited for by Wait until all such goroutines return.

If f returns non-nil error, Cancel is called immediately with that error.

f should watch ctx.Done() channel and return quickly when the channel is closed.

func (*Environment) GoWithID added in v1.3.0 

func (e *Environment) GoWithID(f func(ctx context.Context) error)

GoWithID calls Go with a context having a new request tracking ID.

func (*Environment) Stop 

func (e *Environment) Stop()

Stop just declares no further Go will be called.

Calling Stop is optional if and only if Cancel is guaranteed to be called at some point. For instance, if the program runs until SIGINT or SIGTERM, Stop is optional.

func (*Environment) Wait 

func (e *Environment) Wait() error

Wait waits for Stop or Cancel, and for all goroutines started by Go to finish.

The returned err is the one passed to Cancel, or nil. err can be tested by IsSignaled to determine whether the program got SIGINT or SIGTERM.

type ExecLog added in v1.1.0 

type ExecLog struct {
	Topic    string    `json:"topic"`
	LoggedAt time.Time `json:"logged_at"`
	Severity string    `json:"severity"` // "error" if exec failed.
	Utsname  string    `json:"utsname"`
	Message  string    `json:"message"`

	Type      string   `json:"type"`          // "exec"
	Elapsed   float64  `json:"response_time"` // floating point number of seconds.
	Command   string   `json:"command"`
	Args      []string `json:"args"`
	RequestID string   `json:"request_id"`
	Error     string   `json:"error"`
	Stderr    string   `json:"stderr"`
}

ExecLog is a struct to decode command execution log from LogCmd. The struct is tagged for JSON format.

type Graceful added in v1.2.0 

type Graceful struct {
	// Listen is a function to create listening sockets.
	// This function is called in the master process.
	Listen func() ([]net.Listener, error)

	// Serve is a function to accept connections from listeners.
	// This function is called in child processes.
	// In case of errors, use os.Exit to exit.
	Serve func(listeners []net.Listener)

	// ExitTimeout is duration before Run gives up waiting for
	// a child to exit.  Zero disables timeout.
	ExitTimeout time.Duration

	// Env is the environment for the master process.
	// If nil, the global environment is used.
	Env *Environment
}

Graceful is a struct to implement graceful restart servers.

On Windows, this is just a dummy to make porting easy.

func (*Graceful) Run added in v1.2.0 

func (g *Graceful) Run()

Run runs the graceful restarting server.

If this is the master process, Run starts a child process, and installs SIGHUP handler to restarts the child process.

If this is a child process, Run simply calls g.Serve.

Run returns immediately in the master process, and never returns in the child process.

type HTTPClient added in v1.1.0 

type HTTPClient struct {
	*http.Client

	// Severity is used to log successful requests.
	//
	// Zero suppresses logging.  Valid values are one of
	// log.LvDebug, log.LvInfo, and so on.
	//
	// Errors are always logged with log.LvError.
	Severity int

	// Logger for HTTP request.  If nil, the default logger is used.
	Logger *log.Logger
}

HTTPClient is a thin wrapper for *http.Client.

This overrides Do method to add the request tracking header if the passed request's context brings a request tracking ID. Do also records the request log to Logger.

Do not use Get/Head/Post/PostForm. They panics.

func (*HTTPClient) Do added in v1.1.0 

func (c *HTTPClient) Do(req *http.Request) (*http.Response, error)

Do overrides http.Client.Do.

req's context should have been set by http.Request.WithContext for request tracking and context-based cancelation.

func (*HTTPClient) Get added in v1.1.0 

func (c *HTTPClient) Get(url string) (*http.Response, error)

Get panics.

func (*HTTPClient) Head added in v1.1.0 

func (c *HTTPClient) Head(url string) (*http.Response, error)

Head panics.

func (*HTTPClient) Post added in v1.1.0 

func (c *HTTPClient) Post(url, bodyType string, body io.Reader) (*http.Response, error)

Post panics.

func (*HTTPClient) PostForm added in v1.1.0 

func (c *HTTPClient) PostForm(url string, data url.Values) (*http.Response, error)

PostForm panics.

type HTTPServer 

type HTTPServer struct {
	*http.Server

	// AccessLog is a logger for access logs.
	// If this is nil, the default logger is used.
	AccessLog *log.Logger

	// ShutdownTimeout is the maximum duration the server waits for
	// all connections to be closed before shutdown.
	//
	// Zero duration disables timeout.
	ShutdownTimeout time.Duration

	// Env is the environment where this server runs.
	//
	// The global environment is used if Env is nil.
	Env *Environment
	// contains filtered or unexported fields
}

HTTPServer is a wrapper for http.Server.

This struct overrides Serve and ListenAndServe* methods.

http.Server members are replaced as following:

  • Handler is replaced with a wrapper handler.
  • ReadTimeout is set to 30 seconds if it is zero.
  • ConnState is replaced with the one provided by the framework.

func (*HTTPServer) ListenAndServe 

func (s *HTTPServer) ListenAndServe() error

ListenAndServe overrides http.Server's method.

Unlike the original, this method returns immediately just after starting a goroutine to accept connections. To stop listening, call the environment's Cancel.

ListenAndServe returns non-nil error if and only if net.Listen failed.

func (*HTTPServer) ListenAndServeTLS 

func (s *HTTPServer) ListenAndServeTLS(certFile, keyFile string) error

ListenAndServeTLS overrides http.Server's method.

Unlike the original, this method returns immediately just after starting a goroutine to accept connections. To stop listening, call the environment's Cancel.

Another difference from the original is that certFile and keyFile must be specified. If not, configure http.Server.TLSConfig manually and use Serve().

HTTP/2 is always enabled.

ListenAndServeTLS returns non-nil error if net.Listen failed or failed to load certificate files.

func (*HTTPServer) Serve 

func (s *HTTPServer) Serve(l net.Listener) error

Serve overrides http.Server's Serve method.

Unlike the original, this method returns immediately just after starting a goroutine to accept connections.

The framework automatically closes l when the environment's Cancel is called.

Serve always returns nil.

func (*HTTPServer) ServeHTTP 

func (s *HTTPServer) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP implements http.Handler interface.

func (*HTTPServer) TimedOut 

func (s *HTTPServer) TimedOut() bool

TimedOut returns true if the server shut down before all connections got closed.

type IDGenerator added in v1.1.0 

type IDGenerator struct {
	// contains filtered or unexported fields
}

IDGenerator generates ID suitable for request tracking.

func NewIDGenerator added in v1.1.0 

func NewIDGenerator() *IDGenerator

NewIDGenerator creates a new IDGenerator.

func (*IDGenerator) Generate added in v1.1.0 

func (g *IDGenerator) Generate() string

Generate generates an ID. Multiple goroutines can safely call this.

type LogCmd added in v1.1.0 

type LogCmd struct {
	*exec.Cmd

	// Severity is used to log successful requests.
	//
	// Zero suppresses logging.  Valid values are one of
	// log.LvDebug, log.LvInfo, and so on.
	//
	// Errors are always logged with log.LvError.
	Severity int

	// Fields is passed to Logger as log fields.
	Fields map[string]interface{}

	// Logger for execution results.  If nil, the default logger is used.
	Logger *log.Logger
}

LogCmd is a wrapper for *exec.Cmd to record command execution results. If command fails, log level will be log.LvError. If command succeeds, log level will be log.LvInfo.

In most cases, use CommandContext function to prepare LogCmd.

func CommandContext added in v1.1.0 

func CommandContext(ctx context.Context, name string, args ...string) *LogCmd

CommandContext is similar to exec.CommandContext, but returns *LogCmd with its Context set to ctx.

LogCmd.Severity is set to log.LvInfo.

LogCmd.Logger is left nil. If you want to use another logger, set it manually.

func (*LogCmd) CombinedOutput added in v1.1.0 

func (c *LogCmd) CombinedOutput() ([]byte, error)

CombinedOutput overrides exec.Cmd.CombinedOutput to record the result.

func (*LogCmd) Output added in v1.1.0 

func (c *LogCmd) Output() ([]byte, error)

Output overrides exec.Cmd.Output to record the result. If Cmd.Stderr is nil, Output logs outputs to stderr as well.

func (*LogCmd) Run added in v1.1.0 

func (c *LogCmd) Run() error

Run overrides exec.Cmd.Run to record the result. If both Cmd.Stdout and Cmd.Stderr are nil, this calls Output instead to log stderr.

func (*LogCmd) Wait added in v1.1.0 

func (c *LogCmd) Wait() error

Wait overrides exec.Cmd.Wait to record the result.

type LogConfig 

type LogConfig struct {
	Filename string `toml:"filename" json:"filename"`
	Level    string `toml:"level"    json:"level"`
	Format   string `toml:"format"   json:"format"`
}

LogConfig configures cybozu-go/log's default logger.

Filename, if not an empty string, specifies the output filename.

Level is the log threshold level name. Valid levels are "critical", "error", "warning", "info", and "debug". Empty string is treated as "info".

Format specifies log formatter to be used. Available formatters are "plain", "logfmt", and "json". Empty string is treated as "plain".

For details, see https://godoc.org/github.com/cybozu-go/log .

Example

Load logging configurations from TOML file. 

package main

import (
	"flag"

	"github.com/BurntSushi/toml"
	"github.com/cybozu-go/cmd"
	"github.com/cybozu-go/log"
)

func main() {
	// compile-time defaults
	config := &cmd.LogConfig{
		Level:  "error",
		Format: "json",
	}

	// load from TOML
	_, err := toml.DecodeFile("/path/to/config.toml", config)
	if err != nil {
		log.ErrorExit(err)
	}

	// Apply gives priority to command-line flags, if any.
	flag.Parse()
	err = config.Apply()
	if err != nil {
		log.ErrorExit(err)
	}
}

Output:

func (LogConfig) Apply 

func (c LogConfig) Apply() error

Apply applies configurations to the default logger.

Command-line flags take precedence over the struct member values.

type RequestLog added in v1.1.0 

type RequestLog struct {
	Topic    string    `json:"topic"`
	LoggedAt time.Time `json:"logged_at"`
	Severity string    `json:"severity"` // "error" if request failed.
	Utsname  string    `json:"utsname"`
	Message  string    `json:"message"`

	Type         string    `json:"type"`             // "http"
	ResponseTime float64   `json:"response_time"`    // floating point number of seconds.
	StatusCode   int       `json:"http_status_code"` // 200, 404, 500, ...
	Method       string    `json:"http_method"`
	URLString    string    `json:"url"`
	StartAt      time.Time `json:"start_at"`
	RequestID    string    `json:"request_id"`
	Error        string    `json:"error"`
}

RequestLog is to decode request log from HTTPClient. The struct is tagged for JSON format.

type Server 

type Server struct {

	// Handler handles a connection.  This must not be nil.
	//
	// ctx is a derived context from the base context that will be
	// canceled when Handler returns.
	//
	// conn will be closed when Handler returns.
	Handler func(ctx context.Context, conn net.Conn)

	// ShutdownTimeout is the maximum duration the server waits for
	// all connections to be closed before shutdown.
	//
	// Zero duration disables timeout.
	ShutdownTimeout time.Duration

	// Env is the environment where this server runs.
	//
	// The global environment is used if Env is nil.
	Env *Environment
	// contains filtered or unexported fields
}

Server is a generic network server that accepts connections and invokes Handler in a goroutine for each connection.

In addition, Serve method gracefully waits all its goroutines to complete before returning.

func (*Server) Serve 

func (s *Server) Serve(l net.Listener)

Serve starts a managed goroutine to accept connections.

Serve itself returns immediately. The goroutine continues to accept and handle connections until the base context is canceled.

If the listener is *net.TCPListener, TCP keep-alive is automatically enabled.

The listener l will be closed automatically when the environment's Cancel is called.

func (*Server) TimedOut 

func (s *Server) TimedOut() bool

TimedOut returns true if the server shut down before all connections got closed.

type StdResponseWriter 

type StdResponseWriter interface {
	http.ResponseWriter
	io.ReaderFrom
	http.Flusher
	http.CloseNotifier
	http.Hijacker
	WriteString(data string) (int, error)
}

StdResponseWriter is the interface implemented by the ResponseWriter from http.Server.

HTTPServer's ResponseWriter implements this as well.

Source Files

View all Source files

Directories

Path Synopsis
test
restart

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.