README

GitHub release GoDoc CircleCI Go Report Card

Go Command Framework

This is a framework to create well-behaving commands.

Features

  • Context-based goroutine management.
  • Signal handlers.
  • Graceful stop/restart for any kind of network servers.
  • Logging options.
  • Enhanced http.Server.
  • Ultra fast UUID-like ID generator.
  • Activity tracking.
  • Support for systemd socket activation.

Requirements

Go 1.7 or better.

Specifications

Commands using this framework implement these external specifications:

Command-line options
  • -logfile FILE

    Output logs to FILE instead of standard error.

  • -loglevel LEVEL

    Change logging threshold to LEVEL. Default is info.
    LEVEL is one of critical, error, warning, info, or debug.

  • -logformat FORMAT

    Change log formatter. Default is plain.
    FORMAT is one of plain, logfmt, or json.

Signal Handlers
  • SIGUSR1

    If -logfile is specified, this signal make the program reopen the log file to cooperate with an external log rotation program.

    On Windows, this is not implemented.

  • SIGINT and SIGTERM

    These signals cancel the context of the global environment, and hence goroutines registered with the environment. Usually this will result in graceful stop of network servers, if any.

    On Windows, only SIGINT is handled.

  • SIGHUP

    This signal is used to restart network servers gracefully. Internally, the main (master) process restarts its child process. The PID of the master process thus will not change.

    There is one limitation: the location of log file cannot be changed by graceful restart. To change log file location, the server need to be (gracefully) stopped and started.

    On Windows, this is not implemented.

  • SIGPIPE

    The framework changes the way Go handles SIGPIPE slightly. If a program using this framework receives SIGPIPE when writing to stdout or stderr, the program exits with status code 2. See #15 for details.

Environment variables
  • REQUEST_ID_HEADER

    The value of this variable is used as HTTP header name. The HTTP header is used to track activities across services. The default header name is "X-Cybozu-Request-ID".

  • CYBOZU_LISTEN_FDS

    This is used internally for graceful restart.

Usage

Read Tutorial, the design notes and godoc.

Real world examples

Pull requests are welcome to add your project to this list!

License

MIT

Expand ▾ Collapse ▴

Documentation

Overview

Package cmd provides a framework that helps implementation of commands having these features:

Better logging:

By using github.com/cybozu-go/log package, logs can be structured in JSON or logfmt format. HTTP servers log accesses automatically.

Graceful exit:

The framework provides functions to manage goroutines and network server implementation that can be shutdown gracefully.

Signal handlers:

The framework installs SIGINT/SIGTERM signal handlers for graceful exit, and SIGUSR1 signal handler to reopen log files.

Environment

Environment is the heart of the framework. It provides a base context.Context that will be canceled before program stops, and methods to manage goroutines.

To use the framework easily, the framework provides an instance of Environment as the default, and functions to work with it.

Example (Basic)

    The most basic usage of the framework.

    Output:
    
    
    Example (Http)

      HTTP server that stops gracefully.

      Output:
      
      

      Index

      Examples

      Constants

      View Source
      const (
      	// RequestIDContextKey is a context key for request ID.
      	RequestIDContextKey contextKey = "request_id"
      )

      Variables

      This section is empty.

      Functions

      func BackgroundWithID

      func BackgroundWithID(ctx context.Context) context.Context

        BackgroundWithID returns a new background context with an existing request ID in ctx, if any.

        func Cancel

        func Cancel(err error) bool

          Cancel cancels the base context of the global environment.

          Passed err will be returned by Wait(). Once canceled, Go() will not start new goroutines.

          Note that calling Cancel(nil) is perfectly valid. Unlike Stop(), Cancel(nil) cancels the base context and can gracefully stop goroutines started by Server.Serve or HTTPServer.ListenAndServe.

          This returns true if the caller is the first that calls Cancel. For second and later calls, Cancel does nothing and returns false.

          func FieldsFromContext

          func FieldsFromContext(ctx context.Context) map[string]interface{}

            FieldsFromContext returns a map of fields containing context information. Currently, request ID field is included, if any.

            func GenerateID

            func GenerateID() string

              GenerateID genereates an ID using the default generator. Multiple goroutines can safely call this.

              func Go

              func Go(f func(ctx context.Context) error)

                Go starts a goroutine that executes f in the global environment.

                f takes a drived context from the base context. The context will be canceled when f returns.

                Goroutines started by this function will be waited for by Wait until all such goroutines return.

                If f returns non-nil error, Cancel is called immediately with that error.

                f should watch ctx.Done() channel and return quickly when the channel is closed.

                func GoWithID

                func GoWithID(f func(ctx context.Context) error)

                  GoWithID calls Go with a context having a new request tracking ID.

                  func IsSignaled

                  func IsSignaled(err error) bool

                    IsSignaled returns true if err returned by Wait indicates that the program has received SIGINT or SIGTERM.

                    func IsSystemdService

                    func IsSystemdService() bool

                      IsSystemdService returns true if the program runs as a systemd service.

                      func Stop

                      func Stop()

                        Stop just declares no further Go will be called.

                        Calling Stop is optional if and only if Cancel is guaranteed to be called at some point. For instance, if the program runs until SIGINT or SIGTERM, Stop is optional.

                        func SystemdListeners

                        func SystemdListeners() ([]net.Listener, error)

                          SystemdListeners returns listeners from systemd socket activation.

                          func UTF8StringFromBytes

                          func UTF8StringFromBytes(b []byte) string

                            UTF8StringFromBytes returns a valid UTF-8 string from maybe invalid slice of bytes.

                            func Wait

                            func Wait() error

                              Wait waits for Stop or Cancel, and for all goroutines started by Go to finish.

                              The returned err is the one passed to Cancel, or nil. err can be tested by IsSignaled to determine whether the program got SIGINT or SIGTERM.

                              func WithRequestID

                              func WithRequestID(ctx context.Context, reqid string) context.Context

                                WithRequestID returns a new context with a request ID as a value.

                                Types

                                type AccessLog

                                type AccessLog struct {
                                	Topic    string    `json:"topic"`
                                	LoggedAt time.Time `json:"logged_at"`
                                	Severity string    `json:"severity"`
                                	Utsname  string    `json:"utsname"`
                                	Message  string    `json:"message"`
                                
                                	Type           string  `json:"type"`             // "access"
                                	Elapsed        float64 `json:"response_time"`    // floating point number of seconds.
                                	Protocol       string  `json:"protocol"`         // "HTTP/1.1" or alike
                                	StatusCode     int     `json:"http_status_code"` // 200, 404, ...
                                	Method         string  `json:"http_method"`
                                	RequestURI     string  `json:"url"`
                                	Host           string  `json:"http_host"`
                                	RequestLength  int64   `json:"request_size"`
                                	ResponseLength int64   `json:"response_size"`
                                	RemoteAddr     string  `json:"remote_ipaddr"`
                                	UserAgent      string  `json:"http_user_agent"`
                                	RequestID      string  `json:"request_id"`
                                }

                                  AccessLog is to decode access log records from HTTPServer. The struct is tagged for JSON format.

                                  type Environment

                                  type Environment struct {
                                  	// contains filtered or unexported fields
                                  }

                                    Environment implements context-based goroutine management.

                                    func NewEnvironment

                                    func NewEnvironment(ctx context.Context) *Environment

                                      NewEnvironment creates a new Environment.

                                      This does *not* install signal handlers for SIGINT/SIGTERM for new environments. Only the global environment will be canceled on these signals.

                                      Example

                                        Barrier wait for gorutines.

                                        Output:
                                        
                                        

                                        func (*Environment) Cancel

                                        func (e *Environment) Cancel(err error) bool

                                          Cancel cancels the base context.

                                          Passed err will be returned by Wait(). Once canceled, Go() will not start new goroutines.

                                          Note that calling Cancel(nil) is perfectly valid. Unlike Stop(), Cancel(nil) cancels the base context and can gracefully stop goroutines started by Server.Serve or HTTPServer.ListenAndServe.

                                          This returns true if the caller is the first that calls Cancel. For second and later calls, Cancel does nothing and returns false.

                                          func (*Environment) Go

                                          func (e *Environment) Go(f func(ctx context.Context) error)

                                            Go starts a goroutine that executes f.

                                            f takes a drived context from the base context. The context will be canceled when f returns.

                                            Goroutines started by this function will be waited for by Wait until all such goroutines return.

                                            If f returns non-nil error, Cancel is called immediately with that error.

                                            f should watch ctx.Done() channel and return quickly when the channel is closed.

                                            func (*Environment) GoWithID

                                            func (e *Environment) GoWithID(f func(ctx context.Context) error)

                                              GoWithID calls Go with a context having a new request tracking ID.

                                              func (*Environment) Stop

                                              func (e *Environment) Stop()

                                                Stop just declares no further Go will be called.

                                                Calling Stop is optional if and only if Cancel is guaranteed to be called at some point. For instance, if the program runs until SIGINT or SIGTERM, Stop is optional.

                                                func (*Environment) Wait

                                                func (e *Environment) Wait() error

                                                  Wait waits for Stop or Cancel, and for all goroutines started by Go to finish.

                                                  The returned err is the one passed to Cancel, or nil. err can be tested by IsSignaled to determine whether the program got SIGINT or SIGTERM.

                                                  type ExecLog

                                                  type ExecLog struct {
                                                  	Topic    string    `json:"topic"`
                                                  	LoggedAt time.Time `json:"logged_at"`
                                                  	Severity string    `json:"severity"` // "error" if exec failed.
                                                  	Utsname  string    `json:"utsname"`
                                                  	Message  string    `json:"message"`
                                                  
                                                  	Type      string   `json:"type"`          // "exec"
                                                  	Elapsed   float64  `json:"response_time"` // floating point number of seconds.
                                                  	Command   string   `json:"command"`
                                                  	Args      []string `json:"args"`
                                                  	RequestID string   `json:"request_id"`
                                                  	Error     string   `json:"error"`
                                                  	Stderr    string   `json:"stderr"`
                                                  }

                                                    ExecLog is a struct to decode command execution log from LogCmd. The struct is tagged for JSON format.

                                                    type Graceful

                                                    type Graceful struct {
                                                    	// Listen is a function to create listening sockets.
                                                    	// This function is called in the master process.
                                                    	Listen func() ([]net.Listener, error)
                                                    
                                                    	// Serve is a function to accept connections from listeners.
                                                    	// This function is called in child processes.
                                                    	// In case of errors, use os.Exit to exit.
                                                    	Serve func(listeners []net.Listener)
                                                    
                                                    	// ExitTimeout is duration before Run gives up waiting for
                                                    	// a child to exit.  Zero disables timeout.
                                                    	ExitTimeout time.Duration
                                                    
                                                    	// Env is the environment for the master process.
                                                    	// If nil, the global environment is used.
                                                    	Env *Environment
                                                    }

                                                      Graceful is a struct to implement graceful restart servers.

                                                      On Windows, this is just a dummy to make porting easy.

                                                      func (*Graceful) Run

                                                      func (g *Graceful) Run()

                                                        Run runs the graceful restarting server.

                                                        If this is the master process, Run starts a child process, and installs SIGHUP handler to restarts the child process.

                                                        If this is a child process, Run simply calls g.Serve.

                                                        Run returns immediately in the master process, and never returns in the child process.

                                                        type HTTPClient

                                                        type HTTPClient struct {
                                                        	*http.Client
                                                        
                                                        	// Severity is used to log successful requests.
                                                        	//
                                                        	// Zero suppresses logging.  Valid values are one of
                                                        	// log.LvDebug, log.LvInfo, and so on.
                                                        	//
                                                        	// Errors are always logged with log.LvError.
                                                        	Severity int
                                                        
                                                        	// Logger for HTTP request.  If nil, the default logger is used.
                                                        	Logger *log.Logger
                                                        }

                                                          HTTPClient is a thin wrapper for *http.Client.

                                                          This overrides Do method to add the request tracking header if the passed request's context brings a request tracking ID. Do also records the request log to Logger.

                                                          Do not use Get/Head/Post/PostForm. They panics.

                                                          func (*HTTPClient) Do

                                                          func (c *HTTPClient) Do(req *http.Request) (*http.Response, error)

                                                            Do overrides http.Client.Do.

                                                            req's context should have been set by http.Request.WithContext for request tracking and context-based cancelation.

                                                            func (*HTTPClient) Get

                                                            func (c *HTTPClient) Get(url string) (*http.Response, error)

                                                              Get panics.

                                                              func (*HTTPClient) Head

                                                              func (c *HTTPClient) Head(url string) (*http.Response, error)

                                                                Head panics.

                                                                func (*HTTPClient) Post

                                                                func (c *HTTPClient) Post(url, bodyType string, body io.Reader) (*http.Response, error)

                                                                  Post panics.

                                                                  func (*HTTPClient) PostForm

                                                                  func (c *HTTPClient) PostForm(url string, data url.Values) (*http.Response, error)

                                                                    PostForm panics.

                                                                    type HTTPServer

                                                                    type HTTPServer struct {
                                                                    	*http.Server
                                                                    
                                                                    	// AccessLog is a logger for access logs.
                                                                    	// If this is nil, the default logger is used.
                                                                    	AccessLog *log.Logger
                                                                    
                                                                    	// ShutdownTimeout is the maximum duration the server waits for
                                                                    	// all connections to be closed before shutdown.
                                                                    	//
                                                                    	// Zero duration disables timeout.
                                                                    	ShutdownTimeout time.Duration
                                                                    
                                                                    	// Env is the environment where this server runs.
                                                                    	//
                                                                    	// The global environment is used if Env is nil.
                                                                    	Env *Environment
                                                                    	// contains filtered or unexported fields
                                                                    }

                                                                      HTTPServer is a wrapper for http.Server.

                                                                      This struct overrides Serve and ListenAndServe* methods.

                                                                      http.Server members are replaced as following:

                                                                      - Handler is replaced with a wrapper handler.
                                                                      - ReadTimeout is set to 30 seconds if it is zero.
                                                                      - ConnState is replaced with the one provided by the framework.
                                                                      

                                                                      func (*HTTPServer) ListenAndServe

                                                                      func (s *HTTPServer) ListenAndServe() error

                                                                        ListenAndServe overrides http.Server's method.

                                                                        Unlike the original, this method returns immediately just after starting a goroutine to accept connections. To stop listening, call the environment's Cancel.

                                                                        ListenAndServe returns non-nil error if and only if net.Listen failed.

                                                                        func (*HTTPServer) ListenAndServeTLS

                                                                        func (s *HTTPServer) ListenAndServeTLS(certFile, keyFile string) error

                                                                          ListenAndServeTLS overrides http.Server's method.

                                                                          Unlike the original, this method returns immediately just after starting a goroutine to accept connections. To stop listening, call the environment's Cancel.

                                                                          Another difference from the original is that certFile and keyFile must be specified. If not, configure http.Server.TLSConfig manually and use Serve().

                                                                          HTTP/2 is always enabled.

                                                                          ListenAndServeTLS returns non-nil error if net.Listen failed or failed to load certificate files.

                                                                          func (*HTTPServer) Serve

                                                                          func (s *HTTPServer) Serve(l net.Listener) error

                                                                            Serve overrides http.Server's Serve method.

                                                                            Unlike the original, this method returns immediately just after starting a goroutine to accept connections.

                                                                            The framework automatically closes l when the environment's Cancel is called.

                                                                            Serve always returns nil.

                                                                            func (*HTTPServer) ServeHTTP

                                                                            func (s *HTTPServer) ServeHTTP(w http.ResponseWriter, r *http.Request)

                                                                              ServeHTTP implements http.Handler interface.

                                                                              func (*HTTPServer) TimedOut

                                                                              func (s *HTTPServer) TimedOut() bool

                                                                                TimedOut returns true if the server shut down before all connections got closed.

                                                                                type IDGenerator

                                                                                type IDGenerator struct {
                                                                                	// contains filtered or unexported fields
                                                                                }

                                                                                  IDGenerator generates ID suitable for request tracking.

                                                                                  func NewIDGenerator

                                                                                  func NewIDGenerator() *IDGenerator

                                                                                    NewIDGenerator creates a new IDGenerator.

                                                                                    func (*IDGenerator) Generate

                                                                                    func (g *IDGenerator) Generate() string

                                                                                      Generate generates an ID. Multiple goroutines can safely call this.

                                                                                      type LogCmd

                                                                                      type LogCmd struct {
                                                                                      	*exec.Cmd
                                                                                      
                                                                                      	// Severity is used to log successful requests.
                                                                                      	//
                                                                                      	// Zero suppresses logging.  Valid values are one of
                                                                                      	// log.LvDebug, log.LvInfo, and so on.
                                                                                      	//
                                                                                      	// Errors are always logged with log.LvError.
                                                                                      	Severity int
                                                                                      
                                                                                      	// Fields is passed to Logger as log fields.
                                                                                      	Fields map[string]interface{}
                                                                                      
                                                                                      	// Logger for execution results.  If nil, the default logger is used.
                                                                                      	Logger *log.Logger
                                                                                      }

                                                                                        LogCmd is a wrapper for *exec.Cmd to record command execution results. If command fails, log level will be log.LvError. If command succeeds, log level will be log.LvInfo.

                                                                                        In most cases, use CommandContext function to prepare LogCmd.

                                                                                        func CommandContext

                                                                                        func CommandContext(ctx context.Context, name string, args ...string) *LogCmd

                                                                                          CommandContext is similar to exec.CommandContext, but returns *LogCmd with its Context set to ctx.

                                                                                          LogCmd.Severity is set to log.LvInfo.

                                                                                          LogCmd.Logger is left nil. If you want to use another logger, set it manually.

                                                                                          func (*LogCmd) CombinedOutput

                                                                                          func (c *LogCmd) CombinedOutput() ([]byte, error)

                                                                                            CombinedOutput overrides exec.Cmd.CombinedOutput to record the result.

                                                                                            func (*LogCmd) Output

                                                                                            func (c *LogCmd) Output() ([]byte, error)

                                                                                              Output overrides exec.Cmd.Output to record the result. If Cmd.Stderr is nil, Output logs outputs to stderr as well.

                                                                                              func (*LogCmd) Run

                                                                                              func (c *LogCmd) Run() error

                                                                                                Run overrides exec.Cmd.Run to record the result. If both Cmd.Stdout and Cmd.Stderr are nil, this calls Output instead to log stderr.

                                                                                                func (*LogCmd) Wait

                                                                                                func (c *LogCmd) Wait() error

                                                                                                  Wait overrides exec.Cmd.Wait to record the result.

                                                                                                  type LogConfig

                                                                                                  type LogConfig struct {
                                                                                                  	Filename string `toml:"filename" json:"filename"`
                                                                                                  	Level    string `toml:"level"    json:"level"`
                                                                                                  	Format   string `toml:"format"   json:"format"`
                                                                                                  }

                                                                                                    LogConfig configures cybozu-go/log's default logger.

                                                                                                    Filename, if not an empty string, specifies the output filename.

                                                                                                    Level is the log threshold level name. Valid levels are "critical", "error", "warning", "info", and "debug". Empty string is treated as "info".

                                                                                                    Format specifies log formatter to be used. Available formatters are "plain", "logfmt", and "json". Empty string is treated as "plain".

                                                                                                    For details, see https://godoc.org/github.com/cybozu-go/log .

                                                                                                    Example

                                                                                                      Load logging configurations from TOML file.

                                                                                                      Output:
                                                                                                      
                                                                                                      

                                                                                                      func (LogConfig) Apply

                                                                                                      func (c LogConfig) Apply() error

                                                                                                        Apply applies configurations to the default logger.

                                                                                                        Command-line flags take precedence over the struct member values.

                                                                                                        type RequestLog

                                                                                                        type RequestLog struct {
                                                                                                        	Topic    string    `json:"topic"`
                                                                                                        	LoggedAt time.Time `json:"logged_at"`
                                                                                                        	Severity string    `json:"severity"` // "error" if request failed.
                                                                                                        	Utsname  string    `json:"utsname"`
                                                                                                        	Message  string    `json:"message"`
                                                                                                        
                                                                                                        	Type         string    `json:"type"`             // "http"
                                                                                                        	ResponseTime float64   `json:"response_time"`    // floating point number of seconds.
                                                                                                        	StatusCode   int       `json:"http_status_code"` // 200, 404, 500, ...
                                                                                                        	Method       string    `json:"http_method"`
                                                                                                        	URLString    string    `json:"url"`
                                                                                                        	StartAt      time.Time `json:"start_at"`
                                                                                                        	RequestID    string    `json:"request_id"`
                                                                                                        	Error        string    `json:"error"`
                                                                                                        }

                                                                                                          RequestLog is to decode request log from HTTPClient. The struct is tagged for JSON format.

                                                                                                          type Server

                                                                                                          type Server struct {
                                                                                                          
                                                                                                          	// Handler handles a connection.  This must not be nil.
                                                                                                          	//
                                                                                                          	// ctx is a derived context from the base context that will be
                                                                                                          	// canceled when Handler returns.
                                                                                                          	//
                                                                                                          	// conn will be closed when Handler returns.
                                                                                                          	Handler func(ctx context.Context, conn net.Conn)
                                                                                                          
                                                                                                          	// ShutdownTimeout is the maximum duration the server waits for
                                                                                                          	// all connections to be closed before shutdown.
                                                                                                          	//
                                                                                                          	// Zero duration disables timeout.
                                                                                                          	ShutdownTimeout time.Duration
                                                                                                          
                                                                                                          	// Env is the environment where this server runs.
                                                                                                          	//
                                                                                                          	// The global environment is used if Env is nil.
                                                                                                          	Env *Environment
                                                                                                          	// contains filtered or unexported fields
                                                                                                          }

                                                                                                            Server is a generic network server that accepts connections and invokes Handler in a goroutine for each connection.

                                                                                                            In addition, Serve method gracefully waits all its goroutines to complete before returning.

                                                                                                            func (*Server) Serve

                                                                                                            func (s *Server) Serve(l net.Listener)

                                                                                                              Serve starts a managed goroutine to accept connections.

                                                                                                              Serve itself returns immediately. The goroutine continues to accept and handle connections until the base context is canceled.

                                                                                                              If the listener is *net.TCPListener, TCP keep-alive is automatically enabled.

                                                                                                              The listener l will be closed automatically when the environment's Cancel is called.

                                                                                                              func (*Server) TimedOut

                                                                                                              func (s *Server) TimedOut() bool

                                                                                                                TimedOut returns true if the server shut down before all connections got closed.

                                                                                                                type StdResponseWriter

                                                                                                                type StdResponseWriter interface {
                                                                                                                	http.ResponseWriter
                                                                                                                	io.ReaderFrom
                                                                                                                	http.Flusher
                                                                                                                	http.CloseNotifier
                                                                                                                	http.Hijacker
                                                                                                                	WriteString(data string) (int, error)
                                                                                                                }

                                                                                                                  StdResponseWriter is the interface implemented by the ResponseWriter from http.Server.

                                                                                                                  HTTPServer's ResponseWriter implements this as well.

                                                                                                                  Directories

                                                                                                                  Path Synopsis
                                                                                                                  test