README
¶
Go Command Framework
This is a framework to create well-behaving commands.
Features
- Context-based goroutine management.
- Signal handlers.
- Graceful stop/restart for any kind of network servers.
- Logging options.
- Enhanced http.Server.
- Ultra fast UUID-like ID generator.
- Activity tracking.
- Support for systemd socket activation.
Requirements
Go 1.7 or better.
Specifications
Commands using this framework implement these external specifications:
Command-line options
-
-logfile FILE
Output logs to FILE instead of standard error.
-
-loglevel LEVEL
Change logging threshold to LEVEL. Default is
info
.
LEVEL is one ofcritical
,error
,warning
,info
, ordebug
. -
-logformat FORMAT
Change log formatter. Default is
plain
.
FORMAT is one ofplain
,logfmt
, orjson
.
Signal Handlers
-
SIGUSR1
If
-logfile
is specified, this signal make the program reopen the log file to cooperate with an external log rotation program.On Windows, this is not implemented.
-
SIGINT
andSIGTERM
These signals cancel the context of the global environment, and hence goroutines registered with the environment. Usually this will result in graceful stop of network servers, if any.
On Windows, only
SIGINT
is handled. -
SIGHUP
This signal is used to restart network servers gracefully. Internally, the main (master) process restarts its child process. The PID of the master process thus will not change.
There is one limitation: the location of log file cannot be changed by graceful restart. To change log file location, the server need to be (gracefully) stopped and started.
On Windows, this is not implemented.
-
SIGPIPE
The framework changes the way Go handles SIGPIPE slightly. If a program using this framework receives SIGPIPE when writing to stdout or stderr, the program exits with status code 2. See #15 for details.
Environment variables
-
REQUEST_ID_HEADER
The value of this variable is used as HTTP header name. The HTTP header is used to track activities across services. The default header name is "X-Cybozu-Request-ID".
-
CYBOZU_LISTEN_FDS
This is used internally for graceful restart.
Usage
Read Tutorial, the design notes and godoc.
Real world examples
github.com/cybozu-go/aptutil
github.com/cybozu-go/goma
github.com/cybozu-go/transocks
github.com/cybozu-go/usocksd
Pull requests are welcome to add your project to this list!
License
Documentation
¶
Overview ¶
Package cmd provides a framework that helps implementation of commands having these features:
Better logging:
By using github.com/cybozu-go/log package, logs can be structured in JSON or logfmt format. HTTP servers log accesses automatically.
Graceful exit:
The framework provides functions to manage goroutines and network server implementation that can be shutdown gracefully.
Signal handlers:
The framework installs SIGINT/SIGTERM signal handlers for graceful exit, and SIGUSR1 signal handler to reopen log files.
Environment ¶
Environment is the heart of the framework. It provides a base context.Context that will be canceled before program stops, and methods to manage goroutines.
To use the framework easily, the framework provides an instance of Environment as the default, and functions to work with it.
Index ¶
- Constants
- func BackgroundWithID(ctx context.Context) context.Context
- func Cancel(err error) bool
- func FieldsFromContext(ctx context.Context) map[string]interface{}
- func GenerateID() string
- func Go(f func(ctx context.Context) error)
- func GoWithID(f func(ctx context.Context) error)
- func IsSignaled(err error) bool
- func IsSystemdService() bool
- func Stop()
- func SystemdListeners() ([]net.Listener, error)
- func UTF8StringFromBytes(b []byte) string
- func Wait() error
- func WithRequestID(ctx context.Context, reqid string) context.Context
- type AccessLog
- type Environment
- type ExecLog
- type Graceful
- type HTTPClient
- func (c *HTTPClient) Do(req *http.Request) (*http.Response, error)
- func (c *HTTPClient) Get(url string) (*http.Response, error)
- func (c *HTTPClient) Head(url string) (*http.Response, error)
- func (c *HTTPClient) Post(url, bodyType string, body io.Reader) (*http.Response, error)
- func (c *HTTPClient) PostForm(url string, data url.Values) (*http.Response, error)
- type HTTPServer
- type IDGenerator
- type LogCmd
- type LogConfig
- type RequestLog
- type Server
- type StdResponseWriter
Examples ¶
Constants ¶
const (
// RequestIDContextKey is a context key for request ID.
RequestIDContextKey contextKey = "request_id"
)
Variables ¶
Functions ¶
func BackgroundWithID ¶
BackgroundWithID returns a new background context with an existing request ID in ctx, if any.
func Cancel ¶
Cancel cancels the base context of the global environment.
Passed err will be returned by Wait(). Once canceled, Go() will not start new goroutines.
Note that calling Cancel(nil) is perfectly valid. Unlike Stop(), Cancel(nil) cancels the base context and can gracefully stop goroutines started by Server.Serve or HTTPServer.ListenAndServe.
This returns true if the caller is the first that calls Cancel. For second and later calls, Cancel does nothing and returns false.
func FieldsFromContext ¶
FieldsFromContext returns a map of fields containing context information. Currently, request ID field is included, if any.
func GenerateID ¶
func GenerateID() string
GenerateID genereates an ID using the default generator. Multiple goroutines can safely call this.
func Go ¶
Go starts a goroutine that executes f in the global environment.
f takes a drived context from the base context. The context will be canceled when f returns.
Goroutines started by this function will be waited for by Wait until all such goroutines return.
If f returns non-nil error, Cancel is called immediately with that error.
f should watch ctx.Done() channel and return quickly when the channel is closed.
func IsSignaled ¶
IsSignaled returns true if err returned by Wait indicates that the program has received SIGINT or SIGTERM.
func IsSystemdService ¶
func IsSystemdService() bool
IsSystemdService returns true if the program runs as a systemd service.
func Stop ¶
func Stop()
Stop just declares no further Go will be called.
Calling Stop is optional if and only if Cancel is guaranteed to be called at some point. For instance, if the program runs until SIGINT or SIGTERM, Stop is optional.
func SystemdListeners ¶
SystemdListeners returns listeners from systemd socket activation.
func UTF8StringFromBytes ¶
UTF8StringFromBytes returns a valid UTF-8 string from maybe invalid slice of bytes.
Types ¶
type AccessLog ¶
type AccessLog struct { Topic string `json:"topic"` LoggedAt time.Time `json:"logged_at"` Severity string `json:"severity"` Utsname string `json:"utsname"` Message string `json:"message"` Type string `json:"type"` // "access" Elapsed float64 `json:"response_time"` // floating point number of seconds. Protocol string `json:"protocol"` // "HTTP/1.1" or alike StatusCode int `json:"http_status_code"` // 200, 404, ... Method string `json:"http_method"` RequestURI string `json:"url"` Host string `json:"http_host"` RequestLength int64 `json:"request_size"` ResponseLength int64 `json:"response_size"` RemoteAddr string `json:"remote_ipaddr"` UserAgent string `json:"http_user_agent"` RequestID string `json:"request_id"` }
AccessLog is to decode access log records from HTTPServer. The struct is tagged for JSON format.
type Environment ¶
type Environment struct {
// contains filtered or unexported fields
}
Environment implements context-based goroutine management.
func NewEnvironment ¶
func NewEnvironment(ctx context.Context) *Environment
NewEnvironment creates a new Environment.
This does *not* install signal handlers for SIGINT/SIGTERM for new environments. Only the global environment will be canceled on these signals.
func (*Environment) Cancel ¶
func (e *Environment) Cancel(err error) bool
Cancel cancels the base context.
Passed err will be returned by Wait(). Once canceled, Go() will not start new goroutines.
Note that calling Cancel(nil) is perfectly valid. Unlike Stop(), Cancel(nil) cancels the base context and can gracefully stop goroutines started by Server.Serve or HTTPServer.ListenAndServe.
This returns true if the caller is the first that calls Cancel. For second and later calls, Cancel does nothing and returns false.
func (*Environment) Go ¶
func (e *Environment) Go(f func(ctx context.Context) error)
Go starts a goroutine that executes f.
f takes a drived context from the base context. The context will be canceled when f returns.
Goroutines started by this function will be waited for by Wait until all such goroutines return.
If f returns non-nil error, Cancel is called immediately with that error.
f should watch ctx.Done() channel and return quickly when the channel is closed.
func (*Environment) GoWithID ¶
func (e *Environment) GoWithID(f func(ctx context.Context) error)
GoWithID calls Go with a context having a new request tracking ID.
func (*Environment) Stop ¶
func (e *Environment) Stop()
Stop just declares no further Go will be called.
Calling Stop is optional if and only if Cancel is guaranteed to be called at some point. For instance, if the program runs until SIGINT or SIGTERM, Stop is optional.
func (*Environment) Wait ¶
func (e *Environment) Wait() error
Wait waits for Stop or Cancel, and for all goroutines started by Go to finish.
The returned err is the one passed to Cancel, or nil. err can be tested by IsSignaled to determine whether the program got SIGINT or SIGTERM.
type ExecLog ¶
type ExecLog struct { Topic string `json:"topic"` LoggedAt time.Time `json:"logged_at"` Severity string `json:"severity"` // "error" if exec failed. Utsname string `json:"utsname"` Message string `json:"message"` Type string `json:"type"` // "exec" Elapsed float64 `json:"response_time"` // floating point number of seconds. Command string `json:"command"` Args []string `json:"args"` RequestID string `json:"request_id"` Error string `json:"error"` Stderr string `json:"stderr"` }
ExecLog is a struct to decode command execution log from LogCmd. The struct is tagged for JSON format.
type Graceful ¶
type Graceful struct { // Listen is a function to create listening sockets. // This function is called in the master process. Listen func() ([]net.Listener, error) // Serve is a function to accept connections from listeners. // This function is called in child processes. // In case of errors, use os.Exit to exit. Serve func(listeners []net.Listener) // ExitTimeout is duration before Run gives up waiting for // a child to exit. Zero disables timeout. ExitTimeout time.Duration // Env is the environment for the master process. // If nil, the global environment is used. Env *Environment }
Graceful is a struct to implement graceful restart servers.
On Windows, this is just a dummy to make porting easy.
func (*Graceful) Run ¶
func (g *Graceful) Run()
Run runs the graceful restarting server.
If this is the master process, Run starts a child process, and installs SIGHUP handler to restarts the child process.
If this is a child process, Run simply calls g.Serve.
Run returns immediately in the master process, and never returns in the child process.
type HTTPClient ¶
type HTTPClient struct { *http.Client // Severity is used to log successful requests. // // Zero suppresses logging. Valid values are one of // log.LvDebug, log.LvInfo, and so on. // // Errors are always logged with log.LvError. Severity int // Logger for HTTP request. If nil, the default logger is used. Logger *log.Logger }
HTTPClient is a thin wrapper for *http.Client.
This overrides Do method to add the request tracking header if the passed request's context brings a request tracking ID. Do also records the request log to Logger.
Do not use Get/Head/Post/PostForm. They panics.
func (*HTTPClient) Do ¶
Do overrides http.Client.Do.
req's context should have been set by http.Request.WithContext for request tracking and context-based cancelation.
func (*HTTPClient) Head ¶
func (c *HTTPClient) Head(url string) (*http.Response, error)
Head panics.
type HTTPServer ¶
type HTTPServer struct { *http.Server // AccessLog is a logger for access logs. // If this is nil, the default logger is used. AccessLog *log.Logger // ShutdownTimeout is the maximum duration the server waits for // all connections to be closed before shutdown. // // Zero duration disables timeout. ShutdownTimeout time.Duration // Env is the environment where this server runs. // // The global environment is used if Env is nil. Env *Environment // contains filtered or unexported fields }
HTTPServer is a wrapper for http.Server.
This struct overrides Serve and ListenAndServe* methods.
http.Server members are replaced as following:
- Handler is replaced with a wrapper handler. - ReadTimeout is set to 30 seconds if it is zero. - ConnState is replaced with the one provided by the framework.
func (*HTTPServer) ListenAndServe ¶
func (s *HTTPServer) ListenAndServe() error
ListenAndServe overrides http.Server's method.
Unlike the original, this method returns immediately just after starting a goroutine to accept connections. To stop listening, call the environment's Cancel.
ListenAndServe returns non-nil error if and only if net.Listen failed.
func (*HTTPServer) ListenAndServeTLS ¶
func (s *HTTPServer) ListenAndServeTLS(certFile, keyFile string) error
ListenAndServeTLS overrides http.Server's method.
Unlike the original, this method returns immediately just after starting a goroutine to accept connections. To stop listening, call the environment's Cancel.
Another difference from the original is that certFile and keyFile must be specified. If not, configure http.Server.TLSConfig manually and use Serve().
HTTP/2 is always enabled.
ListenAndServeTLS returns non-nil error if net.Listen failed or failed to load certificate files.
func (*HTTPServer) Serve ¶
func (s *HTTPServer) Serve(l net.Listener) error
Serve overrides http.Server's Serve method.
Unlike the original, this method returns immediately just after starting a goroutine to accept connections.
The framework automatically closes l when the environment's Cancel is called.
Serve always returns nil.
func (*HTTPServer) ServeHTTP ¶
func (s *HTTPServer) ServeHTTP(w http.ResponseWriter, r *http.Request)
ServeHTTP implements http.Handler interface.
func (*HTTPServer) TimedOut ¶
func (s *HTTPServer) TimedOut() bool
TimedOut returns true if the server shut down before all connections got closed.
type IDGenerator ¶
type IDGenerator struct {
// contains filtered or unexported fields
}
IDGenerator generates ID suitable for request tracking.
func (*IDGenerator) Generate ¶
func (g *IDGenerator) Generate() string
Generate generates an ID. Multiple goroutines can safely call this.
type LogCmd ¶
type LogCmd struct { *exec.Cmd // Severity is used to log successful requests. // // Zero suppresses logging. Valid values are one of // log.LvDebug, log.LvInfo, and so on. // // Errors are always logged with log.LvError. Severity int // Fields is passed to Logger as log fields. Fields map[string]interface{} // Logger for execution results. If nil, the default logger is used. Logger *log.Logger }
LogCmd is a wrapper for *exec.Cmd to record command execution results. If command fails, log level will be log.LvError. If command succeeds, log level will be log.LvInfo.
In most cases, use CommandContext function to prepare LogCmd.
func CommandContext ¶
CommandContext is similar to exec.CommandContext, but returns *LogCmd with its Context set to ctx.
LogCmd.Severity is set to log.LvInfo.
LogCmd.Logger is left nil. If you want to use another logger, set it manually.
func (*LogCmd) CombinedOutput ¶
CombinedOutput overrides exec.Cmd.CombinedOutput to record the result.
func (*LogCmd) Output ¶
Output overrides exec.Cmd.Output to record the result. If Cmd.Stderr is nil, Output logs outputs to stderr as well.
type LogConfig ¶
type LogConfig struct { Filename string `toml:"filename" json:"filename"` Level string `toml:"level" json:"level"` Format string `toml:"format" json:"format"` }
LogConfig configures cybozu-go/log's default logger.
Filename, if not an empty string, specifies the output filename.
Level is the log threshold level name. Valid levels are "critical", "error", "warning", "info", and "debug". Empty string is treated as "info".
Format specifies log formatter to be used. Available formatters are "plain", "logfmt", and "json". Empty string is treated as "plain".
For details, see https://godoc.org/github.com/cybozu-go/log .
type RequestLog ¶
type RequestLog struct { Topic string `json:"topic"` LoggedAt time.Time `json:"logged_at"` Severity string `json:"severity"` // "error" if request failed. Utsname string `json:"utsname"` Message string `json:"message"` Type string `json:"type"` // "http" ResponseTime float64 `json:"response_time"` // floating point number of seconds. StatusCode int `json:"http_status_code"` // 200, 404, 500, ... Method string `json:"http_method"` URLString string `json:"url"` StartAt time.Time `json:"start_at"` RequestID string `json:"request_id"` Error string `json:"error"` }
RequestLog is to decode request log from HTTPClient. The struct is tagged for JSON format.
type Server ¶
type Server struct { // Handler handles a connection. This must not be nil. // // ctx is a derived context from the base context that will be // canceled when Handler returns. // // conn will be closed when Handler returns. Handler func(ctx context.Context, conn net.Conn) // ShutdownTimeout is the maximum duration the server waits for // all connections to be closed before shutdown. // // Zero duration disables timeout. ShutdownTimeout time.Duration // Env is the environment where this server runs. // // The global environment is used if Env is nil. Env *Environment // contains filtered or unexported fields }
Server is a generic network server that accepts connections and invokes Handler in a goroutine for each connection.
In addition, Serve method gracefully waits all its goroutines to complete before returning.
func (*Server) Serve ¶
Serve starts a managed goroutine to accept connections.
Serve itself returns immediately. The goroutine continues to accept and handle connections until the base context is canceled.
If the listener is *net.TCPListener, TCP keep-alive is automatically enabled.
The listener l will be closed automatically when the environment's Cancel is called.
type StdResponseWriter ¶
type StdResponseWriter interface { http.ResponseWriter io.ReaderFrom http.Flusher http.CloseNotifier http.Hijacker WriteString(data string) (int, error) }
StdResponseWriter is the interface implemented by the ResponseWriter from http.Server.
HTTPServer's ResponseWriter implements this as well.