cmd

README

go-cmd/Cmd

This package is a small but very useful wrapper around os/exec.Cmd that makes it safe and simple to run external commands in highly concurrent, asynchronous, real-time applications. It works on Linux, macOS, and Windows. Here's the basic usage:

import (
	"fmt"
	"time"
	"github.com/go-cmd/cmd"
)

func main() {
	// Start a long-running process, capture stdout and stderr
	findCmd := cmd.NewCmd("find", "/", "--name", "needle")
	statusChan := findCmd.Start() // non-blocking

	ticker := time.NewTicker(2 * time.Second)

	// Print last line of stdout every 2s
	go func() {
		for range ticker.C {
			status := findCmd.Status()
			n := len(status.Stdout)
			fmt.Println(status.Stdout[n-1])
		}
	}()

	// Stop command after 1 hour
	go func() {
		<-time.After(1 * time.Hour)
		findCmd.Stop()
	}()

	// Check if command is done
	select {
	case finalStatus := <-statusChan:
		// done
	default:
		// no, still running
	}

	// Block waiting for command to exit, be stopped, or be killed
	finalStatus := <-statusChan
}

That's it, only three methods: Start, Stop, and Status. When possible, it's better to use go-cmd/Cmd than os/exec.Cmd because go-cmd/Cmd provides:

1. Channel-based fire and forget
2. Real-time stdout and stderr
3. Real-time status
4. Complete and consolidated return
5. Proper process termination
6. 100% test coverage, no race conditions

Channel-based fire and forget

As the example above shows, starting a command immediately returns a channel to which the final status is sent when the command exits for any reason. So by default commands run asynchronously, but running synchronously is possible and easy, too:

// Run foo and block waiting for it to exit
c := cmd.NewCmd("foo")
s := <-c.Start()

To achieve similar with os/exec.Cmd requires everything this package already does.

Real-time stdout and stderr

It's common to want to read stdout or stderr while the command is running. The common approach is to call StdoutPipe and read from the provided io.ReadCloser. This works but it's wrong because it causes a race condition (that go test -race detects) and the docs say it's wrong:

It is thus incorrect to call Wait before all reads from the pipe have completed. For the same reason, it is incorrect to call Run when using StdoutPipe.

The proper solution is to set the io.Writer of Stdout. To be thread-safe and non-racey, this requires further work to write while possibly N-many goroutines read. go-cmd/Cmd has done this work.

Real-time status

Similar to real-time stdout and stderr, it's nice to see, for example, elapsed runtime. This package allows that: Status can be called any time by any goroutine, and it returns this struct:

type Status struct {
    Cmd      string
    PID      int
    Complete bool
    Exit     int
    Error    error
    Runtime  float64 // seconds
    Stdout   []string
    Stderr   []string
}

Complete and consolidated return

Speaking of that struct above, Go built-in Cmd does not put all the return information in one place, which is fine because Go is awesome! But to save some time, go-cmd/Cmd uses the Status struct above to convey all information about the command. Even when the command finishes, calling Status returns the final status, the same final status sent to the status channel returned by the call to Start.

Proper process termination

os/exec/Cmd.Wait can block even after the command is killed. That can be surprising and cause problems. But go-cmd/Cmd.Stop reliably terminates the command, no surprises. The issue has to do with process group IDs. It's common to kill the command PID, but usually one needs to kill its process group ID instead. go-cmd/Cmd.Stop implements the necessary low-level magic to make this happen.

100% test coverage, no race conditions

In addition to 100% test coverage and no race conditions, this package is actively used in production environments.

---

Acknowledgements

Brian Ip wrote the original code to get the exit status. Strangely, Go doesn't just provide this, it requires magic like exiterr.Sys().(syscall.WaitStatus) and more.

---

License

MIT © go-Cmd.

Documentation

Overview

Package cmd runs external commands with concurrent access to output and status. It wraps the Go standard library os/exec.Command to correctly handle reading output (STDOUT and STDERR) while a command is running and killing a command. All operations are safe to call from multiple goroutines.

A basic example that runs env and prints its output:

import (
    "fmt"
    "github.com/go-cmd/cmd"
)

func main() {
    // Create Cmd, buffered output
    envCmd := cmd.NewCmd("env")

    // Run and wait for Cmd to return Status
    status := <-envCmd.Start()

    // Print each line of STDOUT from Cmd
    for _, line := range status.Stdout {
        fmt.Println(line)
    }
}

Commands can be ran synchronously (blocking) or asynchronously (non-blocking):

envCmd := cmd.NewCmd("env") // create

status := <-envCmd.Start() // run blocking

statusChan := envCmd.Start() // run non-blocking
// Do other work while Cmd is running...
status <- statusChan // blocking

Start returns a channel to which the final Status is sent when the command finishes for any reason. The first example blocks receiving on the channel. The second example is non-blocking because it saves the channel and receives on it later. Only one final status is sent to the channel; use Done for multiple goroutines to wait for the command to finish, then call Status to get the final status.

Index

• Constants
• Variables
• type Cmd
  • func NewCmd(name string, args ...string) *Cmd
  • func NewCmdOptions(options Options, name string, args ...string) *Cmd
  • func (c *Cmd) Clone() *Cmd
  • func (c *Cmd) Done() <-chan struct{}
  • func (c *Cmd) Start() <-chan Status
  • func (c *Cmd) StartWithStdin(in io.Reader) <-chan Status
  • func (c *Cmd) Status() Status
  • func (c *Cmd) Stop() error
• type ErrLineBufferOverflow
  • func (e ErrLineBufferOverflow) Error() string
• type Options
• type OutputBuffer
  • func NewOutputBuffer() *OutputBuffer
  • func (rw *OutputBuffer) Lines() []string
  • func (rw *OutputBuffer) Write(p []byte) (n int, err error)
• type OutputStream
  • func NewOutputStream(streamChan chan string) *OutputStream
  • func (rw *OutputStream) Flush()
  • func (rw *OutputStream) Lines() <-chan string
  • func (rw *OutputStream) SetLineBufferSize(n int)
  • func (rw *OutputStream) Write(p []byte) (n int, err error)
• type Status

Constants

const (
	// DEFAULT_LINE_BUFFER_SIZE is the default size of the OutputStream line buffer.
	// The default value is usually sufficient, but if ErrLineBufferOverflow errors
	// occur, try increasing the size by calling OutputBuffer.SetLineBufferSize.
	DEFAULT_LINE_BUFFER_SIZE = 16384

	// DEFAULT_STREAM_CHAN_SIZE is the default string channel size for a Cmd when
	// Options.Streaming is true. The string channel size can have a minor
	// performance impact if too small by causing OutputStream.Write to block
	// excessively.
	DEFAULT_STREAM_CHAN_SIZE = 1000
)

Variables

var (
	// ErrNotStarted is returned by Stop if called before Start or StartWithStdin.
	ErrNotStarted = errors.New("command not running")
)

Types

type Cmd

type Cmd struct {
	// Name of binary (command) to run. This is the only required value.
	// No path expansion is done.
	// Used to set underlying os/exec.Cmd.Path.
	Name string

	// Commands line arguments passed to the command.
	// Args are optional.
	// Used to set underlying os/exec.Cmd.Args.
	Args []string

	// Environment variables set before running the command.
	// Env is optional.
	Env []string

	// Current working directory from which to run the command.
	// Dir is optional; default is current working directory
	// of calling process.
	// Used to set underlying os/exec.Cmd.Dir.
	Dir string

	// Stdout sets streaming STDOUT if enabled, else nil (see Options).
	Stdout chan string

	// Stderr sets streaming STDERR if enabled, else nil (see Options).
	Stderr chan string

	*sync.Mutex
	// contains filtered or unexported fields
}

Cmd represents an external command, similar to the Go built-in os/exec.Cmd. A Cmd cannot be reused after calling Start. Exported fields are read-only and should not be modified, except Env which can be set before calling Start. To create a new Cmd, call NewCmd or NewCmdOptions.

func NewCmd

func NewCmd(name string, args ...string) *Cmd

NewCmd creates a new Cmd for the given command name and arguments. The command is not started until Start is called. Output buffering is on, streaming output is off. To control output, use NewCmdOptions instead.

func NewCmdOptions

func NewCmdOptions(options Options, name string, args ...string) *Cmd

NewCmdOptions creates a new Cmd with options. The command is not started until Start is called.

func (*Cmd) Clone

func (c *Cmd) Clone() *Cmd

Clone clones a Cmd. All the options are transferred, but the internal state of the original object is lost. Cmd is one-use only, so if you need to restart a Cmd, you need to Clone it.

func (*Cmd) Done

func (c *Cmd) Done() <-chan struct{}

Done returns a channel that's closed when the command stops running. This method is useful for multiple goroutines to wait for the command to finish.Call Status after the command finishes to get its final status.

func (*Cmd) Start

func (c *Cmd) Start() <-chan Status

Start starts the command and immediately returns a channel that the caller can use to receive the final Status of the command when it ends. The caller can start the command and wait like,

status := <-myCmd.Start() // blocking

or start the command asynchronously and be notified later when it ends,

statusChan := myCmd.Start() // non-blocking
// Do other work while Cmd is running...
status := <-statusChan // blocking

Exactly one Status is sent on the channel when the command ends. The channel is not closed. Any Go error is set to Status.Error. Start is idempotent; it always returns the same channel.

func (*Cmd) StartWithStdin

func (c *Cmd) StartWithStdin(in io.Reader) <-chan Status

StartWithStdin is the same as Start but uses in for STDIN.

func (*Cmd) Status

func (c *Cmd) Status() Status

Status returns the Status of the command at any time. It is safe to call concurrently by multiple goroutines.

With buffered output, Status.Stdout and Status.Stderr contain the full output as of the Status call time. For example, if the command counts to 3 and three calls are made between counts, Status.Stdout contains:

"1"
"1 2"
"1 2 3"

The caller is responsible for tailing the buffered output if needed. Else, consider using streaming output. When the command finishes, buffered output is complete and final.

Status.Runtime is updated while the command runs and is final when it finishes.

func (*Cmd) Stop

func (c *Cmd) Stop() error

Stop stops the command by sending its process group a SIGTERM signal. Stop is idempotent. Stopping and already stopped command returns nil.

Stop returns ErrNotStarted if called before Start or StartWithStdin. If the command is very slow to start, Stop can return ErrNotStarted after calling Start or StartWithStdin because this package is still waiting for the system to start the process. All other return errors are from the low-level system function for process termination.

type ErrLineBufferOverflow

type ErrLineBufferOverflow struct {
	Line       string // Unterminated line that caused the error
	BufferSize int    // Internal line buffer size
	BufferFree int    // Free bytes in line buffer
}

ErrLineBufferOverflow is returned by OutputStream.Write when the internal line buffer is filled before a newline character is written to terminate a line. Increasing the line buffer size by calling OutputStream.SetLineBufferSize can help prevent this error.

func (ErrLineBufferOverflow) Error

func (e ErrLineBufferOverflow) Error() string

type Options

type Options struct {
	// If Buffered is true, STDOUT and STDERR are written to Status.Stdout and
	// Status.Stderr. The caller can call Cmd.Status to read output at intervals.
	// See Cmd.Status for more info.
	Buffered bool

	// If CombinedOutput is true, STDOUT and STDERR are written only to Status.Stdout
	// (similar to 2>&1 on Linux), and Status.StdErr will be empty. If CombinedOutput
	// is used Buffered, CombinedOutput takes preference. CombinedOutput does not work
	// with Streaming.
	CombinedOutput bool

	// If Streaming is true, Cmd.Stdout and Cmd.Stderr channels are created and
	// STDOUT and STDERR output lines are written them in real time. This is
	// faster and more efficient than polling Cmd.Status. The caller must read both
	// streaming channels, else lines are dropped silently. Streaming does not work
	// with CombinedOutput.
	Streaming bool

	// BeforeExec is a list of functions called immediately before starting
	// the real command. These functions can be used to customize the underlying
	// os/exec.Cmd. For example, to set SysProcAttr.
	BeforeExec []func(cmd *exec.Cmd)

	// LineBufferSize sets the size of the OutputStream line buffer. The default
	// value DEFAULT_LINE_BUFFER_SIZE is usually sufficient, but if
	// ErrLineBufferOverflow errors occur, try increasing the size with this field.
	LineBufferSize uint
}

Options represents customizations for NewCmdOptions.

type OutputBuffer

type OutputBuffer struct {
	*sync.Mutex
	// contains filtered or unexported fields
}

OutputBuffer represents command output that is saved, line by line, in an unbounded buffer. It is safe for multiple goroutines to read while the command is running and after it has finished. If output is small (a few megabytes) and not read frequently, an output buffer is a good solution.

A Cmd in this package uses an OutputBuffer for both STDOUT and STDERR by default when created by calling NewCmd. To use OutputBuffer directly with a Go standard library os/exec.Command:

import "os/exec"
import "github.com/go-cmd/cmd"
runnableCmd := exec.Command(...)
stdout := cmd.NewOutputBuffer()
runnableCmd.Stdout = stdout

While runnableCmd is running, call stdout.Lines() to read all output currently written.

func NewOutputBuffer

func NewOutputBuffer() *OutputBuffer

NewOutputBuffer creates a new output buffer. The buffer is unbounded and safe for multiple goroutines to read while the command is running by calling Lines.

func (*OutputBuffer) Lines

func (rw *OutputBuffer) Lines() []string

Lines returns lines of output written by the Cmd. It is safe to call while the Cmd is running and after it has finished. Subsequent calls returns more lines, if more lines were written. "\r\n" are stripped from the lines.

func (*OutputBuffer) Write

func (rw *OutputBuffer) Write(p []byte) (n int, err error)

Write makes OutputBuffer implement the io.Writer interface. Do not call this function directly.

type OutputStream

type OutputStream struct {
	// contains filtered or unexported fields
}

OutputStream represents real time, line by line output from a running Cmd. Lines are terminated by a single newline preceded by an optional carriage return. Both newline and carriage return are stripped from the line when sent to a caller-provided channel.

The caller must begin receiving before starting the Cmd. Write blocks on the channel; the caller must always read the channel. The channel is closed when the Cmd exits and all output has been sent.

A Cmd in this package uses an OutputStream for both STDOUT and STDERR when created by calling NewCmdOptions and Options.Streaming is true. To use OutputStream directly with a Go standard library os/exec.Command:

import "os/exec"
import "github.com/go-cmd/cmd"

stdoutChan := make(chan string, 100)
go func() {
    for line := range stdoutChan {
        // Do something with the line
    }
}()

runnableCmd := exec.Command(...)
stdout := cmd.NewOutputStream(stdoutChan)
runnableCmd.Stdout = stdout

While runnableCmd is running, lines are sent to the channel as soon as they are written and newline-terminated by the command.

func NewOutputStream

func NewOutputStream(streamChan chan string) *OutputStream

NewOutputStream creates a new streaming output on the given channel. The caller must begin receiving on the channel before the command is started. The OutputStream never closes the channel.

func (*OutputStream) Flush

func (rw *OutputStream) Flush()

Flush empties the buffer of its last line.

func (*OutputStream) Lines

func (rw *OutputStream) Lines() <-chan string

Lines returns the channel to which lines are sent. This is the same channel passed to NewOutputStream.

func (*OutputStream) SetLineBufferSize

func (rw *OutputStream) SetLineBufferSize(n int)

SetLineBufferSize sets the internal line buffer size. The default is DEFAULT_LINE_BUFFER_SIZE. This function must be called immediately after NewOutputStream, and it is not safe to call by multiple goroutines.

Increasing the line buffer size can help reduce ErrLineBufferOverflow errors.

func (*OutputStream) Write

func (rw *OutputStream) Write(p []byte) (n int, err error)

Write makes OutputStream implement the io.Writer interface. Do not call this function directly.

type Status

type Status struct {
	Cmd      string
	PID      int
	Complete bool     // false if stopped or signaled
	Exit     int      // exit code of process
	Error    error    // Go error
	StartTs  int64    // Unix ts (nanoseconds), zero if Cmd not started
	StopTs   int64    // Unix ts (nanoseconds), zero if Cmd not started or running
	Runtime  float64  // seconds, zero if Cmd not started
	Stdout   []string // buffered STDOUT; see Cmd.Status for more info
	Stderr   []string // buffered STDERR; see Cmd.Status for more info
}

Status represents the running status and consolidated return of a Cmd. It can be obtained any time by calling Cmd.Status. If StartTs > 0, the command has started. If StopTs > 0, the command has stopped. After the command finishes for any reason, this combination of values indicates success (presuming the command only exits zero on success):

Exit     = 0
Error    = nil
Complete = true

Error is a Go error from the underlying os/exec.Cmd.Start or os/exec.Cmd.Wait. If not nil, the command either failed to start (it never ran) or it started but was terminated unexpectedly (probably signaled). In either case, the command failed. Callers should check Error first. If nil, then check Exit and Complete.