shexec

package module
v0.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 16, 2025 License: Apache-2.0 Imports: 8 Imported by: 0

README

shexec Go Report Card Go Reference

Package shexec lets one script a command shell in Go as if a human were running it.

The package separates the problem of orchestrating shell execution from the problem of generating a shell command and parsing the shell's response to said command.

This package hides and solves the first problem (via Shell), and makes the latter easy to do via Go implementations of Commander.

Usage

Roughly:

sh := NewShell(Parameters{
	Params: channeler.Params{Path: "/bin/sh"},
	SentinelOut: Sentinel{
		C: "echo " + unlikelyWord,
		V: unlikelyWord,
	},
})
assertNoErr(sh.Start(timeOut))
assertNoErr(sh.Run(timeOut, commander1))
// consult commander1 getters for whatever purpose,
// optionally use the results to define commander2.
assertNoErr(sh.Run(timeOut, commander2))
// consult commander2, etc.
assertNoErr(sh.Stop(timeOut, ""))

Assumptions

Shell behavior

A shell is any program that accepts newline terminated commands, e.g. bash, and emits lines of output on stdOut and stdErr.

The purpose of a shell, as opposed to a single-purpose program that doesn't prompt for commands, is to allow state that endures over multiple commands. The state contains things like authentication, authorization, secrets obtained from vaults, caches, database connections, etc.

A shell lets a user pay to build that state once, then run multiple commands in the context of that state.

Commands influence commands

There must be an opportunity to examine the output of command n before issuing command n+1.

The choice of command n+1 or its arguments may be influenced by the output of command n.

Command generation and parsing best live together

The code that parses a command's output should live close to the code that generates the command. The parser should have access to command arguments and flags so that it knows what's supposed to happen.

All a Go author need do is implement the Commander interface, then pass instances of the implementation to the Run method of a Shell. When a Run call returns, the Commander instance can be consulted. A commander can offer any number of methods yielding validated data acquired from the shell; it can be viewed as a shell visitor.

A Commander can be tested in isolation (without the need of a shell) for its ability to compose a command and parse the output expected from that command.

Unreliable prompts, unreliable newlines, and command blocks

A human knows that a shell has completed command n and is awaiting command n+1 because they see a prompt following the output of command n. Usually, but not always, the prompt is on a new line.

But in a scripting context, prompts with newlines are unreliable.

When running a shell as a subprocess, e.g. as part of a pipe, the shell can see that stdIn is not a tty, and won't issue a prompt to avoid pipe contamination.

Sometimes command output can accidentally contain data that matches the prompt, making the prompt useless as an output delimiter.

Sometimes a shell will intentionally suppress newline on command completion, e.g. base64 -d, echo -n.

Most importantly, sometimes a user wants to inject a command block, multiple commands with embedded newlines, as a single unit, not caring to know when individual commands in the block finish. Only the whole set matters. This can happen when blindly executing command blocks from some unknown source, e.g. fenced code blocks embedded in markdown documentation.

For these reasons, a Shell cannot depend on prompts and newlines to unambiguously distinguish the data from commands n-1, n and n+1 on stdOut and stdErr.

So instead of relying on prompts or newlines, Shell relies on a Sentinel.

Sentinels
stdOut

A Shell demands the existence of a sentinel command for stdOut.

Such a command

  • does very little,
  • does it quickly,
  • has deterministic, newline terminated output on stdOut.

Example:

$ echo "rumpelstiltskinOut"
rumpelstiltskinOut

Commands that print a program's version, a help message, and/or a copyright message are good candidates for sentinel commands on the stdOut stream.

The unambiguously recognizable output of a sentinel command called the sentinel value.

A Sentinel holds a {command, value} pair.

stdErr

Likewise, a Shell needs a sentinel command for stdErr.

This command differs from the stdOut sentinel only in that its output goes to stdErr.

Example:

$ echo "rumpelstiltskinErr" 1>&2
rumpelstiltskinErr
Command results

The outcome of asking a shell to run a command is one of the following:

  • crash - shell exits with non-zero status.
  • exit - shell exits with zero status.
    If this happens unintentionally, it's treated as a crash.
  • timeout - shell fails to finish the command in a given time period.
    The shell is assumed to be unusable, and should be killed.
  • ready - shell runs the command within the given time period and is ready to accept another command.
    The command can be consulted for whatever results it parsed and saved.

Documentation

Overview

Package shexec lets one script a command shell as if a human were running it. See README.md.

Example (BasicRun)

An error free run using the (locally defined) conch shell. 

sh := NewShell(Parameters{
	Params: channeler.Params{
		WorkingDir: "./conch",
		Path:       "go",
		Args: []string{
			"run", ".",
			// the prompt goes to stdout, so get rid of it in tests.
			"--disable-prompt",
		}},
	SentinelOut: sentinelVersion,
})
assertNoErr(sh.Start(timeOutShort))
assertNoErr(sh.Run(timeOutShort, NewLabellingCommander("query limit 3")))
assertNoErr(sh.Stop(timeOutShort, ""))

Output:

out: Cempedak_|_Bamberga_|_4_|_00000000000000000000000000000001
out: Buddha's hand_|_Hermione_|_6_|_00000000000000000000000000000002
out: African cucumber_|_Ursula_|_6_|_00000000000000000000000000000003
Example (BinShAllowError) 
sh := NewShell(Parameters{
	Params: channeler.Params{
		Path: "/bin/sh",
		// No "-e"; keep going on error.
	},
	SentinelOut: Sentinel{
		C: "echo " + unlikelyStdOut,
		V: unlikelyStdOut,
	},
})
err := sh.Start(timeOutShort)
assertNoErr(err)
assertNoErr(sh.Run(timeOutShort,
	NewLabellingCommander(`
echo alpha
which cat
`)))
assertNoErr(sh.Run(timeOutShort,
	NewLabellingCommander(`
thisCommandDoesNotExist
`,
	)))
assertNoErr(sh.Run(timeOutShort,
	NewLabellingCommander(`
echo beta
which ls
`)))
assertNoErr(sh.Stop(timeOutShort, ""))

Output:

out: alpha
out: /usr/bin/cat
out: beta
out: /usr/bin/ls
Example (BinShDieOnError)

An example using /bin/sh, a shell that's available on most platforms. 

sh := NewShell(Parameters{
	Params: channeler.Params{
		Path: "/bin/sh",
		Args: []string{"-e"},
	},
	SentinelOut: Sentinel{
		C: "echo " + unlikelyStdOut,
		V: unlikelyStdOut,
	},
	SentinelErr: Sentinel{
		C: "echo " + unlikelyStdErr + " 1>&2",
		V: unlikelyStdErr,
	},
	//EnableDetailedLogging: true,
})
err := sh.Start(timeOutShort)
assertNoErr(err)
assertNoErr(sh.Run(timeOutShort,
	NewLabellingCommander(`
echo alpha
which cat
`)))
c := NewLabellingCommander(`
echo "this is output on stderr with no failure"  1>&2
`)
assertNoErr(sh.Run(timeOutShort, c))

c = NewLabellingCommander(`
echo "this is output on stdout with a failure"
echo "this is output on stderr with a failure"  1>&2
exit 1
`)
err = sh.Run(timeOutShort, c)
if err == nil {
	panic("expected an error.")
}
if !strings.Contains(err.Error(), "closed before sentinel") {
	panic(err)
}

Output:

out: alpha
out: /usr/bin/cat
err: this is output on stderr with no failure
err: this is output on stderr with a failure
out: this is output on stdout with a failure
Example (BinShHappy)

An example using /bin/sh, a shell that's available on most platforms. 

sh := NewShell(Parameters{
	Params: channeler.Params{
		Path: "/bin/sh",
		Args: []string{"-e"},
	},
	SentinelOut: Sentinel{
		C: "echo " + unlikelyStdOut,
		V: unlikelyStdOut,
	},
})
err := sh.Start(timeOutShort)
assertNoErr(err)
assertNoErr(sh.Run(timeOutShort,
	NewLabellingCommander(`
echo alpha
which cat
`)))
assertNoErr(sh.Run(timeOutShort,
	NewLabellingCommander(`
echo beta
which find
`,
	)))
assertNoErr(sh.Stop(timeOutShort, ""))

Output:

out: alpha
out: /usr/bin/cat
out: beta
out: /usr/bin/find
Example (BinShTimeout) 
sh := NewShell(Parameters{
	Params: channeler.Params{
		Path: "/bin/sh",
		Args: []string{"-e"},
	},
	SentinelOut: Sentinel{
		C: "echo " + unlikelyStdOut,
		V: unlikelyStdOut,
	},
})
err := sh.Start(timeOutShort)
assertNoErr(err)
assertNoErr(sh.Run(timeOutShort,
	NewLabellingCommander(`
echo alpha
which cat
`)))
// Sleep past the timeout and hit an error.
assertErr(sh.Run(timeOutShort,
	NewLabellingCommander(`
sleep 2
`)))

Output:

out: alpha
out: /usr/bin/cat
Example (SubprocessFailOnStartup)

A shell that crashes on startup. 

sh := NewShell(Parameters{
	Params: channeler.Params{
		WorkingDir: "./conch",
		Path:       "go",
		Args: []string{
			"run", ".",
			"--fail-on-startup",
		},
	},
	SentinelOut: sentinelVersion,
})
err := sh.Start(timeOutShort)
fmt.Println(err.Error())

Output:

shexec infra; stdOut closed before sentinel "v1.2.3" found
Example (SubprocessNonSurvivableError)

A shell that crashes, and is then restarted. 

sh := NewShell(Parameters{
	Params: channeler.Params{
		WorkingDir: "./conch",
		Path:       "go",
		Args: []string{
			"run", ".",
			"--disable-prompt",
			// Using this means any error will cause process exit.
			// So we cannot use an errSentinel, as it by definition causes an error.
			"--exit-on-error",
			"--row-to-error-on", "4",
		}},
	SentinelOut: sentinelVersion,
})
assertNoErr(sh.Start(timeOutShort))

cmdr := NewLabellingCommander("query limit 3")

// The following yields three lines.
assertNoErr(sh.Run(timeOutShort, cmdr))

// Query again, but ask for a row beyond the row that
// triggers a DB error.
// Since flag "exit-on-error" is enabled, this causes the CLI to die.
cmdr.C = "query limit 5"
err := sh.Run(timeOutShort, cmdr)
assertErr(err)
fmt.Println(err.Error())

err = sh.Stop(timeOutShort, "")
assertErr(err)
fmt.Println(err.Error())

// Start it up again and issue a command to show that it works.
assertNoErr(sh.Start(timeOutShort))
cmdr.C = "query limit 2"
assertNoErr(sh.Run(timeOutShort, cmdr))
assertNoErr(sh.Stop(timeOutShort, ""))

Output:

out: Cempedak_|_Bamberga_|_4_|_00000000000000000000000000000001
out: Buddha's hand_|_Hermione_|_6_|_00000000000000000000000000000002
out: African cucumber_|_Ursula_|_6_|_00000000000000000000000000000003
out: Currant_|_Alauda_|_5_|_00000000000000000000000000000001
out: Banana_|_Egeria_|_5_|_00000000000000000000000000000002
out: Bilberry_|_Interamnia_|_2_|_00000000000000000000000000000003
shexec infra; stdOut closed before sentinel "v1.2.3" found
shexec infra; stop called, but shell not started yet
out: Cempedak_|_Bamberga_|_4_|_00000000000000000000000000000001
out: Buddha's hand_|_Hermione_|_6_|_00000000000000000000000000000002
Example (SubprocessSurvivableError)

A shell spits output to stderr. 

sh := NewShell(Parameters{
	Params: channeler.Params{
		WorkingDir: "./conch",
		Path:       "go",
		Args: []string{
			"run", ".",
			"--disable-prompt",
			"--row-to-error-on", "4",
		}},
	SentinelOut: sentinelVersion,
	SentinelErr: sentinelUnknownCommand,
})
assertNoErr(sh.Start(timeOutShort))

cmdr := NewLabellingCommander("query limit 3")

// The following yields three lines.
assertNoErr(sh.Run(timeOutShort, cmdr))

// Query again, but ask for a row beyond the row that
// triggers a DB error.
// Because of the nature of output streams,
// there's no way to know
// when the error will show up in the combined output.
// It might come out first, last, or anywhere in the
// middle relative to lines from stdOut,
// so this test must not be fragile to the order.
// This will yield three "good lines", and one error line.
cmdr.C = "query limit 7"
assertNoErr(sh.Run(timeOutShort, cmdr))

// Yields two lines.
cmdr.C = "query limit 2"
assertNoErr(sh.Run(timeOutShort, cmdr))

assertNoErr(sh.Stop(timeOutShort, ""))

// There should be nine (3 + 3 + 1 + 2) lines in the output.

Output:

out: Cempedak_|_Bamberga_|_4_|_00000000000000000000000000000001
out: Buddha's hand_|_Hermione_|_6_|_00000000000000000000000000000002
out: African cucumber_|_Ursula_|_6_|_00000000000000000000000000000003
out: Currant_|_Alauda_|_5_|_00000000000000000000000000000001
out: Banana_|_Egeria_|_5_|_00000000000000000000000000000002
out: Bilberry_|_Interamnia_|_2_|_00000000000000000000000000000003
err: error! touching row 4 triggers this error
out: Cherimoya_|_Palma_|_6_|_00000000000000000000000000000001
out: Abiu_|_Metis_|_3_|_00000000000000000000000000000002
Example (SubprocessTakesTooLong)

A command takes too long and fails as a result. 

sh := NewShell(Parameters{
	Params: channeler.Params{
		WorkingDir: "./conch",
		Path:       "go",
		Args: []string{
			"run", ".",
			"--disable-prompt",
		}},
	SentinelOut: sentinelVersion,
})

assertNoErr(sh.Start(timeOutShort))

// Send in a sleep command that consumes twice the timeOut.
err := sh.Run(
	timeOutShort,
	NewLabellingCommander("sleep "+(2*timeOutShort).String()))
fmt.Println(err.Error())

Output:

shexec infra; running "sleep 1.6s", no sentinels found after 800ms

Index

Examples

Constants

This section is empty.

Variables

View Source 
var DevNull io.WriteCloser = &discard{}

DevNull is an io.WriteCloser that does nothing.

Functions

This section is empty.

Types

type Commander 

type Commander interface {
	// Command is the actual command to issue to the shell.
	Command() string
	// ParseOut will be written with whatever comes out of
	// the shell's stdOut as the result of issuing Command.
	// Close will be called when the shell believes that
	// all output has been obtained.
	ParseOut() io.WriteCloser
	// ParseErr is like ParseOut, except stdErr is used instead of stdOut.
	ParseErr() io.WriteCloser
}

Commander knows a CLI command, and knows how to parse the command's output.

type DiscardCommander 

type DiscardCommander struct {
	C string
}

DiscardCommander discards everything from its parsers.

func (*DiscardCommander) Command 

func (c *DiscardCommander) Command() string

func (*DiscardCommander) ParseErr 

func (c *DiscardCommander) ParseErr() io.WriteCloser

func (*DiscardCommander) ParseOut 

func (c *DiscardCommander) ParseOut() io.WriteCloser

type LabellingCommander 

type LabellingCommander struct {
	C string
	// contains filtered or unexported fields
}

LabellingCommander passes subprocess output from stdOut and stdErr to the main process' stdOut, adding a prefix to make a distinction.

func NewLabellingCommander 

func NewLabellingCommander(c string) *LabellingCommander

NewLabellingCommander returns an instance of LabellingCommander.

func (*LabellingCommander) Command 

func (c *LabellingCommander) Command() string

func (*LabellingCommander) ParseErr 

func (c *LabellingCommander) ParseErr() io.WriteCloser

func (*LabellingCommander) ParseOut 

func (c *LabellingCommander) ParseOut() io.WriteCloser

type LineAbsorber 

type LineAbsorber struct {
	// contains filtered or unexported fields
}

LineAbsorber remembers all the non-empty lines it sees.

func (*LineAbsorber) Close 

func (ab *LineAbsorber) Close() error

func (*LineAbsorber) Lines 

func (ab *LineAbsorber) Lines() []string

func (*LineAbsorber) Reset 

func (ab *LineAbsorber) Reset()

func (*LineAbsorber) Write 

func (ab *LineAbsorber) Write(data []byte) (int, error)

type Parameters 

type Parameters struct {
	channeler.Params

	// SentinelOut holds the command sent to the shell after every
	// command other than the exit command.
	// SentinelOut is used to be sure that output generated in the
	// course of running command N is swept up and accounted for
	// before looking for output from command N+1.
	SentinelOut Sentinel

	// SentinelErr is a command that intentionally triggers output
	// on stderr, e.g. a misspelled command, a command with a non-extant
	// flag - something that doesn't cause any real trouble.  If non-empty,
	// this is issued after every command other than the exit command,
	// either before or after issuing the OutSentinel command.
	// SentinelErr is used to be sure that any errors generated in the
	// course of running command N are swept up and accounted for before
	// looking for errors from command N+1.
	SentinelErr Sentinel

	// EnableDetailedLogging does what it sounds like
	EnableDetailedLogging bool
}

Parameters is a bag of parameters for a Shell instance. See individual fields for their explanation.

func (*Parameters) Validate 

func (p *Parameters) Validate() error

Validate returns an error if there's a problem in the Parameters.

type PassThruCommander 

type PassThruCommander struct{ C string }

PassThruCommander forwards data to the current process stdOut and stdErr.

func (*PassThruCommander) Command 

func (c *PassThruCommander) Command() string

func (*PassThruCommander) ParseErr 

func (c *PassThruCommander) ParseErr() io.WriteCloser

func (*PassThruCommander) ParseOut 

func (c *PassThruCommander) ParseOut() io.WriteCloser

type RecallCommander 

type RecallCommander struct {
	C string
	// contains filtered or unexported fields
}

RecallCommander remembers all the non-empty lines it sees.

func NewRecallCommander 

func NewRecallCommander(c string) *RecallCommander

NewRecallCommander returns an instance of RecallCommander.

func (*RecallCommander) Command 

func (c *RecallCommander) Command() string

func (*RecallCommander) DataErr 

func (c *RecallCommander) DataErr() []string

func (*RecallCommander) DataOut 

func (c *RecallCommander) DataOut() []string

func (*RecallCommander) ParseErr 

func (c *RecallCommander) ParseErr() io.WriteCloser

func (*RecallCommander) ParseOut 

func (c *RecallCommander) ParseOut() io.WriteCloser

func (*RecallCommander) Reset 

func (c *RecallCommander) Reset()

type Sentinel 

type Sentinel struct {
	// C is a command that should do very little, do it quickly,
	// and have deterministic, newline terminated output.
	C string

	// V is the expected value from Command.
	// Sentinel value comparisons are only made when a
	// newline is encountered in the output stream,
	// and then only working backwards from that newline.
	// E.g. the value "foo" will match "foo\n" in the
	// output stream, but will not match "foo bar".
	V string
}

Sentinel holds a {command, value} pair.

A Sentinel is used to recognize the end of command output on a stream. Examples: 

Command: echo pink elephants dance
Value: pink elephants dance

Command: version
Value: v1.2.3

Command: rumpelstiltskin
Value: rumpelstiltskin: command not found

func (*Sentinel) Validate 

func (s *Sentinel) Validate() error

Validate returns an error if there's a problem in the Sentinel. This validation is critical; if a sentinel value is empty, the infrastructure will hang.

type Shell 

type Shell interface {
	// Start synchronously starts the shell.
	// It assures that the shell runs and that the sentinels work
	// before their first use in the Run method.
	// Errors:
	// * The shell was already started.
	// * Something's wrong in the Parameters, e.g. the shell program
	//   cannot be found.
	// * The sentinels failed to work in the time allotted.
	Start(time.Duration) error

	// Run sends the command in Commander to the shell, and
	// waits for it to complete.  It returns an error if
	// there was some infrastructure problem or if the
	// command timed out because no sentinels were detected
	// in the time given.
	// An error here means that the shell is dead, and in
	// need of fresh call to Start.
	// Errors:
	// * The shell hasn't been started.
	// * The command timed out.
	// * The shell exited, regardless of exit code.
	Run(time.Duration, Commander) error

	// Stop attempts to gracefully stop the shell.
	// It sends the given command to the shell (presumably something
	// like `quit` or `exit`), or just EOF if the command is empty.
	// Stop, unlike Run, treats the shell exiting with a zero status
	// as a success.
	// Errors:
	// * The shell wasn't started or is currently running.
	// * The shell's subprocess didn't finish in the time allotted.
	// * The shell exited with non-zero status.
	Stop(time.Duration, string) error
}

Shell manages a shell program, adding value by allowing the output from different commands to be handled differently.

A Shell is in one of these states:

off: no shell subprocess running.

  • Shell freshly created, Start not yet called.
  • Stop called and finished.
  • An error encountered in any call meaning that the subprocess had to be abandoned (must call Start again).
  • Ok to Start, but not Run or Stop.

idle: shell subprocess healthy and awaiting input.

  • A call to Start finished without error.
  • A call to Run finished without error.
  • Ok to call Run or Stop, but not Start.

All Shell calls block until they finish or their deadlines expire.

func NewShell 

func NewShell(p Parameters) Shell

NewShell returns a new Shell built from Parameters in the off state.

func NewShellRaw 

func NewShellRaw(f channelsMakerF, so Sentinel, se Sentinel) Shell

NewShellRaw returns a new Shell in the off state, built from the given channels-maker function and the two sentinels. Allows testing with injected channels instead of a real shell subprocess.

Source Files

View all Source files

Directories

Path Synopsis
conch module
scripter module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.