README
¶
scripter 
Package scripter lets one script a command
shell in Go as if a human were running it.
The package separates the problem of orchestrating shell execution from the problem of generating a shell command and parsing the shell's response to said command.
This package hides and solves the first problem (via [Shell]),
and makes the latter easy to do via Go
implementations of Commander.
Usage
Roughly:
sh := NewShell(Parameters{
ChParams: channeler.ChParams{Path: "/bin/sh"},
SentinelOut: Sentinel{
C: "echo " + unlikelyWord,
V: unlikelyWord,
},
})
assertNoErr(sh.Start(timeOut))
assertNoErr(sh.Run(timeOut, commander1))
// consult commander1 getters for whatever purpose,
// optionally use the results to define commander2.
assertNoErr(sh.Run(timeOut, commander2))
// consult commander2, etc.
assertNoErr(sh.Stop(timeOut, ""))
Assumptions
Shell behavior
A shell is any program that accepts newline terminated
commands, e.g. bash, and emits lines of output on
stdOut and stdErr.
The purpose of a shell, as opposed to a single-purpose program that doesn't prompt for commands, is to allow state that endures over multiple commands.
State contains things like authentication, authorization, secrets obtained from vaults, caches, database connections, etc.
A shell lets a user pay to build that state once, then run many commands in the context of that state.
Commands influence commands
There must be an opportunity to examine the output of command n before issuing command n+1.
The choice of command n+1 or its arguments may be influenced by the output of command n.
Command generation and parsing best live together
The code that parses a command's output should live close to the code that generates the command. The parser should have access to command arguments and flags so that it knows what's supposed to happen.
All a Go author need do is implement the Commander
interface, then pass instances of the implementation to
the Run method of a [Shell]. When a Run call
returns, the Commander instance can be consulted.
A commander can offer any number of methods yielding
validated data acquired from the shell; it can be
viewed as a shell visitor.
A Commander can be tested in isolation
(without the need of a shell)
for its ability to compose a command and parse the output
expected from that command.
Unreliable prompts, unreliable newlines, and command blocks
A human knows that a shell has completed command n and is awaiting command n+1 because they see a prompt following the output of command n. Usually, but not always, the prompt is on a new line.
But in a scripting context, prompts with newlines are unreliable.
When running a shell as a subprocess,
e.g. as part of a pipe, the shell can see
that stdIn is not a tty, and won't
issue a prompt to avoid pipe contamination.
Sometimes command output can accidentally contain data that matches the prompt, making the prompt useless as an output delimiter.
Sometimes a shell will intentionally suppress
newline on command completion, e.g. base64 -d, echo -n.
Most importantly, sometimes a user wants to inject a command block, multiple commands with embedded newlines, as a single unit, not caring to know when individual commands in the block finish. Only the whole set matters. This can happen when blindly executing command blocks from some unknown source, e.g. fenced code blocks embedded in markdown documentation.
For these reasons, a Shell cannot
depend on prompts and newlines to unambiguously
distinguish the data from commands
n-1, n and n+1 on stdOut and stdErr.
So instead of relying on prompts or newlines,
[Shell] relies on a Sentinel.
Sentinels
stdOut
A Shell demands the existence of
a sentinel command for stdOut.
Such a command
- does very little,
- does it quickly,
- has deterministic, newline terminated output on
stdOut.
Example:
$ echo "rumpelstiltskin" rumpelstiltskin
Commands that print a program's version, a help message,
and/or a copyright message are good candidates for
sentinel commands on the stdOut stream.
The unambiguously recognizable output of a sentinel command called the sentinel value.
A Sentinel holds a {command, value} pair.
stdErr
Likewise, a Shell needs a sentinel
command for stdErr.
This command differs from the stdOut sentinel only
in that its output goes to stdErr.
Usually a shell will complain to stdErr if it sees
a command it doesn't recognize, meaning that
an unrecognized command is also a good sentinel
command for stdErr.
Example:
$ rumpelstiltskin rumpelstiltskin: command not found
Command results
The outcome of asking a shell to run a command is one of the following:
- crash - shell exits with non-zero status.
- exit - shell exits with zero status.
If this happens unintentionally, it's treated as a crash. - timeout - shell fails to finish the command in a given time period.
The shell is assumed to be unusable, and should be killed. - ready - shell runs the command within the given time period and is ready to accept
another command.
The command can be consulted for whatever results it parsed and saved.
Documentation
¶
Overview ¶
Package shell lets one script a command shell in Go as if a human were running it. See README.md.
Example (BasicRun) ¶
An error free run using the (locally defined) conch shell.
sh := NewShell(Parameters{
ChParams: channeler.ChParams{
WorkingDir: "../conch",
Path: "go",
Args: []string{
"run", ".",
// the prompt goes to stdout, so get rid of it in tests.
"--disable-prompt",
}},
SentinelOut: sentinelVersion,
})
assertNoErr(sh.Start(timeOutShort))
assertNoErr(sh.Run(timeOutShort, newPrintingCommander("query limit 3")))
assertNoErr(sh.Stop(timeOutShort, ""))
Output: out: Cempedak_|_Bamberga_|_4_|_00000000000000000000000000000001 out: Buddha's hand_|_Hermione_|_6_|_00000000000000000000000000000002 out: African cucumber_|_Ursula_|_6_|_00000000000000000000000000000003
Example (BinSh) ¶
An example using /bin/sh, a shell that's available on most platforms.
sh := NewShell(Parameters{
ChParams: channeler.ChParams{
Path: "/bin/sh",
},
SentinelOut: Sentinel{
C: "echo " + unlikelyWord,
V: unlikelyWord,
},
})
assertNoErr(sh.Start(timeOutShort))
assertNoErr(sh.Run(timeOutShort,
newPrintingCommander(`
echo alpha
which cat
`)))
assertNoErr(sh.Run(timeOutShort,
newPrintingCommander(`
echo beta
which find
`,
)))
assertNoErr(sh.Stop(timeOutShort, ""))
Output: out: alpha out: /usr/bin/cat out: beta out: /usr/bin/find
Example (SubprocessFailOnStartup) ¶
A shell that crashes on startup.
sh := NewShell(Parameters{
ChParams: channeler.ChParams{
WorkingDir: "../conch",
Path: "go",
Args: []string{
"run", ".",
"--fail-on-startup",
},
},
SentinelOut: sentinelVersion,
})
err := sh.Start(timeOutShort)
fmt.Println(err.Error())
Output: stdOut closed before sentinel "v1.2.3" found
Example (SubprocessNonSurvivableError) ¶
A shell that crashes, and is then restarted.
sh := NewShell(Parameters{
ChParams: channeler.ChParams{
WorkingDir: "../conch",
Path: "go",
Args: []string{
"run", ".",
"--disable-prompt",
// Using this means any error will cause process exit.
// So we cannot use an errSentinel, as it by definition causes an error.
"--exit-on-error",
"--row-to-error-on", "4",
}},
SentinelOut: sentinelVersion,
})
assertNoErr(sh.Start(timeOutShort))
cmdr := newPrintingCommander("query limit 3")
// The following yields three lines.
assertNoErr(sh.Run(timeOutShort, cmdr))
// Query again, but ask for a row beyond the row that
// triggers a DB error.
// Since flag "exit-on-error" is enabled, this causes the CLI to die.
cmdr.Cmd = "query limit 5"
err := sh.Run(timeOutShort, cmdr)
assertErr(err)
fmt.Println(err.Error())
err = sh.Stop(timeOutShort, "")
assertErr(err)
fmt.Println(err.Error())
// Start it up again and issue a command to show that it works.
assertNoErr(sh.Start(timeOutShort))
cmdr.Cmd = "query limit 2"
assertNoErr(sh.Run(timeOutShort, cmdr))
assertNoErr(sh.Stop(timeOutShort, ""))
Output: out: Cempedak_|_Bamberga_|_4_|_00000000000000000000000000000001 out: Buddha's hand_|_Hermione_|_6_|_00000000000000000000000000000002 out: African cucumber_|_Ursula_|_6_|_00000000000000000000000000000003 out: Currant_|_Alauda_|_5_|_00000000000000000000000000000001 out: Banana_|_Egeria_|_5_|_00000000000000000000000000000002 out: Bilberry_|_Interamnia_|_2_|_00000000000000000000000000000003 stdOut closed before sentinel "v1.2.3" found stop called, but shell not started yet out: Cempedak_|_Bamberga_|_4_|_00000000000000000000000000000001 out: Buddha's hand_|_Hermione_|_6_|_00000000000000000000000000000002
Example (SubprocessSurvivableError) ¶
A shell spits output to stderr.
sh := NewShell(Parameters{
ChParams: channeler.ChParams{
WorkingDir: "../conch",
Path: "go",
Args: []string{
"run", ".",
"--disable-prompt",
"--row-to-error-on", "4",
}},
SentinelOut: sentinelVersion,
SentinelErr: sentinelUnknownCommand,
})
assertNoErr(sh.Start(timeOutShort))
cmdr := newPrintingCommander("query limit 3")
// The following yields three lines.
assertNoErr(sh.Run(timeOutShort, cmdr))
// Query again, but ask for a row beyond the row that
// triggers a DB error.
// Because of the nature of output streams,
// there's no way to know
// when the error will show up in the combined output.
// It might come out first, last, or anywhere in the
// middle relative to lines from stdOut,
// so this test must not be fragile to the order.
// This will yield three "good lines", and one error line.
cmdr.Cmd = "query limit 7"
assertNoErr(sh.Run(timeOutShort, cmdr))
// Yields two lines.
cmdr.Cmd = "query limit 2"
assertNoErr(sh.Run(timeOutShort, cmdr))
assertNoErr(sh.Stop(timeOutShort, ""))
// There should be nine (3 + 3 + 1 + 2) lines in the output.
Output: out: Cempedak_|_Bamberga_|_4_|_00000000000000000000000000000001 out: Buddha's hand_|_Hermione_|_6_|_00000000000000000000000000000002 out: African cucumber_|_Ursula_|_6_|_00000000000000000000000000000003 out: Currant_|_Alauda_|_5_|_00000000000000000000000000000001 out: Banana_|_Egeria_|_5_|_00000000000000000000000000000002 out: Bilberry_|_Interamnia_|_2_|_00000000000000000000000000000003 err: error! touching row 4 triggers this error out: Cherimoya_|_Palma_|_6_|_00000000000000000000000000000001 out: Abiu_|_Metis_|_3_|_00000000000000000000000000000002
Example (SubprocessTakesTooLong) ¶
A command takes too long and fails as a result.
sh := NewShell(Parameters{
ChParams: channeler.ChParams{
WorkingDir: "../conch",
Path: "go",
Args: []string{
"run", ".",
"--disable-prompt",
}},
SentinelOut: sentinelVersion,
})
assertNoErr(sh.Start(timeOutShort))
// Send in a sleep command that consumes twice the timeOut.
err := sh.Run(
timeOutShort,
newPrintingCommander("sleep "+(2*timeOutShort).String()))
fmt.Println(err.Error())
Output: running "sleep 1.6s", no sentinels found after 800ms
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var DevNull = &devNullDevice{}
DevNull is an io.WriteCloser that does nothing.
Functions ¶
func VerboseLoggingDisable ¶
func VerboseLoggingDisable()
VerboseLoggingDisable disables detailed logging.
func VerboseLoggingEnable ¶
func VerboseLoggingEnable()
VerboseLoggingEnable enables detailed logging.
Types ¶
type Commander ¶
type Commander interface {
// Command is the actual command to issue to the shell.
Command() string
// ParseOut will be written with whatever comes out of
// the shell's stdOut as the result of issuing Command.
// Close will be called when the executor believes that
// all output has been obtained.
ParseOut() io.WriteCloser
// ParseErr is like ParseOut, except stdErr is used instead of stdOut.
ParseErr() io.WriteCloser
}
Commander knows a CLI command, and knows how to parse the command's output.
type DiscardCommander ¶
type DiscardCommander struct {
C string
}
DiscardCommander discards everything from its parsers.
func (*DiscardCommander) Command ¶
func (c *DiscardCommander) Command() string
func (*DiscardCommander) ParseErr ¶
func (c *DiscardCommander) ParseErr() io.WriteCloser
func (*DiscardCommander) ParseOut ¶
func (c *DiscardCommander) ParseOut() io.WriteCloser
type LabellingCommander ¶
type LabellingCommander struct {
C string
// contains filtered or unexported fields
}
LabellingCommander passes subprocess output from stdOut and stdErr to the main process' stdOut, adding a prefix to make a distinction.
func NewLabellingCommander ¶
func NewLabellingCommander(c string) *LabellingCommander
NewLabellingCommander returns an instance of LabellingCommander.
func (*LabellingCommander) Command ¶
func (c *LabellingCommander) Command() string
func (*LabellingCommander) ParseErr ¶
func (c *LabellingCommander) ParseErr() io.WriteCloser
func (*LabellingCommander) ParseOut ¶
func (c *LabellingCommander) ParseOut() io.WriteCloser
type LineAbsorber ¶
type LineAbsorber struct {
// contains filtered or unexported fields
}
LineAbsorber remembers all the non-empty lines it sees.
func (*LineAbsorber) Close ¶
func (ab *LineAbsorber) Close() error
func (*LineAbsorber) Lines ¶
func (ab *LineAbsorber) Lines() []string
func (*LineAbsorber) Reset ¶
func (ab *LineAbsorber) Reset()
type Parameters ¶
type Parameters struct {
channeler.ChParams
// SentinelOut holds the command sent to the shell after every
// command other than the exit command.
// SentinelOut is used to be sure that output generated in the
// course of running command N is swept up and accounted for
// before looking for output from command N+1.
SentinelOut Sentinel
// SentinelErr is a command that intentionally triggers output
// on stderr, e.g. a misspelled command, a command with a non-extant
// flag - something that doesn't cause any real trouble. If non-empty,
// this is issued after every command other than the exit command,
// either before or after issuing the OutSentinel command.
// SentinelErr is used to be sure that any errors generated in the
// course of running command N are swept up and accounted for before
// looking for errors from command N+1.
SentinelErr Sentinel
}
Parameters is a bag of parameters for Executor. See individual fields for their explanation.
func (*Parameters) Validate ¶
func (p *Parameters) Validate() error
Validate returns an error if there's a problem in the Parameters.
type PassThruCommander ¶
type PassThruCommander struct{ C string }
PassThruCommander forwards data to the current process stdOut and stdErr.
func (*PassThruCommander) Command ¶
func (c *PassThruCommander) Command() string
func (*PassThruCommander) ParseErr ¶
func (c *PassThruCommander) ParseErr() io.WriteCloser
func (*PassThruCommander) ParseOut ¶
func (c *PassThruCommander) ParseOut() io.WriteCloser
type RecallCommander ¶
type RecallCommander struct {
C string
// contains filtered or unexported fields
}
RecallCommander remembers all the non-empty lines it sees.
func NewRecallCommander ¶
func NewRecallCommander(c string) *RecallCommander
NewRecallCommander returns an instance of RecallCommander.
func (*RecallCommander) Command ¶
func (c *RecallCommander) Command() string
func (*RecallCommander) DataErr ¶
func (c *RecallCommander) DataErr() []string
func (*RecallCommander) DataOut ¶
func (c *RecallCommander) DataOut() []string
func (*RecallCommander) ParseErr ¶
func (c *RecallCommander) ParseErr() io.WriteCloser
func (*RecallCommander) ParseOut ¶
func (c *RecallCommander) ParseOut() io.WriteCloser
func (*RecallCommander) Reset ¶
func (c *RecallCommander) Reset()
type Sentinel ¶
type Sentinel struct {
// C is a command that should do very little, do it quickly,
// and have deterministic, newline terminated output.
C string
// V is the expected value from Command.
// Sentinel value comparisons are only made when a
// newline is encountered in the output stream,
// and then only working backwards from that newline.
// E.g. the value "foo" will match "foo\n" in the
// output stream, but will not match "foo bar".
V string
}
Sentinel holds a {command, value} pair.
A Sentinel is used to recognize the end of command output on a stream. Examples:
Command: echo pink elephants dance Value: pink elephants dance Command: version Value: v1.2.3 Command: rumpelstiltskin Value: rumpelstiltskin: command not found
type Shell ¶
type Shell interface {
// Start synchronously starts the shell.
// It assures that the shell runs and that the sentinels work
// before their first use in the Run method.
// Errors:
// * The shell was already started.
// * Something's wrong in the Parameters, e.g. the shell program
// cannot be found.
// * The sentinels failed to work in the time allotted.
Start(time.Duration) error
// Run sends the command in Commander to the shell, and
// waits for it to complete. It returns an error if
// there was some infrastructure problem or if the
// command timed out because no sentinels were detected
// in the time given.
// An error here means that the shell is dead, and in
// need of fresh call to Start.
// Errors:
// * The shell hasn't been started.
// * The command timed out.
// * The shell exited, regardless of exit code.
Run(time.Duration, Commander) error
// Stop attempts to gracefully stop the shell.
// It sends the given command to the shell (presumably something
// like `quit` or `exit`), or just EOF if the command is empty.
// Stop, unlike Run, treats the shell exiting with a zero status
// as a success.
// Errors:
// * The shell wasn't started or is currently running.
// * The shell's subprocess didn't finish in the time allotted.
// * The shell exited with non-zero status.
Stop(time.Duration, string) error
}
Shell manages a shell program, adding value by allowing the output from different commands to be handled differently.
A Shell is in one of these states:
off: no shell subprocess running.
- Shell freshly created, Start not yet called.
- Stop called and finished.
- An error encountered in any call meaning that the subprocess had to be abandoned (must call Start again).
- Ok to Start, but not Run or Stop.
idle: shell subprocess healthy and awaiting input.
- A call to Start finished without error.
- A call to Run finished without error.
- Ok to call Run or Stop, but not Start.
All Shell calls block until they finish or their deadlines expire.
func NewShell ¶
func NewShell(p Parameters) Shell
NewShell returns a new Shell built from Parameters in the off state.
func NewShellRaw ¶
NewShellRaw returns a new Shell in the off state, built from the given channels-maker function and the two sentinels. Allows testing with injected channels instead of a real shell subprocess.
Source Files