Redirected from 
README
¶
wr - workflow runner
wr is a workflow runner. You use it to run the commands in your workflow easily, automatically, reliably, with repeatability, and while making optimal use of your available computing resources.
wr is implemented as a polling-free in-memory job queue with an on-disk acid transactional embedded database, written in go.
Its main benefits over other software workflow management systems are its very low latency and overhead, its high performance at scale, its real-time status updates with a view on all your workflows on one screen, its permanent searchable history of all the commands you have ever run, and its "live" dependencies enabling easy automation of on-going projects.
Furthermore, wr has best-in-class support for OpenStack, providing incredibly easy deployment and auto-scaling without you having to know anything about OpenStack. It also has built-in support for mounting S3-like object stores, providing an easy way of running commands against remote files whilst enjoying high performance.
Download
Alternatively, build it yourself (see go.mod for the minimum version of go required):
git clone https://github.com/VertebrateResequencing/wr.git
cd wr
make
The wr executable should now be in $HOME/go/bin.
Documentation
Complete usage information is available using the -h option to wr and its
sub-commands.
Guided usage, tips, notes and tutorials are available here: https://workflow-runner.readthedocs.io/
Documentation
¶
Overview ¶
Package main is a stub for wr's command line interface, with the actual implementation in the cmd package.
wr is a workflow runner. You use it to run the commands in your workflow easily, automatically, reliably, with repeatability, and while making optimal use of your available computing resources.
wr is implemented as a polling-free in-memory job queue with an on-disk acid transactional embedded database, written in go.
Its main benefits over other software workflow management systems are its very low latency and overhead, its high performance at scale, its real-time status updates with a view on all your workflows on one screen, its permanent searchable history of all the commands you have ever run, and its "live" dependencies enabling easy automation of on-going projects.
Basics ¶
Start up the manager daemon, which gives you a url you can view the web interface on:
wr manager start -s local
In addition to the "local" scheduler, which will run your commands on all available cores of the local machine, you can also have it run your commands on your LSF cluster or in your OpenStack environment (where it will scale the number of servers needed up and down automatically).
Now, stick the commands you want to run in a text file and:
wr add -f myCommands.txt
Arbitrarily complex workflows can be formed by specifying command dependencies. Use the --help option of `wr add` for details.
Package Overview ¶
wr's core is implemented in the queue package. This is the in-memory job queue that holds commands that still need to be run. Its multiple sub-queues enable certain guarantees: a given command will only get run by a single client at any one time; if a client dies, the command will get run by another client instead; if a command cannot be run, it is buried until the user takes action; if a command has a dependency, it won't run until its dependencies are complete.
The jobqueue package provides client+server code for interacting with the in-memory queue from the queue package, and by storing all new commands in an on-disk database, provides an additional guarantee: that (dynamic) workflows won't break because a job that was added got "lost" before it got run. It also retains all completed jobs, enabling searching through of past workflows and allowing for "live" dependencies, triggering the rerunning of previously completed commands if their dependencies change.
The jobqueue package is also what actually does the main "work" of the system: the server component knows how many commands need to be run and what their resource requirements (memory, time, cpus etc.) are, and submits the appropriate number of jobqueue runner clients to the job scheduler.
The jobqueue/scheduler package has the scheduler-specific code that ensures that these runner clients get run on the configured system in the most efficient way possible. Eg. for LSF, if we have 10 commands that need 2GB of memory to run, we will submit a job array of size 10 with 2GB of memory reservation to LSF. The most limited (and therefore potentially least contended) queue capable of running the commands will be chosen. For OpenStack, the cheapest server (in terms of cores and memory) that can run the commands will be spawned, and once there is no more work to do on those servers, they get terminated to free up resources.
The cloud package implements methods for interacting with cloud environments such as OpenStack. The corresponding jobqueue/scheduler package uses these methods to do their work.
The static subdirectory contains the html, css and javascript needed for the web interface. See jobqueue/serverWebI.go for how the web interface backend is implemented.
The internal package contains general utility functions, and most notably config.go holds the code for how the command line interface deals with config options.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
package backoff is used to implement waiting for increasing periods of time between attempts at doing something.
|
package backoff is used to implement waiting for increasing periods of time between attempts at doing something. |
|
mock
package mock contains a mock implementation of backoff.Sleeper
|
package mock contains a mock implementation of backoff.Sleeper |
|
time
package time contains a real time-based implementation of backoff.Sleeper.
|
package time contains a real time-based implementation of backoff.Sleeper. |
|
Package bsub parsers bsub resource strings for easy manipulation.
|
Package bsub parsers bsub resource strings for easy manipulation. |
|
Package client provides Scheduler, which a Go program uses to add commands to the queue of a running wr manager, wait for them, and find them again.
|
Package client provides Scheduler, which a Go program uses to add commands to the queue of a running wr manager, wait for them, and find them again. |
|
package clog is used to do contextual logging with a global logger.
|
package clog is used to do contextual logging with a global logger. |
|
Package cloud provides functions to interact with cloud providers, used to create cloud resources so that you can spawn servers, then delete those resources when you're done.
|
Package cloud provides functions to interact with cloud providers, used to create cloud resources so that you can spawn servers, then delete those resources when you're done. |
|
Package cmd implements wr's command line interface.
|
Package cmd implements wr's command line interface. |
|
wr-testsuite
command
|
|
|
developers
|
|
|
prodsim
command
Command prodsim drives an ISOLATED wr manager with a production-shaped workload made only of harmless commands, and samples the manager's health while it does.
|
Command prodsim drives an ISOLATED wr manager with a production-shaped workload made only of harmless commands, and samples the manager's health while it does. |
|
soak/bboltexp
command
Command bboltexp is soak tooling, not part of wr: it opens a COPY of a manager database the way the manager does (map freelist, NoFreelistSync, a large InitialMmapSize) and either checks it or churns it, so a freelist panic seen through the fusestall mount can be compared with the same file opened directly.
|
Command bboltexp is soak tooling, not part of wr: it opens a COPY of a manager database the way the manager does (map freelist, NoFreelistSync, a large InitialMmapSize) and either checks it or churns it, so a freelist panic seen through the fusestall mount can be compared with the same file opened directly. |
|
soak/dbstart
command
Command dbstart is soak tooling, not part of wr: it opens a STOPPED manager's bolt DB read-only and prints one line per psimjob.sh job in the complete and live buckets (and each relbury.sh job, as kind relb): bucket, key, kind, id, state, exit code, attempts, host, pid, and start and end times in unix ms.
|
Command dbstart is soak tooling, not part of wr: it opens a STOPPED manager's bolt DB read-only and prints one line per psimjob.sh job in the complete and live buckets (and each relbury.sh job, as kind relb): bucket, key, kind, id, state, exit code, attempts, host, pid, and start and end times in unix ms. |
|
soak/fusestall
command
Command fusestall is soak tooling, not part of wr: a FUSE loopback mount of a directory whose fsyncs block while a control file exists, so a manager whose database is opened through the mount has its commits stalled on demand, as a full or hung file system would, and resumes when the file is removed.
|
Command fusestall is soak tooling, not part of wr: a FUSE loopback mount of a directory whose fsyncs block while a control file exists, so a manager whose database is opened through the mount has its commits stalled on demand, as a full or hung file system would, and resumes when the file is removed. |
|
soak/psinspect
command
Command psinspect is soak tooling, not part of wr: it prints, for every job in the given rep groups of the isolated manager that WR_CONFIG_DIR names, the psimjob.sh kind and id from its command with its state, exit code, attempts and fail reason, so run markers can be joined to the manager's view of each job.
|
Command psinspect is soak tooling, not part of wr: it prints, for every job in the given rep groups of the isolated manager that WR_CONFIG_DIR names, the psimjob.sh kind and id from its command with its state, exit code, attempts and fail reason, so run markers can be joined to the manager's view of each job. |
|
package fs is for interacting with file systems.
|
package fs is for interacting with file systems. |
|
local
package local contains a local implementation of VolumeUsageCalculator.
|
package local contains a local implementation of VolumeUsageCalculator. |
|
mock
package mock contains a mock implementation of VolumeUsageCalculator.
|
package mock contains a mock implementation of VolumeUsageCalculator. |
|
Package internal houses code for wr's general utility functions.
|
Package internal houses code for wr's general utility functions. |
|
mangostlstcp
Package mangostlstcp registers a race-clean TLS over TCP transport for mangos.
|
Package mangostlstcp registers a race-clean TLS over TCP transport for mangos. |
|
publishexit
Package publishexit holds how a starting jobqueue server ends the process when it cannot make itself reachable.
|
Package publishexit holds how a starting jobqueue server ends the process when it cannot make itself reachable. |
|
replyproxy
Package replyproxy provides a TCP proxy for tests that need a client's connection to a manager to lose or delay the manager's replies, as when a manager stops after acting on a request but before its reply arrives.
|
Package replyproxy provides a TCP proxy for tests that need a client's connection to a manager to lose or delay the manager's replies, as when a manager stops after acting on a request but before its reply arrives. |
|
testcerts
Package testcerts gives tests a manager's TLS files without paying for new RSA keys every time.
|
Package testcerts gives tests a manager's TLS files without paying for new RSA keys every time. |
|
testimage
Package testimage holds the container image that wr's tests run under a real docker daemon, so that every such test pulls, and so needs, the same one.
|
Package testimage holds the container image that wr's tests run under a real docker daemon, so that every such test pulls, and so needs, the same one. |
|
Package jobqueue provides server/client functions to interact with the queue structure provided by the queue package over a network.
|
Package jobqueue provides server/client functions to interact with the queue structure provided by the queue package over a network. |
|
scheduler
Package scheduler lets the jobqueue server interact with the configured job scheduler (if any) to submit jobqueue runner clients and have them run on a compute cluster (or local machine).
|
Package scheduler lets the jobqueue server interact with the configured job scheduler (if any) to submit jobqueue runner clients and have them run on a compute cluster (or local machine). |
|
Package limiter provides a way of limiting the number of something that belongs to one or more limit groups.
|
Package limiter provides a way of limiting the number of something that belongs to one or more limit groups. |
|
network
|
|
|
Package queue provides an in-memory queue structure suitable for the safe and low latency implementation of a real job queue.
|
Package queue provides an in-memory queue structure suitable for the safe and low latency implementation of a real job queue. |
|
retry is used to keep trying something until it works.
|
retry is used to keep trying something until it works. |
|
Package rp ("resource protector") provides functions that help control access to some limited resource.
|
Package rp ("resource protector") provides functions that help control access to some limited resource. |