README

murex

GoDoc Go Report Card CodeBuild CircleCI codecov

About murex

murex is a shell, like bash / zsh / fish / etc. It follows a similar syntax to POSIX shells like Bash however supports more advanced features than you'd typically expect from a $SHELL.

It aims to be similar enough to traditional shells that you can retain most of your muscle memory, while not being afraid to make breaking changes where "bash-isms" lead to unreadable, hard to maintain, or unsafe code.

murex is designed for DevOps productivity so it isn't suited for high performance workloads beyond what you'd typically run in Bash (eg pipelines forked as concurrent processes).

A non-exhaustive list features would include:

  • Testing frameworks baked right into the language itself
  • Smarter handling of errors (for example try/catch blocks, line numbers included in error messages, errors optionally highlighted in red, etc)
  • Support for typed pipelines - which can be used to work with complex data formats like JSON natively. But also the ability to override or even ignore typed data entirely so it works transparently with standard UNIX tools too
  • Lots of data-mangling tools baked into the shell
  • Parses man pages for a richer user experience
  • In-line spell checking
  • Optional support for event-driven programming
  • Plus every aspect of the shell is easily extendable, inspectable and can be managed via murex's own package manager

The language employs a relatively simple syntax modelled loosely on functional and stack-based programming paradigms (albeit without the LISP-style nested parentheses that scare a lot of developers) while maintaining compatibility with POSIX-like shells (eg Bash) wherever sensible. For example, a program structure could look like the following:

command | command | [ index ] | if { command }

However where murex differs is that, while the pipe token, | is supported for compatibility with existing shells, murex idioms prefer -> used for readability as it clearly demonstrates the direction of the data flow. eg

command -> command -> [ index ] -> if { command }

The language supports multiple data types natively; such as JSON, YAML, CSV, S-Expressions and even loosely tabulated terminal output (eg ps, ls -l). This makes passing data through the pipeline and parsing output easier when dealing with more complex arrangements of data than a simple byte stream in traditional shells like Bash.

Interactive shell

Aside from murex being carefully designed with scripting in mind, the interactive shell itself is also built around productivity. To achieve this, we wrote our own readline library. Below is an example of that library in use:

asciicast

The above demo includes the following features of murex's bespoke readline library:

  • hint text - blue status text below the prompt (the colour is configurable)
  • syntax highlighting (albeit there isn’t much syntax to highlight in the example). This can also be turned off if your preference is to have colours disabled
  • tab-completion in gridded mode (seen when typing cd)
  • tab-completion in list view (seen when selecting a process name to kill where the process ID was substituted when selected)
  • regex searching through the tab-completion suggestions (seen in both cd and kill - enabled by pressing [CTRL+f])
  • line editing using $EDITOR (vi in the example - enabled by pressing [ESC] followed by [v])
  • readline’s warning before pasting multiple lines of data into the buffer and the preview option that’s available as part of the aforementioned warning
  • and VIM keys (enabled by pressing [ESC])

Concise yet predictable

Despite the amount of features added to shell, we have tried to keep the amount of "magic" to a minimum and follow a pretty standard structure so the language is predictable. However there are times when a little magic goes a long way. For example murex's support for complex data objects of differing formats is managed in the pipeline so you don't need to think about the data format when querying data from them.

open: file.csv  -> [ column_name ] # returns specific columns (or rows) in CSV file
open: file.json -> [ index ]       # returns specific items from JSON

The index function ([) alters its matching algorithm depending on the piped data type and open sets the data type depending on the file extension or MIME type.

Sometimes you will want less guesswork or just the robustness of a forced behavior. On those occasions you can remove one layer of magic by casting the data type:

open: file.txt -> cast csv  -> [ column_name ]
open: file.txt -> cast json -> [ index ]

This awareness of data structures is also utilised in foreach (which will cycle through each index in an array) and formap (key/value iteration against complex objects). See GUIDE.control-structures for more details on these and other control structures.

More robust scripts / shell one liners

murex employs a few methods to make shell scripting more robust:

Bash, for all it's power, is littered with hidden traps. The aim of murex is to address as many of them as we can without taking the flexibility or power away from the interactive command line. This is achieved through a couple of key concepts:

Everything is a function

The biggest breaking change from regular shells is how globbing isn't expanded by the shell by default. This is instead done by inlining functions as arrays:

# Bash
ls -l *.go

# Murex
ls -l @{g *.go}

The advantage of murex's method is that we can now offer other ways of matching file system objects that follows the same idiomatic pattern:

# Match files by regexp pattern
ls -l @{rx \.go$}

# Match only directories
ls -l @{f +d}

(more information on g, rx and f are available in GUIDE.quick-start).

However there will be occasions when you just want an inlined expansion (eg when using an interactive shell) and that can be achieved via the @g command prefix:

@g ls -l *.go
Powerful autocompletion

murex takes a slightly different approach to command line autocompletion, both from a usability perspective as well as defining completion rules.

Inspired by IDEs, murex queries man pages directly for flags as well as "tooltip" descriptions. Custom completions are defined via JSON meaning simple commands are much easier to define and complex commands can still fallback to using dynamic shell code just like they are in other shells.

This makes it easier to write completion rules as well as making the code more readable. An example of gits autocompletion definition:

private git-branch {
    # returns git branches and removes the current one from the list
    git branch -> [ :0 ] -> !match *
}

autocomplete set git { [{
    # define the top level flags
    "Flags": [
        "clone", "init", "add", "mv", "reset", "rm", "bisect", "grep",
        "log", "show", "status", "branch", "checkout", "commit", "diff",
        "merge", "rebase", "tag", "fetch", "pull", "push", "stash"
    ],

    # specify what values those flags support
    "FlagValues": {
        "init": [{
            "Flags": [ "--bare" ]
        }],
        "add": [{
            "IncFiles": true,
            "AllowMultiple": true
        }],
        "mv": [{
            "IncFiles": true
        }],
        "rm": [{
            "IncFiles": true,
            "AllowMultiple": true
        }],
        "checkout": [{
            "Dynamic": ({ git-branch }),
            "Flags": [ "-b" ]
        }],
        "merge": [{
            "Dynamic": ({ git-branch })
        }]
    }
}] }

murex also supports several different styles of completion suggestion "popups" to cater for different scenarios (demo above) as well as built in support for jumping to files within nested directories quickly and easily:

cat [ctrl+f]app.g[return]
# same as typing: cat config/app.go
Error handling

Like traditional shells, murex is verbose with errors by default with options to mute them. However murex also supports cleaner decision structures for when you want you want errors captured in a useful way:

try {
    # do something
}
catch {
    err: "Could not perform action"
}

As well as a saner if syntax:

# compare two strings
if { = `foo`==`bar` } then {
    out: "`foo` matched `bar`"
}

# check if command ran successfully
!if { foobar } then {
    err: "`foobar` could not be run"
}
Test and debugging frameworks

Unlike traditional shells, murex is designed with a test and debugging modes baked into the shell language. This means you can write tests against your shell scripts as part of the shell scripts themselves.

For example:

function: hello-world {
    test: define example {
        "StdoutRegex": (^Hello World$)
    }

    out: <test_example> "Hello Earth"
}

test: run { hello-world }

...will output:

Hello Earth
 Status  Definition Function                                           Line Col. Message
[FAILED] example    out                                                5    9    stdout: regexp did not match 'Hello Earth'

If test mode isn't enabled then any test commands are skipped without being executed so you can liberally include test cases throughout your functions without worrying about any performance impact.

murex also supports unit tests

For example:

test: unit function aliases {
    "PreBlock": ({
        alias ALIAS_UNIT_TEST=example param1 param2 param3
    }),
    "StdoutRegex": "([- _0-9a-zA-Z]+ => .*?\n)+",
    "StdoutType": "str",
    "PostBlock": ({
        !alias ALIAS_UNIT_TEST
    })
}

function: aliases {
    # Output the aliases in human readable format
    runtime: --aliases -> formap: name alias {
        $name -> sprintf: "%10s => ${esccli @alias}\n"
    } -> cast: str
}

test: run aliases

...will output:

 Status  Definition Function                                           Line Col. Message
[PASSED] (unit)     aliases                                            13   1    All test conditions were met

Language guides

  1. GUIDE.syntax is recommended first as it gives an overview if the shell scripting languages syntax and data types.

  2. GUIDE.type-system describes murex's type system. Most of the time you will not need to worry about typing in murex as the shell is designed around productivity.

  3. GUIDE.builtin-functions lists some of the builtin functions available for this shell.

Or if you're already a seasoned Bash developer then you read the Quick Start Guide, GUIDE.quick-start, to jump straight into using murex.

Install instructions

There are various ways you can load murex on to your system. See INSTALL for details.

CI/CD

murex makes heavy use of testing and CI/CD to ensure the latest builds are safe for use.

  1. Git pre-commit and pre-push files exist to help developers catch any regression errors before they even hit the feature branches.

  2. Each and every git push is validated against hundreds of distinct tests and race detectors (run in Circle CI). These tests are run 10 times to shake out any possible timing related bugs.

    Each push to develop and master also creates a new docker container, lmorg/murex:develop and lmorg/murex:latest (for the master branch).

  3. Weekly automated builds are then run against the latest container. These builds are run in AWS CodeBuild and they generate the murex.rocks website and build pre-compiled binaries for download.

Known bugs / TODO

Please see GitHub's issue tracker: https://github.com/lmorg/murex/issues

Documentation

Overview

    Murex is a cross-platform shell like Bash but with greater emphasis on writing safe shell scripts and powerful one-liners while maintaining readability.

    A quick breakdown of the project structure:

    * app is for hardcoded strings like copyright dates
    * builtins contains all the plugins exposed via the languages runtime
    * config is APIs for the runtime config
    * debug is debugging APIs
    * docs is the murex online documentation. Contents in here are autogenerated and some of this will be compiled into the murex executable.
    * examples (does not contain Go code) is murex shell script examples
    * gen is the templates for docgen
    * lang contains the language parsers and runtime environment
    * shell is the code for the interactive shell
    * test is the testing framework
    * utils directory is tools used exclusively within this Go source code
    * vendor contains 3rd party packages used by murex
    

    Directories

    Path Synopsis
    Package builtins is the gatekeeper to the various modules, additional data types and builtin functions within murex The builtins are split into several categories: * core - builtin functions required by murex * events - event hooks for murex code * imports_build - optional builtins you wish to compile * imports_src - optional builtins available to compile (these are just include files that need to be copied to `builtins/imports_build` if you wish to compile them) * optional - builtin functions that might add value to murex but are not required.
    Package builtins is the gatekeeper to the various modules, additional data types and builtin functions within murex The builtins are split into several categories: * core - builtin functions required by murex * events - event hooks for murex code * imports_build - optional builtins you wish to compile * imports_src - optional builtins available to compile (these are just include files that need to be copied to `builtins/imports_build` if you wish to compile them) * optional - builtin functions that might add value to murex but are not required.
    core/arraytools
    Package arraytools provides functions for working with arrays and maps
    Package arraytools provides functions for working with arrays and maps
    core/datatools
    Package datatools provides utilities for manipulating data structures
    Package datatools provides utilities for manipulating data structures
    core/escape
    Package escape provides some handy string escaping utilities
    Package escape provides some handy string escaping utilities
    core/httpclient
    Package httpclient provides useful HTTP functions
    Package httpclient provides useful HTTP functions
    core/management
    Package management provides misc functions for managing your murex runtime environment
    Package management provides misc functions for managing your murex runtime environment
    core/mkarray
    Package mkarray provides functions for rapidly building arrays
    Package mkarray provides functions for rapidly building arrays
    core/openimage
    Package openimage renders bitmap image data on your terminal
    Package openimage renders bitmap image data on your terminal
    core/processes
    Package processes provides core functions for managing processes
    Package processes provides core functions for managing processes
    core/runtime
    Package cmdruntime provides data on murex's runtime state
    Package cmdruntime provides data on murex's runtime state
    core/structs
    Package structs provides code syntax structures for murex
    Package structs provides code syntax structures for murex
    core/textmanip
    Package textmanip provides useful string manipulation functions when working with arrayed data types
    Package textmanip provides useful string manipulation functions when working with arrayed data types
    core/time
    Package time provides some core time based builtins
    Package time provides some core time based builtins
    core/typemgmt
    Package typemgmt provides core functions for managing murex types and variables
    Package typemgmt provides core functions for managing murex types and variables
    events
    Package events provides a basic event framework for murex
    Package events provides a basic event framework for murex
    optional/coreutils
    Package coreutils is an optional package that recreates some basic Linux/UNIX tools for Windows users murex coreutils only covers some of the basic packages you'd expect to find in your POSIX environment and it it not aimed at being a complete like for like rewrite of GNU coreutils (for example).
    Package coreutils is an optional package that recreates some basic Linux/UNIX tools for Windows users murex coreutils only covers some of the basic packages you'd expect to find in your POSIX environment and it it not aimed at being a complete like for like rewrite of GNU coreutils (for example).
    optional/encoders
    Package encoders provides some handy builtins for encoding and decoding streams in various different formats
    Package encoders provides some handy builtins for encoding and decoding streams in various different formats
    optional/inline
    Package inline allows you to inline code from other languages
    Package inline allows you to inline code from other languages
    optional/qr
    Package qrimage generates a QR code image
    Package qrimage generates a QR code image
    optional/select
    Package sqlselect provides the SELECT builtin for working with tables in sqlite3
    Package sqlselect provides the SELECT builtin for working with tables in sqlite3
    optional/time
    Package time provides some optional time based builtins
    Package time provides some optional time based builtins
    pipes/null
    Package null provides the null interface (akin to /dev/null).
    Package null provides the null interface (akin to /dev/null).
    pipes/streams
    Package streams provides the standard streams used by murex and thus is REQUIRED by murex.
    Package streams provides the standard streams used by murex and thus is REQUIRED by murex.
    pipes/term
    Package term provides the TTY STDOUT and STDERR interfaces.
    Package term provides the TTY STDOUT and STDERR interfaces.
    types/apachelogs
    Package apachelogs provides definitions for the `commonlog` and `errorlog` data types
    Package apachelogs provides definitions for the `commonlog` and `errorlog` data types
    types/binary
    Package binary provides definitions for the `bin` data type
    Package binary provides definitions for the `bin` data type
    types/bson
    Package bson provides definitions for the `bson` data type
    Package bson provides definitions for the `bson` data type
    types/columns
    Package columns provides definitions for the column, `column`, data type
    Package columns provides definitions for the column, `column`, data type
    types/csv
    Package csv provides definitions for the `csv` data type
    Package csv provides definitions for the `csv` data type
    types/csv-bad
    Package csvbad provides definitions for the `csv-bad` data type
    Package csvbad provides definitions for the `csv-bad` data type
    types/generic
    Package generic provides definitions for the generic, `*`, data type
    Package generic provides definitions for the generic, `*`, data type
    types/hcl
    Package hcl provides definitions for the `hcl` data type
    Package hcl provides definitions for the `hcl` data type
    types/json
    Package json provides definitions for the `json` data type
    Package json provides definitions for the `json` data type
    types/jsonlines
    Package jsonlines provides definitions for the `jsonlines` data type
    Package jsonlines provides definitions for the `jsonlines` data type
    types/numeric
    Package numeric provides definitions for numeric data types (int, float, num)
    Package numeric provides definitions for numeric data types (int, float, num)
    types/querystring
    Package string provides definitions for the `str` data type
    Package string provides definitions for the `str` data type
    types/sexp
    Package sexp provides definitions for the S-Expression data types: `sexpr` and `csexp`
    Package sexp provides definitions for the S-Expression data types: `sexpr` and `csexp`
    types/string
    Package string provides definitions for the `str` data type
    Package string provides definitions for the `str` data type
    types/toml
    Package toml provides definitions for the `toml` data type
    Package toml provides definitions for the `toml` data type
    types/unicode
    Package unicode provides definitions for the unicode, `utf8`, data type
    Package unicode provides definitions for the unicode, `utf8`, data type
    types/yaml
    Package yaml provides definitions for the `yaml` data type
    Package yaml provides definitions for the `yaml` data type
    Package config provides APIs for managing the shell's runtime config
    Package config provides APIs for managing the shell's runtime config
    defaults
    Package defaults defines the default state for many run time config
    Package defaults defines the default state for many run time config
    profile
    Package profile is used to read the various non-default murex user profiles and modules
    Package profile is used to read the various non-default murex user profiles and modules
    Package debug provides debugging APIs
    Package debug provides debugging APIs
    Package lang provides the parser for the murex shell scripting language
    Package lang provides the parser for the murex shell scripting language
    proc/parameters
    Package parameters provides parsing for language command line parameters within murex
    Package parameters provides parsing for language command line parameters within murex
    proc/pipes
    Package pipes provides runtime information about murex named pipes
    Package pipes provides runtime information about murex named pipes
    proc/runmode
    Package runmode provides constants used to describe the run mode of the murex interpreter
    Package runmode provides constants used to describe the run mode of the murex interpreter
    proc/state
    Package state provides constants used to describe the runtime state of murex functions
    Package state provides constants used to describe the runtime state of murex functions
    ref
    Package ref provides some reference structures required by multiple packages This exists as it's own package to get around cyclic dependencies
    Package ref provides some reference structures required by multiple packages This exists as it's own package to get around cyclic dependencies
    Package shell provides sources for the interactive shell
    Package shell provides sources for the interactive shell
    count
    Package count is used to count the number of test cases run
    Package count is used to count the number of test cases run
    ansi
    Package ansi provides APIs for writing common ASNI escape sequences to the terminal
    Package ansi provides APIs for writing common ASNI escape sequences to the terminal
    cd
    Package cd changes the current working directory and updates the global working
    Package cd changes the current working directory and updates the global working
    consts
    Package consts consolidates common values used throughout the source code
    Package consts consolidates common values used throughout the source code
    counter
    Package counter provides a thread safe counter using mutexes
    Package counter provides a thread safe counter using mutexes
    dedup
    Package dedup provides de-duplication routines
    Package dedup provides de-duplication routines
    docgen/api
    Package docgen contains all the executing code of the docgen CLI but in API form
    Package docgen contains all the executing code of the docgen CLI but in API form
    envvars
    Package envvars provides a more pleasant framework around Go's stdlibs for working with environmental variables
    Package envvars provides a more pleasant framework around Go's stdlibs for working with environmental variables
    escape
    Package escape provides tools for escaping command line snippets of code
    Package escape provides tools for escaping command line snippets of code
    gopath
    Package gopath is used during testing which points test files towards go source code
    Package gopath is used during testing which points test files towards go source code
    home
    Package home is used to return the users home directory
    Package home is used to return the users home directory
    humannumbers
    Package humannumbers is used to return human readable representations of numbers
    Package humannumbers is used to return human readable representations of numbers
    inject
    Package inject is used to insert one string inside another
    Package inject is used to insert one string inside another
    json
    Package json is a custom json parser with an aim to provide more descriptive errors when reading malformed json
    Package json is a custom json parser with an aim to provide more descriptive errors when reading malformed json
    man
    Package man is murex's man page parser to provide flag auto-complete suggestions
    Package man is murex's man page parser to provide flag auto-complete suggestions
    mxjson
    Package mxjson is a custom json superset format used by some routines in murex
    Package mxjson is a custom json superset format used by some routines in murex
    parser
    Package parser is a faster murex parser for real time context hints
    Package parser is a faster murex parser for real time context hints
    posix
    Package posix is a quick helper function to determine if the running platform is POSIX or not
    Package posix is a quick helper function to determine if the running platform is POSIX or not
    readall
    Package readall provides an alterative to ioutil.Readall but with support for context.Context
    Package readall provides an alterative to ioutil.Readall but with support for context.Context
    readline
    Package readline is a pure-Go re-imagining of the UNIX readline API
    Package readline is a pure-Go re-imagining of the UNIX readline API
    spellcheck
    Package spellcheck provides functions for spellchecking
    Package spellcheck provides functions for spellchecking
    spellcheck/userdictionary
    Package userdictionary provides `config` hooks for the spellchecker user dictionary
    Package userdictionary provides `config` hooks for the spellchecker user dictionary
    which
    Package which provides similar functionality to the UNIX command of the same name
    Package which provides similar functionality to the UNIX command of the same name