shorthand

README

CLI Shorthand Syntax

CLI shorthand syntax is a contextual shorthand syntax for passing structured data into commands that require e.g. JSON/YAML. While you can always pass full JSON or other documents through stdin, you can also specify or modify them by hand as arguments to the command using this shorthand syntax. For example:

$ my-cli do-something foo.bar[0].baz: 1, .hello: world

Would result in the following body contents being sent on the wire (assuming a JSON media type is specified in the OpenAPI spec):

{
  "foo": {
    "bar": [
      {
        "baz": 1,
        "hello": "world"
      }
    ]
  }
}

The shorthand syntax supports the following features, described in more detail with examples below:

• Automatic type coercion & forced strings
• Nested object creation
• Object property grouping
• Nested array creation
• Appending to arrays
• Both object and array backreferences
• Loading property values from files
  • Supports structured, forced string, and base64 data

Alternatives & Inspiration

The CLI shorthand syntax is not the only one you can use to generate data for CLI commands. Here are some alternatives:

• jo
• jarg

For example, the shorthand example given above could be rewritten as:

$ jo -p foo=$(jo -p bar=$(jo -a $(jo baz=1 hello=world))) | my-cli do-something

The shorthand syntax implementation described herein uses those and the following for inspiration:

• YAML
• W3C HTML JSON Forms
• jq
• JMESPath

It seems reasonable to ask, why create a new syntax?

1. Built-in. No extra executables required. Your CLI ships ready-to-go.
2. No need to use sub-shells to build complex structured data.
3. Syntax is closer to YAML & JSON and mimics how you do queries using tools like jq and jmespath.

Features in Depth

You can use the included j executable to try out the shorthand format examples below. Examples are shown in JSON, but the shorthand parses into structured data that can be marshalled as other formats, like YAML or TOML if you prefer.

go get -u github.com/danielgtaylor/shorthand/cmd/j

Also feel free to use this tool to generate structured data for input to other commands.

Keys & Values

At its most basic, a structure is built out of key & value pairs. They are separated by commas:

$ j hello: world, question: how are you?
{
  "hello": "world",
  "question": "how are you?"
}

Types and Type Coercion

Well-known values like null, true, and false get converted to their respective types automatically. Numbers also get converted. Similar to YAML, anything that doesn't fit one of those is treated as a string. If needed, you can disable this automatic coercion by forcing a value to be treated as a string with the ~ operator. Note: the ~ modifier must come directly after the colon.

# With coercion
$ j empty: null, bool: true, num: 1.5, string: hello
{
  "bool": true,
  "empty": null,
  "num": 1.5,
  "string": "hello"
}

# As strings
$ j empty:~ null, bool:~ true, num:~ 1.5, string:~ hello
{
  "bool": "true",
  "empty": "null",
  "num": "1.5",
  "string": "hello"
}

# Passing the empty string
$ j blank:~
{
  "blank": ""
}

# Passing a tilde using whitespace
$ j foo: ~/Documents
{
  "foo": "~/Documents"
}

# Passing a tilde using forced strings
$ j foo:~~/Documents
{
  "foo": "~/Documents"
}

Objects

Nested objects use a . separator when specifying the key.

$ j foo.bar.baz: 1
{
  "foo": {
    "bar": {
      "baz": 1
    }
  }
}

Properties of nested objects can be grouped by placing them inside { and }.

$ j foo.bar{id: 1, count.clicks: 5}
{
  "foo": {
    "bar": {
      "count": {
        "clicks": 5
      },
      "id": 1
    }
  }
}

Arrays

Simple arrays use a , between values. Nested arrays use square brackets [ and ] to specify the zero-based index to insert an item. Use a blank index to append to the array.

# Array shorthand
$ j a: 1, 2, 3
{
  "a": [
    1,
    2,
    3
  ]
}

# Nested arrays
$ j a[0][2][0]: 1
{
  "a": [
    [
      null,
      null,
      [
        1
      ]
    ]
  ]
}

# Appending arrays
$ j a[]: 1, a[]: 2, a[]: 3
{
  "a": [
    1,
    2,
    3
  ]
}

Backreferences

Since the shorthand syntax is context-aware, it is possible to use the current context to reference back to the most recently used object or array when creating new properties or items.

# Backref with object properties
$ j foo.bar: 1, .baz: 2
{
  "foo": {
    "bar": 1,
    "baz": 2
  }
}

# Backref with array appending
$ j foo.bar[]: 1, []: 2, []: 3
{
  "foo": {
    "bar": [
      1,
      2,
      3
    ]
  }
}

# Easily build complex structures
$ j name: foo, tags[]{id: 1, count.clicks: 5, .sales: 1}, []{id: 2, count.clicks: 8, .sales: 2}
{
  "name": "foo",
  "tags": [
    {
      "count": {
        "clicks": 5,
        "sales": 1
      },
      "id": 1
    },
    {
      "count": {
        "clicks": 8,
        "sales": 2
      },
      "id": 2
    }
  ]
}

Loading from Files

Sometimes a field makes more sense to load from a file than to be specified on the commandline. The @ preprocessor and ~ & % modifiers let you load structured data, strings, and base64-encoded data into values.

# Load a file's value as a parameter
$ j foo: @hello.txt
{
  "foo": "hello, world"
}

# Load structured data
$ j foo: @hello.json
{
  "foo": {
    "hello": "world"
  }
}

# Force loading a string
$ j foo: @~hello.json
{
  "foo": "{\n  \"hello\": \"world\"\n}"
}

# Load as base 64 data
$ j foo: @%hello.json
{
  "foo": "ewogICJoZWxsbyI6ICJ3b3JsZCIKfQ=="
}

Remember, it's possible to disable this behavior with the string modifier ~:

$ j twitter:~ @user
{
  "twitter": "@user"
}

Library Usage

The GetInput function provides an all-in-one quick and simple way to get input from both stdin and passed arguments:

package main

import (
  "fmt"
  "github.com/danielgtaylor/shorthand"
)

func main() {
  result, err := shorthand.GetInput(os.Args[1:])
  if err != nil {
    panic(err)
  }

  fmt.Println(result)
}

It's also possible to get the shorthand representation of an input, for example:

example := map[string]interface{}{
  "hello": "world",
  "labels": []interface{}{
    "one",
    "two",
  },
}

// Prints "hello: world, labels: one, two"
fmt.Println(shorthand.Get(example))

Implementation

The shorthand syntax is implemented as a PEG grammar which creates an AST-like object that is used to build an in-memory structure that can then be serialized out into formats like JSON, YAML, TOML, etc.

The shorthand.peg file implements the parser while the shorthand.go file implements the builder. Here's how you can test local changes to the grammar:

# One-time setup: install PEG compiler
$ go get -u github.com/mna/pigeon

# Make your shorthand.peg edits. Then:
$ go generate

# Next, rebuild the j executable.
$ go install ./cmd/j

# Now, try it out!
$ j <your-new-thing>

Documentation

Overview

Package shorthand provides a quick way to generate structured data via command line parameters.

Index

• func Build(ast AST, existing ...map[string]interface{}) (map[string]interface{}, error)
• func DeepAssign(target, source map[string]interface{})
• func Get(input map[string]interface{}) string
• func GetInput(args []string) (map[string]interface{}, error)
• func Parse(filename string, b []byte, opts ...Option) (interface{}, error)
• func ParseAndBuild(filename, input string, existing ...map[string]interface{}) (map[string]interface{}, error)
• func ParseFile(filename string, opts ...Option) (i interface{}, err error)
• func ParseReader(filename string, r io.Reader, opts ...Option) (interface{}, error)
• type AST
• type Cloner
• type Key
• type KeyPart
• type KeyValue
• type Option
  • func AllowInvalidUTF8(b bool) Option
  • func Debug(b bool) Option
  • func Entrypoint(ruleName string) Option
  • func GlobalStore(key string, value interface{}) Option
  • func InitState(key string, value interface{}) Option
  • func MaxExpressions(maxExprCnt uint64) Option
  • func Memoize(b bool) Option
  • func Recover(b bool) Option
  • func Statistics(stats *Stats, choiceNoMatch string) Option
• type Stats

Functions

func Build

func Build(ast AST, existing ...map[string]interface{}) (map[string]interface{}, error)

Build an AST of key-value pairs into structured data.

func DeepAssign

func DeepAssign(target, source map[string]interface{})

DeepAssign recursively merges a source map into the target.

func Get

func Get(input map[string]interface{}) string

Get the shorthand representation of an input map.

func GetInput

func GetInput(args []string) (map[string]interface{}, error)

GetInput loads data from stdin (if present) and from the passed arguments, returning the final structure.

func Parse

func Parse(filename string, b []byte, opts ...Option) (interface{}, error)

Parse parses the data from b using filename as information in the error messages.

func ParseAndBuild

func ParseAndBuild(filename, input string, existing ...map[string]interface{}) (map[string]interface{}, error)

ParseAndBuild takes a string and returns the structured data it represents.

func ParseFile

func ParseFile(filename string, opts ...Option) (i interface{}, err error)

ParseFile parses the file identified by filename.

func ParseReader

func ParseReader(filename string, r io.Reader, opts ...Option) (interface{}, error)

ParseReader parses the data from r using filename as information in the error messages.

Types

type AST

type AST []*KeyValue

AST contains all of the key-value pairs in the document.

type Cloner

type Cloner interface {
	Clone() interface{}
}

Cloner is implemented by any value that has a Clone method, which returns a copy of the value. This is mainly used for types which are not passed by value (e.g map, slice, chan) or structs that contain such types.

This is used in conjunction with the global state feature to create proper copies of the state to allow the parser to properly restore the state in the case of backtracking.

type Key

type Key struct {
	ResetContext bool
	Parts        []*KeyPart
}

Key contains parts and key-specific configuration.

type KeyPart

type KeyPart struct {
	Key   string
	Index []int
}

KeyPart has a name and optional indices.

type KeyValue

type KeyValue struct {
	PostProcess bool
	Key         *Key
	Value       interface{}
}

KeyValue groups a Key with the key's associated value.

type Option

type Option func(*parser) Option

Option is a function that can set an option on the parser. It returns the previous setting as an Option.

func AllowInvalidUTF8

func AllowInvalidUTF8(b bool) Option

AllowInvalidUTF8 creates an Option to allow invalid UTF-8 bytes. Every invalid UTF-8 byte is treated as a utf8.RuneError (U+FFFD) by character class matchers and is matched by the any matcher. The returned matched value, c.text and c.offset are NOT affected.

The default is false.

func Debug

func Debug(b bool) Option

Debug creates an Option to set the debug flag to b. When set to true, debugging information is printed to stdout while parsing.

The default is false.

func Entrypoint

func Entrypoint(ruleName string) Option

Entrypoint creates an Option to set the rule name to use as entrypoint. The rule name must have been specified in the -alternate-entrypoints if generating the parser with the -optimize-grammar flag, otherwise it may have been optimized out. Passing an empty string sets the entrypoint to the first rule in the grammar.

The default is to start parsing at the first rule in the grammar.

func GlobalStore

func GlobalStore(key string, value interface{}) Option

GlobalStore creates an Option to set a key to a certain value in the globalStore.

func InitState

func InitState(key string, value interface{}) Option

InitState creates an Option to set a key to a certain value in the global "state" store.

func MaxExpressions

func MaxExpressions(maxExprCnt uint64) Option

MaxExpressions creates an Option to stop parsing after the provided number of expressions have been parsed, if the value is 0 then the parser will parse for as many steps as needed (possibly an infinite number).

The default for maxExprCnt is 0.

func Memoize

func Memoize(b bool) Option

Memoize creates an Option to set the memoize flag to b. When set to true, the parser will cache all results so each expression is evaluated only once. This guarantees linear parsing time even for pathological cases, at the expense of more memory and slower times for typical cases.

The default is false.

func Recover

func Recover(b bool) Option

Recover creates an Option to set the recover flag to b. When set to true, this causes the parser to recover from panics and convert it to an error. Setting it to false can be useful while debugging to access the full stack trace.

The default is true.

func Statistics

func Statistics(stats *Stats, choiceNoMatch string) Option

Statistics adds a user provided Stats struct to the parser to allow the user to process the results after the parsing has finished. Also the key for the "no match" counter is set.

Example usage:

input := "input"
stats := Stats{}
_, err := Parse("input-file", []byte(input), Statistics(&stats, "no match"))
if err != nil {
    log.Panicln(err)
}
b, err := json.MarshalIndent(stats.ChoiceAltCnt, "", "  ")
if err != nil {
    log.Panicln(err)
}
fmt.Println(string(b))

type Stats

type Stats struct {
	// ExprCnt counts the number of expressions processed during parsing
	// This value is compared to the maximum number of expressions allowed
	// (set by the MaxExpressions option).
	ExprCnt uint64

	// ChoiceAltCnt is used to count for each ordered choice expression,
	// which alternative is used how may times.
	// These numbers allow to optimize the order of the ordered choice expression
	// to increase the performance of the parser
	//
	// The outer key of ChoiceAltCnt is composed of the name of the rule as well
	// as the line and the column of the ordered choice.
	// The inner key of ChoiceAltCnt is the number (one-based) of the matching alternative.
	// For each alternative the number of matches are counted. If an ordered choice does not
	// match, a special counter is incremented. The name of this counter is set with
	// the parser option Statistics.
	// For an alternative to be included in ChoiceAltCnt, it has to match at least once.
	ChoiceAltCnt map[string]map[string]int
}

Stats stores some statistics, gathered during parsing