monster

README

README

Monster is a grammar based production tool. At present I am using this to generate json documents using production grammar.

Can be invoked via command-line like,

$ go run monster/monster.go -h
	  -bagdir="": directory path containing bags
	  -n=1: generate n combinations
	  -nonterm="": evaluate the non-terminal
	  -o="-": specify an output file
	  -seed=37: seed value

Example production grammar, that generate randomized json documents.

s 		   : value.
object     : "{" properties "}".
properties : properties "," property
		   | property.
property   : DQ (bag "./web2") DQ ":" value.
array      : "[" values "]".
values     : value "," value
		   | (weigh 0.8 0.2) values.
value      : (weigh 0.1) basic
		   | (weigh 0.45 0.1) array
		   | (weigh 0.45 0.1) object.
basic      : "true"
		   | "false"
		   | "null"
		   | number
		   | string.
string     : DQ (bag "./web2") DQ.
number     : (range 0 100000)
		   | (rangef 0.0 100.0).

The grammar can be invoked via command line like,

go run monster/monster.go -bagdir ./bags -n 10 ./prods/json.prod

use -n switch to generate as many document as needed, documents will be output to stdout by default, to redirect them to a file use -o switch.

different commands to debug monster,

// To generate GCTRACE.
GCDEBUG=gctrace=2 go run monster/monster.go -bagdir bags -count 1000000 -o o prods/users.prod

// To generate mem-profile.
go run monster/monster.go -bagdir bags -count 1000000 -memprof apr21.1.mprof -o o prods/users.prod

// To analyse alloc-space.
go tool pprof --svg --alloc_space monster/monster apr21.1.mprof > apr21.1.mprof.alloc.svg

Documentation

Overview

Use production grammer with a rudimentary S expression like language to generate permutations and combinations. Every element in the grammer including the S expressions will be converted into a Form object.

Parser uses `parsec` tool to parse production grammar and construct generator tree.

Monster grammar

bnf        : forms nterminal*
nterminal  : ":" rules "."
rules      : ruletok+
           : rules "|" ruletok+
ruletok    : ident
           |  ref
           |  terminal
           |  string
           |  form

forms      : form*
form       : "(" formarg+ ")"
ident      : `[a-z0-9]+`
terminal   : `[A-Z][A-Z0-9]*`
formarg    : `[^ \t\r\n\(\)]+`
           |  form
ws         : `[ \t\r\n]+`

nomenclature of forms constructed while compiling a production file,

forms["##literaltok"] evaluating INT/HEX/OCT/FLOAT/TRUE/FALSE in form-args forms["##formtok"] evaluating any other form-args, an open ended spec forms["##ident"] evaluating non-terminal identifiers in rule-args forms["##term"] evaluating terminals like DQ, NL in farm-args or rule-args forms["##string"] evaluating literal strings in form-args and rule-args forms["##ref"] evaluating references into local/global namespace forms["##rule"] evaluating a non-terminal rule forms[<symbol>] evaluating builtin or application supplied functions forms["#<ident>] evaluating non-terminal forms from top-level forms["##var"] in unused as of now

these forms are evaluated from one of top-level forms to generate permutations and combinations.

Form evaluation happens at the following points using Eval() or EvalForms() APIs,

• when a global S-expression invokes a builtin function
• when a global S-expression invokes a non-terminal
• before passing S-expression arguments to a form.
• when special form `weigh` is identified at the begining of a non-terminal rule definition.
• when a non-terminal rule is randomly picked by EvalForms()
• before passing rule arguments to a rule-form.

references:

$<symbol> specified in a rule or,
#<symbol> specified in a form argument,
    will evaluate a lookup into local or global scope.

Programmatically invoking monster {

   root, _ := monster.Y(parsec.NewScanner(text)).(common.Scope)
   scope := monster.BuildContext(root, seed, bagdir, prodfile)
   nterms := scope["_nonterminals"].(common.NTForms)
   for i := 0; i < count; i++ {
       scope = scope.RebuildContext()
       val := monster.EvalForms("root", scope, nterms["s"])
   }
}

Index

• Variables
• func BuildContext(scope common.Scope, seed uint64, bagdir, prodfile string) common.Scope
• type Nt

Variables

var EvalForms = common.EvalForms

EvalForms refer to common.EvalForms for more documentation

var Y parsec.Parser

Y root combinator for monster.

Functions

func BuildContext

func BuildContext(
	scope common.Scope,
	seed uint64,
	bagdir, prodfile string) common.Scope

BuildContext to initialize a new scope for evaluating production grammars. The scope contains the following elements,

_globalForms:  list of top-level S-expression definitions
_nonterminals: list of top-level non-terminals in production
_weights:      running weights for each non-terminal rule
_globals:      global scope
    _bagdir:       absolute path to directory containing bags of data
    _prodfile:     absolute path to production file
    _random:       reference to seeded *math.rand.Rand object

Types

type Nt

type Nt [2]interface{}

Nt is intermediate data structure.