jsonnet

package module
v0.20.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 17, 2023 License: Apache-2.0 Imports: 26 Imported by: 247

README

go-jsonnet

GoDoc Widget Travis Widget Coverage Status Widget

This an implementation of Jsonnet in pure Go. It is a feature complete, production-ready implementation. It is compatible with the original Jsonnet C++ implementation. Bindings to C and Python are available (but not battle-tested yet).

This code is known to work on Go 1.12 and above. We recommend always using the newest stable release of Go.

Installation instructions

# go >= 1.17
# Using `go get` to install binaries is deprecated.
# The version suffix is mandatory.
go install github.com/google/go-jsonnet/cmd/jsonnet@latest

# Or other tools in the 'cmd' directory
go install github.com/google/go-jsonnet/cmd/jsonnet-lint@latest

# go < 1.17
go get github.com/google/go-jsonnet/cmd/jsonnet

It's also available on Homebrew:

brew install go-jsonnet

jsonnetfmt and jsonnet-lint are also available as pre-commit hooks. Example .pre-commit-config.yaml:

- repo: https://github.com/google/go-jsonnet
  rev: # ref you want to point at, e.g. v0.17.0
  hooks:
    - id: jsonnet-format
    - id: jsonnet-lint

It can also be embedded in your own Go programs as a library:

package main

import (
	"fmt"
	"log"

	"github.com/google/go-jsonnet"
)

func main() {
	vm := jsonnet.MakeVM()

	snippet := `{
		person1: {
		    name: "Alice",
		    welcome: "Hello " + self.name + "!",
		},
		person2: self.person1 { name: "Bob" },
	}`

	jsonStr, err := vm.EvaluateAnonymousSnippet("example1.jsonnet", snippet)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(jsonStr)
	/*
	   {
	     "person1": {
	         "name": "Alice",
	         "welcome": "Hello Alice!"
	     },
	     "person2": {
	         "name": "Bob",
	         "welcome": "Hello Bob!"
	     }
	   }
	*/
}

Build instructions (go 1.12+)

git clone git@github.com:google/go-jsonnet.git
cd go-jsonnet
go build ./cmd/jsonnet
go build ./cmd/jsonnetfmt
go build ./cmd/jsonnet-deps

To build with Bazel instead:

git clone git@github.com:google/go-jsonnet.git
cd go-jsonnet
git submodule init
git submodule update
bazel build //cmd/jsonnet
bazel build //cmd/jsonnetfmt
bazel build //cmd/jsonnet-deps

The resulting jsonnet program will then be available at a platform-specific path, such as bazel-bin/cmd/jsonnet/darwin_amd64_stripped/jsonnet for macOS.

Bazel also accommodates cross-compiling the program. To build the jsonnet program for various popular platforms, run the following commands:

Target platform Build command
Current host bazel build //cmd/jsonnet
Linux bazel build --platforms=@io_bazel_rules_go//go/toolchain:linux_amd64 //cmd/jsonnet
macOS bazel build --platforms=@io_bazel_rules_go//go/toolchain:darwin_amd64 //cmd/jsonnet
Windows bazel build --platforms=@io_bazel_rules_go//go/toolchain:windows_amd64 //cmd/jsonnet

For additional target platform names, see the per-Go release definitions here in the rules_go Bazel package.

Additionally if any files were moved around, see the section Keeping the Bazel files up to date.

Building libjsonnet.wasm

GOOS=js GOARCH=wasm go build -o libjsonnet.wasm ./cmd/wasm 

Or if using bazel:

bazel build //cmd/wasm:libjsonnet.wasm

Running tests

./tests.sh  # Also runs `go test ./...`

Running Benchmarks

Method 1
go get golang.org/x/tools/cmd/benchcmp
  1. Make sure you build a jsonnet binary prior to making changes.
go build -o jsonnet-old ./cmd/jsonnet
  1. Make changes (iterate as needed), and rebuild new binary
go build ./cmd/jsonnet
  1. Run benchmark:
# e.g. ./benchmark.sh Builtin
./benchmark.sh <TestNameFilter>
Method 2
  1. get benchcmp
go get golang.org/x/tools/cmd/benchcmp
  1. Make sure you build a jsonnet binary prior to making changes.
make build-old
  1. iterate with (which will also automatically rebuild the new binary ./jsonnet)

replace the FILTER with the name of the test you are working on

FILTER=Builtin_manifestJsonEx make benchmark

Implementation Notes

We are generating some helper classes on types by using http://clipperhouse.github.io/gen/. Do the following to regenerate these if necessary:

go get github.com/clipperhouse/gen
go get github.com/clipperhouse/set
export PATH=$PATH:$GOPATH/bin  # If you haven't already
go generate

Update cpp-jsonnet sub-repo

This repo depends on the original Jsonnet repo. Shared parts include the standard library, headers files for C API and some tests.

You can update the submodule and regenerate dependent files with one command:

./update_cpp_jsonnet.sh

Note: It needs to be run from repo root.

Updating and modifying the standard library

Standard library source code is kept in cpp-jsonnet submodule, because it is shared with Jsonnet C++ implementation.

For performance reasons we perform preprocessing on the standard library, so for the changes to be visible, regeneration is necessary:

go run cmd/dumpstdlibast/dumpstdlibast.go cpp-jsonnet/stdlib/std.jsonnet > astgen/stdast.go

**The

The above command creates the astgen/stdast.go file which puts the desugared standard library into the right data structures, which lets us avoid the parsing overhead during execution. Note that this step is not necessary to perform manually when building with Bazel; the Bazel target regenerates the astgen/stdast.go (writing it into Bazel's build sandbox directory tree) file when necessary.

Keeping the Bazel files up to date

Note that we maintain the Go-related Bazel targets with the Gazelle tool. The Go module (go.mod in the root directory) remains the primary source of truth. Gazelle analyzes both that file and the rest of the Go files in the repository to create and adjust appropriate Bazel targets for building Go packages and executable programs.

After changing any dependencies within the files covered by this Go module, it is helpful to run go mod tidy to ensure that the module declarations match the state of the Go source code. In order to synchronize the Bazel rules with material changes to the Go module, run the following command to invoke Gazelle's update-repos command:

bazel run //:gazelle -- update-repos -from_file=go.mod -to_macro=bazel/deps.bzl%jsonnet_go_dependencies

Similarly, after adding or removing Go source files, it may be necessary to synchronize the Bazel rules by running the following command:

bazel run //:gazelle

Documentation

Overview

Package jsonnet implements a parser and evaluator for jsonnet.

Jsonnet is a domain specific configuration language that helps you define JSON data. Jsonnet lets you compute fragments of JSON within the structure, bringing the same benefit to structured data that templating languages bring to plain text.

See http://jsonnet.org/ for a full language description and tutorial.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func SnippetToAST

func SnippetToAST(filename string, snippet string) (ast.Node, error)

SnippetToAST parses a snippet and returns the resulting AST.

func Version

func Version() string

Version returns the Jsonnet version number.

Types

type ColorFormatter

type ColorFormatter func(w io.Writer, f string, a ...interface{}) (n int, err error)

ColorFormatter represents a function that writes to the terminal using color.

type Contents added in v0.11.2

type Contents struct {
	// contains filtered or unexported fields
}

Contents is a representation of imported data. It is a simple byte wrapper, which makes it easier to enforce the caching policy.

func MakeContents added in v0.11.2

func MakeContents(s string) Contents

MakeContents creates Contents from a string.

func MakeContentsRaw added in v0.19.0

func MakeContentsRaw(bytes []byte) Contents

MakeContentsRaw creates Contents from (possibly non-utf8) []byte data.

func (Contents) Data added in v0.19.0

func (c Contents) Data() []byte

Data returns content bytes

func (Contents) String added in v0.11.2

func (c Contents) String() string

type ErrorFormatter

type ErrorFormatter interface {
	// Format static, runtime, and unexpected errors prior to printing them.
	Format(err error) string

	// Set the the maximum length of stack trace before cropping.
	SetMaxStackTraceSize(size int)

	// Set the color formatter for the location color.
	SetColorFormatter(color ColorFormatter)
}

An ErrorFormatter formats errors with stacktraces and color.

type FileImporter

type FileImporter struct {
	JPaths []string
	// contains filtered or unexported fields
}

FileImporter imports data from the filesystem.

func (*FileImporter) Import

func (importer *FileImporter) Import(importedFrom, importedPath string) (contents Contents, foundAt string, err error)

Import imports file from the filesystem.

type Importer

type Importer interface {
	// Import fetches data from a given path. It may be relative
	// to the file where we do the import. What "relative path"
	// means depends on the importer.
	//
	// It is required that:
	// a) for given (importedFrom, importedPath) the same
	//    (contents, foundAt) are returned on subsequent calls.
	// b) for given foundAt, the contents are always the same
	//
	// It is recommended that if there are multiple locations that
	// need to be probed (e.g. relative + multiple library paths)
	// then all results of all attempts will be cached separately,
	// both nonexistence and contents of existing ones.
	// FileImporter may serve as an example.
	//
	// IMPORTANT: The passed importedFrom might be "" (an empty string).
	// It means that the import is coming from an ad-hoc snippet, e.g.
	// code passed on the command line, read from stdin or passed as
	// a snippet to execute. Importer may have a "default" path to use
	// in such case or it may only allow absolute imports from such
	// "anonymous locations".
	//
	// Importing the same file multiple times must be a cheap operation
	// and shouldn't involve copying the whole file - the same buffer
	// should be returned.
	Import(importedFrom, importedPath string) (contents Contents, foundAt string, err error)
}

An Importer imports data from a path. TODO(sbarzowski) caching of errors (may require breaking changes)

type LineReader added in v0.18.0

type LineReader struct {
	// contains filtered or unexported fields
}

LineReader reads single lines.

func (*LineReader) Read added in v0.18.0

func (r *LineReader) Read() ([]byte, error)

Read returns a single line (with '\n' ended) from the underlying reader. An error is returned iff there is an error with the underlying reader.

type MemoryImporter

type MemoryImporter struct {
	Data map[string]Contents
}

MemoryImporter "imports" data from an in-memory map.

func (*MemoryImporter) Import

func (importer *MemoryImporter) Import(importedFrom, importedPath string) (contents Contents, foundAt string, err error)

Import fetches data from a map entry. All paths are treated as absolute keys.

type NativeFunction

type NativeFunction struct {
	Name   string
	Func   func([]interface{}) (interface{}, error)
	Params ast.Identifiers
}

NativeFunction represents a function implemented in Go.

type Reader added in v0.18.0

type Reader interface {
	Read() ([]byte, error)
}

Reader reads bytes

type RuntimeError

type RuntimeError struct {
	Msg        string
	StackTrace []traceFrame
}

RuntimeError is an error discovered during evaluation of the program

func (RuntimeError) Error

func (err RuntimeError) Error() string

type VM

type VM struct {
	MaxStack int

	ErrorFormatter ErrorFormatter
	StringOutput   bool
	// contains filtered or unexported fields
}

VM is the core interpreter and is the touchpoint used to parse and execute Jsonnet.

func MakeVM

func MakeVM() *VM

MakeVM creates a new VM with default parameters.

func (*VM) Evaluate added in v0.13.0

func (vm *VM) Evaluate(node ast.Node) (val string, err error)

Evaluate evaluates a Jsonnet program given by an Abstract Syntax Tree and returns serialized JSON as string. TODO(sbarzowski) perhaps is should return JSON in standard Go representation

func (*VM) EvaluateAnonymousSnippet added in v0.17.0

func (vm *VM) EvaluateAnonymousSnippet(filename string, snippet string) (json string, formattedErr error)

EvaluateAnonymousSnippet evaluates a string containing Jsonnet code, return a JSON string.

The filename parameter is only used for error messages.

func (*VM) EvaluateAnonymousSnippetMulti added in v0.17.0

func (vm *VM) EvaluateAnonymousSnippetMulti(filename string, snippet string) (files map[string]string, formattedErr error)

EvaluateAnonymousSnippetMulti evaluates a string containing Jsonnet code to key-value pairs. The keys are field name strings and the values are JSON strings.

The filename parameter is only used for error messages.

func (*VM) EvaluateAnonymousSnippetStream added in v0.17.0

func (vm *VM) EvaluateAnonymousSnippetStream(filename string, snippet string) (docs []string, formattedErr error)

EvaluateAnonymousSnippetStream evaluates a string containing Jsonnet code to an array. The array is returned as an array of JSON strings.

The filename parameter is only used for error messages.

func (*VM) EvaluateFile added in v0.17.0

func (vm *VM) EvaluateFile(filename string) (json string, formattedErr error)

EvaluateFile evaluates Jsonnet code in a file and returns a JSON string.

The importer is used to fetch the contents of the file.

func (*VM) EvaluateFileMulti added in v0.17.0

func (vm *VM) EvaluateFileMulti(filename string) (files map[string]string, formattedErr error)

EvaluateFileMulti evaluates Jsonnet code in a file to key-value pairs. The keys are field name strings and the values are JSON strings.

The importer is used to fetch the contents of the file.

func (*VM) EvaluateFileStream added in v0.17.0

func (vm *VM) EvaluateFileStream(filename string) (docs []string, formattedErr error)

EvaluateFileStream evaluates Jsonnet code in a file to an array. The array is returned as an array of JSON strings.

The importer is used to fetch the contents of the file.

func (*VM) EvaluateMulti added in v0.13.0

func (vm *VM) EvaluateMulti(node ast.Node) (output map[string]string, err error)

EvaluateMulti evaluates a Jsonnet program given by an Abstract Syntax Tree and returns key-value pairs. The keys are strings and the values are JSON strigns (serialized JSON).

func (*VM) EvaluateSnippet deprecated

func (vm *VM) EvaluateSnippet(filename string, snippet string) (json string, formattedErr error)

EvaluateSnippet evaluates a string containing Jsonnet code, return a JSON string.

The filename parameter is used for resolving relative imports and for errors messages.

Deprecated: Use EvaluateFile or EvaluateAnonymousSnippet instead.

func (*VM) EvaluateSnippetMulti deprecated

func (vm *VM) EvaluateSnippetMulti(filename string, snippet string) (files map[string]string, formattedErr error)

EvaluateSnippetMulti evaluates a string containing Jsonnet code to key-value pairs. The keys are field name strings and the values are JSON strings.

The filename parameter is used for resolving relative imports and for errors messages.

Deprecated: Use EvaluateFileMulti or EvaluateAnonymousSnippetMulti instead.

func (*VM) EvaluateSnippetStream deprecated

func (vm *VM) EvaluateSnippetStream(filename string, snippet string) (docs []string, formattedErr error)

EvaluateSnippetStream evaluates a string containing Jsonnet code to an array. The array is returned as an array of JSON strings.

The filename parameter is used for resolving relative imports and for errors messages.

Deprecated: Use EvaluateFileStream or EvaluateAnonymousSnippetStream instead.

func (*VM) EvaluateStream added in v0.13.0

func (vm *VM) EvaluateStream(node ast.Node) (output []string, err error)

EvaluateStream evaluates a Jsonnet program given by an Abstract Syntax Tree and returns an array of JSON strings.

func (*VM) ExtCode

func (vm *VM) ExtCode(key string, val string)

ExtCode binds a Jsonnet external code var to the given code.

func (*VM) ExtNode added in v0.18.0

func (vm *VM) ExtNode(key string, node ast.Node)

ExtNode binds a Jsonnet external code var to the given AST node.

func (*VM) ExtReset added in v0.18.0

func (vm *VM) ExtReset()

ExtReset rests all external variables registered for this VM.

func (*VM) ExtVar

func (vm *VM) ExtVar(key string, val string)

ExtVar binds a Jsonnet external var to the given value.

func (*VM) FindDependencies added in v0.17.0

func (vm *VM) FindDependencies(importedFrom string, importedPaths []string) ([]string, error)

FindDependencies returns a sorted array of unique transitive dependencies (via import/importstr/importbin) from all the given `importedPaths` which are themselves excluded from the returned array. The `importedPaths` are parsed as if they were imported from a Jsonnet file located at `importedFrom`.

func (*VM) ImportAST added in v0.15.0

func (vm *VM) ImportAST(importedFrom, importedPath string) (contents ast.Node, foundAt string, err error)

ImportAST fetches the Jsonnet AST just as if it was imported from a Jsonnet file located at `importedFrom`. It shares the cache with the actual evaluation.

func (*VM) ImportData added in v0.15.0

func (vm *VM) ImportData(importedFrom, importedPath string) (contents string, foundAt string, err error)

ImportData fetches the data just as if it was imported from a Jsonnet file located at `importedFrom`. It shares the cache with the actual evaluation.

func (*VM) Importer

func (vm *VM) Importer(i Importer)

Importer sets Importer to use during evaluation (import callback).

func (*VM) NativeFunction

func (vm *VM) NativeFunction(f *NativeFunction)

NativeFunction registers a native function.

func (*VM) ResolveImport added in v0.15.0

func (vm *VM) ResolveImport(importedFrom, importedPath string) (foundAt string, err error)

ResolveImport finds the actual path where the imported file can be found. It will cache the contents of the file immediately as well, to avoid the possibility of the file disappearing after being checked.

func (*VM) SetTraceOut added in v0.18.0

func (vm *VM) SetTraceOut(traceOut io.Writer)

SetTraceOut sets the output stream for the builtin function std.trace().

func (*VM) TLACode

func (vm *VM) TLACode(key string, val string)

TLACode binds a Jsonnet top level argument to the given code.

func (*VM) TLANode added in v0.18.0

func (vm *VM) TLANode(key string, node ast.Node)

TLANode binds a Jsonnet top level argument to the given AST node.

func (*VM) TLAReset added in v0.18.0

func (vm *VM) TLAReset()

TLAReset resets all TLAs registered for this VM.

func (*VM) TLAVar

func (vm *VM) TLAVar(key string, val string)

TLAVar binds a Jsonnet top level argument to the given value.

type YAMLReader added in v0.18.0

type YAMLReader struct {
	// contains filtered or unexported fields
}

YAMLReader reads YAML

func NewYAMLReader added in v0.18.0

func NewYAMLReader(r *bufio.Reader) *YAMLReader

NewYAMLReader creates a new YAMLReader

func (*YAMLReader) Read added in v0.18.0

func (r *YAMLReader) Read() ([]byte, error)

Read returns a full YAML document.

type YAMLToJSONDecoder added in v0.18.0

type YAMLToJSONDecoder struct {
	// contains filtered or unexported fields
}

YAMLToJSONDecoder decodes YAML documents from an io.Reader by separating individual documents. It first converts the YAML body to JSON, then unmarshals the JSON.

func NewYAMLToJSONDecoder added in v0.18.0

func NewYAMLToJSONDecoder(r io.Reader) *YAMLToJSONDecoder

NewYAMLToJSONDecoder decodes YAML documents from the provided stream in chunks by converting each document (as defined by the YAML spec) into its own chunk, converting it to JSON via yaml.YAMLToJSON, and then passing it to json.Decoder.

func (*YAMLToJSONDecoder) Decode added in v0.18.0

func (d *YAMLToJSONDecoder) Decode(into interface{}) error

Decode reads a YAML document as JSON from the stream or returns an error. The decoding rules match json.Unmarshal, not yaml.Unmarshal.

Directories

Path Synopsis
Package ast provides AST nodes and ancillary structures and algorithms.
Package ast provides AST nodes and ancillary structures and algorithms.
cmd
internal/cmd
Package cmd provides utilities for parsing and handling command line arguments.
Package cmd provides utilities for parsing and handling command line arguments.
Package formatter is what powers jsonnetfmt, a Jsonnet formatter.
Package formatter is what powers jsonnetfmt, a Jsonnet formatter.
internal
dump
Package dump can dump a Go data structure to Go source file, so that it can be statically embedded into other code.
Package dump can dump a Go data structure to Go source file, so that it can be statically embedded into other code.
errors
Package errors provides internal representation of errors.
Package errors provides internal representation of errors.
formatter
Package formatter provides API for producing pretty-printed source from AST.
Package formatter provides API for producing pretty-printed source from AST.
parser
Package parser reads Jsonnet files and parses them into AST nodes.
Package parser reads Jsonnet files and parses them into AST nodes.
pass
Package pass provides a visitor framework for source code analysis and transformation.
Package pass provides a visitor framework for source code analysis and transformation.
program
Package program provides API for AST pre-processing (desugaring, static analysis).
Package program provides API for AST pre-processing (desugaring, static analysis).
testutils
Package testutils provides general testing utilities.
Package testutils provides general testing utilities.
Package linter analyses Jsonnet code for code "smells".
Package linter analyses Jsonnet code for code "smells".
internal/common
Package common provides utilities to be used in multiple linter subpackages.
Package common provides utilities to be used in multiple linter subpackages.
internal/traversal
Package traversal provides relatively lightweight checks which can all fit within one traversal of the AST.
Package traversal provides relatively lightweight checks which can all fit within one traversal of the AST.
internal/types
Package types provides type inference functionality.
Package types provides type inference functionality.
internal/variables
Package variables allows collecting the information about how variables are used.
Package variables allows collecting the information about how variables are used.
Package toolutils includes several utilities handy for use in code analysis tools
Package toolutils includes several utilities handy for use in code analysis tools

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL