Version: v0.0.0-...-2ef5af0 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Aug 9, 2022 License: Apache-2.0 Imports: 77 Imported by: 4


LUCI Configuration Generator

lucicfg is a tool for generating low-level LUCI configuration files based on a high-level configuration given as a Starlark script that uses APIs exposed by lucicfg. In other words, it takes a *.star file (or files) as input and spits out a bunch of *.cfg files (such us cr-buildbucket.cfg and luci-scheduler.cfg) as outputs.

Overview of the design

lucicfg follows a "microkernel" architecture. The kernel is implemented in Go. It provides a private interface (used internally by the lucicfg's Starlark standard library) by registering a bunch of builtins. The functionality provided by the kernel is pretty generic:

  • A notion of Starlark packages, load(...) and exec(...) implementation.
  • lucicfg.var(...) implementation.
  • A node graph structure to carry the state across module boundaries.
  • Support for traversing the node graph and emitting files to an output.
  • Support for Protobuf messages.
  • Implementation of lucicfg generate and lucicfg validate logic.
  • Various utilities (regexp, hashes, Go templates, etc.)

The builtins are wrapped in two layers of Starlark code:

  • starlark/stdlib/internal, excluding .../luci: generic (not LUCI specific) Starlark standard library that documents and exposes the builtins via a nicer API. It can be used to build all kinds of config generators (1, 2, 3), as well as extend the LUCI config generator. This API surface is currently marked as "internal" (meaning there's no backward compatibility guarantees for it), but it will some day become a part of lucicfg's public interface, so it should be treated as such (no hacks, no undocumented functions, adequate test coverage, etc).
  • starlark/stdlib/internal/luci: all LUCI-specific APIs and declarations, implementing the logic of generating LUCI configs specifically. It is built

The standard library and LUCI configs library are bundled with lucicfg binary via starlark/assets.gen.go file generated from the contents of starlark/stdlib/internal by go generate.

Making changes to the Starlark portion of the code

  1. Modify a *.star file.
  2. In the lucicfg directory, where this file is, run ./ to auto-format and lint new code. Fix all linter warnings.
  3. In the same directory run go generate ./... to regenerate examples/.../generated and doc/
  4. Run go test ./... to verify existing tests pass. If your change modifies the format of emitted files you need to update the expected output in test case files. It will most likely happen for testdata/ Update Expect configs: section there.
  5. If your change warrants a new test, add a file somewhere under testdata/. See existing files there for examples. There are two kinds of tests:
    • "Expectation style" tests. They have Expect configs: or Expect errors like: sections at the bottom. The test runner will execute the Starlark code and compare the produced output (or errors) to the expectations.
    • More traditional unit tests that use assert.eq(...) etc. See testdata/misc/ for an example.
  6. Once you are done with the change, evaluate whether you need to bump lucicfg version. See the section below.

Updating lucicfg version

lucicfg uses a variant of semantic versioning to identify its own version and a version of the bundled Starlark libraries. The version string is set in version.go and looks like MAJOR.MINOR.PATCH.

If a user script is relying on a feature which is available only in some recent lucicfg version, it can perform a check, like so:

lucicfg.check_version('1.7.8', 'Please update depot_tools')

That way if the script is executed by an older lucicfg version, the user will get a nice actionable error message instead of some obscure stack trace.

Thus it is very important to update version.go before releasing changes:

  • Increment PATCH version when you make backward compatible changes. A change is backward compatible if it doesn't reduce Starlark API surface and doesn't affect lucicfg's emitted output (assuming inputs do not change). In particular, adding a new feature that doesn't affect existing features is backward compatible.
  • Increment MINOR version when you make backward incompatible changes. Releasing such changes may require modifying user scripts or asking users to regenerate their configs to get an updated output. Note that both these things are painful, since there are dozens of repositories with lucicfg scripts and, strictly speaking, all of them should be eventually updated.
  • MAJOR version is reserved for major architecture changes or rewrites.

If your new feature is experimental and you don't want to commit to any backward compatibility promises, hide it behind an experiment. Users will need to opt-in to use it. See starlark/stdlib/internal/ for more info.

Making a release

  1. Land the code change. For concreteness sake let's assume it resulted in this luci-go commit.
  2. Wait until it is rolled into infra.git. The roll will look like this infra.git commit. Notice its git hash 86afde8bddae....
  3. Wait until the CIPD package builders produce per-platform lucicfg CIPD packages tagged with infra.git's hash git_revision:86afde8bddae.... Like this one.
  4. Land a depot_tools CL to release the new version to developer workstations:
    1. Modify cipd_manifest.txt: infra/tools/luci/lucicfg/${platform} git_revision:86afde8bddae....
    2. As instructed in the comments, regenerate cipd_manifest.versions: cipd ensure-file-resolve -ensure-file cipd_manifest.txt.
    3. Send a CL like this, describing in the commit message what's new.
  5. Modify cr-buildbucket/settings.cfg like so to release the change to bots. This step is necessary since bots don't use depot_tools.

Steps 2 and 3 usually take about 30 minutes total, and steps 4 and 5 verify CIPD packages actually exist. So in practice it is OK to just land a lucicfg CL, go do other things, then come back >30 min later, look up the revision of the necessary infra.git DEPS roll (or just use the latest one) and proceed to steps 4 and 5.



Package lucicfg contains LUCI config generator.

All Starlark code is executed sequentially in a single goroutine from inside Generate function, thus this package doesn't used any mutexes or other synchronization primitives. It is safe to call Generate concurrently though, since there's no global shared state, each Generate call operates on its own state.



View Source
const (
	TOK_LB    = iota // non-escaped '{'
	TOK_RB           // non-escaped '}'
	TOK_NUM          // a sequence of digits
	TOK_RUNES        // an arbitrary sequence of non-special runes
	TOK_COMMA        // ','
	TOK_DOTS         // '..'
View Source
const (
	// Version is the version of lucicfg tool.
	// It ends up in CLI output and in User-Agent headers.
	Version = "1.32.2"

	// UserAgent is used for User-Agent header in HTTP requests from lucicfg.
	UserAgent = "lucicfg v" + Version


This section is empty.


func FindTrackedFiles

func FindTrackedFiles(dir string, patterns []string) ([]string, error)

FindTrackedFiles recursively discovers all regular files in the given directory whose names match given patterns.

See TrackedSet for the format of `patterns`. If the directory doesn't exist, returns empty slice.

Returned file names are sorted, slash-separated and relative to `dir`.

func TrackedSet

func TrackedSet(patterns []string) func(string) (bool, error)

TrackedSet returns a predicate that classifies whether a slash-separated path belongs to a tracked set or not.

Each entry in `patterns` is either `<glob pattern>` (a "positive" glob) or `!<glob pattern>` (a "negative" glob). A path is considered tracked if its base name matches any of the positive globs and none of the negative globs. If `patterns` is empty, no paths are considered tracked. If all patterns are negative, single `**/*` positive pattern is implied as well.

The predicate returns an error if some pattern is malformed.


type BacktracableError

type BacktracableError interface {

	// Backtrace returns a user-friendly error message describing the stack
	// of calls that led to this error, along with the error message itself.
	Backtrace() string

BacktracableError is an error that has a starlark backtrace attached to it.

Implemented by Error here, by starlark.EvalError and graph errors.

type BlobDatum

type BlobDatum []byte

BlobDatum is a Datum which is just a raw byte blob.

func (BlobDatum) Bytes

func (b BlobDatum) Bytes() ([]byte, error)

Bytes is a raw file body to put on disk.

func (BlobDatum) Compare

func (b BlobDatum) Compare(other []byte) (CompareResult, error)

Compare is Identical if 'other == b' else it is Different.

type CompareResult

type CompareResult int

CompareResult is returned by Datum.Compare.

const (
	UnknownResult     CompareResult = iota // used as a placeholder on errors
	Identical                              // datums are byte-to-byte identical
	SemanticallyEqual                      // datums are byte-to-byte different, but semantically equal
	Different                              // datums are semantically different

type ConfigSet

type ConfigSet struct {
	// Name is a name of this config set, e.g. "projects/something".
	// It is used by LUCI Config to figure out how to validate files in the set.
	Name string

	// Data is files belonging to the config set.
	//  Keys are slash-separated filenames, values are corresponding file bodies.
	Data map[string][]byte

ConfigSet is an in-memory representation of a single config set.

func ReadConfigSet

func ReadConfigSet(dir, name string) (ConfigSet, error)

ReadConfigSet reads all regular files in the given directory (recursively) and returns them as a ConfigSet with given name.

func (ConfigSet) AsOutput

func (cs ConfigSet) AsOutput(root string) Output

AsOutput converts this config set into Output that have it at the given root path (usually ".").

func (ConfigSet) Files

func (cs ConfigSet) Files() []string

Files returns a sorted list of file names in the config set.

func (ConfigSet) Validate

Validate sends the config set for validation to LUCI Config service.

Returns ValidationResult with a list of validation message (errors, warnings, etc). The list of messages may be empty if the config set is 100% valid.

If the RPC call itself failed, ValidationResult is still returned, but it has only ConfigSet and RPCError fields populated.

type ConfigSetValidator

type ConfigSetValidator interface {
	// Validate sends the validation request to the service.
	// Returns errors only on RPC errors. Actual validation errors are
	// communicated through []*ValidationMessage.
	Validate(ctx context.Context, req *ValidationRequest) ([]*ValidationMessage, error)

ConfigSetValidator is primarily implemented through config.Service, but can also be mocked in tests.

func RemoteValidator

func RemoteValidator(svc *config.Service) ConfigSetValidator

RemoteValidator returns ConfigSetValidator that makes RPCs to LUCI Config.

type Datum

type Datum interface {
	// Bytes is a raw file body to put on disk.
	Bytes() ([]byte, error)
	// Compare semantically compares this datum to 'other'.
	Compare(other []byte) (CompareResult, error)

Datum represents one generated output file.

type Error

type Error struct {
	Msg   string
	Stack *builtins.CapturedStacktrace

Error is a single error message emitted by the config generator.

It holds a stack trace responsible for the error.

func (*Error) Backtrace

func (e *Error) Backtrace() string

Backtrace is part of BacktracableError interface.

func (*Error) Error

func (e *Error) Error() string

Error is part of 'error' interface.

type Inputs

type Inputs struct {
	Code  interpreter.Loader // a package with the user supplied code
	Path  string             // absolute path to the main package, if known
	Entry string             // a name of the entry point script in this package
	Meta  *Meta              // defaults for lucicfg own parameters
	Vars  map[string]string  // var values passed via `-var key=value` flags
	// contains filtered or unexported fields

Inputs define all inputs for the config generator.

type MessageDatum

type MessageDatum struct {
	Header  string
	Message *starlarkproto.Message
	// contains filtered or unexported fields

MessageDatum is a Datum constructed from a proto message.

func (*MessageDatum) Bytes

func (m *MessageDatum) Bytes() ([]byte, error)

Bytes is a raw file body to put on disk.

func (*MessageDatum) Compare

func (m *MessageDatum) Compare(other []byte) (CompareResult, error)

Compare deserializes `other` and compares it to `m.Message`.

If `other` can't be deserialized as a proto message at all returns Different. Returns an error if `m` can't be serialized.

type Meta

type Meta struct {
	ConfigServiceHost string   `json:"config_service_host"` // LUCI config host name
	ConfigDir         string   `json:"config_dir"`          // output directory to place generated files or '-' for stdout
	TrackedFiles      []string `json:"tracked_files"`       // e.g. ["*.cfg", "!*-dev.cfg"]
	FailOnWarnings    bool     `json:"fail_on_warnings"`    // true to treat validation warnings as errors
	LintChecks        []string `json:"lint_checks"`         // active lint checks
	// contains filtered or unexported fields

Meta contains configuration for the configuration generator itself.

It influences how generator produces output configs. It is settable through lucicfg.config(...) statements on the Starlark side or through command line flags. Command line flags override what was set via lucicfg.config(...).

See @stdlib//internal/ for full meaning of fields.

func (*Meta) AddFlags

func (m *Meta) AddFlags(fs *flag.FlagSet)

AddFlags registers command line flags that correspond to Meta fields.

func (*Meta) Copy

func (m *Meta) Copy() Meta

Copy returns an "untouched" copy of `m`.

In the returned copy WasTouched reports all fields as untouched.

func (*Meta) Log

func (m *Meta) Log(ctx context.Context)

Log logs the values of the meta parameters to Debug logger.

func (*Meta) PopulateFromTouchedIn

func (m *Meta) PopulateFromTouchedIn(t *Meta)

PopulateFromTouchedIn takes all touched values in `t` and copies them to `m`, overriding what's in `m`.

func (*Meta) RebaseConfigDir

func (m *Meta) RebaseConfigDir(root string)

RebaseConfigDir changes ConfigDir, if it is set, to be absolute by appending it to the given root.

Doesn't touch "-", which indicates "stdout".

func (*Meta) WasTouched

func (m *Meta) WasTouched(field string) bool

WasTouched returns true if the field (given by its Starlark snake_case name) was explicitly set via CLI flags or via lucicfg.config(...) in Starlark.

Panics if the field is unrecognized.

type Output

type Output struct {
	// Data is all output files.
	// Keys are slash-separated filenames, values are corresponding file bodies.
	Data map[string]Datum

	// Roots is mapping "config set name => its root".
	// Roots are given as slash-separated paths relative to the output root, e.g.
	// '.' matches ALL output files.
	Roots map[string]string

Output is an in-memory representation of all generated output files.

Output may span zero or more config sets, each defined by its root directory. Config sets may intersect (though this is rare).

func (Output) Compare

func (o Output) Compare(dir string, semantic bool) (map[string]CompareResult, error)

Compare compares files on disk to what's in the output.

If 'semantic' is true, for output files based on proto messages uses semantic comparison, i.e. loads the file on disk as a proto message and compares it to the output message. If 'semantic' is false, just always compares files as byte blobs.

For each file in the output set, the resulting map has a CompareResult describing how it compares to the file on disk. They can either be identical as byte blobs (Identical), different as byte blobs, but semantically the same (SemanticallyEqual), or totally different (Different).

Note that when 'semantic' is false, only Identical and Different can appear in the result, since we compare files as byte blobs only, so there's no notion of being "semantically the same".

Files on disk that are not in the output set are totally ignored. Files in the output set that are missing on disk as Different.

Returns an error if some file on disk can't be read or some output file can't be serialized.

func (Output) ConfigSets

func (o Output) ConfigSets() ([]ConfigSet, error)

ConfigSets partitions this output into 0 or more config sets based on Roots.

Returns an error if some output Datum can't be serialized.

func (Output) DebugDump

func (o Output) DebugDump()

DebugDump writes the output to stdout in a format useful for debugging.

func (Output) DiscardChangesToUntracked

func (o Output) DiscardChangesToUntracked(ctx context.Context, tracked []string, dir string) error

DiscardChangesToUntracked replaces bodies of the files that are in the output set, but not in the `tracked` set (per TrackedSet semantics) with what's on disk in the given `dir`.

This allows to construct partially generated output: some configs (the ones in the tracked set) are generated, others are loaded from disk.

If `dir` is "-" (which indicates that the output is going to be dumped to stdout rather then to disk), just removes untracked files from the output.

func (Output) Files

func (o Output) Files() []string

Files returns a sorted list of file names in the output.

func (Output) Read

func (o Output) Read(dir string) error

Read replaces values in o.Data by reading them from disk as blobs.

Returns an error if some file can't be read.

func (Output) Write

func (o Output) Write(dir string, force bool) (written, untouched []string, err error)

Write updates files on disk to match the output.

Returns a list of written files and a list of files that were left untouched.

If 'force' is false, compares files on disk to the generated files using the semantic comparison. If they are all up-to-date (semantically) does nothing. If at least one file is stale, rewrites *all* not already identical files. That way all output files always have consistent formatting, but `lucicfg generate` still doesn't produce noop formatting changes by default (it piggy backs formatting changes onto real changes).

If 'force' is true, compares files as byte blobs and rewrites all files that changed as blobs. No semantic comparison is done.

Creates missing directories. Not atomic. All files have mode 0666.

type State

type State struct {
	Inputs  Inputs   // all inputs, exactly as passed to Generate.
	Output  Output   // all generated config files, populated at the end
	Meta    Meta     // lucicfg parameters, settable through Starlark
	Visited []string // visited Starlark modules from Inputs
	// contains filtered or unexported fields

State is mutated throughout execution of the script and at the end contains the final execution result.

It is available in the implementation of native functions exposed to the Starlark side. Starlark code operates with the state exclusively through these functions.

All Starlark code is executed sequentially in a single goroutine, thus the state is not protected by any mutexes.

func Generate

func Generate(ctx context.Context, in Inputs) (*State, error)

Generate interprets the high-level config.

Returns a multi-error with all captured errors. Some of them may implement BacktracableError interface.

type ValidationMessage

Alias some ridiculously long type names that we round-trip in the public API.

type ValidationRequest

Alias some ridiculously long type names that we round-trip in the public API.

type ValidationResult

type ValidationResult struct {
	ConfigSet string               `json:"config_set"`          // a config set being validated
	Failed    bool                 `json:"failed"`              // true if the config is bad
	Messages  []*ValidationMessage `json:"messages"`            // errors, warnings, infos, etc.
	RPCError  string               `json:"rpc_error,omitempty"` // set if the RPC itself failed

ValidationResult is what we get after validating a config set.

func (*ValidationResult) Format

func (vr *ValidationResult) Format() string

Format formats the validation result as a multi-line string

func (*ValidationResult) OverallError

func (vr *ValidationResult) OverallError(failOnWarnings bool) error

OverallError is nil if the validation succeeded or non-nil if failed.

Beware: mutates Failed field accordingly.


Path Synopsis
Package buildifier implements processing of Starlark files via buildifier.
Package buildifier implements processing of Starlark files via buildifier.
Package cli contains command line interface for lucicfg tool.
Package cli contains command line interface for lucicfg tool.
Package base contains code shared by other CLI subpackages.
Package base contains code shared by other CLI subpackages.
Package fmt implements 'fmt' subcommand.
Package fmt implements 'fmt' subcommand.
Package generate implements 'generate' subcommand.
Package generate implements 'generate' subcommand.
Package lint implements 'lint' subcommand.
Package lint implements 'lint' subcommand.
Package validate implements 'validate' subcommand.
Package validate implements 'validate' subcommand.
Command docgen is the documentation generator.
Command docgen is the documentation generator.
Command lucicfg is CLI for LUCI config generator.
Command lucicfg is CLI for LUCI config generator.
Package doc generates starlark documentation.
Package doc generates starlark documentation.
Package docgen generates documentation from Starlark code.
Package docgen generates documentation from Starlark code.
Package ast defines AST relevant for the documentation generation.
Package ast defines AST relevant for the documentation generation.
Package docstring parses docstrings into more structured representation.
Package docstring parses docstrings into more structured representation.
Package symbols defines a data model representing Starlark symbols.
Package symbols defines a data model representing Starlark symbols.
Package graph implements a DAG used internally to represent config objects.
Package graph implements a DAG used internally to represent config objects.
Package starlark contains Starlark code embedded into lucicfg.
Package starlark contains Starlark code embedded into lucicfg.
Package vars implement lucicfg.var() support.
Package vars implement lucicfg.var() support.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL