cv/

directory
v0.0.0-...-9804853 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 8, 2024 License: Apache-2.0

README

LUCI Change Verifier

LUCI Change Verifier (CV) is the LUCI microservice that is responsible for running pre-submit tests and submitting CLs when they pass all checks.

What's here?

  • api: Protobuf files specifying the public API of CV, such as per-project config spec, PubSub messages and RPCs definitions. APIs organization is a bit messy right now, but of note are 2 versions:
    • v0: experimental API under development. Request an approval from CV owners before using it.
    • v1: stable API, which won't be broken unless absolutely necessary. As of Sep 2021, it's under the development.
  • appengine: the entry point for a GAE app.
  • internal: GAE-agnostic implementation details in Go. See godoc-based overview here. Notably:
    • internal/cvtesting/e2e high level end-to-end level CV tests, which cover the most important business logic in a fairly readable and concise way. The test are run against a fake Gerrit.
    • internal/changelist: encapsulates a changelist (a.k.a. CL, patch, Gerrit Change).
    • internal/acls: enforces ALCs.
    • internal/tryjob: manages tryjobs (i.e. Buildbucket builds) which are used to verify a CL.
    • internal/prjmanager and sub-packages: decides when to start new Runs given the state of all CLs in an individual LUCI project.
    • internal/run and sub-packages: handles individual Run from its creation to completion.
    • internal/rpc pRPC handlers, including "admin" API not exposed in public api dir.

Developer Guide

Error handling guide

Like all other LUCI go code:

  • SHOULD wrap errors with errors.Annotate to provide additional context. Unlike other error-wrapping packages, it also captures the call stack.

  • SHOULD avoid ignoring errors. If it's justified, add a comment in code and if necessary log the error.

  • SHOULD use appstatus on errors in RPC-handling codepaths.

LUCI CV also follows these conventions, which in some cases differ from other LUCI Go code:

  • SHOULD use CV's common.LogError(ctx, err) for logging error + stack trace instead of errors.Log(ctx, err). The CV's version packs the entire stack into as few log entries as possible, leading to a better debugging experience with Cloud Logging.

  • Tag the error with transient.Tag if the function call can succeed on a retry. For example, most Datastore errors different from NoSuchEntity SHOULD be thus tagged.

  • SHOULD use common.TQifyError or its custom version common.TQIfy{...}.Error(ctx, err) before returning from a TQ task handler. This func logs the error with appropriate severity and with stack trace as necessary (via common.LogError). Additionally, it converts the error to an appropriate server/tq tag as needed. The custom common.TQIfy SHOULD be used to reduce noise in logs and oncall alerting from the frequent errors during normal operation which don't critically harm the service, e.g. "ErrStaleGerritData".

  • MAY panic when a function precondition fails, e.g. sqrt(x) may panic if x is negative. This is like an assert in Python / C / C++, which is contrary to Go's standard guidance.

  • MUST return a singular error from a function instead of errors.MultiError or similar multi-error holders unless all of the below hold true, which was so far very rare in CV code:

    • the function signature clearly states that a multi-error is returned, e.g., func loadMany(clids ... int64) ([]*CL, errors.MultiError);
    • the caller must examine each individual error, e.g. choosing to create missing CLs based on loadMany() errors;
    • no transient.Tag or annotation attached to the mutli-error itself, though an individual sub-error may have them.

  • SHOULD use common.MostSevereError on a multi-error before returning. For example, a common pattern to choose 1 error after parallelizing is return common.MostSevereError(parallel.FanOut(...));

    • Individual errors which are thus ignored MAY be logged. This isn't required because in practice most such errors are usually correlated, and the most severe one is thus typically sufficient for debugging.
Full end to end testing of new features

  1. Land your code, which gets auto-deployed to luci-change-verifier-dev project, You may also just upload a tainted version and switch all traffic to it, but beware that the next auto-deployment will override it. Thus, adding unit- and e2e tests with fake Gerrit and Cloud dependencies is a good first step and at times cheaper/faster to do.

  2. cq-test LUCI project can be used for any such tests. It's already connected to only luci-change-verifier-dev. The cq-test project's config tells CV to watch 2 repositories:

    Creating a CL on refs/heads/main will use combinable config group, meaning multi-CL Runs in ChromeOS style can be created. For single-CL Runs, create CLs on refs/heads/single ref:

        git new-branch --upstream origin/single
    echo "fail, please" > touch_to_fail_tryjob
    echo "fail with INFRA_FAILURE, please" > touch_to_infra_fail_tryjob
    git commit -a -m "test a signle CL dry run with failing tryjobs"
    git cl upload -d

    You can see recent Runs in https://luci-change-verifier-dev.appspot.com/ui/recents/cq-test.

  3. (Optional) internal/cvtesting/e2e/manual contains tools to automate common tasks, e.g. creating and CQ-ing large CL stacks. Feel free to contribute :)

UI

tl;dr

cd appengine
go run main.go
# See output for URLs to http URLs.

To work with Runs from the -dev project, connect to its Datastore by adding these arguments:

go run main.go \
    -http-addr localhost:8800
    -cloud-project luci-change-verifier-dev \
    -root-secret devsecret://base64anything \
    -primary-tink-aead-key devsecret-gen://tink/aead

NOTE: if you want the old page tokens to work on subsequent go run main.go ... invocations, when you first invoke it, observe output of the first invocation which should mention a devsecret://veeeeeeeeeeeeeeery-looooooong-base64-line, which you can use on subsequent invocations instead of devsecret-gen://tink/aead.

For a quick check, eyeball these two pages:

Finally, you can deploy your work-in-progress to -dev project directly.

How does this code end up in production?

tl;dr this the usual way for LUCI GAE apps. Roughly,

  1. CL lands in luci-go (this) repo.

  2. Autoroller rolls it into infra/infra repo (example).

    • pro-tip: If autoroller is stuck, you can create your own DEPS roll with roll-dep go/src/go.chromium.org/luci in your infra/infra checkout.

  3. Tarball with only the necessary files is created by the Google-internal infra-gae-tarballs-continuous builder.

  4. Newest tarball is automatically rolled into also Google-internal infradata-gae repo (example).

  5. Google-internal gae-deploy builder auto-deploys the newest tarball to luci-change-verifier-dev GAE project.

  6. Someone manually bumps tarball version to deploy to production luci-change-verifier via a CL (example).

    • Recommended use bump-dev-to-prod.py tool to make such a CL. For example, 
      cd gae/app/luci-change-verifier
./bump-dev-to-prod.py -- --bug $BUG -r <REVIEWER> -d -s

  7. The same Google-internal gae-deploy builder deploys the desired version to production luci-change-verifier GAE project.

LUCI CV Command Line utils

LUCI Change Verifier provides a command line interface luci-cv intended for LUCI integrators to debug CV configurations.

Getting the luci-cv CLI binary.

There are two ways of getting the latest version of the binary at the moment:

  • Building it yourself from this package: go.chromium.org/luci/cv/cmd/luci-cv e.g.

    go build go.chromium/org/luci/cv/cmd/luci-cv

  • Getting it from CIPD e.g. via a command such as:

    cipd ensure -ensure-file - -root cv-cli <<< 'infra/tools/luci-cv/${platform} latest'

For the appropriate syntax and flags information refer to the binary's built-in documentation. e.g. luci-cv help match-config.

Examples
Check if CL is watched by CV and which config group applies
# (Assuming luci-cv was installed in the current dir)
~/infra/infra$ ./luci-cv match-config infra/config/generated/commit-queue.cfg https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/3214613

https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/3214613:
  Location: Host: chromium-review.googlesource.com, Repo: infra/luci/luci-go, Ref: refs/heads/main
  Matched: luci-go
  • Additional Google-internal docs are found at go/luci/cv.

Directories

Path Synopsis
api
bigquery/v1
config/legacy
Package tricium has a simplified version of Tricium Project Config proto.
 Package tricium has a simplified version of Tricium Project Config proto.
config/v2
Package cfgpb contains the CQ config schema.
 Package cfgpb contains the CQ config schema.
recipe/v1
Package recipe contains CQ Recipe input proto.
 Package recipe contains CQ Recipe input proto.
v0
Package cvpb contains v0 (preliminary) version of CV API.
 Package cvpb contains v0 (preliminary) version of CV API.
v1
Package cvpb contains v1 version of CV API.
 Package cvpb contains v1 version of CV API.
listener
Package main is the main entry point for the app.
 Package main is the main entry point for the app.
monitor
Package main is the main entry point for the app.
 Package main is the main entry point for the app.
cli is a package implementing command line utilities for CV.
 cli is a package implementing command line utilities for CV.
cmd
luci-cv
internal
acls
Package acls enforces CV ACLs.
 Package acls enforces CV ACLs.
aggrmetrics
Package aggrmetrics computes & reports aggregated metrics.
 Package aggrmetrics computes & reports aggregated metrics.
buildbucket
Package buildbucket contains Buildbucket-related utility functions.
 Package buildbucket contains Buildbucket-related utility functions.
buildbucket/facade
Package bbfacade provides a facade for CV Tryjob support, hiding Buildbucket-specifc implementation details.
 Package bbfacade provides a facade for CV Tryjob support, hiding Buildbucket-specifc implementation details.
buildbucket/fake
Package bbfake implements fake Buildbucket server and the client connects to the fake for needs of CV tests.
 Package bbfake implements fake Buildbucket server and the client connects to the fake for needs of CV tests.
buildbucket/listener
Package bblistener listens to build update notifications from Buildbucket Pub/Sub.
 Package bblistener listens to build update notifications from Buildbucket Pub/Sub.
changelist
Package changelist implements operations on a single CL.
 Package changelist implements operations on a single CL.
common
Package common contains widely used CV utilities & types.
 Package common contains widely used CV utilities & types.
common/bq
Package bq handles sending rows to BigQuery.
 Package bq handles sending rows to BigQuery.
common/eventbox
Package eventbox batches incoming events for a single Datastore entity for processing.
 Package eventbox batches incoming events for a single Datastore entity for processing.
common/eventbox/dsset
Package dsset implements a particular flavor of Datastore-on-Firestore backed set.
 Package dsset implements a particular flavor of Datastore-on-Firestore backed set.
common/lease
Package lease provides a way to "lock" an external resource with expiration time so that concurrent processes/task executions can achieve exclusive privilege to make mutations (generally long-running and non-idempotent) on that resource.
 Package lease provides a way to "lock" an external resource with expiration time so that concurrent processes/task executions can achieve exclusive privilege to make mutations (generally long-running and non-idempotent) on that resource.
common/pubsub
Package pubsub provides a generic way to batch pubsub pull notifications.
 Package pubsub provides a generic way to batch pubsub pull notifications.
common/tree
Package tree implements fetching tree status from Tree Status App.
 Package tree implements fetching tree status from Tree Status App.
common/tree/treetest
Package treetest implements fake Tree for testing in CV.
 Package treetest implements fake Tree for testing in CV.
configs/prjcfg
Package prjcfg handles project-scoped CV config.
 Package prjcfg handles project-scoped CV config.
configs/prjcfg/prjcfgtest
Package prjcfgtest eases controlling of project configs in test.
 Package prjcfgtest eases controlling of project configs in test.
configs/prjcfg/refresher
Package refresher handles RefreshProjectConfigTask.
 Package refresher handles RefreshProjectConfigTask.
configs/srvcfg
Package srvcfg provides service-wide configs.
 Package srvcfg provides service-wide configs.
configs/validation
Package validation validates CV config files.
 Package validation validates CV config files.
cvtesting
Package cvtesting reduces boilerplate in tests.
 Package cvtesting reduces boilerplate in tests.
cvtesting/benchmarks
Package benchmarks contains benchmarks for heavy CV use cases.
 Package benchmarks contains benchmarks for heavy CV use cases.
cvtesting/e2e
Package e2e contains all CV end-to-end tests.
 Package e2e contains all CV end-to-end tests.
cvtesting/saferesembletest
Package saferesembletest exists to test better cvtesting.SafeShouldResemble
 Package saferesembletest exists to test better cvtesting.SafeShouldResemble
gerrit
Package gerrit contains Gerrit utility functions.
 Package gerrit contains Gerrit utility functions.
gerrit/botdata
Package botdata implements parsing and generation logic for BotData.
 Package botdata implements parsing and generation logic for BotData.
gerrit/cfgmatcher
Package cfgmatcher efficiently matches a CL to 0+ ConfigGroupID for a single LUCI project.
 Package cfgmatcher efficiently matches a CL to 0+ ConfigGroupID for a single LUCI project.
gerrit/cqdepend
Package cqdepend parses CQ-Depend directives in CL description.
 Package cqdepend parses CQ-Depend directives in CL description.
gerrit/gerritfake
Package gerritfake implements fake Gerrit for needs of CV tests.
 Package gerritfake implements fake Gerrit for needs of CV tests.
gerrit/gobmap
Package gobmap finds relevant LUCI project config given a Gerrit CL.
 Package gobmap finds relevant LUCI project config given a Gerrit CL.
gerrit/gobmap/gobmaptest
Package gobmaptest eases use of gobmap package in tests.
 Package gobmaptest eases use of gobmap package in tests.
gerrit/listener
Package listener listens to Gerrit events.
 Package listener listens to Gerrit events.
gerrit/metadata
Package metadata can extract metadata from Gerrit CLs.
 Package metadata can extract metadata from Gerrit CLs.
gerrit/poller
Package poller polls Gerrit for interesting CV changes.
 Package poller polls Gerrit for interesting CV changes.
gerrit/trigger
Package trigger determines if and how Gerrit CL is triggered.
 Package trigger determines if and how Gerrit CL is triggered.
gerrit/updater
Package updater implements changelist.Updater backend for Gerrit.
 Package updater implements changelist.Updater backend for Gerrit.
metrics
Package metrics defines metrics and implements reporting utilities.
 Package metrics defines metrics and implements reporting utilities.
prjmanager
Package prjmanager implements public API for Project Manager.
 Package prjmanager implements public API for Project Manager.
prjmanager/clpurger
Package clpurger purges CLs with a trigger for which Runs can't be started.
 Package clpurger purges CLs with a trigger for which Runs can't be started.
prjmanager/cltriggerer
Package cltriggerer implements logic for triggering CLs.
 Package cltriggerer implements logic for triggering CLs.
prjmanager/copyonwrite
Package copyonwrite providers helpers for modifying slices in Copy-on-Write way.
 Package copyonwrite providers helpers for modifying slices in Copy-on-Write way.
prjmanager/itriager
Package itriager defines interface of a CL component triage process.
 Package itriager defines interface of a CL component triage process.
prjmanager/manager
Package manager implements a ProjectManager.
 Package manager implements a ProjectManager.
prjmanager/pmtest
Package pmtest implements tests for working with Project Manager.
 Package pmtest implements tests for working with Project Manager.
prjmanager/prjpb
Package prjpb stores protos for event processing of ProjectManager.
 Package prjpb stores protos for event processing of ProjectManager.
prjmanager/state
Package state implements state machine of a Project Manager.
 Package state implements state machine of a Project Manager.
prjmanager/triager
Package triager proposes concrete actions on a group of related CLs.
 Package triager proposes concrete actions on a group of related CLs.
quota
Package quota manages run and tryjobs quotas per user
 Package quota manages run and tryjobs quotas per user
retention
Package retention implements data retention for LUCI CV.
 Package retention implements data retention for LUCI CV.
rpc/admin
Package admin implements CV admin API.
 Package admin implements CV admin API.
rpc/admin/api
Package adminpb contains LUCI CV admin/maintainer APIs definitions.
 Package adminpb contains LUCI CV admin/maintainer APIs definitions.
rpc/pagination
Package pagination implements handling of page sizes & tokens in CV APIs.
 Package pagination implements handling of page sizes & tokens in CV APIs.
rpc/v0
Package rpc implements CV V0 pRPC APIs.
 Package rpc implements CV V0 pRPC APIs.
rpc/versioning
Package versioning contains utilities to convert types to/from CV API versions as well as tests to avoid regressions.
 Package versioning contains utilities to convert types to/from CV API versions as well as tests to avoid regressions.
run
Package run implements public interface for Run Manager.
 Package run implements public interface for Run Manager.
run/bq
Package bq provides functionality for sending finished Run rows to BigQuery.
 Package bq provides functionality for sending finished Run rows to BigQuery.
run/eventpb
Package eventpb stores protos for event processing of RunManager.
 Package eventpb stores protos for event processing of RunManager.
run/impl
Package impl implements a RunManager.
 Package impl implements a RunManager.
run/impl/handler
Package handler implements handlers that handles run events.
 Package handler implements handlers that handles run events.
run/impl/longops
Package longops implements long operations of the Run Manager.
 Package longops implements long operations of the Run Manager.
run/impl/state
Package state defines the model for a Run state.
 Package state defines the model for a Run state.
run/impl/submit
Package submit contains utilities for Run submission.
 Package submit contains utilities for Run submission.
run/impl/util
Package util contains the common utility functions shared by RunManager.
 Package util contains the common utility functions shared by RunManager.
run/postaction
Package postaction executes post actions for run termination events.
 Package postaction executes post actions for run termination events.
run/pubsub
Package pubsub implements utility functions for publishing various Run events to Cloud PubSub.
 Package pubsub implements utility functions for publishing various Run events to Cloud PubSub.
run/rdb
Package rdb provides functionality for notifying ResultDB of successfully merged invocations.
 Package rdb provides functionality for notifying ResultDB of successfully merged invocations.
run/runcreator
Package runcreator creates new Runs.
 Package runcreator creates new Runs.
run/runquery
Package runquery contains logic to query runs.
 Package runquery contains logic to query runs.
run/runtest
Package runtest implements tests for working with Run Manager.
 Package runtest implements tests for working with Run Manager.
tracing
Package tracing contains helper for reporting OpenTelemetry tracing spans.
 Package tracing contains helper for reporting OpenTelemetry tracing spans.
tryjob
Package tryjob handles individual Tryjob handling in CV.
 Package tryjob handles individual Tryjob handling in CV.
tryjob/execute
Package execute implements logic to execute tryjob requirement for Runs.
 Package execute implements logic to execute tryjob requirement for Runs.
tryjob/requirement
Package requirement computes Tryjob Requirements and provides various utility functions related to Tryjob Requirements.
 Package requirement computes Tryjob Requirements and provides various utility functions related to Tryjob Requirements.
tryjob/tjcancel
Package tjcancel contains code in charge of cancelling stale tryjobs.
 Package tjcancel contains code in charge of cancelling stale tryjobs.
tryjob/update
Package tjupdate contains Updater, which handles an UpdateTryjobTask.
 Package tjupdate contains Updater, which handles an UpdateTryjobTask.
userhtml
Package userhtml contains what's needed to render frontend content.
 Package userhtml contains what's needed to render frontend content.
usertext
Package usertext generates messages for users (humans) using Go templating system.
 Package usertext generates messages for users (humans) using Go templating system.
settings
listener
Package listenerpb controls the settings for LUCI CV listener.
 Package listenerpb controls the settings for LUCI CV listener.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.