README

Go Build Tools

This subrepository holds the source for various packages and tools that support Go's build system and the development of the Go programming language.

Report Issues / Send Patches

This repository uses Gerrit for code changes. To contribute, see https://golang.org/doc/contribute.html.

The main issue tracker for the blog is located at https://github.com/golang/go/issues. Prefix your issue with "x/build/DIR: " in the subject line.

Overview

The main components of the Go build system are:

  • The dashboard, in app/, serves https://build.golang.org/. It runs on App Engine and holds the state for which builds passed or failed, and stores the build failure logs for post-submit failures. (Trybot build failure logs are stored elsewhere). The dashboard does not execute any builds on its own.

  • The coordinator, in cmd/coordinator/, serves https://farmer.golang.org/. It runs on GKE and coordinates the whole build system. It finds work to do (both pre-submit "TryBot" work, and post-submit work) and executes builds, allocating machines to run the builds. It is the owner of all machines.

  • The Go package in buildenv/ contains constants for where the dashboard and coordinator run, for prod, staging, and local development.

  • The buildlet, in cmd/buildlet/, is the HTTP server that runs on each worker machine to execute builds on the coordinator's behalf. This runs on every possible GOOS/GOARCH value. The buildlet binaries are stored on Google Cloud Storage and fetched per-build, so we can update the buildlet binary independently of the underlying machine images. The buildlet is the most insecure server possible: it has HTTP handlers to read & write arbitrary content to disk, and to execute any file on disk. It also has an SSH tunnel handler. The buildlet must never be exposed to the Internet. The coordinator provisions buildlets in one of three ways:

    1. by creating VMs on Google Compute Engine (GCE) with custom images configured to fetch & run the buildlet on boot, listening on port 80 in a private network.

    2. by running Linux containers (on either Google Kubernetes Engine or GCE with the Container-Optimized OS image), with the container images configured to fetch & run the buildlet on start, also listening on port 80 in a private network.

    3. by taking buildlets out of a pool of connected, dedicated machines. The buildlet can run in either listen mode (as on GCE and GKE) or in reverse mode. In reverse mode, the buildlet connects out to https://farmer.golang.org/ and registers itself with the coordinator. The TCP connection is then logically reversed (using revdial and when the coordinator needs to do a build, it makes HTTP requests to the coordinator over the already-open TCP connection.

    These three pools can be viewed at the coordinator's http://farmer.golang.org/#pools

  • The env/ directory describes build environments. It contains scripts to create VM images, Dockerfiles to create Kubernetes containers, and instructions and tools for dedicated machines.

  • maintner in maintner/ is a library for slurping all of Go's GitHub and Gerrit state into memory. The daemon maintnerd in maintner/maintnerd/ runs on GKE and serves https://maintner.golang.org/. The daemon watches GitHub and Gerrit and apps to a mutation log whenever it sees new activity. The logs are stored on GCS and served to clients.

  • The godata package in maintner/godata/ provides a trivial API to let anybody write programs against Go's maintner corpus (all of our GitHub and Gerrit history), live up to the second. It takes a few seconds to load into memory and a few hundred MB of RAM after it downloads the mutation log from the network.

  • pubsubhelper in cmd/pubsubhelper/ is a dependency of maintnerd. It runs on GKE, is available at https://pubsubhelper.golang.org/, and runs an HTTP server to receive Webhook updates from GitHub on new activity and an SMTP server to receive new activity emails from Gerrit. It then is a pubsub system for maintnerd to subscribe to.

  • The gitmirror server in cmd/gitmirror/ mirrors Gerrit to GitHub, and also serves a mirror of the Gerrit code to the coordinator for builds, so we don't overwhelm Gerrit and blow our quota.

  • The Go gopherbot bot logic runs on GKE. The code is in cmd/gopherbot. It depends on maintner via the godata package.

  • The developer dashboard at https://dev.golang.org/ runs on GKE. Its code is in devapp/. It also depends on maintner via the godata package.

  • cmd/retrybuilds: a Go client program to delete build results from the dashboard

Adding a Go Builder

If you wish to run a Go builder, please email golang-dev@googlegroups.com first. There is documentation at https://golang.org/wiki/DashboardBuilders, but depending on the type of builder, we may want to run it ourselves, after you prepare an environment description (resulting in a VM image) of it. See the env directory.

Documentation

Overview

    Package build contains constants for the Go continuous build system.

    Index

    Constants

    View Source
    const DevCoordinatorCA = `` /* 955-byte string literal not displayed */
    

      DevCoordinatorCA is the cert used by the coordinator and buildlet in development mode. (Not to be confused with the staging "dev" instance under GCE project "go-dashboard-dev")

      View Source
      const DevCoordinatorKey = `` /* 1674-byte string literal not displayed */
      

        DevCoordinatorKey is the key used by the coordinator and buildlet in development mode. (Not to be confused with the staging "dev" instance under GCE project "go-dashboard-dev")

        View Source
        const ProdCoordinatorCA = `` /* 1114-byte string literal not displayed */
        

          ProdCoordinatorCA is the production CA cert for farmer.golang.org.

          View Source
          const StagingCoordinatorCA = `` /* 1073-byte string literal not displayed */
          

            StagingCoordinatorCA is the cert used on GCE for the go-dashboard-dev project.

            Variables

            This section is empty.

            Functions

            This section is empty.

            Types

            type CoordinatorInstance

            type CoordinatorInstance string

              CoordinatorInstance is either "prod", "staging", or "localhost:<port>".

              const (
              	ProdCoordinator    CoordinatorInstance = "prod"
              	StagingCoordinator CoordinatorInstance = "staging"
              )

              func (CoordinatorInstance) CACert

              func (ci CoordinatorInstance) CACert() string

                CACert returns the public certificate of the CA used to sign this coordinator instance's certificate.

                func (CoordinatorInstance) TLSDialer

                func (ci CoordinatorInstance) TLSDialer() func(network, addr string) (net.Conn, error)

                func (CoordinatorInstance) TLSHostPort

                func (ci CoordinatorInstance) TLSHostPort() (string, error)

                Source Files

                Directories

                Path Synopsis
                Package autocertcache contains autocert.Cache implementations for golang.org/x/crypto/autocert.
                Package autocertcache contains autocert.Cache implementations for golang.org/x/crypto/autocert.
                Package buildenv contains definitions for the environments the Go build system can run in.
                Package buildenv contains definitions for the environments the Go build system can run in.
                Package buildlet contains client tools for working with a buildlet server.
                Package buildlet contains client tools for working with a buildlet server.
                cloudfns
                issueswebhook Module
                sendwikidiff Module
                wikiwebhook Module
                cmd
                builder
                The builder binary is the old continuous build client for the Go project.
                The builder binary is the old continuous build client for the Go project.
                buildlet
                The buildlet is an HTTP server that untars content to disk and runs commands it has untarred, streaming their output back over HTTP.
                The buildlet is an HTTP server that untars content to disk and runs commands it has untarred, streaming their output back over HTTP.
                buildlet/stage0
                The stage0 command looks up the buildlet's URL from its environment (GCE metadata service, scaleway, etc), downloads it, and runs it.
                The stage0 command looks up the buildlet's URL from its environment (GCE metadata service, scaleway, etc), downloads it, and runs it.
                buildstats
                The buildstats command syncs build logs from Datastore to Bigquery.
                The buildstats command syncs build logs from Datastore to Bigquery.
                cl
                CL prints a list of open Go code reviews (also known as change lists, or CLs).
                CL prints a list of open Go code reviews (also known as change lists, or CLs).
                coordinator
                The coordinator runs the majority of the Go build system.
                The coordinator runs the majority of the Go build system.
                coordinator/metrics
                Package metrics enumerates the set of Stackdriver metrics used by the Go build system.
                Package metrics enumerates the set of Stackdriver metrics used by the Go build system.
                coordinator/spanlog
                Package spanlog provides span and event logger interfaces that are used by the build coordinator infrastructure.
                Package spanlog provides span and event logger interfaces that are used by the build coordinator infrastructure.
                debugnewvm
                The debugnewvm command creates and destroys a VM-based GCE buildlet with lots of logging for debugging.
                The debugnewvm command creates and destroys a VM-based GCE buildlet with lots of logging for debugging.
                docker2boot
                The docker2boot command converts a Docker image into a bootable GCE VM image.
                The docker2boot command converts a Docker image into a bootable GCE VM image.
                fetchlogs
                Fetchlogs downloads build failure logs from the Go dashboard so they can be accessed and searched from the local file system.
                Fetchlogs downloads build failure logs from the Go dashboard so they can be accessed and searched from the local file system.
                genbootstrap
                The genbootstrap command prepares GO_BOOTSTRAP tarballs suitable for use on builders.
                The genbootstrap command prepares GO_BOOTSTRAP tarballs suitable for use on builders.
                gerritbot
                The gerritbot binary converts GitHub Pull Requests to Gerrit Changes, updating the PR and Gerrit Change as appropriate.
                The gerritbot binary converts GitHub Pull Requests to Gerrit Changes, updating the PR and Gerrit Change as appropriate.
                gitlock
                The gitlock command helps write Dockerfiles with a bunch of lines to lock git dependencies in place.
                The gitlock command helps write Dockerfiles with a bunch of lines to lock git dependencies in place.
                gitmirror
                The gitmirror binary watches the specified Gerrit repositories for new commits and reports them to the build dashboard.
                The gitmirror binary watches the specified Gerrit repositories for new commits and reports them to the build dashboard.
                gomote
                The gomote command is a client for the Go builder infrastructure.
                The gomote command is a client for the Go builder infrastructure.
                gopherbot
                The gopherbot command runs Go's gopherbot role account on GitHub and Gerrit.
                The gopherbot command runs Go's gopherbot role account on GitHub and Gerrit.
                makemac
                The makemac command starts OS X VMs for the builders.
                The makemac command starts OS X VMs for the builders.
                perfrun
                perfrun interacts with the buildlet coordinator to run the go1 benchmarks on a buildlet slave for the most recent successful commits according to the build dashboard.
                perfrun interacts with the buildlet coordinator to run the go1 benchmarks on a buildlet slave for the most recent successful commits according to the build dashboard.
                pubsubhelper
                The pubsubhelper is an SMTP server for Gerrit updates and an HTTP server for Github webhook updates.
                The pubsubhelper is an SMTP server for Gerrit updates and an HTTP server for Github webhook updates.
                pubsubhelper/pubsubtypes
                Package pubsubtypes contains types published by pubsubhelper.
                Package pubsubtypes contains types published by pubsubhelper.
                racebuild
                racebuild builds the race runtime (syso files) on all supported OSes using gomote.
                racebuild builds the race runtime (syso files) on all supported OSes using gomote.
                release
                Command release builds a Go release.
                Command release builds a Go release.
                releasebot
                Releasebot manages the process of defining, packaging, and publishing Go releases.
                Releasebot manages the process of defining, packaging, and publishing Go releases.
                relnote
                The relnote command summarizes the Go changes in Gerrit marked with RELNOTE annotations for the release notes.
                The relnote command summarizes the Go changes in Gerrit marked with RELNOTE annotations for the release notes.
                retrybuilds
                The retrybuilds command clears build failures from the build.golang.org dashboard to force them to be rebuilt.
                The retrybuilds command clears build failures from the build.golang.org dashboard to force them to be rebuilt.
                rmplaysnippet
                The rmplaysnippet binary removes a code snippet from play.golang.org given its URL or ID.
                The rmplaysnippet binary removes a code snippet from play.golang.org given its URL or ID.
                rundockerbuildlet
                The rundockerbuildlet command loops forever and creates and cleans up Docker containers running reverse buildlets.
                The rundockerbuildlet command loops forever and creates and cleans up Docker containers running reverse buildlets.
                scaleway
                The scaleway command creates ARM servers on Scaleway.com.
                The scaleway command creates ARM servers on Scaleway.com.
                tip
                Command tip is the tip.golang.org server, serving the latest HEAD straight from the Git oven.
                Command tip is the tip.golang.org server, serving the latest HEAD straight from the Git oven.
                updatecontrib
                The updateac command updates the CONTRIBUTORS file in the Go repository.
                The updateac command updates the CONTRIBUTORS file in the Go repository.
                upload
                The upload command writes a file to Google Cloud Storage.
                The upload command writes a file to Google Cloud Storage.
                xb
                crfs module
                Package dashboard contains shared configuration and logic used by various pieces of the Go continuous build system.
                Package dashboard contains shared configuration and logic used by various pieces of the Go continuous build system.
                Devapp is the server running dev.golang.org.
                Devapp is the server running dev.golang.org.
                Package envutil provides utilities for working with environment variables.
                Package envutil provides utilities for working with environment variables.
                Package gerrit contains code to interact with Gerrit servers.
                Package gerrit contains code to interact with Gerrit servers.
                Package kubernetes contains a minimal client for the Kubernetes API.
                Package kubernetes contains a minimal client for the Kubernetes API.
                api
                Package api contains the Kubernetes v1 API types.
                Package api contains the Kubernetes v1 API types.
                gke
                Package gke contains code for interacting with Google Container Engine (GKE), the hosted version of Kubernetes on Google Cloud Platform.
                Package gke contains code for interacting with Google Container Engine (GKE), the hosted version of Kubernetes on Google Cloud Platform.
                Package livelog provides a buffer that can be simultaneously written to by one writer and read from by many readers.
                Package livelog provides a buffer that can be simultaneously written to by one writer and read from by many readers.
                Package maintner mirrors, searches, syncs, and serves Git, Github, and Gerrit metadata.
                Package maintner mirrors, searches, syncs, and serves Git, Github, and Gerrit metadata.
                cmd/maintserve
                maintserve is a program that serves Go issues and CLs over HTTP, so they can be viewed in a browser.
                maintserve is a program that serves Go issues and CLs over HTTP, so they can be viewed in a browser.
                godata
                Package godata loads the Go project's corpus of Git, Github, and Gerrit activity into memory to allow easy analysis without worrying about APIs and their pagination, quotas, and other nuisances and limitations.
                Package godata loads the Go project's corpus of Git, Github, and Gerrit activity into memory to allow easy analysis without worrying about APIs and their pagination, quotas, and other nuisances and limitations.
                gostats
                The gostats command computes stats about the Go project.
                The gostats command computes stats about the Go project.
                maintnerd
                The maintnerd command serves project maintainer data from Git, Github, and/or Gerrit.
                The maintnerd command serves project maintainer data from Git, Github, and/or Gerrit.
                maintnerd/apipb
                Package apipb is a generated protocol buffer package.
                Package apipb is a generated protocol buffer package.
                maintnerd/gcslog
                Package gcslog is an implementation of maintner.MutationSource and Logger for Google Cloud Storage.
                Package gcslog is an implementation of maintner.MutationSource and Logger for Google Cloud Storage.
                maintnerd/maintapi
                Package maintapi exposes a gRPC maintner service for a given corpus.
                Package maintapi exposes a gRPC maintner service for a given corpus.
                maintnerd/maintapi/version
                Package version implements logic to parse version of Go tags and release branches.
                Package version implements logic to parse version of Go tags and release branches.
                maintpb
                Package maintpb is a generated protocol buffer package.
                Package maintpb is a generated protocol buffer package.
                maintq
                The maintq command queries a maintnerd gRPC server.
                The maintq command queries a maintnerd gRPC server.
                reclog
                Package reclog contains readers and writers for a record wrapper format used by maintner.
                Package reclog contains readers and writers for a record wrapper format used by maintner.
                Package pargzip contains a parallel gzip writer implementation.
                Package pargzip contains a parallel gzip writer implementation.
                Package revdial implements a Dialer and Listener which work together to turn an accepted connection (for instance, a Hijacked HTTP request) into a Dialer which can then create net.Conns connecting back to the original dialer, which then gets a net.Listener accepting those conns.
                Package revdial implements a Dialer and Listener which work together to turn an accepted connection (for instance, a Hijacked HTTP request) into a Dialer which can then create net.Conns connecting back to the original dialer, which then gets a net.Listener accepting those conns.
                Package status contains code for monitoring the golang.org infrastructure.
                Package status contains code for monitoring the golang.org infrastructure.
                statusserver
                Package statusserver contains the status server for the golang.org properties and build infrastructure.
                Package statusserver contains the status server for the golang.org properties and build infrastructure.
                Package tarutil contains utilities for working with tar archives.
                Package tarutil contains utilities for working with tar archives.
                Package types contains common types used by the Go continuous build system.
                Package types contains common types used by the Go continuous build system.
                vcs-test
                The version package permits running a specific version of Go.
                The version package permits running a specific version of Go.
                go1.10
                The go1.10 command runs the go command from go1.10.
                The go1.10 command runs the go command from go1.10.
                go1.10.1
                The go1.10.1 command runs the go command from Go 1.10.1.
                The go1.10.1 command runs the go command from Go 1.10.1.
                go1.10.2
                The go1.10.2 command runs the go command from go1.10.2.
                The go1.10.2 command runs the go command from go1.10.2.
                go1.10.3
                The go1.10.3 command runs the go command from go1.10.3.
                The go1.10.3 command runs the go command from go1.10.3.
                go1.10beta1
                The go1.10beta1 command runs the go command from go1.10beta1.
                The go1.10beta1 command runs the go command from go1.10beta1.
                go1.10beta2
                The go1.10beta2 command runs the go command from go1.10beta2.
                The go1.10beta2 command runs the go command from go1.10beta2.
                go1.10rc1
                The go1.10rc1 command runs the go command from go1.10rc1.
                The go1.10rc1 command runs the go command from go1.10rc1.
                go1.10rc2
                The go1.10rc2 command runs the go command from go1.10rc2.
                The go1.10rc2 command runs the go command from go1.10rc2.
                go1.11beta1
                The go1.11beta1 command runs the go command from go1.11beta1.
                The go1.11beta1 command runs the go command from go1.11beta1.
                go1.8
                The go1.8 command runs the go command from go1.8.
                The go1.8 command runs the go command from go1.8.
                go1.8.1
                The go1.8.1 command runs the go command from go1.8.1.
                The go1.8.1 command runs the go command from go1.8.1.
                go1.8.2
                The go1.8.2 command runs the go command from go1.8.2.
                The go1.8.2 command runs the go command from go1.8.2.
                go1.8.3
                The go1.8.3 command runs the go command from go1.8.3.
                The go1.8.3 command runs the go command from go1.8.3.
                go1.8.4
                The go1.8.4 command runs the go command from go1.8.4.
                The go1.8.4 command runs the go command from go1.8.4.
                go1.8.5
                The go1.8.5 command runs the go command from go1.8.5.
                The go1.8.5 command runs the go command from go1.8.5.
                go1.8.6
                The go1.8.6 command runs the go command from go1.8.6.
                The go1.8.6 command runs the go command from go1.8.6.
                go1.8.7
                The go1.8.7 command runs the go command from go1.8.7.
                The go1.8.7 command runs the go command from go1.8.7.
                go1.8beta1
                The go1.8beta1 command runs the go command from go1.8beta1.
                The go1.8beta1 command runs the go command from go1.8beta1.
                go1.8beta2
                The go1.8beta2 command runs the go command from go1.8beta2.
                The go1.8beta2 command runs the go command from go1.8beta2.
                go1.8rc1
                The go1.8rc1 command runs the go command from go1.8rc1.
                The go1.8rc1 command runs the go command from go1.8rc1.
                go1.8rc2
                The go1.8rc2 command runs the go command from go1.8rc2.
                The go1.8rc2 command runs the go command from go1.8rc2.
                go1.8rc3
                The go1.8rc3 command runs the go command from go1.8rc3.
                The go1.8rc3 command runs the go command from go1.8rc3.
                go1.9
                The go1.9 command runs the go command from go1.9.
                The go1.9 command runs the go command from go1.9.
                go1.9.1
                The go1.9.1 command runs the go command from go1.9.1.
                The go1.9.1 command runs the go command from go1.9.1.
                go1.9.2
                The go1.9.2 command runs the go command from go1.9.2.
                The go1.9.2 command runs the go command from go1.9.2.
                go1.9.3
                The go1.9.3 command runs the go command from go1.9.3.
                The go1.9.3 command runs the go command from go1.9.3.
                go1.9.4
                The go1.9.4 command runs the go command from go1.9.4.
                The go1.9.4 command runs the go command from go1.9.4.
                go1.9.5
                The go1.9.5 command runs the go command from Go 1.9.5.
                The go1.9.5 command runs the go command from Go 1.9.5.
                go1.9.6
                The go1.9.6 command runs the go command from go1.9.6.
                The go1.9.6 command runs the go command from go1.9.6.
                go1.9.7
                The go1.9.7 command runs the go command from go1.9.7.
                The go1.9.7 command runs the go command from go1.9.7.
                go1.9beta1
                The go1.9beta1 command runs the go command from go1.9beta1.
                The go1.9beta1 command runs the go command from go1.9beta1.
                go1.9beta2
                The go1.9beta2 command runs the go command from go1.9beta2.
                The go1.9beta2 command runs the go command from go1.9beta2.
                go1.9rc1
                The go1.9rc1 command runs the go command from go1.9rc1.
                The go1.9rc1 command runs the go command from go1.9rc1.
                go1.9rc2
                The go1.9rc2 command runs the go command from go1.9rc2.
                The go1.9rc2 command runs the go command from go1.9rc2.
                internal
                buildgo
                Package buildgo provides tools for pushing and building the Go distribution on buildlets.
                Package buildgo provides tools for pushing and building the Go distribution on buildlets.
                buildstats
                Package buildstats contains code to sync the coordinator's build logs from Datastore to BigQuery.
                Package buildstats contains code to sync the coordinator's build logs from Datastore to BigQuery.
                gitauth
                Package gitauth writes gitcookies files so git will authenticate to Gerrit as gopherbot for quota purposes.
                Package gitauth writes gitcookies files so git will authenticate to Gerrit as gopherbot for quota purposes.
                gophers
                Package gophers is a list of names, emails, and Github usernames of people from the Go git repos and issue trackers.
                Package gophers is a list of names, emails, and Github usernames of people from the Go git repos and issue trackers.
                httpdl
                Package httpdl downloads things from HTTP to local disk.
                Package httpdl downloads things from HTTP to local disk.
                https
                Package https contains helpers for starting an HTTPS server.
                Package https contains helpers for starting an HTTPS server.
                loghash
                Package loghash provides the shared information for computing a log hash (as in https://build.golang.org/log/HASH).
                Package loghash provides the shared information for computing a log hash (as in https://build.golang.org/log/HASH).
                lru
                Package lru implements an LRU cache.
                Package lru implements an LRU cache.
                singleflight
                Package singleflight provides a duplicate function call suppression mechanism.
                Package singleflight provides a duplicate function call suppression mechanism.
                sourcecache
                Package sourcecache provides a cache of code found in Git repositories.
                Package sourcecache provides a cache of code found in Git repositories.
                untar
                Package untar untars a tarball to disk.
                Package untar untars a tarball to disk.