README

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

[TOC]

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's internal Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image would be something like

FROM marketing.gcr.io/google/ubuntu1804:latest
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get -y update \
  && \
  apt-get install -f -y build-essential lsb-release python \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --allowed-users.

Log messages will be output to stderr.

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o

Directories

Path Synopsis
Package auth provides authentication service.
Package auth provides authentication service.
account
Package account manages service account.
Package account manages service account.
acl
Package acl performs access control with ACL.
Package acl performs access control with ACL.
authdb
Package authdb provides access to authentication database.
Package authdb provides access to authentication database.
enduser
Package enduser manages end user information with context.
Package enduser manages end user information with context.
Package backend is goma service backend of /cxx-compiler-service/*.
Package backend is goma service backend of /cxx-compiler-service/*.
Package bytestreamio provides io interfaces on bytestream service.
Package bytestreamio provides io interfaces on bytestream service.
Package cache provides cache service.
Package cache provides cache service.
gcs
Package gcs provides cache service by google cloud storage.
Package gcs provides cache service by google cloud storage.
redis
Package redis provides cache service by redis (cloud memorystore).
Package redis provides cache service by redis (cloud memorystore).
cmd
auth_server
Binary auth_server provides auth service via gRPC.
Binary auth_server provides auth service via gRPC.
cache_server
Binary cache_server provides cache service via gRPC.
Binary cache_server provides cache service via gRPC.
exec_server
Binary exec_server provides goma exec service via gRPC.
Binary exec_server provides goma exec service via gRPC.
execlog_server
Binary execlog_server provides goma execlog service via gRPC.
Binary execlog_server provides goma execlog service via gRPC.
file_server
Binary file_server provides goma file service via gRPC.
Binary file_server provides goma file service via gRPC.
frontend
Binary frontend is goma frontend.
Binary frontend is goma frontend.
goma_grpc_client
Binary goma_grpc_client is a simple gRPC client of goma api.
Binary goma_grpc_client is a simple gRPC client of goma api.
goma_replay
Binary goma_replay is a simple goma client for load testing etc.
Binary goma_replay is a simple goma client for load testing etc.
remoteexec_proxy
Binary remoteexec-proxy is a proxy server between Goma client and Remote Execution API.
Binary remoteexec-proxy is a proxy server between Goma client and Remote Execution API.
Package command manages commands/toolchains.
Package command manages commands/toolchains.
descriptor
Package descriptor provides command descriptor utilities.
Package descriptor provides command descriptor utilities.
descriptor/posixpath
Package posixpath handles posix-path (Unix style; slash separeted path).
Package posixpath handles posix-path (Unix style; slash separeted path).
descriptor/winpath
Package winpath handles windows-path (backslash separated path).
Package winpath handles windows-path (backslash separated path).
normalizer
Package normalizer provides functions to normalize target.
Package normalizer provides functions to normalize target.
pathconv
Package pathconv provides path converter between client and server.
Package pathconv provides path converter between client and server.
Package exec provides goma exec service implementation.
Package exec provides goma exec service implementation.
Package execlog provides goma execlog service implementation.
Package execlog provides goma execlog service implementation.
Package file provides goma file service implementation.
Package file provides goma file service implementation.
Package frontend is goma service frontend (/cxx-compiler-service/*).
Package frontend is goma service frontend (/cxx-compiler-service/*).
Package fswatch watches directory with fsnotify.
Package fswatch watches directory with fsnotify.
Package hash provides a hash function used in goma.
Package hash provides a hash function used in goma.
Package httprpc provides goma httprpc implementation.
Package httprpc provides goma httprpc implementation.
authdb
Package authdb implements authdb service for goma httprpc.
Package authdb implements authdb service for goma httprpc.
bytestream
Package bytestream implements bytestream for goma http.
Package bytestream implements bytestream for goma http.
exec
Package exec implements exec service for goma httprpc.
Package exec implements exec service for goma httprpc.
execlog
Package execlog implements log service for goma httprpc.
Package execlog implements log service for goma httprpc.
file
Package file implements file service for goma httprpc.
Package file implements file service for goma httprpc.
settings
Package settings implements settings service for goma httprpc.
Package settings implements settings service for goma httprpc.
log
Package log provides logging mechanism for goma servers.
Package log provides logging mechanism for goma servers.
errorreporter
Package errorreporter provides error reporting functionality.
Package errorreporter provides error reporting functionality.
Package profiler provides convenient function to enable cloud profiler.
Package profiler provides convenient function to enable cloud profiler.
Package proto is top directory for proto packages.
Package proto is top directory for proto packages.
api
Package remoteexec provides proxy to remoteexec server.
Package remoteexec provides proxy to remoteexec server.
cas
Package cas manages content addressable storage.
Package cas manages content addressable storage.
datasource
Package datasource provides data source from local file, bytes etc.
Package datasource provides data source from local file, bytes etc.
digest
Package digest handles content digest for remote executon API, https://github.com/bazelbuild/remote-apis/blob/c1c1ad2c97ed18943adb55f06657440daa60d833/build/bazel/remote/execution/v2/remote_execution.proto#L633
Package digest handles content digest for remote executon API, https://github.com/bazelbuild/remote-apis/blob/c1c1ad2c97ed18943adb55f06657440daa60d833/build/bazel/remote/execution/v2/remote_execution.proto#L633
merkletree
Package merkletree operates on a merkle tree for remote execution API, https://github.com/bazelbuild/remote-apis/blob/c1c1ad2c97ed18943adb55f06657440daa60d833/build/bazel/remote/execution/v2/remote_execution.proto#L838 see https://en.Wikipedia.org/wiki/Merkle_tree
Package merkletree operates on a merkle tree for remote execution API, https://github.com/bazelbuild/remote-apis/blob/c1c1ad2c97ed18943adb55f06657440daa60d833/build/bazel/remote/execution/v2/remote_execution.proto#L838 see https://en.Wikipedia.org/wiki/Merkle_tree
rpc
Package rpc provides goma specific rpc features on gRPC.
Package rpc provides goma specific rpc features on gRPC.
grpctest
Package grpctest provides a test server for unit tests that use gRPC.
Package grpctest provides a test server for unit tests that use gRPC.
Package server provides functions for goma servers.
Package server provides functions for goma servers.
healthz
Package healthz provides /healthz for grpc server.
Package healthz provides /healthz for grpc server.