README
¶
Goma server
Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.
This is reference implementation of server code to be used with Goma client.
[TOC]
Dependencies
The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's internal Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.
How to build
Goma server can be built on Linux.
$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy
You will get the binary in $(go env GOPATH)/bin.
How to run
remoteexec_proxy is a single server that acts as proxy server
between Goma client and Remote Execution API.
$ remoteexec_proxy --port $PORT \
--platform-container-image "docker://...@sha256:..." \
--remoteexec-addr $REMOTEEXEC_ADDR \
--remote-instance-name $REMOTE_INSTANCE_NAME
for chromium, platform container image would be something like
FROM marketing.gcr.io/google/ubuntu1804:latest
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get -y update \
&& \
apt-get install -f -y build-essential lsb-release python \
&& \
rm -rf /var/lib/apt/lists/*
If Remote Execution API requires service account,
specify service account JSON file for Remote Execution API by
--service-account-json.
Running user is granted by default. If you need to allow other users, you
need to specify them by --allowed-users.
Log messages will be output to stderr.
How to use
Install goma client. We provide prebuilt binary with cipd, which
is available in depot_tools.
$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma
or follow the
build instructions to
build your own local version of Goma client before running the server code,
and install it in $HOME/goma.
Need to authenticate Goma client before use.
$ $HOME/goma/goma_auth.py login
Specify hostname in $GOMA_SERVER_HOST
and port in $GOMA_SERVER_PORT, along with a few other environment flags.
$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true
For example, if you are running remoteexec_proxy locally with --port 5050, use:
$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050
Finally, start Goma client:
$ $HOME/goma/goma_ctl.py ensure_start
and in chromium tree.
$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o
Directories
¶
| Path | Synopsis |
|---|---|
|
Package auth provides authentication service.
|
Package auth provides authentication service. |
|
account
Package account manages service account.
|
Package account manages service account. |
|
acl
Package acl performs access control with ACL.
|
Package acl performs access control with ACL. |
|
authdb
Package authdb provides access to authentication database.
|
Package authdb provides access to authentication database. |
|
enduser
Package enduser manages end user information with context.
|
Package enduser manages end user information with context. |
|
Package backend is goma service backend of /cxx-compiler-service/*.
|
Package backend is goma service backend of /cxx-compiler-service/*. |
|
Package bytestreamio provides io interfaces on bytestream service.
|
Package bytestreamio provides io interfaces on bytestream service. |
|
Package cache provides cache service.
|
Package cache provides cache service. |
|
gcs
Package gcs provides cache service by google cloud storage.
|
Package gcs provides cache service by google cloud storage. |
|
redis
Package redis provides cache service by redis (cloud memorystore).
|
Package redis provides cache service by redis (cloud memorystore). |
|
cmd
|
|
|
auth_server
command
Binary auth_server provides auth service via gRPC.
|
Binary auth_server provides auth service via gRPC. |
|
cache_server
command
Binary cache_server provides cache service via gRPC.
|
Binary cache_server provides cache service via gRPC. |
|
exec_server
command
Binary exec_server provides goma exec service via gRPC.
|
Binary exec_server provides goma exec service via gRPC. |
|
execlog_server
command
Binary execlog_server provides goma execlog service via gRPC.
|
Binary execlog_server provides goma execlog service via gRPC. |
|
file_server
command
Binary file_server provides goma file service via gRPC.
|
Binary file_server provides goma file service via gRPC. |
|
frontend
command
Binary frontend is goma frontend.
|
Binary frontend is goma frontend. |
|
goma_grpc_client
command
Binary goma_grpc_client is a simple gRPC client of goma api.
|
Binary goma_grpc_client is a simple gRPC client of goma api. |
|
goma_replay
command
Binary goma_replay is a simple goma client for load testing etc.
|
Binary goma_replay is a simple goma client for load testing etc. |
|
remoteexec_proxy
command
Binary remoteexec-proxy is a proxy server between Goma client and Remote Execution API.
|
Binary remoteexec-proxy is a proxy server between Goma client and Remote Execution API. |
|
Package command manages commands/toolchains.
|
Package command manages commands/toolchains. |
|
descriptor
Package descriptor provides command descriptor utilities.
|
Package descriptor provides command descriptor utilities. |
|
descriptor/posixpath
Package posixpath handles posix-path (Unix style; slash separeted path).
|
Package posixpath handles posix-path (Unix style; slash separeted path). |
|
descriptor/winpath
Package winpath handles windows-path (backslash separated path).
|
Package winpath handles windows-path (backslash separated path). |
|
normalizer
Package normalizer provides functions to normalize target.
|
Package normalizer provides functions to normalize target. |
|
pathconv
Package pathconv provides path converter between client and server.
|
Package pathconv provides path converter between client and server. |
|
Package exec provides goma exec service implementation.
|
Package exec provides goma exec service implementation. |
|
Package execlog provides goma execlog service implementation.
|
Package execlog provides goma execlog service implementation. |
|
Package file provides goma file service implementation.
|
Package file provides goma file service implementation. |
|
Package frontend is goma service frontend (/cxx-compiler-service/*).
|
Package frontend is goma service frontend (/cxx-compiler-service/*). |
|
Package fswatch watches directory with fsnotify.
|
Package fswatch watches directory with fsnotify. |
|
Package hash provides a hash function used in goma.
|
Package hash provides a hash function used in goma. |
|
Package httprpc provides goma httprpc implementation.
|
Package httprpc provides goma httprpc implementation. |
|
authdb
Package authdb implements authdb service for goma httprpc.
|
Package authdb implements authdb service for goma httprpc. |
|
bytestream
Package bytestream implements bytestream for goma http.
|
Package bytestream implements bytestream for goma http. |
|
exec
Package exec implements exec service for goma httprpc.
|
Package exec implements exec service for goma httprpc. |
|
execlog
Package execlog implements log service for goma httprpc.
|
Package execlog implements log service for goma httprpc. |
|
file
Package file implements file service for goma httprpc.
|
Package file implements file service for goma httprpc. |
|
settings
Package settings implements settings service for goma httprpc.
|
Package settings implements settings service for goma httprpc. |
|
Package log provides logging mechanism for goma servers.
|
Package log provides logging mechanism for goma servers. |
|
errorreporter
Package errorreporter provides error reporting functionality.
|
Package errorreporter provides error reporting functionality. |
|
Package profiler provides convenient function to enable cloud profiler.
|
Package profiler provides convenient function to enable cloud profiler. |
|
Package proto is top directory for proto packages.
|
Package proto is top directory for proto packages. |
|
Package remoteexec provides proxy to remoteexec server.
|
Package remoteexec provides proxy to remoteexec server. |
|
cas
Package cas manages content addressable storage.
|
Package cas manages content addressable storage. |
|
datasource
Package datasource provides data source from local file, bytes etc.
|
Package datasource provides data source from local file, bytes etc. |
|
digest
Package digest handles content digest for remote executon API, https://github.com/bazelbuild/remote-apis/blob/c1c1ad2c97ed18943adb55f06657440daa60d833/build/bazel/remote/execution/v2/remote_execution.proto#L633
|
Package digest handles content digest for remote executon API, https://github.com/bazelbuild/remote-apis/blob/c1c1ad2c97ed18943adb55f06657440daa60d833/build/bazel/remote/execution/v2/remote_execution.proto#L633 |
|
merkletree
Package merkletree operates on a merkle tree for remote execution API, https://github.com/bazelbuild/remote-apis/blob/c1c1ad2c97ed18943adb55f06657440daa60d833/build/bazel/remote/execution/v2/remote_execution.proto#L838
|
Package merkletree operates on a merkle tree for remote execution API, https://github.com/bazelbuild/remote-apis/blob/c1c1ad2c97ed18943adb55f06657440daa60d833/build/bazel/remote/execution/v2/remote_execution.proto#L838 |
|
Package rpc provides goma specific rpc features on gRPC.
|
Package rpc provides goma specific rpc features on gRPC. |
|
grpctest
Package grpctest provides a test server for unit tests that use gRPC.
|
Package grpctest provides a test server for unit tests that use gRPC. |
|
Package server provides functions for goma servers.
|
Package server provides functions for goma servers. |
|
healthz
Package healthz provides /healthz for grpc server.
|
Package healthz provides /healthz for grpc server. |
Click to show internal directories.
Click to hide internal directories.