greenhouse

command
Version: v0.0.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 7, 2018 License: Apache-2.0 Imports: 15 Imported by: 0

README

Greenhouse

Greenhouse is our bazel remote caching setup. We use this to provide faster build & test presubmits with a Globally shared cache (per repo).

We have a dashboard with metrics at: velodrome.k8s.io/dashboard/db/cache-monitoring

Most Bazel users should probably visit the official docs and select one of the options outlined there, with Prow/Kubernetes we are using a custom setup to explore:

  • better support for multiple repos / cache invalidation by changing the cache URL suffix (see also: images/bootstrap/create_bazel_cache_rcs.sh)
  • customized cache eviction / management
  • integration with our logging and metrics stacks

Setup (on a Kubernetes Cluster)

We use this with Prow, to set it up we do the following:

  • Install kubectl and bazel and Point KUBECONFIG at your cluster.
    • for k8s.io use make -C prow get-build-cluster-credentials
  • Create a dedicated node. We use a GKE node-pool with a single node. Tag this node with label dedicated=greenhouse and taint dedicated=greenhouse:NoSchedule so your other tasks don't schedule on it.
    • for k8s.io (running on GKE) this is:
    gcloud beta container node-pools create greenhouse --cluster=prow --project=k8s-prow-builds --zone=us-central1-f --node-taints=dedicated=greenhouse:NoSchedule --node-labels=dedicated=greenhouse --machine-type=n1-standard-32 --num-nodes=1
    
    • if you're not on GKE you'll probably want to pick a node to dedicate and do something like:
    kubectl label nodes $GREENHOUSE_NODE_NAME dedicated=greenhouse
    kubectl taint nodes $GREENHOUSE_NODE_NAME dedicated=greenhouse:NoSchedule
    
  • Create the Kubernetes service so jobs can talk to it conveniently: kubectl apply -f greenhouse/service.yaml
  • Create a StorageClass / PersistentVolumeClaim for fast cache storage, we use kubectl apply -f greenhouse/gce-fast-storage.yaml for 3TB of pd-ssd storage
  • Finally build, push, and deploy with bazel run //greenhouse:production.apply --platforms=@io_bazel_rules_go//go/toolchain:linux_amd64
    • NOTE: other uses will likely need to tweak this step to their needs, particular the service and storage definitions

Optional Setup:

  • tweak metrics-service.yaml and point prometheus at this service to collect metrics

Documentation

Overview

greenhouse implements a bazel remote cache service [1] supporting arbitrarily many workspaces stored within the same top level directory.

the first path segment in each {PUT,GET} request is mapped to an individual workspace cache, the remaining segments should follow [2].

nursery assumes you are using SHA256

[1] https://docs.bazel.build/versions/master/remote-caching.html [2] https://docs.bazel.build/versions/master/remote-caching.html#http-caching-protocol

Directories

Path Synopsis
Package diskcache implements disk backed cache storage for use in greenhouse
Package diskcache implements disk backed cache storage for use in greenhouse
Package diskutil implements disk related utilities for greenhouse
Package diskutil implements disk related utilities for greenhouse

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
t or T : Toggle theme light dark auto
y or Y : Canonical URL