envbuilder

README

envbuilder

Build development environments from a Dockerfile on Docker, Kubernetes, and OpenShift. Allow developers to modify their environment in a tight feedback loop.

• Supports devcontainer.json and Dockerfile
• Cache image layers with registries for speedy builds
• Runs on Kubernetes, Docker, and OpenShift

Quickstart

The easiest way to get started is to run the envbuilder Docker container that clones a repository, builds the image from a Dockerfile, and runs the $INIT_SCRIPT in the freshly built container.

/tmp/envbuilder is used to persist data between commands for the purpose of this demo. You can change it to any directory you want.

docker run -it --rm \
    -v /tmp/envbuilder:/workspaces \
    -e GIT_URL=https://github.com/coder/envbuilder-starter-devcontainer \
    -e INIT_SCRIPT=bash \
    ghcr.io/coder/envbuilder

Edit .devcontainer/Dockerfile to add htop:

$ vim .devcontainer/Dockerfile

- RUN apt-get install vim sudo -y
+ RUN apt-get install vim sudo htop -y

Exit the container, and re-run the docker run command... after the build completes, htop should exist in the container! 🥳

Container Registry Authentication

envbuilder uses Kaniko to build containers. You should follow their instructions to create an authentication configuration.

After you have a configuration that resembles the following:

{
  "auths": {
    "https://index.docker.io/v1/": {
      "auth": "base64-encoded-username-and-password"
    }
  }
}

base64 encode the JSON and provide it to envbuilder as the DOCKER_CONFIG_BASE64 environment variable.

Alternatively, if running envbuilder in Kubernetes, you can create an ImagePullSecret and pass it into the pod as a volume mount. This example will work for all registries.

# Artifactory example
kubectl create secret docker-registry regcred \
  --docker-server=my-artifactory.jfrog.io \
  --docker-username=read-only \
  --docker-password=secret-pass \
  --docker-email=me@example.com \
  -n coder

resource "kubernetes_deployment" "example" {
  metadata {
    namespace = coder
  }
  spec {
    spec {
      container {  
        # Define the volumeMount with the pull credentials
        volume_mount {
          name       = "docker-config-volume"
          mount_path = "/envbuilder/config.json"
          sub_path   = ".dockerconfigjson"
        }
      }
      # Define the volume which maps to the pull credentials
      volume {
        name = "docker-config-volume"
        secret {
          secret_name = "regcred"
        }
      }
    }
  }
}

Docker Hub

Authenticate with docker login to generate ~/.docker/config.json. Encode this file using the base64 command:

$ base64 -w0 ~/.docker/config.json
ewoJImF1dGhzIjogewoJCSJodHRwczovL2luZGV4LmRvY2tlci5pby92MS8iOiB7CgkJCSJhdXRoIjogImJhc2U2NCBlbmNvZGVkIHRva2VuIgoJCX0KCX0KfQo=

Provide the encoded JSON config to envbuilder:

DOCKER_CONFIG_BASE64=ewoJImF1dGhzIjogewoJCSJodHRwczovL2luZGV4LmRvY2tlci5pby92MS8iOiB7CgkJCSJhdXRoIjogImJhc2U2NCBlbmNvZGVkIHRva2VuIgoJCX0KCX0KfQo=

Git Authentication

GIT_USERNAME and GIT_PASSWORD are environment variables to provide Git authentication for private repositories.

For access token-based authentication, follow the following schema (if empty, there's no need to provide the field):

Provider	GIT_USERNAME	GIT_PASSWORD
GitHub	[access-token]
GitLab	oauth2	[access-token]
BitBucket	x-token-auth	[access-token]
Azure DevOps	[access-token]

If using envbuilder inside of Coder, you can use the coder_external_auth Terraform resource to automatically provide this token on workspace creation:

data "coder_external_auth" "github" {
    id = "github"
}

resource "docker_container" "dev" {
    env = [
        GIT_USERNAME = data.coder_external_auth.github.access_token,
    ]
}

Layer Caching

Cache layers in a container registry to speed up builds. To enable caching, authenticate with your registry and set the CACHE_REPO environment variable.

CACHE_REPO=ghcr.io/coder/repo-cache

To experiment without setting up a registry, use LAYER_CACHE_DIR:

docker run -it --rm \
  -v /tmp/envbuilder-cache:/cache \
  -e LAYER_CACHE_DIR=/cache
  ...

Each layer is stored in the registry as a separate image. The image tag is the hash of the layer's contents. The image digest is the hash of the image tag. The image digest is used to pull the layer from the registry.

The performance improvement of builds depends on the complexity of your Dockerfile. For coder/coder, uncached builds take 36m while cached builds take 40s (~98% improvement).

Image Caching

When the base container is large, it can take a long time to pull the image from the registry. You can pre-pull the image into a read-only volume and mount it into the container to speed up builds.

# Pull your base image from the registry to a local directory.
docker run --rm \
  -v /tmp/kaniko-cache:/cache \
  gcr.io/kaniko-project/warmer:latest \
    --cache-dir=/cache \
    --image=<your-image>

# Run envbuilder with the local image cache.
docker run -it --rm \
  -v /tmp/kaniko-cache:/image-cache:ro \
  -e BASE_IMAGE_CACHE_DIR=/image-cache

In Kubernetes, you can pre-populate a persistent volume with the same warmer image, then mount it into many workspaces with the ReadOnlyMany access mode.

Setup Script

The SETUP_SCRIPT environment variable dynamically configures the user and init command (PID 1) after the container build process.

Note TARGET_USER is passed to the setup script to specify who will execute INIT_COMMAND (e.g., code).

Write the following to $ENVBUILDER_ENV to shape the container's init process:

• TARGET_USER: Identifies the INIT_COMMAND executor (e.g.root).
• INIT_COMMAND: Defines the command executed by TARGET_USER (e.g. /bin/bash).
• INIT_ARGS: Arguments provided to INIT_COMMAND (e.g. -c 'sleep infinity').

# init.sh - change the init if systemd exists
if command -v systemd >/dev/null; then
  echo "Hey 👋 $TARGET_USER"
  echo INIT_COMMAND=systemd >> $ENVBUILDER_ENV
else
  echo INIT_COMMAND=bash >> $ENVBUILDER_ENV
fi

# run envbuilder with the setup script
docker run -it --rm \
  -v ./:/some-dir \
  -e SETUP_SCRIPT=/some-dir/init.sh \
  ...

Custom Certificates

• SSL_CERT_FILE: Specifies the path to an SSL certificate.
• SSL_CERT_DIR: Identifies which directory to check for SSL certificate files.
• SSL_CERT_BASE64: Specifies a base64-encoded SSL certificate that will be added to the global certificate pool on start.

Documentation

Index

• Constants
• Variables
• func CloneRepo(ctx context.Context, opts CloneRepoOptions) (bool, error)
• func DefaultWorkspaceFolder(repoURL string) (string, error)
• func HijackLogrus(callback func(entry *logrus.Entry))
• func Run(ctx context.Context, options Options) error
• type CloneRepoOptions
• type DockerConfig
• type Options
  • func OptionsFromEnv(getEnv func(string) (string, bool)) Options

Constants

const (
	// WorkspacesDir is the path to the directory where
	// all workspaces are stored by default.
	WorkspacesDir = "/workspaces"

	// EmptyWorkspaceDir is the path to a workspace that has
	// nothing going on... it's empty!
	EmptyWorkspaceDir = WorkspacesDir + "/empty"

	// MagicDir is where all envbuilder related files are stored.
	// This is a special directory that must not be modified
	// by the user or images.
	MagicDir = "/.envbuilder"
)

Variables

var (
	ErrNoFallbackImage = errors.New("no fallback image has been specified")

	// MagicFile is a file that is created in the workspace
	// when envbuilder has already been run. This is used
	// to skip building when a container is restarting.
	// e.g. docker stop -> docker start
	MagicFile = filepath.Join(MagicDir, "built")
)

Functions

func CloneRepo

func CloneRepo(ctx context.Context, opts CloneRepoOptions) (bool, error)

CloneRepo will clone the repository at the given URL into the given path. If a repository is already initialized at the given path, it will not be cloned again.

The bool returned states whether the repository was cloned or not.

func DefaultWorkspaceFolder

func DefaultWorkspaceFolder(repoURL string) (string, error)

DefaultWorkspaceFolder returns the default workspace folder for a given repository URL.

func HijackLogrus

func HijackLogrus(callback func(entry *logrus.Entry))

HijackLogrus hijacks the logrus logger and calls the callback for each log entry. This is an abuse of logrus, the package that Kaniko uses, but it exposes no other way to obtain the log entries.

func Run

func Run(ctx context.Context, options Options) error

Run runs the envbuilder.

Types

type CloneRepoOptions

type CloneRepoOptions struct {
	Path    string
	Storage billy.Filesystem

	RepoURL      string
	RepoAuth     transport.AuthMethod
	Progress     sideband.Progress
	Insecure     bool
	SingleBranch bool
	Depth        int
	CABundle     []byte
	ProxyOptions transport.ProxyOptions
}

type DockerConfig

type DockerConfig configfile.ConfigFile

DockerConfig represents the Docker configuration file.

type Options

type Options struct {

	// SetupScript is ran as the root user prior to the init script.
	// It is used to configure envbuilder dynamically during the runtime.
	// e.g. specifying whether to start `systemd` or `tiny init` for PID 1.
	SetupScript string `env:"SETUP_SCRIPT"`

	// InitScript is the script to run to initialize the workspace.
	InitScript string `env:"INIT_SCRIPT"`

	// InitCommand is the command to run to initialize the workspace.
	InitCommand string `env:"INIT_COMMAND"`

	// InitArgs are the arguments to pass to the init command.
	// They are split according to `/bin/sh` rules with
	// https://github.com/kballard/go-shellquote
	InitArgs string `env:"INIT_ARGS"`

	// CacheRepo is the name of the container registry
	// to push the cache image to. If this is empty, the cache
	// will not be pushed.
	CacheRepo string `env:"CACHE_REPO"`

	// BaseImageCacheDir is the path to a directory where the base
	// image can be found. This should be a read-only directory
	// solely mounted for the purpose of caching the base image.
	BaseImageCacheDir string `env:"BASE_IMAGE_CACHE_DIR"`

	// LayerCacheDir is the path to a directory where built layers
	// will be stored. This spawns an in-memory registry to serve
	// the layers from.
	//
	// It will override CacheRepo if both are specified.
	LayerCacheDir string `env:"LAYER_CACHE_DIR"`

	// DevcontainerDir is a path to the folder containing
	// the devcontainer.json file that will be used to build the
	// workspace and can either be an absolute path or a path
	// relative to the workspace folder. If not provided, defaults to
	// `.devcontainer`.
	DevcontainerDir string `env:"DEVCONTAINER_DIR"`

	// DevcontainerJSONPath is a path to a devcontainer.json file
	// that is either an absolute path or a path relative to
	// DevcontainerDir. This can be used in cases where one wants
	// to substitute an edited devcontainer.json file for the one
	// that exists in the repo.
	DevcontainerJSONPath string `env:"DEVCONTAINER_JSON_PATH"`

	// DockerfilePath is a relative path to the Dockerfile that
	// will be used to build the workspace. This is an alternative
	// to using a devcontainer that some might find simpler.
	DockerfilePath string `env:"DOCKERFILE_PATH"`

	// CacheTTLDays is the number of days to use cached layers before
	// expiring them. Defaults to 7 days.
	CacheTTLDays int `env:"CACHE_TTL_DAYS"`

	// DockerConfigBase64 is a base64 encoded Docker config
	// file that will be used to pull images from private
	// container registries.
	DockerConfigBase64 string `env:"DOCKER_CONFIG_BASE64"`

	// FallbackImage specifies an alternative image to use when neither
	// an image is declared in the devcontainer.json file nor a Dockerfile is present.
	// If there's a build failure (from a faulty Dockerfile) or a misconfiguration,
	// this image will be the substitute.
	// Set `ExitOnBuildFailure` to true to halt the container if the build faces an issue.
	FallbackImage string `env:"FALLBACK_IMAGE"`

	// ExitOnBuildFailure terminates the container upon a build failure.
	// This is handy when preferring the `FALLBACK_IMAGE` in cases where
	// no devcontainer.json or image is provided. However, it ensures
	// that the container stops if the build process encounters an error.
	ExitOnBuildFailure bool `env:"EXIT_ON_BUILD_FAILURE"`

	// ForceSafe ignores any filesystem safety checks.
	// This could cause serious harm to your system!
	// This is used in cases where bypass is needed
	// to unblock customers!
	ForceSafe bool `env:"FORCE_SAFE"`

	// Insecure bypasses TLS verification when cloning
	// and pulling from container registries.
	Insecure bool `env:"INSECURE"`

	// IgnorePaths is a comma separated list of paths
	// to ignore when building the workspace.
	IgnorePaths []string `env:"IGNORE_PATHS"`

	// SkipRebuild skips building if the MagicFile exists.
	// This is used to skip building when a container is
	// restarting. e.g. docker stop -> docker start
	// This value can always be set to true - even if the
	// container is being started for the first time.
	SkipRebuild bool `env:"SKIP_REBUILD"`

	// GitURL is the URL of the Git repository to clone.
	// This is optional!
	GitURL string `env:"GIT_URL"`

	// GitCloneDepth is the depth to use when cloning
	// the Git repository.
	GitCloneDepth int `env:"GIT_CLONE_DEPTH"`

	// GitCloneSingleBranch clones only a single branch
	// of the Git repository.
	GitCloneSingleBranch bool `env:"GIT_CLONE_SINGLE_BRANCH"`

	// GitUsername is the username to use for Git authentication.
	// This is optional!
	GitUsername string `env:"GIT_USERNAME"`

	// GitPassword is the password to use for Git authentication.
	// This is optional!
	GitPassword string `env:"GIT_PASSWORD"`

	// GitHTTPProxyURL is the url for the http proxy.
	// This is optional!
	GitHTTPProxyURL string `env:"GIT_HTTP_PROXY_URL"`

	// WorkspaceFolder is the path to the workspace folder
	// that will be built. This is optional!
	WorkspaceFolder string `env:"WORKSPACE_FOLDER"`

	// SSLCertBase64 is the content of an SSL cert file.
	// This is useful for self-signed certificates.
	SSLCertBase64 string `env:"SSL_CERT_BASE64"`

	// ExportEnvFile is an optional file path to a .env file where
	// envbuilder will dump environment variables from devcontainer.json and
	// the built container image.
	ExportEnvFile string `env:"EXPORT_ENV_FILE"`

	// PostStartScriptPath is the path to a script that will be created by
	// envbuilder based on the `postStartCommand` in devcontainer.json, if any
	// is specified (otherwise the script is not created). If this is set, the
	// specified InitCommand should check for the presence of this script and
	// execute it after successful startup.
	PostStartScriptPath string `env:"POST_START_SCRIPT_PATH"`

	// Logger is the logger to use for all operations.
	Logger func(level codersdk.LogLevel, format string, args ...interface{})

	// Filesystem is the filesystem to use for all operations.
	// Defaults to the host filesystem.
	Filesystem billy.Filesystem
}

func OptionsFromEnv

func OptionsFromEnv(getEnv func(string) (string, bool)) Options

OptionsFromEnv returns a set of options from environment variables.