README

TaskCluster Worker

logo

Build Status GoDoc Coverage Status License

A worker for TaskCluster, written in go.

This is our next generation worker, that has a pluggable architecture for adding support for new engines (think Docker™ engine, Windows™ native engine, OS X™ native engine, KVM™/Xen™ engine) and adding engine-independent plugins (think livelogs, caches/volumes, auth proxies, interactive ssh/vnc).

Architecture

See https://docs.taskcluster.net/reference/workers/taskcluster-worker/docs/architecture

Installing From Binary

See https://github.com/taskcluster/taskcluster-worker/releases

Installing From Source

  1. Install go 1.10 or higher
  2. go get -u -t -d github.com/taskcluster/taskcluster-worker/...
  3. cd "${GOPATH}/src/github.com/taskcluster/taskcluster-worker"
  4. go get -u github.com/kardianos/govendor
  5. govendor sync
  6. make rebuild

Testing

make rebuild
Conditional Tests

Certain tests are disabled by default because they are:

  • only runs under certain conditions (e.g. requires root credentials),
  • requires special configuration (for example secret tokens),
  • have system-wide side-effects (like deleting the $HOME folder), or,
  • very slow, noisy or needs fixing.

Tests that fall into one or more of these categories are enabled with build tags. Below is a list of such build tags and description of what they test and why these tests are disabled by default.

  • localtunnel, tests a WebHookServer implementation based on localtunnel.me, tests are somewhat buggy and due to the nature of using a remote service this isn't reliable.
  • native, tests the native-engine, disabled because tests cleans up system folders such as HOME.
  • qemu, tests qemu-engine, disabled because it requires QEMU installed and needs to run as root (run tests with ./docker-tests.sh).
  • network, tests network configuration for qemu-engine, disabled because it can leave the system in a dirty state and requires root (run tests with ./docker-tests.sh).
  • monitor, tests sentry reporting, statsum submission and logging, requires credentials to run successfully.
  • docker, tests docker-engine, disabled because tests requires docker installed and must run as root (run tests with ./docker-tests.sh).

Releasing

Simply create a tag, and push to github.

git tag v1.0.3
git push --tags

Freezing Dependencies

You need govendor to manage vendor dependencies.

govendor sync

Adding Dependencies

go get <package>
govendor add +external
git add vendor/vendor.json
git commit -m 'My new package.'

Updating Dependencies

go get -u -t ./...   # update versions
govendor update

Contributing

We welcome Pull Requests and Issues!

Find us in #taskcluster-worker on irc.mozilla.org

Documentation

Overview

    Package main hosts the main function for taskcluter-worker.

    The code is structured in 4 kinds of extension registries. The commands, config, engines and plugins packages each define interfaces and registries where implementations of these interfaces can be registered.

    To add a new command to taskcluster-worker you must create new sub-package of commands/ which implements and registers commands.CommandProvider with the commands.Register(name, provider) method. The same pattern is followed for implementation of config transformers, engines, and plugins.

    All the sub-packages are then imported here, which ensure that they'll all be included in the respective extension registries. Exceptions to this pattern is the runtime and worker packages. The runtime package and its sub-packages implements generic abstractions and utilities to be used by all other packages. The worker package implements task execution flow to be used by commands.

    Source Files

    Directories

    Path Synopsis
    Package commands exposes a run method for main() to call
    Package commands exposes a run method for main() to call
    help
    Package help provides the help command.
    Package help provides the help command.
    qemu-guest-tools
    Package qemuguesttools implements the command that runs inside a QEMU VM.
    Package qemuguesttools implements the command that runs inside a QEMU VM.
    shell
    Package shell provides a CommandProvider that implements a CLI tool for opening to a interactive shell to an interactive taskcluster-worker task in your terminal.
    Package shell provides a CommandProvider that implements a CLI tool for opening to a interactive shell to an interactive taskcluster-worker task in your terminal.
    version
    Package version provides a CommandProvider that displays version number and git revision, these values are also exported through methods so that they can be read from other packages.
    Package version provides a CommandProvider that displays version number and git revision, these values are also exported through methods so that they can be read from other packages.
    Package config provides configuration loading logic.
    Package config provides configuration loading logic.
    abs
    Package configabs implements a TransformationProvider that replaces objects on the form: {$abs: "path"} with the value of current working folder + path.
    Package configabs implements a TransformationProvider that replaces objects on the form: {$abs: "path"} with the value of current working folder + path.
    configtest
    Package configtest provides structs and logic for declarative configuration tests.
    Package configtest provides structs and logic for declarative configuration tests.
    env
    Package configenv implements a TransformationProvider that replaces objects on the form: {$env: "VAR"} with the value of the environment variable VAR.
    Package configenv implements a TransformationProvider that replaces objects on the form: {$env: "VAR"} with the value of the environment variable VAR.
    hostcredentials
    Package hostcredentials implements a TransformationProvider that fetches credentials from the (oddly named) `host-secrets` service and replaces objects of the form: {$hostcredentials: [url, url]} with the credentials.
    Package hostcredentials implements a TransformationProvider that fetches credentials from the (oddly named) `host-secrets` service and replaces objects of the form: {$hostcredentials: [url, url]} with the credentials.
    packet
    Package configpacket implements a TransformationProvider that replaces objects on the form: {$packet: "VARIABLE"} with a value loaded from https://metadata.packet.net/metadata, following VARIABLE values are supported: - instance-id - hostname - facility - instance-type - public-ipv4 - public-ipv6 If configuration property 'packetMetaDataUrl' this will be used instead of 'https://metadata.packet.net/metadata'.
    Package configpacket implements a TransformationProvider that replaces objects on the form: {$packet: "VARIABLE"} with a value loaded from https://metadata.packet.net/metadata, following VARIABLE values are supported: - instance-id - hostname - facility - instance-type - public-ipv4 - public-ipv6 If configuration property 'packetMetaDataUrl' this will be used instead of 'https://metadata.packet.net/metadata'.
    secrets
    Package configsecrets implements a TransformationProvider that replaces objects on the form: {$secret: "NAME", key: "KEY"} with the value of the key "KEY" taken from the secret NAME loaded from taskcluster-secrets.
    Package configsecrets implements a TransformationProvider that replaces objects on the form: {$secret: "NAME", key: "KEY"} with the value of the key "KEY" taken from the secret NAME loaded from taskcluster-secrets.
    Package engines specifies the interfaces that each engine must implement.
    Package engines specifies the interfaces that each engine must implement.
    docker
    Package dockerengine implements a docker based engine for taskcluster-worker.
    Package dockerengine implements a docker based engine for taskcluster-worker.
    docker/imagecache
    Package imagecache handles loading and caching of docker images.
    Package imagecache handles loading and caching of docker images.
    docker/network
    Package network wraps docker network and ensures exposure of HTTP end-points to containers attached to said network.
    Package network wraps docker network and ensures exposure of HTTP end-points to containers attached to said network.
    enginetest
    Package enginetest provides utilities for testing generic engine implementations.
    Package enginetest provides utilities for testing generic engine implementations.
    mock
    Package mockengine implements a MockEngine that doesn't really do anything, but allows us to test plugins without having to run a real engine.
    Package mockengine implements a MockEngine that doesn't really do anything, but allows us to test plugins without having to run a real engine.
    mock/mocknet
    Package mocknet implements a net.Listener interface that can reached with mocknet.Dial() and establishes connections using net.Pipe() This is useful for testing things that needs net.Listener and net.Conn instances without creating a TCP listener on localhost.
    Package mocknet implements a net.Listener interface that can reached with mocknet.Dial() and establishes connections using net.Pipe() This is useful for testing things that needs net.Listener and net.Conn instances without creating a TCP listener on localhost.
    native
    Package nativeengine provides an engine with minimal sandboxing relying on per-task user accounts, temporary folders and process isolation.
    Package nativeengine provides an engine with minimal sandboxing relying on per-task user accounts, temporary folders and process isolation.
    native/system
    Package system implements cross-platform abstractions for user-management access-control and sub-process execution geared at executing sub-process with best-effort sandboxing.
    Package system implements cross-platform abstractions for user-management access-control and sub-process execution geared at executing sub-process with best-effort sandboxing.
    native/unpack
    Package unpack contains utilities for unpacking files.
    Package unpack contains utilities for unpacking files.
    qemu
    Package qemuengine implements a QEMU based engine for taskcluster-worker.
    Package qemuengine implements a QEMU based engine for taskcluster-worker.
    qemu/image
    Package image exposes methods and abstractions for extracting and managing virtual machine images.
    Package image exposes methods and abstractions for extracting and managing virtual machine images.
    qemu/metaservice
    Package metaservice implements the meta-data service that the guests use to talk to the host.
    Package metaservice implements the meta-data service that the guests use to talk to the host.
    qemu/network
    Package network contains scripts and abstractions for setting up TAP-device based networks for a set of QEMU virtual machines.
    Package network contains scripts and abstractions for setting up TAP-device based networks for a set of QEMU virtual machines.
    qemu/network/openvpn
    Package openvpn provides a wrapper around the openvpn client.
    Package openvpn provides a wrapper around the openvpn client.
    qemu/vm
    Package vm provides virtual machine abstractions using QEMU.
    Package vm provides virtual machine abstractions using QEMU.
    script
    Package scriptengine provides an engine that can be configured with a script and a JSON schema, such that the worker executes declarative tasks.
    Package scriptengine provides an engine that can be configured with a script and a JSON schema, such that the worker executes declarative tasks.
    Package plugins defines interfaces to be implemented by feature plugins.
    Package plugins defines interfaces to be implemented by feature plugins.
    artifacts
    Package artifacts provides a taskcluster-worker plugin that uploads artifacts when sandbox execution has stopped.
    Package artifacts provides a taskcluster-worker plugin that uploads artifacts when sandbox execution has stopped.
    cache
    Package cache provides a cache plugin for taskcluster-worker
    Package cache provides a cache plugin for taskcluster-worker
    env
    Package env provides a taskcluster-worker plugin that injects environment variables into the task environments.
    Package env provides a taskcluster-worker plugin that injects environment variables into the task environments.
    interactive
    Package interactive implements the plugin that serves the interactive display and shell sessions over websockets.
    Package interactive implements the plugin that serves the interactive display and shell sessions over websockets.
    interactive/displayclient
    Package displayclient provides a golang implementation of websockify, transforming a websocket connection to an ioext.ReadWriteCloser object.
    Package displayclient provides a golang implementation of websockify, transforming a websocket connection to an ioext.ReadWriteCloser object.
    interactive/shellclient
    Package shellclient provides a wrapper for demuxing a shell websocket and exposing the stdout/stderr streams as well as offering a way to provide the stdin stream.
    Package shellclient provides a wrapper for demuxing a shell websocket and exposing the stdout/stderr streams as well as offering a way to provide the stdin stream.
    interactive/shellconsts
    Package shellconsts contains constants shared between shell server and client which is split into different packages to reduce the binary size of potential commandline clients.
    Package shellconsts contains constants shared between shell server and client which is split into different packages to reduce the binary size of potential commandline clients.
    livelog
    Package livelog provides a taskcluster-worker plugin that makes the task log available as a live log during task execution and finally uploads it as a static log.
    Package livelog provides a taskcluster-worker plugin that makes the task log available as a live log during task execution and finally uploads it as a static log.
    logprefix
    Package logprefix provides a taskcluster-worker plugin that prefixes all task logs with useful debug information such as taskId, workerType, as well as configurable constants.
    Package logprefix provides a taskcluster-worker plugin that prefixes all task logs with useful debug information such as taskId, workerType, as well as configurable constants.
    maxruntime
    Package maxruntime provides a plugin for taskcluster-worker which can enforce a maximum runtime upon tasks.
    Package maxruntime provides a plugin for taskcluster-worker which can enforce a maximum runtime upon tasks.
    reboot
    Package reboot provides a taskcluster-worker plugin that stops the worker after certain number of tasks or given amount of time.
    Package reboot provides a taskcluster-worker plugin that stops the worker after certain number of tasks or given amount of time.
    relengapi
    Package relengapi provides a taskcluster-worker plugin that exposes a proxy that forward requests to relengapi.
    Package relengapi provides a taskcluster-worker plugin that exposes a proxy that forward requests to relengapi.
    stoponerror
    Package stoponerror implements a very simple plugin that stops the worker gracefully if an non-fatal error is encountered.
    Package stoponerror implements a very simple plugin that stops the worker gracefully if an non-fatal error is encountered.
    success
    Package success implements a very simple plugin that looks that the ResultSet.Success() value to determine if the process from the sandbox exited successfully.
    Package success implements a very simple plugin that looks that the ResultSet.Success() value to determine if the process from the sandbox exited successfully.
    tasklog
    Package tasklog provides a taskcluster-worker plugin that uploads a static task.log when the task is finished.
    Package tasklog provides a taskcluster-worker plugin that uploads a static task.log when the task is finished.
    tcproxy
    Package tcproxy provides a taskcluster-worker plugin that exposes a proxy that signs requests with taskcluster credentials matching task.scopes.
    Package tcproxy provides a taskcluster-worker plugin that exposes a proxy that signs requests with taskcluster credentials matching task.scopes.
    watchdog
    Package watchdog provides a taskcluster-worker plugin that pokes a watchdog whenever a task makes progress or the worker reports that it's idle.
    Package watchdog provides a taskcluster-worker plugin that pokes a watchdog whenever a task makes progress or the worker reports that it's idle.
    Package runtime contains the generic functionality that an engine and plugins use.
    Package runtime contains the generic functionality that an engine and plugins use.
    atomics
    Package atomics provides types that can be concurrently accessed and modified, without caller code needing to implement locking.
    Package atomics provides types that can be concurrently accessed and modified, without caller code needing to implement locking.
    caching
    Package caching provides an easy to make a cache on top of the gc package used to track idle resources in taskcluster-worker.
    Package caching provides an easy to make a cache on top of the gc package used to track idle resources in taskcluster-worker.
    fetcher
    Package fetcher provides means for plugins and engines to fetch resources with generic references.
    Package fetcher provides means for plugins and engines to fetch resources with generic references.
    gc
    Package gc contains the GarbageCollector which allows cacheable resources to register themselves for disposal when we run low on resources.
    Package gc contains the GarbageCollector which allows cacheable resources to register themselves for disposal when we run low on resources.
    ioext
    Package ioext contains interfaces and implementations for when the default io types are not sufficient.
    Package ioext contains interfaces and implementations for when the default io types are not sufficient.
    mocks
    Package mocks contains mock implementations of various interfaces useful for writing unit-tests.
    Package mocks contains mock implementations of various interfaces useful for writing unit-tests.
    monitoring
    Package monitoring provides multiple implementations of runtime.Monitor.
    Package monitoring provides multiple implementations of runtime.Monitor.
    util
    Package util contains a few simple utilites that has no internal dependencies.
    Package util contains a few simple utilites that has no internal dependencies.
    webhookserver
    Package webhookserver provides implementations of the WebHookServer interface.
    Package webhookserver provides implementations of the WebHookServer interface.
    dockertest
    Package dockertest provides integration tests for a few common configuration of docker engine and common plugins.
    Package dockertest provides integration tests for a few common configuration of docker engine and common plugins.
    nativetest
    Package nativetest provides integration tests for a few common configuration of native and common plugins.
    Package nativetest provides integration tests for a few common configuration of native and common plugins.
    scripttest
    Package scripttest provides integration tests for a few common configuration of script and common plugins.
    Package scripttest provides integration tests for a few common configuration of script and common plugins.
    Package worker is responsible for managing the claiming, executing, and resolution of tasks.
    Package worker is responsible for managing the claiming, executing, and resolution of tasks.
    taskrun
    Package taskrun provides abstractions to execute a run of a task given a task, engine, plugin, and other runtime objects required by plugin and engine.
    Package taskrun provides abstractions to execute a run of a task given a task, engine, plugin, and other runtime objects required by plugin and engine.
    workertest
    Package workertest provides a framework for declarative definition of worker integration tests.
    Package workertest provides a framework for declarative definition of worker integration tests.
    workertest/fakequeue
    Package fakequeue provides a fake implementation of taskcluster-queue in golang, The FakeQueue server stores tasks in-memory, it doesn't validate authentication, but implements most end-points correctly.
    Package fakequeue provides a fake implementation of taskcluster-queue in golang, The FakeQueue server stores tasks in-memory, it doesn't validate authentication, but implements most end-points correctly.