README
¶
Scoot
Scoot is a distributed task runner. The core concept of Scoot is the Snapshot, which represents an immutable filesystem state. Scoot allows for execution of commands against input Snapshots to create new output Snapshots.
Scoot Components
Scheduler
The Scheduler receives and distributes jobs to workers, and maintains state. It is also responsible for serving the Cloud Scoot API for clients.
Worker
The Worker (workerserver) receives information about jobs and runs them, and is responsible for all Snapshot-related functionality.
Client APIs
- Cloud Scoot API
- Worker API
Jobs and Tasks
Scoot work is broken down into independent jobs, which can consist of one or more tasks, which are executed sequentially.
Go Version
Scoot is intended to work with go modules. If this repository is on your GOPATH, please set GO111MODULE=on when working with go tools in this repository, and preferably use the Go distribution pinned in the .github/workflows/ci.yml file.
Example
Setup a scheduler and worker nodes locally:
go run ./setup-cloud-scoot/main.go --strategy local.local
Run a series of randomly generated tests against the local scheduler and workers:
go run ./scheduler/client/scootcl/main.go smoke_test
Scoot Integration Tests
Scoot has a few tests that exercise varying levels of common usages and workflows.
Smoketest
Invokes a scoot client directly to run jobs against a local cluster and waits for the scheduled jobs to complete.
(./scheduler/client/cli/smoketest.go)
Recoverytest
Invokes a scoot client directly to run jobs against a local cluster, kills the cluster, attempts to spin up a new one, and waits for the originally scheduled jobs to complete.
(./binaries/recoverytest/main.go)
Integration
Invokes a scoot and scoot-snapshot-db client via CLI to run a job against a local cluster and waits for the job to complete
(./binaries/scoot-integration/main.go)
Invokes a scoot bazel client (bzutil & fs_util) via CLI to run a job against a local cluster using the bazel API grpc endpoints
(./binaries/bazel-integration/main.go)
Scoot Thrift Code
Thrift Prerequisites Install the Thrift tool and golang thrift repository locally using the following section.
Generating thrift files
See documentation in thrift definition files for specific generation instructions, or make thrift.
Development Installation Instructions
Install 3rd party tools:
Thrift (version == 0.9.3)
macOS:
brew install thrift@0.9
Thrift for go:
go modules will take care of this
Install/Access Scoot Executables and libraries
Install binaries to $GOPATH/bin:
make install
Directories
¶
| Path | Synopsis |
|---|---|
|
Async provides tools for asynchronous callback processing using Goroutines
|
Async provides tools for asynchronous callback processing using Goroutines |
|
Bazel Remote Execution API gRPC
|
Bazel Remote Execution API gRPC |
|
cas
Bazel Remote Execution API gRPC server Contains limited implementation of the ContentAddressableStore API interface
|
Bazel Remote Execution API gRPC server Contains limited implementation of the ContentAddressableStore API interface |
|
cas/mock_bytestream
Package mock_bytestream is a generated GoMock package.
|
Package mock_bytestream is a generated GoMock package. |
|
cas/mock_connection
Package mock_connection is a generated GoMock package.
|
Package mock_connection is a generated GoMock package. |
|
execution
Bazel Remote Execution API gRPC server Contains limited implementation of the Execution API interface
|
Bazel Remote Execution API gRPC server Contains limited implementation of the Execution API interface |
|
execution/bazelapi
Execution Request & Action Result Definitions Domain structures for tracking ExecuteRequest info (use cases include Scheduler JobDefs and Worker RunCommands), ActionResult info (used in Worker RunStatuses), and conversions for internal thrift APIs.
|
Execution Request & Action Result Definitions Domain structures for tracking ExecuteRequest info (use cases include Scheduler JobDefs and Worker RunCommands), ActionResult info (used in Worker RunStatuses), and conversions for internal thrift APIs. |
|
execution/mock_longrunning
Package mock_longrunning is a generated GoMock package.
|
Package mock_longrunning is a generated GoMock package. |
|
execution/mock_remoteexecution
Package mock_remoteexecution is a generated GoMock package.
|
Package mock_remoteexecution is a generated GoMock package. |
|
binaries
|
|
|
Package cleaner provides internal cleanup-related utilities, primarily to limit disk consumption in long-lived workers.
|
Package cleaner provides internal cleanup-related utilities, primarily to limit disk consumption in long-lived workers. |
|
cloud
|
|
|
cluster
Cluster provides the means for coordinating the schedulers and workers that make up a Scoot system.
|
Cluster provides the means for coordinating the schedulers and workers that make up a Scoot system. |
|
cluster/local
Package local provides a cluster Fetcher implementation for obtaining state of nodes running on the local machine.
|
Package local provides a cluster Fetcher implementation for obtaining state of nodes running on the local machine. |
|
allocator
Package allocator provides tools for centralized management of a common resource by multiple clients.
|
Package allocator provides tools for centralized management of a common resource by multiple clients. |
|
dialer
Library for establishing Thrift network connections for clients.
|
Library for establishing Thrift network connections for clients. |
|
endpoints
Wrappers for receivers from the common/stats package and setting up an HTTP server with endpoints to make the stats data accessible.
|
Wrappers for receivers from the common/stats package and setting up an HTTP server with endpoints to make the stats data accessible. |
|
proto
Library for common gRPC protobuf-related tools.
|
Library for common gRPC protobuf-related tools. |
|
stats
This package provides a set of minimal interfaces which both build on and are by default backed by go-metrics.
|
This package provides a set of minimal interfaces which both build on and are by default backed by go-metrics. |
|
thrifthelpers
Utils for JSON and binary serialize/deserialize functions for Thrift structures
|
Utils for JSON and binary serialize/deserialize functions for Thrift structures |
|
config
|
|
|
jsonconfig
Jsonconfig implements configuration, reading json into an ice Module.
|
Jsonconfig implements configuration, reading json into an ice Module. |
|
ice is a lightweight Dependency Injection Framework ice's cenral metaphor is a "Magic Bag".
|
ice is a lightweight Dependency Injection Framework ice's cenral metaphor is a "Magic Bag". |
|
integration-tests
|
|
|
perftests contains tooling for running platform performance tests.
|
perftests contains tooling for running platform performance tests. |
|
apiserver
perftests/apiserver contains apiserver performance test tooling
|
perftests/apiserver contains apiserver performance test tooling |
|
apiserver/loadtest
The loadtest package implements running performance benchmarking against the apiserver.
|
The loadtest package implements running performance benchmarking against the apiserver. |
|
apiserver/loadtest/apiserver_benchmarker
apiserver_benchmarker is the cli for generating load on the apiserver.
|
apiserver_benchmarker is the cli for generating load on the apiserver. |
|
scheduler_simulator
Scheduling algorithm evaluation framework.
|
Scheduling algorithm evaluation framework. |
|
Package runner provides for execution of Scoot work and retrieval of the results of that work.
|
Package runner provides for execution of Scoot work and retrieval of the results of that work. |
|
mocks
Package mocks is a generated GoMock package.
|
Package mocks is a generated GoMock package. |
|
Package saga provides a generic implementation of the Saga pattern.
|
Package saga provides a generic implementation of the Saga pattern. |
|
sagalogs
Package sagalogs provides implementations of SagaLog.
|
Package sagalogs provides implementations of SagaLog. |
|
api
Package api implements the Scoot thrift API, Bazel grpc remote execution and longrunning operation APIs, and the TwitterServer http API.
|
Package api implements the Scoot thrift API, Bazel grpc remote execution and longrunning operation APIs, and the TwitterServer http API. |
|
api/thrift
Package thrift implements the Scoot API, see scoot.thrift for details
|
Package thrift implements the Scoot API, see scoot.thrift for details |
|
client/cli
Package client provides a CLI implementation of a client for the Cloud Scoot API.
|
Package client provides a CLI implementation of a client for the Cloud Scoot API. |
|
domain
Package domain provides definitions for Scoot Jobs and Tasks
|
Package domain provides definitions for Scoot Jobs and Tasks |
|
server
Package server provides the main job scheduling interface for Scoot Package server is a generated GoMock package.
|
Package server provides the main job scheduling interface for Scoot Package server is a generated GoMock package. |
|
setup
Package setup provides Cloud Scoot Setup, which is a mechanism for running Scoot with local components, or initializing a connection to remote ones, depending on configuration.
|
Package setup provides Cloud Scoot Setup, which is a mechanism for running Scoot with local components, or initializing a connection to remote ones, depending on configuration. |
|
package snapshot offers access to Snapshot.
|
package snapshot offers access to Snapshot. |
|
git/repo
Package repo provides utilities for operating on a git repo.
|
Package repo provides utilities for operating on a git repo. |
|
store
This package defines the Store interfaces for reading and writing bundles (or any artifact data) to some underlying system, and Store implementations.
|
This package defines the Store interfaces for reading and writing bundles (or any artifact data) to some underlying system, and Store implementations. |
|
tests
|
|
|
worker
|
|
|
client
The client package provides an interface and implementation for the Scoot Worker API, as well as a CLI client that wraps it.
|
The client package provides an interface and implementation for the Scoot Worker API, as well as a CLI client that wraps it. |
|
starter
Package server provides the implementation of the Scoot Worker Server, which implements the Worker API and starts the actual worker.
|
Package server provides the implementation of the Scoot Worker Server, which implements the Worker API and starts the actual worker. |
Click to show internal directories.
Click to hide internal directories.