scoot

module
Version: v0.0.0-...-71d7ec8 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 1, 2021 License: Apache-2.0

README

Scoot

GoDoc Build Status GoReportCard codecov.io

Scoot is a distributed task runner. The core concept of Scoot is the Snapshot, which represents an immutable filesystem state. Scoot allows for execution of commands against input Snapshots to create new output Snapshots.

Scoot Components
Scheduler

The Scheduler receives and distributes jobs to workers, and maintains state. It is also responsible for serving the Cloud Scoot API for clients.

Worker

The Worker (workerserver) receives information about jobs and runs them, and is responsible for all Snapshot-related functionality.

Client APIs
  • Cloud Scoot API
  • Worker API
Jobs and Tasks

Scoot work is broken down into independent jobs, which can consist of one or more tasks, which are executed sequentially.

Go Version

Scoot is intended to work with go modules. If this repository is on your GOPATH, please set GO111MODULE=on when working with go tools in this repository, and preferably use the Go distribution pinned in the .github/workflows/ci.yml file.

Example

Setup a scheduler and worker nodes locally:

go run ./setup-cloud-scoot/main.go --strategy local.local

Run a series of randomly generated tests against the local scheduler and workers:

go run ./scheduler/client/scootcl/main.go smoke_test

Scoot Integration Tests

Scoot has a few tests that exercise varying levels of common usages and workflows.

Smoketest

Invokes a scoot client directly to run jobs against a local cluster and waits for the scheduled jobs to complete.

(./scheduler/client/cli/smoketest.go)

Recoverytest

Invokes a scoot client directly to run jobs against a local cluster, kills the cluster, attempts to spin up a new one, and waits for the originally scheduled jobs to complete.

(./binaries/recoverytest/main.go)

Integration

Invokes a scoot and scoot-snapshot-db client via CLI to run a job against a local cluster and waits for the job to complete

(./binaries/scoot-integration/main.go)

Invokes a scoot bazel client (bzutil & fs_util) via CLI to run a job against a local cluster using the bazel API grpc endpoints

(./binaries/bazel-integration/main.go)

Scoot Thrift Code

Thrift Prerequisites Install the Thrift tool and golang thrift repository locally using the following section.

Generating thrift files See documentation in thrift definition files for specific generation instructions, or make thrift.

Development Installation Instructions

Install 3rd party tools:

Thrift (version == 0.9.3)

macOS:

brew install thrift@0.9
Thrift for go:

go modules will take care of this

Install/Access Scoot Executables and libraries

Install binaries to $GOPATH/bin:

make install

Directories

Path Synopsis
Async provides tools for asynchronous callback processing using Goroutines
Async provides tools for asynchronous callback processing using Goroutines
Bazel Remote Execution API gRPC
Bazel Remote Execution API gRPC
cas
Bazel Remote Execution API gRPC server Contains limited implementation of the ContentAddressableStore API interface
Bazel Remote Execution API gRPC server Contains limited implementation of the ContentAddressableStore API interface
cas/mock_bytestream
Package mock_bytestream is a generated GoMock package.
Package mock_bytestream is a generated GoMock package.
cas/mock_connection
Package mock_connection is a generated GoMock package.
Package mock_connection is a generated GoMock package.
execution
Bazel Remote Execution API gRPC server Contains limited implementation of the Execution API interface
Bazel Remote Execution API gRPC server Contains limited implementation of the Execution API interface
execution/bazelapi
Execution Request & Action Result Definitions Domain structures for tracking ExecuteRequest info (use cases include Scheduler JobDefs and Worker RunCommands), ActionResult info (used in Worker RunStatuses), and conversions for internal thrift APIs.
Execution Request & Action Result Definitions Domain structures for tracking ExecuteRequest info (use cases include Scheduler JobDefs and Worker RunCommands), ActionResult info (used in Worker RunStatuses), and conversions for internal thrift APIs.
execution/mock_longrunning
Package mock_longrunning is a generated GoMock package.
Package mock_longrunning is a generated GoMock package.
execution/mock_remoteexecution
Package mock_remoteexecution is a generated GoMock package.
Package mock_remoteexecution is a generated GoMock package.
binaries
Package cleaner provides internal cleanup-related utilities, primarily to limit disk consumption in long-lived workers.
Package cleaner provides internal cleanup-related utilities, primarily to limit disk consumption in long-lived workers.
cloud
cluster
Cluster provides the means for coordinating the schedulers and workers that make up a Scoot system.
Cluster provides the means for coordinating the schedulers and workers that make up a Scoot system.
cluster/local
Package local provides a cluster Fetcher implementation for obtaining state of nodes running on the local machine.
Package local provides a cluster Fetcher implementation for obtaining state of nodes running on the local machine.
allocator
Package allocator provides tools for centralized management of a common resource by multiple clients.
Package allocator provides tools for centralized management of a common resource by multiple clients.
dialer
Library for establishing Thrift network connections for clients.
Library for establishing Thrift network connections for clients.
endpoints
Wrappers for receivers from the common/stats package and setting up an HTTP server with endpoints to make the stats data accessible.
Wrappers for receivers from the common/stats package and setting up an HTTP server with endpoints to make the stats data accessible.
proto
Library for common gRPC protobuf-related tools.
Library for common gRPC protobuf-related tools.
stats
This package provides a set of minimal interfaces which both build on and are by default backed by go-metrics.
This package provides a set of minimal interfaces which both build on and are by default backed by go-metrics.
thrifthelpers
Utils for JSON and binary serialize/deserialize functions for Thrift structures
Utils for JSON and binary serialize/deserialize functions for Thrift structures
config
jsonconfig
Jsonconfig implements configuration, reading json into an ice Module.
Jsonconfig implements configuration, reading json into an ice Module.
ice is a lightweight Dependency Injection Framework ice's cenral metaphor is a "Magic Bag".
ice is a lightweight Dependency Injection Framework ice's cenral metaphor is a "Magic Bag".
integration-tests
perftests contains tooling for running platform performance tests.
perftests contains tooling for running platform performance tests.
apiserver
perftests/apiserver contains apiserver performance test tooling
perftests/apiserver contains apiserver performance test tooling
apiserver/loadtest
The loadtest package implements running performance benchmarking against the apiserver.
The loadtest package implements running performance benchmarking against the apiserver.
apiserver/loadtest/apiserver_benchmarker
apiserver_benchmarker is the cli for generating load on the apiserver.
apiserver_benchmarker is the cli for generating load on the apiserver.
scheduler_simulator
Scheduling algorithm evaluation framework.
Scheduling algorithm evaluation framework.
Package runner provides for execution of Scoot work and retrieval of the results of that work.
Package runner provides for execution of Scoot work and retrieval of the results of that work.
mocks
Package mocks is a generated GoMock package.
Package mocks is a generated GoMock package.
Package saga provides a generic implementation of the Saga pattern.
Package saga provides a generic implementation of the Saga pattern.
sagalogs
Package sagalogs provides implementations of SagaLog.
Package sagalogs provides implementations of SagaLog.
api
Package api implements the Scoot thrift API, Bazel grpc remote execution and longrunning operation APIs, and the TwitterServer http API.
Package api implements the Scoot thrift API, Bazel grpc remote execution and longrunning operation APIs, and the TwitterServer http API.
api/thrift
Package thrift implements the Scoot API, see scoot.thrift for details
Package thrift implements the Scoot API, see scoot.thrift for details
client/cli
Package client provides a CLI implementation of a client for the Cloud Scoot API.
Package client provides a CLI implementation of a client for the Cloud Scoot API.
domain
Package domain provides definitions for Scoot Jobs and Tasks
Package domain provides definitions for Scoot Jobs and Tasks
server
Package server provides the main job scheduling interface for Scoot Package server is a generated GoMock package.
Package server provides the main job scheduling interface for Scoot Package server is a generated GoMock package.
setup
Package setup provides Cloud Scoot Setup, which is a mechanism for running Scoot with local components, or initializing a connection to remote ones, depending on configuration.
Package setup provides Cloud Scoot Setup, which is a mechanism for running Scoot with local components, or initializing a connection to remote ones, depending on configuration.
package snapshot offers access to Snapshot.
package snapshot offers access to Snapshot.
cli
git/repo
Package repo provides utilities for operating on a git repo.
Package repo provides utilities for operating on a git repo.
store
This package defines the Store interfaces for reading and writing bundles (or any artifact data) to some underlying system, and Store implementations.
This package defines the Store interfaces for reading and writing bundles (or any artifact data) to some underlying system, and Store implementations.
tests
worker
api
Package server provides the implementation of the Scoot Worker Server, which implements the Worker API and starts the actual worker.
Package server provides the implementation of the Scoot Worker Server, which implements the Worker API and starts the actual worker.
client
The client package provides an interface and implementation for the Scoot Worker API, as well as a CLI client that wraps it.
The client package provides an interface and implementation for the Scoot Worker API, as well as a CLI client that wraps it.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL