README
¶
Scoot
Scoot is infrastructure to make developer tools smaller, simpler, and more distributed. The core concept of Scoot is the Snapshot, which represents an immutable filesystem state. Scoot allows for execution of commands against input Snapshots to create new output Snapshots.
Scoot Components
Scheduler
The Scheduler receives and distrubutes jobs to workers, and maintains state. It is also responsible for serving the Cloud Scoot API for clients.
Worker
The Worker (workerserver) receives information about jobs and runs them, and is responsible for all Snapshot-related functionality.
Client APIs
- Cloud Scoot API
- Worker API
Jobs and Tasks
Scoot work is broken down into independent jobs, which can consist of one or more tasks, which are executed sequentially.
Example
Setup a scheduler and worker nodes locally:
go run ./binaries/setup-cloud-scoot/main.go --strategy local.local
Run a series of randomly generated tests against the local scheduler and workers:
go run ./binaries/scootapi/main.go run_smoke_test
Scoot Integration Tests
Scoot has a few tests that exercise varying levels of common usages and workflows.
Smoketest/Swarmtest
Invokes a scootapi client directly to run jobs against a local cluster and waits for the scheduled jobs to complete.
(./scootapi/client/smoke_test_cmd.go)
Recoverytest
Invokes a scootapi client directly to run jobs against a local cluster, kills the cluster, attempts to spin up a new one, and waits for the originally scheduled jobs to complete.
(./binaries/recoverytest/main.go)
Integration
Invokes a scootapi and scoot-snapshot-db client via CLI to run a job against a local cluster and waits for the job to complete
(./tests/integration_test.go)
Scoot Thrift Code
Thrift Prerequisites Install the Thrift tool and golang thrift repository locally using the following section.
Generating thrift files (scootapi used as an example)
See documentation in thrift definition files for specific generation instructions, or make thrift.
Installation Instructions
Install 3rd party tools:
Thrift (version >= 0.9.3)
macOS:
brew install thrift
Thrift for go:
go get github.com/apache/thrift/lib/go/thrift
Install/Access Scoot Executables and libraries
*cd to scoot directory (workspace/github.com/twitter/scoot) *run: go install ./binaries/... ** the binaries will be installed in workspace/bin
Directories
¶
| Path | Synopsis |
|---|---|
|
Async provides tools for asynchronous callback processing using Goroutines
|
Async provides tools for asynchronous callback processing using Goroutines |
|
Bazel Remote Execution API gRPC
|
Bazel Remote Execution API gRPC |
|
cas
Bazel Remote Execution API gRPC server Contains limited implementation of the ContentAddressableStore API interface
|
Bazel Remote Execution API gRPC server Contains limited implementation of the ContentAddressableStore API interface |
|
cas/mock_bytestream
Package mock_bytestream is a generated GoMock package.
|
Package mock_bytestream is a generated GoMock package. |
|
execution
Bazel Remote Execution API gRPC server Contains limited implementation of the Execution API interface
|
Bazel Remote Execution API gRPC server Contains limited implementation of the Execution API interface |
|
execution/bazelapi
Execution Request & Action Result Definitions Domain structures for tracking ExecuteRequest info (use cases include Scheduler JobDefs and Worker RunCommands), ActionResult info (used in Worker RunStatuses), and conversions for internal thrift APIs.
|
Execution Request & Action Result Definitions Domain structures for tracking ExecuteRequest info (use cases include Scheduler JobDefs and Worker RunCommands), ActionResult info (used in Worker RunStatuses), and conversions for internal thrift APIs. |
|
execution/mock_longrunning
Package mock_longrunning is a generated GoMock package.
|
Package mock_longrunning is a generated GoMock package. |
|
execution/mock_remoteexecution
Package mock_remoteexecution is a generated GoMock package.
|
Package mock_remoteexecution is a generated GoMock package. |
|
binaries
|
|
|
apiserver
command
|
|
|
bazel-integration
command
|
|
|
bzutil
command
|
|
|
minfs
command
|
|
|
recoverytest
command
|
|
|
scheduler
command
|
|
|
scoot-integration
command
|
|
|
scoot-snapshot-db
command
|
|
|
scootapi
command
|
|
|
setup-cloud-scoot
command
|
|
|
workercl
command
|
|
|
workerserver
command
|
|
|
cloud
|
|
|
cluster
Cluster provides the means for coordinating the schedulers and workers that make up a Scoot system.
|
Cluster provides the means for coordinating the schedulers and workers that make up a Scoot system. |
|
cluster/local
Package local provides a cluster Fetcher implementation for obtaining state of nodes running on the local machine.
|
Package local provides a cluster Fetcher implementation for obtaining state of nodes running on the local machine. |
|
dialer
Library for establishing Thrift network connections for clients.
|
Library for establishing Thrift network connections for clients. |
|
endpoints
Wrappers for receivers from the common/stats package and setting up an HTTP server with endpoints to make the stats data accessible.
|
Wrappers for receivers from the common/stats package and setting up an HTTP server with endpoints to make the stats data accessible. |
|
proto
Library for common gRPC protobuf-related tools.
|
Library for common gRPC protobuf-related tools. |
|
stats
This package provides a set of minimal interfaces which both build on and are by default backed by go-metrics.
|
This package provides a set of minimal interfaces which both build on and are by default backed by go-metrics. |
|
thrifthelpers
Utils for JSON and binary serialize/deserialize functions for Thrift structures
|
Utils for JSON and binary serialize/deserialize functions for Thrift structures |
|
config
|
|
|
jsonconfig
Jsonconfig implements configuration, reading json into an ice Module.
|
Jsonconfig implements configuration, reading json into an ice Module. |
|
fs
|
|
|
Package fuse enables writing FUSE file systems on Linux, OS X, and FreeBSD.
|
Package fuse enables writing FUSE file systems on Linux, OS X, and FreeBSD. |
|
syscallx
Package syscallx provides wrappers that make syscalls on various platforms more interoperable.
|
Package syscallx provides wrappers that make syscalls on various platforms more interoperable. |
|
ice is a lightweight Dependency Injection Framework
|
ice is a lightweight Dependency Injection Framework |
|
os
|
|
|
temp
Package temp makes hierarchical temporary directories it offers the same interface as ioutil, but recursive.
|
Package temp makes hierarchical temporary directories it offers the same interface as ioutil, but recursive. |
|
Package runner provides for execution of Scoot work and retrieval of the results of that work.
|
Package runner provides for execution of Scoot work and retrieval of the results of that work. |
|
mocks
Package mocks is a generated GoMock package.
|
Package mocks is a generated GoMock package. |
|
Package saga provides a generic implementation of the Saga pattern.
|
Package saga provides a generic implementation of the Saga pattern. |
|
sagalogs
Package sagalogs provides implementations of SagaLog.
|
Package sagalogs provides implementations of SagaLog. |
|
Package sched provides definitions for Scoot Jobs and Tasks
|
Package sched provides definitions for Scoot Jobs and Tasks |
|
scheduler
package scheduler provides StatefulScheduler which distributes tasks to a cluster of nodes.
|
package scheduler provides StatefulScheduler which distributes tasks to a cluster of nodes. |
|
client
Package client provides a CLI implementation of a client for the Cloud Scoot API.
|
Package client provides a CLI implementation of a client for the Cloud Scoot API. |
|
server
Package server implements the Scoot Cloud API.
|
Package server implements the Scoot Cloud API. |
|
setup
Package setup provides Cloud Scoot Setup, which is a mechanism for running Scoot with local components, or initializing a connection to remote ones, depending on configuration.
|
Package setup provides Cloud Scoot Setup, which is a mechanism for running Scoot with local components, or initializing a connection to remote ones, depending on configuration. |
|
package snapshot offers access to Snapshot.
|
package snapshot offers access to Snapshot. |
|
git/gitfiler
Package gitfiler offers Scoot Snapshot Filer operations access to git.
|
Package gitfiler offers Scoot Snapshot Filer operations access to git. |
|
git/repo
Package repo provides utilities for operating on a git repo.
|
Package repo provides utilities for operating on a git repo. |
|
store
This package defines the Store interfaces for reading and writing bundles (or any artifact data) to some underlying system, and Store implementations.
|
This package defines the Store interfaces for reading and writing bundles (or any artifact data) to some underlying system, and Store implementations. |
|
utils/checkout
command
|
|
|
utils/countfiles
command
|
|
|
tests
|
|
|
client
The client package provides an interface and implementation for the Scoot Worker API, as well as a CLI client that wraps it.
|
The client package provides an interface and implementation for the Scoot Worker API, as well as a CLI client that wraps it. |
|
server
Package server provides the implementation of the Scoot Worker Server, which implements the Worker API and starts the actual worker.
|
Package server provides the implementation of the Scoot Worker Server, which implements the Worker API and starts the actual worker. |
Click to show internal directories.
Click to hide internal directories.