README
¶
Scoot
Scoot is infrastructure to make developer tools smaller, simpler, and more distributed. The core concept of Scoot is the Snapshot, which represents an immutable filesystem state. Scoot allows for execution of commands against input Snapshots to create new output Snapshots.
Scoot Components
Scheduler
The Scheduler receives and distrubutes jobs to workers, and maintains state. It is also responsible for serving the Cloud Scoot API for clients.
Worker
The Worker (workerserver) receives information about jobs and runs them, and is responsible for all Snapshot-related functionality.
Daemon
The local daemon serves a Protobuf interface and can either send Scoot jobs to a remote scheduler or to embedded local workers.
Client APIs
- Cloud Scoot API
- Worker API
- Daemon API
Jobs and Tasks
Scoot work is broken down into independent jobs, which can consist of one or more tasks, which are executed sequentially.
Example
Setup a scheduler and worker nodes locally:
go run ./binaries/setup-cloud-scoot/main.go --strategy local.local
Run a series of randomly generated tests against the local scheduler and workers:
go run ./binaries/scootapi/main.go run_smoke_test
Scoot Thrift Code
(open source scoot code is in workspace/github.com/scootdev/scoot)
Generating thrift files (scootapi used as an example)
- To Generate files run from scoot's scootapi directory:
thrift --gen go:package_prefix=github.com/scootdev/scoot/scootapi/gen-go/,package=scoot,thrift_import=github.com/apache/thrift/lib/go/thrift scoot.thrift
Scoot Protobuf Code
Generating go protobuf files (for local Scoot Daemon) cd to scoot's daemon/protocol
protoc -I . daemon.proto --go_out=plugins=grpc:.
Generating python client files (for client library accessing local Scoot Daemon) cd to scoot's daemon/protocol
python -m grpc.tools.protoc -I. --python_out=./python/scoot --grpc_python_out=./python/scoot daemon.proto
Installation Instructions
Install 3rd party tools:
###Thrift (version >= 0.9.3) macOS:
brew install thrift
Thrift for go:
go get github.com/apache/thrift/lib/go/thrift
Protobuf for the Scoot Daemon
- If necessary, remove any conflicting or older versions of Protobuf:
brew uninstall protobuf
grpcio for python
pip install grpcio-tools
docopt for python
pip install docopt==0.6.2
Install/Access Scoot Executables and libraries
Scoot Local Daemon, Local Scheduler and Local Worker
*cd to scoot directory (workspace/github.com/scootdev/scoot) *run: go install ./binaries/... ** the binaries will be installed in workspace/bin
Scoot Local Daemon Command Line Client
### Python client library
Can be found at workspace/github.com/scootdev/scoot/daemon/protocol/python/scoot/client_lib.py
Directories
¶
| Path | Synopsis |
|---|---|
|
Async provides tools for asynchronous callback processing using Goroutines
|
Async provides tools for asynchronous callback processing using Goroutines |
|
binaries
|
|
|
apiserver
command
|
|
|
daemon
command
|
|
|
minfs
command
|
|
|
recoverytest
command
|
|
|
scheduler
command
|
|
|
scoot-snapshot-db
command
|
|
|
scootapi
command
|
|
|
setup-cloud-scoot
command
|
|
|
workercl
command
|
|
|
workerserver
command
|
|
|
cloud
|
|
|
cluster
Cluster provides the means for coordinating the schedulers and workers that make up a Scoot system.
|
Cluster provides the means for coordinating the schedulers and workers that make up a Scoot system. |
|
cluster/local
Package local provides a cluster Fetcher implementation for obtaining state of nodes running on the local machine.
|
Package local provides a cluster Fetcher implementation for obtaining state of nodes running on the local machine. |
|
common
|
|
|
dialer
Library for establishing Thrift network connections for clients.
|
Library for establishing Thrift network connections for clients. |
|
endpoints
Wrappers for receivers from the common/stats package and setting up an HTTP server with endpoints to make the stats data accessible.
|
Wrappers for receivers from the common/stats package and setting up an HTTP server with endpoints to make the stats data accessible. |
|
stats
This package provides a set of minimal interfaces which both build on and are by default backed by go-metrics.
|
This package provides a set of minimal interfaces which both build on and are by default backed by go-metrics. |
|
thrifthelpers
Utils for JSON and binary serialize/deserialize functions for Thrift structures
|
Utils for JSON and binary serialize/deserialize functions for Thrift structures |
|
config
|
|
|
jsonconfig
Jsonconfig implements configuration, reading json into an ice Module.
|
Jsonconfig implements configuration, reading json into an ice Module. |
|
daemon
|
|
|
integration/python
Dummy file to avoid: 'no buildable Go source files in /Users/jschiller/.go.d/gp/src/github.com/scootdev/scoot/daemon/integration/python'
|
Dummy file to avoid: 'no buildable Go source files in /Users/jschiller/.go.d/gp/src/github.com/scootdev/scoot/daemon/integration/python' |
|
protocol
Package protocol is a generated protocol buffer package.
|
Package protocol is a generated protocol buffer package. |
|
server
Package server provides the daemon Server interface and an implementation, which includes an API Handler and Protobuf server, that fulfils daemon.proto.
|
Package server provides the daemon Server interface and an implementation, which includes an API Handler and Protobuf server, that fulfils daemon.proto. |
|
fs
|
|
|
Package fuse enables writing FUSE file systems on Linux, OS X, and FreeBSD.
|
Package fuse enables writing FUSE file systems on Linux, OS X, and FreeBSD. |
|
syscallx
Package syscallx provides wrappers that make syscalls on various platforms more interoperable.
|
Package syscallx provides wrappers that make syscalls on various platforms more interoperable. |
|
ice is a lightweight Dependency Injection Framework
|
ice is a lightweight Dependency Injection Framework |
|
os
|
|
|
temp
Package temp makes hierarchical temporary directories it offers the same interface as ioutil, but recursive.
|
Package temp makes hierarchical temporary directories it offers the same interface as ioutil, but recursive. |
|
Package runner provides for execution of Scoot work and retrieval of the results of that work.
|
Package runner provides for execution of Scoot work and retrieval of the results of that work. |
|
Package saga provides a generic implementation of the Saga pattern.
|
Package saga provides a generic implementation of the Saga pattern. |
|
sagalogs
Package sagalogs provides an implementation of Saga Log.
|
Package sagalogs provides an implementation of Saga Log. |
|
Package sched provides definitions for Scoot Jobs and Tasks
|
Package sched provides definitions for Scoot Jobs and Tasks |
|
scheduler
Package scheduler provides the main job scheduling interface for Scoot
|
Package scheduler provides the main job scheduling interface for Scoot |
|
Package scootapi is where the Cloud Scoot Thrift definition lives, and also contains some simple addressing utils.
|
Package scootapi is where the Cloud Scoot Thrift definition lives, and also contains some simple addressing utils. |
|
client
Package client provides a CLI implementation of a client for the Cloud Scoot API.
|
Package client provides a CLI implementation of a client for the Cloud Scoot API. |
|
gen-go/scoot/cloud_scoot-remote
command
|
|
|
server
Package server implements the Scoot Cloud API.
|
Package server implements the Scoot Cloud API. |
|
setup
Package setup provides Cloud Scoot Setup, which is a mechanism for running Scoot with local components, or initializing a connection to remote ones, depending on configuration.
|
Package setup provides Cloud Scoot Setup, which is a mechanism for running Scoot with local components, or initializing a connection to remote ones, depending on configuration. |
|
package snapshot offers access to Snapshot.
|
package snapshot offers access to Snapshot. |
|
git/gitfiler
Package gitfiler offers Scoot Snapshot Filer operations access to git.
|
Package gitfiler offers Scoot Snapshot Filer operations access to git. |
|
git/repo
Package repo provides utilities for operating on a git repo.
|
Package repo provides utilities for operating on a git repo. |
|
utils/checkout
command
|
|
|
utils/countfiles
command
|
|
|
tests
|
|
|
client
The client package provides an interface and implementation for the Scoot Worker API, as well as a CLI client that wraps it.
|
The client package provides an interface and implementation for the Scoot Worker API, as well as a CLI client that wraps it. |
|
gen-go/worker/worker-remote
command
|
|
|
server
Package server provides the implementation of the Scoot Worker Server, which implements the Worker API and starts the actual worker.
|
Package server provides the implementation of the Scoot Worker Server, which implements the Worker API and starts the actual worker. |
Click to show internal directories.
Click to hide internal directories.