Simulator
A distributed systems and infrastructure simulator for attacking and debugging
Kubernetes: simulator
creates a kuberntes cluster for you in your AWS
account; runs scenarios which misconfigure it and/or leave it vulnerable to
compromise and trains you in mitigating against these vulnerabilities.
For details on why we created this project and our ultimate goals take a look at the vision statement.
Before you start
Usage
Ensure the AWS requirements are met and configured.
The quickest way to get up and running is to simply:
source <(curl https://raw.githubusercontent.com/kubernetes-simulator/simulator/master/kubesim)
or clone this repository and run:
make run
This will drop you into a bash shell in a launch container. You will have a
program on the PATH
named simulator
to interact with.
Refer to Simulator CLI Usage
Simulator CLI Usage
Creating Environment And Lauching Scenario
Before you launch your environment, please see the How It All Works section to ensure you have the necessary credentials and permissions in place and know what you are standing up.
Create a remote state bucket for terraform
simulator init
You will be asked for a name of a S3 bucket, which must be globally unique as pre AWS standards. If this does not exist it will be created, else the existing bucket will be used. This is used to store Terraform state to.
Create the infra if it isn't there
simulator infra create
This will standup the infrastructure, including an initial Kubernetes Cluster
List available scenarios
simulator scenario list
This will list all currently available scenrios
To get a better idea of what is involved in each scenario, e.g. node-shock-tactics
simulator scenario describe node-shock-tactics
Launch a scenario (sets up your cluster)
simulator scenario launch node-shock-tactics
This will launch your selected scenario.
Login to the environment
simulator ssh attack
Running simulator ssh attack
logs you into a container running on the Bastion host. Upon login, an outline of the challenge will be displayed. In addition, shortcuts for logging into the master, or nodes, of the Kubernetes cluster and how to show hints and start tasks are displayed. starting_point
logs you into the correct starting point as per the challenge.
From within the ssh attack container you have access to a range of helper commands such as start_task
, next_hint
and show_hints
.
Start task
start_task 1
The start_task
command is used to inform the simulator which task you are undertaking with the number associated, and therefore what hints are available to aid you with that task.
Accessing hints
next_hint
The next_hint
command will provide a hint to help you complete the task you have started with the start_task
command.
Viewing all hints that have been requested
show_hints
The show_hints
command will display all the hints you have requested to that point, in the task you have started.
Ending a task
end_task
or
start_task 2
The end_task
command will end the task you are currently on and score you. This will also happen if you move onto the a new task with the start_task
command. When you end a task in one of these ways, you will be scored 100 points minus the amount of hints you have displayed for that task at a value of -10 each. This is customizable through the task yamls.
Cleaning Up Environment
Destroy your cluster when you are done
simulator infra destroy
Once you have finished you must destroy the environment to ensure that no additional costs are incurred. This command is actioned from within the launch container.
Scenarios
While you can write your own, the following scenarios are currently shipped with the simulator:
container-ambush
container-defeat-in-detail
container-phalanx-formation
etcd-inverted-wedge
master-shell-scrape
master-encirclement
network-feint
network-hammer-and-anvil
network-hedgehog-defence
network-swarming
node-amphibious-operations
node-raiding
node-shock-tactics
policy-echelon-formation
policy-fire-support
policy-force-dispersal
policy-vertical-envelopment
rbac-contact-drill
rbac-sangar
rbac-flanking-maneuver
rbac-shoot-and-scoot
secret-high-ground
secret-tank-desant
How It All Works
Infrastructure Design
AWS Configuration
Simulator uses terraform to provision its infrastructure. Terraform in turn
honours all the rules for configuring the AWS
CLI.
Please follow these.
You can provide your credentials via the AWS_ACCESS_KEY_ID
and
AWS_SECRET_ACCESS_KEY
, environment variables, representing your AWS Access Key
and AWS Secret Key, respectively. Note that setting your AWS credentials using
either these (or legacy) environment variables will override the use of
AWS_SHARED_CREDENTIALS_FILE
and AWS_PROFILE
. The AWS_DEFAULT_REGION
and
AWS_SESSION_TOKEN
environment variables are also used, if applicable.
https://www.terraform.io/docs/backends/types/s3.html
All the AWS_*
configuration environment variables you have set will be propagated into the container
Troubleshooting AWS
- If you get a timeout when running
simulator infra create
after about 10 minutes, the region you are using
is probably running slowly. You must run simulator infra destroy
and then retry simulator infra
create
AWS_REGION
vs AWS_DEFAULT_REGION
- There have been
some issues with the
Go AWS client region configuration
- Multi-account
Refer to the simulator terraform documentation
SSH
Simulator whether run in the launch container or on the host machine will generate its own SSH RSA key pair. It will configure the cluster to allow access only with this keypair and automates writing SSH config and keyscanning the bastion on your behalf using custom SSH config and known_hosts files. This keeps all simulator-related SSH config separate from any other configs you may have. All simulator-related SSH files are written to ~/.ssh
and are files starting cp_simulator_
If you delete any of the files then simulator will recreate them and reconfigure the infrastructure as necessary on the
next run
Scenario Launching (perturb)
Refer to the scenario launch documentation
Launch Flow Sequence
Refer to the Launch scenario flow documentation
Scenario Task Definition
Refer to the Tasks YAML file Format documentation
The kubesim script
kubesim
is a small script written in BASH for getting users up and running with simulator as fast as possible. It pulls the latest version of the simulator container and sets up some options for running the image. It can be installed with the following steps:
# cURL the script from GitHub
curl -o kubesim https://raw.githubusercontent.com/kubernetes-simulator/simulator/master/kubesim
# Make it executeable
chmod a+x kubesim
# Place the script on your path
cp kubesim /usr/local/bin
Development Workflow
Git hooks
To ensure all Git hooks are in place run the following
make setup-dev
Development targets are specified in the Makefile.
setup-dev Initialise simulation tree with git hooks
devtools-deps: # Install devtools dependencies
devtools Install devtools
test-devtools: # Run all the unit tests for the devtools
reset Clean up files left over by simulator
validate-requirements Verify all requirements are met
previous-tag:
release-tag:
gpg-preflight:
run Run the simulator - the build stage of the container runs all the cli tests
docker-build-no-cache Builds the launch container
docker-build Builds the launch container
docker-test Run the tests
dep Install dependencies for other targets
build Run golang build for the CLI program
is-in-launch-container checks you are running in the launch container
test run all tests except goss tests
test-acceptance Run tcl acceptance tests for the CLI program
test-smoke Run expect smoke test to check happy path works end-to-end
test-unit Run golang unit tests for the CLI program
test-cleanup cleans up automated test artefacts if e.g. you ctrl-c abort a test run
test-coverage Run golang unit tests with coverage and opens a browser with the results
docs Generate documentation
release: gpg-preflight previous-tag release-tag docker-test docker-build build
Git commits
We follow the conventional commit specification.
Please ensure your commit messages adhere to this spec. If you submit a pull request
without following them be aware we will squash and merge your work with a message
that does.
Architecture
Components
Specifications
-
tasks.yaml format
-
If you need to make changes to the format you should update this documentation.
-
Any changes should be accompanied by a bump of the version in the kind
property
-
Use the migrate-hints
devtool to update the existing scenarios en-masse.
You can make this tool available on your PATH by running make devtools
Simulator API Documentation
The simulator API docs are available on godoc.org:
Contributing
Guidelines for contributors, and instructions on working with specific
parts of Simulator are in the Contributor Guide.