README
¶
Monogon Operating System
A cluster operating system. Codename: metropolis. Linux kernel, stateless userland, API-driven management, high integrity. Designed to run Kubernetes and other workload scheduling systems.
Documentation
The canonical documentation for Monogon OS is the Monogon OS Handbook.
Developer Quick Start
Follow the setup instructions in the top-level README.md. We recommend using nix-shell either via Nix installed on an existing distribution or on NixOS.
Start a test cluster by running:
$ bazel run //metropolis:launch-cluster
This will build all the required code and run a fully userspace test cluster consisting of four qemu VMs (three for Monogon OS nodes, one for a swich/router emulator). No root access on the host required.
.--------. .--------. .--------.
| node 0 | | node 1 | | node 2 |
'--------' '--------' '--------'
^ ^ ^
| | | Virtual Ethernet (10.1.0.0/24)
V V V
.-------------------------.
| nanoswitch |
|-------------------------| .-------------------.
| Router, switch, |--->| Internet via host |
| SOCKS proxy | '-------------------'
'-------------------------'
^
| gRPC over SOCKS to nodes
.----------.
| metroctl |
'----------'
The launch tool will output information on how to connect to the cluster:
Launch: Cluster running!
To access cluster use: metroctl --config /tmp/metroctl-3733429479 --proxy 127.0.0.1:42385 --endpoints 10.1.0.2 --endpoints 10.1.0.3 --endpoints 10.1.0.4
Or use this handy wrapper: /tmp/metroctl-3733429479/metroctl.sh
To access Kubernetes, use kubectl --context=launch-cluster
You can use the metroctl wrapper to then look at the node list per the Monogon OS cluster control plane:
$ alias metroctl=/tmp/metroctl-3733429479/metroctl.sh
$ metroctl node describe
NODE ID STATE ADDRESS HEALTH ROLES TPM VERSION HEARTBEAT
metropolis-067651202d00b79fffe92df0001aabff UP 10.1.0.4 HEALTHY yes v0.1.0-dev494.g0d8a8a4f.dirty 1s
metropolis-7ccd2437c50696ea9a9b6543dc163f84 UP 10.1.0.3 HEALTHY yes v0.1.0-dev494.g0d8a8a4f.dirty 3s
metropolis-ec101152c48c5f761534c1910cf66200 UP 10.1.0.2 HEALTHY ConsensusMember,KubernetesController yes v0.1.0-dev494.g0d8a8a4f.dirty 3s
We have a node running the Monogon OS control plane (ConsensusMember role) and Kubernetes control plane ( KubernetesController role), but no Kubernetes worker nodes. But changing that is a simple API call (or metroctl invocation) away:
$ metroctl node add role KubernetesWorker metropolis-067651202d00b79fffe92df0001aabff
2024/02/12 17:42:33 Updated node metropolis-067651202d00b79fffe92df0001aabff.
$ metroctl node add role KubernetesWorker metropolis-7ccd2437c50696ea9a9b6543dc163f84
2024/02/12 17:42:36 Updated node metropolis-7ccd2437c50696ea9a9b6543dc163f84.
$ metroctl node describe
NODE ID STATE ADDRESS HEALTH ROLES TPM VERSION HEARTBEAT
metropolis-067651202d00b79fffe92df0001aabff UP 10.1.0.4 HEALTHY KubernetesWorker yes v0.1.0-dev494.g0d8a8a4f.dirty 0s
metropolis-7ccd2437c50696ea9a9b6543dc163f84 UP 10.1.0.3 HEALTHY KubernetesWorker yes v0.1.0-dev494.g0d8a8a4f.dirty 3s
metropolis-ec101152c48c5f761534c1910cf66200 UP 10.1.0.2 HEALTHY ConsensusMember,KubernetesController yes v0.1.0-dev494.g0d8a8a4f.dirty 3s
And just like that, we can now see these nodes in Kubernetes, too:
$ kubectl --context=launch-cluster get nodes
NAME STATUS ROLES AGE VERSION
metropolis-067651202d00b79fffe92df0001aabff Ready <none> 15s v1.24.2+mngn
metropolis-7ccd2437c50696ea9a9b6543dc163f84 Ready <none> 13s v1.24.2+mngn
$ kubectl --context=launch-cluster run -it --image=ubuntu:22.04 test -- bash
If you don't see a command prompt, try pressing enter.
root@test:/# uname -a
Linux test 6.1.69-metropolis #1 SMP PREEMPT_DYNAMIC Tue Jan 30 14:43:23 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
root@test:/#
With the test launch tooling, you can now start iterating on the codebase. Regardless of whether you're changing the
Linux kernel config or implementing a new RPC, testing your changes interactively is a single bazel command away.
End-to-end tests
We have an end-to-end test suite. It's run automatically on CI. Any new logic should be exercised there.
$ bazel run //metropolis/test/e2e:e2e_test
These tests operate on a fully virtualized cluster just like the launch tooling, so be patient.
Directories
¶
| Path | Synopsis |
|---|---|
|
cli
|
|
|
dbg
command
|
|
|
flagdefs
Package flagdefs contains shared flag definitions for Metropolis.
|
Package flagdefs contains shared flag definitions for Metropolis. |
|
metroctl
command
|
|
|
metroctl/core
Package core contains parts of metroctl which do significant amounts of work beyond just providing a CLI for them.
|
Package core contains parts of metroctl which do significant amounts of work beyond just providing a CLI for them. |
|
takeover
command
|
|
|
Installer creates a Metropolis image at a suitable block device based on the OS image present in the installation medium's ESP, after which it reboots.
|
Installer creates a Metropolis image at a suitable block device based on the OS image present in the installation medium's ESP, after which it reboots. |
|
install
Package install allows planning and executing the installation of Metropolis to a block device.
|
Package install allows planning and executing the installation of Metropolis to a block device. |
|
test/testos
command
TestOS is a tiny "operating system" which is packaged the exact same way as an actual Metropolis node but only outputs a single flag before exiting.
|
TestOS is a tiny "operating system" which is packaged the exact same way as an actual Metropolis node but only outputs a single flag before exiting. |
|
allocs
Package allocs contains allocations of various types of identifiers used in a node.
|
Package allocs contains allocations of various types of identifiers used in a node. |
|
bios_bootcode/genlogo
command
|
|
|
core
command
|
|
|
core/cluster
Package cluster implements low-level clustering logic, especially logic regarding to bootstrapping, registering into and joining a cluster.
|
Package cluster implements low-level clustering logic, especially logic regarding to bootstrapping, registering into and joining a cluster. |
|
core/consensus/client
package client implements a higher-level client for consensus/etcd that is to be used within the Metropolis node code for unprivileged access (ie.
|
package client implements a higher-level client for consensus/etcd that is to be used within the Metropolis node code for unprivileged access (ie. |
|
core/curator
package curator implements the Curator, a service responsible for management of the Metropolis cluster that it is running on.
|
package curator implements the Curator, a service responsible for management of the Metropolis cluster that it is running on. |
|
core/devmgr
Package devmgr is the userspace pendant to the kernel device management system.
|
Package devmgr is the userspace pendant to the kernel device management system. |
|
core/metrics/fake_exporter
command
fake_exporter is a tiny Prometheus-compatible metrics exporter which exports a single metric with a value configured at startup.
|
fake_exporter is a tiny Prometheus-compatible metrics exporter which exports a single metric with a value configured at startup. |
|
core/mgmt
Package mgmt implements the node-local management service, a.k.a.
|
Package mgmt implements the node-local management service, a.k.a. |
|
core/network/hostsfile
Package hostsfile implements a service which owns and writes all node-local files/interfaces used by the system to resolve the local node's name and the names of other nodes in the cluster:
|
Package hostsfile implements a service which owns and writes all node-local files/interfaces used by the system to resolve the local node's name and the names of other nodes in the cluster: |
|
core/network/ipam
Package ipam is just a stub right now which contains a type for describing prefix lists.
|
Package ipam is just a stub right now which contains a type for describing prefix lists. |
|
core/network/overlay
Package overlay implements a Cluster Networking mesh service running on all Metropolis nodes.
|
Package overlay implements a Cluster Networking mesh service running on all Metropolis nodes. |
|
core/roleserve
Package roleserve implements the roleserver/“Role Server”.
|
Package roleserve implements the roleserver/“Role Server”. |
|
core/tconsole/standalone
command
|
|
|
core/time
Package time implements a supervisor runnable which is responsible for keeping both the system clock and the RTC accurate.
|
Package time implements a supervisor runnable which is responsible for keeping both the system clock and the RTC accurate. |
|
core/update/e2e/testos
command
|
|
|
kubernetes/authproxy
Package authproxy implements an authenticating proxy in front of the K8s API server converting Metropolis credentials into authentication headers.
|
Package authproxy implements an authenticating proxy in front of the K8s API server converting Metropolis credentials into authentication headers. |
|
kubernetes/clusternet
Package clusternet implements a WireGuard-based overlay network for Kubernetes.
|
Package clusternet implements a WireGuard-based overlay network for Kubernetes. |
|
kubernetes/containerd/cniproxy
Package cni implements an adapter between the go-cni interface and the Monogon gRPC Workload Attachment interface.
|
Package cni implements an adapter between the go-cni interface and the Monogon gRPC Workload Attachment interface. |
|
kubernetes/hyperkube
command
This is the entry point for our multicall Kubernetes binary.
|
This is the entry point for our multicall Kubernetes binary. |
|
kubernetes/metricsprovider
Package metricsprovider provides a Prometheus registry for code in K8s client-go capable of providing metrics.
|
Package metricsprovider provides a Prometheus registry for code in K8s client-go capable of providing metrics. |
|
kubernetes/metricsproxy
Package metricsproxy implements an authenticating proxy in front of the K8s controller-manager and scheduler providing unauthenticated access to the metrics via local ports
|
Package metricsproxy implements an authenticating proxy in front of the K8s controller-manager and scheduler providing unauthenticated access to the metrics via local ports |
|
kubernetes/nfproxy
Package nfproxy is a Kubernetes Service IP proxy based exclusively on the Linux nftables interface.
|
Package nfproxy is a Kubernetes Service IP proxy based exclusively on the Linux nftables interface. |
|
kubernetes/pki
package pki builds upon osbase/pki/ to provide an etcd-backed implementation of all x509 PKI Certificates/CAs required to run Kubernetes.
|
package pki builds upon osbase/pki/ to provide an etcd-backed implementation of all x509 PKI Certificates/CAs required to run Kubernetes. |
|
kubernetes/plugins/kvmdevice
Package kvmdevice implements a Kubernetes device plugin for the virtual KVM device.
|
Package kvmdevice implements a Kubernetes device plugin for the virtual KVM device. |
|
kubernetes/reconciler
The reconciler ensures that a base set of K8s resources is always available in the cluster.
|
The reconciler ensures that a base set of K8s resources is always available in the cluster. |
|
proto
|
|
|
test
|
|
|
e2e/connectivity/agent
command
|
|
|
e2e/httpserver
command
httpserver serves a test HTTP endpoint for E2E testing.
|
httpserver serves a test HTTP endpoint for E2E testing. |
|
e2e/k8s_cts
command
This package launches a Metropolis cluster with two nodes and spawns in the CTS container.
|
This package launches a Metropolis cluster with two nodes and spawns in the CTS container. |
|
e2e/persistentvolume
command
This is a test for PersistentVolumes provided by our provisioner.
|
This is a test for PersistentVolumes provided by our provisioner. |
|
e2e/preseedtest
command
|
|
|
e2e/selftest
command
|
|
|
launch
cluster builds on the launch package and implements launching Metropolis nodes and clusters in a virtualized environment using qemu.
|
cluster builds on the launch package and implements launching Metropolis nodes and clusters in a virtualized environment using qemu. |
|
launch/cli/launch
command
|
|
|
launch/cli/launch-cluster
command
|
|
|
localregistry
Package localregistry implements a read-only OCI Distribution / Docker V2 container image registry backed by local layers.
|
Package localregistry implements a read-only OCI Distribution / Docker V2 container image registry backed by local layers. |
|
nanoswitch
command
nanoswitch is a virtualized switch/router combo intended for testing.
|
nanoswitch is a virtualized switch/router combo intended for testing. |
|
swtpm/certtool
command
|
|
|
swtpm/swtpm_cert
command
|
|
|
util
This file implements test helper functions that augment the way any given test is run.
|
This file implements test helper functions that augment the way any given test is run. |
Click to show internal directories.
Click to hide internal directories.