cluster

README

Vanadium Cluster

This package and its sub-packages implement the functionality required to run Vanadium applications in a cluster environment.

In this kind of environment, the application is composed of one or more replicas. Each replica has its own Principal with Blessings that are extended from the application's Blessings.

If the application's Blessing is root/my-service, the replicas' Blessings could be root/my-service/XXX and root/my-service/YYY.

The design was first proposed in this document.

Cluster agent

The Cluster agent is in charge of storing the Blessings of the applications. When a replica starts, it fetches its Blessings from the cluster agent.

The Cluster agent interface is defined in service.vdl.

vkube

vkube is a command-line tool that helps manage Vanadium applications on Kubernetes.

Before vkube starts an application, it creates a new secret in the cluster agent and saves the Secret Key in a Kubernetes Secret object. This Secret Key will later be used by the replicas to get their Blessings.

Then, it takes the user-specified Deployment and adds a pod-agent container to its Pod template.

The pod-agent uses the Secret Key to get the blessings for the replica, and makes them available to the other container(s) via the agent protocol.

Walkthrough for GKE

This walkthrough shows everything you need to start a test Vanadium application on Kubernetes.

It assumes that you are using the Container Engine on Google Cloud (aka GKE). The tools should work with Kubernetes on other platforms, but have not been tested outside of GKE.

Environment

This walkthrough assumes that the JIRI_ROOT is set and that $JIRI_ROOT/release/go/bin is in your PATH.

# Our work directory
mkdir $HOME/cluster-walkthrough
cd $HOME/cluster-walkthrough

# For jiri and vanadium binaries
export PATH=$JIRI_ROOT/.jiri_root/scripts:$JIRI_ROOT/release/go/bin:$PATH

# Adjust these to your environment
export PROJECT=my-project
export ZONE=my-zone
export CLUSTER=my-cluster

Tools

The following tools are required to work with GKE:

• gcloud: follow the instructions at cloud.google.com/sdk
• docker: follow the instructions at docker.com
• kubectl

gcloud components install kubectl

• vkube

jiri go install v.io/x/ref/services/cluster/vkube

Create a cluster

You can create a kubernetes cluster from the cloud console, or with the gcloud container clusters create command.

This command creates a cluster named $CLUSTER.

gcloud container clusters create $CLUSTER --num-nodes 1 --project $PROJECT --zone $ZONE

Create a persistent disk for the cluster agent

gcloud compute disks create test-cluster-agent-disk --size 200GB --project $PROJECT --zone $ZONE

Create test Vanadium credentials

jiri go install v.io/x/ref/cmd/principal
mkdir test-root
principal create --with-passphrase=false test-root test-root
export V23_CREDENTIALS=$(pwd)/test-root
principal dump

Create vkube.cfg

By default, the vkube command looks for vkube.cfg in the current directory. If vkube.cfg is not in your current directory, you can specify where it is with --config.

 cat - <<EOF >vkube.cfg
{
  "project": "$PROJECT",
  "zone": "$ZONE",
  "cluster": "$CLUSTER",
  "clusterAgent": {
    "namespace": "default",
    "image": "gcr.io/$PROJECT/cluster-agent:latest",
    "cpu": "0.1",
    "memory": "250M",
    "blessing": "test-root:cluster-agent",
    "admin": "test-root",
    "persistentDisk": "test-cluster-agent-disk"
  },
  "podAgent": {
    "image": "gcr.io/$PROJECT/pod-agent:latest"
  }
}
EOF

The blessings and admin values should match the credentials that you created earlier.

Build the cluster-agent and pod-agent images

This command will build two docker images and push them to your project's registry.

vkube build-docker-images -v

Start the cluster agent

This command creates a Service and a Deployment for the cluster agent. Make sure that V23_CREDENTIALS is set and points at your test credentials.

vkube start-cluster-agent --wait

Claim the cluster agent

The first time that the cluster agent is started, it doesn't have any blessings. The claim-cluster-agent command sends the blessings defined in vkube.cfg to the cluster agent.

vkube claim-cluster-agent

Build a docker image

In this example, we use fortuned as our test application.

Let's create a docker image that contains a statically linked fortuned binary, and push it to the project's registry.

mkdir fortune-docker

 cat - <<EOF >fortune-docker/Dockerfile
FROM busybox
COPY fortuned /usr/local/bin/
EOF

IMAGE=gcr.io/$PROJECT/fortuned
jiri go build -o fortune-docker/fortuned -ldflags "-extldflags -static" \
  v.io/x/ref/examples/fortune/fortuned
docker build -t $IMAGE fortune-docker
gcloud docker push $IMAGE

Run your first application

Create fortuned-deployment.json

The Pod template contains only the fortuned container. There is no need to specify the pod-agent container. The vkube command adds it for us transparently.

 cat - <<EOF >fortuned-deployment.json
{
  "apiVersion": "extensions/v1beta1",
  "kind": "Deployment",
  "metadata": {
    "name": "fortuned"
  },
  "spec": {
    "template": {
      "metadata": {
        "labels": {
          "application": "fortuned"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "fortuned",
            "image": "gcr.io/$PROJECT/fortuned:latest",
            "command": [ "fortuned", "--v23.tcp.address=:2345" ],
            "ports": [ { "containerPort": 2345 } ]
          }
        ]
      }
    }
  }
}
EOF

Start the application with blessing extension fortuned

vkube start -f fortuned-deployment.json fortuned --wait

Verify that fortuned is running.

vkube kubectl get pods

Create the fortune service (load balancer)

 cat - <<EOF >fortuned-service.json
{
  "apiVersion": "v1",
  "kind": "Service",
  "metadata": {
    "name": "fortune"
  },
  "spec": {
    "ports": [
      { "port": 2345, "targetPort": 2345 }
    ],
    "selector": {
      "application": "fortuned"
    },
    "type": "LoadBalancer"
  }
}
EOF

vkube kubectl create -f fortuned-service.json

vkube kubectl get service fortune

The last command will show the service's external IP address after a minute or two.

Verify that the service is working

jiri go install v.io/x/ref/cmd/vrpc

IPADDR=$(vkube kubectl get service fortune --template '{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}')
vrpc signature /$IPADDR:2345
vrpc identify /$IPADDR:2345

This should show something like

$ vrpc signature /$IPADDR:2345
// Fortune is the interface to a fortune-telling service.
type "v.io/x/ref/examples/fortune".Fortune interface {
	// Adds a fortune to the set used by Get().
	Add(fortune string) error
	// Returns a random fortune.
	Get() (fortune string | error)
	// Returns whether or not a fortune exists.
	Has(fortune string) (bool | error)
}

$ vrpc identify /$IPADDR:2345
PRESENTED: test-root:fortuned:10.64.0.4:54945
VALID:     [test-root:fortuned:10.64.0.4:54945]
PUBLICKEY: 3d:92:fc:5f:b3:ee:72:1f:7b:bf:22:a2:d4:d8:86:1a

Clean up

When you're done, these commands will turn off everything.

vkube stop -f fortuned-deployment.json
vkube kubectl delete service fortune
vkube stop-cluster-agent
gcloud compute disks delete test-cluster-agent-disk --project $PROJECT --zone $ZONE

If you created a test cluster, you can delete it with gcloud container clusters create ...

Documentation

Index

• Variables
• type ClusterAgentAdminClientMethods
• type ClusterAgentAdminClientStub
  • func ClusterAgentAdminClient(name string) ClusterAgentAdminClientStub
• type ClusterAgentAdminServerMethods
• type ClusterAgentAdminServerStub
  • func ClusterAgentAdminServer(impl ClusterAgentAdminServerMethods) ClusterAgentAdminServerStub
• type ClusterAgentAdminServerStubMethods
• type ClusterAgentClientMethods
• type ClusterAgentClientStub
  • func ClusterAgentClient(name string) ClusterAgentClientStub
• type ClusterAgentServerMethods
• type ClusterAgentServerStub
  • func ClusterAgentServer(impl ClusterAgentServerMethods) ClusterAgentServerStub
• type ClusterAgentServerStubMethods

Variables

var ClusterAgentAdminDesc rpc.InterfaceDesc = descClusterAgentAdmin

ClusterAgentAdminDesc describes the ClusterAgentAdmin interface.

var ClusterAgentDesc rpc.InterfaceDesc = descClusterAgent

ClusterAgentDesc describes the ClusterAgent interface.

Types

type ClusterAgentAdminClientMethods

type ClusterAgentAdminClientMethods interface {
	ClusterAgentClientMethods
	// Creates a new "secret" that can be used to retrieve extensions
	// of the blessings granted on this RPC, e.g. with the rpc.Granter
	// ClientCallOpt in Go.
	NewSecret(*context.T, ...rpc.CallOpt) (secret string, _ error)
	// Forgets a secret and its associated blessings.
	ForgetSecret(_ *context.T, secret string, _ ...rpc.CallOpt) error
}

ClusterAgentAdminClientMethods is the client interface containing ClusterAgentAdmin methods.

type ClusterAgentAdminClientStub

type ClusterAgentAdminClientStub interface {
	ClusterAgentAdminClientMethods
	rpc.UniversalServiceMethods
}

ClusterAgentAdminClientStub adds universal methods to ClusterAgentAdminClientMethods.

func ClusterAgentAdminClient

func ClusterAgentAdminClient(name string) ClusterAgentAdminClientStub

ClusterAgentAdminClient returns a client stub for ClusterAgentAdmin.

type ClusterAgentAdminServerMethods

type ClusterAgentAdminServerMethods interface {
	ClusterAgentServerMethods
	// Creates a new "secret" that can be used to retrieve extensions
	// of the blessings granted on this RPC, e.g. with the rpc.Granter
	// ClientCallOpt in Go.
	NewSecret(*context.T, rpc.ServerCall) (secret string, _ error)
	// Forgets a secret and its associated blessings.
	ForgetSecret(_ *context.T, _ rpc.ServerCall, secret string) error
}

ClusterAgentAdminServerMethods is the interface a server writer implements for ClusterAgentAdmin.

type ClusterAgentAdminServerStub

type ClusterAgentAdminServerStub interface {
	ClusterAgentAdminServerStubMethods
	// Describe the ClusterAgentAdmin interfaces.
	Describe__() []rpc.InterfaceDesc
}

ClusterAgentAdminServerStub adds universal methods to ClusterAgentAdminServerStubMethods.

func ClusterAgentAdminServer

func ClusterAgentAdminServer(impl ClusterAgentAdminServerMethods) ClusterAgentAdminServerStub

ClusterAgentAdminServer returns a server stub for ClusterAgentAdmin. It converts an implementation of ClusterAgentAdminServerMethods into an object that may be used by rpc.Server.

type ClusterAgentAdminServerStubMethods

type ClusterAgentAdminServerStubMethods ClusterAgentAdminServerMethods

ClusterAgentAdminServerStubMethods is the server interface containing ClusterAgentAdmin methods, as expected by rpc.Server. There is no difference between this interface and ClusterAgentAdminServerMethods since there are no streaming methods.

type ClusterAgentClientMethods

type ClusterAgentClientMethods interface {
	// Retrieves all the blessings associated with a particular secret.
	// The only authorization required to access this method is the secret
	// itself.
	// TODO(rthellend): Consider adding other side-channel authorization
	// mechanisms, e.g. verify that the IP address of the client belongs to
	// an authorized user.
	SeekBlessings(_ *context.T, secret string, _ ...rpc.CallOpt) (security.Blessings, error)
}

ClusterAgentClientMethods is the client interface containing ClusterAgent methods.

type ClusterAgentClientStub

type ClusterAgentClientStub interface {
	ClusterAgentClientMethods
	rpc.UniversalServiceMethods
}

ClusterAgentClientStub adds universal methods to ClusterAgentClientMethods.

func ClusterAgentClient

func ClusterAgentClient(name string) ClusterAgentClientStub

ClusterAgentClient returns a client stub for ClusterAgent.

type ClusterAgentServerMethods

type ClusterAgentServerMethods interface {
	// Retrieves all the blessings associated with a particular secret.
	// The only authorization required to access this method is the secret
	// itself.
	// TODO(rthellend): Consider adding other side-channel authorization
	// mechanisms, e.g. verify that the IP address of the client belongs to
	// an authorized user.
	SeekBlessings(_ *context.T, _ rpc.ServerCall, secret string) (security.Blessings, error)
}

ClusterAgentServerMethods is the interface a server writer implements for ClusterAgent.

type ClusterAgentServerStub

type ClusterAgentServerStub interface {
	ClusterAgentServerStubMethods
	// Describe the ClusterAgent interfaces.
	Describe__() []rpc.InterfaceDesc
}

ClusterAgentServerStub adds universal methods to ClusterAgentServerStubMethods.

func ClusterAgentServer

func ClusterAgentServer(impl ClusterAgentServerMethods) ClusterAgentServerStub

ClusterAgentServer returns a server stub for ClusterAgent. It converts an implementation of ClusterAgentServerMethods into an object that may be used by rpc.Server.

type ClusterAgentServerStubMethods

type ClusterAgentServerStubMethods ClusterAgentServerMethods

ClusterAgentServerStubMethods is the server interface containing ClusterAgent methods, as expected by rpc.Server. There is no difference between this interface and ClusterAgentServerMethods since there are no streaming methods.