deployment

README

NAIS deployment

Overview

NAIS deploy facilitates application deployment into NAV's Kubernetes clusters.

Developers push or merge code into the master branch of a Git repository, triggering an automated build using (preferably) Github actions. A successful build produces a Docker image artifact, which is uploaded onto Github package registry. The final step in the build pipeline sends an API request to NAIS deploy to deploy the Docker image onto one of our Kubernetes clusters.

How it works

1. The teams pipeline use deploy to send a deployment request to hookd.
2. hookd receives the deployment request, verifies its integrity and authenticity, and passes the message on to deployd via gRPC.
3. deployd receives the message from hookd, assumes the identity of the deploying team, and applies your Kubernetes resources into the specified cluster.
4. If the Kubernetes resources contained any Application or Deployment resources, deployd will wait until these are rolled out successfully, or a timeout occurs.

Any fatal error will short-circuit the process with a error or failure status posted back to Github. A successful deployment will result in a success status. Intermediary statuses will be posted, indicating the current state of the deployment.

Usage

The usage documentation has been moved to NAIS platform documentation.

Deploy API

We strongly recommend that teams use the deploy cli to deploy, rather than posting directly to hookd.

Post to /api/v1/deploy to deploy one or more resources into one of our Kubernetes clusters.

Successful requests result in creation of a deployment object on GitHub. Use this object to track the status of your deployment.

Request specification

{
  "resources": [
    {
      "kind": "Application",
      "apiVersion": "nais.io/v1alpha1",
      "metadata": { ... },
      "spec": { ... },
    }
  ],
  "team": "nobody",
  "cluster": "local",
  "environment": "dev-fss:default",
  "owner": "navikt",
  "repository": "deployment",
  "ref": "master",
  "timestamp": 1572942789,
}

Field	Type	Description
resources	list[object]	Array of Kubernetes resources
team	string	Team tag
cluster	string	Kubernetes cluster, see NAIS clusters
environment	string	GitHub environment
owner	string	GitHub repository owner
repository	string	GitHub repository name
ref	string	GitHub commit hash or tag
timestamp	int64	Current Unix timestamp

Additionally, the header X-NAIS-Signature must contain a keyed-hash message authentication code (HMAC). The code can be derived by hashing the request body using the SHA256 algorithm together with your team's NAIS Deploy API key.

Response specification

{
  "logURL": "http://localhost:8080/logs?delivery_id=9a0d1702-e7c5-448f-8a90-1e5ee29a043b&ts=1572437924",
  "correlationID": "9a0d1702-e7c5-448f-8a90-1e5ee29a043b",
  "message": "successful deployment",
  "githubDeployment": { ... }
}

Field	Type	Description
logURL	string	Direct link to human readable frontend where logs for this specific deployment can be read
correlationID	string	UUID used for correlation tracking across systems, especially in logs
message	string	Human readable indication of API result
githubDeployment	object	Data returned from GitHub Deployments API

Response status codes

Code	Retriable	Description
201	N/A	The request was valid and will be deployed. Track the status of your deployment using the GitHub Deployments API.
400	NO	The request contains errors and cannot be processed. Check the message field for details.
403	MAYBE	Authentication failed. Check that you're supplying the correct team; that the team is present on GitHub and has admin access to your repository; that you're using the correct API key; and properly HMAC signing the request.
404	NO	Wrong URL.
5xx	YES	NAIS deploy is having problems and is currently being fixed. Retry later.

Application components

hookd

This service communicates with Github, and acts as a relay between the Internet and our Kubernetes clusters.

Its main tasks are to:

• validate deployment events
• relay deployment requests to deployd using gRPC
• report deployment status back to GitHub

The validation part is done by checking if the signature attached to the deployment event is valid, and by checking the format of the deployment. Refer to the GitHub documentation as to how webhooks are secured.

deployd

Deployd's responsibility is to deploy resources into a Kubernetes cluster, and report state changes back to hookd using gRPC.

gRPC

gRPC is used as a communication protocol between hookd and deployd. Hookd starts a gRPC server with a deployment stream and a status service. Deployd registers as a client on Hookd. When Hookd receives a deployment request, it adds it to the relevant gRPC deployment stream. Deployd acts on the information, and then sends a deployment status to the gRPC status service on Hookd. Hookd publishes the deployment status to Github.

Compiling

Install Golang 1.15 or newer.

Check out the repository and run make. Dependencies will download automatically, and you should have three binary files at bin/hookd, bin/deployd, bin/deploy and bin/provision.

Running locally

Postgres

Start the database by running docker-compose up. This will start an empty local database. When you start hookd, the schema will be patched to current version. If this is the first time running this database, you have to add a dummy deployment key for deploy to work. In this case, the deployment key is 20cefcd6bd0e8b8860c4ea90e75d7123019ed7866c61bd09e23821948878a11d.

psql -U postgres -h localhost -p 5432 hookd <<< EOF
insert into
    apikey (key, team, team_azure_id, created, expires)
    values ('1608bf2caf81bb68d50bfb094a8e0d90de2b27260767a64a0103c6255077eb446f4fabcb7ae94514380b4fdc006bd50dfe2ea73f4b60c0c55891a60f',
            'aura',
            '59dbf5e6-8243-471c-bd57-162346f08d75', now(),now()+interval '10 years');
EOF

Hookd

run ./bin/hookd For local development, hookd will by default start with an insecure listener, Azure AD token validation turned off and github integration disabled. secure listener and Azure AD token validation can be turned on passing the following flags:

--azure.clientid string              Azure ClientId.
--azure.clientsecret string          Azure ClientSecret
--azure.discoveryurl string          Azure DiscoveryURL (default "https://login.microsoftonline.com/common/discovery/v2.0/keys")
--azure.preAuthorizedApps string     Preauthorized Applications as Json
--azure.teamMembershipAppID string   Application ID of canonical team list
--azure.tenant string                Azure Tenant
--grpc-authentication                Validate tokens on gRPC connection.

If you wish to test with these flags, you should have an application with Azure AD enabled.

Github integration can be turned on using the following flags:

--github-app-id int                  Github App ID.
--github-client-id string            Client ID of the Github App.
--github-client-secret string        Client secret of the GitHub App.
--github-enabled                     Enable connections to Github.
--github-install-id int              Github App installation ID.
--github-key-file string             Path to PEM key owned by Github App. (default "private-key.pem")

Deployd

To enable secure listener and Azure AD token validation on deployd, the following flags apply:

--azure.clientid string         Azure ClientId.
--azure.clientsecret string     Azure ClientSecret
--azure.tenant string           Azure Tenant
--grpc-authentication           Use token authentication on gRPC connection.
--grpc-use-tls                  Use secure connection when connecting to gRPC server.

Deploy

Once the above components are running and configured, you can deploy using the following command: ./bin/deploy --resource res.yaml --cluster local --apikey 20cefcd6bd0e8b8860c4ea90e75d7123019ed7866c61bd09e23821948878a11d --deploy-server http://localhost:8080 --wait