hostdb-server

README

HostDB Server

HostDB (hostdb.pdxfixit.com) is a metadata store of hosts and host-like devices. It is stored in a MariaDB instance, and accessed through either a GUI or a REST-ish API.

Configuration

A default configuration can be found in config.yaml. A JSON file can be substituted if desired. Connection details (such as app port, and db host/port) will be silently overridden in a k8s cluster.

Testing

There are integration tests in hostdb_test.go. Those tests will create a fresh database for testing. The tests expect a MariaDB instance (currently v10.3) to be accessible at 127.0.0.1:3306, with no password for the root user.

$ make mariadb_start
$ make test
$ make mariadb_stop

Data Format

The v0 API data format consists of the following fields:

• id – A unique GUID, generated by the server if not provided. (typically 40 chars, must be less than 64)
• type – The source from where the record was collected. Usually openstack, aws, etc. Cannot contain spaces.
• hostname
• ip
• timestamp – Timestamp for when the record was collected from the source. If not supplied by the collector/client, it will be generated by the server.
• committer – Optional, but recommended. Defaults to remote IP and User-Agent string.
• context – Related to type; provides information about the record (e.g. what origin endpoint was used).
• data – JSON data. Often referred to as the 'payload.' This should be unmodified from the source.
• hash – A SHA256 hash of the data/payload, generated by the server. This can be used to quickly determine if the payload has changed. If a hash is provided during a PUT/POST, it will be discarded!

Using the API

Currently, the only supported API version is v0. All write requests must use Basic auth.

There are four primary endpoints for the service;

• catalog – Provides a list of unique values for a requested data point (e.g. flavor).
• detail – Returns detailed information about the requested records.
• list – Returns a list of the requested records, omitting the data payload.
• records – Used for write operations and record management; accepts GET, PUT (single record), POST (multiple/bulk), DELETE verbs.

For examples on interacting with the API, please see EXAMPLES.md.

CI Builds & Deployment

Builds ran in CircleCI, and were defined in .circleci/config.yml. A successful build should result in the updated application getting deployed to k8s.

Required Environment Variables

• KUBE_CONFIG -- contents of a valid ~/.kube/config file, in JSON format
• NEWRELIC_APIKEY -- a New Relic API key used for deployment notifications
• REGISTRY_USER -- user with write permissions to the image repository
• REGISTRY_PASS -- password for the user

Optional Environment Variables

• REGISTRY_HOST -- defaults to registry.pdxfixit.com

Build

hostdb-server is written in go, and is compiled using gox. The app is compiled in such a way that it does not require a base operating system to run, and can be placed into a scratch container.

A database is required for the integration tests, so the first step is to stand up a database instance. After running tests, the binary is compiled.

Next, we run the binary, and post the contents of the sample_data folder to it.

Containerize

In this step we simply run docker build to create a container image, and docker push it up to the registry.

Please ensure REGISTRY_USER and REGISTRY_PASS environment variables have valid credentials with write permissions to the container registry.

Deploy

Deployment of hostdb-server is accomplished by installing a Helm chart into a Kubernetes cluster, into a dedicated namespace. A MariaDB replication pair are also installed in the namespace, using the public Bitnami chart.

Configuration for kubectl should be provided in an environment variable named KUBE_CONFIG, which gets written into the user space of the build. Please be aware, build systems like CircleCI don't support multi-line environment variables, so you may need to convert the kube config to JSON.

The project hostdb-server-chart gets cloned into the workspace, config.yaml is copied from this project into the chart, and then make commands are invoked from within that project.

Finally, upon a successful deployment, a notification is sent to New Relic. Please ensure the environment variable NEWRELIC_APIKEY is set.

Development

If you need to run a local instance to test the API, ensure you have a current installation of Docker and Go, then run the following commands:

$ make run
$ curl http://localhost:8080/health

This will clear the workspace, compile the binary, build it into a local container image, start MariaDB, and then start the service. Finally, we run curl to verify the status of the service.

make is your friend.

The Makefile can be used for the following tasks:

• get – Retrieves any golang dependencies.
• test – Runs the golang tests.
• compile – Compiles the source code into a binary.
• build and push – Build a container image, and push it to the registry, respectively.
• deploy – Installs kubectl and helm, clones hostdb-server-chart, and then upgrades the existing deployment. Requires KUBECONFIG env var to be set.
• clean – Removes junk from previous builds, stops any running containers, and removes any locally-built hostdb containers.
• run and stop – Will run the container, including a MariaDB instance. Note that stop will halt the app, but not MariaDB.
• sample_data – Import sample data for testing. Override the destination by setting REMOTE_HOST and REMOTE_PORT.
• validate – After sample_data has been imported, examine each of the sample files, and ensure that what is returned matches.
• mariadb_start, mariadb_stop and mariadb_restart – Start/stop a MariaDB container, fully configured for HostDB.
• mariadb_client – Starts a MariaDB container, and starts a client session. Will start a database instance if one is not already running.

Populating a database with sample data

In the sample-data folder, you'll find example datasets which can be imported for use during development and testing.

To import sample data into a local instance, simply run make sample_data. Remote destinations are possible too. For example, if one wishes to load sample OpenStack data into a local instance of the service, one might run a command similar to this:

$ make sample_data_openstack REMOTE_HOST=10.10.10.10 REMOTE_PORT=80 REMOTE_PASS=password

The following environment variables are supported for sample_data recipes:

• REMOTE_HOST
• REMOTE_PASS
• REMOTE_PORT
• REMOTE_USER

Prerequisites

Please ensure that you have the following apps ready for use.

• Golang
• Docker Desktop

If you're working with deployments of hostdb, you'll also need:

• Kubernetes and kubectl
• Helm

Dependencies (partial list)

• https://github.com/pdxfixit/hostdb (shared golang code)
• https://github.com/gin-gonic/gin (HTTP server)
• https://github.com/go-sql-driver/mysql (MariaDB)
• https://github.com/spf13/viper (configuration)
• https://github.com/satori/go.uuid (uuids)
• https://github.com/bitnami/charts/tree/master/bitnami/mariadb

See Also

• https://github.com/pdxfixit/hostdb-collector-aws
• https://github.com/pdxfixit/hostdb-collector-oneview
• https://github.com/pdxfixit/hostdb-collector-openstack
• https://github.com/pdxfixit/hostdb-collector-ucs
• https://github.com/pdxfixit/hostdb-collector-vrops
• https://github.com/pdxfixit/hostdb-server-chart

TODO

• richer auth for writes, reads
• expose historical data thru the api
• Provide method to export/dump database