README ¶
Databricks - Grafana Data Source Backend Plugin
Grafana Databricks integration allowing direct connection to Databricks to query and visualize Databricks data in Grafana.
Get started with the plugin
Set up the Databricks Data Source
Requirements
- Grafana Version >= 9.1.0
If you are using an earlier Grafana version try the v1.1.7 release of this plugin, which is the latest release supporting Grafana > 7.0
Install the Data Source
- Install the plugin into the grafana plugin folder:
grafana-cli --pluginUrl https://github.com/mullerpeter/databricks-grafana/releases/latest/download/mullerpeter-databricks-datasource.zip plugins install mullerpeter-databricks-datasource
or
cd /var/lib/grafana/plugins/
wget https://github.com/mullerpeter/databricks-grafana/releases/latest/download/mullerpeter-databricks-datasource.zip
unzip mullerpeter-databricks-datasource.zip
- Edit the grafana configuration file to allow unsigned plugins:
- Linux:/etc/grafana/grafana.ini
- macOS:/usr/local/etc/grafana/grafana.ini
[plugins]
allow_loading_unsigned_plugins = mullerpeter-databricks-datasource
Or with docker
docker run -d \
-p 3000:3000 \
-v "$(pwd)"/grafana-plugins:/var/lib/grafana/plugins \
--name=grafana \
-e "GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=mullerpeter-databricks-datasource" \
grafana/grafana
- Restart grafana
Configure the Datasource
- Open the side menu by clicking the Grafana icon in the top header.
- In the side menu under the Configuration icon you should find a link named Data Sources.
- Click the
+ Add data source
button in the top header. - Select Databricks.
To configure the plugin use the values provided under JDBC/ODBC in the advanced options of the Databricks Cluster (or SQL Warehouse) and the the Credentials according to the chosen Authentication Method.
You can either authenticate the plugin using a Personal Access Token (PAT) or via M2M OAuth using a Service Principal Client ID and Client Secret.
Available configuration fields are as follows:
Name | Description |
---|---|
Server Hostname | Databricks Server Hostname (without http). i.e. XXX.cloud.databricks.com |
Server Port | Databricks Server Port (default 443 ) |
HTTP Path | HTTP Path value for the existing cluster or SQL warehouse. i.e. sql/1.0/endpoints/XXX |
Authentication Method | PAT (Personal Access Token) or M2M (Machine to Machine) OAuth Authentication |
Client ID | Databricks Service Principal Client ID. (only if M2M OAuth is chosen as Auth Method) |
Client Secret | Databricks Service Principal Client Secret. (only if M2M OAuth is chosen as Auth Method) |
Access Token | Personal Access Token for Databricks. (only if PAT is chosen as Auth Method) |
Code Auto Completion | If enabled the SQL editor will fetch catalogs/schemas/tables/columns from Databricks to provide suggestions. |
Supported Macros
All variables used in the SQL query get replaced by their respective values. See Grafana documentation for Global Variables.
Additionally the following Macros can be used within a query to simplify syntax and allow for dynamic parts.
Macro example | Description |
---|---|
$__timeFilter(time_column) |
Will be replaced by an expression to filter on the selected timerange. i.e. time_column BETWEEN '2021-12-31 23:00:00' AND '2022-01-01 22:59:59' |
$__timeWindow(time_column) |
Will be replaced by an expression to group by the selected interval. i.e. window(time_column, '2 HOURS') |
$__timeFrom |
Will be replaced by the start of the selected timerange. i.e. '2021-12-31 23:00:00' |
$__timeTo |
Will be replaced by the end of the selected timerange. i.e. '2022-01-01 22:59:59' |
Write a query
Use the query editor to write a query, you can use sparksql syntax according to the Databricks SQL Reference.
Long to Wide Transformation
By default, the plugin will return the results in wide format. This behavior can be changed in the advanced options of the query editor.
Code Auto Completion
Auto Completion for the code editor is still in development. Basic functionality is implemented, but might not always work perfectly. When enabled, the editor will make requests to Databricks while typing to get the available catalogs, schemas, tables and columns. Only the tables present in the current query will be fetched. Additionally, the editor will also make suggestions for Databricks SQL functions & keywords and Grafana macros.
The feature can be enabled in the Datasource Settings.
Examples
Single Value Time Series
SELECT $__time(time_column), avg(value_column)
FROM catalog.default.table_name
WHERE $__timeFilter(time_column)
GROUP BY $__timeWindow(time_column);
Multiple Values Time Series
SELECT window.start, avg(o_totalprice), o_orderstatus
FROM samples.tpch.orders
WHERE $__timeFilter(o_orderdate)
GROUP BY $__timeWindow(o_orderdate), o_orderstatus
ORDER BY start ASC;
Development
What is Grafana Data Source Backend Plugin?
Grafana supports a wide range of data sources, including Prometheus, MySQL, and even Datadog. There’s a good chance you can already visualize metrics from the systems you have set up. In some cases, though, you already have an in-house metrics solution that you’d like to add to your Grafana dashboards. Grafana Data Source Plugins enables integrating such solutions with Grafana.
For more information about backend plugins, refer to the documentation on Backend plugins.
Getting started
A data source backend plugin consists of both frontend and backend components.
Frontend
-
Install dependencies
yarn install
-
Build plugin in development mode or run in watch mode
yarn dev
or
yarn watch
-
Build plugin in production mode
yarn build
Backend
-
Update Grafana plugin SDK for Go dependency to the latest minor version:
go get -u github.com/grafana/grafana-plugin-sdk-go go mod tidy
-
Build backend plugin binaries for Linux, Windows and Darwin:
mage -v
-
List all available Mage targets for additional commands:
mage -l
Learn more
- Build a data source backend plugin tutorial
- Grafana documentation
- Grafana Tutorials - Grafana Tutorials are step-by-step guides that help you make the most of Grafana
- Grafana UI Library - UI components to help you build interfaces using Grafana Design System
- Grafana plugin SDK for Go