qubeley

README

Qubeley

Qubeley is a real-time systems metrics pipeline, powered by Kafka, Clickhouse, and Grafana.

Overview

• Collector (cmd/qubeley) runs on the host and gathers CPU, memory, temperature, and system log metrics every 5 seconds, publishing each as a JSON message to Kafka
• Consumer (cmd/consumer) reads from Kafka and writes structured rows into ClickHouse
• Grafana queries ClickHouse directly and displays a live system metrics dashboard

Below is how the final product will look like:

Requirements

• Go 1.24+
• Docker with the Compose plugin

Getting started

1. Clone the repo

git clone https://github.com/andrewwkimm/qubeley.git
cd qubeley

2. Start the infrastructure

make up

Wait for Kafka and ClickHouse to become healthy:

docker compose ps

Both should show healthy before proceeding. This typically takes 30–60 seconds.

3. Run the ClickHouse migrations

The migrations run automatically on first start via docker-entrypoint-initdb.d. If the tables are missing (e.g. the volume pre-existed), run them manually:

make migrations

Verify:

docker exec -i qubeley-clickhouse clickhouse-client --user qubeley --password qubeley_dev --query "SHOW TABLES FROM qubeley"

You should see: cpu_metrics, logs, memory_metrics, raw_metrics, temperature_metrics.

4. Start the collector and consumer

In two separate terminals:

# Terminal 1 — collects metrics and publishes to Kafka
make run

# Terminal 2 — reads from Kafka and writes to ClickHouse
make consume

5. Open Grafana

Navigate to http://localhost:3000 and log in with admin / admin.

The System Metrics dashboard is auto-provisioned. Set the time range to Last 5 minutes to see live data.

6. Stopping

# Ctrl+C in both terminal windows, then:
make down

Data is preserved in Docker volumes. make up will resume from where you left off.

Project structure

Path	Description
cmd/qubeley	Collector binary; gathers host metrics and publishes to Kafka
cmd/consumer	Consumer binary; reads from Kafka and writes to ClickHouse
internal/collectors	Metric collector implementations (CPU, memory, temperature, logs)
internal/kafka	Kafka producer client
internal/models	Shared metric types used across the pipeline
clickhouse/migrations	ClickHouse schema migration SQL files
grafana	Auto-provisioned Grafana datasource and dashboard definitions

Contributing

To contribute to this project, run the setup target after cloning the repository.

make setup

To test changes locally, run:

make build