01 / Case study

ClusterPulse

Monitor GPU health, utilization, and incidents across AI clusters.

Interactive project preview

Explore a fleet snapshot

Representative synthetic data. The full platform runs locally with Docker, Kafka, Spark, Airflow, dbt, and Grafana.

Thermal pressureHot GPUs trigger critical reliability incidents for investigation.

Active GPUs48

Avg utilization81.8%

Max temperature111.4°

Critical incidents8

GPU utilization82%

−30mNow

Temperature111°C

−30mNow

Cluster	Active GPUs	Utilization	Max temp	State
cluster-a	24	82%	105.8°C	attention
cluster-b	24	81.6%	111.4°C	attention

RoleData engineering

Year2026

StackPython · Kafka · PySpark · Delta Lake · Airflow · dbt

Overview

ClusterPulse turns synthetic telemetry into reliable insights for GPU health and capacity planning.

ClusterPulse analytical dashboard with cluster filters, health metrics, and utilization charts — 01 / Analytics surface — fleet health, capacity, and reliability

Grafana dashboard monitoring ClusterPulse event volume, quarantined records, freshness, and processing lag — 02 / Operations surface — freshness, lag, throughput, and quarantine

222,452valid events reached Silver

37models and tests passed

45,348events/sec contract baseline

The challenge

The project needed to demonstrate more than a happy-path pipeline. Late, duplicate, malformed, and replayed events had to be handled safely while the resulting analytics stayed explainable.

The approach

Kafka provides the durable event boundary. Spark applies event-time watermarks, validates contracts, routes bad records, and maintains independent sink checkpoints. Airflow owns bounded analytics and backfill work; dbt builds the dimensional Gold layer.

The result

Repeated runs produced identical trusted totals. Invalid events were quarantined without interrupting processing, and replaying duplicate IDs did not change the Silver record count.

See the implementation, documentation, and decisions.

Explore the repository ↗