The challenge
The project needed to demonstrate more than a happy-path pipeline. Late, duplicate, malformed, and replayed events had to be handled safely while the resulting analytics stayed explainable.
01 / Case study
ClusterPulse
Monitor GPU health, utilization, and incidents across AI clusters.
Interactive project preview
Explore a fleet snapshot
Representative synthetic data. The full platform runs locally with Docker, Kafka, Spark, Airflow, dbt, and Grafana.
Thermal pressureHot GPUs trigger critical reliability incidents for investigation.
Active GPUs48
Avg utilization81.8%
Max temperature111.4°
Critical incidents8
GPU utilization82%
−30mNow
Temperature111°C
−30mNow
| Cluster | Active GPUs | Utilization | Max temp | State |
|---|---|---|---|---|
| cluster-a | 24 | 82% | 105.8°C | attention |
| cluster-b | 24 | 81.6% | 111.4°C | attention |
Overview
ClusterPulse turns synthetic telemetry into reliable insights for GPU health and capacity planning.
222,452valid events reached Silver
37models and tests passed
45,348events/sec contract baseline
The approach
The software engineering focus is explicit service boundaries, automated validation, and recovery under failure, with Docker supporting the local environment. Kafka provides the durable event boundary. Spark validates contracts, routes bad records, and maintains independent sink checkpoints. Airflow orchestrates analytics and backfills; dbt builds the dimensional Gold layer.
The result
Repeated runs produced identical trusted totals. Invalid events were quarantined without interrupting processing, and replaying duplicate IDs did not change the Silver record count.
See the implementation, documentation, and decisions.
Explore the repository ↗