GPU Cluster Observability & Diagnostics

This guide covers GPU debugging, burn tests, NCC tests, and using the observability dashboard to monitor and troubleshoot your cluster.

​

Table of Contents

• Observability Dashboard
• GPU Debugging
• Burn Test
• NCC Test

​

Observability Dashboard

Real-time visualization of cluster health metrics, allowing you to track resource utilization across specific nodes and timeframes to ensure optimal performance. Access the Observability dashboard from the left sidebar under MORE > Observability.

​

Selecting Nodes to Monitor

1. Select Cluster: Choose your cluster from the dropdown (e.g., k8s-7xdu304ghh-v1)
2. Select Nodes: Choose one or more nodes to monitor (e.g., k8s-7xdu304ghh-v1-node-1)
3. Set Time Range: Use the time range buttons to select the monitoring window:
   • Quick options: 6h, 12h, 24h
   • Or use Custom for a specific range

​

GPU Metrics (Time Series)

​

CPU Metrics (Gauges)

​

GPU Debugging

Systematic workflows to identify performance bottlenecks, diagnose thermal throttling, and resolve common training failures by correlating resource metrics.

​

Steps to Debug on the Observability Screen

1. Navigate to Observability
   • Click Observability in the left sidebar under MORE
2. Select Your Target
   • Choose the cluster from Select Cluster dropdown
   • Select the specific node(s) from Select Nodes dropdown
3. Set Appropriate Time Range
   • For recent issues: Use 6h, 12h, or 24h
   • For historical analysis: Use Custom
4. Analyze GPU Utilisation
   • Look for patterns correlating with your task progress
   • Sustained ~100% indicates the GPU is consistently executing CUDA kernels
   • Periodic dips suggest idle time between kernel launches (data loading, synchronization)
5. Check GPU Temperature and SM Clocks
   • If temperature approaches 80-85°C and SM Clocks drop, thermal throttling is occurring
   • Normal operation: SM Clocks stable around 2100 MHz or as per GPU specs
6. Monitor GPU DRAM
   • Gradual increase without release may indicate memory leak
   • Sudden spikes near capacity risk OOM errors
7. Correlate with CPU Metrics
   • High CPU Usage + Low GPU Utilisation = Data pipeline bottleneck (GPU waiting for data)
   • Low CPU Usage + Low GPU Utilisation = Task may be idle or blocked

​

Common Workload Patterns

​

Burn Test

Stress-testing protocols designed to push GPUs to their thermal and power limits to ensure hardware stability before production deployment.

​

Overview

Burn tests stress-test GPUs to verify stability under sustained high load. This helps identify hardware issues before deploying production workloads.

​

Steps to Perform Burn Test

1. SSH into the Node
   ssh <node-address>
   
2. Install gpu-burn
   git clone https://github.com/wilicc/gpu-burn.git
   cd gpu-burn
   make
   
3. Run the Burn Test
   # Run for 60 seconds (adjust duration as needed)
   ./gpu_burn 60
   
4. Monitor on Observability Dashboard Open the Observability dashboard and watch for:

   Metric	Expected During Burn Test
   GPU Utilisation	~100% sustained (continuous CUDA kernel execution)
   GPU Power	Near TDP (max power)
   GPU Temperature	Rising, should stay < 85°C
   GPU SM Clocks	Stable at boost frequency (~2100 MHz)
   GPU DRAM	Elevated but stable

5. Interpret Results
   • PASS: All metrics stable, no SM Clock drops, temperature under control
   • FAIL: SM Clocks dropping (throttling), temperature exceeding limits, or computation errors in terminal

​

What to Capture

​

NCC Test

Benchmarks for validating inter-GPU communication bandwidth and latency (NVLink/PCIe), which are essential for distributed training performance.

​

Overview

NCC (NCCL - NVIDIA Collective Communications Library) tests verify GPU-to-GPU communication performance, essential for distributed training workloads.

​

Steps to Perform NCC Test

1. SSH into the Node
   ssh <node-address>
   
2. Install nccl-tests
   git clone https://github.com/NVIDIA/nccl-tests.git
   cd nccl-tests
   make
   
3. Run All-Reduce Test
   # Test with all available GPUs
   ./build/all_reduce_perf -b 8 -e 256M -f 2 -g <num_gpus>
   
4. Monitor on Observability Dashboard

   Metric	Expected During NCC Test
   GPU Utilisation	Periodic bursts during collective operations
   GPU DRAM	Active memory transfers visible
   CPU Usage	Low (GPU-to-GPU communication)

5. Analyze Terminal Output The test outputs bandwidth numbers. Compare against expected values:

   Interconnect	Expected Bandwidth
   NVLink 3.0	~600 GB/s
   NVLink 4.0	~900 GB/s
   PCIe 4.0	~25 GB/s