From Monitoring to Observability: Grafana Lab Part 2

In part 1, I got Grafana, Prometheus, and Node Exporter talking to each other. Dashboards were live, metrics were flowing, and it looked good. But looking at a graph isn’t observability. Observability is knowing the moment something breaks, not the moment you happen to check.

Part 2 is about closing that gap: writing an alert rule, deliberately breaking something to trigger it, and walking through the full detect, alert, resolve lifecycle. Same shape as a real major incident, just self-inflicted on purpose so I could watch every stage of it happen.

Step 1: Write the alert rule

1. In Grafana, go to Alerting > Alert rules > New alert rule.
2. Set the data source to Prometheus and use a query against the CPU metric Node Exporter exposes, something like:

   100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100)

This gives you CPU utilization as a percentage instead of idle time, which is easier to reason about in a threshold. 3. Set the condition: fire when the value is above 80 for 1 minute. Short enough to see it happen quickly in a lab, long enough to avoid noise from a one-second spike. 4. Add labels and annotations (severity, summary, description) so the alert is self-explanatory when it fires, not just a number with no context. 5. Point the notification policy at a contact point. I used a simple webhook/email route since this is a single-node lab, not a production on-call chain. 6. Save and confirm the rule shows as Normal on the alert list before moving on.

Step 2: Trigger it on purpose

I used stress-ng on the monitored VM to force sustained CPU load:

bash

sudo apt install stress-ng -y
stress-ng --cpu 4 --timeout 120s

Four workers for two minutes was enough to push utilization comfortably past the 80% threshold and hold it there long enough for the evaluation window to catch it.

Step 3: Watch the lifecycle

This is the part that actually matters:

• Detect – the Prometheus query starts returning values above 80% almost immediately once the stress load kicks in.
• Alert – after the 1-minute evaluation window, the rule flips from Normal to Pending, then to Firing. The notification goes out.
• Resolve – once stress-ng hits its timeout and load drops, the query falls back under threshold, and after the same evaluation window the alert flips back to Normal and a resolved notification fires.

Watching that full cycle end to end, on metrics I generated myself, made the alerting pipeline feel real in a way that reading documentation never does. It’s a small thing, but it’s the difference between “I set up Grafana” and “I know what happens when this breaks.”

Closing notes

One honest update since part 1: I’ve since removed the Docker containers running this stack. Not because Docker was the wrong choice, but because of storage constraints on the machine I’m running this lab on. That’s the reality of home labbing on a budget in Sri Lanka: sometimes the limiting factor isn’t the architecture, it’s the disk.

The lab will come back in some form, likely as native services instead of containers, and I’ll write that up when it happens. For now, part 2 stands as a record of what the stack did while it was up.