Alerting on symptoms rather than causes

Aim for as few alerts as possible, alerting on symptoms associated with end-user pain rather than trying to catch every possible cause; alerts should link to relevant consoles and allow slack for small blips. For online serving systems, alert on high latency and error rates as high up in the stack as possible, paging on latency at only one point — a slow lower-level component needs no page while overall user latency is fine — and on user-visible errors, with separate pages only for severe non-visible failures that require human involvement. For offline processing, page when the time data takes to get through the system causes user impact. For batch jobs, page when the job has not succeeded recently enough, allowing at least time for two full runs — for a job that runs every 4 hours and takes an hour, 10 hours is a reasonable threshold — and if one failed run is intolerable, run the job more frequently. Also alert on approaching capacity, and metamonitor: a blackbox check that alerts travel from PushGateway to Prometheus to Alertmanager to email beats per-component alerts. Scope: Prometheus. Reviewed October 4, 2026.