{"dataset":{"id":"00000000-0000-0000-0000-000000000040","slug":"prometheus-best-practices","name":"Prometheus best practices","description":"Source-linked guidance on Prometheus fundamentals: metric and label naming, instrumentation, alerting, histograms versus summaries, and recording rules. Scoped to Prometheus; not a general monitoring or PromQL corpus.","visibility":"public","version":1,"documentCount":5,"publisher":"Koskamo editorial","license":"Original summaries; linked sources retain their own terms.","reviewedAt":"2026-10-04T00:00:00.000Z","readOnly":true,"createdAt":"2026-10-04T00:00:00.000Z","updatedAt":"2026-10-04T00:00:00.000Z"},"total":5,"documents":[{"title":"Recording rule naming and aggregation","sourceUrl":"https://prometheus.io/docs/practices/rules/","text":"Recording rules follow the naming form level:metric:operations: level is the aggregation level and labels of the rule output, metric is the metric name unchanged except for stripping _total off counters when using rate() or irate(), and operations lists the applied operations newest first — omitting _sum when other operations exist, merging associative operations, defaulting to sum when no operation is obvious, and joining divided metrics with _per_ plus the operation ratio for ratios. When aggregating a ratio, aggregate the numerator and denominator separately and then divide; never average a ratio or an average of averages, which is not statistically valid. When computing average observation size from a Summary's _count and _sum, keep the metric name without those suffixes and replace rate with mean in the operation. Always specify a without clause with the labels being aggregated away so other labels such as job are preserved, avoiding conflicts. Aggregating removes the without-clause labels from the output level; when there is no aggregation the levels always match, and a mismatch signals a mistake. Scope: Prometheus. Reviewed October 4, 2026.","id":"prometheus-best-practices:Recording rule naming and aggregation","version":1,"createdAt":"2026-10-04T00:00:00.000Z","updatedAt":"2026-10-04T00:00:00.000Z"},{"title":"Histograms versus summaries","sourceUrl":"https://prometheus.io/docs/practices/histograms/","text":"Summaries calculate pre-configured quantiles over sliding windows inside the instrumented program: accurate and fast to query, but the quantiles cannot be recalculated later or aggregated across instances, and each observation is relatively expensive due to streaming quantile calculation. Histograms count observations in buckets — cheap to observe — and quantiles are computed later on the server with histogram_quantile(), so they aggregate freely. Both track count and sum of observations, giving averages as rate of sum over rate of count. Classic histograms and summaries expose separate _count, _sum, and per-quantile or per-bucket (_bucket with the le label) series, and every configured classic bucket creates a series even when unpopulated. Native histograms instead store count, sum, and a dynamic number of exponential buckets in a single composite series; you configure a resolution, not bucket boundaries. Native histogram support is still rare and requires protobuf exposition, though the server can ingest classic histograms as NHCBs (native histograms with custom bucket boundaries), which store more efficiently while keeping the same PromQL handling. Scope: Prometheus. Reviewed October 4, 2026.","id":"prometheus-best-practices:Histograms versus summaries","version":1,"createdAt":"2026-10-04T00:00:00.000Z","updatedAt":"2026-10-04T00:00:00.000Z"},{"title":"Alerting on symptoms rather than causes","sourceUrl":"https://prometheus.io/docs/practices/alerting/","text":"Aim for as few alerts as possible, alerting on symptoms associated with end-user pain rather than trying to catch every possible cause; alerts should link to relevant consoles and allow slack for small blips. For online serving systems, alert on high latency and error rates as high up in the stack as possible, paging on latency at only one point — a slow lower-level component needs no page while overall user latency is fine — and on user-visible errors, with separate pages only for severe non-visible failures that require human involvement. For offline processing, page when the time data takes to get through the system causes user impact. For batch jobs, page when the job has not succeeded recently enough, allowing at least time for two full runs — for a job that runs every 4 hours and takes an hour, 10 hours is a reasonable threshold — and if one failed run is intolerable, run the job more frequently. Also alert on approaching capacity, and metamonitor: a blackbox check that alerts travel from PushGateway to Prometheus to Alertmanager to email beats per-component alerts. Scope: Prometheus. Reviewed October 4, 2026.","id":"prometheus-best-practices:Alerting on symptoms rather than causes","version":1,"createdAt":"2026-10-04T00:00:00.000Z","updatedAt":"2026-10-04T00:00:00.000Z"},{"title":"Instrumentation by service type, labels, and cardinality","sourceUrl":"https://prometheus.io/docs/practices/instrumentation/","text":"Services fit three main types. Online-serving systems need queries, errors, and latency; count queries when they end so they line up with error and latency stats. Offline processing tracks items in, in progress, and out per stage, with a heartbeat item carrying its insertion timestamp to detect stalls. Batch jobs export gauges — last success time, stage durations, overall runtime — pushed to a PushGateway; jobs running more often than every 15 minutes should be considered for conversion into daemons. Libraries should provide instrumentation with no additional configuration, tracking query count, errors, and latency for external resources at minimum. Increment a counter for every line of logging code and every failure, with a separate attempts metric so failure ratios are easy to compute. Combine related series into one metric with labels (http_responses_total with a code label) and never procedurally generate metric names. Keep cardinality below 10 per metric as a general guideline, investigate anything over 100, and start with no labels. Values that can go down are gauges; never rate() a gauge, and export Unix timestamps rather than time-since-event, computing time() - my_timestamp_metric. Scope: Prometheus. Reviewed October 4, 2026.","id":"prometheus-best-practices:Instrumentation by service type, labels, and cardinality","version":1,"createdAt":"2026-10-04T00:00:00.000Z","updatedAt":"2026-10-04T00:00:00.000Z"},{"title":"Metric and label naming conventions","sourceUrl":"https://prometheus.io/docs/practices/naming/","text":"The Prometheus naming conventions are a style guide and collection of best practices, not a requirement. A metric name MUST comply with the data model for valid characters, SHOULD carry a single-word application prefix (called a namespace by client libraries, as in prometheus_notifications_total or process_cpu_seconds_total), MUST refer to a single unit and a single quantity, SHOULD use base units (seconds, bytes, meters — not milliseconds or megabytes), and SHOULD end with the unit in plural form, with total as the suffix for accumulating counts and _info for pseudo-metrics exposing metadata. Common name components MAY be ordered first so related metrics sort together. Each metric SHOULD represent the same logical thing-being-measured across all label dimensions — sum() or avg() over a metric's dimensions should be meaningful. Use labels to differentiate characteristics (operation=\"create|update|delete\") and never put label names in the metric name. Every unique label combination is a new time series, so avoid high-cardinality values like user IDs or email addresses. Scope: Prometheus. Reviewed October 4, 2026.","id":"prometheus-best-practices:Metric and label naming conventions","version":1,"createdAt":"2026-10-04T00:00:00.000Z","updatedAt":"2026-10-04T00:00:00.000Z"}]}