Instrumentation by service type, labels, and cardinality

Services fit three main types. Online-serving systems need queries, errors, and latency; count queries when they end so they line up with error and latency stats. Offline processing tracks items in, in progress, and out per stage, with a heartbeat item carrying its insertion timestamp to detect stalls. Batch jobs export gauges — last success time, stage durations, overall runtime — pushed to a PushGateway; jobs running more often than every 15 minutes should be considered for conversion into daemons. Libraries should provide instrumentation with no additional configuration, tracking query count, errors, and latency for external resources at minimum. Increment a counter for every line of logging code and every failure, with a separate attempts metric so failure ratios are easy to compute. Combine related series into one metric with labels (http_responses_total with a code label) and never procedurally generate metric names. Keep cardinality below 10 per metric as a general guideline, investigate anything over 100, and start with no labels. Values that can go down are gauges; never rate() a gauge, and export Unix timestamps rather than time-since-event, computing time() - my_timestamp_metric. Scope: Prometheus. Reviewed October 4, 2026.