TL;DR: Check active-series growth and churn alongside query and ingestion load. An unbounded identifier label is a common cause, but the graph alone does not prove it. Use TSDB status to find the new labelsets; drop the offending metric temporarily if needed, then fix its instrumentation. Removing a label does not aggregate colliding samples.
How to approach it
Name the mechanism in one sentence, because everything else follows: each observed labelset creates a series; the Cartesian product is only an upper bound. Then go to the API that names the offender, rather than guessing.
A strong answer
Prometheus holds an in-memory index and recent chunks for every active series. A series is a unique combination of metric name and label values, so http_requests_total{service, route, method, status} with 20 services, 50 routes, 5 methods and 8 statuses permits up to 40,000 combinations if every combination is emitted. Adding user_id can multiply that upper bound, but actual series follow observed combinations. Compare recent series creation, retention and rollout changes before attributing the memory jump.
Find it without guessing:
curl -s localhost:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName[:10]'
curl -s localhost:9090/api/v1/status/tsdb | jq '.data.labelValueCountByLabelName[:10]'
The responses identify candidates by metric and label; correlate them with the changed instrumentation. Common sources include user ID, request ID, full URL path with IDs embedded, email address, pod name on a metric that is aggregated anyway, or a Kubernetes label propagated automatically by an exporter.
A scoped emergency mitigation is to drop the known offending metric at scrape time:
metric_relabel_configs:
- source_labels: [__name__]
regex: 'expensive_metric_.*'
action: drop
When a label is redundant and removing it preserves sample uniqueness, the valid syntax is:
metric_relabel_configs:
- action: labeldrop
regex: 'user_id'
Use that second rule only after checking all affected metrics in this scrape job. labeldrop matches label names and does not use source_labels. If two samples differ only by user_id, removing it creates the same series identity twice; Prometheus does not sum them. Aggregate in the application/exporter instead, or drop the unsafe metric until a corrected version is deployed. Relabelling limits new ingestion but does not instantly remove historical blocks or reclaim all head memory. Prometheus configuration.
Then the structural part, which is what the question is really asking. Three controls, all worth having.
A limit that fails loudly. sample_limit bounds samples after metric relabelling; exceeding it fails the entire target scrape. Alert on scrape failures and choose the limit from measured target output. This is not a bound on unique series over time: a target can stay below the limit on every scrape while emitting fresh identifiers continuously.
A review point. Cardinality is introduced in application code, so the check belongs where labels are added. A short rule that a label may only take values from a bounded, known set catches almost everything, and "is this ever unbounded?" is a good code review question.
The right tool for high-cardinality data. Identifiers belong in traces and logs, not metrics. Traces are designed for exactly this: per-request data with unbounded attributes, sampled. Exemplars link a metric bucket to a specific trace, so you keep the aggregate cheap and can still jump to the individual request. Explaining that trade is the senior half of the answer.
If the scale justifies it, the architectural answers are remote write to Mimir, Thanos or Cortex, which move long-term storage and query fan-out off the scraping instance, and vertical sharding of scrape jobs across several Prometheus servers. Neither fixes cardinality; they raise the ceiling.
What interviewers probe next
"Why is a metric worse than a log here?" Stored samples and index entries can remain through retention even after a series stops being active. A log line has different indexing and retention costs; identifiers can still be expensive there. Distinguish active head memory from historical storage.
"What does churn mean?" Series that appear and disappear, typically from pod names on redeployment. High churn inflates the index even when the point-in-time count looks fine, and it is why pod as a label on an aggregated metric is expensive.
"How would you have caught this before production?" Cardinality checks in staging with realistic data, plus the sample limit set from day one.
Common mistakes
Adding memory. It buys weeks and the growth continues.
Removing the label in the query rather than at ingestion. The cost is paid at ingestion; querying differently changes nothing.
Not knowing the TSDB status API, which turns a five-minute diagnosis into an afternoon of guessing.