TL;DR: Samples land in a write-ahead log plus an in-memory head block covering roughly the last two hours; every two hours the head truncates into an immutable on-disk block, background compaction merges blocks into progressively larger ones, and retention expires whole blocks rather than individual samples. This explains slow restarts, cold dashboards and why a full disk stops ingestion entirely.
How to approach it
Treat it as a storage-engine question with four beats: write path, durability, read path, lifecycle on disk. Saying "pull model" here answers nothing; the interviewer is checking whether you have operated Prometheus or only installed it.
A strong answer
The write path is WAL first, memory second. A scrape response is parsed into samples and appended to the head block, the in-memory structure holding recent chunks. Every sample also goes to the write-ahead log in segment files around 128 MB, used for recovery. Durability depends on filesystem, storage and failure mode; the WAL does not establish a universal few-seconds loss bound. Head chunks are written to mmapped files as they fill, which is why some recent data survives restarts even before WAL replay finishes. On startup Prometheus replays the WAL since its last checkpoint to rebuild the head; checkpoints are cut when the head truncates, so replay time tracks how much high-cardinality churn happened since the last two-hour boundary.
Blocks are immutable and grow by merging. Every two hours the head truncates into a persistent block: a directory named by ULID containing metadata, a chunk file set and an inverted index. Background compaction then merges adjacent blocks into larger ones through a few generations, from two hours up to a cap around thirty-six hours at default settings. Queries spanning many small blocks pay iterator-merging overhead, so a freshly scraped metric is cheap to read but a three-week-old range query benefits from having been compacted into fewer, bigger blocks.
Retention deletes whole blocks, not points. When a block's time range falls fully outside --storage.tsdb.retention.time, it is deleted as a unit. Two consequences worth saying out loud: disk usage overshoots the nominal retention by up to one max-size block, and the optional admin API supports delete_series with label/time selectors using tombstones, and clean_tombstones reclaims deleted data. Secure that API and handle remote copies/backups separately, which is why cardinality mistakes are expensive to undo.
The failure modes are what they are really asking about. A full disk breaks WAL appends, ingestion stops, and Prometheus keeps serving increasingly stale data while looking alive; alerts silently age out. Monitor WAL fsync latency and disk headroom, give the data directory a dedicated volume, and treat remote_write to a long-term store (Thanos, Mimir) as durability insurance rather than a nice-to-have. Backfilling history uses promtool tsdb create-blocks-from openmetrics, dropping pre-built blocks into the data dir; anything older than retention gets discarded at startup regardless of what you just imported. HA deduplication depends on the receiving/query backend and its configured replica labels; it is not an automatic property of any remote-write store.
One texture point interviewers love: queries crawl right after a restart not because TSDB logic broke but because the OS page cache is cold and index lookups hit disk for the first time again. Warm-up traffic or a chunk cache fixes it.
What interviewers probe next
"Two HA replicas scrape the same targets. What does each store?" Both store everything locally without issue; duplicates only matter where their data meets, so give replicas distinct replica labels and configure Thanos Query or the selected backend’s HA deduplication; sidecars do not themselves perform query deduplication, never by trying to stop one side scraping.
"Why is my brand-new counter's increase() wrong?" It is not storage, but the same windowing logic applies; increase extrapolates from observed samples with boundary handling, but it does not simply scale every young counter to a whole window. Check sample count, resets and scrape gaps before judging the result.
"When does WAL replay hurt?" After unclean shutdowns with large un-checkpointed heads, typically cardinality-heavy servers; the durable fix is bounding series count, which is exactly why cardinality discipline and crash-recovery time are the same conversation.
Common mistakes
Reciting "pull model, exporters, Grafana" as architecture. That describes collection, not storage.
Claiming Prometheus loses everything on crash because data lives in memory. The WAL exists precisely for this and replay usually takes seconds to minutes.
Assuming retention trims individual samples per series. It expires whole blocks, which changes how you reason about both disk sizing and deletion requests.
References