TL;DR: Inventory who writes what before touching anything; most of the volume is debug-level noise and duplicated stacks nobody queries. Route by value into a hot searchable tier (days) and a cold object-storage tier (months) queried on demand, prioritize errors with bounded burst/dedup policies while sampling healthy traffic at source, and charge teams for their own ingest so the incentive survives you.
How to approach it
Refuse to start with a tool choice. Ask three questions: what is the monthly volume split by service and level, what fraction of stored logs was queried in the last quarter, and which compliance rules actually mandate retention. The answers usually show the bill is not buying diagnosability.
A strong answer
Measure first. Group ingest by producer and level. The pattern is depressingly consistent: two or three services emit half the bytes, much of it at debug or info level in production, request-success logs logged per call with full payloads, or a library default nobody tuned. Query rates against stored volume are typically abysmal; low query frequency suggests a tiering candidate, but audit and incident evidence can be valuable despite rarely being read.
Cut at the source, where deletion is free. Enforce levels: debug never ships to production log sinks, or ships behind a per-request flag when diagnosing that specific user's problem. Drop or aggregate high-frequency success lines; a service doing 5k requests per second gains little from 5k near-identical "200 OK" lines when a counter would do. Deduplicate repeated stack traces with occurrence counts. Structured JSON with consistent fields (service, trace_id, level) belongs everywhere, because structured logs can be filtered cheaply anywhere downstream.
Route by value instead of storing uniformly.
| Tier | Contents | Retention | Store |
|---|---|---|---|
| Hot | Errors, warnings, auth events, sampled requests | 7 to 14 days | Searchable (managed logging, Elasticsearch, ClickHouse) |
| Cold | Selected retained streams, compressed | 90 days to a year | Object storage, columnar (Parquet), queried on demand |
Model the actual contract: ingest, indexing, retention, retrieval and query charges. Moving 70 percent of bytes does not imply halving the bill when ingestion or committed spend dominates. Estimate savings by charge category and verify them in the pilot. Queries against cold data use Athena, DuckDB or ClickHouse over Parquet and take seconds to minutes, which is fine because investigations start in the hot tier and reach cold only when reconstructing history.
Sample like you mean it. Prefer error and slow-request evidence, with rate limits and repeated-error summaries during storms. A 1 percent healthy sample is an example, not a universal rate. Trace-aware retention needs candidate data available until the decision; head-dropped data cannot be recovered later.
Fix incentives or relive this in eighteen months. Per-team chargeback on ingest volume turns each engineering team into a stakeholder in its own verbosity. Pair it with a visible leaderboard; nothing reduces debug logging like the team's name appearing atop the cost list.
The position worth stating plainly: halving by shortening retention is the wrong first move and sometimes actively harmful, because the expensive part is ingestion pricing, not storage duration. Cut what you accept, then tier what remains.
What interviewers probe next
"Where does the pipeline itself break?" Agents (Vector, Fluent Bit, OTel collector) buffer to disk and drop by policy under pressure; make drop behaviour explicit and monitored rather than silent. A Kafka buffer between agents and sinks absorbs bursts but is one more system to run; at modest scale, disk-buffered agents suffice.
"Compliance wants seven years." Seven years of everything is almost never the requirement; usually specific event classes (auth, financial, access) need it. Route those streams separately from application chatter and archive them cheaply.
"How do you prove signal survived the cuts?" Re-run the last five incident investigations against the new pipeline and confirm each could reach its root cause. That test, not gut feel, is the acceptance criterion.
Common mistakes
Shortening retention as step one. It saves less than expected, destroys long-tail forensics, and leaves the firehose running.
Sampling errors along with successes. Head sampling applied blindly to logs deletes the evidence trail; sampling must be value-aware.
One vendor price renegotiation instead of architecture. Discounts get eaten within a year if volume keeps compounding.