TL;DR: Schedule plan runs against every environment and alert on any diff, because detection is cheap. The real work is attribution: distinguishing incident hotfixes from shadow IT from another team's automation. Then make the deliberate policy call on what console changes are ever legitimate, and give each class a paved road back into code.
How to approach it
Separate three problems that blur together: detecting drift (solved with scheduled plans), attributing it (process plus cloud audit logs), and preventing it (governance choice). Interviewers watch for whether you know prevention-by-policy beats prevention-by-scolding.
A strong answer
Detection: nightly plans, alerting on any non-empty diff. A CI job applies nothing; it runs terraform plan -detailed-exitcode per environment and posts nonzero exits to the owning team with the plan attached. Exit code 2 means planned differences; exit code 1 means the check failed and must alert separately. Differences can reflect desired configuration or provider changes as well as drift; use a known configuration revision. This catches drift within 24 hours while context still exists, versus quarterly when nobody remembers. Two refinements: use read-only provider permissions where supported, while granting the backend narrowly scoped lock access (which can require writes), and suppress known-noisy attributes centrally rather than teaching people to ignore alerts.
Attribution: the diff tells you what, audit logs tell you who. CloudTrail or its equivalents name the principal behind out-of-band changes, which sorts drift into its actual categories:
| Cause | Signature | Right response |
|---|---|---|
| Incident hotfix | Change during an outage, by on-call, documented in the incident ticket | Legitimate; needs backport to code within days |
| Another automation | Regular pattern, service principal, e.g. autoscaling or a cost tool | Reconcile ownership: import its writes or disable them |
| Console convenience | Developer mid-task, one-off | Close the gap that pushed them there |
| Shadow infrastructure | Resources nobody claims, sometimes unattached | Investigate then remove or adopt |
The incident-hotfix case deserves emphasis because it is where teams destroy their own credibility: if you punish the on-call engineer who scaled instances manually during a Sev1, drift continues but stops being visible. The policy must be explicit that emergency changes are correct behaviour with one obligation: open the pull request before end of next shift, reverting or codifying as appropriate.
Prevention is a policy call, made openly. The mature positions are two, and pretending both can hold is how drift wins:
- Read-only consoles for managed accounts. Humans cannot drift what they cannot change; break-glass roles exist for emergencies and their use alarms. Strongest guarantee, real cultural adjustment.
- Everything permitted, everything reconciled. Consoles stay open; scheduled reconciliation reverts uncodified changes automatically after a grace window. Softer, works where autonomy matters more than strictness, but requires nerve: reverts must actually fire or the policy teaches contempt.
Either is coherent. The incoherent middle ("please do not change things in the console") is what most organisations run, and their drift graphs prove it.
Closing gaps that cause drift. Every recurring drift source names a missing capability in the paved road: developers editing security groups means self-service SG changes via code are too slow; someone adding cron jobs by hand means your scheduler product has a hole. Fix the road and the walking-off-it stops. This reframing, drift as UX feedback, is the senior move in this answer.
What interviewers probe next
"Plan-based detection misses resources not yet in state." True: pair it with inventory comparison (cloud API listing diffed against tracked resources) so untracked creation also alerts. Two detectors, two failure modes covered.
"Doesn't auto-revert fight legitimate urgency?" Grace windows sized per environment (minutes for prod, hours for dev), plus revert exemptions tied to active incident tickets. Automation respects declared exceptions, never silence.
"Terraform state says clean but reality differs?" Investigate the detection boundary first: unmanaged resources, ignored fields, provider read behavior or stale inputs can explain it. State tampering and partial applies are possibilities, not the only explanation. State integrity monitoring belongs in the same alarm family.
Common mistakes
Quarterly manual drift reviews. Drift compounds; anything slower than a day loses attribution and turns reconciliation into archaeology.
Treating drifters as the problem. People route around friction; permanent fixes change the friction, not the people.
Auto-revert without communication. A reversion that surprises its author breeds a workaround, which is drift with better opsec.
Sources: Plan exit codes and refresh.