DevOpsInterviewPrep logo
CI/CD, Release Engineering & GitOps / 09
hard★ EssentialNewRed HatRazorpaySalesforce

A teammate kubectl-applied a fix during an incident and Argo CD reverted it a minute later. What happened?

Self-heal did exactly what it was configured to do. The senior answer is not switching it off; it is designing the Git path and the sync order so nobody bypasses the system at 3am again.

Updated Sep 2026 · Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: The application runs with automated sync and self-heal enabled, so the controller diffed live state against Git, saw the manual patch as drift, and restored the declared state. Git never saw the fix. The mature response keeps enforcement on and makes the commit path fast enough for incidents, with sync waves making reconciliation safe rather than flappy.

How to approach it

Name the mechanism in your first sentence: automated sync plus self-heal did this on purpose. Then pivot to design, because the real question is whether your incident process has a first-class Git lane.

A strong answer

Repository refresh timing and live-state self-heal are separate settings. Argo CD compares desired and live state and marks managed differences OutOfSync; a Git webhook speeds detection of a repository change, not every kind of drift. With automated.selfHeal enabled it acts on those differences: a field someone patched by hand gets restored to the manifest, and pruning can delete an application-tracked resource that is absent from the desired manifests. An unrelated, untracked resource is not automatically owned or pruned merely because it shares the namespace. Nothing mysterious happened to your teammate. The controller enforced exactly the invariant you asked for, against a fix that existed only in the cluster's memory.

rendering diagram…

So the design work is making sure going around Git never looks attractive:

An expedited lane for incidents. Use an incident-specific authorization path for a small config change, with the operator, approval and incident reference recorded. A Git commit preserves the diff; it does not by itself prove who authorized production access. Rehearse the path and measure its elapsed time.

Sync waves with health gates. Annotate resources with argocd.argoproj.io/sync-wave: negative for namespaces and config, then migration Jobs, then Deployments, then Ingress. Each wave pauses until the previous one reports Healthy, using built-in health checks for known kinds and Lua checks for CRDs. Without ordering, a rollout racing its ConfigMap is how self-healing turns into a flap loop that makes everyone distrust the controller.

Documented break-glass. Temporarily disable auto-sync on that application for the duration, where drift detection keeps reporting so the exception stays visible, then commit the emergency state or revert the patch before restoring automation. For generated Applications, change the controlling ApplicationSet policy as needed rather than fighting its reconciliation. IgnoreExtraneous changes sync-status calculation only: it is not a self-heal pause or prune exemption. Argo CD documents that distinction.

Where I would reverse the whole stance: if merging to the config repo takes fifteen minutes because every change queues behind human review of a shared monorepo, engineers will keep patching clusters, and self-heal becomes a fight you lose nightly. Fix the lane first, keep the enforcement second. Enforcement without a fast legitimate path produces shadow operations, which is worse than either alternative alone.

The line interviewers are really scoring: do you understand that GitOps moves operational truth during an incident? Before, truth was whatever the cluster ran. Now truth is what the repository says plus a short reconciliation lag, and every emergency procedure must be expressed through that lens rather than around it.

What interviewers probe next

"Manual versus automatic sync?" Choose from change risk and the required approval model. Automated reconciliation can suit a database with tested migrations; a stateless service can still require an approval window. Drift detection runs regardless, so even manual-sync apps surface divergence.

"Hooks versus waves?" Waves order steady-state resources. Hooks run one-shot work: a PreSync migration Job before anything changes, PostSync smoke tests whose failure marks the sync failed. Sync phases do not supply an automatic rollback phase; restoration needs an explicit, rehearsed procedure or another rollout controller.

"Pruning worries you?" It should, it is how a renamed manifest deletes the old object. Scope it per application, enable it deliberately, rehearse it in staging first.

Common mistakes

Answering "disable self-heal", which trades a permanent guarantee away to solve one bad night.

Conflating self-heal with prune: one reverts modified fields, the other deletes extraneous objects, and they fail differently.

Claiming GitOps means nobody ever touches the cluster, which is false. It means cluster edits become deliberate, visible exceptions instead of silent ones.

That one was free, and so are 10 answers per topic without an account. Signing in doubles that to 20, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
DISCUSSION · 0

Nothing here yet. Say how you would answer it.