DevOpsInterviewPrep logo
Incident Response & Production Debugging / 01
medium★ EssentialNewGoogleUberFlipkart

A pod is in CrashLoopBackOff and the logs are empty. Debug it, step by step.

The most-asked scenario question in DevOps interviews, and the one where candidates most often list commands instead of reasoning. Empty logs is the clue, not the obstacle.

Updated Sep 2026 · Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: Empty logs can mean an early failure, the wrong container instance, buffering or a different log destination. Read the last terminated state and events. Exit 137 commonly means SIGKILL, which can come from OOM handling or another actor; exit codes are clues, not unique diagnoses.

How to approach it

Narrate a hypothesis and the observation that would eliminate it, rather than listing every command you know. The interviewer will withhold information deliberately, so ask for specific output. Say what you would check first and why that first.

A strong answer

First, get the exit code and the events, because they partition the whole search space:

kubectl describe pod api-7f9c | tail -30
kubectl get pod api-7f9c -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'
kubectl logs api-7f9c --previous   # the crashed instance, not the restarting one

kubectl logs without --previous reads the container that is currently backing off, which frequently has produced nothing yet. That alone explains a large share of "the logs are empty" reports.

Now branch on what comes back.

Exit code 137 with reason OOMKilled: the kernel killed it, but establish whether this was a cgroup OOM or a node-wide OOM. Check the node kernel log and cgroup memory events before changing the limit. A leak or a changed working set may be the cause. A JVM or Node process that reads host memory rather than its cgroup limit will happily size a heap larger than the container is allowed, and dies during warm-up every time.

Exit code 1 or 2 with no output: check initialization and logging configuration; the exit code alone does not establish either cause. A missing environment variable, a Secret or ConfigMap that does not exist in this namespace, or a dependency it connects to at startup and cannot reach. kubectl describe events will show FailedMount if it is a volume, and silence if it is a network dependency.

Exit code 126 or 127: the entrypoint is not executable or does not exist. Wrong architecture image on an arm node, a missing shared library in a distroless base, a script without the executable bit.

No exit code but repeated Killing events: a liveness probe is failing. The container starts, the probe fires before the app is ready, kubelet kills it, and the cycle repeats forever. The tell is that the restart interval matches the probe's period and threshold rather than the app's startup time. The fix is a startupProbe with a generous failure threshold, so slow starts are tolerated while liveness stays aggressive once running.

If none of the above is conclusive, use a debug container, or start a disposable copy if the application image actually contains a shell:

kubectl debug -it api-7f9c --image=busybox --target=api
kubectl run tmp --rm -it --image=myapp:1.4.2 --command -- sh

The BusyBox shell belongs to the debug image. Sharing a process namespace does not give it the target container's root filesystem; inspect target paths through /proc only where runtime support and permissions allow. A distroless application image may have no sh at all.

Then change one thing supported by the evidence. In production the order is mitigate first: roll back to the last known-good image tag, get the service healthy, and diagnose the bad image outside the critical path.

What interviewers probe next

"Why did raising the memory limit fix it in staging and not production?" Because production has a different working set, or because the limit change moved the pod to a node where it now cannot be scheduled, or because the real cause is a leak and staging never runs long enough to reach it.

"How do you tell an OOMKill of your container from a node under memory pressure?" A cgroup OOM and a node-wide kernel OOM can both produce OOMKilled. Kubelet node-pressure eviction is a separate path with eviction events. Correlate termination details, cgroup memory events and node kernel logs.

"What would you check if kubectl logs --previous says the container has not restarted?" Confirm the container name, pod UID and restart count. There may be no retained previous instance, or you may be looking at a replacement pod; use events and the runtime's available logs.

Common mistakes

Reading logs of the running container and concluding there is nothing to see. --previous is the entire trick.

Jumping straight to raising the memory limit because 137 was mentioned once in a runbook. Confirm OOMKilled first; 137 is also what you get from an external SIGKILL.

Removing the liveness probe as a permanent fix rather than as a diagnostic. Removing it to prove the probe is the cause is fine. Leaving it removed is how a hung process runs for a week.

Listing fifteen commands with no hypothesis. The round scores reasoning, and an unordered command dump reads as memorisation.

References

Kubernetes pod lifecycle.

That one was free, and so are 10 answers per topic without an account. Signing in doubles that to 20, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
DISCUSSION · 0

Nothing here yet. Say how you would answer it.