Both configurations fit the weights. Compare cache capacity, collective overhead and replica placement under the actual traffic shape; neither layout wins every workload.
an account raises the per-topic limit · no card
← AI & GPU Infrastructure / 14
expertNewNVIDIADatabricksMeta
You have eight 80GB GPUs and a 70B model. Tensor parallel across all eight, or two replicas of four? Defend the choice.
Both configurations fit the weights. Compare cache capacity, collective overhead and replica placement under the actual traffic shape; neither layout wins every workload.
Updated Sep 2026 · Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.
UP NEXT ON YOUR JOURNEY
Next in this trackOne customer's batch job is making your interactive chat slow. Fix it at the serving layer.Next in this trackYour inference replica takes nine minutes from pod scheduled to first token served. Where does the time go and what do you fix?Next in this trackSomeone proposes serving the quantised model to halve your GPU bill. What has to be true before you ship it?
DISCUSSION · 0
Nothing here yet. Say how you would answer it.