Multi-LoRA inference serving — vLLM panel design
Expected question
"Design serving for dozens of fine-tuned LoRA adapters on one base model — how does vLLM do it and when does it break?"
Variant forms
- "Explain PagedAttention and KV cache."
- "Continuous batching vs static batching?"
- "Multi-LoRA vs spinning up N model replicas?"
- "What happens when max_loras is exceeded?"
- "TTFT vs throughput — what do you optimize for?"
- "How do you prove vLLM experience?" (common Principal trap)
Where this actually gets asked
LLM inference / ML platform loops — well-documented archetype in vLLM docs, CalibreOS MLSD serving guides, and 2025–2026 vLLM interview prep material. Production multi-LoRA at scale is increasingly discussed in vLLM/AWS blog posts (2026).
Org grounding (O): vllm_cuda.json · vLLM Lab Path B · ADR-022
Executive summary
30-second thesis
One base replica + N small adapters beats N full replicas on GPU dollars. vLLM shares base weights and KV infrastructure; adapters swap per request with --enable-lora. The failure mode is adapter eviction when the working set exceeds max_loras — fix with sizing, sticky routing, or sharding.
2-minute answer
Clarifying questions first: single model or multi-tenant LoRA? Interactive or batch? That changes scheduling and memory.
Core mechanics: PagedAttention blocks KV cache like OS paging — cuts fragmentation, enables prefix caching for shared system prompts. Continuous batching adds requests at iteration boundaries instead of waiting for the whole batch to finish — higher GPU utilization.
Multi-LoRA: request carries adapter id; base forward + low-rank side paths. Flags: --enable-lora, --max-loras, --max-lora-rank (pre-alloc buffers — don't oversize). CPU pool for cold adapters optional.
Org honesty: CUDA metrics live in ModelForge vllm_cuda.json — rented L4 serving Mistral-7B-Instruct-v0.3 (vllm-20260903T225117Z, 13.74 tok/s, TTFT p50 371.67ms). That is a dated single-request serve receipt, not a multi-LoRA fleet. Path B in vLLM Lab is educational adapter registry — not CUDA multi-LoRA kernels. Say both aloud in a panel.
What I'd refuse: silent wrong-adapter after eviction, or claiming we production-serve a lab Path B. Clock: lock routing + eviction; skip MoE unless they steer.
Migration and rollout
Wrong-adapter after eviction is worse than a 404.
- Single-adapter serve first. Multi-LoRA only after
max_loras/ rank buffers are sized from memory, not a blog. - Route by adapter id; fail closed if the id is missing or evicted — never silently fall through to base or a neighbor.
- Quality parity on a fixture before an adapter is routable.
- Say the receipt: dated L4 serve is not a multi-LoRA fleet. Path B is a lab registry.
- Rollback = previous replica / merged weights. Kill switch = disable the adapter id.
Org ownership and operating model
- Serving owns routing, eviction, and the fail-closed miss.
- Adapter owners own quality parity before routable.
- Platform owns the registry vs lab Path B honesty.
- SRE owns TTFT when many adapters are hot.
- FinOps owns tok/s → $/token labeled H unless measured.
What I'd ask them
- How many hot adapters vs GPU memory?
- Interactive TTFT SLO?
- Quality parity test before an adapter is routable?
Requirements
Functional
- Route requests to named adapters on shared base.
- OpenAI-compatible API for app integration.
- Dynamic or static adapter registry; runtime load optional.
Non-functional
- TTFT SLO for interactive; throughput for offline.
- Graceful degradation on eviction (no silent wrong adapter).
- Cost-per-token estimable:
(GPU $/hr) / (tok/s × 3600)— label H unless measured.
Deep dive 1: Multi-LoRA vs N replicas
| Approach | GPU memory | Ops complexity | Best when |
|---|---|---|---|
| N replicas | N × base model | Low per variant | Few variants, hard isolation |
| Multi-LoRA | 1 × base + hot adapters | Eviction/routing | Many small adapters, shared base |
Illustrative H: 50 adapters × 7B separate ≈ 50 GPUs; multi-LoRA ≈ 1–few GPUs with 5–15% throughput penalty vs dedicated.
Deep dive 2: Failure modes
- Eviction thrashing — working set > max_loras → reload latency (30–50ms class, H).
- Rank mismatch — max_lora_rank too low → load fail; too high → wasted VRAM.
- Quality parity — multi-LoRA kernels vs merged adapter; need per-adapter eval before promote.
Deep dive 3: MoE + multi-LoRA (2026 frontier)
Compound sparsity: expert routing + adapter selection. vLLM fused_moe_lora kernels — mention as "where the industry is going" without claiming org runs MoE LoRA in prod unless evidenced.
Trap answers
| Trap | Answer |
|---|---|
| "We production-serve Path B" | "Path B is educational; CUDA receipt is upstream vLLM metrics" |
| "PagedAttention is optional optimization" | "It's the memory model that makes continuous batching practical at scale" |
| "Always maximize batch size" | "Interactive TTFT suffers; split queues by SLO tier" |
Follow-up questions to expect
- "What do you skip in 45 minutes?" MoE theater. Cover multi-LoRA vs N replicas, eviction, honesty on receipts.
Staff+/Principal signal rubric
- Mid-level: names LoRA vs a dedicated replica.
- Senior: multi-LoRA memory math vs N copies of the base.
- Staff+: eviction thrashing, rank mismatch, per-adapter quality parity before promote.
- Principal: org adapter registry other teams inherit; fail-closed on silent wrong-adapter; honesty on CUDA receipts vs employer production. Clock: lock routing + eviction; skip MoE theater unless they steer there.