Multi-LoRA inference serving — vLLM panel design

Expected question

"Design serving for dozens of fine-tuned LoRA adapters on one base model — how does vLLM do it and when does it break?"

Variant forms

"Explain PagedAttention and KV cache."
"Continuous batching vs static batching?"
"Multi-LoRA vs spinning up N model replicas?"
"What happens when max_loras is exceeded?"
"TTFT vs throughput — what do you optimize for?"
"How do you prove vLLM experience?" (common Principal trap)

Where this actually gets asked

LLM inference / ML platform loops — well-documented archetype in vLLM docs, CalibreOS MLSD serving guides, and 2025–2026 vLLM interview prep material. Production multi-LoRA at scale is increasingly discussed in vLLM/AWS blog posts (2026).

Org grounding (O): vllm_cuda.json · vLLM Lab Path B · ADR-022

Executive summary

30-second thesis

One base replica + N small adapters beats N full replicas on GPU dollars. vLLM shares base weights and KV infrastructure; adapters swap per request with --enable-lora. The failure mode is adapter eviction when the working set exceeds max_loras — fix with sizing, sticky routing, or sharding.

2-minute answer

Clarifying questions first: single model or multi-tenant LoRA? Interactive or batch? That changes scheduling and memory.

Core mechanics: PagedAttention blocks KV cache like OS paging — cuts fragmentation, enables prefix caching for shared system prompts. Continuous batching adds requests at iteration boundaries instead of waiting for the whole batch to finish — higher GPU utilization.

Multi-LoRA: request carries adapter id; base forward + low-rank side paths. Flags: --enable-lora, --max-loras, --max-lora-rank (pre-alloc buffers — don't oversize). CPU pool for cold adapters optional.

Org honesty: CUDA metrics live in ModelForge vllm_cuda.json (T4 TinyLlama). Path B in vLLM Lab is educational adapter registry — not CUDA multi-LoRA kernels. Say both aloud in a panel.

Requirements

Functional

Route requests to named adapters on shared base.
OpenAI-compatible API for app integration.
Dynamic or static adapter registry; runtime load optional.

Non-functional

TTFT SLO for interactive; throughput for offline.
Graceful degradation on eviction (no silent wrong adapter).
Cost-per-token estimable: (GPU $/hr) / (tok/s × 3600) — label H unless measured.

Deep dive 1: Multi-LoRA vs N replicas

Approach	GPU memory	Ops complexity	Best when
N replicas	N × base model	Low per variant	Few variants, hard isolation
Multi-LoRA	1 × base + hot adapters	Eviction/routing	Many small adapters, shared base

Illustrative H: 50 adapters × 7B separate ≈ 50 GPUs; multi-LoRA ≈ 1–few GPUs with 5–15% throughput penalty vs dedicated.

Deep dive 2: Failure modes

Eviction thrashing — working set > max_loras → reload latency (30–50ms class, H).
Rank mismatch — max_lora_rank too low → load fail; too high → wasted VRAM.
Quality parity — multi-LoRA kernels vs merged adapter; need per-adapter eval before promote.

Deep dive 3: MoE + multi-LoRA (2026 frontier)

Compound sparsity: expert routing + adapter selection. vLLM fused_moe_lora kernels — mention as "where the industry is going" without claiming org runs MoE LoRA in prod unless evidenced.

Trap answers

Trap	Answer
"We production-serve Path B"	"Path B is educational; CUDA receipt is upstream vLLM metrics"
"PagedAttention is optional optimization"	"It's the memory model that makes continuous batching practical at scale"
"Always maximize batch size"	"Interactive TTFT suffers; split queues by SLO tier"