Model Plane panel question bank (industry archetypes)
Use with: Model Plane taxonomy · ModelForge · DomainForge · panel scripts
Sourcing honesty: Public prep guides (123ofAI, Interview Coder, ShadeCoder, SharpSkill, CalibreOS, vLLM docs/blog) converge on these archetypes for 2025–2026 Staff+ / LLM engineer loops. Few companies publish verbatim question lists for Principal model-plane roles — treat this as high-frequency practice priority, not a leaked question dump.
Evidence classes: O = org open proof · R = researched industry pattern · H = hypothetical numbers unless you defend a baseline · P = employer production (do not claim without primary records)
A. Model plane & strategy (CAIO / Principal)
| # | Question (as asked) | What they probe | Strong answer sketch | Proof |
|---|---|---|---|---|
| A1 | "You're agents-only — where's your model plane?" | Orchestration vs weights/serve/economics | Agents call tools; model plane decides buy/RAG/PEFT/self-host with receipts — ModelForge posture + ADR-034 | O: ModelForge posture |
| A2 | "When do you fine-tune vs RAG vs prompt?" | Decision tree before GPU spend | Prompt first; RAG for changing facts + cites; PEFT for stable schema/behavior; buy when frontier reasoning wins | O: ADR-019 DomainForge split |
| A3 | "Why not stuff SOPs into fine-tune weights?" | Fact/behavior separation | Stale policies + hallucinated chunk_ids; RAG holds facts, PEFT holds JSON discipline | O: DomainForge S0→S4 |
| A4 | "Show me you've run GPU training — not a Colab screenshot." | CUDA receipt integrity | peft_gpu.json with cuda=true; say T4 micro honesty; DomainForge 7B is depth path | O: peft_gpu.json |
| A5 | "When is self-host cheaper than API?" | Cost/latency/residency math | SLM bake-off for schema tasks; vLLM receipt for tok/s; gateway FinOps — label H unless measured | O + H |
| A6 | "Design model routing for 50 tenant adapters." | Multi-tenant serve economics | One base + multi-LoRA vs N replicas; max_loras, eviction, sticky routing | R: vLLM multi-LoRA docs |
| A7 | "How do apps pick models without N routers?" | Federated plane | Apps select; gateway enforces + records (ADR-028/029) | O: aegis-llm-gateway |
Deep entry: 01 ModelForge flagship
B. LoRA · QLoRA · DPO · full fine-tune
| # | Question | What they probe | Strong answer sketch | Proof |
|---|---|---|---|---|
| B1 | "Explain LoRA — why is it cheap?" | Math + intuition | Freeze W₀; train low-rank ΔW=BA; rank r ≪ d; ~100–1000× fewer trainable params | R |
| B2 | "LoRA vs QLoRA?" | Memory vs quality | QLoRA: 4-bit NF4 base + fp16/bf16 adapters; fits 65–70B on single 48–80GB GPU; slight train overhead | R |
| B3 | "What are rank and alpha?" | Hyperparameter literacy | r = expressiveness; alpha scales adapter contribution; diminishing returns past r≈16–64 for many tasks | R |
| B4 | "When does LoRA lose to full fine-tune?" | Honest limits | Heavy domain shift / tasks needing large weight updates; often within 1–2 pts on standard tasks | R |
| B5 | "Why DPO after SFT?" | Alignment stage | SFT teaches schema; DPO fixes plausible-wrong outputs via preference pairs | O: DomainForge S4 |
| B6 | "Can fine-tuning inject new facts?" | RAG vs PEFT trap | No — use RAG; PEFT for format/intent/action codes, not memorizing policies | O: ADR-019 |
| B7 | "Merge adapter vs serve dynamically?" | Deploy trade-off | Merge = zero adapter overhead; multi-LoRA = fleet economics, eviction risk | R + O: Path B edu |
Deep entry: 02 PEFT adaptation panel grill
C. Multi-LoRA · vLLM · inference serving
| # | Question | What they probe | Strong answer sketch | Proof |
|---|---|---|---|---|
| C1 | "Explain PagedAttention / KV cache." | Inference fundamentals | Fixed-size blocks; dynamic alloc; cuts fragmentation; enables prefix cache | R: vLLM paper/blog |
| C2 | "Continuous batching vs static?" | Throughput | Iteration-level scheduling; new requests join as others finish | R |
| C3 | "How does multi-LoRA serving work in vLLM?" | Fleet economics | Shared base weights + per-request adapter; --enable-lora, --max-loras | R: vLLM LoRA docs |
| C4 | "Failure mode when max_loras exceeded?" | Production scar | Adapter eviction/reload latency; fix: working-set sizing, sticky routing, shard adapters | R |
| C5 | "TTFT vs throughput — what do you optimize?" | SLO literacy | Interactive: TTFT; batch/offline: throughput; cite receipt metrics honestly | O: vllm_cuda.json |
| C6 | "Path A CUDA vs Path B educational?" | Honesty under grill | Path B = mock adapter swap; Path A = upstream vLLM CUDA metrics — don't conflate | O: ADR-022 |
Deep entry: 03 Multi-LoRA serving panel · ai-system-design/01 inference serving
D. Task types — classification · regression · JSON · eval
| # | Question | What they probe | Strong answer sketch | Proof |
|---|---|---|---|---|
| D1 | "Is your triage model 'classification'?" | Task taxonomy precision | LLM JSON intent classification (~27 intents) — not sklearn tabular softmax | O: DomainForge |
| D2 | "Design tabular fraud classification at scale." | Classical ML lane | Feature store + registry + batch/online parity + champion/challenger | R: mlops-llmops/04 |
| D3 | "Regression vs classification in MLOps?" | Classical vs LLM vocab | Tabular regression = numeric target + skew monitoring; eval regression = fixture CI gates | O: golden-eval |
| D4 | "How do you score format vs faithfulness?" | Separate eval dimensions | schema_pass vs citation faithfulness — don't lump into one accuracy | O: DomainForge eval |
| D5 | "When does an SLM beat GPT-4?" | Bake-off discipline | Structured/schema tasks; cost/latency/privacy; public SLM table | O: ModelForge bake-off |
Deep entry: 04 Task types & classical ML crossover
E. Classical MLOps crossover (Staff AI Platform)
| # | Question | What they probe | Strong answer sketch | Proof |
|---|---|---|---|---|
| E1 | "Design a model registry with promotion gates." | Registry authority | Immutable digests; stage/alias; eval evidence required; rollback = retarget alias | R: mlops-llmops/01 |
| E2 | "What is training–serving skew?" | Feature parity | Same feature name, different computation offline vs online; feature store fixes | R: mlops-llmops/02 |
| E3 | "Data drift vs concept drift?" | Monitoring literacy | Input distribution shift vs P(y | x) change; PSI common for data drift |
| E4 | "When do you retrain after drift?" | Gated promote | Shadow/challenger beat champion + no contract regression + approver for risk class | R |
| E5 | "How does LLM plane connect to classical MLOps?" | Stack integration | Adapters in registry; gateway metering; golden-eval as CI gate bridging both | O |
Deep entries: mlops-llmops/01–05
F. Trap questions (say these cleanly)
| Trap | Wrong answer | Right answer |
|---|---|---|
| "Have you trained your own LLM?" | "Yes, we fine-tune 7B from scratch" | "I adapt open bases with PEFT; I don't pretrain foundations — here's adapter receipt + eval Δ" |
| "Is Path B multi-LoRA production?" | "Yes, we serve adapters on GPU" | "Educational OpenAI-shaped swap — CUDA Path A is the metrics receipt" |
| "Put policies in QLoRA weights" | "Fine-tune on SOP PDFs" | "RAG for facts; PEFT for JSON discipline only" |
| "Eval regression = linear regression?" | Confuse terms | "Fixture/metric regression in CI — different from tabular regression models" |
| "Higher LoRA rank always better?" | "Use r=256 everywhere" | "Diminishing returns + VRAM; task-dependent; defend with ablation" |
Rehearsal order (45 min panel prep)
- Open venkat-ai.com/model-plane — walk Adaptation + Task tabs (5 min)
- Cold-open ModelForge posture — say ready vs smoke (2 min)
- Drill A1, B2, C3, D1, E2 aloud (15 min)
- Close with buy/RAG/PEFT/self-host tree + three links (3 min)
- Optional: Practice Arena rubric for linked playbook entries