Playbook / Model Plane / Model Plane flagship — buy vs RAG vs PEFT vs self-host (ModelForge)

Model Plane flagship — buy vs RAG vs PEFT vs self-host (ModelForge)

Expected question

"You've built a lot of agents — where's your model plane? How do you decide buy vs fine-tune vs self-host, and can you prove you've run GPU training and serving?"

Variant forms

Interviewers often ask the same design with different framing — recognize the archetype:

  • "Are you agents-only, or do you own LLMOps?"
  • "Walk me through your PEFT / vLLM experience — not a diagram, receipts."
  • "When would you use a small local model instead of GPT-4?"
  • "Design the control UI for model triage across RAG, adapters, and serve paths."
  • "How do apps pick models without every team building its own router?"
  • "Show me honest smoke vs CUDA-ready posture — no invented metrics."

Where this actually gets asked

Principal / Staff AI architect loops at companies that separate agent orchestration from model economics — especially CAIO-style grills where "multi-agent demo" is not enough.

Org grounding (O): ModelForge · DomainForge · ADR-034 · panel scripts

Executive summary

30-second thesis

I'd decide buy vs RAG vs PEFT vs self-host on constraints — cost, privacy, how fast facts change, whether behavior has to stick — then show one receipt that matches the claim. I will not open with a product tour.

2-minute answer

Say this first: PEFT and vLLM receipts are real GPU runs on a rented L4 serving the pipeline's actual model — Mistral-7B-Instruct-v0.3. PEFT is real QLoRA SFT + DPO (peft-20260904T055541Z). vLLM is vllm-20260903T225117Z, 13.74 tok/s, TTFT p50 371.67ms. Those are training config/timing and a dated serve number — not an adapter win-rate vs RAG, and not a fleet I on-call.

Then the tree. Buy when an API meets latency, cost, and privacy. RAG when the facts move and you need citations — access before rank, or don't bother. PEFT when schema or tone has to stick after the prompt stopped working. Self-host when residency, batch economics, or adapter serving actually wins.

If they want a click: ModelForge posture — ready vs smoke vs planned, said aloud. One O cite. The rest of the stack stays in Related.

What I'd refuse: a “we fine-tuned” slide with no eval slice, or treating a training receipt as a quality score.

Quantitative trade-offs

DecisionTrade-off and reversal evidenceEvidence class
Buy API vs self-hostBuy wins on time-to-ship and frontier quality; self-host wins on residency, batch $/token, and adapter control — reverse when API price drops or SLM bake-off shows schema tasks win local.H
RAG vs PEFTRAG holds facts; PEFT holds behavior — merging them creates stale weights and citation theater.O
SLM vs frontierSLM wins on cost/latency for structured tasks; frontier wins on open-ended reasoning — bake-off table is public on ModelForge.O
Gateway enforce vs app routersCentral enforce+record beats N team routers; reverse only for research sandboxes without production alias.O

Numbers in delivery should be labeled H unless defended from a measured baseline. ModelForge receipts are O; do not upgrade a dated L4 run into an always-on production SLO.

Migration and rollout

The thing you migrate is the default, not a new brand.

  1. Write the tree on one card: Buy / RAG / PEFT / self-host, with the eval that reverses each.
  2. Dual-run the new default on one task class (schema vs facts vs residency). Don't flip the org on a slide.
  3. Posture API stays honest: ready vs smoke vs planned. A training receipt is not a quality score.
  4. Gateway records the route so you can unwind without hunting team routers.
  5. Rollback = previous default. Kill switch = Buy/API or RAG-only; never "we fine-tuned" with no slice.

Org ownership and operating model

  • Model plane owns the decision tree and posture honesty.
  • Eval owns the slice that can reverse a default.
  • Platform / gateway owns the route record.
  • Legal / privacy own self-host vs API when residency is the constraint.
  • Product owns which task class is in scope this quarter.

What I'd ask them

  • Is the job facts (RAG), behavior (PEFT), or control of weights (self-host)?
  • What eval would reverse the default?
  • Honesty: what have we actually trained vs routed?

Requirements

Functional

  • Posture API: honest ready / smoke / planned per capability (PEFT, vLLM CUDA, SLM, gateway).
  • Receipt gallery with downloadable JSON artifacts and cuda flag integrity.
  • SLM bake-off table with task class, model, latency, cost estimate.
  • Buy / RAG / PEFT / self-host decision card linked to live proof.
  • Compose DomainForge train path + vLLM serve path + gateway route without merging governance planes.

Non-functional

  • Fail closed on posture: never show "GPU trained" without cuda=true in receipt JSON.
  • Label Free-tier cold starts and dated-receipt scope on every panel path.
  • Single hire-facing flagship — no new model-zoo repos after ModelForge (ADR-034 freeze).

Core entities

  • PostureCapability: name, status (ready | smoke | planned), evidence_href.
  • ReceiptArtifact: type (peft_gpu | vllm_cuda | slm_row | gateway_route), digest, cuda, metrics.
  • DecisionNode: buy | rag | peft | self_host, when_to_use, proof_link.
  • AdapterLineage: base_model, adapter_digest, eval_report_ids (DomainForge).
  • RoutingDecision: app_selection, gateway_enforcement, audit_record_id.

API / interface

GET /api/v1/posture
→ 200 { "peft": "ready", "vllm_cuda": "ready", "slm_bakeoff": "ready", "gateway": "ready" }

GET /receipts/peft_gpu.json
→ 200 { "cuda": true, "device": "NVIDIA L4", "base_model": "mistralai/Mistral-7B-Instruct-v0.3", ... }

GET /receipts/vllm_cuda.json
→ 200 { "cuda": true, "tok_per_s": 13.74, "ttft_p50_ms": 371.67, ... }

Staff+ callout: posture is the panel contract — say smoke vs ready before clicking receipts.

Data Flow

Rendering architecture diagram…

Deep dive 1: agents-only rebuttal

Orchestration (VAP) and governance (AegisAI) are separate from weights, serve, and route. ModelForge is the sixth spine layer — peer to GOV/ORCH/RAG/PUB/ADR — not a lab buried in teaching drawers.

Deep dive 2: honest CUDA receipts

peft_smoke on CPU is not a GPU claim. Published receipts must carry cuda=true and device class. The live pair is a rented L4 running Mistral-7B — real QLoRA SFT+DPO training and a dated upstream vLLM serve. That is not a quality/win-rate score and not always-on production serve — say that without apology theater.

Deep dive 3: SLM bake-off as architecture evidence

Schema-heavy tasks often win on small local models for cost/latency/privacy. The bake-off table is how I defend "not always GPT-4" with numbers, not vibes.

Deep dive 4: gateway vs second router

Apps select models; aegis-llm-gateway enforces + records via routing contract (ADR-028/029). AegisAI stays tool gateway + HITL — it does not become a duplicate model router.

Staff+/Principal signal rubric

  • Mid-level: names OpenAI API + "we might fine-tune someday."
  • Senior: RAG vs fine-tune split; mentions evals.
  • Staff+: posture honesty, receipt gallery, SLM bake-off, gateway enforce+record, decision tree.
  • Principal: org-wide model-plane default, freeze discipline, 30s CAIO rebuttal with honest O vs P. Who inherits: every team picking a model path. Fail-closed: don't self-host to look sophisticated without a cost/eval packet. Clock: lock the decision tree + reversal evidence; skip a catalog of seventeen demos.

Trap answers

TrapAnswer
"Have you trained your own LLM?""I adapt open bases with PEFT; I don't pretrain 7B+ from scratch. Here's the adapter receipt and eval delta."
"Show me vLLM experience""Educational lab for internals; ModelForge Serve receipt for CUDA metrics from upstream vLLM on a rented L4 serving Mistral-7B (13.74 tok/s)."
"Why not always GPT-4?""SLM bake-off: schema tasks often win on small local models — table is public."
"Isn't this just agents?""Agents call tools; ModelForge is weights, serve, and route — different plane, same org."

Follow-up questions to expect

  • "What do you skip in 45 minutes?" A catalog of demos. Lock the decision tree and one reversal.

  • "What's still smoke?" Name planned vs ready from posture — never bluff.

  • "GPU quota story?" GCP spot L4 (g2-standard-4), teardown after capture — operator receipt, not fantasy infra.

  • "How does this map to interview playbook?" Point to inference serving (01), LLM gateway (07), registry (mlops-llmops/01), and this Model Plane entry.

PEFT adaptation methods — LoRA, QLoRA, DPO panel grill →