Playbook / Model Plane / Model Plane flagship — buy vs RAG vs PEFT vs self-host (ModelForge)

Model Plane flagship — buy vs RAG vs PEFT vs self-host (ModelForge)

Expected question

"You've built a lot of agents — where's your model plane? How do you decide buy vs fine-tune vs self-host, and can you prove you've run GPU training and serving?"

Variant forms

Interviewers often ask the same design with different framing — recognize the archetype:

  • "Are you agents-only, or do you own LLMOps?"
  • "Walk me through your PEFT / vLLM experience — not a diagram, receipts."
  • "When would you use a small local model instead of GPT-4?"
  • "Design the control UI for model triage across RAG, adapters, and serve paths."
  • "How do apps pick models without every team building its own router?"
  • "Show me honest smoke vs CUDA-ready posture — no invented metrics."

Where this actually gets asked

Principal / Staff AI architect loops at companies that separate agent orchestration from model economics — especially CAIO-style grills where "multi-agent demo" is not enough.

Org grounding (O): ModelForge · DomainForge · ADR-034 · panel scripts

Executive summary

30-second thesis

Agents are how work gets done; the model plane is how I decide buy vs RAG vs PEFT vs self-host — with a hire-facing UI (ModelForge), training depth in DomainForge, CUDA serve receipts, SLM bake-off evidence, and a gateway that enforces + records instead of letting every app invent routing.

2-minute answer

I'd open ModelForge posture and say each status aloud — ready vs smoke vs planned. Facts stay in Enterprise RAG (authorization before ranking). Behavior and JSON discipline go in DomainForge (PEFT ladder). ModelForge composes the plane: SLM bake-off for when small models win, PEFT + vLLM receipt gallery for GPU proof, gateway path for enforce+record.

Honesty matters: published CUDA receipts are T4 micro-runs (peft_gpu.json, vllm_cuda.json, cuda=true) — not a fictional always-on 7B production fleet. DomainForge 7B QLoRA+DPO is the deeper panel story when they want training depth; ModelForge is the decision surface.

Close with the decision tree: buy when latency/cost/privacy allow API; RAG when facts change and citations matter; PEFT when schema/format behavior must stick; self-host when residency, batch economics, or adapter serving wins — and show the receipt that matches the claim.

Quantitative trade-offs

DecisionTrade-off and reversal evidenceEvidence class
Buy API vs self-hostBuy wins on time-to-ship and frontier quality; self-host wins on residency, batch $/token, and adapter control — reverse when API price drops or SLM bake-off shows schema tasks win local.H
RAG vs PEFTRAG holds facts; PEFT holds behavior — merging them creates stale weights and citation theater.O
SLM vs frontierSLM wins on cost/latency for structured tasks; frontier wins on open-ended reasoning — bake-off table is public on ModelForge.O
Gateway enforce vs app routersCentral enforce+record beats N team routers; reverse only for research sandboxes without production alias.O

Numbers in delivery should be labeled H unless defended from a measured baseline. ModelForge receipts are O; do not upgrade T4 micro-runs into employer P production SLOs.

Requirements

Functional

  • Posture API: honest ready / smoke / planned per capability (PEFT, vLLM CUDA, SLM, gateway).
  • Receipt gallery with downloadable JSON artifacts and cuda flag integrity.
  • SLM bake-off table with task class, model, latency, cost estimate.
  • Buy / RAG / PEFT / self-host decision card linked to live proof.
  • Compose DomainForge train path + vLLM serve path + gateway route without merging governance planes.

Non-functional

  • Fail closed on posture: never show "GPU trained" without cuda=true in receipt JSON.
  • Label Free-tier cold starts and micro-run scope on every panel path.
  • Single hire-facing flagship — no new model-zoo repos after ModelForge (ADR-034 freeze).

Core entities

  • PostureCapability: name, status (ready | smoke | planned), evidence_href.
  • ReceiptArtifact: type (peft_gpu | vllm_cuda | slm_row | gateway_route), digest, cuda, metrics.
  • DecisionNode: buy | rag | peft | self_host, when_to_use, proof_link.
  • AdapterLineage: base_model, adapter_digest, eval_report_ids (DomainForge).
  • RoutingDecision: app_selection, gateway_enforcement, audit_record_id.

API / interface

GET /api/v1/posture
→ 200 { "peft": "ready", "vllm_cuda": "ready", "slm_bakeoff": "ready", "gateway": "ready" }

GET /receipts/peft_gpu.json
→ 200 { "cuda": true, "device": "Tesla T4", ... }

GET /receipts/vllm_cuda.json
→ 200 { "cuda": true, "ttft_ms": ~122, "tokens_per_sec": ~54, ... }

Staff+ callout: posture is the panel contract — say smoke vs ready before clicking receipts.

Data Flow

Rendering architecture diagram…

Deep dive 1: agents-only rebuttal

Orchestration (VAP) and governance (AegisAI) are separate from weights, serve, and route. ModelForge is the sixth spine layer — peer to GOV/ORCH/RAG/PUB/ADR — not a lab buried in teaching drawers.

Deep dive 2: honest CUDA receipts

peft_smoke on CPU is not a GPU claim. Published receipts must carry cuda=true and device class. T4 fp16 LoRA micro-run is valid proof of operator path; it is not DomainForge 7B S0/S3/S4 ladder or always-on production serve — say that without apology theater.

Deep dive 3: SLM bake-off as architecture evidence

Schema-heavy tasks often win on small local models for cost/latency/privacy. The bake-off table is how I defend "not always GPT-4" with numbers, not vibes.

Deep dive 4: gateway vs second router

Apps select models; aegis-llm-gateway enforces + records via routing contract (ADR-028/029). AegisAI stays tool gateway + HITL — it does not become a duplicate model router.

Staff+/Principal signal rubric

  • Mid-level: names OpenAI API + "we might fine-tune someday."
  • Senior: RAG vs fine-tune split; mentions evals.
  • Staff+: posture honesty, receipt gallery, SLM bake-off, gateway enforce+record, decision tree.
  • Principal: org-wide model plane flagship, ADR-034 freeze discipline, CAIO-ready 30s rebuttal with three links.

Trap answers

TrapAnswer
"Have you trained your own LLM?""I adapt open bases with PEFT; I don't pretrain 7B+ from scratch. Here's the adapter receipt and eval delta."
"Show me vLLM experience""Educational lab for internals; ModelForge Serve receipt for CUDA metrics from upstream vLLM on T4."
"Why not always GPT-4?""SLM bake-off: schema tasks often win on small local models — table is public."
"Isn't this just agents?""Agents call tools; ModelForge is weights, serve, and route — different plane, same org."

Follow-up questions to expect

  • "What's still smoke?" Name planned vs ready from posture — never bluff.
  • "GPU quota story?" GCP spot T4, quota raise, teardown after capture — operator receipt, not fantasy infra.
  • "How does this map to interview playbook?" Point to inference serving (01), LLM gateway (07), registry (mlops-llmops/01), and this Model Plane entry.
PEFT adaptation methods — LoRA, QLoRA, DPO panel grill →