Design context engineering: RAG vs long-context vs memory vs tools

Expected question

"Design how an enterprise agent decides what to put in context — RAG, long-context windows, persistent memory, and tool results. How do you budget tokens, stay fresh, and not leak the wrong tenant's knowledge?"

Variant forms

Interviewers often ask the same design with different framing — recognize the archetype:

"We just bought a 1M-token model. Can we delete the RAG platform?"
"Design context compaction for an agent that runs for hours."
"Prompt caching vs retrieval — when do you pin a corpus in the prefix?"
"The agent keeps forgetting the ticket it already read — memory, RAG, or scratchpad?"
"How do you stop tool dumps from blowing the context and the bill?"
"Design a context policy that another team can reuse — not a per-app prompt pile."
"Long-context RAG: retrieve then stuff 200 pages, or retrieve then cite 8 chunks?"
"What's your default when the model can see the whole SharePoint tenant?"

Where this actually gets asked

This is the 2026 Principal version of "design RAG." Long-context models and prompt caching (Anthropic, Gemini, OpenAI public docs — R) made "just paste the corpus" a real executive question. It is not a sourced "Company X always asks this phrase." Treat it as the control-plane question on top of 02, 20, and 13: retrieval quality is necessary; what is allowed into the window is the Principal problem.

Executive summary

30-second thesis

I'd treat context as a budgeted, authorized working set — not a dump. Access-filtered RAG for the corpus, a tiny durable memory for durable facts, tool results as citations not novels, long-context only when the task is one document the user already has rights to. A million tokens is not an ACL.

2-minute answer

Four stores, four jobs. RAG is "what may I know about the company, right now, under this principal." Session scratchpad is this-run working memory — dies with the turn or checkpoints with the run. Persistent memory is a write-gated, principal-scoped list of facts the user (or policy) agreed to keep. Tool results are evidence with ids, not 40k tokens of JSON left in the prompt forever.

Long-context is a product choice when the unit of work is one file or one thread the caller already opened — a contract PDF, a PR diff — and stuffing beats chunking. It is a terrible multi-tenant search strategy. I'd still filter by ACL before anything enters the window. Prompt cache the stable prefix (policy, tool schemas, maybe a pinned style guide). Don't cache tenant B's chunks under a key that tenant A can hit.

Compaction for long agents: summarize done steps into a checkpoint, keep raw ids so you can re-fetch, never summarize away a side-effect log. If the window is full, retrieve again rather than trust a stale summary as the corpus.

What I'd ask them: Is the unit of work one document or a corpus? What's the token budget per call vs per run? Who owns ACL truth? What's the longest HITL pause?

What I'd refuse: stuffing a tenant-wide corpus into a 1M window and calling it architecture. A million tokens is not an ACL.

Quantitative trade-offs

Decision	Trade-off and reversal evidence	Evidence class
RAG vs stuff-the-window	RAG scales corpora and ACLs; stuffing wins on one already-authorized artifact. Reverse to stuffing when chunking destroys structure (tables, legal clauses) and the principal already has the file.	H
Prompt cache vs re-embed	Cache cuts cost on a stable prefix; reverse if the prefix contains tenant data or changes hourly.	R,H
Persistent memory vs RAG	Memory is a small write-gated fact list; RAG is the corpus. Reverse "everything is memory" after poisoning or RTBF failures.	H
Aggressive compaction vs re-retrieve	Compaction saves tokens on long runs; reverse when summaries drop ids and the agent re-does side effects.	H

Numbers in delivery are H unless you can defend a measured baseline. Do not upgrade a vendor context-window blog into "we don't need retrieval."

Migration and rollout

You're moving from one mega-prompt to a context policy.

Inventory what's in the prompt today: policy, tools, RAG, chat, secrets. Secrets out. Tenant ids from the token, not the text.
Split stores: corpus retrieve with access-before-rank; memory write-gated; tool results referenced by id.
Add a token budget + overflow policy (drop oldest tool blobs first, never drop the deny/policy prefix).
Dual-run: same questions, compare citation faithfulness and leak tests (NT-style: tenant A cannot see B even with a 1M window).
Enable prompt cache only on a tenant-safe, versioned prefix. Kill switch = disable cache + disable stuffing; fall back to RAG+decline.

Org ownership and operating model

AI platform owns the context policy, budget meters, cache keying, and negative leak tests.
Knowledge / RAG owns corpus freshness and ACL-before-rank.
Product owns which workflows may stuff a user-attached file vs must retrieve.
Security owns "long context is not a permission" reviews.
FinOps owns $/run caps when agents compact poorly and re-read the world.

Requirements

Functional

Choose, per request class, whether to retrieve, stuff an authorized artifact, load memory, and/or keep tool traces.
Compact long-running agent context without losing side-effect ids.
Decline when the working set is unauthorized or ungrounded.

Non-functional

Token and $ budget per tenant and per run.
Cache keys include tenant + policy version + model; no cross-tenant prefix hits.
Compaction is reversible via ids (re-fetch beats hallucinated summary-as-source).

Core entities

Context policy — per-workflow rules: retrieve / stuff / memory / tools, budgets, overflow
Working set — the actual tokens sent this call, with source tags (rag | memory | tool | user_file)
Prefix pack — cacheable, tenant-safe (instructions, tool schemas)
Compaction checkpoint — summary + id list for a durable run
Citation / evidence id — pointer back to chunk, memory row, or tool payload store

API / interface

POST /v1/context/plan — workflow, principal, attachments → plan (retrieve queries, stuff yes/no, budget)
POST /v1/context/materialize — execute plan, return working set + citations + degraded flags
POST /v1/runs/{id}/compact — checkpoint a long run
Negative evals as CI fixtures, not a dashboard afterthought

Data Flow

Request → verify principal/tenant → classify workflow → filter corpus/memory/files by ACL → retrieve or stuff → assemble prefix + working set under budget → call model → persist citations and (if any) memory writes through write gates → compact if the run continues.

High-level design

Rendering architecture diagram…

Deep dive 1: a million tokens is not an ACL

Hard constraint: stuffing a tenant-wide corpus into a long window is a leak with extra steps.

If the model can attend over everything the retrieval layer would have been asked to filter, you moved AuthZ into "hope the model ignores Tenant B." I wouldn't. Filter first. Long-context is for an artifact the user already opened — and I'd still strip secrets and other-tenant metadata from that file's properties.

Deep dive 2: compaction without losing the side-effect log

Hard constraint: summarizing "emailed the customer" into "handled outreach" is how you double-send on resume.

Keep a durable side-effect ledger (13). Compaction may drop prose; it may not drop ids. Re-fetch tool payloads from the store if the model needs them again.

Deep dive 3: prompt cache as a tenancy footgun

Hard constraint: cache keys that are only hash(prompt) bleed.

Key by tenant + policy version + model + prefix digest. If the prefix includes retrieved chunks, you probably shouldn't cache it as a shared prefix. Cache the instructions; retrieve the rest.

Staff+/Principal signal rubric

Mid-level: "We'll use RAG" or "we'll use the 1M window."
Senior: RAG + some memory; may not budget tokens or cache keys.
Staff+: access-before-rank, write-gated memory, overflow policy.
Principal: context as a platform policy other teams inherit; long-context vs RAG as a product default with reversal evidence; cache/tenancy; compaction that preserves side-effect ids. Fail-closed: a million tokens is not an ACL. Clock: lock policy + access filter + budget; skip stuffing the whole tenant corpus.

Follow-up questions to expect

"Can't we just upload the whole Confluence space?" Not as a shared window. Retrieve with ACLs. Stuff one page the user has open if structure matters.
"Memory vs RAG?" Memory is small, write-gated, user-visible. RAG is the corpus. Don't store the employee handbook as "memory."
"Agent looping and burning $?" Hard token/$ ceiling at the gateway. Compaction is not a budget. Alerts are not enforcement.
"What do you skip in 45 minutes?" Embedding bake-offs. Cover policy, ACL, budget, one compaction/side-effect rule.

What I'd ask them

One document vs corpus vs multi-hour agent?
Max $ per run they'll actually fail closed on?
Do attachments inherit the same ACL as search?