Playbook / AI system design / Design an AI coding assistant (Copilot-style)

Design an AI coding assistant (Copilot-style)

Expected question

"Design an AI coding assistant (Copilot-style). How do you index repos, serve low-latency completions, respect IDE context, and keep suggestions safe?"

Variant forms

Interviewers often ask the same design with different framing — recognize the archetype:

  • "Design inline code completion with <100ms perceived latency in VS Code."
  • "How do you index a monorepo with 50M lines for semantic codebase search?"
  • "Design fill-in-the-middle (FIM) serving for partial edits in the middle of a file."
  • "Our assistant suggested an API key from another file — architect secret leakage prevention."
  • "Design multi-file agent edits with diff preview and user acceptance."
  • "How do you personalize suggestions per repo style without per-repo full fine-tunes?"
  • "Design telemetry to measure suggestion acceptance without storing full source code."

Where this actually gets asked

Explicitly listed among high-frequency 2026 GenAI system-design questions (Microsoft, OpenAI, Apple prep material): "Design GitHub Copilot," "Design a code completion system," "Design Cursor." The critical Staff+ distinction: inline completion (sub-100–200ms TTFT, FIM, small model) is a different system from chat/agent mode (frontier model, tools, multi-file edits). Pin the mode before drawing boxes — conflating them is the #1 failure mode.

Executive summary

30-second thesis

I'd pin the mode before drawing boxes — inline completion (sub-200ms, FIM, small model) is a different product from chat/agent (frontier, tools, multi-file). Conflating them is the failure mode I've seen most often. Secrets leave the IDE redacted or not at all.

2-minute answer

For inline: fill-in-the-middle with a tight context budget — prefix first, then suffix, open-tab signatures, then a little repo retrieval. Never embed the whole monorepo on the hot path. Debounce, cancel abandoned keystrokes (wasted GPU is a real cost bug), and cascade tiny→larger models. Chat/agent gets its own GPU pool so agent traffic can't blow the inline P99.

Index path is cold and incremental: tree-sitter symbols, dirty queue, embeddings of symbols not whole files. Retrieval and agent tools respect VCS permissions — no reading //secret/ because the assistant "found" it. Multi-file edits land as one reviewable diff stack with atomic reject; don't leave half-applied refactors.

Eval: acceptance and persistence@30m beat HumanEval alone. Enterprise may demand on-device / no-train. License and CWE filters on suggestions. Here's where demos lie: a model that wins on small Python repos can fail on monorepo Java/TS — segment online eval.

What I'd ask them: Inline-only or agent in scope? On-prem / no-train? Monorepo size? Latency budget they actually feel in the IDE?

Requirements

Functional

  • Inline ghost-text completions as the user types (prefix + suffix aware).
  • Optional chat/agent mode: explain code, multi-file edit proposals, run tools in a sandbox.
  • Repo-aware context: current file, open tabs, symbols, similar code retrieval.
  • Privacy: secrets never leave the IDE; enterprise may require on-prem / no-train.

Non-functional

  • Inline TTFT budget typically 50–200ms P99 feel; cancel abandoned requests on keystroke.
  • Index large monorepos without blocking the editor; incremental re-index on edit.
  • Acceptance rate / persistence metrics for product quality, not just HumanEval offline.
  • License and vulnerability filters on suggestions.

Core entities

  • Completion request: cursor position, prefix, suffix, language, debounce_id, cancel_token.
  • Context pack: budgeted snippets (current file, imports, retrieved symbols) + token budget.
  • Repo index: AST symbols (tree-sitter), embeddings, file hashes, incremental dirty set.
  • Suggestion: text, range, model_id, latency_ms, filter_flags.
  • Acceptance event: shown / accepted / persisted_after_N_min (online eval).

API / interface

IDE plugin is the primary client. Separate hot path (complete) from cold path (index/chat).

POST /v1/complete
Authorization: Bearer <device_or_user_token>
{ "language":"python", "prefix":"...", "suffix":"...", "repo_id":"r_...",
  "open_files_meta":[{"path":"a.py","hash":"..."}], "max_tokens":64 }
→ 200 text/event-stream (partial suggestion) | 204 cancelled | 429

POST /v1/index/delta
{ "repo_id":"r_...", "changes":[{"path":"...","content_hash":"...","symbols":[...]}] }
→ 202 { "indexed_through":"..." }

POST /v1/chat
{ "repo_id":"r_...", "messages":[...], "tools_allowed":["read_file","apply_patch"] }
→ stream + optional tool_calls (gated)

POST /v1/telemetry/acceptance
{ "suggestion_id":"s_...", "accepted":true, "persisted_30m":false }
→ 204

Staff+ callout: /complete must be cancelable; wasted GPU on abandoned keystrokes is a cost bug.

Data Flow

Keystroke → debounce → context assemble (local redaction) → complete API → stream suggestion → telemetry. Index path is async and separate.

Rendering architecture diagram…

High-level design

Maps to functional requirements from step 1 — the component architecture that makes the API and data flow real.

Rendering architecture diagram…

Deep dives below target non-functional requirements (latency, scale, failure, cost, security).

Deep dive 1: FIM and context budgeting

Fill-in-the-Middle formats <PRE> prefix <SUF> suffix <MID> so the model sees code after the cursor — critical for inline accuracy. Budget ~2–4k tokens: current prefix (highest), suffix, import/signatures from open tabs, then repo retrieval. Never embed the whole repo on the hot path.

Deep dive 2: latency stack

Debounce + speculative start; cancel in-flight on keystroke; model cascade (tiny model for parens/names, larger for multi-line); prefix KV cache for stable file headers; local LRU of recent completions. Chat/agent must not share the inline GPU pool without isolation.

Deep dive 3: privacy, license, eval

Client-side secret redaction before network. Enterprise: on-device or VPC inference (11). Filter license-contaminated and known-CWE patterns. Online metrics: acceptance rate, persistence@30m — offline HumanEval alone is insufficient.

Deep dive 4: repo trust boundaries and multi-file rollback (Staff+)

Index and agent tools must respect VCS permissions — a user without read access to //secret/ must not retrieve those symbols via the assistant. Treat dependency lockfiles and .env examples as secret-adjacent. For multi-file agent edits: propose a single reviewable diff stack with per-file apply; support atomic reject/rollback of the stack (do not leave half-applied refactors). Supply-chain: suggestions that add dependencies need allowlist / SCA gates before "apply." Segment online eval by language and repo size — a model that wins on small Python repos can fail on monorepo Java/TS. Tie agent tool schemas to release gates (19).

What's expected at each level

  • Mid-level: send current file to an LLM API; show completion.
  • Senior: FIM, basic retrieval, streaming, mentions latency.
  • Staff+: cancelation, cascade, incremental index, privacy redaction, acceptance metrics, permission-aware retrieval, multi-file apply/rollback.
  • Principal: separates inline vs agent products and GPU pools; enterprise no-train contracts; supply-chain gates; eval segmentation that matches real customer repos.

Follow-up questions to expect

  • "How do you index a 10M-LOC monorepo?" — Incremental tree-sitter, embed symbols not whole files, dirty queue. Hot path never waits on a full reindex.
  • "What if the model suggests a secret?" — Output filters, client redaction before network, and default no-train on customer code. Still imperfect — treat it as defense in depth.
  • "Agent mode vs autocomplete?" — Different models, tools, and latency SLOs. Shared indexer only.
  • "Overload on complete?" — 204/429 fast; keep chat/agent off the inline GPU pool so agent fan-out can't trash the 50–200ms feel.
← Design a ChatGPT-style conversational serviceDesign an LLM-powered customer support assistant →