Playbook / AI system design / Design an AI coding assistant (Copilot-style)

Design an AI coding assistant (Copilot-style)

Expected question

"Design an AI coding assistant (Copilot-style). How do you index repos, serve low-latency completions, respect IDE context, and keep suggestions safe?"

Variant forms

Interviewers often ask the same design with different framing — recognize the archetype:

  • "Design inline code completion with <100ms perceived latency in VS Code."
  • "How do you index a monorepo with 50M lines for semantic codebase search?"
  • "Design fill-in-the-middle (FIM) serving for partial edits in the middle of a file."
  • "Our assistant suggested an API key from another file — architect secret leakage prevention."
  • "Design multi-file agent edits with diff preview and user acceptance."
  • "How do you personalize suggestions per repo style without per-repo full fine-tunes?"
  • "Design telemetry to measure suggestion acceptance without storing full source code."

Where this actually gets asked

Explicitly listed among high-frequency 2026 GenAI system-design questions (Microsoft, OpenAI, Apple prep material): "Design GitHub Copilot," "Design a code completion system," "Design Cursor." The critical Staff+ distinction: inline completion (sub-100–200ms TTFT, FIM, small model) is a different system from chat/agent mode (frontier model, tools, multi-file edits). Pin the mode before drawing boxes — conflating them is the #1 failure mode.

Requirements

Functional

  • Inline ghost-text completions as the user types (prefix + suffix aware).
  • Optional chat/agent mode: explain code, multi-file edit proposals, run tools in a sandbox.
  • Repo-aware context: current file, open tabs, symbols, similar code retrieval.
  • Privacy: secrets never leave the IDE; enterprise may require on-prem / no-train.

Non-functional

  • Inline TTFT budget typically 50–200ms P99 feel; cancel abandoned requests on keystroke.
  • Index large monorepos without blocking the editor; incremental re-index on edit.
  • Acceptance rate / persistence metrics for product quality, not just HumanEval offline.
  • License and vulnerability filters on suggestions.

Core entities

  • Completion request: cursor position, prefix, suffix, language, debounce_id, cancel_token.
  • Context pack: budgeted snippets (current file, imports, retrieved symbols) + token budget.
  • Repo index: AST symbols (tree-sitter), embeddings, file hashes, incremental dirty set.
  • Suggestion: text, range, model_id, latency_ms, filter_flags.
  • Acceptance event: shown / accepted / persisted_after_N_min (online eval).

API / interface

IDE plugin is the primary client. Separate hot path (complete) from cold path (index/chat).

POST /v1/complete
Authorization: Bearer <device_or_user_token>
{ "language":"python", "prefix":"...", "suffix":"...", "repo_id":"r_...",
  "open_files_meta":[{"path":"a.py","hash":"..."}], "max_tokens":64 }
→ 200 text/event-stream (partial suggestion) | 204 cancelled | 429

POST /v1/index/delta
{ "repo_id":"r_...", "changes":[{"path":"...","content_hash":"...","symbols":[...]}] }
→ 202 { "indexed_through":"..." }

POST /v1/chat
{ "repo_id":"r_...", "messages":[...], "tools_allowed":["read_file","apply_patch"] }
→ stream + optional tool_calls (gated)

POST /v1/telemetry/acceptance
{ "suggestion_id":"s_...", "accepted":true, "persisted_30m":false }
→ 204

Staff+ callout: /complete must be cancelable; wasted GPU on abandoned keystrokes is a cost bug.

Data Flow

Keystroke → debounce → context assemble (local redaction) → complete API → stream suggestion → telemetry. Index path is async and separate.

Rendering architecture diagram…

High-level design

Maps to functional requirements from step 1 — the component architecture that makes the API and data flow real.

Rendering architecture diagram…

Deep dives below target non-functional requirements (latency, scale, failure, cost, security).

Deep dive 1: FIM and context budgeting

Fill-in-the-Middle formats <PRE> prefix <SUF> suffix <MID> so the model sees code after the cursor — critical for inline accuracy. Budget ~2–4k tokens: current prefix (highest), suffix, import/signatures from open tabs, then repo retrieval. Never embed the whole repo on the hot path.

Deep dive 2: latency stack

Debounce + speculative start; cancel in-flight on keystroke; model cascade (tiny model for parens/names, larger for multi-line); prefix KV cache for stable file headers; local LRU of recent completions. Chat/agent must not share the inline GPU pool without isolation.

Deep dive 3: privacy, license, eval

Client-side secret redaction before network. Enterprise: on-device or VPC inference (11). Filter license-contaminated and known-CWE patterns. Online metrics: acceptance rate, persistence@30m — offline HumanEval alone is insufficient.

Deep dive 4: repo trust boundaries and multi-file rollback (Staff+)

Index and agent tools must respect VCS permissions — a user without read access to //secret/ must not retrieve those symbols via the assistant. Treat dependency lockfiles and .env examples as secret-adjacent. For multi-file agent edits: propose a single reviewable diff stack with per-file apply; support atomic reject/rollback of the stack (do not leave half-applied refactors). Supply-chain: suggestions that add dependencies need allowlist / SCA gates before "apply." Segment online eval by language and repo size — a model that wins on small Python repos can fail on monorepo Java/TS. Tie agent tool schemas to release gates (19).

What's expected at each level

  • Mid-level: send current file to an LLM API; show completion.
  • Senior: FIM, basic retrieval, streaming, mentions latency.
  • Staff+: cancelation, cascade, incremental index, privacy redaction, acceptance metrics, permission-aware retrieval, multi-file apply/rollback.
  • Principal: separates inline vs agent products and GPU pools; enterprise no-train contracts; supply-chain gates; eval segmentation that matches real customer repos.

Follow-up questions to expect

  • "How do you index a 10M-LOC monorepo?" (Incremental tree-sitter; embed symbols not files; dirty queue.)
  • "What if the model suggests a secret?" (Output filters + never train on customer code by default.)
  • "Agent mode vs autocomplete?" (Different models, tools, latency SLOs — shared indexer only.)
  • "Overload on complete?" (Answer: return 204/429 fast; isolate chat/agent GPU pool so agent traffic never blows inline 50–200ms P99.)
← Design a ChatGPT-style conversational serviceDesign an LLM-powered customer support assistant →