Playbook / AI system design / Design enterprise PDF Q&A with page citations and grounding

Design enterprise PDF Q&A with page citations and grounding

Expected question

"Design an enterprise PDF Q&A product: users upload multi-page PDFs and get answers with page-specific citations. How do you stop hallucinated citations and enforce decline when evidence is weak? How do Demo vs Strict principal modes differ?"

Variant forms

  • "Our RAG cites the wrong PDF page — redesign the ingest and citation contract."
  • "Design a compliance PDF assistant that must never invent a page number."
  • "Scanned PDFs have no text layer — what do you return, and what do you refuse?"
  • "Show me how Demo body Principal vs Strict JWT changes ingest and answer trust."
  • "How do you evaluate page-citation accuracy and faithfulness in CI?"
  • "Architect a glass-box UI that jumps to the cited page when the reviewer clicks."

Where this actually gets asked

Principal / Staff+ loops for knowledge platforms, legal/compliance copilots, and "grounded GenAI" products. Interviewers increasingly ask for locator-level citations (page/section), not document titles alone — especially after demos that look polished but cannot prove provenance.

Executive summary

30-second thesis

I'd make page-level citations a hard product invariant — refuse when evidence is thin — not a prompt that says "please cite." If ingest flattened away page boundaries, no citation layer can invent them later. Demo helpfulness and Strict enterprise grounding are different postures on purpose.

2-minute answer

I'd start at ingest: server-side page parse, page-bounded chunks with lineage. Client-side flatten + whitespace collapse is the classic scar — page numbers die before retrieval ever runs. At answer time: ACL retrieve, generate only with verifiable spans, decline on low confidence or failed faithfulness. I'd refuse citation spoofing — attaching citations[0] when the model never emitted [S#] just to pad metrics.

Scanned PDFs with no text layer fail closed (ocr_required) until OCR is an explicit path — pretending OCR is free and perfect is how demos lie. Strict mode verifies JWT and binds ingest tenant to claims so ACL on read isn't bypassed by poisoning the corpus. Eval gates measure page citation accuracy, not "citations non-empty."

What I'd ask them: Demo vs Strict in the panel? OCR in scope? Table-heavy docs? What decline rate they'll accept?

Quantitative trade-offs

DecisionTrade-off and reversal evidenceEvidence class
Strict decline vs answer rateDeclines protect trust; reverse only with labeled low-risk corpora and explicit product acceptance.H
Page cites vs chunk citesPage/span cites audit better; reverse to coarser cites only if parser limits force it and UI discloses uncertainty.H
OCR quality vs latency/costHeavier OCR lifts scanned PDFs; reverse when corpus is born-digital and cost dominates.H

Numbers and thresholds in interview delivery should be labeled H unless the candidate can defend a measured baseline. Open repositories are O; researched public patterns are R. Do not upgrade O/R into employer P adoption.

ML fundamentals

Evaluate faithfulness and citation precision by document type; do not optimize BLEU-like overlap alone.

Migration and rollout

PDF QA fails on citations and layout, not on "we have embeddings."

  1. Dual-run page-aware chunking vs old chunker on a labeled PDF set; compare citation page accuracy.
  2. Enforce grounding/decline in shadow before flipping Strict mode for regulated users.
  3. Canary by corpus (policies first, marketing PDFs later) with faithfulness gates.
  4. Drill deletion of a PDF and confirm citations disappear.
  5. Rollback = previous chunker + index alias. Kill switch = force decline when grounding score dips.

Org ownership and operating model

  • Knowledge platform owns PDF parse/chunk pipeline and citation contracts.
  • Product owns Demo vs Strict mode UX and decline copy.
  • Legal / compliance owns which corpora require Strict grounding.
  • Eval owns page-citation and faithfulness suites.
  • Security owns ACL inheritance from document to chunk.

Requirements

Functional

  • Ingest PDFs with page structure preserved end-to-end.
  • Every answer citation includes page (and ideally chunk/span lineage).
  • Decline when retrieve confidence is low or faithfulness fails.
  • Demo mode may use body Principal; Strict mode verifies JWT (exp) and binds ingest tenant to claims.

Non-functional

  • No silent truncation without warnings.
  • Scanned/empty PDFs fail closed (ocr_required) until OCR is an explicit paid path.
  • Eval gates measure page citation accuracy — not just "citations non-empty."

Core entities

  • Page: (page_number, text) from server-side parse (PyMuPDF).
  • Chunk: page-bounded window with page_start/page_end.
  • Citation: {id, title, uri, page, chunk_id, snippet} — never fabricate citations[0].
  • Principal: Demo = body; Strict = JWT claims for retrieve and ingest.

API / interface

POST /v1/ingest/pdf  (multipart)
→ 200 {"chunks_added":N,"lineage":[{"page_start":2,...}],"pdf":{"page_count":12}}
→ 422 {"detail":[{"code":"ocr_required","message":"..."}]}

POST /v1/answer
→ 200 {"answer":"... [S1]","citations":[{"id":"S1","page":2,...}],"grounded":true,"declined":false}
→ 200 {"declined":true,"grounded":false,"risk_flags":["declined_low_confidence"|"declined_unfaithful"]}

Data Flow

Rendering architecture diagram…

High-level design

Rendering architecture diagram…

Deep dive 1: why client-side PDF flatten fails

If the browser extracts pages then join("\n\n") and the server collapses whitespace, page numbers are destroyed. Page citation cannot be added later at the citation layer — it is an ingest contract (ADR-0007).

Deep dive 2: citation spoofing

Attaching citations[0] when the model omitted [S#] inflates citation coverage metrics and lies to users. Prefer empty citations + missing_citation / decline.

Deep dive 3: Demo vs Strict

PRODUCTION_STRICT is process env — two services. Strict must validate JWT exp and bind /v1/ingest* tenant to claims (ADR-0009), or ACL on read is bypassable by poisoning the corpus.

Staff+/Principal signal rubric

  • Mid-level: upload, chunk, retrieve, and answer flow with document-level citations.
  • Senior: page field on citations; server PDF parse.
  • Staff+: page-bounded chunking; decline + faithfulness; no citation spoof.
  • Principal: trust chain on ingest; OCR honesty; eval for page accuracy; Demo/Strict dual posture.

Follow-up questions to expect

  • "How do you prove the cited page actually supports the sentence?" — Faithfulness check against the cited span; if it fails, decline. Clicking the viewer to the page is the human audit loop.
  • "What changes for table-heavy PDFs / layout?" — Page text alone struggles; I'd call out layout/ table extraction as a known limitation and refuse confident answers when structure is gone.
  • "How do you panel this in 3 minutes?" — Ingest contract → page cites → decline/faithfulness → Demo vs Strict. Don't drown in embedding bake-offs.
← Design a real-time voice AI assistantDesign enterprise hybrid retrieval with access-aware ranking →