Playbook / AI system design / Design enterprise hybrid retrieval with access-aware ranking

Design enterprise hybrid retrieval with access-aware ranking

Expected question

"Design hybrid retrieval for enterprise RAG: BM25 + dense, fused with RRF, then cross-encoder rerank. Where does authorization happen relative to scoring? How do you avoid 'semantic' theater (Jaccard pretending to be embeddings)?"

Variant forms

  • "Our 'hybrid' RAG is lexical only — add dense retrieval without breaking ACLs."
  • "Explain RRF vs weighted score fusion when BM25 and cosine live on different scales."
  • "Design Qdrant (or OpenSearch) so tenant filters run before ANN results are trusted."
  • "When do you use a cross-encoder vs a cheap score-boost reranker?"
  • "Calibrate decline thresholds across BM25/RRF scores vs cross-encoder logits."
  • "How do you eval paraphrase recall vs exact-term recall?"

Where this actually gets asked

Staff+/Principal AI infra and search platform loops. Reviewers who have shipped RAG will probe whether "hybrid" means real dual-signal retrieval or a README badge over token overlap.

Executive summary

30-second thesis

I'd push access filters before any ranking signal — BM25 and dense both run under ACL predicates. Fuse with RRF because the score scales don't match, then cross-encode a bounded N. If "hybrid" is Jaccard dressed up as embeddings, I'll say so.

2-minute answer

Lexical catches exact terms; dense catches paraphrase. Their raw scores aren't commensurate, so I'd fuse by rank (RRF) unless we've earned calibrated weighted fusion offline. Cross-encoder reranks a widened candidate set — larger N helps until P99 dies; I'd reverse when the latency budget fails.

The NFR that kills you is authZ: filter-after-ANN still fetched unauthorized neighbors into memory and can leak via traces. Push tenant predicates into the store query; degrade to lexical-only under load rather than dropping authz. Cache keys include principal, policy, and index versions.

Demo can use local/hash embeddings with honesty; Strict uses real vectors. Zero-vector + scroll-as-search is theater — I'd refuse it. Limitation: fixed 0.65/0.30 fusion weights without slice evidence are a smell I'd call out in the room.

What I'd ask them: Index size and P99 budget? Filter pushdown support in the vector store? Decline threshold ownership? Demo vs Strict embedding posture?

Quantitative trade-offs

DecisionTrade-off and reversal evidenceEvidence class
RRF vs tuned weighted fusionRRF works without score calibration; reverse to learned fusion only with offline proof and owned features.H
Cross-encoder depth vs latencyLarger rerank N improves quality; reverse when P99 budget fails and eval allows smaller N.H
Filter pushdown vs app-side filterPushdown prevents leakage windows; app-side filter is a last resort with audited proofs.O/H

Numbers and thresholds in interview delivery should be labeled H unless the candidate can defend a measured baseline. Open repositories are O; researched public patterns are R. Do not upgrade O/R into employer P adoption.

ML fundamentals

Calibrate retrieval/rerank quality on slices; treat fixed 0.65/0.30 weights without evidence as a smell.

Migration and rollout

Hybrid retrieval migrations break when lexical and dense indexes diverge.

  1. Dual-write BM25 + dense with shared doc ids and ACL predicates.
  2. Shadow RRF + rerank; compare slice metrics vs dense-only baseline.
  3. Prove access filter still runs before fusion — fusion must not reintroduce denied docs.
  4. Canary query classes that need sparse (IDs, exact phrases) separately from semantic queries.
  5. Rollback = previous retriever config. Kill switch = lexical-only degraded mode.

Org ownership and operating model

  • Search / RAG platform owns hybrid fusion, rerankers, and access-aware retrieval invariants.
  • Corpus owners own metadata quality that sparse retrieval depends on.
  • Security owns ACL predicate pushdown requirements.
  • Eval owns sparse-vs-dense slice suites.
  • SRE owns index lag and query p99.

Requirements

Functional

  • Lexical retriever (BM25 with k1/b) and dense retriever (real embeddings).
  • Fuse candidate lists with RRF (or documented alternative).
  • Optional cross-encoder second stage on widened top_k.
  • Access filter before any ranking signal is computed for unauthorized chunks.

Non-functional

  • Dense path must work without paid APIs in Demo (local/hash) and with gateway/OpenAI in Strict/prod.
  • Vector upserts use real dimensions; no zero-vector + scroll-as-search.
  • Telemetry exposes fusion/reranker profile on /health.

Core entities

  • Lexical hit list / Dense hit list → RRF fused ranking.
  • RetrievalHit: chunk + score + reasons (bm25, dense, rrf, cross_encoder).
  • AccessPolicy: tenant, clearance, groups.

API / interface

POST /v1/retrieve
{"query":"termination clause","top_k":10,"principal":{"tenant_id":"acme","groups":["legal"]}}
→ 200 {"hits":[{"chunk_id":"c42","score":0.91,"reasons":["bm25","dense","cross_encoder"]}],"profile":"strict"}
→ 403 {"code":"principal_not_authorized"}

GET /health
→ 200 {"retrieval":{"lexical":"bm25","dense":"qdrant","fusion":"rrf","reranker":"cross_encoder"}}

Data Flow

Rendering architecture diagram…

High-level design

Rendering architecture diagram…

Deep dive 1: access-before-ranking

Filtering after vector search still fetched unauthorized neighbors into memory and can leak via bugs/traces. Push tenant predicates into the store query when possible; always enforce policy before returning hits.

Deep dive 2: RRF vs weighted sums

BM25 magnitudes and cosine similarities are incommensurable. Fixed weights (0.65/0.30) without calibration are a smell. RRF ranks are scale-free.

Deep dive 3: cross-encoder ops

Load once at process start; install weights in the image; do not call build_reranker() per request if that reloads the model. ScoreBoost is Demo-only honesty when CE is absent.

Staff+/Principal signal rubric

  • Mid-level: lexical or vector retrieval with basic top-k ranking.
  • Senior: hybrid two-stage + rerank vocabulary.
  • Staff+: RRF rationale; ACL ordering; real embeddings.
  • Principal: dual Demo/Strict embedding posture; decline calibration; Qdrant filter + no scroll theater.

Follow-up questions to expect

  • "What breaks first at 1B chunks?" — Usually ANN fan-out, filter selectivity, or rerank N under P99 — not "BM25 can't scale" in the abstract. I'd ask for their QPS and corpus shape.
  • "How do you A/B fusion strategies safely?" — Shadow retrieve, offline slice eval, then canary with holdback. Don't flip global fusion on a vibe.
  • "Show live /health.retrieval for the profile under test." — Lexical, dense, fusion, reranker names on health — so "hybrid" is observable, not a README claim.
← Design enterprise PDF Q&A with page citations and groundingDesign identity and access management for AI agents and MCP tools →