Design a web crawler

Expected question

"Design a web crawler. How do you discover URLs, fetch politely, deduplicate, store, and scale crawling without overloading target sites?"

Variant forms

Same design, different framing:

• "Design a search-engine crawler for billions of pages with politeness policies."
• "How do you prioritize fresh vs important pages when crawl budget is finite?"
• "Design distributed crawling with URL frontier sharding and dedup."
• "Our crawler got blocked — architect rate limits, robots.txt, and backoff per domain."
• "How do you detect and handle duplicate/near-duplicate content at crawl time?"
• "Design a recrawl scheduler based on page change frequency signals."
• "Architect storing raw HTML vs extracted text for downstream indexing."

Where this actually gets asked

Reported in OpenAI and general Staff system-design loops as "design a polite web crawler" or "crawl the web at scale." Tests frontier queues, politeness, dedupe, and parser pipelines — not AI per se, but often paired with training-data / RAG ingest discussions.

Executive summary

30-second thesis

I'd put politeness and URL canonicalization in front of everything — per-host leases/token buckets, robots cache, leased frontier — because a fast crawler that DDoSes hosts is a failed design.

2-minute answer

Clarify scope (open web vs allowlisted domains), freshness vs coverage, and whether JS render is in budget. Default: seeds → prioritized frontier → host-aware scheduler → fetchers lease URLs with visibility timeout → parse → content_hash dedupe → enqueue outlinks. Canonicalize before insert or you'll storm duplicates. Bloom/URL store blocks recrawl thrash; unchanged hash skips reindex work. Robots: cache with TTL; strict mode fails closed if robots fetch fails. JS-heavy sites get an expensive optional headless tier with hard budgets — not the default path. Poison URLs isolate; max depth / calendar-trap guards. Legal scope is a requirement, not a footnote.

Quantitative trade-offs

What I'd ask them

1. Open web or constrained domain allowlist?
2. Refresh policy — news freshness vs archive completeness?
3. Is crawled data feeding training/RAG (provenance constraints)?

Requirements

Functional

• Given seed URLs, discover and fetch pages continuously.
• Extract links; respect robots.txt and crawl-delay.
• Store raw + parsed content; emit change detection.
• Prioritize important / fresh URLs.

Non-functional

• Politeness: per-host rate limits; no accidental DDoS.
• Scale to billions of URLs; exactly-once-ish fetch semantics via dedupe.
• Fault tolerant workers; poison URL isolation.
• Legal/ToS constraints — stay in allowed domains when scoped.

Core entities

• URL frontier: prioritized queue of URLs to fetch with next_fetch_at.
• Host lease: per-domain token bucket / crawl-delay state.
• Document: content_hash, fetched_at, canonical_url, outlinks[].
• Robots rules: cached per host with TTL.

API / interface

Control-plane APIs for operators; workers pull work.

POST /v1/seeds
{ "urls":["https://example.com"], "scope":"domain" }
→ 202

POST /v1/workers/lease
{ "worker_id":"w1", "max":100 }
→ 200 { "items":[{"url":"...","headers":{...}}] }

POST /v1/workers/complete
{ "url":"...", "status":200, "content_hash":"...", "outlinks":["..."] }
→ 204

GET /v1/stats
→ { "frontier_size":..., "fetch_qps":..., "robots_blocked":... }

Staff+ callout: politeness is a correctness requirement, not an optimization.

Data Flow

Seeds → frontier → host-polite scheduler → fetch → parse → dedupe store → enqueue outlinks.

Rendering architecture diagram…

High-level design

Rendering architecture diagram…

Deep dive 1: frontier and prioritization

Multiple priority queues (news > static). Consistent hashing of hosts to crawlers reduces per-host coordination. Bloom filter / URL store prevents re-crawl storms; content_hash detects unchanged pages.

Deep dive 2: politeness and robots

Cache robots.txt; default deny on fetch failure of robots for strict modes. Per-host token buckets; global QPS caps. Separate political/legal allowlists for enterprise crawlers.

Deep dive 3: parser safety

Sandbox HTML parsers; size limits; no executing JS by default (or isolated headless pool with strict budgets). Connects to training-data provenance when crawl feeds models (../ai-system-design/12).

Deep dive 4: canonicalization and lease-based frontier

Normalize URLs before frontier insert (host, slash, tracking params, redirects, rel=canonical) or you either storm duplicates or miss pages. Workers lease URLs with visibility timeout (same pattern as job queues) so death doesn't lose work or politeness state. In 45 minutes, cover frontier + politeness + dedupe; JS-rendering is an expensive optional tier.

What's expected at each level

• Mid-level: queue of URLs + HTTP GET + store.
• Senior: robots, per-host limits, link extraction.
• Staff+: frontier priority, dedupe, failure isolation, scale partitioning.
• Principal: legal scope, cost of refresh policies, interaction with downstream training/RAG.

Follow-up questions to expect

• "JS-heavy sites?" — Optional headless tier with strict CPU/time budgets. Expensive — don't make it the default fetch path.
• "Crawl traps?" — Max depth, calendar/query-param guards, checksum loops. Assume adversarial HTML.

Related

• 04 Job scheduler
• ../ai-system-design/12 Training data provenance