Design a Web Crawler: System Design Interview 2026

A web crawler is a continuous, policy-constrained traversal of an unknown graph. The hard part is not extracting links. It is deciding what may be fetched, when it is worth fetching again, and how to recover from duplicate delivery without violating a site's rate limit or exposing the crawler network to SSRF.

This guide applies the six-step system design framework to a general public-web crawler. Search ranking and index serving are separate systems; JavaScript rendering is an optional, tightly isolated tier rather than the default fetch path.

Table of Contents

1. Clarify Scope and Policy
2. Estimate the Workload
3. Model URL and Origin State
4. Build the Crawl Loop
5. Schedule Politely
6. Deduplicate Without Preventing Recrawl
7. Fetch Untrusted URLs Safely
8. Handle Traps, Freshness, and Failure
9. Scale Across Regions
10. Common Interview Mistakes
11. Quick Reference
12. Frequently Asked Questions

Step 1 - Clarify Scope and Policy

Define a target corpus instead of saying “crawl the web.” Useful questions include:

• Are seeds curated, submitted, or discovered from sitemaps and links?
• Which schemes, ports, content types, languages, regions, and maximum object sizes are allowed?
• Is the objective broad discovery, news freshness, archive completeness, or vertical-specific coverage?
• Are authenticated pages, cookies, forms, and JavaScript rendering excluded?
• How are deletion, privacy requests, legal restrictions, and retention handled?
• What identifies the crawler and how can an operator contact or block it?

Core requirements:

• ingest seeds and discovered links;
• honor the Robots Exclusion Protocol and product policy;
• schedule new fetches and adaptive recrawls fairly;
• store response evidence and publish downstream indexing events;
• resist traps and hostile content;
• recover without concurrent duplicate requests breaking politeness.

Robots rules are a request from a service operator, not access authorization. A crawler must not use a permissive robots.txt as permission to bypass authentication, network boundaries, terms, privacy obligations, or explicit block decisions.

Do not promise “never crawl the same URL twice.” Freshness requires revisits, redirects create aliases, and at-least-once work delivery can repeat an attempt. Promise instead to avoid unnecessary duplicate fetches and to make permitted retries safe.

Step 2 - Estimate the Workload

For an interview scenario, target ten billion eligible pages with an average revisit interval of 30 days:

10,000,000,000 ÷ (30 × 86,400) ≈ 3,860 fetches/second

Designing for 10,000 fetches/second is a stated peak assumption. The true rate depends on robots exclusions, changes, errors, redirects, body size, rendering, and the distribution of origin budgets.

If mean end-to-end fetch latency is 500 ms, Little's Law gives roughly 5,000 concurrent in-flight fetches at 10,000/s before headroom. That does not imply a fixed worker count: TLS, DNS, decompression, body parsing, bandwidth, sockets, CPU, per-origin limits, and long-tail latency determine process and host sizing.

At a mean retained response size of 100 KB, ten billion bodies represent about 1 PB logical. Add headers, fetch history, URL/origin state, compressed and uncompressed limits, replicas, versions, indexes, and downstream artifacts. Measure the distribution; averages hide large pages and compressed bombs.

If each page exposes 50 links, the system may process hundreds of billions of discovery candidates per crawl cycle even though far fewer are unique. Candidate parsing, routing, and deduplication can therefore exceed fetch QPS by orders of magnitude.

For a Bloom filter with n fingerprints and target false-positive probability p, size it with:

m = -n × ln(p) ÷ (ln(2)^2) bits

At n = 10 billion and p = 1%, this is about 9.6 bits/item or 12 GB. That is a useful prefilter size, not proof that an exact store must occupy a terabyte or that skipping 1% of pages is acceptable.

Step 3 - Model URL and Origin State

Separate fetch identity, scheduling state, fetch evidence, and content identity.

Parse http and https URLs with a standards-conformant URL parser. A conservative network-fetch key can normalize the parsed scheme and host, remove a default port, resolve dot segments, and omit a fragment because fragments are not sent in an HTTP request. Keep the originally observed form for audit.

Do not blindly sort query parameters, delete “session-looking” keys, lowercase paths, decode reserved characters, or add/remove trailing slashes. Those transformations can change the resource. Site-specific query rules, redirects, sitemap data, and rel=canonical can be useful hints, but they need evidence and versioned policy.

URL deduplication asks “is this fetch identity already represented?” Content deduplication asks “did two responses contain the same bytes or near-duplicate text?” Keep both URL records even when bodies share a content object, because status, provenance, canonical hints, freshness, and access policy can differ.

Step 4 - Build the Crawl Loop

The architecture is a feedback loop with a durable scheduler and explicit safety gates.

flowchart TD
    Seeds[Seeds and sitemap hints] --> Parse[Standards URL parse + policy]
    Parse --> Route[Route by origin owner]
    Route --> URLState[(Exact URL state)]
    URLState --> Frontier[Durable priority + next-fetch frontier]
    Frontier --> Origin[Origin and network scheduler]
    Origin -->|fenced lease + permit| Fetch[Sandboxed fetcher]
    Fetch --> Robots[(Robots and DNS state)]
    Fetch --> Evidence[(Fetch evidence + content)]
    Fetch --> Extract[Bounded link extractor]
    Extract --> Parse
    Evidence --> Outbox[(Indexing / policy outbox)]
    Evidence --> Revisit[Freshness model]
    Revisit --> Frontier

Figure 1. Every discovery passes URL policy and exact state before scheduling. Per-origin/network permits gate sandboxed fetchers; durable evidence and a freshness model feed intentional recrawls back into the frontier.

The frontier stores intent; it is not the only record of progress. Fetch evidence, URL state, origin state, and downstream publication all need durable boundaries. A fetch result and its discovered-link outbox should commit before the lease is acknowledged, or reprocessing must be idempotent by attempt ID.

Step 5 - Schedule Politely

A practical frontier has two concerns:

1. Global utility: choose work by next eligible time, page importance, estimated change probability, discovery value, age, and cost.
2. Origin fairness: enforce concurrency and spacing for each scheme://host:port, plus shared budgets for resolved IP ranges or infrastructure when appropriate.

Strict breadth-first search is not a universal goal. It is useful for broad discovery, but a continuous crawler needs priorities and deadlines. A popular news homepage, a newly submitted seed, and an unchanged archive should not wait in one FIFO.

flowchart LR
    In[Eligible URL intents] --> Priority[Priority and next-fetch queues]
    Priority --> Router[Origin-owner router]
    Router --> O1[Origin A queue]
    Router --> O2[Origin B queue]
    Router --> O3[Origin C queue]
    O1 --> Heap[Ready-origin heap by nextAllowedAt]
    O2 --> Heap
    O3 --> Heap
    Heap --> Permit[Origin + IP permit]
    Permit --> Lease[Fenced fetch lease]
    Lease --> Workers[Fetcher pool]

Figure 2. Priority decides value; the ready-origin heap and permit store decide when a request is polite. Ownership and leases allow a worker pool without equating one origin to one permanent worker.

Use a declared User-Agent and contact information. Fetch /robots.txt for the exact service authority and implement RFC 9309 semantics:

• follow at least five consecutive redirects and apply reached rules in the context of the original authority;
• obey parseable rules after a successful fetch;
• a 4xx “unavailable” result permits access under the RFC, though product policy may be stricter;
• a 5xx or network “unreachable” result requires complete disallow while undefined;
• honor HTTP cache controls and normally do not use a cached file for more than 24 hours unless it is unreachable;
• bound parsing while supporting at least the RFC's 500 KiB minimum.

Crawl-delay is not defined by RFC 9309. A product may support it as a documented compatibility extension, but should still have its own conservative scheduler. Adapt delays to observed latency, timeout, 429/503 responses, Retry-After, operator requests, and repeated connection failures. Use slow recovery after throttling instead of immediately returning to the old rate.

The permit must be held for the attempt and fenced by the current origin-owner epoch. A worker crash cannot allow its late completion to release or overwrite a newer permit. Avoid parallel duplicate fetches of the same URL; they waste resources and can violate politeness, so they are not simply “harmless.”

Step 6 - Deduplicate Without Preventing Recrawl

The exact URL-state record is the correctness layer. A discovery performs a conditional create by fingerprint. If the URL exists, it can update priority or provenance without creating a second active fetch intent. After a successful fetch, the same record receives last_fetch_at, validators, and a new next_fetch_at.

A Bloom filter can reduce exact-store reads:

flowchart TD
    U[Canonical URL fingerprint] --> B{Bloom says absent?}
    B -->|yes| Create[Conditional create exact URL state]
    B -->|no, probably present| Exact{Exact state exists?}
    Exact -->|yes| Merge[Merge priority and provenance]
    Exact -->|no, false positive| Create
    Create --> Add[Add fingerprint to filter]
    Add --> Schedule[Schedule when eligible]
    Merge --> Maybe[Reschedule only by state policy]

Figure 3. The Bloom filter accelerates definite misses; exact conditional state resolves positives and races. This preserves coverage without confusing discovery deduplication with permanent suppression.

An intact insert-only Bloom filter has no mathematical false negatives: “absent” means definitely not inserted. Operationally, a stale replica, lost snapshot, filter rotation, unsynchronized write, or deletion can create false-negative behavior. Rebuild or version filters from the exact store, monitor measured false-positive rate, and never let an unverified optimization become the only record of crawl state.

If the product tolerates skipped discoveries, a Bloom-only mode is a deliberate coverage trade-off. Do not casually say “1% false positives means exactly 1% of pages are missed”: candidates are not uniformly independent, high-value pages have many discovery paths, and false-positive probability changes as capacity fills.

Step 7 - Fetch Untrusted URLs Safely

A crawler is an SSRF engine by design. Isolate fetchers from control-plane services and secrets, give them narrowly scoped egress, and validate every target and redirect:

• allow only configured schemes and ports;
• resolve DNS through controlled resolvers and reject loopback, private, link-local, multicast, documentation, cloud-metadata, and internal ranges according to policy;
• re-check the effective address after DNS refresh and every redirect to resist rebinding;
• bound redirects, detect loops, and preserve the complete chain;
• set connect, header, idle, total, and body-size limits;
• cap decompressed bytes and expansion ratio before parsing;
• disable credentials and cookies unless a separately authorized product requires them;
• sandbox HTML, archive, media, PDF, and JavaScript processing with CPU, memory, recursion, and output limits;
• validate TLS and record, rather than silently bypass, certificate errors.

DNS caching should respect TTL and failure behavior; “cache aggressively” can pin stale or poisoned answers. Per-origin policy is not enough because many names may resolve to one fragile server, while many unrelated sites may share an address. Combine origin fairness with network-level guardrails without letting one noisy shared-hosting tenant starve all others.

HTTP redirects, ETag, Last-Modified, If-None-Match, If-Modified-Since, 304, Retry-After, and status-specific retry policy are part of the fetcher. Retry only idempotent fetch attempts, add jitter, cap attempts, and distinguish permanent policy/input errors from transient network failures.

Step 8 - Handle Traps, Freshness, and Failure

Crawler traps

Use several bounded signals rather than one global maximum depth:

• URL length, path depth, repeated segments, and redirect count;
• query-key/value cardinality and combinations per path template;
• calendar/date expansion and monotonically generated identifiers;
• unique-URL growth versus unique-content or downstream-value yield;
• per-origin, prefix, template, and crawl-session budgets;
• response bytes, parse nodes, extracted links, and render cost.

Throttle or quarantine suspicious patterns and retain samples for review. Blind parameter stripping can merge different products, locales, search results, or pagination; pattern rules must be site-aware, observable, and reversible.

Freshness

Estimate revisit value from historical content changes, page importance, status, validators, trusted sitemap lastmod hints, inbound discovery, fetch cost, and origin budget. Conditional GET can return 304 Not Modified and save response bytes, although it still consumes a request and origin permit. A content hash detects change after bytes arrive; it does not replace validators.

New and high-value pages need exploration even without history. Avoid a feedback loop that only revisits already popular pages. Reserve capacity for seeds, newly discovered origins, stale tail pages, and policy rechecks. A sitemap is a hint, not proof of freshness or permission.

Failure recovery

stateDiagram-v2
    [*] --> ELIGIBLE
    ELIGIBLE --> LEASED: origin permit + fenced lease
    LEASED --> FETCHED: durable result committed
    LEASED --> ELIGIBLE: lease expires, retry policy allows
    FETCHED --> DELAYED: compute next_fetch_at
    DELAYED --> ELIGIBLE: due
    LEASED --> BLOCKED: robots or policy changes
    FETCHED --> BLOCKED: deletion or policy
    BLOCKED --> ELIGIBLE: explicit policy re-admission

Figure 4. URL state permits intentional recrawl while fencing active attempts. Failure returns work only through retry policy and the origin scheduler, not directly to an unconstrained queue.

Worker crashes can cause a repeated GET if the result committed but acknowledgment did not. Attempt IDs make content and outbox writes idempotent. Frontier loss is recovered from authoritative URL state and next-fetch indexes, not from a Bloom snapshot alone. Keep enough fetch evidence to explain policy decisions and distinguish a blank page from a failed parser.

Step 9 - Scale Across Regions

Route each normalized origin to one active scheduler authority. A globally consistent directory assigns an owner epoch; every permit and lease includes that epoch. During handoff, transfer robots state, adaptive limits, queued work, and next_allowed_at, fence the old owner, then activate the new one. If ownership is uncertain, pause that origin rather than letting two regions crawl it concurrently.

New discoveries are sent idempotently to the origin owner, while exact URL-state conditional writes resolve duplicates. Async replication of frontier data alone is not enough to preserve politeness. Network budgets may need a separate authority because many origins share infrastructure.

“Fetch from the nearest region” is not automatically correct. Different vantage points can receive geo-personalized content, different consent pages, or different blocks. Choose stable crawl identities and vantage policy deliberately. Account for data residency, cross-region body transfer, legal restrictions, regional IP reputation, DNS view, and downstream index locality.

Regional failure policy names which work pauses, how owners are reassigned, how stale robots state is treated, and what RPO/RTO applies to fetch evidence and URL scheduling. Test failover against an origin with a one-request-at-a-time limit; aggregate throughput alone will not reveal a double-owner bug.

Common Interview Mistakes

• Saying deduplication means a URL is never fetched twice, which makes recrawl impossible.
• Treating strict BFS as the only correct continuous-crawl policy.
• Claiming Crawl-delay is part of the Robots Exclusion Protocol.
• Ignoring RFC 9309's different behavior for 4xx versus 5xx/network robots failures.
• Using a Bloom filter as the sole source of truth without accepting and measuring skipped coverage.
• Saying false positives are false negatives or that 1% probability equals exactly 1% missed pages.
• Sorting or removing arbitrary query parameters as “normalization.”
• Releasing a crashed worker's URL directly to a global queue without the origin scheduler.
• Calling duplicate fetches harmless when they can violate politeness.
• Omitting DNS rebinding, redirects, decompression bombs, parser isolation, and internal-network protection.
• Assuming domain hashing across regions is sufficient without fenced ownership and handoff state.
• Reporting pages/second while ignoring useful coverage, freshness, response bytes, and per-origin behavior.

Quick Reference

Observe frontier age by priority, eligible versus politeness-delayed work, origin concurrency, robots age/result, 429/503 and Retry-After compliance, lease expiry, duplicate fetches, discovery-to-unique ratio, measured Bloom false positives, canonicalization merges, trap throttles, SSRF blocks, DNS changes, conditional-GET savings, content change rate, useful downstream documents, bytes, parser failures, and cost. Define coverage against a named eligible corpus; “99% of the web” is not measurable.

Frequently Asked Questions

What is a URL frontier in a web crawler?

A URL frontier is a durable scheduler of fetch intents, not a FIFO. It combines page priority and next-fetch time with per-origin and network politeness budgets, then leases eligible work to fetchers. Discoveries and scheduled recrawls return to the same frontier, while ownership epochs prevent two regional schedulers from independently exceeding an origin's limits.

How does a crawler avoid duplicate work but still recrawl pages?

Parse and conservatively canonicalize each URL, fingerprint that fetch identity, and conditionally create or update one exact URL-state record. That record tracks last and next fetch rather than declaring the URL permanently done. Leases, attempt IDs, and idempotent writes bound duplicate work after crashes; redirects and content hashes provide separate URL-alias and content-dedup signals.

Why use a Bloom filter in a web crawler?

A Bloom filter can reject definitely absent fingerprints without reading a large exact store. With about 9.6 bits per item, its theoretical false-positive target is roughly 1%, but a positive means only probably present and should be confirmed when coverage matters. An intact insert-only filter has no false negatives; state loss, lag, deletion, or an incorrect distributed implementation can still violate that operational assumption.

How does a web crawler stay polite to servers?

Identify the crawler, honor RFC 9309 robots rules, cache them within the standard's bounds, and schedule with explicit per-origin plus shared-network concurrency and delay limits. Adapt to latency and errors, and honor Retry-After with 429 or 503 responses. Crawl-delay is not part of RFC 9309; supporting it is an optional compatibility policy, not standards compliance.

What is a crawler trap and how do you handle it?

A crawler trap creates an unbounded or low-value URL space through calendars, query combinations, session tokens, redirects, or generated paths. Use per-origin and per-pattern budgets, depth and length guards, query-cardinality and repetition signals, redirect and response-size limits, and declining-yield detection. Do not blindly sort or remove query parameters because that can merge distinct resources.

How does a crawler keep content fresh?

Store a next-fetch time based on page importance, observed change history, HTTP validators and status, trusted sitemap hints, fetch cost, and origin budget. Revisit with If-None-Match or If-Modified-Since when validators exist, then update the change model from 200 or 304 outcomes. Freshness is a scheduling objective under finite capacity, not uniform monthly BFS.

Sources

• RFC 9309: Robots Exclusion Protocol - robots rules, redirects, status handling, caching, limits, and security scope.
• WHATWG URL Standard - current URL parsing, host, path, query, and percent-encoding behavior.
• RFC 9110: HTTP Semantics - validators, conditional requests, redirects, Retry-After, and status semantics.
• RFC 6585: Additional HTTP Status Codes - 429 Too Many Requests and optional Retry-After.
• Sitemaps protocol - sitemap URL and modification-time hints.

Related Articles

• System Design Interview Problems: A Senior's Roadmap - the complete series index.
• System Design Interview Guide: The 6-Step Framework - the method applied here.
• Design a Notification Service - durable delivery, retries, and idempotency.
• Design a Distributed Cache - cache ownership, failure, and hot-key behavior.
• Design Search Autocomplete - downstream retrieval and ranking concerns.
• Apache Kafka Interview Questions - partitions, consumer groups, and delivery semantics.

This is Part 8 of a 16-part system design series. Next: Design Search Autocomplete.