Patterns, Technologies, and Concepts

• Source & Deep Dive

×

Key Takeaways

Requirements

You demonstrated strong fundamentals in this section, correctly identifying fault tolerance, politeness constraints, and scalability targets without any major gaps.

• When designing a web crawler, always mention robots.txt compliance and crawl-delay directives. This is a politeness constraint that prevents your crawler from overwhelming target servers and getting blocked. Many candidates skip this because it feels like a product detail, but interviewers treat it as a core non-functional requirement.

• For fault tolerance in a long-running crawl job, the key pattern is checkpointing progress so failed jobs can resume rather than restart. Store the frontier (list of URLs to visit) in a persistent queue like Kafka or a database so a crashed worker can pick up where it left off instead of re-crawling from scratch.

• Anchor your scalability discussion with a concrete calculation. For example, 10 billion pages in 5 days means roughly 23,000 pages per second. This number drives decisions about parallelism, queue throughput, and storage sizing, and shows the interviewer you are thinking in real engineering terms rather than vague abstractions.

Core Entities

You demonstrated a strong understanding of the core data entities in a web crawler, correctly identifying URLs and page data as the two key things the system needs to track.

• A web crawler has two core entities to persist: URLs (the queue of links to visit and their crawl status) and page data (the extracted content including raw text and any new URLs discovered). Keeping these separate lets you independently scale the crawl queue from the content storage.

System Interface

You correctly identified both the inputs and outputs of a web crawler system, showing a solid understanding of the basic data flow.

• A web crawler takes seed URLs as input and produces extracted text data as output. Seed URLs are the starting points the crawler uses to discover and traverse the web graph, making them the essential first input to define before designing any other component.

Data Flow

You demonstrated a strong understanding of the web crawler data flow, covering all key stages clearly and correctly on your first attempt.

• A web crawler follows a cycle: seed URLs go into a URL frontier (queue), the crawler fetches each URL (including DNS resolution to get the IP), extracts and stores the page content, then parses out new URLs to feed back into the frontier. This loop repeats continuously.

• DNS resolution is a hidden but important step in the fetch phase. Before making an HTTP request, the crawler must resolve the domain name to an IP address. At scale, caching DNS results locally can significantly reduce latency and external DNS load.

High Level Design

You demonstrated a solid understanding of the core web crawler loop and showed good instincts around horizontal scaling, with only minor gaps in naming subcomponents and defining stopping conditions.

• Always name a Parser or Extractor as an explicit subcomponent in your crawler design, even if it lives inside the crawler worker. This makes the responsibility split clear to interviewers and shows you understand that fetching a page and extracting links from it are two distinct operations.

• A crawl loop needs a stopping condition to feel complete. The two most common ones are crawling until the frontier queue is empty, or stopping when a configured crawl budget is reached (for example, a max number of pages or a time limit). Mentioning this shows you understand the system has defined boundaries and does not run forever.

Deep Dives

You demonstrated strong progression across most topics, quickly incorporating feedback on retry logic, deduplication, and distributed rate limiting, but you had notable gaps around time-based scheduling for recrawls and the distinction between a conditional write and a simple consistent read for race condition prevention.

• A strongly consistent read in DynamoDB does NOT prevent a race condition on its own. Between the read and the write, another worker can insert the same item. The correct fix is a conditional write using ConditionExpression: attribute_not_exists(contentHash), which makes the check and insert atomic. If the condition fails, it means a duplicate was found, so you can skip parsing immediately without any extra lookup.

• SQS visibility timeout controls when a message becomes re-visible AFTER it has already been received by a worker. It is not a scheduling tool for future delivery. For recrawl scheduling where intervals can be days or weeks, use a Redis sorted set where the score is the next crawl timestamp. A lightweight scheduler process runs ZRANGEBYSCORE key 0 <now> LIMIT 0 100 periodically to batch-fetch due URLs and push them into the frontier queue. This separates scheduling from execution cleanly.

• In a distributed crawler, per-domain politeness (1 request per second) only works if all workers share state. Each worker must check a centralized Redis key storing the next allowed timestamp for that domain before fetching. Use a Lua script to make the read-check-and-update atomic, so two workers cannot both read an old timestamp and both decide it is safe to proceed at the same time.

• When handling 429 responses in a crawler, treat them as retriable with respectful backoff, not as permanent failures. A 429 means the server is asking you to slow down. Many servers include a Retry-After header in the 429 response, which you can use to set the exact visibility timeout on the SQS message instead of guessing with a fixed exponential schedule.