+7Design a distributed web crawlerSystem designCrawl billions of web pages politely and repeatedly, deduplicating URLs and content, to feed a search index or a training-data pipeline.
Last asked Oct 2026
#deduplication
- +7
- +2Design an image sharing service that stores each file onceSystem designDesign a service where users upload, view, download and delete images, and identical images are stored only once, keyed by a content hash. Explain how metadata and blobs stay…Last asked Sep 2026
Deduplicate and order logs from batch and streaming sourcesCodingLog records come both in batches and as a live stream; some are duplicates and some arrive out of order. Produce a single output that has each record once, in timestamp order…Last asked Jul 2026
Design a file backup systemSystem designDesign a system that backs up files from many machines and restores them on demand. You may assume basic file operations exist; focus on how backups work. Cover full versus…Last asked Apr 2026
+1Web-scale crawl cleaning for pretraining dataSystem designDesign a staged batch pipeline that turns raw web crawl into a deduplicated, quality-filtered, decontaminated pretraining corpus on a fixed cluster and deadline. Reported on…Last asked Mar 2026