Concurrent same-host web crawler

Implement a web crawler that finds every page reachable from a start URL on the same host, and fetches pages concurrently.

You are given a blocking helper:

def get_urls(url: str) -> list[str]:
    """Download the page at `url` and return the absolute URLs it links to.
    Slow (network I/O) and may raise on failure."""

Write:

def crawl(start_url: str, get_urls, max_workers: int = 8) -> list[str]:
    ...

Rules

• Start at start_url, and include it in the result.
• Follow and return only URLs whose hostname is exactly the hostname of start_url. http://docs.shop.io is a different host from http://shop.io.
• Ignore the #fragment part of a URL: http://shop.io/cart#top and http://shop.io/cart are the same page.
• Never fetch the same page twice, even when several threads find it at the same moment.
• If get_urls raises for a page, keep the page in the result and treat it as having no links.
• Return the unique URLs in sorted order. The function must return once every reachable page has been fetched.

Start with a single-threaded version, then make it concurrent with at most max_workers fetches in flight. Explain how your crawler knows it is finished.

Constraints: up to 10,000 reachable pages; each fetch takes 10 ms to 2 s; links can form cycles.

Example 1

http://shop.io/       -> [http://shop.io/cart, http://shop.io/help, http://docs.shop.io/]
http://shop.io/cart   -> [http://shop.io/, http://shop.io/help#returns]
http://shop.io/help   -> [http://other.com/]

crawl("http://shop.io/", get_urls)
-> ["http://shop.io/", "http://shop.io/cart", "http://shop.io/help"]

Each of the three pages is fetched exactly once.

Example 2

If http://shop.io/ has no links, crawl("http://shop.io/", get_urls) returns ["http://shop.io/"].