← Questions
Single-pass cleaning pipeline for a million crawled web documents
0
I was asked this
No reports yet. Sign in to say where you were asked it.
A take-home: input is a set of Parquet files with about a million documents crawled from public web pages. Write a Python pipeline that reads the input in a single pass and writes out the cleaned documents. Choose and justify the cleaning steps (for example filtering low-quality or non-text pages, normalization and deduplication), keep memory bounded, and report what each step removed.
No official answer yet. Write yours on the right and run it.
Nothing published yet.
Sign in to submit your code; every attempt is kept here.
Write and test your solution herePython in your browser, the question's tests level by level, every submission kept for you.Sign in to start coding