← Questions
Migrate 200 TB of media and feed it to training
0
I was asked this
No reports yet. Sign in to say where you were asked it.
Move about 200 terabytes of media files to new storage and build the input pipeline that streams them to model training. Cover how you copy that much data reliably (parallelism, checksums, resuming after failures, cut-over), the format and sharding you store it in for training, and how the training input pipeline reads, decodes and shuffles it fast enough to keep accelerators busy.
No official answer yet. Draw yours on the right; publish it and it shows under Community.
Nothing published yet. Publish your canvas and it shows here for others to open and remix.
Draw your answer hereBoxes, arrows and notes, saved for you to come back to.Sign in to start drawing
