Design a video generation service on a scarce GPU pool

Users submit text-to-video requests that each need minutes of GPU time, and the GPU pool is limited and changes size. Design the request flow, scheduling and workers, how users follow progress, and above all what happens when workers, GPUs or whole jobs fail partway through.