Design a warm pool of GPU containers

On a serverless GPU platform a new container takes about 30 seconds to be ready. Each workload names its image, GPU type, region and tenant, and maybe a model to load. Design a pool of containers kept warm ahead of demand so most calls skip the cold start, without paying for lots of idle GPUs.

Cover:

• predicting demand and sizing the pool per image and GPU type
• handing one healthy container to exactly one call, with tenants kept apart
• noticing crashed, leaked or stale leases and reclaiming or destroying the container
• shrinking idle capacity, and falling back to a cold start on a miss