Scale and observe a production ML serving service

An existing service takes prediction requests, does asynchronous preprocessing and feature lookups, then calls a versioned model. Traffic is bursty, inference is expensive, and when latency or correctness regresses nobody can tell where. The model itself is fixed.

Explain how you would scale it and find its bottlenecks. Cover:

• metrics, traces and logs that follow a request through admission, feature calls, preprocessing, queueing, inference and the response;
• admission, batching, per-tenant limits and deadlines, and what a client gets back when the service is overloaded or a call times out;
• making the async Python code faster without breaking it;
• rolling out a new serving version with canaries, comparison and rollback.