Design an ads ranking system with calibrated predictions

Design the system that ranks ads for a user, using predicted click and conversion probabilities together with advertiser bids.

Because the predictions feed an auction, they must be calibrated, not only well ordered. Cover the labels and features, the model, how you measure and fix calibration, delayed conversions, serving latency, and how you evaluate a new model online.