Add a KV cache to GPT inference

You have a small GPT model with plain autoregressive generation that recomputes attention over the whole sequence for every new token. Add a key-value cache so each step only computes the new token's keys, queries and values and reuses the cached ones. Explain what is stored per layer, how the cache grows and is indexed with positions, how much memory it takes, and how much compute it saves.