Matmul performance modeling on a fixed GPU

Estimate wall-clock time for matrix multiplications of varying shapes on a given accelerator from arithmetic intensity and memory movement, then extend the model to a multi-layer pipeline.

Assume an H100-class GPU: 989 TFLOP/s dense BF16, 3.35 TB/s HBM, 132 SMs, ~4 µs kernel launch.

from dataclasses import dataclass


@dataclass
class GPU:
    peak_flops: float = 989e12
    bandwidth: float = 3.35e12
    sms: int = 132
    launch_s: float = 4e-6
    tile: int = 128


def matmul_time(m, k, n, gpu=GPU(), dtype_bytes=2, waves=False):
    """-> {"flops", "bytes", "intensity", "bound", "seconds"}"""


def layer_time(tokens, d, ffn_mult=4, gpu=GPU()): ...
def pipeline_time(stage_s, stages, microbatches): ...   # -> (seconds, bubble fraction)

Matmul performance modeling on a fixed GPU

I was asked this

Sign in to see the worked answer

Sign in to see the worked answer