Prompt design with evaluation scaffolding

In a notebook, design prompts for a defined task and write the surrounding scaffolding that evaluates them. The exercise tests task framing and measurement more than clever phrasing.

The task used in the worked answer: route support messages to a team and extract an order id, as JSON.

def parse(text, labels):
    """Strictly parse the model's JSON reply -> (team, order_id); raise ValueError."""


def run(model, render, examples, labels):
    """model(prompt) -> text. One record per example with pred_team, pred_order, error, correct."""


def score(records, labels):
    """accuracy, parse_failures, per_class recall and precision, confusion."""


def compare(a, b, iters=2000, seed=0):
    """Paired bootstrap on per-item correctness -> (mean diff, (low, high))."""


def failures_by_cause(records): ...

Prompt design with evaluation scaffolding

I was asked this

Sign in to see the worked answer

Sign in to see the worked answer