Evaluation Engineering: Iterative Strategies to Testing Prompts

Evaluation Engineering is a key part of the prompt engineering iteration cycle. This talk will discuss strategies & real-world examples of how teams evaluate their prompts. Backtesting, regression testing, and test-driven prompt engineering will be major themes. Through examples of real team evaluations, this talk will argue that there is no one-sized fits all eval metric. Evals must be developed iteratively with the prompt.