RAG Hyperparameter Optimization: Translating a Traditional ML Design Pattern to RAG Applications

In this talk, Niels Bantilan explores how traditional ML design patterns, specifically hyperparameter optimization (HPO), can be applied to Retrieval Augmented Generation (RAG) applications. He discusses the shift in focus from model training to inference in the era of Foundation LLMs and RAG architectures. The presentation recasts HPO for RAG pipelines, demonstrating that established techniques like grid search, random search, and Bayesian optimization remain relevant. The talk emphasizes the importance of a high-quality evaluation dataset for systematically improving RAG performance and covers practical steps for bootstrapping an HPO pipeline, including data synthesis, metric definition, and the use of orchestration tools.