Lessons Learned Productionizing LLMs for Stripe Support

In this talk, Sophie Daly from Stripe discusses the lessons learned from productionizing Stripe's first application of Large Language Modelling for their support operations. The presentation covers the challenges of using LLMs for complex business problems, the importance of breaking down problems into manageable steps, prioritizing online feedback and monitoring during development, and the critical role of data quality in achieving scalable and accurate ML solutions. It highlights the need for robust data collection infrastructure, testing, and experimentation, emphasizing that a strong data strategy often outperforms fancy model architectures.