Apache Airflow: Where Data Engineers and ML Engineers Meet

This talk explores how Apache Airflow serves as a unifying platform for data engineers and ML engineers to build and orchestrate generative AI workflows. It delves into the challenges of moving generative AI applications to production, such as data freshness, API changes, and pipeline complexity, and how Airflow addresses these issues. The presentation uses the 'Ask Astro' application as a reference architecture, showcasing best practices for production-quality Generative AI pipelines. It covers concepts like retrieval augmented generation, fine-tuning, and provides guidance on getting started with Airflow locally using the Astro CLI.