Striking the Balance: Leveraging Human Intelligence with LLMs for Cost-Effective Annotations

Data annotation is a crucial but often time-consuming and expensive process for enhancing machine learning (ML) model performance. Large Language Models (LLMs) offer a potential solution for automating this process. However, the inherent complexity of data annotation, including unclear instructions and subjective human judgment on ambiguous data, presents significant challenges.

This session features Chris Stephens, Field CTO and Head of AI Solutions at Appen, who will discuss an experiment conducted by Appen. The experiment aimed to evaluate the trade-off between quality and cost when training ML models using LLMs versus human input. The goal was to identify which utterances could be confidently annotated by LLMs and which required human intervention to ensure diverse perspectives and prevent errors from overly generalized models. Chris will share the dataset used, the experimental methodology, and the company's research findings.