LLM Evaluation to Craft Delightful Content From Messy Inputs

This talk explores an evaluation framework for the quality of LLM outputs, focusing on transforming diverse and messy textual inputs into delightful content. It goes beyond general LLM evaluation metrics like relevance and fluency to include specific metrics such as overall information preservation rate, accuracy of titles/headings understanding, and key information extraction score. This framework aims to provide measurable metrics that can be generalized to similar LLM tasks, particularly for generating detailed summaries from design inputs based on design types. The talk also touches upon the challenges of objectively evaluating LLM outcomes due to the subjective and unstructured nature of content generation.