Abstract
Open-ended surveys offer valuable insights, but they are notoriously difficult to analyze at scale. This study builds on previous work that employed traditional machine learning to classify text, investigating how different large language models (LLMs) understand and analyze NSSE open-ended survey responses. We focus on several cutting-edge LLMs—OpenAI's GPT series, Twitter's roBERTa-base model, and Meta's LLaMA—and compare their performance to previous machine learning models in tasks like sentiment analysis and thematic classification.
Our research analysis assesses model agreement, classification accuracy, and interpretability of reasoning. The findings reveal that current LLMs routinely outperform classic machine learning models in classification accuracy, particularly in understanding complex mood and theme patterns in student replies. While LLMs have superior accuracy, they differ greatly in how explicitly and consistently they justify their predictions and apply category boundaries. These distinctions highlight crucial trade-offs when using LLMs for qualitative analysis: increased predictive strength comes with issues in consistency and explainability.
Our findings illustrate the benefits and drawbacks of utilizing various LLMs for large-scale qualitative research, and we provide practical advice for researchers looking to balance automation and interpretive rigor.
Blogger's Review: This article delves into the strengths and weaknesses of large language models in handling open-ended survey data, particularly the trade-off between accuracy and interpretability. As LLMs become widely adopted, understanding their limitations will help researchers utilize these tools more effectively for data analysis. Notably, finding the optimal balance between automation and human interpretation remains a crucial direction for future research.