In highly competitive software markets, user experience (UX) evaluation is crucial for ensuring software quality and fostering long-term product success. Such evaluations typically combine quantitative metrics from standardized questionnaires with qualitative feedback collected through open-ended questions. While open-ended feedback offers valuable insights for improvement, analyzing large volumes of user comments is challenging and time-consuming.
This paper presents techniques developed during a long-term UX measurement project at a major software company to efficiently process and interpret extensive volumes of user comments. We employ a supervised machine learning approach that assigns meaningful, pre-defined topic labels to each comment, providing a high-level overview of the collected comments. Additionally, we demonstrate how generative AI (GenAI) can be leveraged to create concise and informative summaries of user feedback, facilitating effective communication of findings to the organization, especially upper management.
Finally, we investigate whether the sentiment expressed in user comments can serve as an indicator for overall product satisfaction. Our results show that sentiment analysis alone does not reliably reflect user satisfaction. Instead, product satisfaction needs to be assessed explicitly in surveys to measure the user's perception of the product.
Blogger's Review: This paper presents an innovative approach by integrating multi-label classification with generative AI for user feedback analysis, significantly enhancing analysis efficiency in a rapidly evolving software environment. Notably, the limitations of sentiment analysis remind us that quantifying user satisfaction still relies on direct survey feedback.