Generative UI tools promise to democratize design by transforming natural language descriptions into complete interfaces. These tools not only generate interfaces but also provide user-facing design rationales explaining their layout, accessibility, and design choices. However, it remains unclear if these stated rationales truly reflect in the generated interfaces. We term this disconnect as 'Design Theater': plausible and confident design rationales that bear little relationship to actual implementations.
To study this phenomenon, we introduce a benchmark and three metrics for measuring Design Theater. The benchmark includes 24 UI generation tasks spanning structural, styling, and functional design requirements. Using this benchmark, we evaluated 120 interfaces created by five generative UI tools. On average, over 25\% of user-facing design rationales are not implemented in the generated interface, with the implementation failure rising to 34\% for functional requirements. Tools recognize roughly half of the UX principles embedded in prompts (mean = 0.54), with four out of five tools implementing 6\% or fewer functional principles.
We also measure interface similarity across tools and find convergence in visual appearance and layout organization, with greater variation in color choices. Overall, we contribute: 1) the concept of Design Theater; 2) a benchmark with metrics for assessing whether the stated reasoning of generative UI tools is reflected in their implementations; 3) and findings from a systematic evaluation of these tools. We discuss what these findings mean for the design and evaluation of generative UI tools.
Blogger's Review: The concept of Design Theater highlights the limitations of generative UI tools in practical applications. While these tools theoretically offer convenient design options, the effectiveness and reliability of their implementations need significant improvement to ensure consistency between design rationales and the final interface. Future research should focus on enhancing the functional implementation capabilities of these tools.