Structured representation can characterize semantic objects and relationships in images, providing an effective means for the semantic understanding of Traditional Chinese Paintings (TCPs), which is crucial for archaeology and art history research.
However, existing image-oriented structured representation methods struggle with TCPs due to two major challenges: 1) the substantial differences between objects and events in TCPs and modern natural images lead to semantic misunderstandings; 2) accurate identification of ancient objects and events in TCPs is challenging, even for domain experts.
To address these issues, we propose VisTCP, a visualization framework that integrates a TCP-oriented intelligent model with expert knowledge, enabling art historians to achieve trustworthy structured representations of TCPs in a human-in-the-loop manner.
We begin with a pilot study involving three domain experts to construct a semantic taxonomy of TCPs. Expert-annotated data are then used to train a TCP-oriented structured representation model that can automatically extract meaningful objects and their relationships in TCPs.
To inform users of model uncertainty, we design a joint embedding visualization view to illustrate the differences between expert annotations and model predictions, allowing users to refine the structured representation based on their domain knowledge, thus enabling iterative model optimization.
Finally, we conduct a case study, usage scenario, and expert interviews on a real dataset to demonstrate the effectiveness of VisTCP in supporting the structured representation and semantic understanding of TCPs.
Blogger's Review: By combining domain knowledge with intelligent modeling, VisTCP offers an innovative approach to tackling the complexities of Traditional Chinese Painting, effectively overcoming the limitations of conventional image processing methods. This human-in-the-loop approach not only enhances model accuracy but also enriches the depth and breadth of art historical research. The visualization features provide users with a more intuitive understanding, making it a promising solution for broader applications.