Metadata plays a crucial role in dataset search, making LLM-generated metadata a significant form of synthetic content in retrieval systems. We studied six settings for generating metadata for RDF datasets, ranging from simple rewriting to profile-grounded and agentic graph-based generation, and evaluated them for retrieval effectiveness and faithfulness.
Unconstrained metadata rewriting provided the strongest retrieval gains over the original metadata but exhibited the least faithfulness, indicating that search improvements can stem from unsupported semantic expansion. More grounded settings significantly enhanced faithfulness, with profile-grounded rewriting offering the best balance between retrieval effectiveness and grounding.
These findings position synthetic metadata as a system-level IR problem where effectiveness, provenance, and trust must be evaluated together.
Blogger's Review: This paper delves into the complexities of LLM-generated metadata, revealing the delicate balance between improving retrieval effectiveness and maintaining data faithfulness. Notably, the challenge of effectively integrating both aspects in practical applications remains a pressing issue.