Knowledge Graphs (KGs) are increasingly constructed through automated extraction pipelines; however, such systems often introduce spurious or incomplete triples, which degrade downstream performance. Existing evaluation practices rely heavily on task-specific metrics or small-scale manual verification, offering limited insight into the structural and semantic fidelity of extracted graphs.
We propose a novel, interpretable metric for intrinsic KG quality assessment that measures how closely an automatically extracted graph approximates an "ideal" graph capturing the key noun phrases, predicate relations, and basic linguistic phenomena such as negation expressed in the source text. Our framework integrates two complementary components:
- An entity-level assessment that evaluates completeness, resolution quality, and connectivity;
- A relation-level assessment that judges predicate preservation and multiplicity using lexical similarity, dependency-parse alignment, and lightweight negation handling to ensure semantic faithfulness.
We evaluate our metric across multiple state-of-the-art triple extraction systems and datasets, including WebNLG, TinyButMighty, and BenchIE, demonstrating that it reliably identifies omissions, redundancy, and structural deviations that existing metrics overlook.
Our work offers a scalable, model-agnostic, and interpretable framework for comparing automated KG construction methods and provides a foundation for standardized evaluation. We further validate the metric through an ablation study isolating noun and verb components, and a downstream evaluation showing that KGCQual scores correlate significantly with link prediction performance on the same extracted KGs. The code repository is available at GitHub.
Blogger's Review: KGCQual introduces a novel perspective on evaluating knowledge graph construction quality through interpretable intrinsic metrics. Its dual assessment mechanism effectively captures the structural and semantic features of graphs, laying a solid foundation for future KG construction and evaluation, with broad application potential.