As the availability of image data increases, labeling datasets remains a significant challenge due to high costs and time consumption. Thus, semi-supervised approaches like Graph Convolutional Networks (GCNs), which leverage both labeled and unlabeled data, have emerged as a promising solution. A primary challenge in applying GCNs for image classification is graph construction, as images generally lack predefined structural representations. Most studies construct graphs based on the similarity of feature vectors from pretrained deep learning models, typically using kNN or reciprocal kNN algorithms.
Despite the remarkable capabilities of Large Language Models (LLMs) in capturing high-level semantics, their integration with GCNs for image classification remains underexplored. To address this gap, our approach employs a Vision Language Model (VLM) to generate textual descriptions of images, which are then processed by an LLM to estimate semantic similarity scores between connected images. These scores guide the pruning of edges in kNN and reciprocal kNN graphs, filtering out semantically irrelevant neighbors.
Experimental results reveal that leveraging LLMs for graph refinement can enhance classification accuracy, particularly for kNN graphs and certain backbones. The source code is publicly available at http://gcnllm.lucasvalem.com.
Blogger's Review: This innovative approach of integrating LLMs with GCNs showcases immense potential in the field of image classification. The semantic optimization of graph edges significantly enhances classification performance, indicating a promising future for the convergence of image processing and natural language processing.