We investigate whether socioeconomic indicators, like household wealth, leave recoverable informational imprints in both satellite imagery (capturing features like buildings and roads) and Internet-sourced text (reflecting historical, cultural, and narratives of neighborhoods). Using DHS data from African neighborhoods (clusters), we pair high-resolution Landsat images with textual descriptions generated by LLMs conditioned on location/year, plus text retrieved by an LLM-driven AI Search Agent from web sources. We develop a multimodal framework that predicts household wealth (International Wealth Index; IWI) via five pipelines:
- A vision model on satellite images
- An LLM using only location and year
- An AI agent that searches and synthesizes web text
- A joint image-text encoder
- An ensemble of all signals
Our framework yields three contributions. First, evaluations show that fusing vision and agent/LLM-generated text improves on vision-only baselines in wealth prediction (e.g., R-squared of 0.77 vs. 0.63 on out-of-sample splits), with LLM-internal knowledge (artificial neural memory) proving surprisingly predictive in out-of-country/time generalization.
Second, we find suggestive evidence of partial representational alignment: fused embeddings from vision and language modalities correlate moderately (median cosine similarity across modalities of about 0.60 after alignment). This pattern is broadly consistent with the Platonic Representation Hypothesis, but does not by itself establish convergence to a single shared latent representation. Because agent-retrieved data yields only marginal and unstable gains across splits, our evidence for the Agent-Induced Novelty Hypothesis is limited.
Third, we release a large-scale multimodal dataset of about 60,000 DHS clusters, each linked to satellite images, LLM-generated descriptions, and AI-agent-retrieved texts.
Blogger's Review: This multimodal framework illustrates how to enhance poverty mapping accuracy by integrating visual and textual information, particularly in resource-constrained areas. The effective amalgamation of diverse data sources offers new insights for socioeconomic analysis, showcasing the potential applications of AI in social sciences.