Sensors are valuable not only for perceiving the world but also for deciding the next steps. Existing sensor models largely stop at perception, merely recognizing states or forecasting outcomes, while actions are handled separately through task‑specific and often closed label spaces. To bridge this gap we introduce the Sensor-Language-Action (SLA) framework, which unifies multimodal sensor observations, natural language, and actions within a single model. SLA treats language as a semantic interface between sensing and acting, enabling heterogeneous actions to be represented, predicted, and explained while staying grounded in the underlying sensor evidence.
We assemble a large‑scale SLA benchmark that spans more than 116,000 individuals, 79 sensor modalities, and 60 action groups. A multi‑faceted captioning pipeline aligns user context, sensor dynamics, and action evidence. Building on this benchmark, we present OpenSLA, a unified model capable of hierarchical action prediction, state understanding, and action explanation. Extensive experiments on real‑world tasks—including clinical prediction, operating‑room monitoring, and metabolic health—show that OpenSLA consistently outperforms state‑of‑the‑art baselines across accuracy, recall, and other metrics.
OpenSLA also demonstrates intriguing abilities such as language‑guided evidence grounding and zero‑shot generalization to unseen actions and cohorts, highlighting the power of language in cross‑modal reasoning.
Review