The evolution of e-commerce has fundamentally transformed how users search for products, shifting from simple text-based keyword queries to complex multimodal interactions that seamlessly combine product images, natural language descriptions, and mixed-intent instructions.
However, existing approaches face a critical dilemma: single-modal specialist models, deployed independently for text retrieval, visual search, and voice recognition, operate in isolation and cannot handle cross-modal queries, while general-purpose vision-language models lack the domain-specific knowledge necessary for fine-grained product understanding, user behavior modeling, and commercial intent reasoning.
In this work, we present Pailitao-MMSearch, a native e-commerce multimodal search foundation model designed to bridge this gap. Our approach introduces three key innovations:
- HybSID (Hybrid Semantic ID);
- A two-stage continual pre-training strategy;
- A hybrid reasoning post-training pipeline.
Built upon Qwen and deployed on Taobao's Pailitao multimodal search platform, Pailitao-MMSearch achieves substantial improvements in online A/B testing, including up to +13.61% in Gross Merchandise Volume (GMV) and +8.21% in transaction volume compared to traditional multimodal search pipeline, demonstrating the effectiveness of our native e-commerce multimodal search large language models.
Blogger's Review: The introduction of Pailitao-MMSearch not only showcases technological innovation in the e-commerce search domain but also emphasizes the importance of cross-modal models in real-world applications. By integrating various data forms, e-commerce platforms can better understand user needs, thereby enhancing conversion rates and user satisfaction.
Looking ahead, optimizing the model for more diverse search scenarios will be a key area of research.