Customer support systems are rapidly evolving with the advent of agentic chatbots, yet they encounter significant limitations in accessing enterprise data without predefined API endpoints. This paper introduces SAFAARI (Schema-Aware Framework for Accelerated Advertiser Response Intelligence), a multi-agent framework that tackles the critical bottleneck of schema linking in Natural Language to SQL (NL-to-SQL) systems through specialized content, metadata, and orchestration agents.
We also present SEAL (Schema Evaluation and Accuracy in Language-to-SQL), a novel composite metric that comprehensively evaluates system performance while penalizing inconsistent results. Through systematic experimentation with five feature set configurations, SAFAARI achieves an 81.66% SEAL score (a 6.65% improvement over the baseline), with notable gains in datapoint accuracy (5.51%) and schema-linking precision (4.69%).
The framework's effectiveness is validated through human-in-the-loop evaluation with domain experts, demonstrating its adaptability across diverse support domains. By automating the labor-intensive process of schema linking and query generation, our framework achieves an 8x reduction in development time while maintaining high accuracy. This solution streamlines API development and enhances self-service capabilities, particularly benefiting customer support enterprises with complex data ecosystems.
Blogger's Review: The SAFAARI framework, with its introduction of the SEAL metric, not only enhances system performance but also significantly reduces development time, showcasing immense potential in handling natural language queries within complex data environments, marking a significant advancement in customer support systems.