Abstract
Neural operators have become a common approach for learning PDE solution maps and accelerating numerical simulations. Transformer-based neural operators are of particular interest, as attention can learn long-range dependencies in the computational domain. However, standard attention has two major limitations when applied to PDEs: it scales quadratically with the number of computational nodes and lacks an explicit bias toward local interactions. To address these issues, we introduce the Local Linear Transformer (LLT) for PDE operator learning.
The architecture combines linear global attention with local spatial mixing and incorporates coordinate and geometry information. We evaluate LLT on several PDE problems, including elasticity, plasticity, airfoil flow, pipe flow, and Darcy flow. The reference data for these problems span finite-element, finite-volume, and finite-difference discretizations on structured and unstructured meshes. Compared with other neural operator and transformer baselines from prior studies, LLT achieves competitive or lower relative $L_2$ error across these problems. On matched structured discretizations, wall-clock time per training iteration is reduced by factors of 1.8 to 2.5 relative to Transolver.
We also scale the approach and apply it to a three-dimensional car aerodynamics dataset with 32,186 unstructured mesh points per sample. Together, these results indicate that LLT provides an accurate and computationally efficient operator for PDE problems across discretizations, mesh types, and problem settings.
Blogger's Review: The introduction of LLT effectively addresses the limitations of traditional transformers in the PDE domain by enhancing computational efficiency with a local linear attention mechanism while maintaining accuracy. LLT shows potential for broader applications in more complex physical problems in the future.