Recent research indicates that state-of-the-art pre-trained code models exhibit impressive performance, yet the internal encoding of type information remains poorly understood.
We probe the residual streams of these models using a parallel dataset of Java and Python code examples to extract internal type representations. Our findings reveal that cross-lingual type representations emerge even from untyped code.
Moreover, we test whether hidden states linearly encode the result type implied by typed function applications by training probes on one language to infer argument and result types in another. Our results show that this structure is partly robust to lexical perturbations and cross-language syntactic variations.
Notably, prior work on code model interpretability has not directly targeted formal type semantics or cross-lingual type representations. We will release our code and datasets.
Blogger's Review: This paper empirically uncovers the encoding of type information in pre-trained code models, addressing a significant gap in type semantics research while showcasing the potential for cross-lingual features. Future exploration into the intersection of code model interpretability and type systems is warranted.