A new research paper reported by arXiv cs.CL examines how large language models handle code comprehension when identifier names are altered. The study investigates whether current models place disproportionate weight on lexical cues. To test this, the researchers introduced a framework called Face/Off, which performs semantics-preserving identifier renaming to evaluate models across progressive naming conditions.
The Lexical Superstition in Code Models
In naturally occurring software, variable names and identifiers carry significant statistical information, though that information can sometimes be misleading. As large language models become standard tools for software engineering tasks, understanding their reliance on these surface-level lexical elements rather than deeper structural logic is crucial for reliability. The evaluation found that lexical overemphasis is widespread across the tested models.
Unknowns and Open Questions
The announcement did not specify the exact list of large language models evaluated within the Face/Off framework, nor did it name the specific code-comprehension tasks used in the benchmark tests. Furthermore, the report did not detail the performance variance between different model architectures or sizes when subjected to altered identifier names.
Future evaluations may clarify whether fine-tuning methods can successfully reduce this lexical dependency in upcoming AI systems.
Lead image: Lee Campbell / Pexels
