Memorization and Generalization in Neural Code Intelligence Models

Are deeplearning models memorizing your data instead of learning patterns? Recent works suggest memorization risk is high with noisy training data. In this exploration, we see the extent of memorization in neural code intelligence models (models that can automatically do certain coding tasks), and provide insights on how memorization may impact the learning behavior of such models.

• Evaluated the impact of adding training noise on the accuracy of Google’s GREAT family of code models for the variable- misuse task.

• Systematically induced input and output variable noise to the Py150 ETH VarMisuse dataset.

---

---

Image by svstudioart on Freepik