Doctoral colloquium - Endre Hamerlik (27.2.2023)

Monday 27.2.2023 at 13:10, Lecture room I/9

---

Endre Hamerlik:
Morphology Spreads Progressively: Evidence from Probing BERTs

Abstract:
Large language models (LLMs) have become increasingly popular in natural language processing due to their impressive performance on a range of tasks. To better understand the internal representations of LLMs, probing techniques are being used, which involve training diagnostic classifiers to predict specific linguistic features based on the LLM's hidden representations.
In this study, we investigate multilingual LLMs' (mBERT's and XLM-RoBERTa's) internal representations using probing techniques and explore the effect of input perturbations on these representations. We also introduce new controls and ablations to evaluate the impact of these perturbations on the diagnostic classifiers' performance. We utilize Shapley values, a model-agnostic approach, to identify the most influential tokens in the input that affect the LLM's internal representations.
Our results indicate that the diagnostic classifiers are highly sensitive to input perturbations, imlpying that LLMs' representations are highly dependent on specific linguistic features like morphology. The analysis of Shapley values provides insight into which input tokens have the greatest impact on the LLM's representations of morphological features. One of the most intriguing findings that emerges is a strong tendency for the preceding context to hold more morphosyntactic information relevant to the prediction than the following context.

---