[R] Do Llamas Work in English? On the Latent Language of Multilingual Transformers

**Paper**: [https://arxiv.org/abs/2402.10588](https://arxiv.org/abs/2402.10588)

**Code**: [https://github.com/epfl-dlab/llm-latent-language](https://github.com/epfl-dlab/llm-latent-language)

**Dataset**: [https://huggingface.co/datasets/wendlerc/llm-latent-language](https://huggingface.co/datasets/wendlerc/llm-latent-language)

**Colab links**:

**(1)** [https://colab.research.google.com/drive/1l6qN-hmCV4TbTcRZB5o6rUk\_QPHBZb7K?usp=sharing](https://colab.research.google.com/drive/1l6qN-hmCV4TbTcRZB5o6rUk_QPHBZb7K?usp=sharing)

**(2)** [https://colab.research.google.com/drive/1EhCk3\_CZ\_nSfxxpaDrjTvM-0oHfN9m2n?usp=sharing](https://colab.research.google.com/drive/1EhCk3_CZ_nSfxxpaDrjTvM-0oHfN9m2n?usp=sharing)

**Abstract**:

>We ask whether multilingual language models trained on unbalanced, English-dominated corpora use English as an internal pivot language -- a question of key importance for understanding how language models function and the origins of linguistic bias. Focusing on the Llama-2 family of transformer models, our study uses carefully constructed non-English prompts with a unique correct single-token continuation. From layer to layer, transformers gradually map an input embedding …

abstract bias colab english family function importance key language language models links llama machinelearning multilingual pivot prompts question study token transformer transformer models understanding unique