Assessing the Effects of Corrupted Parameters in a Large Language Model: A Case Study of LLAMA 3.2 1B

Abstract

This study explores the effects of parameter corruption in a large language model (LLM) by altering its weights and evaluating performance. Experiments involve corrupting different layers and matrix types, including Self-Attention and Feed-Forward components, with performance assessed using BERT and ROUGE scores. Testing the Llama-3.2-1B-Instruct model was performed on the GLUE-QNLI dataset. Results show that increased corruption leads to greater degradation, with Feed-Forward matrices having a stronger impact especially in the Down matrices. According to the study, later layers are more important for performance than those that come before them. These results shed light on possible future chip implementations of LLM, which may help guide the design of fault-tolerant systems by taking vulnerable parameter placement into account.