LLMs have shown remarkable capabilities but are often too large for consumer devices. Smaller models are trained alongside larger ones, or compression techniques are applied to make them more efficient. While compressing models can significantly speed up inference without sacrificing much performance, the effectiveness of smaller models varies across different trust dimensions. Some studies suggest benefits like reduced biases and privacy risks, while others highlight vulnerabilities like attack susceptibility. Assessing compressed models’ trustworthiness is crucial, as current evaluations often focus on limited aspects, leaving uncertainties about their overall reliability and utility.
Researchers from the University of Texas at Austin, Drexel University, MIT, UIUC, Lawrence Livermore National Laboratory, Center for AI Safety, University of California, Berkeley, and the University of Chicago conducted a comprehensive evaluation of three leading LLMs using five state-of-the-art compression techniques across eight dimensions of trustworthiness. Their study revealed that quantization is more effective than pruning in maintaining efficiency and trustworthiness. Moderate bit-range quantization can enhance certain trust dimensions like ethics and fairness, while extreme quantization to very low bit levels poses risks to trustworthiness. Their insights highlight the importance of holistic trustworthiness evaluation alongside utility performance. They offer practical recommendations for achieving high utility, efficiency, and trustworthiness in compressed LLMs, providing valuable insights for future compression endeavors.
Various compression techniques, like quantization and pruning, aim to make LLMs more efficient. Quantization reduces parameter precision, while pruning removes redundant parameters. These methods have seen advancements like Activation Aware Quantization (AWQ) and SparseGPT. While evaluating compressed LLMs typically focuses on performance metrics like perplexity, their trustworthiness across different scenarios still needs to be explored. The study addresses this gap by comprehensively evaluating how compression techniques impact trustworthiness dimensions, which are crucial for deployment.
The study assesses the trustworthiness of three leading LLMs using five advanced compression techniques across eight trustworthiness dimensions. Quantization reduces parameter precision, employing methods like Int8 matrix multiplication and activation-aware quantization. Pruning reduces redundant parameters, utilizing strategies such as magnitude-based and calibration-based pruning. The impact of compression on trustworthiness is evaluated by comparing compressed models with originals, considering different compression rates and sparsity levels. Additionally, the study explores the interplay between compression, trustworthiness, and dimensions like ethics and fairness, providing valuable insights into optimizing LLMs for real-world deployment.
The study thoroughly examined three prominent LLMs using five advanced compression techniques across eight dimensions of trustworthiness. It revealed that quantization is superior to pruning in maintaining efficiency and trustworthiness. While a 4-bit quantized model preserved original trust levels, pruning notably diminished trust, even with 50% sparsity. Moderate bit ranges in quantization unexpectedly bolstered ethics and fairness dimensions, but extreme quantization compromised trustworthiness. The study underscores the complex relationship between compression and trustworthiness, emphasizing the need for comprehensive evaluation.
In conclusion, the study illuminates the trustworthiness of compressed LLMs, revealing the intricate balance between model efficiency and various trustworthiness dimensions. Through a thorough evaluation of state-of-the-art compression techniques, the researchers highlight the potential of quantization to improve specific trustworthiness aspects with minimal trade-offs. By releasing all benchmarked models, they enhance reproducibility and mitigate score variances. Their findings underscore the importance of developing efficient yet ethically robust AI language models, emphasizing ongoing ethical scrutiny and adaptive measures to address challenges like biases and privacy concerns while maximizing societal benefits.
Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.
If you like our work, you will love our newsletter..
Don’t Forget to join our 39k+ ML SubReddit
Sana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.
