COMPARING THE PERFORMANCE OF QUESTION ANSWERING BY LLMS USING QUANTIZATION AND RETRIEVAL AUGMENTED GENERATION TECHNIQUES - Netisopakul, Ponrudee; Haruethaipree, Sira (My University, 2025)
The development of large language models (LLMs) like ChatGPT and Google Bard has led to the creation of intelligent chatbots and question-answering systems that are gaining widespread popularity. However, there are still limitations in using LLMs to develop applications, including the substantial computational resources required for fine-tuning and deployment. This paper studies and experiments with two techniques to reduce the computing resources required for developing a question-answering system using LLMs. A quantization technique is employed to compress the model’s size, and the application of Retrieval Augmented Generation (RAG) techniques is utilized for information retrieval. The study compares the performance of compressed-size models using Quantization and RAG against the original-sized models. The results show that quantizing the model can compress the VRAM resources used in the GPU between 38% to 57% while still achieving 68.9% accuracies compared to 70% in the non-compress model.
Keywords: Document Question Answering (DQA); Large language models; Quantization; Resource-constrained machine; Retrieval Augmented Generation (RAG)