Evaluation and Optimization of LLM and RAG Components for a Post-Operative Oral Surgery Consultation Chatbot

Abstract

The increasing demand for dental services highlights the need for efficient post-operative oral surgery consultations. Many patients experience anxiety due to limited knowledge of oral care and treatment. This study introduces a chatbot prototype integrating Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs) to provide accurate, context-aware responses. The research evaluates various LLMs, embedding models, and chunking techniques to enhance chatbot performance. The multilingual-e5-large embedding model excelled in retrieval tasks due to its multilingual training, instruction tuning, and contrastive pre-training, ensuring high retrieval precision. The Hybrid Chunking method was selected for its ability to segment text contextually, combining Markdown-based, token-based, and semantic segmentation for optimal chunk relevance. The Llama3.3 (70B) model was chosen for its superior fluency, relevance, and ability to handle complex dependencies. The results demonstrate that combining the multilingual-e5-large embedding model, Hybrid Chunking technique, and Llama3.3 (70B) model improves retrieval precision, response accuracy, and relevance, enhancing patient care and operational effectiveness of dental staffs.