Loading...
Benchmarking Next-Generation Frontier Models for Document-Level Arabic-Thai Medical Translation: A Reliability Study of LLM-as-a-Judge
Author(s)
Date Issued
January 1, 2026
Type
Conference Paper
Abstract
This research targets Arabic-to-Thai medical translation, a low-resource and linguistically distant pair critical for public health. We utilize the OpenWHO dataset, a resource comprising 293 documents. This dataset is shielded from webcrawling to minimize training data contamination and ensure rigorous zero-shot evaluation. We conduct a comparative analysis between 2026-era frontier models including GPT-5.2, Gemini 3 Pro, and Claude Opus 4.5 and industry-standard NMT represented by Google Translate. Using an LLM-as-a-Judge framework with an AI jury of efficient reasoning models such as GPT-5.1, Gemini 3 Flash, and Claude Sonnet 4.5, we performed 3,516 evaluations of fidelity, fluency, and cultural appropriateness. Results demonstrate that frontier models outperform traditional NMT across all dimensions. While high exact agreement suggests LLM-as-a-Judge frameworks are promising for scalable evaluation, human validation reveals persistent opportunities for improving AI-human alignment, necessitating targeted oversight for safety-critical applications.
Citation
2026 8th International Conference on Natural Language Processing Icnlp 2026, 162-166, 2026