Performance validation of deep-learning-based approach in stool examination

Abstract

BACKGROUND: Human intestinal parasitic infections (IPI) pose a significant global health issue caused by parasitic helminths and protozoa, affecting around 3.5 billion people worldwide, with more than 200,000 deaths annually. Despite advancements in molecular methods with higher sensitivity and specificity, the Kato-Katz or formalin-ethyl acetate centrifugation technique (FECT) remains the gold standard and a routine diagnostic procedure suitable for its simplicity and cost-effectiveness. However, these techniques have limitations that must be addressed. Thus, this study evaluated the performance of a deep-learning-based approach for intestinal parasite identification and compared it with that of human experts. METHODS: Human experts performed FECT and Merthiolate-iodine-formalin (MIF) techniques to serve as ground truth and reference for parasite species. Subsequently, a modified direct smear was conducted to gather images for the training (80%) and testing (20%) datasets. State-of-the-art models, including YOLOv4-tiny, YOLOv7-tiny, YOLOv8-m, ResNet-50, and DINOv2 (base, small, and large), were employed and were operated using in-house CIRA CORE platform. Overall performance was evaluated using confusion matrices, the metrics of which were calculated on the basis of the one-versus-rest and micro-averaging approaches. Moreover, the receiver operating characteristic (ROC) and precision-recall (PR) curves were determined for visual comparison. Lastly, Cohen's Kappa and Bland-Altman analyses were used to statistically measure the significant differences and visualize the association levels between the human experts and the deep learning models' classification performance in intestinal parasite identification. RESULTS: Findings demonstrated the potential of a deep-learning-based approach, particularly of models DINOv2-large (accuracy: 98.93%; precision: 84.52%; sensitivity: 78.00%; specificity: 99.57%; F1 score: 81.13%; AUROC: 0.97) and YOLOv8-m (accuracy: 97.59%; precision: 62.02%; sensitivity: 46.78%; specificity: 99.13%; F1 score: 53.33%; AUROC: 0.755; AUPR: 0.556) for their high metric values in intestinal parasite identification. Class-wise prediction showed high precision, sensitivity, and F1 scores for helminthic eggs and larvae due to more distinct morphology. Moreover, all models obtained a > 0.90 k score, which indicates a strong level of agreement compared with the medical technologists. The Bland-Altman analysis also presented the best agreement between FECT performed by medical technologist A and YOLOv4-tiny, while the MIF technique performed by medical technologist B and DINOv2-small demonstrated the best bias-free agreement, with mean differences of 0.0199 and -0.0080, and standard deviation differences of 0.6012 and 0.5588, respectively. CONCLUSIONS: The results highlight the potential of integrating a deep-learning-based approach into parasite identification. The models showcased superiority in automated detection, suggesting a significant leap toward improving diagnostic procedures for IPI. This hybridization could enhance early detection and diagnosis, facilitating timely and targeted interventions to reduce the burden of IPI through more effective management and prevention strategies.