Zero-/Few-Shot Anomaly Classification for Transistor Using Multimodal CLIP Retrieval Augmented

Praneenatthavee, Natdhanai; Ploysuwan, Tuchsanai

doi:10.1109/iSAI-NLP64410.2024.10799271

Zero-/Few-Shot Anomaly Classification for Transistor Using Multimodal CLIP Retrieval Augmented

Author(s)

Praneenatthavee, Natdhanai

Ploysuwan, Tuchsanai

Date Issued

January 1, 2024

Type

Conference Paper

DOI

10.1109/iSAI-NLP64410.2024.10799271

Abstract

Detecting anomalies in transistors is a challenging task due to the intricate features distinguishing normal from abnormal components. This paper introduces a zero-shot and few-shot anomaly classification framework for transistors using a retrieval-augmented multimodal approach. We focus on two critical parts of the transistor: the Component BOX and the Metal Legs. Utilizing Florence2 for prompt-based bounding box generation and Segment Anything Model2 (SAM2) for segmentation, we create precise masks for each part. Embeddings generated through Contrastive Language-Image Pre-training (CLIP) are employed to classify each component effectively. For the few-shot learning scenario, we implement Retrieval-Augmented Generation (RAG) to simulate learning from both images and textual data, enhancing the anomaly classification performance. Our zero-shot model achieved an fl score of 70.5, while the few-shot model attained an improved fl score of 77.0, demonstrating the efficacy of our approach in transistor anomaly detection.

Citation

19th International Joint Symposium on Artificial Intelligence and Natural Language Processing Isai Nlp 2024, 2024