Evaluating and Fine-Tuning Vision-Language Models for Industrial Manufacturing in Low-Data Regimes

Vision-Language Models (VLMs) have demonstrated exceptional zero- and few-shot learning capabilities, as well as strong reasoning skills on multimodal data. These qualities make them ideal candidates for the development of interactive systems in industrial manufacturing, where robots collaborate with human workers to perform tasks requiring precision and adaptability to diverse conditions. In this study, we selected and evaluated four open-source VLMs for their potential to automate the microchip production process in fabrication plants. We assessed the prediction accuracy of each model using their pre-trained parameters. Additionally, we fine-tuned the models to investigate whether their performance could be enhanced by fine-tuning on a domain-specific dataset. This paper presents the methodology used, the experimental results obtained, and a concise discussion of key insights. We also provide general recommendations for applying VLMs in real-world industrial settings, discussing their limitations, and highlighting their potential to enhance efficiency and adaptability in complex manufacturing environments.