MLOps and Software Engineering Automation Challenges in Production

Abstract

As we work on semiconductor manufacturing lines, integrating AI solutions for manufacturing image analysis can significantly enhance chip productivity and yield rates. However, several challenges must be addressed: #1 Machine Learning Model Limitations - No single model can handle all types of image analysis. #2 Stakeholder Alignment in the AI Model Lifecycle. #3 Global Collaboration and Continuous Operations. To effectively address these challenges, establishing a Python-based generic MLOps framework for model lifecycle management with an automated software engineering process is essential. This framework enables developers to build models consistently by implementing predefined Kubeflow interfaces and deployed to GCP Vertex AI Pipelines and GCP components (eg. streaming Dataflow) via CI/CD, as well as to meet 0 data missing in the PROD env. By following this framework, the model development time can be reduced from weeks to hours, and deployment time can shrink from hours to minutes.

Description