As artificial intelligence and machine learning (AI/ML) components are increasingly integrated into complex software products, development teams struggle with traditional software development workflows. This study analyzes the unique challenges faced by software engineers when building, testing, and deploying ML systems in a massive corporate setting, aiming to establish whether traditional SE tools and practices scale to ML workflows.

The authors conduct a large-scale survey and deep interview study across Microsoft to evaluate how AI/ML changes software engineering roles, skills, and tools. They synthesize their findings into a unified maturity model of AI engineering and identify three primary distinctions between ML and traditional software: (1) managing data is significantly more complex than managing code, (2) model customization requires different developer skills, and (3) testing models requires statistical, multi-dimensional validation rather than binary pass/fail unit testing.

Conducted a multi-phase qualitative and quantitative study involving: (1) semi-structured interviews with 14 Microsoft AI practitioners, (2) a comprehensive survey completed by 551 software engineers, managers, and data scientists across 100+ product teams, and (3) an evaluation of the workflows of teams using AI to pinpoint process bottlenecks and best practices.

• Identified three core characteristics of AI system development that make it distinct from standard software engineering (Data Complexity, Model Customization, and Multi-Dimensional validation).
• Created a nine-stage loop representing the standard operational AI workflow in industry, spanning from data acquisition to deployment and monitoring.
• Provided an empirical baseline of common tools, workflows, and developer skills required to achieve high-maturity AI engineering.

• The study is restricted to Microsoft, which operates at a scale and resource level that may not represent small start-ups or open-source environments.
• Focuses primarily on practitioner opinions and self-reported issues rather than direct observation of software failures in production.

• Explore the creation of a 'Statistical Unit Testing' suite for CI/CD pipelines that flags statistical drifts in classification models before deployment.
• Formulate a specification language that developers can use to define expected behavior boundaries for LLM prompts in application architectures.