Logs and models in complex engineering production systems

ASML develops photolithography machines that are used to produce microchips for many electronic devices such as  smartphones. These complex machines are viewed as giant black boxes by some engineers, and, as a result, tools to gain insights into their inner workings are needed.

For her PhD thesis, Nan Yang studied how developers at ASML use logs and models to engineer photolithography machines. Logs act as a diary, recording everything that happens inside the machine, and allow developers to identify if anything goes wrong during production. Models provide a graphical representation of system behavior, which developers use for testing and code generation. Through the use logs and models, engineers gain insight into the behavior of these ‘black box’ devices and optimize the production process for greater precision.

Mixed methods

To understand the challenges faced by developers and the tools they need, Yang used a mixed method that combined qualitative and quantitative methods. She collected empirical evidence from multiple sources, including developers, repositories, and experiments. The aim was to gather information about developers' practices with a diverse set of methods and data sources to design better tools and methodologies that would help them use logs and models more effectively.

Through interviews with 39 software developers from five companies, including ASML, Yang identified common challenges in log analysis practices. This suggests the possibility of instigating a joint effort to address these issues.

Additionally, she studied existing literature to determine the gap between the state-of-the-art and the state-of-the-practice. The empirical knowledge from these studies has been used by TNO-ESI researchers to develop a log analysis technique at ASML. The results of their case studies have shown that researchers can bring effective technique for software developers by applying knowledge learned from empirical studies in industry.

Helping high-tech companies

Moreover, Yang also investigated how to help high-tech companies move from code-based engineering to model-based engineering. Specifically, she studied how to create models automatically from logs and how developers create models manually. Based on the empirical evidence gathered, Yang suggested improvements for techniques that can help developers create models in a more efficient and more maintainable way.

This research demonstrated the usefulness of empirical studies in industry and suggested more action research that involves software developers in the iteration of problem identification, solution design, and technique evaluation to help developers work with embedded production systems using logs and models.

Title of PhD thesis: Logs and Models in Engineering Complex Embedded Production Software Systems. Supervisors: Johan Lukkien, Alexander Serebrenik, and Pieter Cuijpers.