Jean-Guy Schneider

Tuan Dung Lai, Anj Simmons, Scott Barnett et al.

Background: Machine Learning (ML) systems rely on data to make predictions, the systems have many added components compared to traditional software systems such as the data processing pipeline, serving pipeline, and model training. Existing research on software maintenance has studied the issue-reporting needs and resolution process for different types of issues, such as performance and security issues. However, ML systems have specific classes of faults, and reporting ML issues requires domain-specific information. Because of the different characteristics between ML and traditional Software Engineering systems, we do not know to what extent the reporting needs are different, and to what extent these differences impact the issue resolution process. Objective: Our objective is to investigate whether there is a discrepancy in the distribution of resolution time between ML and non-ML issues and whether certain categories of ML issues require a longer time to resolve based on real issue reports in open-source applied ML projects. We further investigate the size of fix of ML issues and non-ML issues. Method: We extract issues reports, pull requests and code files in recent active applied ML projects from Github, and use an automatic approach to filter ML and non-ML issues. We manually label the issues using a known taxonomy of deep learning bugs. We measure the resolution time and size of fix of ML and non-ML issues on a controlled sample and compare the distributions for each category of issue.

Hala Abdelkader, Mohamed Abdelrazek, Scott Barnett et al.

Machine learning (ML), especially with the emergence of large language models (LLMs), has significantly transformed various industries. However, the transition from ML model prototyping to production use within software systems presents several challenges. These challenges primarily revolve around ensuring safety, security, and transparency, subsequently influencing the overall robustness and trustworthiness of ML models. In this paper, we introduce ML-On-Rails, a protocol designed to safeguard ML models, establish a well-defined endpoint interface for different ML tasks, and clear communication between ML providers and ML consumers (software engineers). ML-On-Rails enhances the robustness of ML models via incorporating detection capabilities to identify unique challenges specific to production ML. We evaluated the ML-On-Rails protocol through a real-world case study of the MoveReminder application. Through this evaluation, we emphasize the importance of safeguarding ML models in production.

Jean-Guy Schneider, Peter Mandile, Steve Versteeg

Assuring quality of contemporary software systems is a very challenging task due to the often large complexity of the deployment environments in which they will operate. Service virtualization is an approach to this challenge where services within the deployment environment are emulated by synthesising service response messages from models or by recording and then replaying service interaction messages with the system. Record-and-replay techniques require an approach where (i) message prototypes can be derived from recorded system interactions (i.e. request-response sequences), (ii) a scheme to match incoming request messages against message prototypes, and (iii) the synthesis of response messages based on similarities between incoming messages and the recorded system interactions. Previous approaches in service virtualization have required a multiple sequence alignment (MSA) algorithm as a means of finding common patterns of similarities and differences between messages required by all three steps. In this paper, we present a novel MSA algorithm based on Generalized Suffix Trees (GSTs). We evaluated the accuracy and efficiency of the proposed algorithm against six enterprise service message trace datasets, with the proposed algorithm performing up to 50 times faster than standard MSA approaches. Furthermore, the algorithm has applicability to other domains beyond service virtualization.

Cameron Hine, Jean-Guy Schneider, Jun Han et al.

Constructing testbeds for systems which are interconnected with large networks of other software services is a challenging task. It is particularly difficult to create testbeds facilitating evaluation of the non-functional qualities of a system, such as scalability, that can be expected in production deployments. Software service emulation is an approach for creating such testbeds where service behaviour is defined by emulate-able models executed in an emulation runtime environment. We present (i) a meta-modelling framework supporting emulate-able service modelling (including messages, protocol, behaviour and states), and (ii) Kaluta, an emulation environment able to concurrently execute large numbers (thousands) of service models, providing a testbed which mimics the behaviour and characteristics of large networks of interconnected software services. Experiments show that Kaluta can emulate 10,000 servers using a single physical machine, and is a practical testbed for scalability testing of a real, enterprise-grade identity management suite. The insights gained into the tested enterprise system were used to enhance its design.

Steve Versteeg, Miao Du, Jean-Guy Schneider et al.

Large enterprise software systems make many complex interactions with other services in their environment. Developing and testing for production-like conditions is therefore a very challenging task. Current approaches include emulation of dependent services using either explicit modelling or record-and-replay approaches. Models require deep knowledge of the target services while record-and-replay is limited in accuracy. Both face developmental and scaling issues. We present a new technique that improves the accuracy of record-and-replay approaches, without requiring prior knowledge of the service protocols. The approach uses Multiple Sequence Alignment to derive message prototypes from recorded system interactions and a scheme to match incoming request messages against prototypes to generate response messages. We use a modified Needleman-Wunsch algorithm for distance calculation during message matching. Our approach has shown greater than 99% accuracy for four evaluated enterprise system messaging protocols. The approach has been successfully integrated into the CA Service Virtualization commercial product to complement its existing techniques.

Steve Versteeg, Miao Du, John Bird et al.

Service virtualisation is a supporting tool for DevOps to generate interactive service models of dependency systems on which a system-under-test relies. These service models allow applications under development to be continuously tested against production-like conditions. Generating these virtual service models requires expert knowledge of the service protocol, which may not always be available. However, service models may be generated automatically from network traces. Previous work has used the Needleman-Wunsch algorithm to select a response from the service model to play back for a live request. We propose an extension of the Needleman-Wunsch algorithm, which uses entropy analysis to automatically detect the critical matching fields for selecting a response. Empirical tests against four enterprise protocols demonstrate that entropy weighted matching can improve response accuracy.

Miao Du, Steve Versteeg, Jean-Guy Schneider et al.

Enterprise software systems make complex interactions with other services in their environment. Developing and testing for production-like conditions is therefore a challenging task. Prior approaches include emulations of the dependency services using either explicit modelling or record-and-replay approaches. Models require deep knowledge of the target services while record-and-replay is limited in accuracy. We present a new technique that improves the accuracy of record-and-replay approaches, without requiring prior knowledge of the services. The approach uses multiple sequence alignment to derive message prototypes from recorded system interactions and a scheme to match incoming request messages against message prototypes to generate response messages. We introduce a modified Needleman-Wunsch algorithm for distance calculation during message matching, wildcards in message prototypes for high variability sections, and entropy-based weightings in distance calculations for increased accuracy. Combined, our new approach has shown greater than 99% accuracy for four evaluated enterprise system messaging protocols.