Using Automated Test Generation to Improve Testing of Embedded PLC Software

Testing is an important activity in engineering of industrial control software. In certain application domains (e.g., railway industry) engineering software requires certification according to safety standards. These standards mandate the use of specification-based testing and recommends the demonstration of some level of code coverage on the developed software units: each test suite that contributes to the demonstration that a specified requirement has indeed been satisfied, when fed to the unit under test, should systematically exercise the code (e.g., covering the branches). Naturally, developers want their test cases to be of high quality: test cases should be cost-effective and good at detecting faults. To support developers in software testing, we propose in this thesis two different techniques for producing test cases using an automated test generation approach called CompleteTest. The first technique relies on code coverage criteria and operates towards the goal of exercising the structure of the software. The second technique instead operates using mutation testing, a well known technique for deriving test cases such that certain faults injected into the program are detected. These techniques work with software written in IEC 61131-3 language, a programming standard for industrial control software, commonly used for Programmable Logic Controllers (PLCs) in the engineering of embedded safety-critical software.Further, this thesis presents the results of a series of studies performed in academia and in industry, evaluating these techniques using real industrial programs and comparing tests created manually by human subjects with tests generated automatically using CompleteTest. We found that automated test generation is efficient in terms of time required to generate tests that satisfy code coverage and scales well for most of the software considered. We found that coverage-directed automated test generation can achieve similar code coverage to manual testing but in a fraction of the time. However, these automatically generated tests do not show better fault detection than manually created tests. The results of this thesis indicate that tests generated automatically for achieving high code coverage might even be slightly worse in terms of fault detection compared to manual tests.