You are required to read and agree to the below before accessing a full-text version of an article in the IDE article repository.

The full-text document you are about to access is subject to national and international copyright laws. In most cases (but not necessarily all) the consequence is that personal use is allowed given that the copyright owner is duly acknowledged and respected. All other use (typically) require an explicit permission (often in writing) by the copyright owner.

For the reports in this repository we specifically note that

  • the use of articles under IEEE copyright is governed by the IEEE copyright policy (available at http://www.ieee.org/web/publications/rights/copyrightpolicy.html)
  • the use of articles under ACM copyright is governed by the ACM copyright policy (available at http://www.acm.org/pubs/copyright_policy/)
  • technical reports and other articles issued by M‰lardalen University is free for personal use. For other use, the explicit consent of the authors is required
  • in other cases, please contact the copyright owner for detailed information

By accepting I agree to acknowledge and respect the rights of the copyright owner of the document I am about to access.

If you are in doubt, feel free to contact webmaster@ide.mdh.se

Using Automated Test Generation to Improve Testing of Embedded PLC Software

Publication Type:

Manuscript


Abstract

Testing is an important activity in engineering of industrial control software. In certain application domains (e.g., railway industry) engineering software requires certification according to safety standards. These standards mandate the use of specification-based testing and recommends the demonstration of some level of code coverage on the developed software units: each test suite that contributes to the demonstration that a specified requirement has indeed been satisfied, when fed to the unit under test, should systematically exercise the code (e.g., covering the branches). Naturally, developers want their test cases to be of high quality: test cases should be cost-effective and good at detecting faults. To support developers in software testing, we propose in this thesis two different techniques for producing test cases using an automated test generation approach called CompleteTest. The first technique relies on code coverage criteria and operates towards the goal of exercising the structure of the software. The second technique instead operates using mutation testing, a well known technique for deriving test cases such that certain faults injected into the program are detected. These techniques work with software written in IEC 61131-3 language, a programming standard for industrial control software, commonly used for Programmable Logic Controllers (PLCs) in the engineering of embedded safety-critical software.Further, this thesis presents the results of a series of studies performed in academia and in industry, evaluating these techniques using real industrial programs and comparing tests created manually by human subjects with tests generated automatically using CompleteTest. We found that automated test generation is efficient in terms of time required to generate tests that satisfy code coverage and scales well for most of the software considered. We found that coverage-directed automated test generation can achieve similar code coverage to manual testing but in a fraction of the time. However, these automatically generated tests do not show better fault detection than manually created tests. The results of this thesis indicate that tests generated automatically for achieving high code coverage might even be slightly worse in terms of fault detection compared to manual tests.

Bibtex

@unpublished{Enoiu4419,
author = {Eduard Paul Enoiu},
title = {Using Automated Test Generation to Improve Testing of Embedded PLC Software},
month = {March},
year = {2016},
url = {http://www.es.mdh.se/publications/4419-}
}