You are required to read and agree to the below before accessing a full-text version of an article in the IDE article repository.

The full-text document you are about to access is subject to national and international copyright laws. In most cases (but not necessarily all) the consequence is that personal use is allowed given that the copyright owner is duly acknowledged and respected. All other use (typically) require an explicit permission (often in writing) by the copyright owner.

For the reports in this repository we specifically note that

  • the use of articles under IEEE copyright is governed by the IEEE copyright policy (available at http://www.ieee.org/web/publications/rights/copyrightpolicy.html)
  • the use of articles under ACM copyright is governed by the ACM copyright policy (available at http://www.acm.org/pubs/copyright_policy/)
  • technical reports and other articles issued by M‰lardalen University is free for personal use. For other use, the explicit consent of the authors is required
  • in other cases, please contact the copyright owner for detailed information

By accepting I agree to acknowledge and respect the rights of the copyright owner of the document I am about to access.

If you are in doubt, feel free to contact webmaster@ide.mdh.se

A Controlled Experiment in Testing of Safety-Critical Embedded Software

Fulltext:


Note:

Acceptance rate 35/130=26%, submitted 2015This material is presented for the dissemination of scientific work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In some cases, this work may not be reposted without the explicit permission of the copyright holder.

Publication Type:

Conference/Workshop Paper

Venue:

IEEE International Conference on Software Testing, Verification and Validation


Abstract

In engineering of safety critical systems, regulatory standards often put requirements on both traceable specification-based testing, and structural coverage on program units. Automated test input generation techniques can be used to generate test data to cover the structural aspects of a program. However, there is no conclusive evidence on how automated test input generation compares to manual test generation, or how testing based on the program implementation relates to specification-based testing.In this paper, we investigate specification- and implementation-based testing of embedded software written in the IEC 1131-3 language, a programming standard used in many embedded safety critical software systems. Further, we measure the efficiency and effectiveness in terms of fault detection. For this purpose, a controlled experiment was conducted, comparing tests created by a total of twenty-three software engineering master students. The participants worked individually on manually designing and automatically generating tests for two IEC 1131-3 programs. Tests created by the participants in the experiment were collected and analyzed in terms of mutation score, decision coverage, number of tests, and testing duration.We found that, when compared to implementation-based testing, specification-based testing yields significantly more effective tests in terms of the number of faults detected. Specifically, specification-based tests more effectively detect comparison and value replacement type of faults, compared to implementation-based tests. On the other hand, implementation-based automated testing leads to fewer tests (up to 85% improvement) created in shorter time than the ones manually created based on the specification.

Bibtex

@inproceedings{Enoiu4269,
author = {Eduard Paul Enoiu and Adnan Causevic and Daniel Sundmark and Paul Pettersson},
title = {A Controlled Experiment in Testing of Safety-Critical Embedded Software},
note = {Acceptance rate 35/130=26{\%}, submitted 2015This material is presented for the dissemination of scientific work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In some cases, this work may not be reposted without the explicit permission of the copyright holder.},
month = {April},
year = {2016},
booktitle = {IEEE International Conference on Software Testing, Verification and Validation},
url = {http://www.es.mdh.se/publications/4269-}
}