Systematic Literature Review on Search Based Mutation Testing

• Autorzy: Nishtha J. , Bharti S. , Shweta R.

Identyfikatory

DOI

10.5277/e-Inf170103

• Języki publikacji: EN

Abstrakty

EN

Search based techniques have been widely applied in the domain of software testing. This Systematic Literature Review aims to present the research carried out in the field of search based approaches applied particularly to mutation testing. During the course of literature review, renowned databases were searched for the relevant publications in the field to include relevant studies up to the year 2014. Few studies for the year 2015-16, gathered by performing snowball search, have also been included. For reviewing the literature in the field, 43 studies were evaluated, out of which 18 studies were thoroughly studied and analysed. The result of this SLR shows that search based techniques were applied to mutation testing primarily for two purposes, either for mutant optimisation or for test case optimisation. The future directions of this SLR suggests the application of search based techniques for other issues related to mutation testing, like, solution to equivalents mutants, generation of non-trivial mutants, multi-objective test data generation and non-functional testing.

Słowa kluczowe

EN

software testing; analysis; verification; systematic review; mapping studies

• Wydawca: Oficyna Wydawnicza Politechniki Wrocławskiej
• Czasopismo: e-Informatica Software Engineering Journal
• Rocznik: 2017
• Tom: Vol. 11, nr 1
• Strony: 59--76
• Opis fizyczny: Bibliogr. 83 poz., tab., rys.

Twórcy

autor

Nishtha J.

• Research Scholar, USICT and Assistant Professor, Department of Computer Science and Engineering, MSIT, New Delhi, India

autor

Bharti S.

• Research Scholar, USICT and Assistant Professor, Department of Computer Science and Engineering, MSIT, New Delhi, India

autor

Shweta R.

• USICT, GGS Indraprastha University, New Delhi, India

Bibliografia

• [1] G.J. Myers, C. Sandler, and T. Badgett, The art of software testing. John Wiley & Sons, 2011.
• [2] J. Edvardsson, “A survey on automatic test data generation,” in Proceedings of the 2nd Conference on Computer Science and Engineering, 1999, pp. 21–28.
• [3] M. Prasanna, S. Sivanandam, R. Venkatesan, and R. Sundarrajan, “A survey on automatic test case generation,” Academic Open Internet Journal, Vol. 15, No. part 6, 2005.
• [4] H. Tahbildar and B. Kalita, “Automated software test data generation: Direction of research,” International Journal of Computer Science and Engineering Survey, Vol. 2, No. 1, 2011, pp. 99–120.
• [5] S. Anand, E.K. Burke, T.Y. Chen, J. Clark, M.B. Cohen, W. Grieskamp, M. Harman, M.J. Harrold, P. McMinn et al., “An orchestrated survey of methodologies for automated software test case generation,” Journal of Systems and Software, Vol. 86, No. 8, 2013, pp. 1978–2001.
• [6] C.A.C. Coello, “A comprehensive survey of evolutionary-based multiobjective optimization techniques,” Knowledge and Information systems, Vol. 1, No. 3, 1999, pp. 269–308.
• [7] P. McMinn, “Search-based software test data generation: A survey,” Software Testing Verification and Reliability, Vol. 14, No. 2, 2004, pp. 105–156.
• [8] P. McMinn, “Search-based software testing: Past, present and future,” in IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 2011, pp. 153–163.
• [9] L. Bottaci, “A genetic algorithm fitness function for mutation testing,” in Proceedings of the SEMINALL-workshop at the 23rd International Conference on Software Engineering, Toronto, Canada, 2001.
• [10] M. Papadakis and N. Malevris, “Searching and generating test inputs for mutation testing,” SpringerPlus, Vol. 2, No. 1, 2013, p. 1.
• [11] F. Souza, M. Papadakis, V.H. Durelli, and M.E. Delamaro, “Test data generation techniques for mutation testing: A systematic mapping,” Proceedings of the 11th ESELAW, 2014, pp. 1–14.
• [12] Y. Jia and M. Harman, “An analysis and survey of the development of mutation testing,” IEEE Transactions on Software Engineering, Vol. 37, No. 5, 2011, pp. 649–678.
• [13] N. Jatana, S. Rani, and B. Suri, “State of art in the field of search-based mutation testing,” in 4th International Conference on Reliability, Infocom Technologies and Optimization (ICRITO) (Trends and Future Directions). IEEE, 2015, pp. 1–6.
• [14] R.A. Silva, S. do Rocio Senger de Souza, and P.S.L. de Souza, “A systematic review on search based mutation testing,” Information and Software Technology, 2016.
• [15] C. Wohlin, “Guidelines for snowballing in systematic literature studies and a replication in software engineering,” in Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering. ACM, 2014, p. 38.
• [16] P. Cronin, F. Ryan, and M. Coughlan, “Undertaking a literature review: A step-by-step approach,” British Journal of Nursing, Vol. 17, No. 1, 2008, p. 38.
• [17] K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson, “Systematic mapping studies in software engineering,” in 12th International Conference on Evaluation and Assessment in Software Engineering, Vol. 17, No. 1. sn, 2008.
• [18] B. Kitchenham, “Procedures for performing systematic reviews,” Keele University, Keele University, Keele, Staffs, UK, Joint Technical Report TR/SE-0401, 2004. [Online]. http: //csnotes.upm.edu.my/kelasmaya/pgkm20910. nsf/0/715071a8011d4c2f482577a700386d3a/ $FILE/10.1.1.122.3308[1].pdf
• [19] M.J. Grant and A. Booth, “A typology of reviews: An analysis of 14 review types and associated
• methodologies,” Health Information and Libraries Journal, Vol. 26, No. 2, 2009, pp. 91–108.
• [20] B. Kitchenham and S. Charters, “Guidelines for performing systematic literature reviews in software engineering,” Keele University & University of Durham, EBSE Technical Report EBSE 2007-01, 2007.
• [21] W. Orzeszyna, L. Madeyski, and R. Torkar, Protocol for a systematic literature review of methods dealing with equivalent mutant problem. [Online]. http://madeyski.e-informatyka.pl/download/ slr/EquivalentMutantsSLRProtocol.pdf
• [22] L. Madeyski, W. Orzeszyna, R. Torkar, and M. Jozala, “Overcoming the equivalent mutant problem: A systematic literature review and a comparative experiment of second order mutation,” IEEE Transactions on Software Engineering, Vol. 40, No. 1, 2014, pp. 23–42.
• [23] B. Baudry, V. Le Hanh, J.M. Jézéquel, and Y. Le Traon, “Trustable components: Yet anothermutation-based approach,” in Mutation testing for the new century. Springer, 2001, pp. 47–54.
• [24] B. Baudry, F. Fleurey, J.M. Jézéquel, and Y. Le Traon, “Genes and bacteria for automatic test cases optimization in the .NET environment,” in 13th International Symposium on Software Reliability Engineering. IEEE, 2002, pp. 195–206.
• [25] P. May, K. Mander, and J. Timmis, “Software vaccination: An artificial immune system approach to mutation testing,” in International Conference on Artificial Immune Systems. Springer, 2003, pp. 81–92.
• [26] M.C.F. Emer and S.R. Vergilio, “Selection and evaluation of test data based on genetic programming,”Software Quality Journal, Vol. 11, No. 2, 2003, pp. 167–186.
• [27] K. Adamopoulos, M. Harman, and R.M. Hierons, “How to overcome the equivalent mutant problem and achieve tailored selective mutation using co-evolution,” in Genetic and evolutionary computation conference. Springer, 2004, pp. 1338–1349.
• [28] M. Masud, A. Nayak, M. Zaman, and N. Bansal, “Strategy for mutation testing using genetic algorithms,” in Canadian Conference on Electrical and Computer Engineering. IEEE, 2005, pp. 1049–1052.
• [29] K. Ayari, S. Bouktif, and G. Antoniol, “Automatic mutation test input data generation via ant colony,” in Proceedings of the 9th annual conference on Genetic and evolutionary computation. ACM, 2007, pp. 1074–1081.
• [30] Y. Jia and M. Harman, “Constructing subtle faults using higher order mutation testing,” in Eighth IEEE International Working Conference on Source Code Analysis and Manipulation. IEEE, 2008, pp. 249–258.
• [31] K. Mishra, S. Tiwari, A. Kumar, and A. Misra, “An approach for mutation testing using elitist genetic algorithm,” in 3rd IEEE International Conference on Computer Science and Information Technology (ICCSIT), Vol. 5. IEEE, 2010, pp. 426–429.
• [32] B. Schwarz, D. Schuler, and A. Zeller, “Breeding high-impact mutations,” in IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 2011, pp. 382–387.
• [33] M. Harman, Y. Jia, and W.B. Langdon, “Strong higher order mutation-based test data generation,” in Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering. ACM, 2011, pp. 212–222.
• [34] J.J. Domínguez-Jiménez, A. Estero-Botaro, A. García-Domínguez, and I. Medina-Bulo, “Evolutionary mutation testing,” Information and Software Technology, Vol. 53, No. 10, 2011, pp. 1108–1123.
• [35] G. Fraser and A. Zeller, “Mutation-driven generation of unit tests and oracles,” IEEE Transactions on Software Engineering, Vol. 38, No. 2, 2012, pp. 278–292.
• [36] A.A.L. de Oliveira, C.G. Camilo-Junior, and A.M. Vincenzi, “A coevolutionary algorithm to automatic test case selection and mutant in mutation testing,” in IEEE Congress on Evolutionary Computation. IEEE, 2013, pp. 829–836.
• [37] M.B. Bashir and A. Nadeem, “A fitness function for evolutionary mutation testing of object-oriented programs,” in 9th International Conference on Emerging Technologies (ICET). IEEE, 2013, pp. 1–6.
• [38] P.S. Yiasemis and A.S. Andreou, “Locating and correcting software faults in executable code slices via evolutionary mutation testing,” in International Conference on Enterprise Information Systems. Springer, 2012, pp. 207–227.
• [39] E. Omar, S. Ghosh, and D. Whitley, “Comparing search techniques for finding subtle higher order mutants,” in Proceedings of the Annual Conference on Genetic and Evolutionary Computation. ACM, 2014, pp. 1271–1278.
• [40] E. Omar, S. Ghosh, and D. Whitley, “Constructing subtle higher order mutants for Java and AspectJ programs,” in IEEE 24th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2013, pp. 340–349.
• [41] Y.M.B. Ali and F. Benmaiza, “Generating test case for object-oriented software using genetic algorithm and mutation testing method,” International Journal of Applied Metaheuristic Computing (IJAMC), Vol. 3, No. 1, 2012, pp. 15–23.
• [42] A.S. Banzi, T. Nobre, G.B. Pinheiro, J.C.G. Árias, A. Pozo, and S.R. Vergilio, “Selecting mutation operators with a multiobjective approach,” Expert Systems with Applications, Vol. 39, No. 15, 2012, pp. 12 131–12 142.
• [43] B. Baudry, V. Le Hanh, J.M. Jézéquel, and Y. Le Traon, “Building trust into oo components using a genetic analogy,” in 11th International Symposium on Software Reliability Engineering. IEEE, 2000, pp. 4–14.
• [44] S. Subramanian and R. Natarajan, “A tool for generation and minimization of test suite by mutant gene algorithm,” Journal of Computer Sciences, Vol. 7, No. 10, 2011, pp. 1581–1589.
• [45] K.T. Le Thi My Hanh and N.T.B. Tung, “Mutation-based test data generation for simulink models using genetic algorithm and simulated annealing,” International Journal of Computer and Information Technology, Vol. 3,No. 04, 2014, pp. 763–771.
• [46] N.T. Binh, K.T. Tung et al., “A novel test data generation approach based upon mutation testing by using artificial immune system for Simulink models,” in Knowledge and Systems Engineering. Springer, 2015, pp. 169–181.
• [47] L.T.M. Hanh, N.T. Binh, and K.T. Tung, “Applying the meta-heuristic algorithms for mutation-based test data generation for Simulink models,” in Proceedings of the Fifth Symposium on Information and Communication Technology. ACM, 2014, pp. 102–109.
• [48] B.N. Thanh and T.K. Thanh, “Survey on mutation-based test data generation,” International Journal of Electrical and Computer Engineering, Vol. 5, No. 5, 2015.
• [49] M. Patrick, “Metaheuristic optimisation and mutation-driven test data generation,” in Computational Intelligence and Quantitative Software Engineering. Springer, 2016, pp. 89–115.
• [50] F. Popentiu-Vladicescu and G. Albeanu, “Nature-inspired approaches in software faults identification and debugging,” Procedia Computer Science, Vol. 92, 2016, pp. 6–12.
• [51] M. Dave and R. Agrawal, “Search based techniques and mutation analysis in automatic test case generation: A survey,” in IEEE International Advance Computing Conference (IACC). IEEE, 2015, pp. 795–799.
• [52] Y. Jia, F. Wu, M. Harman, and J. Krinke, “Genetic improvement using higher order mutation,” in Proceedings of the Companion Publication of the Annual Conference on Genetic and Evolutionary Computation. ACM, 2015, pp. 803–804.
• [53] Q.V. Nguyen and L. Madeyski, “Searching for strongly subsuming higher order mutants by applying multi-objective optimization algorithm,” in Advanced Computational Methods for Knowledge Engineering. Springer, 2015, pp. 391–402.
• [54] Q.V. Nguyen and L. Madeyski, “Higher order mutation testing to drive development of new test cases: An empirical comparison of three strategies,” in Asian Conference on Intelligent Information and Database Systems. Springer, 2016, pp. 235–244.
• [55] Q.V. Nguyen and L. Madeyski, “Empirical evaluation of multiobjective optimization algorithms searching for higher order mutants,” Cybernetics and Systems, 2016.
• [56] F. Wu, M. Harman, Y. Jia, and J. Krinke,“HOMI: Searching higher order mutants for softwareimprovement,” in International Symposium on Search Based Software Engineering. Springer, 2016, pp. 18–33.
• [57] A. Estero-Botaro, A. García-Domínguez, J.J. Domínguez-Jiménez, F. Palomo-Lozano, and I. Medina-Bulo, “A framework for genetic test-case generation for ws-bpel compositions,” in IFIP International Conference on Testing Software and Systems. Springer, 2014, pp. 1–16.
• [58] C.P. Rao and P. Govindarajulu, “Genetic algorithm for automatic generation of representative test suite for mutation testing,” International Journal of Computer Science and Network Security(IJCSNS), Vol. 15, No. 2, 2015, p. 11.
• [59] S. Rani and B. Suri, “An approach for test data generation based on genetic algorithm and delete mutation operators,” in Second International Conference on Advances in Computing and Communication Engineering (ICACCE). IEEE, 2015, pp. 714–718.
• [60] F.C.M. Souza, M. Papadakis, Y. Le Traon, andM.E. Delamaro, “Strong mutation-based tesdata generation using hill climbing,” in Proceedingof the 9th International Workshop o Search-Based Software Testing. ACM, 2016, pp 45–54.
• [61] N. Jatana, B. Suri, S. Misra, P. Kumar, and A.R. Choudhury, “Particle swarm based evolution and generation of test data using mutation testing,” in International Conference on Computational Science and its Applications. Springer, 2016, pp. 585–594.
• [62] N.T. Binh, K.T. Tung et al., “A novel fitness function of metaheuristic algorithms for test data generation for Simulink models based on mutation analysis,” Journal of Systems and Software, Vol. 120, 2016, pp. 17–30.
• [63] N. Jatana, B. Suri, P. Kumar, and B. Wadhwa, “Test suite reduction by mutation testing mapped to set cover problem,” in Proceedings of the Second International Conference on Information and Communication Technology for Competitive Strategies. ACM, 2016, p. 36.
• [64] P. McMinn, M. Harman, K. Lakhotia, Y. Hassoun, and J. Wegener, “Input domain reduction through irrelevant variable removal and its effect on local, global, and hybrid search-based structural test data generation,” IEEE Transactions on Software Engineering, Vol. 38, No. 2, 2012, pp. 453–477.
• [65] M. Dorigo and G.D. Caro, New ideas in optimization. McGraw-Hill Ltd., 1999, ch. The antcolony optimization meta-heuristic, pp. 11–32.
• [66] M. Dorigo, G. Di Caro, and L.M. Gambardella, “Ant algorithms for discrete optimization,” Artificial life, Vol. 5, No. 2, 1999, pp. 137–172.
• [67] M. Dorigo, “Optimization, learning and natural algorithms,” Ph. D. Thesis, Politecnico di Milano, Italy, 1992.
• [68] T. Stützle, M. López-Ibáñez, and M. Dorigo, “A concise overview of applications of ant colony optimization,” Wiley Encyclopedia of Operations Research and Management Science, 2011.
• [69] B. Suri and S. Singhal, “Literature survey of ant colony optimization in software testing,” in CSI Sixth International Conference on Software Engineering (CONSEG). IEEE, 2012, pp. 1–7.
• [70] A.S. Fraser, “Simulation of genetic systems by automatic digital computers VI. Epistasis,” Australian Journal of Biological Sciences, Vol. 13, No. 2, 1960, pp. 150–162.
• [71] H.J. Bremermann, The evolution of intelligence: The nervous system as a model of its environment. University of Washington, Department of Mathematics, 1958.
• [72] J.H. Holland, Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence. U Michigan Press, 1975.
• [73] C. Sharma, S. Sabharwal, and R. Sibal, “A survey on software testing techniques using genetic algorithm,” CoRR, 2014. [Online]. https://arxiv.org/abs/1411.1154
• [74] E. Omar and S. Ghosh, “An exploratory study of higher order mutation testing in aspect-oriented programming,” in IEEE 23rd International Symposium on Software Reliability Engineering. IEEE, 2012, pp. 1–10.
• [75] B. Baudry, F. Fleurey, J.M. Jézéquel, and Y. Le Traon, “Automatic test case optimization using a bacteriological adaptation model: Application to .NET components,” in 17th IEEE International Conference on Automated Software Engineering. IEEE, 2002, pp. 253–256.
• [76] B. Baudry, F. Fleurey, J.M. Jézéquel, and Y. Le Traon, “From genetic to bacteriological algorithms for mutation-based testing,” Software Testing, Verification and Reliability, Vol. 15, No. 2, 2005, pp. 73–96.
• [77] P. May, J. Timmis, and K. Mander, “Immune and evolutionary approaches to software mutation testing,” in Artificial Immune Systems. Springer, 2007, pp. 336–347.
• [78] P. May, K. Mander, and J. Timmis, “Mutation testing: An artificial immune system approach,” in UK-Softest. UK Software Testing Workshop. Citeseer, 2003.
• [79] M. Papadakis and N. Malevris, “Automatically performing weak mutation with the aid of symbolic execution, concolic testing andsearch-based testing,” Software Quality Journal, Vol. 19, No. 4, 2011, pp. 691–723.
• [80] Y. Jia and M. Harman, “Higher order mutationtesting,” Information and Software Technology, Vol. 51, No. 10, 2009, pp. 1379–1393.
• [81] Y. Jia, “Higher order mutation testing,” Ph.D.dissertation, University College London, 2013.[Online]. http://discovery.ucl.ac.uk/1401264/1/ YuePhDFinal2013.pdf
• [82] Q.V. Nguyen and L. Madeyski, “Problems of mutation testing and higher order mutation testing,” in Advanced computational methods for knowledge engineering. Springer, 2014, pp. 157–172.
• [83] M. Harman, Y. Jia, and Y. Zhang, “Achievements, open problems and challenges for search based software testing,” in IEEE 8th International Conference on Software Testing, Verification and Validation (ICST). IEEE, 2015, pp. 1–12.

Uwagi

PL

Opracowanie ze środków MNiSW w ramach umowy 812/P-DUN/2016 na działalność upowszechniającą naukę (zadania 2017).