The benefits of testing software in SE research : lessons learned from two PHD projects

• Autorzy: Novak Matija , Mijac Marco

Identyfikatory

DOI

10.7494/csci.2025.26.4.6702

• Języki publikacji: EN

Abstrakty

EN

Software engineering (SE) research often involves creating software – either asa primary research output (e.g., in design science research) or as a supporting tool for the traditional research process. Ensuring software quality is essential, as it influences both the research process and the credibility of findings. Integrating software-testing methods into SE research can streamline efforts by addressing the goals of both research and development processes simultaneously. This paper highlights the advantages of incorporating software testingin SE research – particularly for research evaluation. Through qualitative analysis of software artifacts and insights from two PhD projects, we present ten lessons learned. These experiences demonstrate that, when effectively integrated, software testing offers significant benefits for both the research processand its results.

Słowa kluczowe

EN

software testing; research evaluation; lessons learned; SE research

• Wydawca: Wydawnictwa AGH
• Czasopismo: Computer Science
• Rocznik: 2025
• Tom: T. 26 (4)
• Strony: 51--75
• Opis fizyczny: Bibliogr. 54 poz., rys., tab., wykr.

Twórcy

autor

Novak Matija

• University of Zagreb, Faculty of Organization and Informatics, Pavlinska 2 42000 Varaždin, Croatia

autor

Mijac Marco

• University of Zagreb, Faculty of Organization and Informatics, Pavlinska 2 42000 Varaždin, Croatia

Bibliografia

• [1] Arnott D., Pervan G.: Design science in decision support systems research: An assessment using the Hevner, March, Park, and Ram Guidelines, Journal of the Association for Information Systems, vol. 13(11), p. 1, 2012. doi: 10.17705/1jais.00315.
• [2] Arvanitou E.M., Ampatzoglou A., Chatzigeorgiou A., Carver J.C.: Software engineering practices for scientific software development: A systematic mapping study, Journal of Systems and Software, vol. 172, 110848, 2021. doi: 10.1016/j.jss.2020.110848.
• [3] Badolato A.M.: Research software as a pillar of open science, 2022. https://www.ouvrirlascience.fr/research-software-as-a-pillar-of-open-science.
• [4] Barker M., Chue Hong N.P., Katz D.S., Lamprecht A.L., Martinez-Ortiz C., Psomopoulos F., Harrow J., Castro L.J., Gruenpeter M., Martinez P.A., Honeyman T.: Introducing the FAIR Principles for research software, Scientific Data, vol. 9(1), p. 622, 2022. doi: 10.1038/s41597-022-01710-x. Publisher: NaturePublishing Group.
• [5] Beck K.: Test-driven development: by example, Addison-Wesley Professional, 2003.
• [6] Bissi W., Serra Seca Neto A.G., Emer M.C.F.P.: The effects of test driven development on internal quality, external quality and productivity: A systematic review, Information and Software Technology, vol. 74, pp. 45–54, 2016.doi: 10.1016/j.infsof.2016.02.004.
• [7] Booch G., Maksimchuk R.A., Engle M.W., Young B.J., Conallen J., Hous-ton K.A.: Object-Oriented Analysis and Design with Applications, Addison-Wesley Professional, Upper Saddle River, NJ, 3rd ed., 2007.
• [8] Brett A., Croucher M., Haines R., Hettrick S., Hetherington J., Stillwell M., Wyatt C.: Research software engineers: state of the nation report 2017, 2017.
• [9] Corrado E.M.: The Importance of Open Access, Open Source, and Open Standards for Libraries: Theme: Open Access Journals, Issues in Science and Technology Librarianship, vol. Spring 2005(42), 2005. doi: 10.29173/istl2002.
• [10] Cotton R.: Testing R Code, Chapman and Hall/CRC, 2017. doi: 10.1201/9781315380285.
• [11] Dasso A.: Verification, validation and testing in software engineering, IGI global,2006.
• [12] Deng Q., Wang Y., Ji S.: Design science research in information systems: A systematic literature review 2001-2015. In: International Conference on Information Resources Management (CONF-IRM), 2017.
• [13] Ebert C., Gallardo G., Hernantes J., Serrano N.: DevOps, Ieee Software, vol. 33(3), pp. 94–100, 2016. doi: 10.1109/ms.2016.68.
• [14] Engstrom E., Runeson P., Skoglund M.: A systematic review on regression test selection techniques, Information and Software Technology, vol. 52(1), pp. 14–30, 2010. doi: https://doi.org/10.1016/j.infsof.2009.07.001.
• [15] Field A., Miles J., Field Z.: Discovering statistics using R, SAGE Publications Ltd, 2012.
• [16] Fowler M.: Refactoring: improving the design of existing code, Addison-Wesley Professional, 2018.
• [17] Gamma E., Helm R., Johnson R., Vlissides J.: Design patterns: elements of reusable object-oriented languages and systems, Addison-Wesley Reading, 1994.
• [18] Heaton D.: Software engineering for enabling scientific software development, Ph.D. thesis, University of Alabama Libraries, 2015.
• [19] Hettrick S.: softwaresaved/software_in_research_survey_2014: Software in research survey, 2018. doi: 10.5281/zenodo.1183562.
• [20] Hettrick S., Antonioletti M., Carr L., Chue Hong N., Crouch S., De Roure D., Emsley I., et al.: UK Research Software Survey 2014 [dataset], 2014. doi: 10.5281/zenodo.14809.
• [21] Hevner A.R., March S.T., Park J., Ram S.: Design science in information systems research, MIS quarterly, pp. 75–105, 2004. doi: 10.2307/25148625.
• [22] IEEE Standard Glossary of Software Engineering Terminology, IEEE Std 61012-1990, pp. 1–84, 1990. doi: 10.1109/IEEE.1990.101064. Conference Name: IEEEStd 610.12-1990.
• [23] Jalil S., Rafi S., LaToza T.D., Moran K., Lam W.: ChatGPT and Software Testing Education: Promises & Perils. In: 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW),pp. 4130–4137, IEEE, 2023. doi: 10.1109/icstw58534.2023.00078.
• [24] John L.K., Eeckhout L.: Performance evaluation and benchmarking, CRC Press, 2018.
• [25] Kanewala U., Bieman J.M.: Testing scientific software: A systematic literature review, Information and Software Technology, vol. 56(10), pp. 1219–1232, 2014.doi: 10.1016/j.infsof.2014.05.006.
• [26] Kelly D., Sanders R.: The challenge of testing scientific software. In: Proceedings of the 3rd annual conference of the Association for Software Testing (CAST 2008:Beyond the Boundaries), pp. 30–36, 2008.
• [27] Kitchenham B., Pickard L., Pfleeger S.L.: Case studies for method and tool evaluation, IEEE Software, vol. 12(4), pp. 52–62, 1995. doi: 10.1109/52.391832.
• [28] Kochhar P.S., Thung F., Lo D.: Code coverage and test suite effectiveness: Empirical study with real bugs in large systems. In: 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), pp. 560–564, IEEE, 2015. doi: 10.1109/saner.2015.7081877.
• [29] Martin R.C.: Clean code: a handbook of agile software craftsmanship, Pearson Education, 2009.
• [30] Martin R.C.: Clean Architecture: A Craftsman’s Guide to Software Structure and Design, Robert C. Martin Series, Pearson Education, 2017.
• [31] Martin R.C.: Clean agile: back to basics, Pearson Boston, 2019.
• [32] Martin R.C.: Clean Craftsmanship: Disciplines, Standards, and Ethics, Pearson Education, Limited, 2021.
• [33] Meszaros G.: xUnit test patterns: Refactoring test code, Pearson Education, 2007.
• [34] Microsoft: Code coverage testing - Visual Studio, 2024. https : / /docs.microsoft.com/en-us/visualstudio/test/using-code-coverage-to-determine-how-much-code-is-being-tested.
• [35] Mijač M.: Design and evaluation of software framework that improves the management of reactive dependencies in development of object-oriented applications, Ph.D. thesis, University of Zagreb Faculty of Organization and Informatics, 2021. https://urn.nsk.hr/urn:nbn:hr:211:897057.
• [36] Milliken G., Johnson D.: Analysis of messy data Volume 1 Designed Experiments, CRC Press, 2009.
• [37] Novak M.: Effect of source-code preprocessing techniques on plagiarism detection accuracy in student programming assignments, Ph.D. thesis, University of Zagreb Faculty of Organization and Informatics, 2020. https://urn.nsk.hr/urn:nbn:hr:211:528052.
• [38] Novak M., Joy M., Kermek D.: Source-code Similarity Detection and Detection Tools Used in Academia: A Systematic Review, ACM TOCE, vol. 19(3), 2019.doi: 10.1145/3313290.
• [39] Pargaonkar S.: A comprehensive review of performance testing methodologies andbest practices: software quality engineering, International Journal of Science and Research (IJSR), vol. 12(8), pp. 2008–2014, 2023. doi: 10.21275/sr23822111402.
• [40] Prat N., Comyn-Wattiau I., Akoka J.: A Taxonomy of Evaluation Methods for Information Systems Artifacts, Journal of Management Information Systems, vol. 32(3), pp. 229–267, 2015. doi: 10.1080/07421222.2015.1099390.
• [41] Segal J.: Some problems of professional end user developers. In: IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2007), pp. 111–118, IEEE, 2007. doi: 10.1109/vlhcc.2007.17.
• [42] Srinivasan K., Devi T.: Software metrics validation methodologies in software engineering, International Journal of Software Engineering & Applications,vol. 5(6), p. 87, 2014. doi: 10.5121/ijsea.2014.5606.
• [43] Stol K.J., Fitzgerald B.: Guidelines for Conducting Software Engineering Research. In: Contemporary Empirical Methods in Software Engineering, pp. 27–62,Springer, 2020. doi: 10.1007/978-3-030-32489-6_2.
• [44] Todorov V., Boulanger F., Taha S.: Formal verification of automotive embedded software. In: Proceedings of the 6th Conference on Formal Methods in SoftwareEngineering, pp. 84–87, 2018. doi: 10.1145/3193992.3194003.
• [45] Tosun A., Ahmed M., Turhan B., Juristo N.: On the effectiveness of unit tests intestdriven development. In: Proceedings of the 2018 International Conference onSoftware and System Process, pp. 113–122, 2018. doi: 10.1145/3202710.3203153.
• [46] Tremblay M.C., Hevner A.R., Berndt D.J.: Focus groups for artifact refinement and evaluation in design research, Communications of the association for information systems, vol. 26(1), p. 27, 2010. doi: 10.17705/1cais.02627.
• [47] Venable J., Pries-Heje J., Baskerville R.: FEDS: a Framework for Evaluationin Design Science Research, European Journal of Information Systems, 2014.doi: 10.1057/ejis.2014.36.
• [48] Von Krogh G., Spaeth S.: The open source software phenomenon: Characteristics that promote research, The Journal of Strategic Information Systems, vol. 16(3), pp. 236–253, 2007. doi: 10.1016/j.jsis.2007.06.001.
• [49] Wahl N.J.: An Overview of Regression Testing, ACM SIGSOFT Software Engineering Notes, vol. 24(1), pp. 69–73, 1999. doi: 10.1145/308769.308790.
• [50] Wang J., Huang Y., Chen C., Liu Z., Wang S., Wang Q.: Software testing with large language models: Survey, landscape, and vision, IEEE Transactions on Software Engineering, 2024. doi: 10.1109/tse.2024.3368208.
• [51] Wickham H.: testthat: Get started with testing, R Journal, vol. 3(1), 2011.doi: 10.32614/rj-2011-002.
• [52] Wieringa R., Moralı A.: Technical action research as a validation method in information systems design science. In: International Conference on Design Science Research in Information Systems, pp. 220–238, Springer, 2012.doi: 10.1007/978-3-642-29863-9_17.
• [53] Wohlin C., Runeson P., Höst M., Ohlsson M.C., Regnell B., Wesslén A., et al.: Experimentation in software engineering, vol. 236, Springer, 2012. doi: 10.1007/978-3-662-69306-3.
• [54] Zhang H., Maillo A., Khan S.A., Martínez-de Morentin X., Lehmann R., Gomez-Cabrero D., Tegnär J.: Reviewability and supportability: New complementary principles to empower research software practices, Computational and Structural Biotechnology Journal, vol. 23, pp. 3989–3998, 2024. doi: 10.1016/j.csbj.2024.10.034.

Uwagi

Opracowanie rekordu ze środków MNiSW, umowa nr POPUL/SP/0154/2024/02 w ramach programu "Społeczna odpowiedzialność nauki II" - moduł: Popularyzacja nauki (2025).