Seeking the best practices of assessment in maritime simulator training

• Autorzy: Tusher H. M. , Nazir S. , Ghosh S. , Rusli R.

Identyfikatory

DOI

10.12716/1001.17.01.10

• Języki publikacji: EN

Abstrakty

EN

Simulator-based training has become an integral part of Maritime education, and its effectiveness hinges on the use of appropriate assessment protocols. Despite the existence of several subjective and objective assessment techniques, instructors face difficulties in selecting and implementing the best practices that fit different learning contexts. The contextualized utility of the available assessment techniques further complicates the contexts. This study adopts a systematic literature review approach to comprehensively analyse available assessment techniques employed in maritime simulator training and to elicit their relationship with the desired learning outcomes. The study also presents a nuanced understanding of the advantages and limitations of the identified assessment techniques. Further, the state-of-the-art of assessment methods is discussed along with a few proposals for the future considering both research and practical implications. The findings of this study are expected to provide valuable guidance to maritime instructors in selecting and implementing appropriate assessment techniques that align with desired learning outcomes in simulator training.

Słowa kluczowe

EN

maritime education and training; human factor; maritime simulator training; assessments in simulated environments; Performance Assessment; assessment model; STCW; literature review

• Wydawca: Faculty of Navigation, Gdynia Maritime University
• Czasopismo: TransNav : International Journal on Marine Navigation and Safety of Sea Transportation
• Rocznik: 2023
• Tom: Vol. 17 No. 1
• Strony: 105--114
• Opis fizyczny: Bibliogr. 53 poz., rys., tab.

Twórcy

autor

Tusher H. M.

• University of South-Eastern Norway, Borre, Norway

autor

Nazir S.

• University of South-Eastern Norway, Borre, Norway
• Nord University, Nordland, Bodø, Norway

autor

Ghosh S.

• University of Tasmania, Launceston, Australia

autor

Rusli R.

• Universiti Teknologi Petronas, , Seri Iskandar, Perak, Malaysia

Bibliografia

• [1] M. Hontvedt and H. C. Arnseth, ‘On the bridge to learn: Analysing the social organization of nautical instruction in a ship simulator’, Int. J. Comput.-Support. Collab. Learn., vol. 8, no. 1, pp. 89–112, Mar. 2013, doi: 10.1007/s11412-013-9166-3.
• [2] T. Kim et al., ‘The continuum of simulator-based maritime training and education’, WMU J. Marit. Aff., vol. 20, no. 2, pp. 135–150, Jun. 2021, doi: 10.1007/s13437-021-00242-2.
• [3] J. Ernstsen and S. Nazir, ‘Performance assessment in full-scale simulators–A case of maritime pilotage operations’, Saf. Sci., vol. 129, p. 104775, 2020, doi: https://doi.org/10.1016/j.ssci.2020.104775.
• [4] C. Sellberg, ‘Simulators in bridge operations training and assessment: a systematic review and qualitative synthesis’, WMU J. Marit. Aff., vol. 16, no. 2, pp. 247–263, May 2017, doi: 10.1007/s13437-016-0114-8.
• [5] K. Kraiger, J. K. Ford, and E. Salas, ‘Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation.’, J. Appl. Psychol., vol. 78, no. 2, pp. 311–328, Apr. 1993, doi: 10.1037/0021-9010.78.2.311.
• [6] S. Ghosh, M. Bowles, D. Ranmuthugala, and B. Brooks, ‘Reviewing seafarer assessment methods to determine the need for authentic assessment’, Aust. J. Marit. Ocean Aff., vol. 6, no. 1, pp. 49–63, Jan. 2014, doi: 10.1080/18366503.2014.888133.
• [7] B. S. Bell, S. I. Tannenbaum, J. K. Ford, R. A. Noe, and K. Kraiger, ‘100 years of training and development research: What we know and where we should go.’, J. Appl. Psychol., vol. 102, no. 3, pp. 305–323, 2017, doi: 10.1037/apl0000142.
• [8] C. Gipps, ‘Developments in Educational Assessment: what makes a good test?’, Assess. Educ. Princ. Policy Pract., vol. 1, no. 3, pp. 283–292, Jan. 1994, doi: 10.1080/0969594940010304.
• [9] W. L. Sanders and S. P. Horn, ‘Educational Assessment Reassessed: The Usefulness of Standardized and Alternative Measures of Student Achievement as Indicators for the Assessment of Educational Outcomes’, 1995.
• [10] M. Yorke, ‘Summative assessment: dealing with the “measurement fallacy”’, Stud. High. Educ., vol. 36, no. 3, pp. 251–273, 2011.
• [11] R. M. Gagne, ‘Learning outcomes and their effects: Useful categories of human performance.’, Am. Psychol., vol. 39, no. 4, p. 377, 1984.
• [12] K. Hjelmervik, S. Nazir, and A. Myhrvold, ‘Simulator training for maritime complex tasks: an experimental study’, WMU J. Marit. Aff., vol. 17, no. 1, pp. 17–30, Mar. 2018, doi: 10.1007/s13437-017-0133-0.
• [13] K. I. ØvergÁrd, S. Nazir, and A. Solberg, ‘Towards Automated Performance Assessment for Maritime Navigation’, TransNav Int. J. Mar. Navig. Saf. Sea Transp., vol. 11, no. 2, pp. 43–48, 2017, doi: 10.12716/1001.11.02.03.
• [14] C. Sellberg, ‘Pedagogical dilemmas in dynamic assessment situations: perspectives on video data from simulator-based competence tests’, WMU J. Marit. Aff., vol. 19, no. 4, pp. 493–508, Dec. 2020, doi: 10.1007/s13437-020-00210-2.
• [15] C. Sellberg, M. Lundin, and R. Säljö, ‘Assessment in the zone of proximal development: simulator-based competence tests and the dynamic evaluation of knowledge-in-action’, Classr. Discourse, pp. 1–21, Nov. 2021, doi: 10.1080/19463014.2021.1981957.
• [16] Y. Liu et al., ‘Psychophysiological evaluation of seafarers to improve training in maritime virtual simulator’, Adv. Eng. Inform., vol. 44, p. 101048, Apr. 2020, doi: 10.1016/j.aei.2020.101048.
• [17] L. Orlandi, B. Brooks, and M. Bowles, ‘The development of a shiphandling assessment tool (SAT): A methodology and an integrated approach to assess manoeuvring expertise in a full mission bridge simulator’, in 15th Annual General Assembly of the International Association of Maritime Universities, IAMU AGA 2014-Looking Ahead: Innovation in Maritime Education, Training and Research, 2014, pp. 131–140.
• [18] V. O. Gekara, M. Bloor, and H. Sampson, ‘Computer-based assessment in safety-critical industries: the case of shipping’, J. Vocat. Educ. Train., vol. 63, no. 1, pp. 87–100, 2011.
• [19] S. Ghosh, ‘Can authentic assessment find its place in seafarer education and training?’, Aust. J. Marit. Ocean Aff., vol. 9, no. 4, pp. 213–226, Oct. 2017, doi: 10.1080/18366503.2017.1320828.
• [20] S. Ghosh and M. Bowles, ‘Challenges and implications in achieving content validity of an authentic assessment task designed to assess seafarer’s leadership and managerial skills’, WMU J. Marit. Aff., vol. 19, no. 3, pp. 373–391, Sep. 2020, doi: 10.1007/s13437-020-00209-9.
• [21] C. Sellberg, A. C. Wiig, and R. Säljö, ‘Mastering the artful practice of navigation: The situated endorsement of professional competence in post-simulation evaluations’, Stud. Educ. Eval., vol. 72, p. 101111, Mar. 2022, doi: 10.1016/j.stueduc.2021.101111.
• [22] M. G. Jamil and Z. Bhuiyan, ‘Deep learning elements in maritime simulation programmes: a pedagogical exploration of learner experiences’, Int. J. Educ. Technol. High. Educ., vol. 18, no. 1, p. 18, Dec. 2021, doi: 10.1186/s41239-021-00255-0.
• [23] S. K. Renganayagalu, S. Mallam, S. Nazir, J. Ernstsen, and P. Haavardtun, ‘Impact of Simulation Fidelity on Student Self-efficacy and Perceived Skill Development in Maritime Training’, TransNav Int. J. Mar. Navig. Saf. Sea Transp., vol. 13, no. 3, pp. 663–669, 2019, doi: 10.12716/1001.13.03.25.
• [24] C. Sellberg, ‘From briefing, through scenario, to debriefing: the maritime instructor’s work during simulator-based training’, Cogn. Technol. Work, vol. 20, no. 1, pp. 49–62, Feb. 2018, doi: 10.1007/s10111-017-0446-y.
• [25] E.-R. Saus, B. H. Johnsen, J. Eid, and J. F. Thayer, ‘Who benefits from simulator training: Personality and heart rate variability in relation to situation awareness during navigation training’, Comput. Hum. Behav., vol. 28, no. 4, pp. 1262–1268, Jul. 2012, doi: 10.1016/j.chb.2012.02.009.
• [26] K. Benedict, M. Baldauf, C. Felsenstein, and M. Kirchhoff, ‘Computer-based support for the evaluation of ship handling exercise results’, WMU J. Marit. Aff., vol. 5, no. 1, pp. 17–35, Apr. 2006, doi: 10.1007/BF03195079.
• [27] H. Kobayashi, ‘Use of simulators in assessment, learning and teaching of mariners’, WMU J. Marit. Aff., vol. 4, no. 1, pp. 57–75, Apr. 2005, doi: 10.1007/BF03195064.
• [28] C. Sellberg and M. Lundin, ‘Demonstrating professional intersubjectivity: The instructor’s work in simulator-based learning environments’, Learn. Cult. Soc. Interact., vol. 13, pp. 60–74, Jun. 2017, doi: 10.1016/j.lcsi.2017.02.003.
• [29] D. R. Sadler, ‘Interpretations of criteria-based assessment and grading in higher education’, Assess. Eval. High. Educ., vol. 30, no. 2, pp. 175–194, 2005.
• [30] D. Moher, A. Liberati, J. Tetzlaff, D. G. Altman, and The PRISMA Group, ‘Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement’, PLoS Med., vol. 6, no. 7, p. e1000097, Jul. 2009, doi: 10.1371/journal.pmed.1000097.
• [31] J. P. Chan, R. Norman, K. Pazouki, and D. Golightly, ‘Autonomous maritime operations and the influence of situational awareness within maritime navigation’, WMU J. Marit. Aff., vol. 21, no. 2, pp. 121–140, Jun. 2022, doi: 10.1007/s13437-022-00264-4.
• [32] O. Atik and O. Arslan, ‘Use of eye tracking for assessment of electronic navigation competency in maritime training’, J. Eye Mov. Res., vol. 12, no. 3, Jul. 2019, doi: 10.16910/jemr.12.3.2.
• [33] O. Atik, ‘Eye tracking for assessment of situational awareness in bridge resource management training’, J. Eye Mov. Res., vol. 12, no. 3, Apr. 2020, doi: 10.16910/jemr.12.3.7.
• [34] G. Li, R. Mao, H. P. Hildre, and H. Zhang, ‘Visual Attention Assessment for Expert-in-the-Loop Training in a Maritime Operation Simulator’, IEEE Trans. Ind. Inform., vol. 16, no. 1, pp. 522–531, Jan. 2020, doi: 10.1109/TII.2019.2945361.
• [35] F. Sanfilippo, ‘A multi-sensor fusion framework for improving situational awareness in demanding maritime training’, Reliab. Eng. Syst. Saf., vol. 161, pp. 12–24, May 2017, doi: 10.1016/j.ress.2016.12.015.
• [36] A. M. Nizar, T. Miwa, and M. Uchida, ‘Measurement of situation awareness in engine control room: approach for non-technical skill assessment in engine resource management’, WMU J. Marit. Aff., vol. 21, no. 3, pp. 401–419, Sep. 2022, doi: 10.1007/s13437-022-00270-6.
• [37] J. Jung and Y. J. Ahn, ‘Effects of interface on procedural skill transfer in virtual training: Lifeboat launching operation study: A comparative assessment interfaces in virtual training’, Comput. Animat. Virtual Worlds, vol. 29, no. 3–4, p. e1812, May 2018, doi: 10.1002/cav.1812.
• [38] C. Sellberg, O. Lindmark, and M. Lundin, ‘Certifying Navigational Skills: A Video-based Study on Assessments in Simulated Environments’, TransNav Int. J. Mar. Navig. Saf. Sea Transp., vol. 13, no. 4, pp. 881–886, 2019, doi: 10.12716/1001.13.04.23.
• [39] C. Kandemir, O. Soner, and M. Celik, ‘Proposing a practical training assessment technique to adopt simulators into marine engineering education’, WMU J. Marit. Aff., vol. 17, no. 1, pp. 1–15, Mar. 2018, doi: 10.1007/s13437-018-0137-4.
• [40] C. Kandemir, and M. Celik, ‘A Human Reliability Assessment of Marine Engineering Students through Engine Room Simulator Technology’, Simul. Gaming,
• [41] F. Saeed, A. Wall, C. Roberts, R. Riahi, and A. Bury, ‘A proposed quantitative methodology for the evaluation of the effectiveness of Human Element, Leadership and Management (HELM) training in the UK’, WMU J. Marit. Aff., vol. 16, no. 1, pp. 115–138, Jan. 2017, doi: 10.1007/s13437-016-0107-7.
• [42] E.-R. Saus, B. H. Johnsen, J. E.-R. Saus, and J. Eid, ‘Perceived learning outcome: The relationship between experience, realism and situation awareness during simulator training’, Int. Marit. Health, vol. 62, no. 4, pp. 258–264, 2010.
• [43] G. Emad and W. M. Roth, ‘Contradictions in the practices of training for and assessment of competency: A case study from the maritime domain’, Educ. Train., vol. 50, no. 3, pp. 260–272, Apr. 2008, doi: 10.1108/00400910810874026.
• [44] S. Fan, J. Zhang, E. Blanco-Davis, Z. Yang, J. Wang, and X. Yan, ‘Effects of seafarers’ emotion on human performance using bridge simulation’, Ocean Eng., vol. 170, pp. 111–119, Dec. 2018, doi: 10.1016/j.oceaneng.2018.10.021.
• [45] S. Jensen, M. Lutzen, L. L. Mikkelsen, H. B. Rasmussen, P. V. Pedersen, and P. Schamby, ‘Energy-efficient operational training in a ship bridge simulator’, J. Clean. Prod., vol. 171, pp. 175–183, Jan. 2018, doi: 10.1016/j.jclepro.2017.10.026.
• [46] T. Kim, A. K. Sydnes, and B.-M. Batalden, ‘Development and validation of a safety leadership Self-Efficacy Scale (SLSES) in maritime context’, Saf. Sci., vol. 134, p. 105031, Feb. 2021, doi: 10.1016/j.ssci.2020.105031.
• [47] S. Fan and Z. Yang, ‘Towards objective human performance measurement for maritime safety: A new psychophysiological data-driven machine learning method’, Reliab. Eng. Syst. Saf., vol. 233, p. 109103, May 2023, doi: 10.1016/j.ress.2023.109103.
• [48] H. M. Tusher, S. Nazir, S. Mallam, and Z. H. Munim, ‘Artificial Neural Network (ANN) for Performance Assessment in Virtual Reality (VR) Simulators: From Surgical to Maritime Training’, in 2022 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM), Kuala Lumpur, Malaysia, Dec. 2022, pp. 0334–0338. doi: 10.1109/IEEM55944.2022.9989816.
• [49] A. M. Wahl and T. Kongsvik, ‘Crew resource management training in the maritime industry: a literature review’, WMU J. Marit. Aff., vol. 17, no. 3, pp. 377–396, Sep. 2018, doi: 10.1007/s13437-018-0150-7.
• [50] O. Chernikova, N. Heitzmann, M. Stadler, D. Holzberger, T. Seidel, and F. Fischer, ‘Simulation-Based Learning in Higher Education: A Meta-Analysis’, Rev. Educ. Res., vol. 90, no. 4, pp. 499–541, Aug. 2020, doi: 10.3102/0034654320933544.
• [51] J. Herrington and L. Kervin, ‘Authentic Learning Supported by Technology: Ten suggestions and cases of integration in classrooms’, Educ. Media Int., vol. 44, no. 3, pp. 219–236, Sep. 2007, doi: 10.1080/09523980701491666.
• [52] T. C. Reeves, J. Herrington, and R. Oliver, ‘A development research agenda for online collaborative learning’, Educ. Technol. Res. Dev., vol. 52, no. 4, pp. 53–65, 2004.
• [53] M. A. Flores, A. M. Veiga Simão, A. Barros, and D. Pereira, ‘Perceptions of effectiveness, fairness and feedback of assessment methods: a study in higher education’, Stud. High. Educ., vol. 40, no. 9, pp. 1523–1534, 2015.

Uwagi

Opracowanie rekordu ze środków MEiN, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2022-2023).