Assessment Methods for Evaluation of Recommender Systems: A Survey

• Autorzy: Kuanr Madhusree , Mohapatra Puspanjali

Identyfikatory

DOI

10.2478/fcds-2021-0023

• Języki publikacji: EN

Abstrakty

EN

The recommender system (RS) filters out important information from a large pool of dynamically generated information to set some important decisions in terms of some recommendations according to the user’s past behavior, preferences, and interests. A recommender system is the subclass of information filtering systems that can anticipate the needs of the user before the needs are recognized by the user in the near future. But an evaluation of the recommender system is an important factor as it involves the trust of the user in the system. Various incompatible assessment methods are used for the evaluation of recommender systems, but the proper evaluation of a recommender system needs a particular objective set by the recommender system. This paper surveys and organizes the concepts and definitions of various metrics to assess recommender systems. Also, this survey tries to find out the relationship between the assessment methods and their categorization by type.

Słowa kluczowe

EN

recommender system; assessment methods; information filtering

• Wydawca: Wydawnictwo Politechniki Poznańskiej
• Czasopismo: Foundations of Computing and Decision Sciences
• Rocznik: 2021
• Tom: Vol. 46, No. 4
• Strony: 393--421
• Opis fizyczny: Bibliogr. 95 poz., rys., tab.

Twórcy

autor

Kuanr Madhusree

• Department of Computer Science and Engineering, IIIT Bhubaneswar, India

autor

Mohapatra Puspanjali

• Department of Computer Science and Engineering, IIIT Bhubaneswar, India

• https://orcid.org/puspanjali%40iiit-bh.ac.in

Bibliografia

• [1] Abu-Issa, A., Nawawreh, H., Shreteh, L., Salman, Y., Hassouneh, Y., Tumar, I., Hussein, M. (2017, August). A smart city mobile application for multitype, proactive, and context-aware recommender system. In 2017 International Conference on Engineering and Technology (ICET) (pp. 1-5). IEEE.
• [2] Adomavicius, G., Tuzhilin, A. (2005). Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE transactions on knowledge and data engineering, 17(6), 734-749.
• [3] Adomavicius, G., Sankaranarayanan, R., Sen, S., Tuzhilin, A. (2005). Incorporating contextual information in recommender systems using a multidimensional approach. ACM Transactions on Information Systems (TOIS), 23(1), 103-145.
• [4] Adamopoulos, P., Tuzhilin, A. (2014). On unexpectedness in recommender systems: Or how to better expect the unexpected. ACM Transactions on Intelligent Systems and Technology (TIST), 5(4), 1-32.
• [5] Ajesh, A., Nair, J., Jijin, P.S. (2016, September). A random forest approach for rating-based recommender system. In 2016 International conference on advances in computing, communications and informatics (ICACCI) (pp. 1293-1297). IEEE.
• [6] Ardagelou, P., Arampatzis, A. (2017). A Half-Life Decaying Model for Recommender Systems with Matrix Factorization. In TDDL/MDQual/Futurity@ TPDL.
• [7] Armstrong, R., Freitag, D., Joachims, T., Mitchell, T. (1995, March). Webwatcher: A learning apprentice for the world wide web. In AAAI Spring symposium on Information gathering from Heterogeneous, distributed environments (Vol. 93, p. 107).
• [8] Asim, M., Khusro, S. (2018). Content Based Call for Papers Recommendation to Researchers. 2018 12th International Conference on Open Source Systems and Technologies (ICOSST), 42-47.
• [9] Baeza-Yates, R., Ribeiro-Neto, B. (1999). Modern information retrieval. Essex: Addison Wesley.
• [10] Billsus, D., Pazzani, M.J. (1998, July). Learning collaborative information filters. In Icml (Vol. 98, pp. 46-54).
• [11] Bobadilla, J., Ortega, F., Hernando, A., Gutiérrez, A. (2013). Recommender systems survey. Knowledge-based systems, 46, 109-132.
• [12] Breese, J., Heckerman, D., Kadie, C. (1998). Empirical analysis of predictive algorithms for collaborative filtering. Uncertainty in artificial intelligence, proceedings of the fourteenth conference (pp. 43-52). Morgan Kaufman.
• [13] Bridge, D., Göker, M.H., McGinty, L., Smyth, B. (2005). Case-based recommender systems. The Knowledge Engineering Review, 20(3), 315-320.
• [14] Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) (2007). The Adaptive Web: Methods and Strategies of Web Personalization, Lecture Notes in Computer Science, vol. 4321. Springer-Verlag, Berlin.
• [15] Burke, R. (2002). Hybrid recommender systems. User Modeling and UserAdapted Interaction, 12(4), 331-370.
• [16] Cami, B.R., Hassanpour, H., Mashayekhi, H. (2017, December). A content-based movie recommender system based on temporal user preferences. In 2017 3rd Iranian Conference on Intelligent Systems and Signal Processing (ICSPIS) (pp. 121-125). IEEE.
• [17] Castells, P., Vargas, S., Wang, J. (2011). Novelty and diversity metrics for recommender systems: choice, discovery and relevance.
• [18] Chelliah, M., Sarkar, S. (2017, August). Product recommendations enhanced with reviews. In Proceedings of the Eleventh ACM Conference on Recommender Systems (pp. 398-399).
• [19] Chen, H.W., Wu, Y.L., Hor, M.K., Tang, C.Y. (2017, July). Fully content-based movie recommender system with feature extraction using neural network. In 2017 International conference on machine learning and cybernetics (ICMLC) (Vol. 2, pp. 504-509). IEEE.
• [20] Chen, J., Zhao, C., Chen, L. (2019). Collaborative filtering recommendation algorithm based on user correlation and evolutionary clustering. Complex Intelligent Systems, 1-10.
• [21] Chen, L., Pu, P. (2006, July). Evaluating critiquing-based recommender agents. In AAAI (Vol. 6, pp. 157-162).
• [22] Chen, L., Nayak, R., Xu, Y. (2011, July). A recommendation method for online dating networks based on social relations and demographic information. In 2011 International Conference on Advances in Social Networks Analysis and Mining (pp. 407-411). IEEE.
• [23] Chen, S., Peng, Y., Mi, H., Wang, C., Huang, Z. (2018, March). A cluster feature based approach for QoS prediction in Web service recommendation. In 2018 IEEE Symposium on Service-Oriented System Engineering (SOSE) (pp. 246-251). IEEE.
• [24] Chu, W., Park, S.T. (2009, April). Personalized recommendation on dynamic content using predictive bilinear models. In Proceedings of the 18th international conference on World wide web (pp. 691-700).
• [25] Davis, J., Goadrich, M. (2006). The relationship between precision, recall and roc curves. In Proceedings of the 23rd international conference on machine learning (ICML).
• [26] Ding, J., Wang, Y., Wang, Q., Cao, Y. (2017, October). Football video recommendation system with automatic rating based on user behavior. In 2017 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI) (pp. 1-5). IEEE.
• [27] Fawcett, T. (2006). An introduction to ROC analysis. Pattern recognition letters, 27(8), 861-874.
• [28] Fisher, M.J., Fieldsend, J.E., Everson, R. M. (2004). Precision and recall optimisation for information access tasks. In First workshop on roc analysis in AI. European conference on artificial intelligence (ECAI’2004), Valencia, Spain, August.
• [29] Ge, M., Delgado-Battenfeld, C., Jannach, D. (2010, September). Beyond accuracy: evaluating recommender systems by coverage and serendipity. In Proceedings of the fourth ACM conference on Recommender systems (pp. 257-260).
• [30] Giustozzi, F., Casali, A., Deco, C., dos Santos, H.L. and Cechinel, C. (2016). Recommender system of educational resources: A critiquing-based proposal. 2016 XI Latin American Conference on Learning Objects and Technology (LACLO), San Carlos, 2016, pp. 1-8, doi: 10.1109/LACLO.2016.7751779.
• [31] Goldberg, K., Roeder, T., Gupta, D., Perkins, C. (2001). Eigentaste: A constant time collaborative filtering algorithm. Information Retrieval, 4(2), 133-151.
• [32] Gong, S. (2009, May). Joining case-based reasoning and item-based collaborative filtering in recommender systems. In 2009 Second International Symposium on Electronic Commerce and Security (Vol. 1, pp. 40-42). IEEE.
• [33] Grasch, P., Felfernig, A., Reinfrank, F. (2013, October). Recomment: Towards critiquing-based recommendation with speech interaction. In Proceedings of the 7th ACM Conference on Recommender Systems (pp. 157-164).
• [34] Gravino, P., Monechi, B., Loreto, V. (2019). Towards novelty-driven recommender systems. Comptes Rendus Physique, 20(4), 371-379.
• [35] Guo, Y., Deng, G., Zhang, G., Luo, C. (2007, September). Using case-based reasoning and social trust to improve the performance of recommender system in e-commerce. In Second International Conference on Innovative Computing, Informatio and Control (ICICIC 2007) (pp. 484-484). IEEE.
• [36] Gupta, J., Gadge, J. (2014, April). A framework for a recommendation system based on collaborative filtering and demographics. In 2014 international conference on circuits, systems, communication and information technology applications (CSCITA) (pp. 300-304). IEEE.
• [37] Gupta, J. Gadge, J. (2015). Performance analysis of recommendation system based on collaborative filtering and demographics. Proceedings - 2015 International Conference on Communication, Information and Computing Technology, ICCICT 2015. 10.1109/ICCICT.2015.7045675.
• [38] Hariadi, A.I., Nurjanah, D. (2017, November). Hybrid attribute and personality based recommender system for book recommendation. In 2017 International Conference on Data and Software Engineering (ICoDSE) (pp. 1-5). IEEE.
• [39] Hayashi, A., Itoh, T., Nakamura, S. (2013, July). A visual analytics tool for system logs adopting variable recommendation and feature-based filtering. In 2013 17th International Conference on Information Visualisation (pp. 1-10). IEEE.
• [40] Herlocker, J.L., Konstan, J.A., Borchers, A., Riedl, J. (1999). An algorithmic framework for performing collaborative filtering. In SIGIR’99: Proceedings of the 22nd annual international ACM SIGIR conference on research and development in information retrieval (pp. 230-237). New York, NY, USA: ACM Press.
• [41] Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T. (2004). Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems (TOIS), 22(1), 5-53.
• [42] Hu, R., Pu, P. (2010, June). A study on user perception of personality-based recommender systems. In International conference on user modeling, adaptation, and personalization (pp. 291-302). Springer, Berlin, Heidelberg.
• [43] Huang, X., Tang, Y., Qu, R., Li, C., Yuan, C., Sun, S., Xu, B. (2018, May). Course Recommendation Model in Academic Social Networks Based on Association Rules and Multi-similarity. In 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design ((CSCWD)) (pp. 277-282). IEEE.
• [44] Hurley, N., Zhang, M. (2011). Novelty and diversity in top-n recommendation–analysis and evaluation. ACM Transactions on Internet Technology (TOIT), 10(4), 1-30.
• [45] Hurley, N.J. (2013, October). Personalised ranking with diversity. In Proceedings of the 7th ACM conference on Recommender systems (pp. 379-382).
• [46] Jarvelin, K., Kekalainen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS), 20(4), 422-446.
• [47] Kaminskas, M., Bridge, D. (2014, October). Measuring surprise in recommender systems. In Proceedings of the workshop on recommender systems evaluation: dimensions and design (Workshop programme of the 8th ACM conference on recommender systems).
• [48] Katarya, R., Verma, O.P. (2015, October). Restaurant recommender system based on psychographic and demographic factors in mobile environment. In 2015 International Conference on Green Computing and Internet of Things (ICGCIoT) (pp. 907-912). IEEE.
• [49] Kotkov, D., Veijalainen, J., Wang, S. (2016). Challenges of serendipity in recommender systems. In WEBIST 2016: Proceedings of the 12th International conference on web information systems and technologies. Volume 2, ISBN 978-989-758-186-1. SCITEPRESS.
• [50] Kuanr, M., Mohapatra, P. (2021). Recent Challenges in Recommender Systems: A Survey. In Progress in Advanced Computing and Intelligent Engineering (pp. 353-365). Springer, Singapore.
• [51] Kunaver, M., Požrl, T. (2017). Diversity in recommender systems–A survey. Knowledge-Based Systems, 123, 154-162.
• [52] Li, X. Li, D. (2019). An Improved Collaborative Filtering Recommendation Algorithm and Recommendation Strategy. Mobile Information Systems. 2019. 1-11. 10.1155/2019/3560968.
• [53] Li, X., Murata, T. (2010, October). Customizing knowledge-based recommender system by tracking analysis of user behavior. In 2010 IEEE 17Th International Conference on Industrial Engineering and Engineering Management (pp. 65-69). IEEE.
• [54] Liang, S., Liu, Y., Jian, L., Gao, Y., Lin, Z. (2011, August). A utility-based recommendation approach for academic literatures. In 2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology (Vol. 3, pp. 229-232). IEEE.
• [55] Ma, W., Shi, J., Zhao, R. (2017). Normalizing item-based collaborative filter using context-aware Scaled baseline predictor. Mathematical Problems in Engineering, 2017.
• [56] Moghaddam, M.G., Elahian, A. (2014, May). A novel temporal trust-based recommender system. In 2014 22nd Iranian Conference on Electrical Engineering (ICEE) (pp. 1142-1146). IEEE.
• [57] Murakami, T., Mori, K., Orihara, R. (2007, June). Metrics for evaluating the serendipity of recommendation lists. In Annual conference of the Japanese society for artificial intelligence (pp. 40-46). Springer, Berlin, Heidelberg.
• [58] Nakatsuji, M., Fujiwara, Y., Tanaka, A., Uchiyama, T., Fujimura, K., Ishida, T. (2010, October). Classical music for rock fans? Novel recommendations for expanding user interests. In Proceedings of the 19th ACM international conference on Information and knowledge management (pp. 949-958).
• [59] Ng, P.C., She, J., Cheung, M., Cebulla, A. (2016, April). An images-textual hybrid recommender system for vacation rental. In 2016 IEEE Second International Conference on Multimedia Big Data (BigMM) (pp. 60-63). IEEE.
• [60] Pal, A., Parhi, P., Aggarwal, M. (2017, August). An improved content based collaborative filtering algorithm for movie recommendations. In 2017 tenth international conference on contemporary computing (IC3) (pp. 1-3). IEEE.
• [61] Pazzani, M.J., Muramatsu, J., Billsus, D. (1996). Syskil Webert: Identifying interesting web sites. In Proceedings of the national conference on artificial intelligence (Vol. 1, pp. 54-61).
• [62] Pandey, A.K., Rajpoot, D.S. (2016, December). Resolving cold start problem in recommendation system using demographic approach. In 2016 International Conference on Signal Processing and Communication (ICSC) (pp. 213-218). IEEE.
• [63] Prangl, M., Bachlechner, R., Hellwagner, H. (2007, July). A hybrid recommender strategy for personalized utility-based cross-modal multimedia adaptation. In 2007 IEEE International Conference on Multimedia and Expo (pp. 1707-1710). IEEE.
• [64] Pujahari, A., Padmanabhan, V. (2015, December). Group Recommender Systems: Combining user-user and item-item Collaborative filtering techniques. In 2015 International Conference on Information Technology (ICIT) (pp. 148-152). IEEE.
• [65] Ricci, F., Nguyen, Q.N. (2005). Critique-based mobile recommender systems. OEGAI Journal, 24(4), 1-7.
• [66] Ricci, F., Rokach, L., Shapira, B. Kantor, P. (2011) Recommender systems handbook. Springer, Berlin.
• [67] Rogers, S., Flechter, C., Langley, P. (1999). An adaptive interactive agent for route advice. In Proceedings of the third international conference on autonomous agents (Agents’99) (pp. 198-205). Seattle, WA, USA: ACM Press.
• [68] Roshchina, A. (2012). TWIN: Personality-based Recommender System. Institute of Technology Tallaght, Dublin.
• [69] Salman, Y., Abu-Issa, A., Tumar, I., Hassouneh, Y. (2015, October). A proactive multi-type context-aware recommender system in the environment of internet of things. In 2015 IEEE International Conference on Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing (pp. 351-355). IEEE.
• [70] Samin, H., Azim, T. (2019). Knowledge based recommender system for academia using machine learning: A case study on higher education landscape of Pakistan. IEEE Access, 7, 67081-67093.
• [71] Schafer, J.B., Frankowski, D., Herlocker, J., Sen, S. (2007). Collaborative filtering recommender systems. In The adaptive web (pp. 291-324). Springer, Berlin, Heidelberg.
• [72] Shahjalal, M.A., Ahmad, Z., Arefin, M.S., Hossain, M.R.T. (2017, December). A user rating based collaborative filtering approach to predict movie preferences. In 2017 3rd International Conference on Electrical Information and Communication Technology (EICT) (pp. 1-5). IEEE.
• [73] Shani, G., Gunawardana, A. (2009) Evaluating recommender systems. Microsoft Research. https://www.microsoft.com/en-us/research/publication/evaluating-recommender-systems/.
• [74] Shardanand, U., Maes, P. (1995). Social information filtering: Algorithms for automating ’word of mouth’. In CHI’95: Proceedings of the conference of human factors in computing systems. ACM Press.
• [75] Silveira, T., Zhang, M., Lin, X., Liu, Y., Ma, S. (2019). How good your recommender system is? A survey on evaluations in recommendation. International Journal of Machine Learning and Cybernetics, 10(5), 813-831.
• [76] Spiekermann, S. Paraschiv, C. (2002). Motivating Human–Agent Interaction: Transferring Insights from Behavioral Marketing to Interface Design. Electronic Commerce Research. 2. 255-285. 10.2139/ssrn.1346865.
• [77] Supic, H. (2012, October). An approach to integration of contextual information in case-based recommender systems. In 2012 IX International Symposium on Telecommunications (BIHTEL) (pp. 1-5). IEEE.
• [78] Tian, F., Chen, Y., Wang, X., Lan, T., Zheng, Q., Chao, K. M. (2015, October). Common Features Based Volunteer and Voluntary Activity Recommendation Algorithm. In 2015 IEEE 12th International Conference on e-Business Engineering (pp. 43-47). IEEE.
• [79] Thiprak, S., Kurutach, W. (2015, June). Ubiquitous computing technologies and context aware recommender systems for ubiquitous learning. In 2015 12th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON) (pp. 1-6). IEEE.
• [80] Thuan, T.T., Puntheeranurak, S. (2014, October). Hybrid recommender system with review helpfulness features. In TENCON 2014-2014 IEEE Region 10 Conference (pp. 1-5). IEEE.
• [81] Tyagi, S., Bharadwaj, K.K. (2012, December). Trust-enhanced recommender system based on case-based reasoning and collaborative filtering. In 2012 2nd International Conference on Power, Control and Embedded Systems (pp. 1-4). IEEE.
• [82] Uddin, M.F., Banerjee, S., and Lee, J.(2016).Recommender System Framework for Academic Choices: Personality Based Recommendation Engine (PBRE). 2016 IEEE 17th International Conference on Information Reuse and Integration (IRI), Pittsburgh, PA, 2016, pp. 476-483, doi: 10.1109/IRI.2016.70.
• [83] Vargas, S., Baltrunas, L., Karatzoglou, A., Castells, P. (2014, October). Coverage, redundancy and size-awareness in genre diversity for recommender systems. In Proceedings of the 8th ACM Conference on Recommender systems (pp. 209-216).
• [84] Vargas, S., Castells, P. (2011, October). Rank and relevance in novelty and diversity metrics for recommender systems. In Proceedings of the fifth ACM conference on Recommender systems (pp. 109-116).
• [85] Wongkhamchan, T., Namvong, A., Surawanitkun, C. (2019, July). Personalized Recommender System Using A Social Network Based Collaborative Filtering Technique. In 2019 16th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON) (pp. 846-849). IEEE.
• [86] Xie, F., Xu, M., Chen, Z. (2012, March). RBRA: A simple and efficient rating-based recommender algorithm to cope with sparsity in recommender systems. In 2012 26th International Conference on Advanced Information Networking and Applications Workshops (pp. 306-311). IEEE.
• [87] Yakhchi, S., Beheshti, A., Ghafari, S.M., Orgun, M. (2020). Enabling the Analysis of Personality Aspects in Recommender Systems. arXiv preprint arXiv:2001.04825.
• [88] Yu, C., Tang, Q., Liu, Z., Dong, B., Wei, Z. (2018, July). A recommender system for ordering platform based on an improved collaborative filtering algorithm. In 2018 International Conference on Audio, Language and Image Processing (ICALIP) (pp. 298-302). IEEE.
• [89] Zhang, F. (2011, November). Analysis of bandwagon and average hybrid attack model against trust-based recommender systems. In 2011 Fifth International Conference on Management of e-Commerce and e-Government (pp. 269-273). IEEE.
• [90] Zhang, Y.C., Séaghdha, D.Ó., Quercia, D., Jambor, T. (2012, February). Auralist: introducing serendipity into music recommendation. In Proceedings of the fifth ACM international conference on Web search and data mining (pp. 13-22).
• [91] Zhang, Y., Liu, X., Liu, W., Zhu, C. (2016, September). Hybrid recommender system using semi-supervised clustering based on Gaussian mixture model. In 2016 International Conference on Cyberworlds (CW) (pp. 155-158). IEEE.
• [92] Zheng, X.L., Chen, C.C., Hung, J.L., He, W., Hong, F.X., Lin, Z. (2015). A hybrid trust-based recommender system for online communities of practice. IEEE Transactions on Learning Technologies, 8(4), 345-356.
• [93] Zheng, Y., Pu, A. (2018, December). Utility-based multi-stakeholder recommendations by multi-objective optimization. In 2018 IEEE/WIC/ACM International Conference on Web Intelligence (WI) (pp. 128-135). IEEE.
• [94] Zhou, G., Zhu, X., Song, C., Fan, Y., Zhu, H., Ma, X., ... Gai, K. (2018, July). Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery Data Mining (pp. 1059-1068).
• [95] Zielinski, A. (2015, July). A utility-based semantic recommender for technology-enhanced learning. In 2015 IEEE 15th International Conference on Advanced Learning Technologies (pp. 394-396). IEEE.

Uwagi

Opracowanie rekordu ze środków MNiSW, umowa Nr 461252 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2021).