Collaborative Rankings

• Autorzy: Andrejczuk E. , Rodriguez-Aguilar J. A. , Sierra C.

Identyfikatory

DOI

10.3233/FI-2018-1649

• Języki publikacji: EN

Abstrakty

EN

In this paper we introduce a new ranking algorithm, called Collaborative Judgement (CJ), that takes into account peer opinions of agents and/or humans on objects (e.g. products, exams, papers) as well as peer judgements over those opinions. The combination of these two types of information has not been studied in previous work in order to produce object rankings. Here we apply Collaborative Judgement to the use case of scientific paper assessment and we validate it over simulated data. The results show that the rankings produced by our algorithm improve current scientific paper ranking practice, which is based on averages of opinions weighted by their reviewers’ self-assessments.

Słowa kluczowe

EN

individuality; data analysis; algorithms; algebra; fuzzy algorithms

• Wydawca: IOS Press
• Czasopismo: Fundamenta Informaticae
• Rocznik: 2018
• Tom: Vol. 158, nr 1/3
• Strony: 277--295
• Opis fizyczny: Bibliogr. 20 poz., rys., tab., wykr.

Twórcy

autor

Andrejczuk E.

• Change Management Tool S.L., Barcelona, Spain
• Artificial Intelligence Research Institute (IIIA-CSIC), Bellaterra, Spain
• Universitat Autònoma de Barcelona, Bellaterra, Spain

autor

Rodriguez-Aguilar J. A.

• Artificial Intelligence Research Institute (IIIA-CSIC), Bellaterra, Spain

autor

Sierra C.

• Artificial Intelligence Research Institute (IIIA-CSIC), Bellaterra, Spain

Bibliografia

• [1] Piech C, Huang J, Chen Z, Do C, Ng A, Koller D. Tuned Models of Peer Assessment in MOOCs. Proc. of the 6th International Conference on Educational Data Mining (EDM 2013), 2013. arXiv:1307.2579 [cs.LG].
• [2] de Alfaro L, Shavlovsky M. Crowdgrader: Crowdsourcing the evaluation of homework assignments. Tech. Report 1308.5273, arXiv.org, 2013. doi:10.1145/2538862.2538900.
• [3] Walsh T. The PeerRank Method for Peer Assessment. In: Schaub T, Friedrich G, O’Sullivan B (eds.), ECAI 2014 - 21st European Conference on Artificial Intelligence, 18-22 August 2014, Prague, Czech Republic - Including Prestigious Applications of Intelligent Systems (PAIS 2014), volume 263 of Frontiers in Artificial Intelligence and Applications. IOS Press. 2014, pp. 909-914. ISBN 978-1-61499-418-3.
• [4] Wu J, Chiclana F, Herrera-Viedma E. Trust based consensus model for social network in an incomplete linguistic information context. Applied Soft Computing, 2015;35:827-839. https://doi.org/10.1016/j.asoc.2015.02.023.
• [5] Zhang J, Ghorbani AA, Cohen R. A familiarity-based trust model for effective selection of sellers in multiagent e-commerce systems. Int. J. Inf. Sec., 2007;6(5):333-344. doi:10.1007/s10207-007-0025-y.
• [6] Osman N, Sierra C, McNeill F, Pane J, Debenham JK. Trust and matching algorithms for selecting suitable agents. ACM TIST, 2013;5(1):16. doi:10.1145/2542182.2542198.
• [7] Ramchurn SD, Farinelli A, Macarthur KS, Jennings NR. Decentralized Coordination in RoboCup Rescue. Comput. J., 2010;53(9):1447-1461. doi:10.1093/comjnl/bxq022.
• [8] Nair R, Tambe M, Marsella S. Team Formation for Reformation in Multiagent Domains Like RoboCupRescue. In: Kaminka G, Lima P, Rojas R (eds.), RoboCup 2002: Robot Soccer World Cup VI, volume 2752 of Lecture Notes in Computer Science. Springer Berlin Heidelberg. 2003 pp. 150-161. ISBN: 978-3-540-40666-2.
• [9] Haque M, Egerstedt M, Rahmani A. Multilevel Coalition Formation Strategy for Suppression of Enemy Air Defenses Missions. Journal of Aerospace Information Systems, 2013;10(6):287-296. https://doi.org/10.2514/1.53860.
• [10] Lappas T, Liu K, Terzi E. Finding a Team of Experts in Social Networks. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’09. ACM, New York, NY, USA. 2009 pp. 467-476. ISBN: 978-1-60558-495-9.
• [11] Osman N, Gutierrez P, Sierra C. Trustworthy advice. Knowl.-Based Syst., 2015;82:41-59. https://doi.org/10.1016/j.knosys.2015.02.024.
• [12] Osman N, Sierra C, Sabater-Mir J. Propagation of Opinions in Structural Graphs. In: ECAI 2010 - 19th European Conference on Artificial Intelligence, Lisbon, Portugal, August 16-20, 2010, Proceedings. 2010 pp. 595-600. doi:10.3233/978-1-60750-606-5-595.
• [13] Charlin L, Zemel RS, Boutilier C. A Framework for Optimizing Paper Matching. CoRR, 2012. arXiv: 1202.3706 [cs.IR].
• [14] Fagin R, Kumar R, Mahdian M, Sivakumar D, Vee E. Comparing and Aggregating Rankings with Ties. In: Proceedings of the Twenty-third ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, PODS ’04. ACM, New York, NY, USA. 2004 pp. 47-58. ISBN: 158113858X.
• [15] Fagin R, Kumar R, Mahdian M, Sivakumar D, Vee E. Comparing partial rankings. SIAM Journal on Discrete Mathematics, 2006;20(3):628-648. https://doi.org/10.1137/05063088X.
• [16] Cormen TH, Stein C, Rivest RL, Leiserson CE. Introduction to Algorithms. McGraw-Hill Higher Education, 2nd edition, 2001. ISBN: 0070131511.
• [17] Kamvar SD, Schlosser MT, Garcia-Molina H. The Eigentrust Algorithm for Reputation Management in P2P Networks. In: Proceedings of the 12th International Conference on World Wide Web, WWW ’03. ACM, New York, NY, USA. 2003 pp. 640-651. ISBN: 1-58113-680-3.
• [18] Hill T, P L. Statistics: Methods and Applications. StatSoft, Inc., 2005. ISBN: 10:1884233597, 13:9781884233593.
• [19] Sierra C, Debenham JK. Trust and honour in information-based agency. In: 5th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS 2006), Hakodate, Japan, May 8-12, 2006 pp. 1225-1232. doi:10.1145/1160633.1160855.
• [20] Critchlow DE. Metric Methods for Analyzing Partially Ranked Data. Lecture Notes in Statistics 34. Springer-Verlag New York, 1 edition, 1985. ISBN: 978-0-387-96288-7,978-1-4612-1106-8.

Uwagi

Opracowanie rekordu w ramach umowy 509/P-DUN/2018 ze środków MNiSW przeznaczonych na działalność upowszechniającą naukę (2018).