Інформаційні технології та комп'ютерна техніка
АВТОМАТИЗАЦІЯ ПІДБОРУ НАУКОВИХ РЕЦЕНЗЕНТІВ: ОГЛЯД ЗАДАЧ І МЕТОДІВ
Ключові слова
експертиза
рецензування
задача підбору рецензента
обробка природної мови,
задача призначення
дискретна оптимізація
метрика схожості,
тематичне моделювання
мовні моделі
Анотація
Останнім часом все гостріше постає проблема якісного та своєчасного наукового рецензування. Наукові рецензенти здійснюють експертизу рукописів статей, доповідей конференцій, монографій, дисертацій, запитів на гранти тощо. Наукових рецензентів призначають переважно вручну в умовах дефіциту часу. За великих обсягів рецензування досягти високої якості експертизи вкрай важко. На сьогодні збільшилася кількість досліджень щодо автоматизації підбору рецензентів. При цьому формальна звірка спеціальностей рецензентів та заявок або посимвольне порівняння ключових слів не завжди дозволяють сформувати якісні призначення. В цій роботі здійснено огляд стану сучасних досліджень щодо методів автоматичного призначення наукових рецензентів. Виділяють 3 етапи процесу підбору рецензентів: 1) формування бази рецензентів та структурування інформації про рецензентів та заявки; 2) визначення коефіцієнта схожості між заявкою та рецензентом; 3) розподіл заявок за рецензентами для максимізації агрегованої схожості та комплексного покриття тематики за усіма призначеннями. Розглянуто основні варіанти задачі підбору як одного рецензента, так і колективу рецензентів. Проаналізовано методи для оцінювання відповідності рецензента заявці на основі статистичного аналізу тексту, тематичного моделювання та глибокого навчання. Проаналізовано можливі критерії оптимальності в задачі розподілу рецензентів по заявках, а також обмеження щодо рівня відповідності рецензентів заявці, рівня комплексноcті покриття тематики, справедливих умов експертизи, завантаженості рецензентів та інші. Задача оптимального призначення рецензентів є NP-повною, тому для її розв’язання використовують різноманітні евристичні та мета-евристичні алгоритми. В статті також окреслено перспективи подальших досліджень у сфері автоматичного призначення рецензентів.
Як цитувати
[1]
«АВТОМАТИЗАЦІЯ ПІДБОРУ НАУКОВИХ РЕЦЕНЗЕНТІВ: ОГЛЯД ЗАДАЧ І МЕТОДІВ», Вісник ВПІ, вип. 1, с. 56–64, Лют 2024, doi: 10.31649/1997-9266-2024-172-1-56-64.
Посилання
N. B. Shah, “An Overview of Challenges, Experiments, and Computational Solutions in Peer Review,” Communications of the ACM, vol. 65, no. 6, pp. 76-87, June 2022 https://doi.org/10.1145/3528086 .
S. Price, and P. A. Flach, “Computational Support for Academic Peer Review: A Perspective from Artificial Intelligence,” Communications of the ACM, 2017.
T. Brown, “Peer Review and the Acceptance of New Scientific Ideas: Discussion Paper from a Working Party on Equipping the Public with an Understanding of Peer Review,” Sense About Science, November 2002-May 2004.
А. Т. Ашеров, Диссертации. Экспресс-анализ качества: руководство для экспертов, оппонентов и науч. руководителей, моногр. Украинская инженерно-педагогическая академия. Харків: Кортес-2001, 2008, 53 с.
D. J. Benos, E. Bashari, et al. “The ups and downs of peer review,” Adv Physiol Educ., no. 31(2), pp. 145-152, 2007. https://doi.org/10.1152/advan.00104.2006 .
T. Jefferson, P. Alderson, E. Wager, and F. Davidoff, “ Effects of editorial peer review: a systematic review,” JAMA, no. 287(21), pp. 2784-2786, 2002. https://doi.org/10.1001/jama.287.21.2784 .
L. Bornmann, R. Haunschild, and R. Mutz, “Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases,” Humanit Soc Sci Commun, no. 8, pp. 224, 2021. https://doi.org/10.1057/s41599-021-00903-w .
S. T. Dumais, and J. Nielsen, “Automating the assignment of submitted manuscripts to reviewers,” in Proceedings of the Fifteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 1992, pp. 233-244. Publ by ACM. https://doi.org/10.1145/133160.133205 .
A. C. Ribeiro, A. Sizo, and L. P. Reis, “Investigating the reviewer assignment problem: A systematic literature review,” Journal of Information Science, 2023. https://doi.org/10.1177/01655515231176668 .
Meltem Aksoy, Seda Yanik, and Mehmet Fatih, Amasyali. Reviewer Assignment Problem: A Systematic Review of the Literature. J. Artif. Int. Res. 76, May 2023. https://doi.org/10.1613/jair.1.14318 .
F. Wang, N. Shi, and B. Chen, “A comprehensive survey of the reviewer assignment problem,” International Journal of Information Technology and Decision Making, no. 9 (4), pp. 645-668, 2010. https://doi.org/10.1142/S0219622010003993 .
F. Wang, B. Chen, and Z. Miao, “A survey on reviewer assignment problem,” Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 5027 LNAI(60704048), pp. 718-72, 2008. https://doi.org/10.1007/978-3-540-69052-8_75 .
S. Tan, Z. Duan, S. Zhao, J. Chen, and Y. Zhang, “Improved reviewer assignment based on both word and semantic features,” Information Retrieval Journal, no. 24 (3), pp. 175-204, 2021. https://doi.org/10.1007/s10791-021-09390-8 .
D. Yarowsky, and R. Florian, “Taking the load off the conference chairs: Towards a digital paper-routing assistant,” in Proceedings of the 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, EMNLP 1999, pp. 220-230, 1999. Association for Computational Linguistics (ACL).
M. Karimzadehgan, C. X. Zhai, and G. Belford, “Multi-aspect expertise matching for review assignment,” in Proceedings of International Conference on Information and Knowledge Management, 2008, pp. 1113-1122, https://doi.org/10.1145/1458082.1458230 .
M. Mirzaei, J. Sander, and E. Stroulia, “Multi-Aspect Review-Team Assignment using Latent Research Areas,” Information Processing and Management, no. 56 (3), pp. 858-878, 2019. https://doi.org/10.1016/j.ipm.2019.01.007 .
D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent Dirichlet allocation,” Journal of Machine Learning Research, no. 3 (4-5), pp. 993-1022, 2003. https://doi.org/10.7551/mitpress/1120.003.0082 .
E. Ekinci, and S. I. Omurca, “NET-LDA: A novel topic modeling method based on semantic document similarity,” Turkish Journal of Electrical Engineering and Computer Sciences, no. 28 (4), pp. 2244-2260,2020. https://doi.org/10.3906/ELK-1912-62 .
O. Anjum, H. Gong, S. Bhat, J. Xiong, and W. M. Hwu, “Pare: A paper-reviewer matching approach using a common topic space,” in EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, pp. 518-528, 2019. Association for Computational Linguistics. https://doi.org/10.18653/v1/d19-1049 .
С. Д. Штовба, М. В. Петричко, і М. Ю. Петранова, «Метрика схожості категоріальних розподілів, що враховує спорідненість різних категорій,» Вісник Вінницького політехнічного інституту, № 2 (167), с. 49-57, 2023. https://doi.org/10.31649/1997-9266-2023-167-2-49-57 .
T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations ofwords and phrases and their compositionality,” in Advances in Neural Information Processing Systems. Neural information processing systems foundation, 2013.
J. Pennington, R.Socher, and C. D. Manning, “GloVe: Global vectors for word representation,” in EMNLP 2014–2014 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, 2014, pp. 1532-1543. Association for Computational Linguistics (ACL). https://doi.org/10.3115/v1/d14-1162 .
P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov,. “Enriching Word Vectors with Subword Information,” Transactions of the Association for Computational Linguistics, no. 5, pp. 135-146, 2017.
J. Howard, and S.Ruder, “Universal Language Model Fine-tuning for Text Classification.” ACL, Association for Computational Linguistics. 2018. http:// arxiv.org/abs/1801.06146 .
J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” arXiv Preprint. arXiv:1810.04805v2. 1-16. 2018.
R. Alec, W. Jeffrey, C. Rewon, L. David, A. Dario, and S. Ilya, “Language Models are Unsupervised Multitask Learners | Enhanced Reader,” OpenAI Blog, no. 1(8), p. 9, 2019.
V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108. 2019.
C. Sun,, K. Ng, T. J. Henville, P., and R. Marchant, “Hierarchical word mover distance for collaboration recommender system, in Communications in Computer and Information Science, vol. 996, pp. 289-302, 2019. Springer Verlag. https://doi.org/10.1007/978-981-13-6661-1_23 .
X. Kong, H. Jiang, Z. Yang, Z. Xu, F. Xia, and A. Tolba, “Exploiting Publication Contents and Collaboration Networks for Collaborator Recommendation,” PLoS ONE , no. 11(2): e0148492. 2016. https://doi.org/10.1371/journal.pone.0148492 .
Y. Zhao, J. Tang, and Z. Du, “EFCNN: A Restricted Convolutional Neural Network for Expert Finding,” in: Yang Q., Zhou Z. H., Gong Z., Zhang M. L., Huang S. J., Eds Advances in Knowledge Discovery and Data Mining. PAKDD 2019. Lecture Notes in Computer Science, vol. 11440, 2019. Springer, Cham. https://doi.org/10.1007/978-3-030-16145-3_8 .
D. G. Cattrysse, and L. N. Van Wassenhove, “A survey of algorithms for the generalized assignment problem,” European Journal of Operational Research,no. 60 (3), pp. 260-272, 1992. https://doi.org/10.1016/0377-2217(92)90077-M .
C. Long, R. C. W. Wong, Y. Peng, and L. Ye, “On good and fair paper-reviewer assignment,” in Proceedings IEEE International Conference On Data Mining, 2013, pp. 1145-1150. https://doi.org/10.1109/ICDM.2013.13 .
J. Goldsmith, and R. H. Sloan, “The AI conference paper assignment problem,” in AAAI Workshop – Technical Report, vol. WS-07-10, pp. 53-57, 2007.
W. D.Cook, B. Golany, M. Kress, M. Penn, and Raviv, “Optimal allocation of proposals to reviewers to facilitate effective ranking,” Management Science, no. 51 (4), pp. 655-661, 2005. https://doi.org/10.1287/mnsc.1040.0290 .
T. Kolasa, and D. Krol, “A survey of algorithms for paper-reviewer assignment problem,” IETE Technical Review (Institution of Electronics and Telecommunication Engineers, India, no. 28 (2), pp. 123-134, 2011. https://doi.org/10.4103/0256-4602.78092 .
L. Charlin, R. Zemel, and C. Boutilier, “A framework for optimizing paper matching,” in Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence, pp. 86–95, 2012.
L. Charlin, and R. S. Zemel, “The Toronto Paper Matching System: An automated paper-reviewer assignment system,” in ICML Workshop on Peer Reviewing and Publishing Models (PEER), vol. 28, 2013.
D. K. Tayal, P. C. Saxena, A. Sharma, G. Khanna, and S. Gupta, “New method for solving reviewer assignment problem using type-2 fuzzy sets and fuzzy functions,” Applied Intelligence, no. 40 (1), pp, 54-73, 2014. https://doi.org/10.1007/s10489-013-0445-5 .
M. Karimzadehgan, and C. X. Zhai, “Integer linear programming for Constrained Multi-Aspect Committee Review Assignment,” Information Processing and Management, no. 48 (4), pp. 725-740, 2012. https://doi.org/10.1016/j.ipm.2011.09.004 .
M. Karimzadehgan, and C. X. Zhai, “|Constrained multi-aspect expertise matching for committee review assignment,” in International Conference on Information and Knowledge Management, Proceedings, 2009, pp. 1697-1700. https://doi.org/10.1145/1645953.1646207 .
W. Tang, J. Tang, T. Lei, C. Tan, B. Gao, and T. Li, “On optimization of expertise matching with various constraints,” Neurocomputing, no. 76 (1), pp. 71-83, 2012. https://doi.org/10.1016/j.neucom.2011.04.039 .
W. Tang, J. Tang, and C. Tan, “Expertise matching via constraint-based optimization,” in Proceedings–2010 IEEE/WIC/ACM International Conference on Web Intelligence, WI 2010, vol. 1, pp. 34-41, 2010. https://doi.org/10.1109/WI-IAT.2010.133 .
N. Xue, J. X. Hao, S. L. Jia, and Q. Wang, “An interval fuzzy ontology-based peer review assignment method,” in Proceedings of IEEE International Conference on E-Business Engineering, 2012, 55-60. https://doi.org/10.1109/ICEBE.2012.19 .
A. Kale, R. Kharat, S. Bodkhe, P. Apte, and H. Dhonde, “Automated fair paper reviewer assignment for conference management system,” in Proceedings of 1st International Conference on Computing, Communication, Control and Automation, ICCUBEA 2015, 2015, pp. 408-411. https://doi.org/10.1109/ICCUBEA.2015.85 .
D. Hartvigsen, J. C. Wei, and R. Czuchlewski, “The conference paper-reviewer assignment problem” Decision Sciences, vol. 30, pp. 865-876, 1999. Decision Sciences Institute. https://doi.org/10.1111/j.1540-5915.1999.tb00910.x .
N. Garg, T. Kavitha, A. Kumar, K. Mehlhorn, and J. Mestre, “Assigning papers to referees,” Algorithmica (New York), no. 58 (1), pp. 119-136, 2010. https://doi.org/10.1007/s00453-009-9386-0.
I. Stelmakh, N. B. Shah, and A. Singh, “PeerReview4All: Fair and Accurate Reviewer Assignment,” in Peer Review. arXiv preprint, arXiv:1806.06237, 2019. https://arxiv.org/abs/1806.06237 .
A. Kobren, B. Saha, and A. McCallum, “Paper matching with local fairness constraints,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19, 2019, pp. 1247-1257, New York, USA. ACM.
J. Pay and Y. Zick, “I Will Have Order!” Optimizing Orders for Fair Reviewer Assignment. arXiv preprint, arXiv:2108.02126, 2021. https://doi.org/10.48550/arXiv.2108.02126.
K. Dhull, S.J ecmen, P. Kothari, and N. B. Shah, “Strategyproofing Peer Assessment via Partitioning,” The Price in Terms of Evaluators’ Expertise. arXiv preprint, arXiv:2201.10631, 2022. https://doi.org/10.48550/arXiv.2201.10631 .
N. Alon, F. Fischer, A. Procaccia, and M. Tennenholtz, “Sum of us: Strategyproof selection from the selectors,” in Proceedings of the 13th Conference on Theoretical Aspects of Rationality and Knowledge, 2011, pp. 101-110.
S. Jecmen, H. Zhang, R. Liu, N. B. Shah, V. Conitzer, and F. Fang, “Mitigating Manipulation,” in Peer Review via Randomized Reviewer Assignments. arXiv preprint arXiv:2006.16437, 2020. http://arxiv.org/abs/2006.16437 .
L. Guo, J. Wu, W. Chang, J. Wu, and J. Li, “K-Loop Free Assignment,” in Conference Review Systems. In 2018 International Conference on Computing, Networking and Communications, ICNC 2018, 2018, pp. 542-547. Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/ICCNC.2018.8390343 .
N. Boehmer, R. Bredereck, and A. Nichterlein, “Combating Collusion Rings is Hard but Possible,” arXiv preprint, arXiv:2112.08444, 2021. https://doi.org/10.48550/arXiv.2112.08444 .
K. Leyton-Brown, Mausam, Y. Nandwani, H. Zarkoob, C. Cameron, N. Newman, and D. Raghu, “Matching Papers and Reviewers at Large Conferences,” arXiv preprint, arXiv:2202.12273, 2022. https://arxiv.org/abs/2202.12273 .
T. Silva, M. Jian, and Y. Chen, “Process analytics approach for R&D project selection.” ACM Transactions on Management Information Systems, no. 5(4), 2014. https://doi.org/10.1145/2629436 .
C. J. Taylor, “On the optimal assignment of conference papers to reviewers,” Technical Report MS-CIS-08-30, University of Pennsylvania, 2008.
J. J. Merelo-Guervós, and P. Castillo-Valdivieso,” Conference paper assignment using a combined greedy/evolutionary algorithm,” Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), no. 3242, pp. 602-611, 2004. https://doi.org/10.1007/978-3-540-30217-9_61 .
S. L. Janak, M. S., Taylor, C. A. Floudas, M. K. Burka, and T. J. Mountziaris, “A novel and effective integer optimization approach for the Nsf panel assignment problem: A multi-resource and preference-constrained generalized assignment problem,” in Proceedings of AIChE Annual Meeting, no. 609, pp. 6066, 2005.
N. N. Li, Z. Zhao, J. H. Gu, and B. Y. Liu,. “Ant colony optimization algorithm for expert assignment problem,” in Proceedings of the 7th International Conference on Machine Learning and CyberneticsI 2008, July, pp. 660-664. https://doi.org/10.1109/ICMLC.2008.4620487 .
D. Conry, Y. Koren, and N. Ramakrishnan, “Recommender systems for the conference paper assignment problem,” in Proceedings of the 3rd ACM Conference on Recommender Systems, 2009, pp. 357-360. https://doi.org/10.1145/1639714.1639787.
B. Kat, “An algorithm and a decision support system for the panelist assignment problem: The case of TUBITAK,” Journal of the Faculty of Engineering and Architecture of Gazi University, no. 36 (1), pp. 69-87, 2021. https://doi.org/10.17341/gazimmfd.631071 .
F. Wang, S. Zhou, and N. Shi, “Group-to-group reviewer assignment problem,” Computers and Operations Research, no. 40 (5), pp. 1351-1362, 2013. https://doi.org/10.1016/j.cor.2012.08.005 .
D. K. Pradhan, J. Chakraborty, P. Choudhary, and S. Nandi, “An automated conflict of interest based greedy approach for conference paper assignment system,” Journal of Informetrics, no. 14 (2), 2020. https://doi.org/10.1016/j.joi.2020.101022 .
Y. Xu, J. Ma, Y. Sun, G. Hao, W. Xu, and D. Zhao, “A decision support approach for assigning reviewers to proposals,” Expert Systems with Applications, no. 37 (10), pp. 6948-6956, 2010. https://doi.org/10.1016/j.eswa.2010.03.027 .
N. N. Li, J. N. Zhang, J. H. Gu, and B. Y. Liu, “Solving expert assignment problem using improved genetic algorithm,” in Proceedings of the Sixth International Conference on Machine Learning and Cybernetics, 2007, 2 August, pp. 934-937. https://doi.org/10.1109/ICMLC.2007.4370276 .
J. Li, J. Peng, and Y. Wei, Adaptive parallel genetic algorithm for expert assignment problem. In Proceedings of 6th International Symposium on Computational Intelligence and Design, no. 1, pp. 23-26, 2013. https://doi.org/10.1109/ISCID.2013.13 .
C. Yang, T. Liu, W. Yi, X. Chen, and B. Niu, “Identifying expertise through semantic modeling: A modified BBPSO algorithm for the reviewer assignment problem,” Applied Soft Computing Journal, no. 94, pp. 106483, 2020. https://doi.org/10.1016/j.asoc.2020.106483 .
M. Huang, B. Liu, and L. Hong, “On assigning papers to reviewers,” 2nd International Workshop on Database Technology and Applications, 2010. https://doi.org/10.1109/DBTA.2010.5659009 .
