{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T01:02:13Z","timestamp":1774400533761,"version":"3.50.1"},"reference-count":59,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2022,3,7]],"date-time":"2022-03-07T00:00:00Z","timestamp":1646611200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,3,7]],"date-time":"2022-03-07T00:00:00Z","timestamp":1646611200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000923","name":"australian research council","doi-asserted-by":"crossref","award":["DP190101113"],"award-info":[{"award-number":["DP190101113"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Inf Retrieval J"],"published-print":{"date-parts":[[2022,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Query performance prediction (QPP) has been studied extensively in the IR community over the last two decades. A by-product of this research is a methodology to evaluate the effectiveness of QPP techniques. In this paper, we re-examine the existing evaluation methodology commonly used for QPP, and propose a new approach. Our key idea is to model QPP performance as a distribution instead of relying on point estimates. To obtain such distribution, we exploit the scaled Absolute Ranking Error (sARE) measure, and its mean the scaled Mean Absolute Ranking Error (sMARE). Our work demonstrates important statistical implications, and overcomes key limitations imposed by the currently used correlation-based point-estimate evaluation approaches. We also explore the potential benefits of using multiple query formulations and ANalysis Of VAriance (ANOVA) modeling in order to measure interactions between multiple factors. The resulting statistical analysis combined with a novel evaluation framework demonstrates the merits of modeling QPP performance as distributions, and enables detailed statistical ANOVA models for comparative analyses to be created.<\/jats:p>","DOI":"10.1007\/s10791-022-09407-w","type":"journal-article","created":{"date-parts":[[2022,3,7]],"date-time":"2022-03-07T16:06:47Z","timestamp":1646669207000},"page":"94-122","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":25,"title":["sMARE: a new paradigm to evaluate and understand query performance prediction methods"],"prefix":"10.1007","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5070-2049","authenticated-orcid":false,"given":"Guglielmo","family":"Faggioli","sequence":"first","affiliation":[]},{"given":"Oleg","family":"Zendel","sequence":"additional","affiliation":[]},{"given":"J. Shane","family":"Culpepper","sequence":"additional","affiliation":[]},{"given":"Nicola","family":"Ferro","sequence":"additional","affiliation":[]},{"given":"Falk","family":"Scholer","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,3,7]]},"reference":[{"key":"9407_CR1","first-page":"127","volume-title":"Proceedings of the ECIR","author":"G Amati","year":"2004","unstructured":"Amati, G., Carpineto, C., & Romano, G. (2004). Query difficulty, robustness, and selective application of query expansion. In Proceedings of the ECIR (pp. 127\u2013137). Springer."},{"key":"9407_CR2","first-page":"198","volume-title":"Proceedings of the ECIR","author":"JA Aslam","year":"2007","unstructured":"Aslam, J. A., & Pavlu, V. (2007). Query hardness estimation using Jensen-Shannon divergence among multiple scoring functions. In Proceedings of the ECIR (pp. 198\u2013209). Springer."},{"key":"9407_CR3","doi-asserted-by":"crossref","unstructured":"Bailey, P., Moffat, A., Scholer, F., & Thomas, P. (2016) UQV100: A test collection with query variability. In Proceedings of the SIGIR (pp 725\u2013728).","DOI":"10.1145\/2911451.2914671"},{"key":"9407_CR4","doi-asserted-by":"crossref","unstructured":"Bailey, P., Moffat, A., Scholer, F., & Thomas, P. (2017) Retrieval consistency in the presence of query variations. In Proceedings of the SIGIR (pp 395\u2013404).","DOI":"10.1145\/3077136.3080839"},{"issue":"1\u20132","key":"9407_CR5","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1023\/A:1009984519381","volume":"1","author":"D Banks","year":"1999","unstructured":"Banks, D., Over, P., & Zhang, N. F. (1999). Blind men and elephants: Six approaches to TREC data. Information Retrieval, 1(1\u20132), 7\u201334.","journal-title":"Information Retrieval"},{"key":"9407_CR6","doi-asserted-by":"crossref","unstructured":"Benham, R., Culpepper, J. S. (2017) Risk-Reward Trade-offs in Rank Fusion. In Proceedings of the ADCS (vol. 1, pp. 1\u20138).","DOI":"10.1145\/3166072.3166084"},{"issue":"4","key":"9407_CR7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3345001","volume":"37","author":"R Benham","year":"2019","unstructured":"Benham, R., Mackenzie, J., Moffat, A., & Culpepper, J. S. (2019). Boosting search performance using query variations. ACM Transactions on Information Systems, 37(4), 1\u201325.","journal-title":"ACM Transactions on Information Systems"},{"key":"9407_CR8","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-02272-2","volume-title":"Estimating the query difficulty for information retrieval","author":"D Carmel","year":"2010","unstructured":"Carmel, D., & Yom-Tov, E. (2010). Estimating the query difficulty for information retrieval. Morgan & Claypool Publishers."},{"key":"9407_CR9","doi-asserted-by":"crossref","unstructured":"Carmel, D., Yom-Tov, E., Darlow, A., Pelleg, D. (2006). What makes a query difficult? In Proceedings of the SIGIR (pp. 390\u2013397).","DOI":"10.1145\/1148170.1148238"},{"issue":"1","key":"9407_CR10","doi-asserted-by":"publisher","first-page":"4:1","DOI":"10.1145\/2094072.2094076","volume":"30","author":"BA Carterette","year":"2012","unstructured":"Carterette, B. A. (2012). Multiple testing in statistical analysis of systems-based information retrieval experiments. ACM Transactions on Information Systems, 30(1), 4:1-4:34.","journal-title":"ACM Transactions on Information Systems"},{"key":"9407_CR11","doi-asserted-by":"crossref","unstructured":"Chifu, A. G., Laporte, L., Mothe, J., & Ullah, M. Z. (2018) Query performance prediction focused on summarized Letor features. In Proceedings of the SIGIR (pp. 1177\u20131180).","DOI":"10.1145\/3209978.3210121"},{"key":"9407_CR12","doi-asserted-by":"crossref","unstructured":"Cronen-Townsend, S., Zhou, Y., & Croft, W. B. (2002) Predicting query performance. In Proceedings of the SIGIR (pp. 299\u2013306).","DOI":"10.1145\/564376.564429"},{"key":"9407_CR13","doi-asserted-by":"crossref","unstructured":"Cronen-Townsend, S., Zhou, Y., & Croft, W. B. (2004). A language modeling framework for selective query expansion. Tech. rep.: Center for intelligent information retrieval, University of Massachusetts.","DOI":"10.21236\/ADA477592"},{"issue":"1","key":"9407_CR14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3470563","volume":"40","author":"JS Culpepper","year":"2021","unstructured":"Culpepper, J. S., Faggioli, G., Ferro, N., & Kurland, O. (2021). Topic difficulty: Collection and query formulation effects. ACM Transactions on Information Systems, 40(1), 1\u201336.","journal-title":"ACM Transactions on Information Systems"},{"issue":"1","key":"9407_CR15","doi-asserted-by":"publisher","first-page":"2:1","DOI":"10.1145\/2559170","volume":"32","author":"R Cummins","year":"2014","unstructured":"Cummins, R. (2014). Document score distribution models for query performance inference and prediction. ACM Transactions on Information Systems, 32(1), 2:1-2:28.","journal-title":"ACM Transactions on Information Systems"},{"issue":"19","key":"9407_CR16","doi-asserted-by":"publisher","first-page":"9075","DOI":"10.3390\/app11199075","volume":"11","author":"GM Di Nunzio","year":"2021","unstructured":"Di Nunzio, G. M., & Faggioli, G. (2021). A study of a gain based approach for query aspects in recall oriented tasks. Applied Sciences, 11(19), 9075.","journal-title":"Applied Sciences"},{"issue":"2","key":"9407_CR17","doi-asserted-by":"crossref","first-page":"262","DOI":"10.1111\/j.2517-6161.1977.tb01624.x","volume":"39","author":"P Diaconis","year":"1977","unstructured":"Diaconis, P., & Graham, R. L. (1977). Spearman\u2019s footrule as a measure of disarray. Journal of the Royal Statistical Society, 39(2), 262\u2013268.","journal-title":"Journal of the Royal Statistical Society"},{"key":"9407_CR18","doi-asserted-by":"crossref","unstructured":"Diaz, F. (2007). Performance prediction using spatial autocorrelation. In Proceedings of the SIGIR (pp. 583\u2013590).","DOI":"10.1145\/1277741.1277841"},{"key":"9407_CR19","doi-asserted-by":"publisher","DOI":"10.1201\/9780429246593","volume-title":"An introduction to the bootstrap","author":"B Efron","year":"1994","unstructured":"Efron, B., & Tibshirani, R. J. (1994). An introduction to the bootstrap. Chapman and Hall\/CRC."},{"key":"9407_CR20","doi-asserted-by":"crossref","unstructured":"Faggioli, G., & Ferro, N. (2021) System effect estimation by sharding: A comparison between ANOVA approaches to detect significant differences. In Proceedings of the of ECIR (pp. 33\u201346).","DOI":"10.1007\/978-3-030-72240-1_3"},{"key":"9407_CR21","doi-asserted-by":"crossref","unstructured":"Faggioli, G., Zendel, O., Culpepper, J. S., Ferro, N., & Scholer, F. (2021) An enhanced evaluation framework for query performance prediction. In Proceedings of the of ECIR (pp 115\u2013129).","DOI":"10.1007\/978-3-030-72113-8_8"},{"key":"9407_CR22","doi-asserted-by":"crossref","unstructured":"Ferro, N., & Harman, D. (2010). CLEF 2009: Grid@CLEF pilot track overview. In Proceedings of the CLEF (pp. 552\u2013565).","DOI":"10.1007\/978-3-642-15754-7_68"},{"key":"9407_CR23","doi-asserted-by":"crossref","unstructured":"Ferro, N., & Silvello, G. (2016). A general linear mixed models approach to study system component effects. In Proceedings of the SIGIR (pp 25\u201334).","DOI":"10.1145\/2911451.2911530"},{"key":"9407_CR24","doi-asserted-by":"crossref","unstructured":"Ferro, N., Fuhr, N., Maistro, M., Sakai, T., & Soboroff, I. (2019). CENTRE@CLEF 2019. In Proceedings of the ECIR (pp 283\u2013290).","DOI":"10.1007\/978-3-030-15719-7_38"},{"issue":"3","key":"9407_CR25","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1145\/3190580.3190586","volume":"51","author":"N Fuhr","year":"2017","unstructured":"Fuhr, N. (2017). Some common mistakes in IR evaluation, and how they can be avoided. SIGIR Forum, 51(3), 32\u201341.","journal-title":"SIGIR Forum"},{"key":"9407_CR26","volume-title":"Nonparametric statistical inference","author":"JD Gibbons","year":"2011","unstructured":"Gibbons, J. D., & Chakraborti, S. (2011). Nonparametric statistical inference (5th ed.). Chapman & Hall\/CRC.","edition":"5"},{"key":"9407_CR27","doi-asserted-by":"crossref","unstructured":"Hauff, C., Hiemstra, D., de\u00a0Jong, F. (2008). A survey of pre-retrieval query performance predictors. In Proceedings of the CIKM (pp. 1419\u20131420).","DOI":"10.1145\/1458082.1458311"},{"key":"9407_CR28","first-page":"301","volume-title":"Proceedings of the ECIR","author":"C Hauff","year":"2009","unstructured":"Hauff, C., Azzopardi, L., & Hiemstra, D. (2009). The combination and evaluation of query performance prediction methods. In Proceedings of the ECIR (pp. 301\u2013312). Springer."},{"key":"9407_CR29","first-page":"43","volume-title":"Proceedings of the SPIRE","author":"B He","year":"2004","unstructured":"He, B., & Ounis, I. (2004). Inferring query performance using pre-retrieval predictors. In Proceedings of the SPIRE (pp. 43\u201354). Springer."},{"issue":"1\/2","key":"9407_CR30","doi-asserted-by":"publisher","first-page":"81","DOI":"10.2307\/2332226","volume":"30","author":"MG Kendall","year":"1938","unstructured":"Kendall, M. G. (1938). A new measure of rank correlation. Biometrika, 30(1\/2), 81\u201393.","journal-title":"Biometrika"},{"issue":"3","key":"9407_CR31","doi-asserted-by":"publisher","first-page":"239","DOI":"10.1093\/biomet\/33.3.239","volume":"33","author":"MG Kendall","year":"1945","unstructured":"Kendall, M. G. (1945). The treatment of ties in ranking problems. Biometrika, 33(3), 239\u2013251.","journal-title":"Biometrika"},{"key":"9407_CR32","doi-asserted-by":"crossref","unstructured":"Lin, J., Mackenzie, J., Kamphuis, C., Macdonald, C., Mallia, A., Siedlaczek Micha\u0142 Trotman, A., & de Vries, A. (2020). Supporting interoperability between open-source search engines with the common index file format. In Proceedings of the SIGIR (pp. 2149\u20132152).","DOI":"10.1145\/3397271.3401404"},{"key":"9407_CR33","volume-title":"Designing experiments and analyzing data. A model comparison perspective","author":"S Maxwell","year":"2004","unstructured":"Maxwell, S., & Delaney, H. D. (2004). Designing experiments and analyzing data. A model comparison perspective (2nd ed.). Lawrence Erlbaum Associates.","edition":"2"},{"issue":"1","key":"9407_CR34","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1037\/0033-2909.111.1.172","volume":"111","author":"XL Meng","year":"1992","unstructured":"Meng, X. L., Rosenthal, R., & Rubin, D. B. (1992). Comparing correlated correlation coefficients. Psychological Bulletin, 111(1), 172\u2013175.","journal-title":"Psychological Bulletin"},{"key":"9407_CR35","unstructured":"Mothe, J., & Tanguy, L. (2005). Linguistic features to predict query difficulty. In Proceedings of the SIGIR (pp. 7\u201310)."},{"key":"9407_CR36","doi-asserted-by":"crossref","unstructured":"Robertson, S. E., & Kanoulas, E. (2012). On per-topic variance in IR evaluation. In Proceedings of the SIGIR (pp. 891\u2013900).","DOI":"10.1145\/2348283.2348402"},{"key":"9407_CR37","doi-asserted-by":"crossref","unstructured":"Roitman, H. (2018). An extended query performance prediction framework utilizing passage-level information. In Proceedings of the SIGIR (pp. 35\u201342).","DOI":"10.1145\/3234944.3234946"},{"key":"9407_CR38","doi-asserted-by":"crossref","unstructured":"Roitman, H. (2018). Query performance prediction using passage information. In Proceedings of the SIGIR (pp. 893\u2013896).","DOI":"10.1145\/3209978.3210070"},{"key":"9407_CR39","doi-asserted-by":"crossref","unstructured":"Roitman, H. (2020). ICTIR tutorial: Modern query performance prediction: Theory and practice. In Proceedings of the SIGIR (pp. 195\u2013196).","DOI":"10.1145\/3409256.3409813"},{"key":"9407_CR40","doi-asserted-by":"publisher","DOI":"10.1002\/9781118491683","volume-title":"ANOVA and ANCOVA. A GLM approach","author":"A Rutherford","year":"2011","unstructured":"Rutherford, A. (2011). ANOVA and ANCOVA. A GLM approach (2nd ed.). Wiley.","edition":"2"},{"key":"9407_CR41","doi-asserted-by":"crossref","unstructured":"Sakai, T. (2006). Evaluating Evaluation Metrics based on the Bootstrap. In Proceedings of the SIGIR (pp. 525\u2013532).","DOI":"10.1145\/1148170.1148261"},{"issue":"1","key":"9407_CR42","first-page":"1","volume":"54","author":"T Sakai","year":"2020","unstructured":"Sakai, T. (2020). On Fuhr\u2019s guideline for IR evaluation. In Proceedings of the SIGIR, 54(1), 1\u20138.","journal-title":"Proceedings of the SIGIR"},{"key":"9407_CR43","doi-asserted-by":"crossref","unstructured":"Scholer, F., & Garcia, S. (2009). A case for improved evaluation of query difficulty prediction. In Proceedings of the SIGIR (pp. 640\u2013641).","DOI":"10.1145\/1571941.1572055"},{"issue":"7","key":"9407_CR44","first-page":"637","volume":"55","author":"F Scholer","year":"2004","unstructured":"Scholer, F., Williams, H. E., & Turpin, A. (2004). Query association surrogates for web search. Journal of the Association for Information Science and Technology, 55(7), 637\u2013650.","journal-title":"Journal of the Association for Information Science and Technology"},{"key":"9407_CR45","doi-asserted-by":"crossref","unstructured":"Shtok, A., Kurland, O., & Carmel, D. (2010). Using statistical decision theory and relevance models for query-performance prediction. In Proceedings of the SIGIR (pp. 259\u2013266).","DOI":"10.1145\/1835449.1835494"},{"issue":"2","key":"9407_CR46","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2180868.2180873","volume":"30","author":"A Shtok","year":"2012","unstructured":"Shtok, A., Kurland, O., Carmel, D., Raiber, F., & Markovits, G. (2012). Predicting query performance by query-drift estimation. ACM Transactions on Information Systems, 30(2), 1\u201335.","journal-title":"ACM Transactions on Information Systems"},{"issue":"4","key":"9407_CR47","doi-asserted-by":"publisher","first-page":"19:1","DOI":"10.1145\/2926790","volume":"34","author":"A Shtok","year":"2016","unstructured":"Shtok, A., Kurland, O., & Carmel, D. (2016). Query performance prediction using reference lists. ACM Transactions on Information Systems, 34(4), 19:1-19:34.","journal-title":"ACM Transactions on Information Systems"},{"key":"9407_CR48","doi-asserted-by":"crossref","unstructured":"Smucker, M. D., Allan, J., & Carterette, B. A. (2007). A comparison of statistical significance tests for information retrieval evaluation. In Proceedings of the CIKM (pp. 623\u2013632).","DOI":"10.1145\/1321440.1321528"},{"key":"9407_CR49","unstructured":"Tague-Sutcliffe, J. M., & Blustein, J. (1994). A statistical analysis of the TREC-3 data. In Proceedings of the TREC (pp. 385\u2013398)."},{"key":"9407_CR50","doi-asserted-by":"crossref","unstructured":"Tao, Y., & Wu, S. (2014). Query performance prediction by considering score magnitude and variance together. In Proceedings of the CIKM (pp. 1891\u20131894).","DOI":"10.1145\/2661829.2661906"},{"key":"9407_CR51","doi-asserted-by":"crossref","unstructured":"Thomas, P., Scholer, F., Bailey, P., Moffat, A. (2017). Tasks, queries, and rankers in pre-retrieval performance prediction. In Proceedings of the ADCS.","DOI":"10.1145\/3166072.3166079"},{"key":"9407_CR52","unstructured":"Voorhees, E. M. (2004). Overview of the TREC 2004 Robust track. In Proceedings of the TREC."},{"issue":"2","key":"9407_CR53","doi-asserted-by":"publisher","first-page":"12:1","DOI":"10.1145\/3086701","volume":"36","author":"EM Voorhees","year":"2017","unstructured":"Voorhees, E. M., Samarov, D., & Soboroff, I. (2017). Using replicates in information retrieval evaluation. ACM Transactions on Information Systems, 36(2), 12:1-12:21.","journal-title":"ACM Transactions on Information Systems"},{"key":"9407_CR54","doi-asserted-by":"crossref","unstructured":"Zamani, H., Croft, W. B., & Culpepper, J. S. (2018). Neural query performance prediction using weak supervision from multiple signals. In Proceedings of the SIGIR (pp. 105\u2013114).","DOI":"10.1145\/3209978.3210041"},{"key":"9407_CR55","doi-asserted-by":"crossref","unstructured":"Zendel, O., Shtok, A., Raiber, F., Kurland, O., & Culpepper, J. S. (2019). Information needs, queries, and query performance prediction. In Proceedings of the SIGIR (pp. 395\u2013404).","DOI":"10.1145\/3331184.3331253"},{"key":"9407_CR56","doi-asserted-by":"crossref","unstructured":"Zhai, C., & Lafferty, J. (2001). A study of smoothing methods for language models applied to ad hoc information retrieval. In: Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval, New York, NY, USA, SIGIR \u201901 (pp 334\u2013342).","DOI":"10.1145\/383952.384019"},{"key":"9407_CR57","doi-asserted-by":"crossref","unstructured":"Zhao, Y., Scholer, F., & Tsegay, Y. (2008). Effective pre-retrieval query performance prediction using similarity and variability evidence. In Proceedings of the ECIR (pp. 52\u201364).","DOI":"10.1007\/978-3-540-78646-7_8"},{"key":"9407_CR58","doi-asserted-by":"crossref","unstructured":"Zhou, Y., & Croft, W. B. (2006). Ranking robustness: A novel framework to predict query performance. In Proceedings of the CIKM (pp. 567\u2013574).","DOI":"10.1145\/1183614.1183696"},{"key":"9407_CR59","doi-asserted-by":"crossref","unstructured":"Zhou, Y., & Croft, W. B. (2007). Query performance prediction in web search environments. In Proceedings of the SIGIR (pp. 543\u2013550).","DOI":"10.1145\/1277741.1277835"}],"container-title":["Information Retrieval Journal"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10791-022-09407-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10791-022-09407-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10791-022-09407-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,19]],"date-time":"2024-09-19T21:50:21Z","timestamp":1726782621000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10791-022-09407-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,7]]},"references-count":59,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,6]]}},"alternative-id":["9407"],"URL":"https:\/\/doi.org\/10.1007\/s10791-022-09407-w","relation":{},"ISSN":["1386-4564","1573-7659"],"issn-type":[{"value":"1386-4564","type":"print"},{"value":"1573-7659","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,7]]},"assertion":[{"value":"2 July 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 February 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 March 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 September 2022","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Update","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Missing Open Access funding information has been added in the Funding Note","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}}]}}