{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,26]],"date-time":"2026-06-26T07:49:00Z","timestamp":1782460140009,"version":"3.54.5"},"reference-count":114,"publisher":"Institute for Operations Research and the Management Sciences (INFORMS)","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information Systems Research"],"published-print":{"date-parts":[[2026,6]]},"abstract":"<jats:p>As businesses increasingly rely on large language models (LLMs) for tasks such as customer service and information retrieval, ensuring the accuracy of their responses is a critical challenge. Traditional verification is costly, slow, and often requires scarce domain experts. We introduce the automated quality evaluation based on textual responses (AQER) framework, a novel, cost-effective method to assess the correctness of free-text answers from both LLMs and human workers without needing preexisting correct answers. AQER works by intelligently aggregating multiple responses to the same question, leveraging the wisdom of the crowd to create a reliable synthetic correct answer, followed by an iterative procedure that accounts for response quality cues. AQER obtains state-of-the-art performance compared with existing automated response evaluation baselines. For managers AQER offers a scalable, data-driven method to (i) evaluate and select the best performing LLMs for specific organizational needs and use cases, (ii) continuously monitor artificial intelligence (AI) performance to ensure reliability and accountability across different model versions, and (iii) manage the quality of crowd workers essential for high-quality AI training and validation. AQER, thus, offers a robust mechanism for improving model performance and mitigating the significant financial and reputational risks associated with deploying untrustworthy generative AI technologies.<\/jats:p>","DOI":"10.1287\/isre.2023.0426","type":"journal-article","created":{"date-parts":[[2025,9,23]],"date-time":"2025-09-23T14:33:15Z","timestamp":1758637995000},"page":"782-804","source":"Crossref","is-referenced-by-count":0,"title":["Quality Control for Crowd Workers and for Language Models: A Framework for Free-Text Response Evaluation with No Ground Truth"],"prefix":"10.1287","volume":"37","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1513-017X","authenticated-orcid":false,"given":"Inbal","family":"Yahav","sequence":"first","affiliation":[{"name":"Coller School of Management, Tel-Aviv University, Tel Aviv-Yafo 6997801, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1038-762X","authenticated-orcid":false,"given":"Anat","family":"Goldstein","sequence":"additional","affiliation":[{"name":"Department of Industrial Engineering and Management, Ariel University, Ariel 4070000, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4501-3414","authenticated-orcid":false,"given":"Tomer","family":"Geva","sequence":"additional","affiliation":[{"name":"Coller School of Management, Tel-Aviv University, Tel Aviv-Yafo 6997801, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shahar","family":"Meir","sequence":"additional","affiliation":[{"name":"Coller School of Management, Tel-Aviv University, Tel Aviv-Yafo 6997801, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9594-7819","authenticated-orcid":false,"given":"Onn","family":"Shehory","sequence":"additional","affiliation":[{"name":"School of Business Administration, Bar-Ilan University, Ramat Gan 5290002, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"109","reference":[{"key":"B1","doi-asserted-by":"publisher","DOI":"10.1287\/isre.2024.editorial.v35.n2"},{"key":"B2","unstructured":"Abedissa T, Usbeck R, Assabie Y (2023) AmQA: Amharic question answering dataset. Preprint, submitted March 6, https:\/\/arxiv.org\/abs\/2303.03290."},{"key":"B3","doi-asserted-by":"crossref","unstructured":"Abualsaud M, Chen IX, Ghajar K, Minh LNL, Smucker MD, Tahami AV, Zhang D (2021) UWaterlooMDS at the TREC 2021 health misinformation track.\n                      30th Text REtrieval Conf. Proc.\n                      (National Institute of Standards and Technology, Gaithersburg, MD).\u200f","DOI":"10.6028\/NIST.SP.500-335.misinfo-UWaterlooMDS"},{"key":"B4","doi-asserted-by":"crossref","unstructured":"Alfonseca E, P\u00e9rez D (2004) Automatic assessment of open ended questions with a BLEU-inspired algorithm and shallow NLP. Vicedo JL, Mart\u00ednez-Barco P, Mu\u0144oz R, Saiz Noeda M, eds.\n                      Advances in Natural Language Processing. EsTAL 2004\n                      , Lecture Notes in Computer Science, vol. 3230 (Springer, Berlin, Heidelberg).","DOI":"10.1007\/978-3-540-30228-5_3"},{"key":"B5","unstructured":"Awasthi P, Blum A, Haghtalab N, Mansour Y (2017) Efficient PAC learning from the crowd.\n                      Proc. 2017 Conf. Learn. Theory\n                      , vol. 65 (PMLR, New York), 127\u2013150."},{"key":"B6","unstructured":"Bajaj P, Campos D, Craswell N, Deng L, Gao J, Liu X, Majumder R, et al. (2016) MS MARCO: A human generated machine reading comprehension dataset. Preprint, submitted November 28, https:\/\/arxiv.org\/abs\/1611.09268."},{"key":"B7","doi-asserted-by":"crossref","unstructured":"Berant J, Chou A, Frostig R, Liang P (2013) Semantic parsing on freebase from question-answer pairs.\n                      Proc. 2013 Conf. Empirical Methods Natural Language Processing\n                      (Association for Computational Linguistics, Stroudsburg, PA), 1533\u20131544.","DOI":"10.18653\/v1\/D13-1160"},{"key":"B8","volume-title":"Pattern Recognition and Machine Learning","author":"Bishop CM","year":"2006"},{"key":"B9","doi-asserted-by":"crossref","unstructured":"Bondarenko A, Fr\u00f6be M, Kasturia V, Hagen M, V\u00f6lske M, Stein B (2019) Webis at TREC 2019: Decision Track.\n                      TREC 2019 Proc.\n                      (National Institute of Standards and Technology, Gaithersburg, MD).","DOI":"10.6028\/NIST.SP.1250.decisions-Webis"},{"key":"B10","doi-asserted-by":"crossref","unstructured":"Bonthu S, Rama Sree S, Krishna Prasad MHM (2021) Automated short answer grading using deep learning: A survey.\n                      Internat. Cross-Domain Conf. Machine Learn. Knowledge Extraction\n                      (Springer, Berlin, Heidelberg), 61\u201378.","DOI":"10.1007\/978-3-030-84060-0_5"},{"key":"B12","doi-asserted-by":"crossref","unstructured":"Brand C, Ganian R, Simonov K (2023) A parameterized theory of PAC learning.\n                      Proc. AAAI Conf. Artificial Intelligence\n                      , vol. 37 (AAAI Press, Palo Alto, CA), 6834\u20136841.","DOI":"10.1609\/aaai.v37i6.25837"},{"key":"B13","doi-asserted-by":"crossref","unstructured":"Branson S, van Horn G, Perona P (2017) Lean crowdsourcing: Combining humans and machines in an online system.\n                      Proc. IEEE Conf. Comput. Vision Pattern Recognition\n                      (IEEE, Piscataway, NJ), 7474\u20137483.","DOI":"10.1109\/CVPR.2017.647"},{"key":"B14","doi-asserted-by":"crossref","unstructured":"Braylan A, Lease M (2020) Modeling and aggregation of complex annotations via annotation distances.\n                      Proc. Web Conf\n                      . (Association for Computing Machinery, New York), 1807\u20131818.","DOI":"10.1145\/3366423.3380250"},{"key":"B15","doi-asserted-by":"crossref","unstructured":"Braylan A, Alonso O, Lease M (2022) Measuring annotator agreement generally across complex structured, multi-object, and free-text annotation tasks.\n                      Proc. ACM Web Conf.\n                      (Association for Computing Machinery, New York), 1720\u20131730.","DOI":"10.1145\/3485447.3512242"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.1007\/s40593-014-0026-8"},{"key":"B17","unstructured":"Cer D, Yang Y, Kong SY, Hua N, Limtiaco N, St. John R, Constant N, et al. (2018) Universal sentence encoder. Preprint, submitted March 29, https:\/\/arxiv.org\/abs\/1803.11175."},{"key":"B18","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2022.07.001"},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.1145\/3641289"},{"key":"B20","unstructured":"Chen X, Aksitov R, Alon U, Ren J, Xiao K, Yin P, Prakash S, Sutton C, Wang X, Zhou D (2023) Universal self-consistency for large language model generation. Preprint, submitted November 29, https:\/\/arxiv.org\/abs\/2311.17311."},{"key":"B21","unstructured":"Chiang CH, Lee HY (2023) Can large language models be an alternative to human evaluations? Preprint, submitted May 3, https:\/\/arxiv.org\/abs\/2305.01937."},{"key":"B22","doi-asserted-by":"crossref","unstructured":"Clarke CLA, Maistro M, Smucker MD (2022) Overview of the TREC 2021 health misinformation track.\n                      Proc. Thirtieth Text Retrieval Conf. (TREC 2021)\n                      , Special Publication 500-335 (National Institute of Standards and Technology (NIST), Washington, DC).","DOI":"10.6028\/NIST.SP.500-335.misinfo-overview"},{"key":"B23","doi-asserted-by":"crossref","unstructured":"Clarke CL, Rizvi S, Smucker MD, Maistro M, Zuccon G (2020) Overview of the TREC 2020 health misinformation track.\n                      TREC 2020 Proc\n                      .\u200f","DOI":"10.6028\/NIST.SP.1266.misinfo-overview"},{"key":"B24","doi-asserted-by":"crossref","unstructured":"Dalvi N, Dasgupta A, Kumar R, Rastogi V (2013) Aggregating crowdsourced binary ratings.\n                      Proc. 22nd Internat. Conf. World Wide Web\n                      , 285\u2013294.","DOI":"10.1145\/2488388.2488414"},{"key":"B25","unstructured":"Dam SK, Hong CS, Qiao Y, Zhang C (2024) A Complete Survey on LLM-based AI Chatbots. Preprint, submitted June 17, https:\/\/arxiv.org\/abs\/2406.16937."},{"issue":"1","key":"B27","first-page":"20","volume":"28","author":"Dawid AP","year":"1979","journal-title":"J. Roy. Statist. Soc. Ser. C Appl. Statist."},{"key":"B28","unstructured":"Dekel O, Shamir O (2009) Vox populi: Collecting high-quality labels from a crowd.\n                      22nd Annual Conf. Learn. Theory (COLT) Proc."},{"key":"B29","unstructured":"Devlin J, Chang MW, Lee K, Toutanova K (2018) BERT: Pre-training of deep bidirectional transformers for language understanding. Preprint, submitted October 11, https:\/\/arxiv.org\/abs\/1810.04805."},{"key":"B30","doi-asserted-by":"crossref","unstructured":"d\u2019Hoffschmidt M, Belblidia W, Brendl\u00e9 T, Heinrich Q, Vidal M (2020) FQuAD: French question answering dataset. Preprint, submitted February 14, https:\/\/arxiv.org\/abs\/2002.06071.","DOI":"10.18653\/v1\/2020.findings-emnlp.107"},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.2021.03357"},{"key":"B34","unstructured":"Dzikovska MO, Nielsen R, Brew C (2012) Towards effective tutorial feedback for explanation questions: A dataset and baselines.\n                      Proc. 2012 Conf. North Amer. Chapter Assoc. Comput. Linguistics Human Language Tech.\n                      (ACL, Stroudsburg, PA), 200\u2013210."},{"key":"B35","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2022.105211"},{"key":"B36","doi-asserted-by":"crossref","unstructured":"Fern\u00e1ndez-Pichel M, Losada DE, Pichel JC, Elsweiler D (2020) CiTIUS at the TREC 2020 health misinformation track.\n                      TREC 2020 Proc\n                      .","DOI":"10.6028\/NIST.SP.1266.misinfo-CiTIUS"},{"key":"B37","doi-asserted-by":"crossref","unstructured":"Galhardi LB, Brancher JD (2018) Machine learning approach for automatic short answer grading: A systematic review.\n                      Ibero-Amer. Conf. Artificial Intelligence\n                      (Springer, Cham, Switzerland), 380\u2013391.","DOI":"10.1007\/978-3-030-03928-8_31"},{"key":"B38","doi-asserted-by":"crossref","unstructured":"Gao T, Yao X, Chen D (2021) SimCSE: Simple contrastive learning of sentence embeddings. Preprint, submitted April 18, https:\/\/arxiv.org\/abs\/2104.08821.","DOI":"10.18653\/v1\/2021.emnlp-main.552"},{"key":"B39","unstructured":"Geva T, Saar-Tsechansky M (2016) Who\u2019s a good decision maker? Data-driven expert worker ranking under unobservable quality.\n                      Proc. 37th Internat. Conf. Inform. Systems\n                      (Association for Information Systems, Atlanta)."},{"key":"B40","doi-asserted-by":"publisher","DOI":"10.1111\/poms.13260"},{"key":"B41","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-019-00637-z"},{"issue":"11","key":"B42","volume":"3","author":"Gomaa WH","year":"2012","journal-title":"Internat. J. Advanced Comput. Sci. Appl."},{"key":"B43","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00454"},{"issue":"1","key":"B44","volume":"3","author":"G\u00fctl C","year":"2008","journal-title":"Internat. J. Emerging Tech. Learn."},{"key":"B45","doi-asserted-by":"crossref","unstructured":"Hadi MU, Qureshi R, Shah A, Irfan M, Zafar A, Shaikh MB, Akhtar N, Wu J, Mirjalili S, Shah M (2025) Large language models: A comprehensive survey of its applications, challenges, limitations, and future prospects. Preprint, submitted February 10, http:\/\/dx.doi.org\/10.36227\/techrxiv.23589741.v8.","DOI":"10.36227\/techrxiv.23589741.v8"},{"key":"B46","unstructured":"Haller S, Aldea A, Seifert C, Strisciuglio N (2022) Survey on automated short answer grading with deep learning: From word embeddings to transformers. Preprint, submitted March 11, https:\/\/arxiv.org\/abs\/2204.03503."},{"key":"B47","doi-asserted-by":"crossref","unstructured":"Hanneke S, Green Larsen K, Zhivotovskiy N (2024) Revisiting agnostic PAC learning.\n                      Proc. 65th IEEE Annual Sympos. Foundations Comput. Sci\n                      . 1968\u20131982.","DOI":"10.1109\/FOCS61266.2024.00118"},{"key":"B48","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-84858-7"},{"key":"B49","doi-asserted-by":"crossref","unstructured":"Heilman M, Madnani N (2015) The impact of training data on automated short answer scoring performance.\n                      Proc. 10th Workshop Innovative Use NLP Building Ed. Appl.\n                      (ACL, Stroudsburg, PA), 81\u201385.","DOI":"10.3115\/v1\/W15-0610"},{"key":"B50","doi-asserted-by":"crossref","unstructured":"Heinecke S, Reyzin L (2019) Crowdsourced PAC learning under classification noise.\n                      Proc. Seventh AAAI Conf. Human Comput. Crowdsourcing\n                      (AAAI, Palo Alto, CA), 41\u201349.","DOI":"10.1609\/hcomp.v7i1.5279"},{"key":"B51","doi-asserted-by":"publisher","DOI":"10.2307\/25148625"},{"key":"B52","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1963.10500830"},{"key":"B53","doi-asserted-by":"crossref","unstructured":"Horbach A, Pinkal M (2018) Semi-supervised clustering for short answer scoring.\n                      Proc. 11th Internat. Conf. Language Resources Evaluation\n                      (ACL, Stroudsburg, PA).","DOI":"10.63317\/45z7js2bo39m"},{"key":"B54","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-013-0306-1"},{"key":"B55","volume-title":"Proc. 15th Internat. Comput. Assisted Assessment Conf.","author":"Jordan S","year":"2012"},{"key":"B56","doi-asserted-by":"crossref","unstructured":"Joshi M, Choi E, Weld DS, Zettlemoyer L (2017) TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. Preprint, submitted May 9, https:\/\/arxiv.org\/abs\/1705.03551.","DOI":"10.18653\/v1\/P17-1147"},{"key":"B57","unstructured":"Karchmer A (2024) Agnostic membership query learning with nontrivial savings: New results and techniques.\n                      Proc. 35th Internat. Conf. Algorithmic Learn. Theory. Proc. Machine Learn. Res\n                      ., vol. 237 (PMLR, New York), 654\u2013682."},{"key":"B122","doi-asserted-by":"crossref","unstructured":"Karger DR, Oh S, Shah D (2014) Budget-optimal task allocation for reliable crowdsourcing systems.\n                      Oper. Res.\n                      62(1):1\u201324.","DOI":"10.1287\/opre.2013.1235"},{"key":"B58","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/3897.001.0001"},{"key":"B59","unstructured":"Khetan A, Lipton ZC, Anandkumar A (2017) Learning from noisy singly-labeled data. Preprint, submitted December 13, https:\/\/arxiv.org\/abs\/1712.04577."},{"key":"B60","doi-asserted-by":"crossref","unstructured":"Klein R, Kyrilov A, Tokman M (2011) Automated assessment of short free-text responses in computer science using latent semantic analysis.\n                      Proc. 16th Annual Joint Conf. Innovation Tech. Comput. Sci. Ed.\n                      (ACM, New York), 158\u2013162.","DOI":"10.1145\/1999747.1999793"},{"key":"B121","doi-asserted-by":"crossref","unstructured":"Kittur A, Nickerson JV, Bernstein M, Gerber E, Shaw A, Zimmerman J, Lease M, Horton J (2013) The future of crowd work.\n                      Proc. 2013 Conf. Comput. Supported Cooperative Work\n                      (Association for Computing Machinery, New York), 1301\u20131318.","DOI":"10.1145\/2441776.2441923"},{"key":"B61","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00023"},{"key":"B63","unstructured":"Kumar A, Lease M (2011) Modeling annotator accuracies for supervised learning.\n                      Proc. Workshop Crowdsourcing Search Data Mining Fourth ACM Internat Conf Web Search Data Mining\n                      (ACM, New York), 19\u201322."},{"key":"B64","first-page":"452","volume":"7","author":"Kwiatkowski T","year":"2019","journal-title":"Trans. Assoc. Comput. Linguistics"},{"key":"B65","unstructured":"Larsen KG (2023) Bagging is an optimal PAC learner.\n                      Proc. 36th Annual Conf. Learn. Theory Proc. Machine Learn. Res.\n                      , vol. 195 (PMLR, New York), 1\u201320."},{"key":"B66","doi-asserted-by":"publisher","DOI":"10.1023\/A:1025779619903"},{"key":"B67","doi-asserted-by":"crossref","unstructured":"Lee S, Kang M, Lee J, Hwang SJ (2021) Learning to perturb word embeddings for out-of-distribution QA. Preprint, submitted May 6, https:\/\/arxiv.org\/abs\/2105.02692.","DOI":"10.18653\/v1\/2021.acl-long.434"},{"key":"B68","unstructured":"Lewis M, Liu Y, Goyal N, Ghazvininejad M, Mohamed A, Levy O, Stoyanov V, Zettlemoyer L (2019) Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. Preprint, submitted October 29, https:\/\/arxiv.org\/1910.13461."},{"key":"B69","doi-asserted-by":"crossref","unstructured":"Li J (2020) Crowdsourced text sequence aggregation based on hybrid reliability and representation.\n                      Proc. 43rd Internat. ACM SIGIR Conf. Res. Development Inform. Retrieval\n                      (Association for Computing Machinery, New York), 1761\u20131764.","DOI":"10.1145\/3397271.3401239"},{"key":"B70","doi-asserted-by":"crossref","unstructured":"Li J, Fukumoto F (2019) A dataset of crowdsourced word sequences: Collections and answer aggregation for ground truth creation.\n                      Proc. First Workshop Aggregating Analysing Crowdsourced Annotations NLP\n                      (Association for Computational Linguistics, Stroudsburg, PA), 24\u201328.","DOI":"10.18653\/v1\/D19-5904"},{"key":"B71","unstructured":"Liang P, Bommasani R, Lee T, Tsipras D, Soylu D, Yasunaga M, Zhang Y, et al. (2023) Holistic evaluation of language models. Preprint, submitted November 16, 2022, https:\/\/arxiv.org\/2211.09110."},{"key":"B72","unstructured":"Lima LC, Wright DB, Augenstein I, Maistro M (2021) University of Copenhagen participation in TREC Health Misinformation track 2020. Preprint, submitted March 3, https:\/\/arxiv.org\/2103.02462."},{"key":"B73","doi-asserted-by":"crossref","unstructured":"Lin S, Hilton J, Evans O (2022) Truthfulqa: Measuring how models mimic human falsehoods. Preprint, submitted September 8, 2021, https:\/\/arxiv.org\/2109.07958.","DOI":"10.18653\/v1\/2022.acl-long.229"},{"key":"B75","unstructured":"Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) RoBERTa: A robustly optimized BERT pretraining approach. Preprint, submitted July 26, https:\/\/arxiv.org\/1907.11692."},{"key":"B77","doi-asserted-by":"crossref","unstructured":"Mohler M, Mihalcea R (2009) Text-to-text semantic similarity for automatic short answer grading.\n                      Proc. 12th Conf. Eur. Chapter ACL\n                      (ACL, Stroudsburg, PA), 567\u2013575.","DOI":"10.3115\/1609067.1609130"},{"key":"B78","unstructured":"Mohler M, Bunescu R, Mihalcea R (2011) Learning to grade short answer questions using semantic similarity measures and dependency graph alignments.\n                      Proc. 49th Annual Meeting Assoc. Comput. Linguistics Human Language Tech.\n                      (ACL, Stroudsburg, PA), 752\u2013762."},{"key":"B79","volume-title":"Foundations of Machine Learning","author":"Mohri M","year":"2012"},{"key":"B80","doi-asserted-by":"crossref","unstructured":"M\u00f6ller T, Risch J, Pietsch M (2021) GermanQuAD and GermanDPR: Improving non-English question answering and passage retrieval. Preprint, submitted April 26, https:\/\/arxiv.org\/2104.12741.","DOI":"10.18653\/v1\/2021.mrqa-1.4"},{"key":"B81","unstructured":"Nguyen T, Rosenberg M, Song X, Gao J, Tiwary S, Majumder R, Deng L (2016) MS MARCO: A human generated machine reading comprehension dataset.\n                      Workshop Adv. Neural Inform. Processing Systems\n                      (CEUR-WS.org)."},{"key":"B82","doi-asserted-by":"crossref","unstructured":"Papineni K, Roukos S, Ward T, Zhu W-J (2002) BLEU: A method for automatic evaluation of machine translation.\n                      Proc. 40th Annual Meeting Assoc. Comput. Linguistics\n                      (Association for Computational Linguistics, Stroudsburg, PA), 311\u2013318.","DOI":"10.3115\/1073083.1073135"},{"issue":"2","key":"B83","first-page":"76","volume":"14","author":"Polat M","year":"2020","journal-title":"Novitas-ROYAL"},{"key":"B84","doi-asserted-by":"crossref","unstructured":"Pradeep R, Ma X, Nogueira R, Lin J (2021) Vera: Prediction techniques for reducing harmful misinformation in consumer health search.\n                      Proc. 44th Internat. ACM SIGIR Conf. Res. Development Inform. Retrieval\n                      (ACM, New York), 2066\u20132070.","DOI":"10.1145\/3404835.3463120"},{"key":"B85","doi-asserted-by":"crossref","unstructured":"Rajpurkar P, Jia R, Liang P (2018) Know what you don\u2019t know: Unanswerable questions for SQuAD. Preprint, submitted June 11, https:\/\/arxiv.org\/1806.03822.","DOI":"10.18653\/v1\/P18-2124"},{"key":"B86","doi-asserted-by":"crossref","unstructured":"Rajpurkar P, Zhang J, Lopyrev K, Liang P (2016) Squad: 100,000+ questions for machine comprehension of text.\n                      Proc. 2016 Conf. Empirical Methods Natural Language Processing\n                      (Association for Computational Linguistics, Stroudsburg, PA), 2383\u20132392.","DOI":"10.18653\/v1\/D16-1264"},{"key":"B87","doi-asserted-by":"crossref","unstructured":"Ramachandran L, Cheng J, Foltz P (2015) Identifying patterns for short answer scoring using graph-based lexico-semantic text matching.\n                      Proc. 10th Workshop Innovative Use NLP Building Ed. Appl.\n                      (ACL, Stroudsburg, PA), 97\u2013106.","DOI":"10.3115\/v1\/W15-0612"},{"key":"B88","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-021-10068-2"},{"issue":"4","key":"B89","first-page":"1297","volume":"11","author":"Raykar VC","year":"2010","journal-title":"J. Machine Learn. Res."},{"key":"B90","doi-asserted-by":"crossref","unstructured":"Reimers N, Gurevych I (2019) Sentence-BERT: Sentence embeddings using Siamese BERT-networks. Preprint, submitted August 27, https:\/\/arxiv.org\/1908.10084.","DOI":"10.18653\/v1\/D19-1410"},{"key":"B91","doi-asserted-by":"publisher","DOI":"10.1016\/j.patrec.2013.05.012"},{"key":"B92","doi-asserted-by":"crossref","unstructured":"Roy S, Narahari Y, Deshmukh OD (2015) A perspective on computer assisted assessment techniques for short free-text answers.\n                      Internat. Comput. Assisted Assessment Conf.\n                      (Springer, Cham, Switzerland), 96\u2013109.","DOI":"10.1007\/978-3-319-27704-2_10"},{"key":"B93","unstructured":"Roy S, Dandapat S, Nagesh A, Narahari Y (2016) Wisdom of students: A consistent automatic short answer grading technique.\n                      Proc. 13th Internat. Conf. Natural Language Processing\n                      (ACL, Stroudsburg, PA), 178\u2013187."},{"key":"B94","doi-asserted-by":"crossref","unstructured":"Saha S, Dhamecha TI, Marvaniya S, Sindhgatta R, Sengupta B (2018) Sentence level or token level features for automatic short answer grading? Use both.\n                      Internat. Conf. Artificial Intelligence Ed.\n                      (Springer, Cham, Switzerland), 503\u2013517.","DOI":"10.1007\/978-3-319-93843-1_37"},{"issue":"5","key":"B95","volume":"5","author":"Singh P","year":"2018","journal-title":"Internat. Res. J. Engrg. Tech."},{"key":"B97","unstructured":"Steimel K, Riordan B (2020) Towards instance-based content scoring with pre-trained transformer models.\n                      Proc. 34th AAAI Conf. Artificial Intelligence\n                      (AAAI, Palo Alto, CA)."},{"key":"B98","doi-asserted-by":"crossref","unstructured":"Sultan MA, Salazar C, Sumner T (2016) Fast and easy short answer grading with high accuracy.\n                      Proc. 2016 Conf. North Amer. Chapter Assoc. Comput. Linguistics Human Language Tech.\n                      (ACL, Stroudsburg, PA), 1070\u20131075.","DOI":"10.18653\/v1\/N16-1123"},{"key":"B99","doi-asserted-by":"crossref","unstructured":"Sung C, Dhamecha TI, Mukhi N (2019) Improving short answer grading using transformer-based pre-training.\n                      Internat. Conf. Artificial Intelligence Ed.\n                      (Springer, Cham, Switzerland), 469\u2013481.","DOI":"10.1007\/978-3-030-23204-7_39"},{"key":"B100","doi-asserted-by":"crossref","unstructured":"Tanno R, Saeedi A, Sankaranarayanan S, Alexander DC, Silberman N (2019) Learning from noisy labels by regularized estimation of annotator confusion.\n                      Proc. IEEE Conf. Comput. Vision Pattern Recognition\n                      (IEEE, Piscataway, NJ), 11244\u201311253.","DOI":"10.1109\/CVPR.2019.01150"},{"key":"B101","doi-asserted-by":"publisher","DOI":"10.1145\/1968.1972"},{"key":"B102","doi-asserted-by":"publisher","DOI":"10.2196\/49788"},{"key":"B103","doi-asserted-by":"publisher","DOI":"10.1287\/isre.2016.0661"},{"key":"B104","unstructured":"Wang X, Wei J, Schuurmans D, Le Q, Chi E, Narang S, Chowdhery A, Zhou D (2022) Self-consistency improves chain of thought reasoning in language models. Preprint, submitted March 21, https:\/\/arxiv.org\/2203.11171."},{"key":"B105","unstructured":"Wang P, Li L, Chen L, Cai Z, Zhu D, Lin B, Cao Y, Liu Q, Liu T, Sui Z (2023b) Large language models are not fair evaluators. Preprint, submitted May 29, https:\/\/arxiv.org\/2305.17926."},{"key":"B106","doi-asserted-by":"publisher","DOI":"10.1109\/TMI.2004.828354"},{"key":"B107","first-page":"1800","volume":"24","author":"Wauthier FL","year":"2011","journal-title":"Adv. Neural Inform. Processing Systems"},{"key":"B108","unstructured":"Whitehill J, Wu TF, Bergsma J, Movellan J, Ruvolo P (2009) Whose vote should count more: Optimal integration of labels from labelers of unknown expertise.\n                      Adv. Neural Inform. Processing Systems\n                      , vol. 22 (Curran Associates Inc., Red Hook, NY)."},{"key":"B109","doi-asserted-by":"publisher","DOI":"10.1016\/j.jksuci.2020.05.006"},{"key":"B110","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-66785-6_12"},{"issue":"3","key":"B111","first-page":"752","volume":"32","author":"Yin J","year":"2021","journal-title":"Inform. Systems Res."},{"key":"B112","unstructured":"Zeng S, Shen J (2022) Efficient PAC learning from the crowd with pairwise comparisons.\n                      Proc. Internat. Conf. Machine Learn\n                      . (PMLR, New York), 25973\u201325993."},{"key":"B113","unstructured":"Zeng S, Shen J (2023) Semi-verified PAC learning from the crowd.\n                      Proc. 26th Internat. Conf. Artificial Intelligence. Statist.\n                      (PMLR, New York), 2068\u20132086."},{"key":"B114","doi-asserted-by":"crossref","unstructured":"Zesch T, Heilman M, Cahill A (2015) Reducing annotation efforts in supervised short answer scoring.\n                      Proc. 10th Workshop Innovative Use NLP Building Ed. Appl.\n                      (ACL, Stroudsburg, PA), 124\u2013132.","DOI":"10.3115\/v1\/W15-0615"},{"key":"B115","unstructured":"Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y (2019) BERTscore: Evaluating text generation with BERT. Preprint, submitted April 21, https:\/\/arxiv.org\/1904.09675."},{"key":"B116","doi-asserted-by":"publisher","DOI":"10.1016\/j.aiopen.2022.12.003"},{"key":"B117","doi-asserted-by":"publisher","DOI":"10.52202\/075280-2020"},{"key":"B118","author":"Zhou T","year":"2025","journal-title":"J. Librarianship Inform. Sci."},{"key":"B119","doi-asserted-by":"crossref","unstructured":"Zhu P, Wang Z, Hauff C, Yang J, Anand A (2022) Answer quality aware aggregation for extractive QA crowdsourcing.\n                      Findings Assoc. Comput. Linguistics\n                      (Association for Computational Linguistics, Stroudsburg, PA), 6147\u20136159.","DOI":"10.18653\/v1\/2022.findings-emnlp.457"},{"key":"B120","unstructured":"Ziegler DM, Stiennon N, Wu J, Brown TB, Radford A, Amodei D, Christiano P, Irving G (2019) Fine-tuning language models from human preferences. Preprint, submitted September 18, https:\/\/arxiv.org\/1909.08593."}],"container-title":["Information Systems Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pubsonline.informs.org\/doi\/pdf\/10.1287\/isre.2023.0426","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,26]],"date-time":"2026-06-26T07:20:02Z","timestamp":1782458402000},"score":1,"resource":{"primary":{"URL":"https:\/\/pubsonline.informs.org\/doi\/10.1287\/isre.2023.0426"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6]]},"references-count":114,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2026,6]]}},"alternative-id":["10.1287\/isre.2023.0426"],"URL":"https:\/\/doi.org\/10.1287\/isre.2023.0426","relation":{},"ISSN":["1047-7047","1526-5536"],"issn-type":[{"value":"1047-7047","type":"print"},{"value":"1526-5536","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6]]}}}