{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,9]],"date-time":"2026-02-09T15:44:47Z","timestamp":1770651887614,"version":"3.49.0"},"reference-count":93,"publisher":"MIS Quarterly","issue":"3","license":[{"start":{"date-parts":[[2025,9,2]],"date-time":"2025-09-02T00:00:00Z","timestamp":1756771200000},"content-version":"vor","delay-in-days":1,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,9,1]]},"abstract":"<jats:p>Many real-world process automation environments are rife with high-expertise and limited labeled data. We propose a computational design science artifact to automate systematic review (SR) in such an environment. SR is a manual process that collects and synthesizes data from medical literature to inform medical decisions and improve clinical practice. Existing machine learning solutions for SR automation suffer from a lack of labeled data and a misrepresentation of the high-expertise manual process. Motivated by humans\u2019 impressive capability to learn from limited examples, we propose a principled and generalizable few-shot learning framework\u2014FastSR\u2014to automate the multistep, expertise-intensive SR process using minimal training data. Informed by SR experts\u2019 annotation logic, FastSR extends the traditional few-shot learning framework by including (1) various representations to account for diverse SR knowledge, (2) attention mechanisms to reflect semantic correspondence of medical text fragments, and (3) shared representations to jointly learn interrelated tasks (i.e., sentence classification and sequence tagging). We instantiated and evaluated FastSR on three test beds: full-text articles from Wilson disease (WD) and COVID-19, as well as a public dataset (EBM-NLP) containing clinical trial abstracts on a wide range of diseases. Our experiments demonstrate that FastSR significantly outperforms several benchmarking solutions and expedites the SR project by up to 65%. We critically examine the SR outcomes and practical advantages of FastSR compared to other ML and manual SR solutions and propose a new FastSR-augmented protocol. Overall, our multifaceted evaluation quantitatively and qualitatively underscores the efficacy and applicability of FastSR in expediting SR. Our results have important implications for designing computational artifacts for automating\/augmenting processes in high-expertise, low-label environments.<\/jats:p>","DOI":"10.25300\/misq\/2024\/18573","type":"journal-article","created":{"date-parts":[[2025,6,9]],"date-time":"2025-06-09T17:04:50Z","timestamp":1749488690000},"page":"1049-1094","source":"Crossref","is-referenced-by-count":3,"title":["Automating in High-Expertise, Low-Label Environments: Evidence-Based Medicine by Expert-Augmented Few-Shot Learning"],"prefix":"10.25300","volume":"49","author":[{"given":"Rong","family":"Liu","sequence":"first","affiliation":[{"name":"College of Business, Florida International University Miami, Florida, U.S.A."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jingjing","family":"Li","sequence":"additional","affiliation":[{"name":"Information Technology and Innovation, McIntire School of Commerce University of Virginia, Charlottesville, VA, U.S.A."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marko","family":"Zivkovic","sequence":"additional","affiliation":[{"name":"Genesis Research Group Hoboken, NJ, U.S.A."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ahmed","family":"Abbasi","sequence":"additional","affiliation":[{"name":"Department of IT, Analytics, and Operations, Mendoza College of Business University of Notre Dame, Notre Dame, IN, U.S.A."}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"10933","published-online":{"date-parts":[[2025,9,1]]},"reference":[{"issue":"2","key":"2025090211540016900_b1-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"441","DOI":"10.1287\/isre.2024.editorial.v35.n2","article-title":"Pathways for design research on artificial intelligence","volume":"35","author":"Abbasi","year":"2024","journal-title":"Information Systems Research"},{"issue":"2","key":"2025090211540016900_b2-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"i","DOI":"10.17705\/1jais.00423","article-title":"Big data research in information systems: Toward an inclusive research agenda","volume":"17","author":"Abbasi","year":"2016","journal-title":"Journal of the Association for Information Systems"},{"issue":"10","key":"2025090211540016900_b3-08_ra_10_25300_misq_2024_18573","first-page":"1705","article-title":"Clustering with Bregman divergences","volume":"6","author":"Banerjee","year":"2005","journal-title":"Journal of Machine Learning Research"},{"key":"2025090211540016900_b4-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"5108","DOI":"10.18653\/v1\/2020.coling-main.448","article-title":"Learning to few-shot learn across diverse natural language classification tasks","author":"Bansal","year":"2020"},{"key":"2025090211540016900_b5-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"522","DOI":"10.18653\/v1\/2020.emnlp-main.38","article-title":"Self-supervised meta-learning for few-shot natural language classification tasks","author":"Bansal","year":"2020"},{"key":"2025090211540016900_b6-08_ra_10_25300_misq_2024_18573","unstructured":"Bao, Y., Wu, M., Chang, S., & Barzilay, R. (2019). Few-shot text classification with distributional signatures. In Proceedings of the International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=H1emfT4twB"},{"key":"2025090211540016900_b7-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"3615","DOI":"10.18653\/v1\/D19-1371","article-title":"Scibert: A pretrained language model for scientific text","author":"Beltagy","year":"2019"},{"issue":"3","key":"2025090211540016900_b8-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"1433","DOI":"10.25300\/MISQ\/2021\/16274","article-title":"Managing artificial intelligence","volume":"45","author":"Berente","year":"2021","journal-title":"MIS Quarterly"},{"issue":"2","key":"2025090211540016900_b9-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2016-012545","article-title":"Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry","volume":"7","author":"Borah","year":"2017","journal-title":"BMJ Open"},{"key":"2025090211540016900_b10-08_ra_10_25300_misq_2024_18573","first-page":"1877","article-title":"Language models are few-shot learners","author":"Brown","year":"2020"},{"key":"2025090211540016900_b11-08_ra_10_25300_misq_2024_18573","unstructured":"Chen, Q., Du, J., Hu, Y., Keloth, V. K., Peng, X., Raja, K., Zhang, R., Lu, Z., & Xu, H. (2024a). Large language models in biomedical natural language processing: Benchmarks, baselines, and recommendations. arXiv. https:\/\/arxiv.org\/abs\/2305.16326"},{"issue":"12","key":"2025090211540016900_b12-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3663363","article-title":"Multi-task learning in natural language processing: An overview","volume":"56","author":"Chen","year":"2024","journal-title":"ACM Computing Surveys"},{"key":"2025090211540016900_b13-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","DOI":"10.1109\/MMSP.2019.8901732","article-title":"Multi-label few-shot learning for sound event recognition","author":"Cheng","year":"2019"},{"issue":"2","key":"2025090211540016900_b14-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1258\/135581907780279648","article-title":"How useful are Cochrane reviews in identifying research needs?","volume":"12","author":"Clarke","year":"2007","journal-title":"Journal of Health Services Research and Policy"},{"key":"2025090211540016900_b15-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-2097","article-title":"A discourse-aware attention model for abstractive summarization of long documents","author":"Cohan","year":"2018"},{"key":"2025090211540016900_b16-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1145\/1143844.11438","article-title":"The relationship between precision-recall and ROC curves","author":"Davis","year":"2006"},{"key":"2025090211540016900_b17-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1423","article-title":"BERT: Pre-training of deep bidirectional Transformers for language understanding","author":"Devlin","year":"2019"},{"key":"2025090211540016900_b18-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"123","DOI":"10.18653\/v1\/2020.bionlp-1.13","article-title":"Evidence inference 2.0: More data, better models","author":"DeYoung","year":"2020"},{"issue":"10092","key":"2025090211540016900_b19-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"415","DOI":"10.1016\/S0140-6736(16)31592-6","article-title":"Progress in evidence-based medicine: A quarter century on","volume":"390","author":"Djulbegovic","year":"2017","journal-title":"The Lancet"},{"key":"2025090211540016900_b20-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"12039","DOI":"10.18653\/v1\/2024.findings-acl.716","article-title":"Generalization or memorization: Data contamination and trustworthy evaluation for large language models","author":"Dong","year":"2024","journal-title":"Findings of the Association for Computational Linguistics"},{"key":"2025090211540016900_b21-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"935","DOI":"10.18653\/v1\/2021.eacl-main.79","article-title":"A neural few-shot text classification reality check","author":"Dopierre","year":"2021"},{"key":"2025090211540016900_b22-08_ra_10_25300_misq_2024_18573","first-page":"1126","article-title":"Model-agnostic meta-learning for fast adaptation of deep networks","author":"Finn","year":"2017"},{"key":"2025090211540016900_b23-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2021.acl-long.295","article-title":"making pre-trained language models better few-shot learners","author":"Gao","year":"2021"},{"issue":"13","key":"2025090211540016900_b24-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"1117","DOI":"10.1503\/cmaj.131251","article-title":"The Cochrane collaboration 20 years in","volume":"185","author":"Grimshaw","year":"2013","journal-title":"CMAJ"},{"key":"2025090211540016900_b25-08_ra_10_25300_misq_2024_18573","volume-title":"Users\u2019 guides to the medical literature: A manual for evidence-based clinical practice","author":"Guyatt","year":"2002"},{"key":"2025090211540016900_b26-08_ra_10_25300_misq_2024_18573","unstructured":"He, Z., Wang, Z., Wei, W., Feng, S., Mao, X., & Jiang, S. (2020). A Survey on recent advances in sequence labeling from deep learning models. arXiv. https:\/\/arxiv.org\/abs\/2011.06727"},{"key":"2025090211540016900_b27-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"crossref","DOI":"10.1002\/9781119536604","volume-title":"Cochrane handbook for systematic reviews of interventions","author":"Higgins","year":"2019"},{"issue":"8","key":"2025090211540016900_b28-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Computation"},{"issue":"9","key":"2025090211540016900_b29-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"1812","DOI":"10.1093\/jamia\/ocad259","article-title":"Improving large language models for clinical named entity recognition via prompt engineering","volume":"31","author":"Hu","year":"2024","journal-title":"JAMIA"},{"key":"2025090211540016900_b30-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"crossref","unstructured":"Huang, J., Li, C., Subudhi, K., Jose, D., Balakrishnan, S., Chen, W., Peng, B., Gao, J., & Han, J. (2020). Few-shot Named entity recognition: A comprehensive study. arXiv. https:\/\/arxiv.org\/abs\/2012.14978","DOI":"10.18653\/v1\/2021.emnlp-main.813"},{"issue":"1","key":"2025090211540016900_b31-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1471-2288-13-50","article-title":"The use of systematic reviews in the planning, design and conduct of randomised trials: A retrospective cohort of NIHR HTA funded trials","volume":"13","author":"Jones","year":"2013","journal-title":"BMC Medical Research Methodology"},{"key":"2025090211540016900_b32-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"1746","DOI":"10.3115\/v1\/D14-1181","article-title":"Convolutional neural networks for sentence classification","author":"Kim","year":"2014"},{"issue":"1","key":"2025090211540016900_b33-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","DOI":"10.1186\/1472-6947-10-56","article-title":"ExaCT: Automatic extraction of clinical trial characteristics from journal publications","volume":"10","author":"Kiritchenko","year":"2010","journal-title":"BMC Medical Informatics and Decision Making"},{"issue":"2","key":"2025090211540016900_b34-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"540","DOI":"10.1080\/07421222.2018.1451957","article-title":"Advanced customer analytics: Strategic value through integration of relationship-oriented big data","volume":"35","author":"Kitchens","year":"2018","journal-title":"Journal of Management Information Systems"},{"issue":"4","key":"2025090211540016900_b35-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"70","DOI":"10.1109\/MIS.2006.75","article-title":"Making sense of sensemaking 1: Alternative perspectives","volume":"21","author":"Klein","year":"2006","journal-title":"IEEE Intelligent Systems"},{"key":"2025090211540016900_b36-08_ra_10_25300_misq_2024_18573","first-page":"282","article-title":"Conditional random fields: Probabilistic models for segmenting and labeling sequence data","author":"Lafferty","year":"2001"},{"issue":"6266","key":"2025090211540016900_b37-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"1332","DOI":"10.1126\/science.aab3050","article-title":"Human-level concept learning through probabilistic program induction","volume":"350","author":"Lake","year":"2015","journal-title":"Science"},{"issue":"4","key":"2025090211540016900_b38-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"2025090211540016900_b39-08_ra_10_25300_misq_2024_18573","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive NLP tasks","author":"Lewis","year":"2020"},{"key":"2025090211540016900_b40-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2021.eacl-main.86","article-title":"question and answer test-train overlap in open-domain question answering datasets","author":"Lewis","year":"2021"},{"issue":"4","key":"2025090211540016900_b41-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"1733","DOI":"10.25300\/MISQ\/2020\/15323","article-title":"TheoryOn: A design framework and system for unlocking behavioral knowledge through ontology learning","volume":"44","author":"Li","year":"2020","journal-title":"MIS Quarterly"},{"key":"2025090211540016900_b42-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"6299","DOI":"10.18653\/v1\/D19-1657","article-title":"Multi-task stance detection with sentiment and stance lexicons","author":"Li","year":"2019"},{"issue":"9","key":"2025090211540016900_b43-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3560815","article-title":"Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing","volume":"55","author":"Liu","year":"2023","journal-title":"ACM Computing Surveys"},{"key":"2025090211540016900_b44-08_ra_10_25300_misq_2024_18573","article-title":"Decoupled weight decay regularization","author":"Loshchilov","year":"2019"},{"issue":"2","key":"2025090211540016900_b45-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"623","DOI":"10.25300\/MISQ\/2019\/14439","article-title":"Expecting the unexpected: Effects of data collection design choices on the quality of crowdsourced user-generated content","volume":"43","author":"Lukyanenko","year":"2019","journal-title":"MIS Quarterly"},{"issue":"6","key":"2025090211540016900_b46-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"bbac409","DOI":"10.1093\/bib\/bbac409","article-title":"BioGPT: Generative pre-trained Transformer for biomedical text generation and mining","volume":"23","author":"Luo","year":"2022","journal-title":"Briefings in Bioinformatics"},{"key":"2025090211540016900_b47-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"1412","DOI":"10.18653\/v1\/D15-1166","article-title":"Effective approaches to attention-based neural machine translation","author":"Luong","year":"2015"},{"issue":"12","key":"2025090211540016900_b48-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"1253","DOI":"10.17705\/1jais.00526","article-title":"Data-driven meets theory-driven research in the era of big data: Opportunities and challenges for information systems research","volume":"19","author":"Maass","year":"2018","journal-title":"Journal of the Association for Information Systems"},{"issue":"1","key":"2025090211540016900_b49-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1093\/jamia\/ocv044","article-title":"RobotReviewer: Evaluation of a system for automatically assessing bias in clinical trials","volume":"23","author":"Marshall","year":"2016","journal-title":"JAMIA"},{"issue":"1","key":"2025090211540016900_b50-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1186\/s13643-019-1074-9","article-title":"Toward systematic review automation: A practical guide to using machine learning tools in research synthesis","volume":"8","author":"Marshall","year":"2019","journal-title":"Systematic Reviews"},{"key":"2025090211540016900_b51-08_ra_10_25300_misq_2024_18573","unstructured":"Miyato, T., Dai, A. M., & Goodfellow, I. (2017). Adversarial training methods for semi-supervised text classification. In Proceedings of the 5th International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=r1X3g2_xl"},{"key":"2025090211540016900_b52-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"287","DOI":"10.1016\/j.jclinepi.2021.05.019","article-title":"Resource use during systematic review production varies widely: A scoping review","volume":"139","author":"Nussbaumer-Streit","year":"2021","journal-title":"Journal of Clinical Epidemiology"},{"key":"2025090211540016900_b53-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"197","DOI":"10.18653\/v1\/P18-1019","article-title":"A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature","author":"Nye","year":"2018"},{"key":"2025090211540016900_b54-08_ra_10_25300_misq_2024_18573","first-page":"3239","article-title":"Realistic evaluation of deep semi-supervised learning algorithms","author":"Oliver","year":"2018"},{"issue":"1","key":"2025090211540016900_b55-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","DOI":"10.1186\/2046-4053-4-5","article-title":"Using text mining for study identification in systematic reviews: A systematic review of current approaches","volume":"4","year":"2015","journal-title":"Systematic Reviews"},{"issue":"1","key":"2025090211540016900_b56-08_ra_10_25300_misq_2024_18573","first-page":"iii","article-title":"Editor\u2019s comments: Machine learning in information systems research","volume":"46","author":"Padmanabhan","year":"2022","journal-title":"MIS Quarterly"},{"key":"2025090211540016900_b57-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.n160","article-title":"PRISMA 2020 explanation and elaboration: Updated guidance and exemplars for reporting systematic reviews","volume":"372","author":"Page","year":"2021","journal-title":"BMJ"},{"issue":"1","key":"2025090211540016900_b58-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1007\/BF00763644","article-title":"The principle of semantic compositionality","volume":"13","author":"Pelletier","year":"1994","journal-title":"Topoi"},{"issue":"1","key":"2025090211540016900_b59-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00958-w","article-title":"A study of generative large language model for medical research and healthcare","volume":"6","author":"Peng","year":"2023","journal-title":"NPJ Digital Medicine"},{"issue":"23","key":"2025090211540016900_b60-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"2501","DOI":"10.1001\/jama.2013.281334","article-title":"Enhancing physicians\u2019 use of clinical guidelines","volume":"310","author":"Pronovost","year":"2013","journal-title":"JAMA"},{"key":"2025090211540016900_b61-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","DOI":"10.1101\/2020.03.04.20031401","author":"Qiu","year":"2020","journal-title":"Outcome reporting from protocols of clinical trials of coronavirus disease 2019 (COVID-19): A review"},{"key":"2025090211540016900_b62-08_ra_10_25300_misq_2024_18573","unstructured":"Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training (Working paper). Available at https:\/\/cdn.openai.com\/research-covers\/language-unsupervised\/language_understanding_paper.pdf"},{"issue":"1","key":"2025090211540016900_b63-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"crossref","first-page":"iii","DOI":"10.25300\/MISQ\/2017\/41.1.E0","article-title":"Editor\u2019s comments: Diversity of design science research","volume":"41","author":"Rai","year":"2017","journal-title":"MIS Quarterly"},{"issue":"1","key":"2025090211540016900_b64-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s13721-020-0218-0","article-title":"Sample size determination for biomedical big data with limited labels","volume":"9","author":"Richter","year":"2020","journal-title":"Network Modeling Analysis in Health Informatics and Bioinformatics"},{"key":"2025090211540016900_b65-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1189","article-title":"EMR coding with semi-parametric multi-head matching networks","author":"Rios","year":"2018"},{"key":"2025090211540016900_b66-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"3132","DOI":"10.18653\/v1\/D18-1352","article-title":"Few-shot and zero-shot multi-label learning for structured label spaces","author":"Rios","year":"2018"},{"issue":"1","key":"2025090211540016900_b67-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"1","DOI":"10.2307\/25148826","article-title":"Toward improving the relevance of information systems research to practice: The role of applicability checks","volume":"32","author":"Rosemann","year":"2008","journal-title":"MIS Quarterly"},{"key":"2025090211540016900_b68-08_ra_10_25300_misq_2024_18573","first-page":"269","article-title":"The cost structure of sensemaking","author":"Russell","year":"1993"},{"issue":"3","key":"2025090211540016900_b69-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0118432","article-title":"the precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets","volume":"10","author":"Saito","year":"2015","journal-title":"PLOS One"},{"issue":"1","key":"2025090211540016900_b70-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"271","DOI":"10.1080\/07421222.2023.2172772","article-title":"Deep learning for information systems research","volume":"40","author":"Samtani","year":"2023","journal-title":"Journal of Management Information Systems"},{"key":"2025090211540016900_b71-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","DOI":"10.12688\/f1000research.51117.2","article-title":"Data extraction methods for systematic review (semi) automation: A living systematic review","volume":"10","author":"Schmidt","year":"2025","journal-title":"F1000Research"},{"key":"2025090211540016900_b72-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"83","DOI":"10.5220\/0008945700830094","article-title":"Data mining in clinical trial text: Transformers for classification and question answering tasks","author":"Schmidt","year":"2020"},{"key":"2025090211540016900_b73-08_ra_10_25300_misq_2024_18573","unstructured":"Seo, M., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2017). Bidirectional attention flow for machine comprehension. In Proceedings of the 5th International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=HJ0UKP9ge"},{"issue":"1","key":"2025090211540016900_b74-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-016-0315-4","article-title":"Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews","volume":"5","author":"Shemilt","year":"2016","journal-title":"Systematic Reviews"},{"issue":"3","key":"2025090211540016900_b75-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1016\/j.jclinepi.2014.09.022","article-title":"Survey of new 2007 and 2011 Cochrane reviews found 37% of prespecified outcomes not reported","volume":"68","author":"Smith","year":"2015","journal-title":"Journal of Clinical Epidemiology"},{"key":"2025090211540016900_b76-08_ra_10_25300_misq_2024_18573","first-page":"4080","article-title":"Prototypical networks for few-shot learning","author":"Snell","year":"2017"},{"key":"2025090211540016900_b77-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"3875","DOI":"10.24963\/ijcai.2020\/536","article-title":"Hierarchical multi-task learning for organization evaluation of argumentative student essays","author":"Song","year":"2020"},{"issue":"3","key":"2025090211540016900_b78-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","DOI":"10.2196\/17984","article-title":"Clinical text data in machine learning: Systematic review","volume":"8","author":"Spasic","year":"2020","journal-title":"JMIR Medical Informatics"},{"issue":"2","key":"2025090211540016900_b79-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"373","DOI":"10.1007\/s10994-019-05855-6","article-title":"A survey on semi-supervised learning","volume":"109","author":"van Engelen","year":"2020","journal-title":"Machine Learning"},{"key":"2025090211540016900_b80-08_ra_10_25300_misq_2024_18573","first-page":"3637","article-title":"Matching networks for one shot learning","author":"Vinyals","year":"2016"},{"key":"2025090211540016900_b81-08_ra_10_25300_misq_2024_18573","first-page":"2692","article-title":"Pointer networks","author":"Vinyals","year":"2015"},{"key":"2025090211540016900_b82-08_ra_10_25300_misq_2024_18573","unstructured":"Wang, L., Lo, K., Chandrasekhar, Y., Reas, R., Yang, J., Burdick, D., Eide, D., Funk, K., Katsis, Y., Kinney, R., Li, Y., Liu, Z., Merrill, W., Mooney, P., Murdick, D., Rishi, D., Sheehan, J., Shen, Z., Stilson, B., \u2026 Kohlmeier, S. (2020a). CORD-19: The COVID-19 Open Research dataset. arXiv. https:\/\/arxiv.org\/abs\/2004.10706."},{"issue":"3","key":"2025090211540016900_b83-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3386252","article-title":"Generalizing from a few examples: A survey on few-shot learning","volume":"53","author":"Wang","year":"2020","journal-title":"ACM Computing Surveys"},{"key":"2025090211540016900_b84-08_ra_10_25300_misq_2024_18573","first-page":"6000","article-title":"Attention is all you need","author":"Vaswani","year":"2017"},{"key":"2025090211540016900_b85-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"5363","DOI":"10.18653\/v1\/P19-1533","article-title":"Label-agnostic sequence labeling by copying nearest neighbors","author":"Wiseman","year":"2019"},{"key":"2025090211540016900_b86-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"3011","DOI":"10.18653\/v1\/D19-1298","article-title":"Extractive summarization of long documents by combining global and local context","author":"Xiao","year":"2019"},{"key":"2025090211540016900_b87-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"3413","DOI":"10.1145\/3308558.3313644","article-title":"Open-world learning and application to product classification","author":"Xu","year":"2019"},{"issue":"9","key":"2025090211540016900_b88-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"8934","DOI":"10.1109\/TKDE.2022.3220219","article-title":"A survey on deep semi-supervised learning","volume":"35","author":"Yang","year":"2022","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"2025090211540016900_b89-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1109","article-title":"Diverse few-shot text classification with multiple metrics","author":"Yu","year":"2018"},{"issue":"2","key":"2025090211540016900_b90-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1007\/s11704-017-7031-7","article-title":"Binary relevance for multi-label learning: An overview","volume":"12","author":"Zhang","year":"2018","journal-title":"Frontiers of Computer Science"},{"key":"2025090211540016900_b91-08_ra_10_25300_misq_2024_18573","unstructured":"Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020). BERTScore: Evaluating text generation with BERT. In Proceedings of the 8th International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=SkeHuCVFDr"},{"key":"2025090211540016900_b92-08_ra_10_25300_misq_2024_18573","first-page":"649","article-title":"Character-level Convolutional Networks for Text Classification","author":"Zhang","year":"2015"},{"key":"2025090211540016900_b93-08_ra_10_25300_misq_2024_18573","doi-asserted-by":"publisher","first-page":"6236","DOI":"10.18653\/v1\/P19-1628","article-title":"Sentence centrality revisited for unsupervised summarization","author":"Zheng","year":"2019"}],"container-title":["MIS Quarterly"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/misq.umn.edu\/misq\/article-pdf\/49\/3\/1049\/15305\/08_ra_10.25300_misq_2024_18573.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/misq.umn.edu\/misq\/article-pdf\/49\/3\/1049\/15305\/08_ra_10.25300_misq_2024_18573.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,2]],"date-time":"2025-09-02T15:54:10Z","timestamp":1756828450000},"score":1,"resource":{"primary":{"URL":"https:\/\/misq.umn.edu\/misq\/article\/49\/3\/1049\/3193\/Automating-in-High-Expertise-Low-Label"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,1]]},"references-count":93,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,9,1]]},"published-print":{"date-parts":[[2025,9,1]]}},"URL":"https:\/\/doi.org\/10.25300\/misq\/2024\/18573","relation":{},"ISSN":["0276-7783","2162-9730"],"issn-type":[{"value":"0276-7783","type":"print"},{"value":"2162-9730","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,1]]}}}