{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,25]],"date-time":"2026-04-25T20:44:20Z","timestamp":1777149860003,"version":"3.51.4"},"reference-count":47,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2019,11,18]],"date-time":"2019-11-18T00:00:00Z","timestamp":1574035200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,11,18]],"date-time":"2019-11-18T00:00:00Z","timestamp":1574035200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Human-in-the-loop (HITL) AI may enable an ideal symbiosis of human experts and AI models, harnessing the advantages of both while at the same time overcoming their respective limitations. The purpose of this study was to investigate a novel collective intelligence technology designed to amplify the diagnostic accuracy of networked human groups by forming real-time systems modeled on biological swarms. Using small groups of radiologists, the swarm-based technology was applied to the diagnosis of pneumonia on chest radiographs and compared against human experts alone, as well as two state-of-the-art deep learning AI models. Our work demonstrates that both the swarm-based technology and deep-learning technology achieved superior diagnostic accuracy than the human experts alone. Our work further demonstrates that when used in combination, the swarm-based technology and deep-learning technology outperformed either method alone. The superior diagnostic accuracy of the combined HITL AI solution compared to radiologists and AI alone has broad implications for the surging clinical AI deployment and implementation strategies in future practice.<\/jats:p>","DOI":"10.1038\/s41746-019-0189-7","type":"journal-article","created":{"date-parts":[[2019,11,18]],"date-time":"2019-11-18T11:02:16Z","timestamp":1574074936000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":156,"title":["Human\u2013machine partnership with artificial intelligence for chest radiograph diagnosis"],"prefix":"10.1038","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5157-9903","authenticated-orcid":false,"given":"Bhavik N.","family":"Patel","sequence":"first","affiliation":[]},{"given":"Louis","family":"Rosenberg","sequence":"additional","affiliation":[]},{"given":"Gregg","family":"Willcox","sequence":"additional","affiliation":[]},{"given":"David","family":"Baltaxe","sequence":"additional","affiliation":[]},{"given":"Mimi","family":"Lyons","sequence":"additional","affiliation":[]},{"given":"Jeremy","family":"Irvin","sequence":"additional","affiliation":[]},{"given":"Pranav","family":"Rajpurkar","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9354-9486","authenticated-orcid":false,"given":"Timothy","family":"Amrhein","sequence":"additional","affiliation":[]},{"given":"Rajan","family":"Gupta","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1317-984X","authenticated-orcid":false,"given":"Safwan","family":"Halabi","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8972-8051","authenticated-orcid":false,"given":"Curtis","family":"Langlotz","sequence":"additional","affiliation":[]},{"given":"Edward","family":"Lo","sequence":"additional","affiliation":[]},{"given":"Joseph","family":"Mammarappallil","sequence":"additional","affiliation":[]},{"given":"A. J.","family":"Mariano","sequence":"additional","affiliation":[]},{"given":"Geoffrey","family":"Riley","sequence":"additional","affiliation":[]},{"given":"Jayne","family":"Seekins","sequence":"additional","affiliation":[]},{"given":"Luyao","family":"Shen","sequence":"additional","affiliation":[]},{"given":"Evan","family":"Zucker","sequence":"additional","affiliation":[]},{"given":"Matthew P.","family":"Lungren","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2019,11,18]]},"reference":[{"key":"189_CR1","doi-asserted-by":"publisher","first-page":"1342","DOI":"10.1038\/s41591-018-0107-6","volume":"24","author":"J De Fauw","year":"2018","unstructured":"De Fauw, J. et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat. Med. 24, 1342\u20131350 (2018).","journal-title":"Nat. Med."},{"key":"189_CR2","doi-asserted-by":"publisher","first-page":"2199","DOI":"10.1001\/jama.2017.14585","volume":"318","author":"B Ehteshami Bejnordi","year":"2017","unstructured":"Ehteshami Bejnordi, B. et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318, 2199\u20132210 (2017).","journal-title":"JAMA"},{"key":"189_CR3","doi-asserted-by":"publisher","first-page":"2402","DOI":"10.1001\/jama.2016.17216","volume":"316","author":"V Gulshan","year":"2016","unstructured":"Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316, 2402\u20132410 (2016).","journal-title":"JAMA"},{"key":"189_CR4","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1002686","volume":"15","author":"P Rajpurkar","year":"2018","unstructured":"Rajpurkar, P. et al. Deep learning for chest radiograph diagnosis: a retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLoS Med. 15, e1002686 (2018).","journal-title":"PLoS Med."},{"key":"189_CR5","doi-asserted-by":"crossref","unstructured":"Irvin, J. et al. CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In: Proc. AAAI Conference on Artificial Intelligence, North America (2019).","DOI":"10.1609\/aaai.v33i01.3301590"},{"key":"189_CR6","doi-asserted-by":"publisher","first-page":"1476","DOI":"10.1016\/j.jacr.2017.07.007","volume":"14","author":"M Recht","year":"2017","unstructured":"Recht, M. & Bryan, R. N. Artificial intelligence: threat or boon to radiologists? J. Am. Coll. Radiol. 14, 1476\u20131480 (2017).","journal-title":"J. Am. Coll. Radiol."},{"key":"189_CR7","doi-asserted-by":"publisher","first-page":"1004","DOI":"10.1016\/j.jacr.2018.03.046","volume":"15","author":"R Schier","year":"2018","unstructured":"Schier, R. Artificial intelligence and the practice of radiology: an alternative view. J. Am. Coll. Radiol. 15, 1004\u20131007 (2018).","journal-title":"J. Am. Coll. Radiol."},{"key":"189_CR8","doi-asserted-by":"publisher","first-page":"1216","DOI":"10.1056\/NEJMp1606181","volume":"375","author":"Z Obermeyer","year":"2016","unstructured":"Obermeyer, Z. & Emanuel, E. J. Predicting the future\u2014big data, machine learning, and clinical medicine. N. Engl. J. Med. 375, 1216\u20131219 (2016).","journal-title":"N. Engl. J. Med."},{"key":"189_CR9","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1148\/radiol.2016162471","volume":"282","author":"HY Kressel","year":"2017","unstructured":"Kressel, H. Y. Setting sail: 2017. Radiology 282, 4\u20136 (2017).","journal-title":"Radiology"},{"key":"189_CR10","doi-asserted-by":"publisher","first-page":"2113","DOI":"10.1148\/rg.2017170077","volume":"37","author":"G Chartrand","year":"2017","unstructured":"Chartrand, G. et al. Deep learning: a primer for radiologists. Radiographics 37, 2113\u20132131 (2017).","journal-title":"Radiographics"},{"key":"189_CR11","doi-asserted-by":"publisher","first-page":"1544","DOI":"10.1001\/jamainternmed.2018.3763","volume":"178","author":"MA Gianfrancesco","year":"2018","unstructured":"Gianfrancesco, M. A., Tamang, S., Yazdany, J. & Schmajuk, G. Potential biases in machine learning algorithms using electronic health record data. JAMA Intern. Med. 178, 1544\u20131547 (2018).","journal-title":"JAMA Intern. Med."},{"key":"189_CR12","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0192360","volume":"13","author":"S Gehrmann","year":"2018","unstructured":"Gehrmann, S. et al. Comparing deep learning and concept extraction based methods for patient phenotyping from clinical narratives. PLoS ONE 13, e0192360 (2018).","journal-title":"PLoS"},{"key":"189_CR13","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1001\/jama.2017.19198","volume":"319","author":"A Verghese","year":"2018","unstructured":"Verghese, A., Shah, N. H. & Harrington, R. A. What this computer needs is a physician: humanism and artificial intelligence. JAMA 319, 19\u201320 (2018).","journal-title":"JAMA"},{"key":"189_CR14","doi-asserted-by":"publisher","first-page":"152","DOI":"10.1016\/j.ejrad.2018.03.019","volume":"102","author":"C Liew","year":"2018","unstructured":"Liew, C. The future of radiology augmented with artificial intelligence: a strategy for success. Eur. J. Radiol. 102, 152\u2013156 (2018).","journal-title":"Eur. J. Radiol."},{"key":"189_CR15","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1002699","volume":"15","author":"N Bien","year":"2018","unstructured":"Bien, N. et al. Deep-learning-assisted diagnosis for knee magnetic resonance imaging: development and retrospective validation of MRNet. PLoS Med. 15, e1002699 (2018).","journal-title":"PLoS Med."},{"key":"189_CR16","doi-asserted-by":"publisher","first-page":"574","DOI":"10.1148\/radiol.2017162326","volume":"284","author":"P Lakhani","year":"2017","unstructured":"Lakhani, P. & Sundaram, B. Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology 284, 574\u2013582 (2017).","journal-title":"Radiology"},{"key":"189_CR17","doi-asserted-by":"crossref","unstructured":"Beni, G. From Swarm Intelligence to Swarm Robotics. 1\u20139 (Springer, Berlin, Heidelberg, 2005).","DOI":"10.1007\/978-3-540-30552-1_1"},{"key":"189_CR18","unstructured":"Wang, J. & Beni, G. Pattern generation in cellular robotic systems. In: Proc. IEEE International Symposium on Intelligent Control, 63\u201369 (IEEE, 1988)."},{"key":"189_CR19","doi-asserted-by":"crossref","unstructured":"Rosenberg, L. Artificial swarm intelligence, a human-in-the-loop approach to A.I. In: Proc. 13th AAAI Conference on Artificial Intelligence, 4381\u20134382 (AAAI Press, Phoenix, AZ, 2016).","DOI":"10.1609\/aaai.v30i1.9833"},{"key":"189_CR20","doi-asserted-by":"publisher","first-page":"1065","DOI":"10.1098\/rsif.2008.0511","volume":"6","author":"JA Marshall","year":"2009","unstructured":"Marshall, J. A. et al. On optimal decision-making in brains and social insect colonies. J. R. Soc. Interface 6, 1065\u20131074 (2009).","journal-title":"J. R. Soc. Interface"},{"key":"189_CR21","doi-asserted-by":"publisher","first-page":"108","DOI":"10.1126\/science.1210361","volume":"335","author":"TD Seeley","year":"2012","unstructured":"Seeley, T. D. et al. Stop signals provide cross inhibition in collective decision-making by honeybee swarms. Science 335, 108\u2013111 (2012).","journal-title":"Science"},{"key":"189_CR22","doi-asserted-by":"publisher","first-page":"416","DOI":"10.1007\/s002650000299","volume":"49","author":"TD Seeley","year":"2001","unstructured":"Seeley, T. D. & Buhrman, S. C. Nest-site selection in honey bees: how well do swarms implement the \u201cbest-of-N\u201d decision rule? Behav. Ecol. Sociobiol. 49, 416\u2013427 (2001).","journal-title":"Behav. Ecol. Sociobiol."},{"key":"189_CR23","doi-asserted-by":"crossref","unstructured":"Rosenberg, L. Artificial Swarm Intelligence, a Human-in-the-Loop Approach to A.I. Thirtieth AAAI Conference on Artificial Intelligence. (2016).","DOI":"10.1109\/IJCNN.2016.7727517"},{"key":"189_CR24","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1038\/nature21056","volume":"542","author":"A Esteva","year":"2017","unstructured":"Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115\u2013118 (2017).","journal-title":"Nature"},{"key":"189_CR25","doi-asserted-by":"publisher","first-page":"1337","DOI":"10.1038\/s41591-018-0147-y","volume":"24","author":"JJ Titano","year":"2018","unstructured":"Titano, J. J. et al. Automated deep-neural-network surveillance of cranial images for acute neurologic events. Nat. Med. 24, 1337\u20131341 (2018).","journal-title":"Nat. Med."},{"key":"189_CR26","doi-asserted-by":"publisher","first-page":"450","DOI":"10.1038\/075450a0","volume":"75","author":"F Galton","year":"1907","unstructured":"Galton, F. Vox populi (The wisdom of crowds). Nature 75, 450\u2013451 (1907).","journal-title":"Nature"},{"key":"189_CR27","doi-asserted-by":"publisher","first-page":"337","DOI":"10.1037\/h0042344","volume":"55","author":"I Lorge","year":"1958","unstructured":"Lorge, I., Fox, D., Davitz, J. & Brenner, M. A survey of studies contrasting the quality of group performance and individual performance, 1920\u20131957. Psychol. Bull. 55, 337\u2013372 (1958).","journal-title":"Psychol. Bull."},{"key":"189_CR28","doi-asserted-by":"publisher","first-page":"9020","DOI":"10.1073\/pnas.1008636108","volume":"108","author":"J Lorenz","year":"2011","unstructured":"Lorenz, J., Rauhut, H., Schweitzer, F. & Helbing, D. How social influence can undermine the wisdom of crowd effect. Proc. Natl Acad. Sci. USA 108, 9020\u20139025 (2011).","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"189_CR29","first-page":"351","volume":"27","author":"T Miner","year":"2005","unstructured":"Miner, T. The wisdom of crowds: why the many are smarter than the few, and how collective wisdom shapes business, economies, societies, and nations. J. Exp. Educ. 27, 351 (2005).","journal-title":"J. Exp. Educ."},{"key":"189_CR30","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1016\/j.jmp.2010.10.002","volume":"55","author":"H Rauhut","year":"2011","unstructured":"Rauhut, H. & Lorenz, J. The wisdom of crowds in one mind: how individuals can simulate the knowledge of diverse societies to reach better decisions. J. Math. Psychol. 55, 191\u2013197 (2011).","journal-title":"J. Math. Psychol."},{"key":"189_CR31","doi-asserted-by":"crossref","first-page":"590","DOI":"10.1227\/NEU.0000000000001374","volume":"80","author":"AM Sonabend","year":"2017","unstructured":"Sonabend, A. M. et al. Defining glioblastoma resectability through the wisdom of the crowd: a proof-of-principle study. Neurosurgery 80, 590\u2013601 (2017).","journal-title":"Neurosurgery"},{"key":"189_CR32","doi-asserted-by":"publisher","first-page":"979","DOI":"10.1016\/j.canep.2013.09.004","volume":"37","author":"AJ King","year":"2013","unstructured":"King, A. J., Gehl, R. W., Grossman, D. & Jensen, J. D. Skin self-examinations and visual identification of atypical nevi: comparing individual and crowdsourcing approaches. Cancer Epidemiol. 37, 979\u2013984 (2013).","journal-title":"Cancer Epidemiol."},{"key":"189_CR33","doi-asserted-by":"publisher","first-page":"1280","DOI":"10.1016\/j.media.2012.04.007","volume":"16","author":"MT McKenna","year":"2012","unstructured":"McKenna, M. T. et al. Strategies for improved interpretation of computer-aided detections for CT colonography utilizing distributed human intelligence. Med. Image Anal. 16, 1280\u20131292 (2012).","journal-title":"Med. Image Anal."},{"key":"189_CR34","doi-asserted-by":"publisher","first-page":"2595","DOI":"10.1002\/cam4.1165","volume":"6","author":"YJ Lee","year":"2017","unstructured":"Lee, Y. J., Arida, J. A. & Donovan, H. S. The application of crowdsourcing approaches to cancer research: a systematic review. Cancer Med. 6, 2595\u20132605 (2017).","journal-title":"Cancer Med."},{"key":"189_CR35","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0078433","volume":"8","author":"M Moussaid","year":"2013","unstructured":"Moussaid, M., Kammer, J. E., Analytis, P. P. & Neth, H. Social influence and the collective dynamics of opinion formation. PLoS ONE 8, e78433 (2013).","journal-title":"PLoS ONE"},{"key":"189_CR36","doi-asserted-by":"publisher","first-page":"281","DOI":"10.1098\/rstb.2009.0169","volume":"365","author":"M Baddeley","year":"2010","unstructured":"Baddeley, M. Herding, social influence and economic decision-making: socio-psychological and neuroscientific analyses. Philos. Trans. R. Soc. Lond. B 365, 281\u2013290 (2010).","journal-title":"Philos. Trans. R. Soc. Lond. B"},{"key":"189_CR37","doi-asserted-by":"publisher","first-page":"E5070","DOI":"10.1073\/pnas.1621512114","volume":"114","author":"J Becker","year":"2017","unstructured":"Becker, J., Brackbill, D. & Centola, D. Network dynamics of social influence in the wisdom of crowds. Proc. Natl Acad. Sci. USA 114, E5070\u2013E5076 (2017).","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"189_CR38","unstructured":"Rajpurkar, P. et al. CheXNet: radiologist-level pneumonia detection on chest X-rays with deep learning. Preprint at arXiv:1711.05225 (2017)."},{"key":"189_CR39","doi-asserted-by":"crossref","unstructured":"Wang, X. et al. ChestX-ray8: Hospital-scale Chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. Proceedings of the IEEE conference on computer vision and pattern recognition. Preprint at arXiv:1705.02315 (2017).","DOI":"10.1109\/CVPR.2017.369"},{"key":"189_CR40","doi-asserted-by":"publisher","first-page":"500","DOI":"10.1038\/s41568-018-0016-5","volume":"18","author":"A Hosny","year":"2018","unstructured":"Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H. & Aerts, H. Artificial intelligence in radiology. Nat. Rev. Cancer 18, 500\u2013510 (2018).","journal-title":"Nat. Rev. Cancer"},{"key":"189_CR41","doi-asserted-by":"publisher","first-page":"2353","DOI":"10.1001\/jama.2016.17438","volume":"316","author":"S Jha","year":"2016","unstructured":"Jha, S. & Topol, E. J. Adapting to artificial intelligence: radiologists and pathologists as information specialists. JAMA 316, 2353\u20132354 (2016).","journal-title":"JAMA"},{"key":"189_CR42","doi-asserted-by":"publisher","first-page":"1097","DOI":"10.1016\/S0140-6736(15)60733-4","volume":"386","author":"E Prina","year":"2015","unstructured":"Prina, E., Ranzani, O. T. & Torres, A. Community-acquired pneumonia. Lancet 386, 1097\u20131108 (2015).","journal-title":"Lancet"},{"key":"189_CR43","doi-asserted-by":"publisher","first-page":"1440","DOI":"10.1001\/jama.1997.03550170070035","volume":"278","author":"JP Metlay","year":"1997","unstructured":"Metlay, J. P., Kapoor, W. N. & Fine, M. J. Does this patient have community-acquired pneumonia? Diagnosing pneumonia by history and physical examination. JAMA 278, 1440\u20131445 (1997).","journal-title":"JAMA"},{"key":"189_CR44","unstructured":"Yao, L. et al. Learning to diagnose from scratch by exploiting dependencies among labels. Preprint at arXiv:1710.10501 (2017)."},{"key":"189_CR45","doi-asserted-by":"crossref","unstructured":"Rosenberg, L. & Pescetelli, N. Amplifying Prediction Accuracy Using Swarm A.I. In: Intelligent Systems Conference (IntelliSys). (IEEE, 2017).","DOI":"10.1109\/IntelliSys.2017.8324329"},{"key":"189_CR46","doi-asserted-by":"crossref","unstructured":"Rosenberg, L., Baltaxe, D. & Pescetelli, N. Crowds vs. swarms, a comparison of intelligence. In: Proc. 2016 Swarm\/Human Blended Intelligence Workshop (SHBI), 1\u20134 (2016).","DOI":"10.1109\/SHBI.2016.7780278"},{"key":"189_CR47","doi-asserted-by":"crossref","unstructured":"Rosenberg, L. et al. Artificial swarm intelligence employed to amplify diagnostic accuracy in radiology. In: Proc. 2018 IEEE 9th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON) 1186\u20131191 (2018).","DOI":"10.1109\/IEMCON.2018.8614883"}],"updated-by":[{"DOI":"10.1038\/s41746-019-0198-6","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2019,12,10]],"date-time":"2019-12-10T00:00:00Z","timestamp":1575936000000}}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0189-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0189-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0189-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,17]],"date-time":"2022-12-17T18:34:37Z","timestamp":1671302077000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0189-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11,18]]},"references-count":47,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2019,12]]}},"alternative-id":["189"],"URL":"https:\/\/doi.org\/10.1038\/s41746-019-0189-7","relation":{"correction":[{"id-type":"doi","id":"10.1038\/s41746-019-0198-6","asserted-by":"object"}]},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,11,18]]},"assertion":[{"value":"25 February 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 October 2019","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 November 2019","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 December 2019","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Correction","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"An amendment to this paper has been published and can be accessed via a link at the top of the paper.","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors had control of the data and the information submitted for publication. Four authors (L.R., D.B., G.W. and M.L.) are employees of Unanimous AI, who developed the swarm platform used in this study. All other authors are not employees of or consultants for Unanimous AI and had control of the study methodology, data analysis, and results. There was no industry support specifically for this study. This study was supported in part by NSF through Award ID 1840937.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"111"}}