{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,29]],"date-time":"2026-10-29T00:52:05Z","timestamp":1793235125461,"version":"4.5.0"},"reference-count":76,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2022,6,29]],"date-time":"2022-06-29T00:00:00Z","timestamp":1656460800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000180","name":"U.S. Department of Homeland Security","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000180","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:p>This work investigates how different forms of input elicitation obtained from crowdsourcing can be utilized to improve the quality of inferred labels for image classification tasks, where an image must be labeled as either positive or negative depending on the presence\/absence of a specified object. Five types of input elicitation methods are tested: binary classification (positive or negative); the (<jats:italic>x, y<\/jats:italic>)-coordinate of the position participants believe a target object is located; level of confidence in binary response (on a scale from 0 to 100%); what participants believe the majority of the other participants' binary classification is; and participant's perceived difficulty level of the task (on a discrete scale). We design two crowdsourcing studies to test the performance of a variety of input elicitation methods and utilize data from over 300 participants. Various existing voting and machine learning (ML) methods are applied to make the best use of these inputs. In an effort to assess their performance on classification tasks of varying difficulty, a systematic synthetic image generation process is developed. Each generated image combines items from the <jats:italic>MPEG-7 Core Experiment CE-Shape-1 Test Set<\/jats:italic> into a single image using multiple parameters (e.g., density, transparency, etc.) and may or may not contain a target object. The difficulty of these images is validated by the performance of an automated image classification method. Experiment results suggest that more accurate results can be achieved with smaller training datasets when both the crowdsourced binary classification labels and the average of the self-reported confidence values in these labels are used as features for the ML classifiers. Moreover, when a relatively larger properly annotated dataset is available, in some cases augmenting these ML algorithms with the results (i.e., probability of outcome) from an automated classifier can achieve even higher performance than what can be obtained by using any one of the individual classifiers. Lastly, supplementary analysis of the collected data demonstrates that other performance metrics of interest, namely reduced false-negative rates, can be prioritized through special modifications of the proposed aggregation methods.<\/jats:p>","DOI":"10.3389\/frai.2022.848056","type":"journal-article","created":{"date-parts":[[2022,6,29]],"date-time":"2022-06-29T05:07:38Z","timestamp":1656479258000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Improving Crowdsourcing-Based Image Classification Through Expanded Input Elicitation and Machine Learning"],"prefix":"10.3389","volume":"5","author":[{"given":"Romena","family":"Yasmin","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Md Mahmudulla","family":"Hassan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joshua T.","family":"Grassel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Harika","family":"Bhogaraju","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adolfo R.","family":"Escobedo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Olac","family":"Fuentes","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2022,6,29]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2001.08856","article-title":"Stochastic optimization of plain convolutional neural networks with simple methods","author":"Assiri","year":"2020","journal-title":"arXiv [Preprint] arXiv:2001.08856"},{"key":"B2","doi-asserted-by":"publisher","first-page":"54","DOI":"10.4401\/ag-5324","article-title":"Crowdsourcing earthquake damage assessment using remote sensing imagery","author":"Barrington","year":"2012","journal-title":"Ann. Geophys"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2103.07579","article-title":"Revisiting resnets: improved training and scaling strategies","author":"Bello","year":"2021","journal-title":"arXiv [Preprint] arXiv:2103.07579"},{"key":"B4","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9781107446984","volume-title":"Handbook of Computational Social Choice","author":"Brandt","year":"2016"},{"key":"B5","doi-asserted-by":"crossref","first-page":"2334","DOI":"10.1145\/3025453.3026044","article-title":"Revolt: collaborative crowdsourcing for labeling machine learning datasets","volume-title":"Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems","author":"Chang","year":"2017"},{"key":"B6","doi-asserted-by":"publisher","first-page":"280","DOI":"10.1016\/j.media.2019.03.009","article-title":"Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis","volume":"54","author":"Cheplygina","year":"2019","journal-title":"Med. Image Anal"},{"key":"B7","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1007\/978-3-030-01364-6_12","article-title":"Crowd disagreement about medical images is informative","volume-title":"Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis","author":"Cheplygina","year":"2018"},{"key":"B8","doi-asserted-by":"publisher","first-page":"e0252604","DOI":"10.1371\/journal.pone.0252604","article-title":"An experimental characterization of workers' behavior and accuracy in crowdsourced tasks","volume":"16","author":"Christoforou","year":"2021","journal-title":"PLoS ONE"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2106.04803","article-title":"Coatnet: Marrying convolution and attention for all data sizes","author":"Dai","year":"2021","journal-title":"ArXiv, abs\/2106.04803"},{"key":"B10","doi-asserted-by":"crossref","first-page":"248","DOI":"10.1109\/CVPR.2009.5206848","article-title":"Imagenet: A large-scale hierarchical image database","volume-title":"2009 IEEE Conference on Computer Vision and Pattern Recognition","author":"Deng","year":"2009"},{"key":"B11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/ICCCN.2017.8038465","article-title":"A study and comparison of human and deep learning recognition performance under visual distortions","volume-title":"2017 26th International Conference on Computer Communication and Networks (ICCCN)","author":"Dodge","year":"2017"},{"key":"B12","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1145\/3159652.3159654","article-title":"Cognitive biases in crowdsourcing","volume-title":"Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining","author":"Eickhoff","year":"2018"},{"key":"B13","doi-asserted-by":"publisher","first-page":"322","DOI":"10.1016\/j.ins.2021.12.124","article-title":"An axiomatic distance methodology for aggregating multimodal evaluations","volume":"590","author":"Escobedo","year":"2022","journal-title":"Inform. Sci"},{"key":"B14","first-page":"226","article-title":"A density-based algorithm for discovering clusters in large spatial databases with noise","volume-title":"KDD, Vol. 96","author":"Ester","year":"1996"},{"key":"B15","doi-asserted-by":"publisher","first-page":"303","DOI":"10.1007\/s11263-009-0275-4","article-title":"The pascal visual object classes (VOC) challenge","volume":"88","author":"Everingham","year":"2010","journal-title":"Int. J. Comput. Vis"},{"key":"B16","doi-asserted-by":"publisher","first-page":"80","DOI":"10.3390\/ijgi7030080","article-title":"Increasing the accuracy of crowdsourced information on land cover via a voting procedure weighted by information inferred from the contributed data","volume":"7","author":"Foody","year":"2018","journal-title":"ISPRS Int. J. Geo-Inform"},{"key":"B17","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1706.06969","article-title":"Comparing deep neural networks against humans: object recognition when the signal gets weaker","author":"Geirhos","year":"2017","journal-title":"arXiv [Preprint] arXiv:1706.06969"},{"key":"B18","doi-asserted-by":"publisher","first-page":"4571","DOI":"10.1073\/pnas.1906831117","article-title":"Expert-augmented machine learning","volume":"117","author":"Gennatas","year":"2020","journal-title":"Proc. Natl. Acad. Sci. U.S.A"},{"key":"B19","volume-title":"Extracting the Wisdom From the Crowd: A Comparison of Approaches to Aggregating Collective Intelligence","author":"G\u00f6rzen","year":"2019"},{"key":"B20","doi-asserted-by":"publisher","first-page":"411","DOI":"10.1016\/0010-0285(92)90013-R","article-title":"The weighing of evidence and the determinants of confidence","volume":"24","author":"Griffin","year":"1992","journal-title":"Cogn. Psychol"},{"key":"B21","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1007\/BF00125672","article-title":"Thirteen theorems in search of the truth","volume":"15","author":"Grofman","year":"1983","journal-title":"Theory Decis"},{"key":"B22","doi-asserted-by":"crossref","first-page":"1169","DOI":"10.1109\/WACV.2015.160","article-title":"How to collect segmentations for biomedical images? A benchmark evaluating the performance of experts, crowdsourced non-experts, and algorithms","volume-title":"2015 IEEE Winter Conference on Applications of Computer Vision","author":"Gurari","year":"2015"},{"key":"B23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s41235-020-00248-z","article-title":"Wisdom of crowds and collective decision-making in a survival situation with complex information integration","volume":"5","author":"Hamada","year":"2020","journal-title":"Cogn. Res"},{"key":"B24","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1145\/2384916.2384989","article-title":"A feasibility study of crowdsourcing and google street view to determine sidewalk accessibility","volume-title":"Proceedings of the 14th International ACM SIGACCESS Conference on Computers and Accessibility","author":"Hara","year":"2012"},{"key":"B25","doi-asserted-by":"publisher","first-page":"494","DOI":"10.1037\/0033-295X.112.2.494","article-title":"The robust beauty of majority rules in group decisions","volume":"112","author":"Hastie","year":"2005","journal-title":"Psychol. Rev"},{"key":"B26","first-page":"57","article-title":"Do you need experts in the crowd? A case study in image annotation for marine biology","volume-title":"Proceedings of the 10th Conference on Open Research Areas in Information Retrieval","author":"He","year":"2013"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90","article-title":"Deep residual learning for image recognition","author":"He","year":"2015","journal-title":"arXiv [Preprint] arXiv:1512.03385"},{"key":"B28","doi-asserted-by":"crossref","first-page":"114","DOI":"10.1016\/j.ejca.2019.07.019","article-title":"Superior skin cancer classification by the combination of human and artificial intelligence","volume":"120","author":"Hekler","year":"2019","journal-title":"Eur. J. Cancer"},{"key":"B29","doi-asserted-by":"publisher","first-page":"361","DOI":"10.1002\/rse2.84","article-title":"Economical crowdsourcing for camera trap image classification","volume":"4","author":"Hsing","year":"2018","journal-title":"Remote Sens. Ecol. Conserv"},{"key":"B30","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1145\/1837885.1837906","article-title":"Quality management on amazon mechanical Turk","volume-title":"Proceedings of the ACM SIGKDD Workshop on Human Computation","author":"Ipeirotis","year":"2010"},{"key":"B31","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/srep43286","article-title":"Crowdsourcing scoring of immunohistochemistry images: evaluating performance of the crowd and an automated computational method","volume":"7","author":"Irshad","year":"2017","journal-title":"Sci. Rep"},{"key":"B32","volume-title":"Description of Core Experiments for MPEG-7 Motion\/Shape","author":"Jeannin","year":"1999"},{"key":"B33","article-title":"Iterative learning for reliable crowdsourcing systems","volume-title":"Neural Information Processing Systems","author":"Karger","year":"2011"},{"key":"B34","first-page":"73","article-title":"Enhancing collective estimates by aggregating cardinal and ordinal inputs","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 8","author":"Kemmer","year":"2020"},{"key":"B35","first-page":"5","article-title":"Quality control of crowd labeling through expert evaluation","volume-title":"Proceedings of the NIPS 2nd Workshop on Computational Social Science and the Wisdom of Crowds, Vol. 2","author":"Khattak","year":"2011"},{"key":"B36","doi-asserted-by":"crossref","first-page":"2093","DOI":"10.1109\/TENCON.2017.8228206","article-title":"Preliminary investigation of augmented intelligence for remote assistance using a wearable display","volume-title":"TENCON 2017-2017 IEEE Region 10 Conference","author":"Koh","year":"2017"},{"key":"B37","doi-asserted-by":"publisher","first-page":"80","DOI":"10.1037\/a0025648","article-title":"The self-consistency model of subjective confidence","volume":"119","author":"Koriat","year":"2012","journal-title":"Psychol. Rev"},{"key":"B38","article-title":"Imagenet classification with deep convolutional neural networks","volume-title":"Advances in Neural Information Processing Systems, Vol. 25","author":"Krizhevsky","year":"2012"},{"key":"B39","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1145\/3055601.3055607","article-title":"Does confidence reporting from the crowd benefit crowdsourcing performance?","volume-title":"Proceedings of the 2nd International Workshop on Social Sensing","author":"Li","year":"2017"},{"key":"B40","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1037\/dec0000119","article-title":"How the \u201cwisdom of the inner crowd\u201d can boost accuracy of confidence judgments","volume":"7","author":"Litvinova","year":"2020","journal-title":"Decision"},{"key":"B41","doi-asserted-by":"publisher","first-page":"276","DOI":"10.1037\/a0036677","article-title":"The wisdom of select crowds","volume":"107","author":"Mannes","year":"2014","journal-title":"J. Pers. Soc. Psychol"},{"key":"B42","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v27i1.8460","article-title":"Better human computation through principled voting","volume-title":"AAAI","author":"Mao","year":"2013"},{"key":"B43","volume-title":"Wisdom of the crowd: comparison of the CWM, simple average and surprisingly popular answer method","author":"Matoulkova","year":"2017"},{"key":"B44","doi-asserted-by":"publisher","first-page":"e37245","DOI":"10.1371\/journal.pone.0037245","article-title":"Distributed medical image analysis and diagnosis through crowd-sourced games: a malaria case study","volume":"7","author":"Mavandadi","year":"2012","journal-title":"PLoS ONE"},{"key":"B45","doi-asserted-by":"publisher","first-page":"68","DOI":"10.1109\/MSP.2016.51","article-title":"Machine learning in adversarial settings","volume":"14","author":"McDaniel","year":"2016","journal-title":"IEEE Secur. Privacy"},{"key":"B46","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s41235-021-00279-0","article-title":"Group decisions based on confidence weighted majority voting","volume":"6","author":"Meyen","year":"2021","journal-title":"Cogn. Res"},{"key":"B47","doi-asserted-by":"publisher","first-page":"e71154","DOI":"10.1371\/journal.pone.0071154","article-title":"Crowdsourcing as a novel technique for retinal fundus photography classification: analysis of images in the epic norfolk cohort on behalf of the ukbiobank eye and vision consortium","volume":"8","author":"Mitry","year":"2013","journal-title":"PLoS ONE"},{"key":"B48","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1167\/tvst.5.5.6","article-title":"The accuracy and reliability of crowdsource annotations of digital retinal images","volume":"5","author":"Mitry","year":"2016","journal-title":"Transl. Vis. Sci. Technol"},{"key":"B49","doi-asserted-by":"crossref","first-page":"309","DOI":"10.5220\/0010214503090318","article-title":"Who wants to use an augmented reality shopping assistant application?\u201d","volume-title":"Proceedings of the 4th International Conference on Computer-Human Interaction Research and Applications - WUDESHI-DR","author":"Mora","year":"2020"},{"key":"B50","doi-asserted-by":"publisher","first-page":"366","DOI":"10.1002\/jrsm.1252","article-title":"An exploration of crowdsourcing citation screening for systematic reviews","volume":"8","author":"Mortensen","year":"2017","journal-title":"Res. Synthes. Methods"},{"key":"B51","doi-asserted-by":"publisher","first-page":"824","DOI":"10.1148\/radiol.11110938","article-title":"Distributed human intelligence for colonic polyp classification in computer-aided detection for CT colonography","volume":"262","author":"Nguyen","year":"2012","journal-title":"Radiology"},{"key":"B52","doi-asserted-by":"crossref","first-page":"557","DOI":"10.1145\/1743384.1743478","article-title":"How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation","volume-title":"Proceedings of the International Conference on Multimedia Information Retrieval","author":"Nowak","year":"2010"},{"key":"B53","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1145\/2615569.2615644","article-title":"Crowdsourcing knowledge-intensive tasks in cultural heritage","volume-title":"Proceedings of the 2014 ACM Conference on Web Science","author":"Oosterman","year":"2014"},{"key":"B54","article-title":"Accurate integration of crowdsourced labels using workers' self-reported confidence scores","volume-title":"Twenty-Third International Joint Conference on Artificial Intelligence","author":"Oyama","year":"2013"},{"key":"B55","doi-asserted-by":"crossref","first-page":"372","DOI":"10.1109\/EuroSP.2016.36","article-title":"The limitations of deep learning in adversarial settings","volume-title":"2016 IEEE European Symposium on Security and Privacy (EuroS&P)","author":"Papernot","year":"2016"},{"key":"B56","doi-asserted-by":"publisher","first-page":"2825","DOI":"10.48550\/arXiv.1201.0490","article-title":"Scikit-learn: machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res"},{"key":"B57","doi-asserted-by":"publisher","first-page":"532","DOI":"10.1038\/nature21054","article-title":"A solution to the single-question crowd wisdom problem","volume":"541","author":"Prelec","year":"2017","journal-title":"Nature"},{"key":"B58","unstructured":"Mpeg-7 core experiment ce-shape-1 test set\n            RalphR\n          1999"},{"key":"B59","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1002\/ejsp.2420100303","article-title":"A comparison of ranking and rating procedures for value system measurement","volume":"10","author":"Rankin","year":"1980","journal-title":"Eur. J. Soc. Psychol"},{"key":"B60","doi-asserted-by":"publisher","first-page":"E1980","DOI":"10.1175\/BAMS-D-19-0324.1","article-title":"Combining crowdsourcing and deep learning to explore the mesoscale organization of shallow convection","volume":"101","author":"Rasp","year":"2020","journal-title":"Bull. Am. Meteorol. Soc"},{"key":"B61","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"Imagenet large scale visual recognition challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vis"},{"key":"B62","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.cogsys.2019.04.004","article-title":"Modelling cognitive bias in crowdsourcing systems","volume":"58","author":"Saab","year":"2019","journal-title":"Cogn. Syst. Res"},{"key":"B63","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-020-80500-0","article-title":"Wisdom of crowds benefits perceptual decision making across difficulty levels","volume":"11","author":"Saha Roy","year":"2021","journal-title":"Sci. Rep"},{"key":"B64","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v27i1.8465","article-title":"Hotspotting-a probabilistic graphical model for image object localization through crowdsourcing","volume-title":"Twenty-Seventh AAAI Conference on Artificial Intelligence","author":"Salek","year":"2013"},{"key":"B65","first-page":"1015","article-title":"Beyond accuracy, f-score and roc: a family of discriminant measures for performance evaluation","volume-title":"Australasian Joint Conference on Artificial Intelligence","author":"Sokolova","year":"2006"},{"key":"B66","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1002\/qj.3662","article-title":"Sugar, gravel, fish and flowers: mesoscale cloud patterns in the trade winds","volume":"146","author":"Stevens","year":"2020","journal-title":"Q. J. R. Meteorol. Soc"},{"key":"B67","volume-title":"The Wisdom of Crowds","author":"Surowiecki","year":"2005"},{"key":"B68","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/sdata.2015.26","article-title":"Snapshot serengeti, high-frequency annotated camera trap images of 40 mammalian species in an African savanna","volume":"2","author":"Swanson","year":"2015","journal-title":"Sci. Data"},{"key":"B69","first-page":"6105","article-title":"EfficientNet: rethinking model scaling for convolutional neural networks","volume-title":"International Conference on Machine Learning","author":"Tan","year":"2019"},{"key":"B70","doi-asserted-by":"crossref","first-page":"1183","DOI":"10.1145\/2145204.2145380","article-title":"A reference-based scoring model for increasing the findability of promising ideas in innovation pipelines","volume-title":"Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work","author":"Xu","year":"2012"},{"key":"B71","first-page":"166","article-title":"Enhancing image classification capabilities of crowdsourcing-based methods through expanded input elicitation","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 9","author":"Yasmin","year":"2021"},{"key":"B72","doi-asserted-by":"publisher","first-page":"452","DOI":"10.1111\/j.1551-6709.2011.01223.x","article-title":"The wisdom of the crowd in combinatorial problems","volume":"36","author":"Yi","year":"2012","journal-title":"Cogn. Sci"},{"key":"B73","doi-asserted-by":"publisher","first-page":"1025","DOI":"10.1016\/j.ejor.2020.02.027","article-title":"A new correlation coefficient for comparing and aggregating non-strict and incomplete rankings","volume":"285","author":"Yoo","year":"2020","journal-title":"Eur. J. Oper. Res"},{"key":"B74","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2106.04560","article-title":"Scaling vision transformers","author":"Zhai","year":"2021","journal-title":"arXiv [Preprint] arXiv:2106.04560"},{"key":"B75","doi-asserted-by":"publisher","first-page":"487","DOI":"10.1101\/265918","article-title":"Learning deep features for scene recognition using places database","volume":"27","author":"Zhou","year":"2014","journal-title":"Adv. Neural Inform. Process. Syst"},{"key":"B76","doi-asserted-by":"publisher","first-page":"e1006337","DOI":"10.1371\/journal.pcbi.1006337","article-title":"Crowdsourcing image analysis for plant phenomics to generate ground truth data for machine learning","volume":"14","author":"Zhou","year":"2018","journal-title":"PLoS Comput. Biol"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2022.848056\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,6,29]],"date-time":"2022-06-29T05:08:42Z","timestamp":1656479322000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2022.848056\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,29]]},"references-count":76,"alternative-id":["10.3389\/frai.2022.848056"],"URL":"https:\/\/doi.org\/10.3389\/frai.2022.848056","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,6,29]]},"article-number":"848056"}}