{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T14:48:35Z","timestamp":1786978115376,"version":"build-2736575974"},"reference-count":46,"publisher":"Wiley","issue":"5","license":[{"start":{"date-parts":[[2023,6,20]],"date-time":"2023-06-20T00:00:00Z","timestamp":1687219200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"funder":[{"DOI":"10.13039\/501100002424","name":"Fujifilm Corporation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100002424","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Research Synthesis Methods"],"published-print":{"date-parts":[[2023,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>There are currently no abstract classifiers, which can be used for new diagnostic test accuracy (DTA) systematic reviews to select primary DTA study abstracts from database searches. Our goal was to develop machine\u2010learning\u2010based abstract classifiers for new DTA systematic reviews through an open competition. We prepared a dataset of abstracts obtained through database searches from 11 reviews in different clinical areas. As the reference standard, we used the abstract lists that required manual full\u2010text review. We randomly splitted the datasets into a train set, a public test set, and a private test set. Competition participants used the training set to develop classifiers and validated their classifiers using the public test set. The classifiers were refined based on the performance of the public test set. They could submit as many times as they wanted during the competition. Finally, we used the private test set to rank the submitted classifiers. To reduce false exclusions, we used the Fbeta measure with a beta set to seven for evaluating classifiers. After the competition, we conducted the external validation using a dataset from a cardiology DTA review. We received 13,774 submissions from 1429 teams or persons over 4 months. The top\u2010honored classifier achieved a Fbeta score of 0.4036 and a recall of 0.2352 in the external validation. In conclusion, we were unable to develop an abstract classifier with sufficient recall for immediate application to new DTA systematic reviews. Further studies are needed to update and validate classifiers with datasets from other clinical areas.<\/jats:p>","DOI":"10.1002\/jrsm.1649","type":"journal-article","created":{"date-parts":[[2023,6,20]],"date-time":"2023-06-20T03:16:23Z","timestamp":1687230983000},"page":"707-717","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["An open competition involving thousands of competitors failed to construct useful abstract classifiers for new diagnostic test accuracy systematic reviews"],"prefix":"10.1002","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7982-5213","authenticated-orcid":false,"given":"Yuki","family":"Kataoka","sequence":"first","affiliation":[{"name":"Department of Internal Medicine Kyoto Min\u2010iren Asukai Hospital  Kyoto Japan"},{"name":"Scientific Research Works Peer Support Group (SRWS\u2010PSG)  Osaka Japan"},{"name":"Section of Clinical Epidemiology, Department of Community Medicine Kyoto University Graduate School of Medicine  Kyoto Japan"},{"name":"Department of Healthcare Epidemiology Kyoto University Graduate School of Medicine\/School of Public Health  Kyoto Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1218-4225","authenticated-orcid":false,"given":"Shunsuke","family":"Taito","sequence":"additional","affiliation":[{"name":"Scientific Research Works Peer Support Group (SRWS\u2010PSG)  Osaka Japan"},{"name":"Division of Rehabilitation, Department of Clinical Practice and Support Hiroshima University Hospital  Hiroshima Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7902-9994","authenticated-orcid":false,"given":"Norio","family":"Yamamoto","sequence":"additional","affiliation":[{"name":"Scientific Research Works Peer Support Group (SRWS\u2010PSG)  Osaka Japan"},{"name":"Department of Orthopedic Surgery Miyamoto Orthopedic Hospital  Okayama Japan"},{"name":"Department of Epidemiology Graduate School of Medicine, Dentistry and Pharmaceutical Sciences, Okayama University  Okayama Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9838-350X","authenticated-orcid":false,"given":"Ryuhei","family":"So","sequence":"additional","affiliation":[{"name":"Scientific Research Works Peer Support Group (SRWS\u2010PSG)  Osaka Japan"},{"name":"Department of Psychiatry Okayama Psychiatric Medical Center  Okayama Japan"},{"name":"CureApp, Inc.  Tokyo Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9160-0241","authenticated-orcid":false,"given":"Yusuke","family":"Tsutsumi","sequence":"additional","affiliation":[{"name":"Scientific Research Works Peer Support Group (SRWS\u2010PSG)  Osaka Japan"},{"name":"Department of Healthcare Epidemiology Kyoto University Graduate School of Medicine\/School of Public Health  Kyoto Japan"},{"name":"Department of Emergency Medicine National Hospital Organization Mito Medical Center  Ibaraki Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2701-7334","authenticated-orcid":false,"given":"Keisuke","family":"Anan","sequence":"additional","affiliation":[{"name":"Scientific Research Works Peer Support Group (SRWS\u2010PSG)  Osaka Japan"},{"name":"Division of Respiratory Medicine Saiseikai Kumamoto Hospital  Kumamoto Japan"},{"name":"Department of Healthcare Epidemiology Graduate School of Medicine and Public Health, Kyoto University  Kyoto Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2539-1031","authenticated-orcid":false,"given":"Masahiro","family":"Banno","sequence":"additional","affiliation":[{"name":"Scientific Research Works Peer Support Group (SRWS\u2010PSG)  Osaka Japan"},{"name":"Department of Psychiatry Seichiryo Hospital  Nagoya Japan"},{"name":"Department of Psychiatry Nagoya University Graduate School of Medicine  Nagoya Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7214-5589","authenticated-orcid":false,"given":"Yasushi","family":"Tsujimoto","sequence":"additional","affiliation":[{"name":"Scientific Research Works Peer Support Group (SRWS\u2010PSG)  Osaka Japan"},{"name":"Oku Medical Clinic  Osaka Japan"},{"name":"Department of Health Promotion and Human Behavior Kyoto University Graduate School of Medicine\/School of Public Health, Kyoto University  Kyoto Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2191-3629","authenticated-orcid":false,"given":"Yoshitaka","family":"Wada","sequence":"additional","affiliation":[{"name":"Scientific Research Works Peer Support Group (SRWS\u2010PSG)  Osaka Japan"},{"name":"Department of Rehabilitation Medicine School of Medicine, Fujita Health University  Toyoake Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1202-8161","authenticated-orcid":false,"given":"Shintaro","family":"Sagami","sequence":"additional","affiliation":[{"name":"Center for Advanced IBD Research and Treatment Kitasato University Kitasato Institute Hospital  Tokyo Japan"},{"name":"Department of Gastroenterology and Hepatology Kitasato University Kitasato Institute Hospital  Tokyo Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5847-4132","authenticated-orcid":false,"given":"Hiraku","family":"Tsujimoto","sequence":"additional","affiliation":[{"name":"Hospital Care Research Unit Hyogo Prefectural Amagasaki General Medical Center  Amagasaki Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Takashi","family":"Nihashi","sequence":"additional","affiliation":[{"name":"Department of Radiology Komaki City Hospital  Komaki Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Motoki","family":"Takeuchi","sequence":"additional","affiliation":[{"name":"Department of Emergency and General Internal Medicine Fujita Health University School of Medicine  Toyoake Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Teruhiko","family":"Terasawa","sequence":"additional","affiliation":[{"name":"Section of General Internal Medicine, Department of Emergency and General Internal Medicine Fujita Health University School of Medicine  Toyoake Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Masahiro","family":"Iguchi","sequence":"additional","affiliation":[{"name":"Department of Neurology Fukushima Medical University  Fukushima Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4619-945X","authenticated-orcid":false,"given":"Junji","family":"Kumasawa","sequence":"additional","affiliation":[{"name":"Human Health Sciences Kyoto University Graduate School of Medicine  Kyoto Japan"},{"name":"Department of Critical Care Medicine Sakai City Medical Center  Sakai Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Takumi","family":"Ichikawa","sequence":"additional","affiliation":[{"name":"Yahoo Japan Corporation  Tokyo Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ryuki","family":"Furukawa","sequence":"additional","affiliation":[{"name":"Yahoo Japan Corporation  Tokyo Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jun","family":"Yamabe","sequence":"additional","affiliation":[{"name":"Self\u2010employed Engineer  Tokyo Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2159-3776","authenticated-orcid":false,"given":"Toshi A.","family":"Furukawa","sequence":"additional","affiliation":[{"name":"Department of Health Promotion and Human Behavior Kyoto University Graduate School of Medicine\/School of Public Health, Kyoto University  Kyoto Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2023,6,20]]},"reference":[{"key":"e_1_2_12_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejrad.2022.110447"},{"key":"e_1_2_12_3_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-020-01174-9"},{"key":"e_1_2_12_4_1","doi-asserted-by":"publisher","DOI":"10.37737\/ace.22014"},{"key":"e_1_2_12_5_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-022-01895-z"},{"key":"e_1_2_12_6_1","doi-asserted-by":"publisher","DOI":"10.2196\/26911"},{"key":"e_1_2_12_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2022.04.027"},{"key":"e_1_2_12_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2021.01.010"},{"key":"e_1_2_12_9_1","doi-asserted-by":"publisher","DOI":"10.1002\/jrsm.1537"},{"key":"e_1_2_12_10_1","doi-asserted-by":"publisher","DOI":"10.1002\/jrsm.1486"},{"key":"e_1_2_12_11_1","article-title":"Search strategies to identify diagnostic accuracy studies in MEDLINE and EMBASE","volume":"9","author":"Beynon R","year":"2013","journal-title":"Cochrane Database Syst Rev"},{"key":"e_1_2_12_12_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2016-012799"},{"key":"e_1_2_12_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2021.12.014"},{"key":"e_1_2_12_14_1","unstructured":"Data science bowl 2017 [Internet]. [cited 2022 Sep 1]. Available from:https:\/\/www.kaggle.com\/c\/data-science-bowl-2017"},{"key":"e_1_2_12_15_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2017.14585"},{"key":"e_1_2_12_16_1","doi-asserted-by":"publisher","DOI":"10.1002\/mrm.28338"},{"key":"e_1_2_12_17_1","doi-asserted-by":"publisher","DOI":"10.7326\/M14-0697"},{"key":"e_1_2_12_18_1","unstructured":"[cited 2022 Sep 9]. Available from:http:\/\/eppi.ioe.ac.uk\/COVID19_MAP\/covid_map_v37.html"},{"key":"e_1_2_12_19_1","doi-asserted-by":"publisher","DOI":"10.1183\/23120541.00464-2020"},{"key":"e_1_2_12_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cgh.2020.07.067"},{"key":"e_1_2_12_21_1","doi-asserted-by":"publisher","DOI":"10.1111\/cea.13685"},{"issue":"4","key":"e_1_2_12_22_1","article-title":"Ultrasonography for confirmation of gastric tube placement","volume":"4","author":"Tsujimoto H","year":"2017","journal-title":"Cochrane Database Syst Rev"},{"key":"e_1_2_12_23_1","doi-asserted-by":"publisher","DOI":"10.1177\/2048872620901831"},{"key":"e_1_2_12_24_1","doi-asserted-by":"publisher","DOI":"10.3174\/ajnr.A3324"},{"key":"e_1_2_12_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00259-019-04480-8"},{"key":"e_1_2_12_26_1","doi-asserted-by":"publisher","DOI":"10.3233\/JAD-150675"},{"key":"e_1_2_12_27_1","doi-asserted-by":"publisher","DOI":"10.2967\/jnumed.116.174391"},{"issue":"6","key":"e_1_2_12_28_1","article-title":"Diagnostic test accuracy of jolt accentuation for headache in acute meningitis in the emergency setting","volume":"6","author":"Iguchi M","year":"2020","journal-title":"Cochrane Database Syst Rev"},{"key":"e_1_2_12_29_1","unstructured":"PowersDMW.What the F\u2010measure doesn't measure: features flaws fallacies and fixes [Internet]. arXiv [cs.IR].2015Available from:http:\/\/arxiv.org\/abs\/1503.06410"},{"key":"e_1_2_12_30_1","doi-asserted-by":"publisher","DOI":"10.5195\/jmla.2020.912"},{"issue":"5","key":"e_1_2_12_31_1","article-title":"Doppler trans\u2010thoracic echocardiography for detection of pulmonary hypertension in adults","volume":"5","author":"Tsujimoto Y","year":"2022","journal-title":"Cochrane Database Syst Rev"},{"key":"e_1_2_12_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2005.50"},{"key":"e_1_2_12_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2009.11.009"},{"key":"e_1_2_12_34_1","doi-asserted-by":"publisher","DOI":"10.1097\/EDE.0b013e3181c30fb2"},{"key":"e_1_2_12_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4842-4470-8_7"},{"key":"e_1_2_12_36_1","unstructured":"Posit Team.Posit Cloud [Internet]. Posit Cloud: Integrated Development for R. RStudio PBC Boston MA.2022Available from:https:\/\/posit.cloud\/"},{"key":"e_1_2_12_37_1","unstructured":"DevlinJ ChangM\u2010W LeeK ToutanovaK.BERT: Pre\u2010training of deep bidirectional Transformers for language understanding.2018Available from:10.48550\/ARXIV.1810.04805."},{"key":"e_1_2_12_38_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-022-01606-z"},{"key":"e_1_2_12_39_1","unstructured":"Kaggle competitions [Internet]. [cited 2022 Oct 24]. Available from:https:\/\/www.kaggle.com\/competitions?sortOption=numTeams&searchQuery=medicine"},{"key":"e_1_2_12_40_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.m1328"},{"key":"e_1_2_12_41_1","article-title":"Rapid, point\u2010of\u2010care antigen tests for diagnosis of SARS\u2010CoV\u20102 infection","volume":"7","author":"Dinnes J","year":"2022","journal-title":"Cochrane Database Syst Rev"},{"issue":"5","key":"e_1_2_12_42_1","article-title":"Signs and symptoms to determine if a patient presenting in primary care or hospital outpatient settings has COVID\u201019","volume":"5","author":"Struyf T","year":"2022","journal-title":"Cochrane Database Syst Rev"},{"key":"e_1_2_12_43_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-022-01961-6"},{"key":"e_1_2_12_44_1","first-page":"402","volume-title":"Lecture Notes in Computer Science","author":"Li Z","year":"2019"},{"key":"e_1_2_12_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13347-019-00382-7"},{"key":"e_1_2_12_46_1","doi-asserted-by":"publisher","DOI":"10.1136\/heartjnl-2011-301247"},{"key":"e_1_2_12_47_1","doi-asserted-by":"crossref","unstructured":"LuoR SunL XiaY et al.BioGPT: generative pre\u2010trained transformer for biomedical text generation and mining [Internet]. arXiv [cs.CL].2022Available from:http:\/\/arxiv.org\/abs\/2210.10341","DOI":"10.1093\/bib\/bbac409"}],"container-title":["Research Synthesis Methods"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/jrsm.1649","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,14]],"date-time":"2023-09-14T09:10:21Z","timestamp":1694682621000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/jrsm.1649"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,20]]},"references-count":46,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2023,9]]}},"alternative-id":["10.1002\/jrsm.1649"],"URL":"https:\/\/doi.org\/10.1002\/jrsm.1649","archive":["Portico"],"relation":{},"ISSN":["1759-2879","1759-2887"],"issn-type":[{"value":"1759-2879","type":"print"},{"value":"1759-2887","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,20]]},"assertion":[{"value":"2022-12-14","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-06-07","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-06-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}