{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,26]],"date-time":"2026-02-26T20:34:56Z","timestamp":1772138096647,"version":"3.50.1"},"reference-count":59,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2019,9,3]],"date-time":"2019-09-03T00:00:00Z","timestamp":1567468800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/100000002","name":"US National Institute of Health","doi-asserted-by":"publisher","award":["U54GM114833"],"award-info":[{"award-number":["U54GM114833"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Scripps Translational Science Institute"},{"DOI":"10.13039\/100006108","name":"NIH-NCATS Clinical and Translational Science Award","doi-asserted-by":"publisher","award":["5 UL1 RR025774"],"award-info":[{"award-number":["5 UL1 RR025774"]}],"id":[{"id":"10.13039\/100006108","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,2,15]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Motivation<\/jats:title>\n                    <jats:p>Biomedical literature is growing at a rate that outpaces our ability to harness the knowledge contained therein. To mine valuable inferences from the large volume of literature, many researchers use information extraction algorithms to harvest information in biomedical texts. Information extraction is usually accomplished via a combination of manual expert curation and computational methods. Advances in computational methods usually depend on the time-consuming generation of gold standards by a limited number of expert curators. Citizen science is public participation in scientific research. We previously found that citizen scientists are willing and capable of performing named entity recognition of disease mentions in biomedical abstracts, but did not know if this was true with relationship extraction (RE).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>In this article, we introduce the Relationship Extraction Module of the web-based application Mark2Cure (M2C) and demonstrate that citizen scientists can perform RE. We confirm the importance of accurate named entity recognition on user performance of RE and identify design issues that impacted data quality. We find that the data generated by citizen scientists can be used to identify relationship types not currently available in the M2C Relationship Extraction Module. We compare the citizen science-generated data with algorithm-mined data and identify ways in which the two approaches may complement one another. We also discuss opportunities for future improvement of this system, as well as the potential synergies between citizen science, manual biocuration and natural language processing.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Availability and implementation<\/jats:title>\n                    <jats:p>Mark2Cure platform: https:\/\/mark2cure.org; Mark2Cure source code: https:\/\/github.com\/sulab\/mark2cure; and data and analysis code for this article: https:\/\/github.com\/gtsueng\/M2C_rel_nb.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Supplementary information<\/jats:title>\n                    <jats:p>Supplementary data are available at Bioinformatics online.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btz678","type":"journal-article","created":{"date-parts":[[2019,8,30]],"date-time":"2019-08-30T07:48:44Z","timestamp":1567151324000},"page":"1226-1233","source":"Crossref","is-referenced-by-count":8,"title":["Applying citizen science to gene, drug and disease relationship extraction from biomedical abstracts"],"prefix":"10.1093","volume":"36","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9536-9115","authenticated-orcid":false,"given":"Ginger","family":"Tsueng","sequence":"first","affiliation":[{"name":"Department of Integrative Structural and Computational Biology, The Scripps Research Institute , La Jolla, CA 92037, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Max","family":"Nanis","sequence":"additional","affiliation":[{"name":"Department of Integrative Structural and Computational Biology, The Scripps Research Institute , La Jolla, CA 92037, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jennifer T","family":"Fouquier","sequence":"additional","affiliation":[{"name":"Department of Integrative Structural and Computational Biology, The Scripps Research Institute , La Jolla, CA 92037, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Mayers","sequence":"additional","affiliation":[{"name":"Department of Integrative Structural and Computational Biology, The Scripps Research Institute , La Jolla, CA 92037, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Benjamin M","family":"Good","sequence":"additional","affiliation":[{"name":"Department of Integrative Structural and Computational Biology, The Scripps Research Institute , La Jolla, CA 92037, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andrew I","family":"Su","sequence":"additional","affiliation":[{"name":"Department of Integrative Structural and Computational Biology, The Scripps Research Institute , La Jolla, CA 92037, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2019,9,3]]},"reference":[{"key":"2023013110135920300_btz678-B1","author":"Angeli","year":"2014"},{"key":"2023013110135920300_btz678-B2","volume-title":"ACM Web Science Conference","author":"Aroyo","year":"2013"},{"key":"2023013110135920300_btz678-B3","doi-asserted-by":"crossref","first-page":"2376","DOI":"10.1093\/mnras\/stw1067","article-title":"Radio galaxy zoo: discovery of a poor cluster through a giant wide-angle tail radio galaxy","volume":"460","author":"Banfield","year":"2016","journal-title":"Mon. Not. R. Astron. Soc"},{"key":"2023013110135920300_btz678-B4","volume-title":"Natural Language Processing with Python","author":"Bird","year":"2009"},{"key":"2023013110135920300_btz678-B301"},{"key":"2023013110135920300_btz678-B5","doi-asserted-by":"crossref","DOI":"10.1093\/database\/bau094","article-title":"Hybrid curation of gene\u2013mutation relations combining automated extraction and crowdsourcing","volume":"2014","author":"Burger","year":"2014","journal-title":"Database"},{"key":"2023013110135920300_btz678-B6","doi-asserted-by":"crossref","first-page":"W484","DOI":"10.1093\/nar\/gkx462","article-title":"LimTox: a web tool for applied text mining of adverse event and toxicity associations of compounds, drugs and genes","volume":"45","author":"Ca\u00f1ada","year":"2017","journal-title":"Nucleic Acids Res"},{"key":"2023013110135920300_btz678-B7","doi-asserted-by":"crossref","first-page":"681","DOI":"10.1016\/j.ebiom.2015.05.009","article-title":"Crowdsourcing the general public for large scale molecular pathology studies in cancer","volume":"2","author":"Candido dos Reis","year":"2015","journal-title":"Ebiomedicine"},{"key":"2023013110135920300_btz678-B8","doi-asserted-by":"crossref","first-page":"bav104.","DOI":"10.1093\/database\/bav104","article-title":"PhenoMiner: from text to a database of phenotypes associated with OMIM diseases","volume":"2015","author":"Collier","year":"2015","journal-title":"Database"},{"key":"2023013110135920300_btz678-B9","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1109\/MCSE.2015.65","article-title":"Defining and measuring success in online citizen science: a case study of zooniverse projects","volume":"17","author":"Cox","year":"2015","journal-title":"Comput. Sci. Eng"},{"key":"2023013110135920300_btz678-B10","volume-title":"International Workshop on Biomedical Data Mining, Modeling, and Semantic Integration: A Promising Approach to Solving Unmet Medical Needs","author":"Dumitrache","year":"2015"},{"key":"2023013110135920300_btz678-B11","doi-asserted-by":"crossref","first-page":"758","DOI":"10.1093\/jamia\/ocw030","article-title":"Automated identification of molecular effects of drugs (AIMED)","volume":"23","author":"Fathiamini","year":"2016","journal-title":"J. Am. Med. Inform. Assoc"},{"key":"2023013110135920300_btz678-B12","doi-asserted-by":"crossref","DOI":"10.3389\/conf.FENVS.2016.01.00011","article-title":"Data quality in citizen science projects: challenges and solutions","volume":"4","author":"Gabriele","year":"2016","journal-title":"Front. Environ. Sci"},{"key":"2023013110135920300_btz678-B13","article-title":"Microtask crowdsourcing for disease mention annotation in PubMed abstracts","author":"Good","year":"2015","journal-title":"Pac. Symp. Biocomput"},{"key":"2023013110135920300_btz678-B14","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1007\/978-94-007-4587-2_7","volume-title":"Crowdsourcing Geographic Knowledge","author":"Haklay","year":"2013"},{"key":"2023013110135920300_btz678-B15","doi-asserted-by":"crossref","DOI":"10.1186\/s13326-017-0153-x","article-title":"Semantic annotation in biomedicine: the current landscape","volume":"8","author":"Jovanovi\u0107","year":"2017","journal-title":"J. Biomed. Semantics"},{"key":"2023013110135920300_btz678-B16","doi-asserted-by":"crossref","DOI":"10.1093\/database\/bav016","article-title":"Scaling drug indication curation through crowdsourcing","volume":"2015","author":"Khare","year":"2015","journal-title":"Database"},{"key":"2023013110135920300_btz678-B17","doi-asserted-by":"crossref","first-page":"3158","DOI":"10.1093\/bioinformatics\/bts591","article-title":"SemMedDB: a PubMed-scale repository of biomedical semantic predications","volume":"28","author":"Kilicoglu","year":"2012","journal-title":"Bioinformatics"},{"key":"2023013110135920300_btz678-B18","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1038\/nature13240","article-title":"Space\u2013time wiring specificity supports direction selectivity in the retina","volume":"509","author":"Kim","year":"2014","journal-title":"Nature"},{"key":"2023013110135920300_btz678-B19","doi-asserted-by":"crossref","first-page":"551","DOI":"10.1002\/fee.1436","article-title":"Assessing data quality in citizen science","volume":"14","author":"Kosmala","year":"2016","journal-title":"Front. Ecol. Environ"},{"key":"2023013110135920300_btz678-B20","doi-asserted-by":"crossref","first-page":"84.","DOI":"10.3847\/0004-637X\/830\/2\/84","article-title":"Disk detective: discovery of new circumstellar disk candidates through citizen science","volume":"830","author":"Kuchner","year":"2016","journal-title":"Astrophys. J"},{"key":"2023013110135920300_btz678-B21","doi-asserted-by":"crossref","first-page":"e1000450","DOI":"10.1371\/journal.pcbi.1000450","article-title":"Building disease-specific drug-protein connectivity maps from molecular interaction networks and PubMed abstracts","volume":"5","author":"Li","year":"2009","journal-title":"PLoS Comput. Biol"},{"key":"2023013110135920300_btz678-B22","doi-asserted-by":"crossref","first-page":"baw051.","DOI":"10.1093\/database\/baw051","article-title":"A crowdsourcing workflow for extracting chemical-induced disease relations from free text","volume":"2016","author":"Li","year":"2016","journal-title":"Database"},{"key":"2023013110135920300_btz678-B23","author":"Liu","year":"2016"},{"key":"2023013110135920300_btz678-B24","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1186\/s12911-018-0635-5","article-title":"OC-2-KB: integrating crowdsourcing into an obesity and cancer knowledge base curation system","volume":"18","author":"Lossio-Ventura","year":"2018","journal-title":"BMC Med. Inform. Decis. Mak"},{"key":"2023013110135920300_btz678-B25","doi-asserted-by":"crossref","first-page":"2363","DOI":"10.1093\/bioinformatics\/btx172","article-title":"A transition-based joint model for disease named entity recognition and normalization","volume":"33","author":"Lou","year":"2017","journal-title":"Bioinformatics"},{"key":"2023013110135920300_btz678-B26","doi-asserted-by":"crossref","first-page":"e167","DOI":"10.2196\/jmir.2338","article-title":"Crowdsourcing malaria parasite quantification: an online game for analyzing images of infected thick blood smears","volume":"14","author":"Luengo-Oroz","year":"2012","journal-title":"J. Med. Internet. Res"},{"key":"2023013110135920300_btz678-B27","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1016\/j.biocon.2016.05.015","article-title":"Citizen science can improve conservation science, natural resource management, and environmental protection","volume":"208","author":"McKinley","year":"2017","journal-title":"Biol. Conserv"},{"key":"2023013110135920300_btz678-B28","author":"Mintz","year":"2009"},{"key":"2023013110135920300_btz678-B29","doi-asserted-by":"crossref","first-page":"S3.","DOI":"10.1186\/gb-2008-9-s2-s3","article-title":"Overview of BioCreative II gene normalization","volume":"9 (Suppl. 2)","author":"Morgan","year":"2008","journal-title":"Genome Biol"},{"key":"2023013110135920300_btz678-B30","volume-title":"OSC","author":"Murray-Rust","year":"2017"},{"key":"2023013110135920300_btz678-B31","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1155\/2015\/910423","article-title":"A relation extraction framework for biomedical text using hybrid feature set","volume":"2015","author":"Muzaffar","year":"2015","journal-title":"Comput. Math. Methods Med"},{"key":"2023013110135920300_btz678-B32","doi-asserted-by":"crossref","first-page":"baw005.","DOI":"10.1093\/database\/baw005","article-title":"EXTRACT: interactive extraction of environment metadata and term suggestion for metagenomic sample annotation","volume":"2016","author":"Pafilis","year":"2016","journal-title":"Database"},{"key":"2023013110135920300_btz678-B33","doi-asserted-by":"crossref","first-page":"e0172587","DOI":"10.1371\/journal.pone.0172587","article-title":"A natural user interface to integrate citizen science and physical exercise","volume":"12","author":"Palermo","year":"2017","journal-title":"PLoS One"},{"key":"2023013110135920300_btz678-B34","doi-asserted-by":"crossref","first-page":"132","DOI":"10.1016\/j.jbi.2017.06.017","article-title":"Predicting biomedical metadata in CEDAR: a study of Gene Expression Omnibus (GEO)","volume":"72","author":"Panahiazar","year":"2017","journal-title":"J. Biomed. Inform"},{"key":"2023013110135920300_btz678-B35","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1093\/database\/bay073","article-title":"Extracting chemical\u2013protein relations with ensembles of SVM and deep learning models","volume":"2018","author":"Peng","year":"2018","journal-title":"Database"},{"key":"2023013110135920300_btz678-B36","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1016\/j.ymeth.2014.11.020","article-title":"DISEASES: text mining and data integration of disease\u2013gene associations","volume":"74","author":"Pletscher-Frankild","year":"2015","journal-title":"Methods"},{"key":"2023013110135920300_btz678-B37","doi-asserted-by":"crossref","first-page":"462","DOI":"10.1016\/j.jbi.2003.11.003","article-title":"The interaction of domain knowledge and linguistic structure in natural language processing: interpreting hypernymic propositions in biomedical text","volume":"36","author":"Rindflesch","year":"2003","journal-title":"J. Biomed. Inform"},{"key":"2023013110135920300_btz678-B38","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1007\/978-1-4939-3743-1_6","article-title":"Text mining to support gene ontology curation and vice versa","volume":"1446","author":"Ruch","year":"2016","journal-title":"Methods Mol. Biol"},{"key":"2023013110135920300_btz678-B39","doi-asserted-by":"crossref","first-page":"679","DOI":"10.1073\/pnas.1408907112","article-title":"Crowd science user contribution patterns and their implications","volume":"112","author":"Sauermann","year":"2015","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2023013110135920300_btz678-B40","doi-asserted-by":"crossref","first-page":"506","DOI":"10.1111\/cobi.12661","article-title":"Contributions of paraecologists and parataxonomists to research, conservation, and social development","volume":"30","author":"Schmiedel","year":"2016","journal-title":"Conserv. Biol"},{"key":"2023013110135920300_btz678-B41","article-title":"Giving citizen scientists a chance: a study of volunteer-led scientific discovery","volume":"1","author":"Straub","year":"2016","journal-title":"Citiz. Sci"},{"key":"2023013110135920300_btz678-B42","doi-asserted-by":"crossref","first-page":"1740005.","DOI":"10.1142\/S0219720017400054","article-title":"MPTM: a tool for mining protein post-translational modifications from literature","volume":"15","author":"Sun","year":"2017","journal-title":"J. Bioinform. Comput. Biol"},{"key":"2023013110135920300_btz678-B43","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1353\/pbm.1986.0087","article-title":"Fish oil, Raynaud's syndrome, and undiscovered public knowledge","volume":"30","author":"Swanson","year":"1986","journal-title":"Perspect. Biol. Med"},{"key":"2023013110135920300_btz678-B44","doi-asserted-by":"crossref","DOI":"10.1186\/s12859-015-0871-y","article-title":"NOBLE\u2014flexible concept recognition for large-scale biomedical natural language processing","volume":"17","author":"Tseytlin","year":"2016","journal-title":"BMC Bioinform"},{"key":"2023013110135920300_btz678-B45","first-page":"14.","article-title":"Citizen science for mining the biomedical literature","volume":"1","author":"Tsueng","year":"2016","journal-title":"Citiz. Sci"},{"key":"2023013110135920300_btz678-B46","doi-asserted-by":"crossref","first-page":"S1","DOI":"10.1186\/1471-2164-11-S5-S1","article-title":"Tumor and reproductive traits are linked by RNA metabolism genes in the mouse ovary: a transcriptome-phenotype association analysis","volume":"11 (Suppl. 5)","author":"Urz\u00faa","year":"2010","journal-title":"BMC Genomics"},{"key":"2023013110135920300_btz678-B47","volume-title":"52nd Proc. Conf. Assoc. Comput. Linguist. Meet","author":"Wang","year":"2014"},{"key":"2023013110135920300_btz678-B48","doi-asserted-by":"crossref","first-page":"W518","DOI":"10.1093\/nar\/gkt441","article-title":"PubTator: a web-based text mining tool for assisting biocuration","volume":"41","author":"Wei","year":"2013","journal-title":"Nucleic Acids Res"},{"key":"2023013110135920300_btz678-B49","first-page":"1","article-title":"GNormPlus: an integrative approach for tagging genes, gene families, and protein domains","volume":"2015","author":"Wei","year":"2015","journal-title":"Biomed. Res. Int"},{"key":"2023013110135920300_btz678-B50","doi-asserted-by":"crossref","DOI":"10.1093\/database\/baw032","article-title":"Assessing the state of the art in biomedical relation extraction: overview of the BioCreative V chemical-disease relation (CDR) task","volume":"2016","author":"Wei","year":"2016","journal-title":"Database"},{"key":"2023013110135920300_btz678-B51","author":"Williams","year":"2014"},{"key":"2023013110135920300_btz678-B52","doi-asserted-by":"crossref","first-page":"i386","DOI":"10.1093\/bioinformatics\/bty263","article-title":"A gene\u2013phenotype relationship extraction pipeline from the biomedical literature using a representation learning approach","volume":"34","author":"Xing","year":"2018","journal-title":"Bioinformatics"},{"key":"2023013110135920300_btz678-B53","first-page":"488","article-title":"Literature-based discovery of new candidates for drug repurposing","volume":"18","author":"Yang","year":"2016","journal-title":"Brief. Bioinform"},{"key":"2023013110135920300_btz678-B54","first-page":"103\u2013111.","article-title":"Exploiting literature-derived knowledge and semantics to identify potential prostate cancer drugs","volume":"13 (Suppl. 1)","author":"Zhang","year":"2014","journal-title":"Cancer Inform"},{"key":"2023013110135920300_btz678-B55","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1016\/j.jbi.2014.01.004","article-title":"Using semantic predications to uncover drug\u2013drug interactions in clinical data","volume":"49","author":"Zhang","year":"2014","journal-title":"J. Biomed. Inform"},{"key":"2023013110135920300_btz678-B56","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1016\/j.jbi.2018.07.007","article-title":"Chemical-induced disease relation extraction with dependency information and prior knowledge","volume":"84","author":"Zhou","year":"2018","journal-title":"J. Biomed. Inform"},{"key":"2023013110135920300_btz678-B57","doi-asserted-by":"crossref","first-page":"200","DOI":"10.1016\/j.jbi.2012.10.007","article-title":"Biomedical text mining and its applications in cancer research","volume":"46","author":"Zhu","year":"2013","journal-title":"J. Biomed. Inform"},{"key":"2023013110135920300_btz678-B58","doi-asserted-by":"crossref","first-page":"1547","DOI":"10.1093\/bioinformatics\/btx815","article-title":"GRAM-CNN: a deep learning approach with local context for named entity recognition in biomedical text","volume":"34","author":"Zhu","year":"2018","journal-title":"Bioinformatics"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btz678\/29522887\/btz678.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/36\/4\/1226\/48982766\/bioinformatics_36_4_1226.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/36\/4\/1226\/48982766\/bioinformatics_36_4_1226.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,31]],"date-time":"2023-01-31T15:14:09Z","timestamp":1675178049000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/36\/4\/1226\/5559491"}},"subtitle":[],"editor":[{"given":"Jonathan","family":"Wren","sequence":"additional","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2019,9,3]]},"references-count":59,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,2,15]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btz678","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/564187","asserted-by":"object"}]},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,2,15]]},"published":{"date-parts":[[2019,9,3]]}}}