{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T05:10:03Z","timestamp":1772169003752,"version":"3.50.1"},"reference-count":24,"publisher":"F1000 Research Ltd","license":[{"start":{"date-parts":[[2020,7,30]],"date-time":"2020-07-30T00:00:00Z","timestamp":1596067200000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["UH3TR000506-05"],"award-info":[{"award-number":["UH3TR000506-05"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000057","name":"National Institute of General Medical Sciences","doi-asserted-by":"publisher","award":["R01GM097618-05"],"award-info":[{"award-number":["R01GM097618-05"]}],"id":[{"id":"10.13039\/100000057","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Marv Conney"}],"content-domain":{"domain":["f1000research.com"],"crossmark-restriction":false},"short-container-title":["F1000Res"],"abstract":"<ns3:p>\n                    Many important scientific discoveries require lengthy experimental processes of trial and error and could benefit from intelligent prioritization based on deep domain understanding. While exponential growth in the scientific literature makes it difficult to keep current in even a single domain, that same rapid growth in literature also presents an opportunity for automated extraction of knowledge via text mining. We have developed a web application implementation of the KinderMiner algorithm for proposing ranked associations between a list of target terms and a key phrase. Any key phrase and target term list can be used for biomedical inquiry. We built the web application around a text index derived from PubMed. It is the first publicly available implementation of the algorithm, is fast and easy to use, and includes an interactive analysis tool. The KinderMiner web application is a public resource offering scientists a cohesive summary of what is currently known about a particular topic within the literature, and helping them to prioritize experiments around that topic. It performs comparably or better to similar state-of-the-art text mining tools, is more flexible, and can be applied to any biomedical topic of interest. It is also continually improving with quarterly updates to the underlying text index and through response to suggestions from the community. The web application is available at\n                    <ns3:ext-link xmlns:ns4=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" ns4:href=\"https:\/\/www.kinderminer.org\">https:\/\/www.kinderminer.org<\/ns3:ext-link>\n                    .\n                  <\/ns3:p>","DOI":"10.12688\/f1000research.25523.1","type":"journal-article","created":{"date-parts":[[2020,7,30]],"date-time":"2020-07-30T10:55:46Z","timestamp":1596106546000},"page":"832","update-policy":"https:\/\/doi.org\/10.12688\/f1000research.crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["KinderMiner Web: a simple web tool for ranking pairwise associations in biomedical applications"],"prefix":"10.12688","volume":"9","author":[{"given":"Finn","family":"Kuusisto","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daniel","family":"Ng","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"John","family":"Steill","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ian","family":"Ross","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Miron","family":"Livny","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"James","family":"Thomson","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David","family":"Page","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ron","family":"Stewart","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"2560","published-online":{"date-parts":[[2020,7,30]]},"reference":[{"key":"ref-1","doi-asserted-by":"publisher","first-page":"3234-3247","DOI":"10.3390\/su4123234","article-title":"Publication growth in biological sub-\ufb01elds: patterns, predictability and sustainability.","volume":"4","author":"M Pautasso","year":"2012","journal-title":"Sustainability."},{"key":"ref-2","doi-asserted-by":"publisher","first-page":"2215-2222","DOI":"10.1002\/asi.23329","article-title":"Growth rates of modern science: a bibliometric analysis based on the number of publications and cited references.","volume":"66","author":"L Bornmann","year":"2015","journal-title":"J Assoc Inf Sci Technol."},{"key":"ref-3","first-page":"166-174","article-title":"A simple text mining approach for ranking pairwise associations in biomedical applications.","author":"F Kuusisto","year":"2017","journal-title":"AMIA Jt Summits Transl Sci Proc."},{"key":"ref-4","article-title":"Medline\/pubmed citation records","year":"2019"},{"key":"ref-5","doi-asserted-by":"publisher","first-page":"D1042-D1048","DOI":"10.1093\/nar\/gku1061","article-title":"Europe pmc: a full- life sciences and platform for innovation.","volume":"43","year":"2014","journal-title":"Nucleic Acids Res."},{"key":"ref-6","doi-asserted-by":"publisher","first-page":"663-676","DOI":"10.1016\/j.cell.2006.07.024","article-title":"Induction of pluripotent stem cells from mouse embryonic and adult \ufb01broblast cultures by de\ufb01ned factors.","volume":"126","author":"K Takahashi","year":"2006","journal-title":"Cell."},{"key":"ref-7","doi-asserted-by":"publisher","first-page":"1917-1920","DOI":"10.1126\/science.1151526","article-title":"Induced pluripotent stem cell lines derived from human somatic cells.","volume":"318","author":"J Yu","year":"2007","journal-title":"Science."},{"key":"ref-8","doi-asserted-by":"publisher","first-page":"861-872","DOI":"10.1016\/j.cell.2007.11.019","article-title":"Induction of pluripotent stem cells from adult human \ufb01broblasts by de\ufb01ned factors.","volume":"131","author":"K Takahashi","year":"2007","journal-title":"Cell."},{"key":"ref-9","doi-asserted-by":"publisher","first-page":"1269-1275","DOI":"10.1038\/nbt.1502","article-title":"Induction of pluripotent stem cells from primary human \ufb01broblasts with only oct4 and sox2.","volume":"26","author":"D Huangfu","year":"2008","journal-title":"Nat Biotechnol."},{"key":"ref-10","doi-asserted-by":"publisher","first-page":"375-386","DOI":"10.1016\/j.cell.2010.07.002","article-title":"Direct reprogramming of \ufb01broblasts into functional cardiomyocytes by de\ufb01ned factors.","volume":"142","author":"M Ieda","year":"2010","journal-title":"Cell."},{"key":"ref-11","doi-asserted-by":"publisher","first-page":"97-106","DOI":"10.1016\/j.yjmcc.2013.04.004","article-title":"Optimization of direct \ufb01broblast reprogramming to cardiomyocytes using calcium activity as a functional measure of success.","volume":"60","author":"R Addis","year":"2013","journal-title":"J Mol Cell Cardiol."},{"key":"ref-12","doi-asserted-by":"publisher","first-page":"386-389","DOI":"10.1038\/nature10116","article-title":"Induction of functional hepatocyte-like cells from mouse \ufb01broblasts by de\ufb01ned factors.","volume":"475","author":"P Huang","year":"2011","journal-title":"Nature."},{"key":"ref-13","doi-asserted-by":"publisher","first-page":"937-944","DOI":"10.1007\/s12072-013-9432-5","article-title":"Transdifferentiation of human \ufb01broblasts into hepatocyte-like cells by de\ufb01ned transcriptional factors.","volume":"7","author":"T Kogiso","year":"2013","journal-title":"Hepatol Int."},{"key":"ref-14","doi-asserted-by":"publisher","first-page":"i111-i119","DOI":"10.1093\/bioinformatics\/btr214","article-title":"Discovering and visualizing indirect associations between biomedical concepts.","volume":"27","author":"Y Tsuruoka","year":"2011","journal-title":"Bioinformatics."},{"key":"ref-15","doi-asserted-by":"publisher","first-page":"W535-W542","DOI":"10.1093\/nar\/gkv383","article-title":"PPolySearch2: a significantly improved text-mining system for discovering associations between human diseases, genes, drugs, metabolites, toxins and more.","volume":"43","author":"Y Liu","year":"2015","journal-title":"Nucleic Acids Res."},{"key":"ref-16","doi-asserted-by":"publisher","first-page":"e0164680","DOI":"10.1371\/journal.pone.0164680","article-title":"Best: next-generation biomedical entity search tool for knowledge discovery from biomedical literature.","volume":"11","author":"S Lee","year":"2016","journal-title":"PLoS One."},{"key":"ref-17","first-page":"19-24","article-title":"Deeplife: An entity-aware search, analytics and exploration platform for health and life sciences.","author":"P Ernst","year":"2016"},{"key":"ref-18","first-page":"55-60","article-title":"Life-inet: A structured network-based knowledge exploration and analytics system for life sciences.","author":"X Ren","year":"2017"},{"key":"ref-19","doi-asserted-by":"publisher","first-page":"eaaw7195","DOI":"10.1126\/sciadv.aaw7195","article-title":"Data-driven phenotype discovery of FMR1 premutation carriers in a population-based sample.","volume":"5","author":"A Movaghar","year":"2019","journal-title":"Sci Adv."},{"key":"ref-20","first-page":"572-581","article-title":"Machine learning assisted discovery of novel predictive lab tests using electronic health record data.","volume":"2019","author":"R Kleiman","year":"2019","journal-title":"AMIA Jt Summits Transl Sci Proc."},{"key":"ref-21","doi-asserted-by":"publisher","first-page":"13-34","DOI":"10.1007\/978-1-4939-9873-9_2","article-title":"Automated extraction and visualization of protein\u2013protein interaction networks and beyond: A text-mining protocol.","volume":"2074","author":"K Raja","year":"2020","journal-title":"Methods Mol Biol."},{"key":"ref-22","doi-asserted-by":"publisher","first-page":"264-271","DOI":"10.1093\/bioinformatics\/btz490","article-title":"Cocoscore: Context-aware co-occurrence scoring for text mining applications using distant supervision.","volume":"36","author":"A Junge","year":"2020","journal-title":"Bioinformatics."},{"key":"ref-23","article-title":"iross\/km_indexer: Creating a new release, now that zenodo is activated.","author":"I Ross","year":"2020"},{"key":"ref-24","article-title":"stewart-lab\/kinderminer_webapp: First release for publication.","author":"F Kuusisto","year":"2020"}],"updated-by":[{"DOI":"10.12688\/f1000research.25523.2","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2021,12,20]],"date-time":"2021-12-20T00:00:00Z","timestamp":1639958400000}}],"container-title":["F1000Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/f1000research.com\/articles\/9-832\/v1\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/f1000research.com\/articles\/9-832\/v1\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/f1000research.com\/articles\/9-832\/v1\/iparadigms","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,12,20]],"date-time":"2021-12-20T03:15:25Z","timestamp":1639970125000},"score":1,"resource":{"primary":{"URL":"https:\/\/f1000research.com\/articles\/9-832\/v1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,30]]},"references-count":24,"URL":"https:\/\/doi.org\/10.12688\/f1000research.25523.1","relation":{"has-review":[{"id-type":"doi","id":"10.5256\/f1000research.28167.r83813","asserted-by":"subject"},{"id-type":"doi","id":"10.5256\/f1000research.28167.r82000","asserted-by":"subject"},{"id-type":"doi","id":"10.5256\/f1000research.28167.r83813","asserted-by":"object"},{"id-type":"doi","id":"10.5256\/f1000research.28167.r82000","asserted-by":"object"}]},"ISSN":["2046-1402"],"issn-type":[{"value":"2046-1402","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,7,30]]},"assertion":[{"value":"Approved with reservations, Approved","URL":"https:\/\/f1000research.com\/articles\/9-832\/v1#article-reports","order":0,"name":"referee-status","label":"Referee status","group":{"name":"current-referee-status","label":"Current Referee Status"}},{"value":"10.5256\/f1000research.28167.r82000, Qingyu Chen, National Center for Biotechnology Information (NCBI), National Library of Medicine (NLM), National Institutes of Health (NIH), Bethesda, MD, USA, 19 Apr 2021, version 1, 1 approved, 1 approved with reservations","URL":"https:\/\/f1000research.com\/articles\/9-832\/v1#referee-response-82000","order":0,"name":"referee-response-82000","label":"Referee Report","group":{"name":"article-reports","label":"Article Reports"}},{"value":"<b>Finn Kuusisto<\/b>; \n<i>Posted: 13 Dec 2021<\/i>; \n<i>We greatly appreciate Dr. Chen's patience, time, and effort in providing useful feedback and corrections.<\/i>  &gt; A primary comment is the limitation should also specify the application is limited to the abstract level only.  \n<i>We did not emphasize the fact that our search is only at the abstract level in the limitations section. This was simply an oversight, and we have now added it to the limitations section.<\/i>  &gt; Please consider providing an API so that potential users can query the associations systematically.  \n<i>This is certainly a feature we have considered, and we intend to update the software with new backend data and features according to user demand. If it becomes a common request, we will absolutely implement it.<\/i>  &gt; In addition, does the indexing part incorporate synonyms?  \n<i>We do not perform any synonym matching as implemented but have explored it as an option and do ultimately intend to incorporate some version of it in the future. One major issue that we have found with synonym matching though is that, while it has the potential to increase recall, it can also substantially decrease precision. For example, one synonym of the gene name POU5F1 is OCT3, but OCT3 is also a synonym of SLC22A3, and this is not a particularly unique case even in the simple domain of gene names. Similar situations also frequently occur with acronyms, such as DR for Diabetic Retinopathy.  To elaborate further, in our exploration of synonym matching we ran the \u201cembryonic stem cell\u201d example with and without synonyms. While POU5F1, NANOG, and SOX2 show up in the top 7 hits without synonyms, the search with synonyms didn\u2019t even include SOX2 in the top 20 and NANOG was pushed to hit 16. This is due at least in part to several hits getting inflated counts due to synonym collisions with other words and each other. SPI1, TAF1, PAX6, and NR4A2 show up as hits number 3, 4, 5, and 7 respectively when we ran the search with synonyms. This is likely because both SPI1 and TAF1 have the synonym \u201cOF,\u201d PAX6 has a synonym \u201cAN,\u201d and NR4A2 has a synonym \u201cNOT.\u201d It appears then that synonym matching alone could very possibly lead to a substantial tradeoff in precision. We have clarified that KinderMiner does not perform synonym matching and increased emphasis of this point in the limitations section.  Given that this a software paper rather than a full research paper, we chose to share this tool with the simpler approach of exact string matching without synonyms because of its flexibility and because it has been demonstrated to work well for us under most circumstances.<\/i>  &gt; My primary comment is the precision and recall of the term matching part should be quantified. Given only a simple string matching method is used, it will potentially miss identifying the same entities using different expressions or wrongly identify the entities using exact terms but represent differently (for example, some genes share the same names with chemicals). Importantly, this is the first step of the algorithm; errors would propagate to the later stages.  \n<i>Dr. Chen and Dr. Orimaye have both commented on the limitations of using exact string matching. While we agree that exact string matching has potential limitations, we argue that exact string matching has advantages as well. Primarily, exact string matching allows for rapid exploration of the literature versus ontologies or semantic matching methods. Building ontologies or semantic models for any category of biomedical entity requires a great deal of time and effort, whereas exact matching allows for the quick creation of custom lists of entities of interest, which we found to be a common desire among our colleagues at the bench. Instead of looking for a tool that provides a predefined entity set that seems closest to their interests, they can simply create a list on the fly and run a search. Furthermore, as new terms arise (e.g., genes or drugs), the ontologies or semantic models need to be reworked\/retrained rather than simply adding a string to the end of a list. The trade-off is that some entities may require manual curation and a possible sacrifice in recall in some cases. Further, semantic matching approaches, or even simple synonym matching as we described with the embryonic transcription factor above, have the potential to introduce errors of their own when the simple exact match works as is. That said, we understand and appreciate the concern for our approach, have attempted to clarify this in the limitations section, and have elaborated on how we are considering more advanced matching for future versions.<\/i>  &gt; It is therefore critical to provide a detailed evaluation on this part.  \n<i>Dr. Chen has requested a detailed evaluation of our choice to use exact string matching versus others. Given that this paper is on a software implementation of a previously published algorithm, and because the flexibility afforded by allowing any list of entities makes an exhaustive evaluation of recall and precision an expansive research undertaking, we feel this request is out of scope for this paper. Even an evaluation of recall for our exact string matching on a single gene, such as HNF1A, would require a gold standard labeling of HNF1A-related abstracts for our entire index. In order to more generally evaluate how our exact matching performs on the entire human gene domain would then require nearly 20000 more distinct gold standard abstract labelings of our index. This in turn would be true of every other domain that our algorithm might encounter, including entity lists like proteins, drugs, diseases, cell types, species names, and so on. Because the output of the algorithm is an ordered subset of entities of interest, we have opted instead to present compelling results on several gold standard lists for important discoveries, rather than focusing on how the algorithm may miss individual entities of interest. Exact string matching provides great flexibility by allowing a user to search any list of entities, but this comes at the risk that some important entities may not surface due to say a particularly unpopular spelling. We think the flexibility is worth the tradeoff and that the promising results we have seen justify the decision.  We greatly appreciate the Dr. Chen's patience, time, and feedback. We intend for this software tool to give more researchers access to a simple algorithm that has helped us to prioritize some of our own research over the years. Overall, despite limitations, we find that the empirical results speak for themselves and hope that others will find it useful too.<\/i>","URL":"https:\/\/f1000research.com\/articles\/9-832\/v1#referee-comment-7568","order":1,"name":"referee-comment-7568","label":"Referee Comment","group":{"name":"article-reports","label":"Article Reports"}},{"value":"10.5256\/f1000research.28167.r83813, Sylvester O. Orimaye, College of Public Health, East Tennessee State University, Johnson City, TN, USA, 12 May 2021, version 1, 1 approved, 1 approved with reservations","URL":"https:\/\/f1000research.com\/articles\/9-832\/v1#referee-response-83813","order":2,"name":"referee-response-83813","label":"Referee Report","group":{"name":"article-reports","label":"Article Reports"}},{"value":"<b>Finn Kuusisto<\/b>; \n<i>Posted: 13 Dec 2021<\/i>; \n<i>We greatly appreciate the Dr. Orimaye's patience, time, and effort in providing useful feedback and corrections.<\/i>  &gt; The authors could also explain the rationale behind setting the Fisher's exact threshold to 0.00005.  \n<i>We used 0.00001 for the Fisher\u2019s Exact Test threshold. In this case we simply used the same search parameters as those that we used in the original KinderMiner algorithm publication. That said, we did not have a rigorous statistical justification for the threshold in the original paper. Instead, we chose the 1e-5 threshold, as it frequently resulted in final ranked lists in the range of 50 to 100 hits, which was a reasonable size when presenting results to collaborators working at the bench. We stuck with that choice for our own internal use of the algorithm and for comparison to the original here, but we provide the interactive slider in the webapp for flexibility and exploration of resultant list sizes. We have elaborated on this choice in the paper.<\/i>  &gt; It cannot be over-emphasized that semantically related concepts found in many ontologies have increasingly become an essential part of search engines.  \n<i>Dr. Chen and Dr. Orimaye have both commented on the limitations of using exact string matching. While we agree that exact string matching has potential limitations, we argue that exact string matching has advantages as well. Primarily, exact string matching allows for rapid exploration of the literature versus ontologies or semantic matching methods. Building ontologies or semantic models for any category of biomedical entity requires a great deal of time and effort, whereas exact matching allows for the quick creation of custom lists of entities of interest, which we found to be a common desire among our colleagues at the bench. Instead of looking for a tool that provides a predefined entity set that seems closest to their interests, they can simply create a list on the fly and run a search. Furthermore, as new terms arise (e.g., genes or drugs), the ontologies or semantic models need to be reworked\/retrained rather than simply adding a string to the end of a list. The trade-off is that some entities may require manual curation and a possible sacrifice in recall in some cases. Further, semantic matching approaches, or even simple synonym matching as we described with the embryonic transcription factor above, have the potential to introduce errors of their own when the simple exact match works as is. That said, we understand and appreciate the concern for our approach, have attempted to clarify this in the limitations section, and have elaborated on how we are considering more advanced matching for future versions.<\/i>  &gt; I found minor typos such as &quot; we need to validate...&quot; even though the authors described the experiment in the past. Please check for other typos in the final version of the manuscript.  \n<i>We have corrected that typo and have checked the manuscript for other typos.  We greatly appreciate Dr. Orimaye's patience, time, and feedback. We intend for this software tool to give more researchers access to a simple algorithm that has helped us to prioritize some of our own research over the years. Overall, despite limitations, we find that the empirical results speak for themselves and hope that others will find it useful too.<\/i>","URL":"https:\/\/f1000research.com\/articles\/9-832\/v1#referee-comment-7569","order":3,"name":"referee-comment-7569","label":"Referee Comment","group":{"name":"article-reports","label":"Article Reports"}},{"value":"This work was supported by the National Institutes of Health (NIH) grant number UH3TR000506-05 and the National Institute of General Medical Sciences (NIGMS) grant number R01GM097618-05.\nThe authors also thank Marv Conney for a grant to RS, JT, and FK. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.","order":4,"name":"grant-information","label":"Grant Information"},{"value":"This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.","order":0,"name":"copyright-info","label":"Copyright"}]}}