{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2023,10,24]],"date-time":"2023-10-24T11:00:32Z","timestamp":1698145232544},"reference-count":0,"publisher":"IOS Press","license":[{"start":{"date-parts":[[2021,5,27]],"date-time":"2021-05-27T00:00:00Z","timestamp":1622073600000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,5,27]]},"abstract":"<jats:p>Unstructured medical text labeling technologies are expected to be highly demanded since the interest in artificial intelligence and natural language processing arises in the medical domain. Our study aimed to assess the agreement between experts who judged on the fact of pulmonary embolism (PE) in neurosurgical cases retrospectively based on electronic health records and assess the utility of the machine learning approach to automate this process. We observed a moderate agreement between 3 independent raters on PE detection (Light\u2019s kappa = 0.568, p = 0). Labeling sentences with the method we proposed earlier might improve the machine learning results (accuracy = 0.97, ROC AUC = 0.98) even in those cases that could not be agreed between 3 independent raters. Medical text labeling techniques might be more efficient when strict rules and semi-automated approaches are implemented. Machine learning might be a good option for unstructured text labeling when the reliability of textual data is properly addressed. This project was supported by the RFBR grant 18-29-22085.<\/jats:p>","DOI":"10.3233\/shti210132","type":"book-chapter","created":{"date-parts":[[2021,5,27]],"date-time":"2021-05-27T12:39:33Z","timestamp":1622119173000},"source":"Crossref","is-referenced-by-count":1,"title":["Inter-Rater Reliability of Unstructured Text Labeling: Artificially vs. Naturally Intelligent Approaches"],"prefix":"10.3233","author":[{"given":"Gleb","family":"Danilov","sequence":"first","affiliation":[{"name":"Laboratory of Biomedical Informatics and Artificial Intelligence, National Medical Research Center for Neurosurgery named after N.N. Burdenko, Moscow, Russian Federation"}]},{"given":"Alexandra","family":"Kosyrkova","sequence":"additional","affiliation":[{"name":"Laboratory of Biomedical Informatics and Artificial Intelligence, National Medical Research Center for Neurosurgery named after N.N. Burdenko, Moscow, Russian Federation"}]},{"given":"Maria","family":"Shults","sequence":"additional","affiliation":[{"name":"Laboratory of Biomedical Informatics and Artificial Intelligence, National Medical Research Center for Neurosurgery named after N.N. Burdenko, Moscow, Russian Federation"}]},{"given":"Semen","family":"Melchenko","sequence":"additional","affiliation":[{"name":"Laboratory of Biomedical Informatics and Artificial Intelligence, National Medical Research Center for Neurosurgery named after N.N. Burdenko, Moscow, Russian Federation"}]},{"given":"Tatyana","family":"Tsukanova","sequence":"additional","affiliation":[{"name":"Laboratory of Biomedical Informatics and Artificial Intelligence, National Medical Research Center for Neurosurgery named after N.N. Burdenko, Moscow, Russian Federation"}]},{"given":"Michael","family":"Shifrin","sequence":"additional","affiliation":[{"name":"Laboratory of Biomedical Informatics and Artificial Intelligence, National Medical Research Center for Neurosurgery named after N.N. Burdenko, Moscow, Russian Federation"}]},{"given":"Alexander","family":"Potapov","sequence":"additional","affiliation":[{"name":"Laboratory of Biomedical Informatics and Artificial Intelligence, National Medical Research Center for Neurosurgery named after N.N. Burdenko, Moscow, Russian Federation"}]}],"member":"7437","container-title":["Studies in Health Technology and Informatics","Public Health and Informatics"],"original-title":[],"link":[{"URL":"https:\/\/ebooks.iospress.nl\/pdf\/doi\/10.3233\/SHTI210132","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,10,25]],"date-time":"2021-10-25T13:10:51Z","timestamp":1635167451000},"score":1,"resource":{"primary":{"URL":"https:\/\/ebooks.iospress.nl\/doi\/10.3233\/SHTI210132"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,27]]},"references-count":0,"URL":"https:\/\/doi.org\/10.3233\/shti210132","relation":{},"ISSN":["0926-9630","1879-8365"],"issn-type":[{"value":"0926-9630","type":"print"},{"value":"1879-8365","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,27]]}}}