{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T06:23:48Z","timestamp":1773815028594,"version":"3.50.1"},"reference-count":26,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2021,2,10]],"date-time":"2021-02-10T00:00:00Z","timestamp":1612915200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Informatics"],"abstract":"<jats:p>Cochrane produces independent research to improve healthcare decisions. It translates its research summaries into different languages to enable wider access, relying largely on volunteers. Machine translation (MT) could facilitate efficiency in Cochrane\u2019s low-resource environment. We compared three off-the-shelf machine translation engines (MTEs)\u2014DeepL, Google Translate and Microsoft Translator\u2014for Russian translations of Cochrane plain language summaries (PLSs) by assessing the quantitative human post-editing effort within an established translation workflow and quality assurance process. 30 PLSs each were pre-translated with one of the three MTEs. Ten volunteer translators post-edited nine randomly assigned PLSs each\u2014three per MTE\u2014in their usual translation system, Memsource. Two editors performed a second editing step. Memsource\u2019s Machine Translation Quality Estimation (MTQE) feature provided an artificial intelligence (AI)-powered estimate of how much editing would be required for each PLS, and the analysis feature calculated the amount of human editing after each editing step. Google Translate performed the best with highest average quality estimates for its initial MT output, and the lowest amount of human post-editing. DeepL performed slightly worse, and Microsoft Translator worst. Future developments in MT research and the associated industry may change our results.<\/jats:p>","DOI":"10.3390\/informatics8010009","type":"journal-article","created":{"date-parts":[[2021,2,12]],"date-time":"2021-02-12T16:12:10Z","timestamp":1613146330000},"page":"9","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Assessing Human Post-Editing Efforts to Compare the Performance of Three Machine Translation Engines for English to Russian Translation of Cochrane Plain Language Health Information: Results of a Randomised Comparison"],"prefix":"10.3390","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1999-0705","authenticated-orcid":false,"given":"Liliya Eugenevna","family":"Ziganshina","sequence":"first","affiliation":[{"name":"Cochrane Russia, St Albans House, 57-59 Haymarket, London SW1Y 4QX, UK"},{"name":"Centre for Knowledge Translation, Institute for Methodology of Professional Development, Federal State Budgetary Educational Institution of Continuing Professional Education \u201cRussian Medical Academy of Continuing Professional Education\u201d of the Ministry of Health of the Russian Federation (RMANPO): 2\/1, Barrikadnaya Street, 123995 Moscow, Russia"},{"name":"Department of Pharmacology, Kazan State Medical University of the Ministry of Health of the Russian Federation (KSMU), 49 Butlerov Street, 420012 Kazan, Russia"},{"name":"Department of Basic and Clinical Pharmacology, Kazan Federal University of the Ministry of Science and Higher Education of the Russian Federation (KFU), 18 Kremlevskaya Street, 420008 Kazan, Russia"}]},{"given":"Ekaterina V.","family":"Yudina","sequence":"additional","affiliation":[{"name":"Cochrane Russia, St Albans House, 57-59 Haymarket, London SW1Y 4QX, UK"},{"name":"Children\u2019s Hospital N 1 of the City of Kazan, 125a Dekabristov Street, 420034 Kazan, Russia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0362-3700","authenticated-orcid":false,"given":"Azat I.","family":"Gabdrakhmanov","sequence":"additional","affiliation":[{"name":"Cochrane Russia, St Albans House, 57-59 Haymarket, London SW1Y 4QX, UK"},{"name":"Department of Pharmacology, Kazan State Medical University of the Ministry of Health of the Russian Federation (KSMU), 49 Butlerov Street, 420012 Kazan, Russia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6455-0701","authenticated-orcid":false,"given":"Juliane","family":"Ried","sequence":"additional","affiliation":[{"name":"Cochrane, 79110 Freiburg, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2021,2,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Useem, J., Brennan, A., LaValley, M., Vickery, M., Ameli, O., Reinen, N., and Gill, C.J. (2015). Systematic Differences between Cochrane and Non-Cochrane Meta-Analyses on the Same Topic: A Matched Pair Analysis. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0144980"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1016\/j.jdent.2018.05.003","article-title":"Heterogeneity in Cochrane and non-Cochrane meta-analyses in orthodontics","volume":"74","author":"Koletsi","year":"2018","journal-title":"J. Dent."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"von Elm, E., Ravaud, P., Maclehose, H., Mbuagbaw, L., Garner, P., Ried, J., and Bonfill, X. (2013). Translating Cochrane reviews to ensure that healthcare decision-making is informed by high-quality research evidence. PLoS Med., 10.","DOI":"10.1371\/journal.pmed.1001516"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"221","DOI":"10.3233\/JRS-170732","article-title":"The first ever Cochrane event in Russia and Russian speaking countries\u2014Cochrane Russia Launch\u2014Evidence-based medicine: Achievements and barriers (QiQUM 2015) International Conference, Kazan, 7\u20138 December 2015","volume":"28","author":"Ziganshina","year":"2016","journal-title":"Int. J. Risk Saf. Med."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1038\/scientificamerican0306-92","article-title":"The elusive goal of machine translation","volume":"294","author":"Stix","year":"2006","journal-title":"Sci. Am."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2214","DOI":"10.1021\/acs.jcim.8b00534","article-title":"Is Machine Translation a Reliable Tool for Reading German Scientific Databases and Research Articles?","volume":"58","author":"Zulfiqar","year":"2018","journal-title":"J. Chem. Inf. Model."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1016\/j.compmedimag.2015.09.005","article-title":"Telemedicine as a special case of machine translation","volume":"46","author":"Marasek","year":"2015","journal-title":"Comput. Med. Imaging Graph."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1016\/j.jbi.2018.07.018","article-title":"Development of machine translation technology for assisting health communication: A systematic review","volume":"85","author":"Dew","year":"2018","journal-title":"J. Biomed. Inform."},{"key":"ref_9","first-page":"492","article-title":"PHAST: A Collaborative Machine Translation and Post-Editing Tool for Public Health","volume":"2015","author":"Dew","year":"2015","journal-title":"AMIA Annu. Symp. Proc."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1478","DOI":"10.1093\/jamia\/ocz110","article-title":"Neural machine translation of clinical texts between long distance languages","volume":"26","author":"Soto","year":"2019","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_11","first-page":"1290","article-title":"Statistical machine translation for biomedical text: Are we there yet?","volume":"2011","author":"Wu","year":"2011","journal-title":"AMIA Annu. Symp. Proc."},{"key":"ref_12","unstructured":"(2021, January 31). Health in My Language. Available online: http:\/\/www.himl.eu\/."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Martikainen, H. (2019). Post-Editing Neural MT in Medical LSP: Lexico-Grammatical Patterns and Distortion in the Communication of Specialized Knowledge. Informatics, 6.","DOI":"10.3390\/informatics6030026"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"247","DOI":"10.1177\/0009922818809494","article-title":"Dangers of Machine Translation: The Need for Professionally Translated Anticipatory Guidance Resources for Limited English Proficiency Caregivers","volume":"58","author":"Das","year":"2019","journal-title":"Clin. Pediatr."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1016\/j.jclinepi.2020.07.021","article-title":"Using an evidence-based research approach to place your results into context after the study is performed to ensure usefulness of the conclusion","volume":"129","author":"Lund","year":"2020","journal-title":"J. Clin. Epidemiol."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"158","DOI":"10.1016\/j.jclinepi.2020.07.019","article-title":"Using an evidence-based research approach before a new study is conducted to ensure value","volume":"129","author":"Lund","year":"2020","journal-title":"J. Clin. Epidemiol."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1905","DOI":"10.1016\/S0140-6736(20)30172-0","article-title":"Doing meaningful systematic reviews is no gravy train","volume":"395","author":"Lund","year":"2020","journal-title":"Lancet"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"i5440","DOI":"10.1136\/bmj.i5440","article-title":"Towards evidence based research","volume":"355","author":"Lund","year":"2016","journal-title":"BMJ"},{"key":"ref_19","unstructured":"Memsource (2021, January 31). Machine Translation Quality Estimation. Memsource Help Center. Available online: https:\/\/help.memsource.com\/hc\/en-us\/articles\/360012527380."},{"key":"ref_20","unstructured":"Memsource (2021, January 31). Analysis Overview. Memsource Help Center. Available online: https:\/\/help.memsource.com\/hc\/en-us\/articles\/360013675760."},{"key":"ref_21","unstructured":"Wikipedia (2021, January 31). \u201cDeepL Translator\u201d. Available online: https:\/\/en.wikipedia.org\/wiki\/DeepL_Translator."},{"key":"ref_22","unstructured":"Wikipedia (2021, January 31). Google Neural Machine Translation. Available online: https:\/\/en.wikipedia.org\/wiki\/Google_Neural_Machine_Translation."},{"key":"ref_23","unstructured":"Wikipedia (2021, January 31). Microsoft Translator. Available online: https:\/\/en.wikipedia.org\/wiki\/Microsoft_Translator."},{"key":"ref_24","unstructured":"TAUS (2021, January 31). Automated MT Evaluation Metrics. Available online: https:\/\/blog.taus.net\/knowledgehub\/automated-mt-evaluation-metrics."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Jelicic Kadic, A., Fidahic, M., Vujcic, M., Saric, F., Propadalo, I., Marelja, I., Dosenovic, S., and Puljak, L. (2016). Cochrane plain language summaries are highly heterogeneous with low adherence to the standards. BMC Med. Res. Methodol., 16.","DOI":"10.1186\/s12874-016-0162-y"},{"key":"ref_26","unstructured":"DeepL (2021, January 31). DeepL Pro\u2014Terms and Conditions. Available online: https:\/\/www.deepl.com\/en\/pro-license\/."}],"container-title":["Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-9709\/8\/1\/9\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:22:41Z","timestamp":1760160161000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-9709\/8\/1\/9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,10]]},"references-count":26,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,3]]}},"alternative-id":["informatics8010009"],"URL":"https:\/\/doi.org\/10.3390\/informatics8010009","relation":{},"ISSN":["2227-9709"],"issn-type":[{"value":"2227-9709","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,10]]}}}