{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,22]],"date-time":"2026-01-22T13:14:36Z","timestamp":1769087676877,"version":"3.49.0"},"reference-count":45,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,5,13]],"date-time":"2024-05-13T00:00:00Z","timestamp":1715558400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neuroinform."],"abstract":"<jats:sec><jats:title>Introduction<\/jats:title><jats:p>Polysomnographic recordings are essential for diagnosing many sleep disorders, yet their detailed analysis presents considerable challenges. With the rise of machine learning methodologies, researchers have created various algorithms to automatically score and extract clinically relevant features from polysomnography, but less research has been devoted to how exactly the algorithms should be incorporated into the workflow of sleep technologists. This paper presents a sophisticated data collection platform developed under the Sleep Revolution project, to harness polysomnographic data from multiple European centers.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>A tripartite platform is presented: a user-friendly web platform for uploading three-night polysomnographic recordings, a dedicated splitter that segments these into individual one-night recordings, and an advanced processor that enhances the one-night polysomnography with contemporary automatic scoring algorithms. The platform is evaluated using real-life data and human scorers, whereby scoring time, accuracy, and trust are quantified. Additionally, the scorers were interviewed about their trust in the platform, along with the impact of its integration into their workflow.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>We found that incorporating AI into the workflow of sleep technologists both decreased the time to score by up to 65 min and increased the agreement between technologists by as much as 0.17 <jats:italic>\u03ba<\/jats:italic>.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>We conclude that while the inclusion of AI into the workflow of sleep technologists can have a positive impact in terms of speed and agreement, there is a need for trust in the algorithms.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fninf.2024.1379932","type":"journal-article","created":{"date-parts":[[2024,5,13]],"date-time":"2024-05-13T04:55:53Z","timestamp":1715576153000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["An optimized framework for processing multicentric polysomnographic data incorporating expert human oversight"],"prefix":"10.3389","volume":"18","author":[{"given":"Benedikt","family":"Holm","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gabriel","family":"Jouan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Emil","family":"Hardarson","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sigr\u00ed\u00f0ur","family":"Sigur\u00f0ardottir","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kenan","family":"Hoelke","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Conor","family":"Murphy","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Erna Sif","family":"Arnard\u00f3ttir","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mar\u00eda","family":"\u00d3skarsd\u00f3ttir","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Anna Sigr\u00ed\u00f0ur","family":"Islind","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2024,5,13]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"e0275530","DOI":"10.1371\/journal.pone.0275530","article-title":"Computer-assisted analysis of polysomnographic recordings improves inter-scorer associated agreement and scoring times","volume":"17","author":"Alvarez-Estevez","year":"2022","journal-title":"PLoS ONE"},{"key":"B2","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1016\/j.jsmc.2021.05.004","article-title":"The future of sleep measurements: a review and perspective","volume":"16","author":"Arnardottir","year":"2021","journal-title":"Sleep Med. Clin"},{"key":"B3","doi-asserted-by":"publisher","first-page":"e13630","DOI":"10.1111\/jsr.13630","article-title":"The sleep revolution project: the concept and objectives","volume":"31","author":"Arnardottir","year":"2022","journal-title":"J. Sleep Res"},{"key":"B4","doi-asserted-by":"publisher","first-page":"zsac154","DOI":"10.1093\/sleep\/zsac154","article-title":"Scoring sleep with artificial intelligence enables quantification of sleep stage ambiguity: hypnodensity based on multiple expert scorers and auto-scoring","volume":"46","author":"Bakker","year":"2023","journal-title":"Sleep"},{"key":"B5","doi-asserted-by":"publisher","first-page":"687","DOI":"10.1016\/S2213-2600(19)30198-5","article-title":"Estimation of the global prevalence and burden of obstructive sleep apnoea: a literature-based analysis","volume":"7","author":"Benjafield","year":"2019","journal-title":"Lancet Respir. Med"},{"key":"B6","doi-asserted-by":"publisher","first-page":"976","DOI":"10.1007\/s10618-023-00985-x","article-title":"Anomaly detection in sleep: detecting mouth breathing in children","volume":"38","author":"Biedebach","year":"2024","journal-title":"Data Min. Knowl. Disc"},{"key":"B7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s40463-022-00566-w","article-title":"Barriers of artificial intelligence implementation in the diagnosis of obstructive sleep apnea","volume":"51","author":"Brennan","year":"2022","journal-title":"J. Otolaryngol.-Head Neck Surg"},{"key":"B8","doi-asserted-by":"publisher","first-page":"1123935","DOI":"10.3389\/fneur.2023.1123935","article-title":"Benchmarking performance of an automatic polysomnography scoring system in a population with suspected sleep disorders","volume":"14","author":"Choo","year":"2023","journal-title":"Front. Neurol"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2023.09.026","article-title":"Explainable AI for operational research: a defining framework, methods, applications, and a research agenda","author":"De Bock","year":"2023","journal-title":"Eur. J. Oper. Res"},{"key":"B10","doi-asserted-by":"publisher","first-page":"346","DOI":"10.1097\/YCO.0b013e3283473375","article-title":"Medical comorbidity of sleep disorders","volume":"24","author":"Dikeos","year":"2011","journal-title":"Curr. Opin. Psychiatry"},{"key":"B11","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1037\/h0031619","article-title":"Measuring nominal scale agreement among many raters","volume":"76","author":"Fleiss","year":"1971","journal-title":"Psychol. Bull"},{"key":"B12","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1016\/j.jneumeth.2019.01.013","article-title":"Iterative expert-in-the-loop classification of sleep PSG recordings using a hierarchical clustering","volume":"317","author":"Gerla","year":"2019","journal-title":"J. Neurosci. Methods"},{"key":"B13","doi-asserted-by":"crossref","first-page":"2590","DOI":"10.1109\/BIBM.2018.8621557","article-title":"\u201cExpert-in-the-loop learning for sleep EEG data,\u201d","volume-title":"2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)","author":"Gerla","year":"2018"},{"key":"B14","doi-asserted-by":"publisher","first-page":"111031","DOI":"10.1016\/j.jss.2021.111031","article-title":"A software engineering perspective on engineering machine learning systems: state of the art and challenges","volume":"180","author":"Giray","year":"2021","journal-title":"J. Syst. Softw"},{"key":"B15","doi-asserted-by":"publisher","first-page":"609","DOI":"10.5664\/jcsm.8388","article-title":"Artificial intelligence in sleep medicine: background and implications for clinicians","volume":"16","author":"Goldstein","year":"2020","journal-title":"J. Clin. Sleep Med"},{"key":"B16","doi-asserted-by":"publisher","first-page":"101614","DOI":"10.1016\/j.jsis.2020.101614","article-title":"Augmenting the algorithm: emerging human-in-the-loop work configurations","volume":"29","author":"Gr\u00f8nsund","year":"2020","journal-title":"J. Strateg. Inf. Syst"},{"key":"B17","unstructured":"HolmB.\n          BreathFinder2020"},{"key":"B18","doi-asserted-by":"publisher","first-page":"1704","DOI":"10.1109\/TBME.2022.3225268","article-title":"A comparison of signal combinations for deep learning-based simultaneous sleep staging and respiratory event detection","volume":"70","author":"Huttunen","year":"2022","journal-title":"IEEE Trans. Biomed. Eng"},{"key":"B19","unstructured":"\u201cBalancing overreliance and mistrust in data-driven decision making: a critical view on the role of quantified self in diabetes management,\u201d\n            IslindA. S.\n            HultH. V.\n          8th International Workshop on Socio-Technical Perspective in IS Development (STPIS 2022)2022"},{"key":"B20","doi-asserted-by":"publisher","first-page":"e2123427119","DOI":"10.1073\/pnas.2123427119","article-title":"Recurrent hippocampo-neocortical sleep-state divergence in humans","volume":"119","author":"Jang","year":"2022","journal-title":"Proc. Natl. Acad. Sci"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.12688\/wellcomeopenres.17550.1","article-title":"Influences on user trust in healthcare artificial intelligence: a systematic review","author":"Jermutus","year":"2022","journal-title":"Wellcome Open Res"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2023.09.039","article-title":"An algorithmic approach to identification of gray areas: analysis of sleep scoring expert ensemble non agreement areas using a multinomial mixture model","author":"Jouan","year":"2023","journal-title":"Eur. J. Oper. Res"},{"key":"B23","doi-asserted-by":"publisher","first-page":"1264","DOI":"10.1016\/j.ophtha.2018.01.034","article-title":"Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy","volume":"125","author":"Krause","year":"2018","journal-title":"Ophthalmology"},{"key":"B24","doi-asserted-by":"publisher","first-page":"271","DOI":"10.3390\/ijerph18010271","article-title":"Application of artificial intelligence-based technologies in the healthcare industry: opportunities and challenges","volume":"18","author":"Lee","year":"2021","journal-title":"Int. J. Environ. Res. Public Health"},{"key":"B25","doi-asserted-by":"publisher","first-page":"e0218948","DOI":"10.1371\/journal.pone.0218948","article-title":"Development of a human-computer collaborative sleep scoring system for polysomnography recordings","volume":"14","author":"Liang","year":"2019","journal-title":"PLoS ONE"},{"key":"B26","doi-asserted-by":"publisher","first-page":"107161","DOI":"10.1016\/j.cmpb.2022.107161","article-title":"Application of explainable artificial intelligence for healthcare: a systematic review of the last decade (2011-2022)","volume":"226","author":"Loh","year":"2022","journal-title":"Comput. Methods Programs Biomed"},{"key":"B27","doi-asserted-by":"publisher","first-page":"2","DOI":"10.5555\/2600239.2600241","article-title":"Docker: lightweight linux containers for consistent development and deployment","volume":"2014","author":"Merkel","year":"2014","journal-title":"Linux J"},{"key":"B28","doi-asserted-by":"publisher","first-page":"3005","DOI":"10.1007\/s10462-022-10246-w","article-title":"Human-in-the-loop machine learning: a state of the art","volume":"56","author":"Mosqueira-Rey","year":"2023","journal-title":"Artif. Intell. Rev"},{"key":"B29","doi-asserted-by":"publisher","first-page":"e13956","DOI":"10.1111\/jsr.13956","article-title":"Multicentre sleep-stage scoring agreement in the sleep revolution project","volume":"33","author":"Nikkonen","year":"2024","journal-title":"J. Sleep Res"},{"key":"B30","doi-asserted-by":"publisher","first-page":"e17971","DOI":"10.2196\/17971","article-title":"Attitudes of patients and health professionals regarding screening algorithms: qualitative study","volume":"5","author":"Oxholm","year":"2021","journal-title":"JMIR Form Res"},{"key":"B31","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1038\/s41746-021-00440-5","article-title":"U-sleep: resilient high-frequency sleep staging","volume":"4","author":"Perslev","year":"2021","journal-title":"NPJ Digit. Med"},{"key":"B32","doi-asserted-by":"publisher","first-page":"zsae011","DOI":"10.1093\/sleep\/zsae011","article-title":"Looking for clues in the hypnogram - the human eye and the machine","volume":"47","author":"Pevernagie","year":"2024","journal-title":"Sleep"},{"key":"B33","doi-asserted-by":"publisher","first-page":"4748","DOI":"10.1109\/JBHI.2023.3303197","article-title":"L-seqsleepnet: whole-cycle long sequence modelling for automatic sleep staging","volume":"27","author":"Phan","year":"2023","journal-title":"IEEE J Biomed Health Inform"},{"key":"B34","first-page":"114","article-title":"\u201cDesigning for scalability and trustworthiness in mhealth systems,\u201d","volume-title":"Distributed Computing and Internet Technology. ICDCIT 2015. Lecture Notes in Computer Science, Vol. 8956","author":"Prasad","year":"2015"},{"key":"B35","unstructured":"35960154RabbitMQ Documentation2007"},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.1093\/sleep\/zsad275","article-title":"The pros and cons of using automated sleep scoring in sleep research: comparative analysis of automated sleep scoring in human and rodents: advantages and limitations","author":"Rayan","year":"2023","journal-title":"Sleep 47"},{"key":"B37","doi-asserted-by":"publisher","first-page":"169","DOI":"10.5664\/jcsm.26818","article-title":"The scoring of respiratory events in sleep: reliability and validity","volume":"3","author":"Redline","year":"2007","journal-title":"J. Clin. Sleep Med"},{"key":"B38","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3472291","article-title":"A survey of deep active learning","volume":"54","author":"Ren","year":"2021","journal-title":"ACM Comput. Surveys"},{"key":"B39","doi-asserted-by":"publisher","first-page":"81","DOI":"10.5664\/jcsm.2350","article-title":"The American academy of sleep medicine inter-scorer reliability program: sleep stage scoring","volume":"9","author":"Rosenberg","year":"2013","journal-title":"J. Clin. Sleep Med"},{"key":"B40","unstructured":"RusanenM.\n            JouanG.\n            HuttunenR.\n            NikkonenS.\n            Sigur\u00f0ard\u00f3ttirS.\n            T\u00f6yr\u00f0sJ.\n          asaga: Automatic sleep analysis with gray areas2023"},{"key":"B41","unstructured":"SettlesB.\n          Active learning literature survey2009"},{"key":"B42","unstructured":"Revolution of sleep diagnostics and personalized health care based on digital diagnostics and therapeutics with health data integration2021"},{"key":"B43","unstructured":"Fastapi2023"},{"key":"B44","volume-title":"The AASM Manual for the Scoring of Sleep and Associated Events: Rules, Terminology and Technical Specifications, volume Version 3.0","author":"Troester","year":"2023"},{"key":"B45","doi-asserted-by":"publisher","first-page":"1971","DOI":"10.5664\/jcsm.9606","article-title":"There is no question about it, sleep disorders increase health care costs","volume":"17","author":"Wickwire","year":"2021","journal-title":"J. Clin. Sleep Med"}],"container-title":["Frontiers in Neuroinformatics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fninf.2024.1379932\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,13]],"date-time":"2024-05-13T04:55:58Z","timestamp":1715576158000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fninf.2024.1379932\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,13]]},"references-count":45,"alternative-id":["10.3389\/fninf.2024.1379932"],"URL":"https:\/\/doi.org\/10.3389\/fninf.2024.1379932","relation":{},"ISSN":["1662-5196"],"issn-type":[{"value":"1662-5196","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,13]]},"article-number":"1379932"}}