{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,13]],"date-time":"2025-11-13T18:32:45Z","timestamp":1763058765566,"version":"3.41.0"},"reference-count":18,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2022,10,31]],"date-time":"2022-10-31T00:00:00Z","timestamp":1667174400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"NSF","award":["1915398"],"award-info":[{"award-number":["1915398"]}]},{"name":"NIH","award":["R18 HS026620"],"award-info":[{"award-number":["R18 HS026620"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Healthcare"],"published-print":{"date-parts":[[2022,10,31]]},"abstract":"<jats:p>Classification of clinical alarms is at the heart of prioritization, suppression, integration, postponement, and other methods of mitigating alarm fatigue. Since these methods directly affect clinical care, alarm classifiers, such as intelligent suppression systems, need to be evaluated in terms of their sensitivity and specificity, which is typically calculated on a labeled dataset of alarms. Unfortunately, the collection and particularly labeling of such datasets requires substantial effort and time, thus deterring hospitals from investigating mitigations of alarm fatigue. This article develops a lightweight method for evaluating alarm classifiers without perfect alarm labels. The method relies on probabilistic labels obtained from data programming\u2014a labeling paradigm based on combining noisy and cheap-to-obtain labeling heuristics. Based on these labels, the method produces confidence bounds for the sensitivity\/specificity values from a hypothetical evaluation with manual labeling. Our experiments on five alarm datasets collected at Children\u2019s Hospital of Philadelphia show that the proposed method provides accurate bounds on the classifier\u2019s sensitivity\/specificity, appropriately reflecting the uncertainty from noisy labeling and limited sample sizes.<\/jats:p>","DOI":"10.1145\/3549942","type":"journal-article","created":{"date-parts":[[2022,7,22]],"date-time":"2022-07-22T11:13:37Z","timestamp":1658488417000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Evaluating Alarm Classifiers with High-confidence Data Programming"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1465-983X","authenticated-orcid":false,"given":"Sydney","family":"Pugh","sequence":"first","affiliation":[{"name":"University of Pennsylvania, Philadelphia, PA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3546-414X","authenticated-orcid":false,"given":"Ivan","family":"Ruchkin","sequence":"additional","affiliation":[{"name":"University of Pennsylvania, Philadelphia, PA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2823-5883","authenticated-orcid":false,"given":"Christopher","family":"Bonafide","sequence":"additional","affiliation":[{"name":"Children\u2019s Hospital of Philadelphia, Philadelphia, PA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1647-4000","authenticated-orcid":false,"given":"Sara","family":"Demauro","sequence":"additional","affiliation":[{"name":"Children\u2019s Hospital of Philadelphia, Philadelphia, PA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5282-0658","authenticated-orcid":false,"given":"Oleg","family":"Sokolsky","sequence":"additional","affiliation":[{"name":"University of Pennsylvania, Philadelphia, PA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2672-1132","authenticated-orcid":false,"given":"Insup","family":"Lee","sequence":"additional","affiliation":[{"name":"University of Pennsylvania, Philadelphia, PA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8167-9163","authenticated-orcid":false,"given":"James","family":"Weimer","sequence":"additional","affiliation":[{"name":"University of Pennsylvania, Philadelphia, PA"}]}],"member":"320","published-online":{"date-parts":[[2022,11,3]]},"reference":[{"issue":"1","key":"e_1_3_3_2_2","first-page":"1","article-title":"Reduction of false alarms in the intensive care unit using an optimized machine learning based approach","volume":"2","author":"Au-Yeung Wan-Tai M.","year":"2019","unstructured":"Wan-Tai M. Au-Yeung, Ashish K. Sahani, Eric M. Isselbacher, and Antonis A. Armoundas. 2019. Reduction of false alarms in the intensive care unit using an optimized machine learning based approach. NPJ Digital Medicine 2, 1 (2019), 1\u20135.","journal-title":"NPJ Digital Medicine"},{"key":"e_1_3_3_3_2","doi-asserted-by":"crossref","DOI":"10.1155\/2015\/934362","article-title":"Confidence interval estimation of an ROC curve: An application of Generalized Half Normal and Weibull distributions","volume":"2015","author":"Balaswamy S.","year":"2015","unstructured":"S. Balaswamy and R. Vishnu Vardhan. 2015. Confidence interval estimation of an ROC curve: An application of Generalized Half Normal and Weibull distributions. Journal of Probability and Statistics 2015 (2015).","journal-title":"Journal of Probability and Statistics"},{"issue":"1","key":"e_1_3_3_4_2","doi-asserted-by":"crossref","first-page":"524","DOI":"10.1001\/jamapediatrics.2016.5123","article-title":"Video analysis of factors associated with response time to physiologic monitor alarms in a children\u2019s hospital","volume":"171","author":"Bonafide Christopher P.","year":"2017","unstructured":"Christopher P. Bonafide, A. Russell Localio, John H. Holmes, Vinay M. Nadkarni, Shannon Stemler, Matthew MacMurchy, Miriam Zander, Kathryn E. Roberts, Richard Lin, and Ron Keren. 2017. Video analysis of factors associated with response time to physiologic monitor alarms in a children\u2019s hospital. JAMIA Pediatrics 171, 1 (2017), 524\u2013531.","journal-title":"JAMIA Pediatrics"},{"key":"e_1_3_3_5_2","article-title":"Confidence intervals for the area under the ROC curve","volume":"17","author":"Cortes Corinna","year":"2004","unstructured":"Corinna Cortes and Mehryar Mohri. 2004. Confidence intervals for the area under the ROC curve. Advances in Neural Information Processing Systems 17 (2004).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0110274"},{"key":"e_1_3_3_7_2","first-page":"1","article-title":"Computer aided clinical trials for implantaule cardiac devices","volume":"2018","author":"Jang Kuk","year":"2018","unstructured":"Kuk Jang, James Weimer, Houssam Abbas, Zhihao Jiang, Jackson Liang, Sanjay Dixit, and Rahul Mangharam. 2018. Computer aided clinical trials for implantaule cardiac devices. Annual International Conference of the IEEE Engineering in Medicine and Biology Society (2018), 1\u20134.","journal-title":"Annual International Conference of the IEEE Engineering in Medicine and Biology Society"},{"issue":"4","key":"e_1_3_3_8_2","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1177\/1937586716673829","article-title":"Development and application of a clinical microsystem simulation methodology for human factors-based research of alarm fatigue","volume":"10","author":"Kobayashi Leo","year":"2017","unstructured":"Leo Kobayashi, John W. Gosbee, and Derek L. Merck. 2017. Development and application of a clinical microsystem simulation methodology for human factors-based research of alarm fatigue. HERD: Health Environments Research and Design Journal 10, 4 (2017), 91\u2013104.","journal-title":"HERD: Health Environments Research and Design Journal"},{"key":"e_1_3_3_9_2","first-page":"5829","article-title":"Suppression of intensive care unit false alarms based on the arterial blood pressure signal","volume":"5","author":"Lameski Petre","year":"2017","unstructured":"Petre Lameski, Eftim Zdravevski, Saso Koceski, Andrea Kulakov, and Vladimir Trajkovik. 2017. Suppression of intensive care unit false alarms based on the arterial blood pressure signal. IEEE Access 5, 4 (2017), 5829\u20135836.","journal-title":"IEEE Access"},{"issue":"1","key":"e_1_3_3_10_2","doi-asserted-by":"crossref","first-page":"25","DOI":"10.2345\/0899-8205-51.1.25","article-title":"Research: Acceptability, feasibility, and cost of using video to evaluate alarm fatigue","volume":"51","author":"MacMurchy Matt","year":"2017","unstructured":"Matt MacMurchy, Shannon Stemler, Mimi Zander, and Christopher P. Bonafide. 2017. Research: Acceptability, feasibility, and cost of using video to evaluate alarm fatigue. Biomedical Instrumentation and Technology 51, 1 (2017), 25\u201333.","journal-title":"Biomedical Instrumentation and Technology"},{"key":"e_1_3_3_11_2","article-title":"Reducing pulse oximetry false alarms without missing life-threatening events","volume":"9","author":"Nguyen Hung","year":"2018","unstructured":"Hung Nguyen, Sooyong Jang, Radoslav Ivanov, Christopher Bonafide, James Weimer, and Insup Lee. 2018. Reducing pulse oximetry false alarms without missing life-threatening events. Smart Health 9\u201310 (2018), 287\u2013296.","journal-title":"Smart Health"},{"issue":"2","key":"e_1_3_3_12_2","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1002\/jhm.2520","article-title":"Systematic review of physiologic monitor alarm characteristics and pragmatic interventions to reduce alarm frequency","volume":"11","author":"Paine Christine Weirich","year":"2016","unstructured":"Christine Weirich Paine, Veena V. Goel, Elizabeth Ely, Christopher D. Stave, Shannon Stemler, Miriam Zander, and Christopher P. Bonafide. 2016. Systematic review of physiologic monitor alarm characteristics and pragmatic interventions to reduce alarm frequency. Journal of Hospital Medicine 11, 2 (2016), 136\u2013144.","journal-title":"Journal of Hospital Medicine"},{"key":"e_1_3_3_13_2","first-page":"1556","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Pal Arghya","year":"2018","unstructured":"Arghya Pal and Vineeth N. Balasubramanian. 2018. Adversarial data programming: Using gans to relax the bottleneck of curated labeled data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1556\u20131565."},{"key":"e_1_3_3_14_2","unstructured":"Arghya Pal and Vineeth N. Balasubramanian. 2020. Generative adversarial data programming. arXiv:2005.00364. Retrieved from https:\/\/arxiv.org\/abs\/2005.00364."},{"key":"e_1_3_3_15_2","volume-title":"Proceedings of the Conference on Connected Health: Applications, Systems and Engineering Technologies","author":"Pugh Sydney","year":"2021","unstructured":"Sydney Pugh, Ivan Ruchkin, Christopher Bonafide, Sara DeMauro, Oleg Sokolsky, Insup lee, and James Weimer. 2021. High-confidence data programming for evaluating suppression of physiological alarms. In Proceedings of the Conference on Connected Health: Applications, Systems and Engineering Technologies."},{"issue":"2","key":"e_1_3_3_16_2","doi-asserted-by":"crossref","first-page":"709","DOI":"10.1007\/s00778-019-00552-1","article-title":"Snorkel: Rapid training data creation with weak supervision","volume":"29","author":"Ratner Alexander","year":"2020","unstructured":"Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher R\u00e9. 2020. Snorkel: Rapid training data creation with weak supervision. The VLDB Journal 29, 2 (2020), 709\u2013730.","journal-title":"The VLDB Journal"},{"key":"e_1_3_3_17_2","first-page":"3567","article-title":"Data programming: Creating large training sets, quickly","volume":"29","author":"Ratner Alexander","year":"2016","unstructured":"Alexander Ratner, Christopher De Sa, Sen Wu, Daniel Selsam, and Christopher R\u00e9. 2016. Data programming: Creating large training sets, quickly. Advances in Neural Information Processing Systems 29 (2016), 3567.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_18_2","unstructured":"Alexander Ratner Braden Hancock Jared Dunnmon Frederic Sala Shreyash Pandey and Christopher R\u00e9. 2018. Training Complex Models with Multi-Task Weak Supervision. arXiv:1810.02840. Retrieved from https:\/\/arxiv.org\/abs\/1810.02840."},{"issue":"1","key":"e_1_3_3_19_2","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1097\/CCM.0000000000002803","article-title":"Technological distractions (part 2): A summary of approaches to manage clinical alarms with intent to reduce alarm fatigue","volume":"46","author":"Winters Bradford D.","year":"2018","unstructured":"Bradford D. Winters, Maria M. Cvach, Christopher P. Bonafide, Xiao Hu, Avinash Konkani, Michael F. O\u2019Connor, Jeffrey M. Rothschild, Nicholas M. Selby, Michele M. Pelter, Barbara McLean, et\u00a0al. 2018. Technological distractions (part 2): A summary of approaches to manage clinical alarms with intent to reduce alarm fatigue. Critical Care Medicine 46, 1 (2018), 130\u2013137. [Accessed January 10, 2022].","journal-title":"Critical Care Medicine"}],"container-title":["ACM Transactions on Computing for Healthcare"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3549942","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3549942","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3549942","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:08:13Z","timestamp":1750183693000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3549942"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,31]]},"references-count":18,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,10,31]]}},"alternative-id":["10.1145\/3549942"],"URL":"https:\/\/doi.org\/10.1145\/3549942","relation":{},"ISSN":["2691-1957","2637-8051"],"issn-type":[{"type":"print","value":"2691-1957"},{"type":"electronic","value":"2637-8051"}],"subject":[],"published":{"date-parts":[[2022,10,31]]},"assertion":[{"value":"2022-01-26","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-06-14","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-11-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}