{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T00:46:38Z","timestamp":1778201198167,"version":"3.51.4"},"reference-count":37,"publisher":"Oxford University Press (OUP)","issue":"12","license":[{"start":{"date-parts":[[2022,10,7]],"date-time":"2022-10-07T00:00:00Z","timestamp":1665100800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"State of Washington Department of Health"},{"DOI":"10.13039\/100007812","name":"University of Washington","doi-asserted-by":"publisher","award":["HED 25742"],"award-info":[{"award-number":["HED 25742"]}],"id":[{"id":"10.13039\/100007812","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,11,14]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>Digital exposure notifications (DEN) systems were an emergency response to the coronavirus disease 2019 (COVID-19) pandemic, harnessing smartphone-based technology to enhance conventional pandemic response strategies such as contact tracing. We identify and describe performance measurement constructs relevant to the implementation of DEN tools: (1) reach (number of users enrolled in the intervention); (2) engagement (utilization of the intervention); and (3) effectiveness in preventing transmissions of COVID-19 (impact of the intervention). We also describe WA State\u2019s experience utilizing these constructs to design data-driven evaluation approaches.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>We conducted an environmental scan of DEN documentation and relevant publications. Participation in multidisciplinary collaborative environments facilitated shared learning. Compilation of available data sources and their relevance to implementation and operation workflows were synthesized to develop implementation evaluation constructs.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>We identified 8 useful performance indicators within reach, engagement, and effectiveness constructs.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>We use implementation science to frame the evaluation of DEN tools by linking the theoretical constructs with the metrics available in the underlying disparate, deidentified, and aggregate data infrastructure. Our challenges in developing meaningful metrics include limited data science competencies in public health, validation of analytic methodologies in the complex and evolving pandemic environment, and the lack of integration with the public health infrastructure.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>Continued collaboration and multidisciplinary consensus activities can improve the utility of DEN tools for future public health emergencies.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocac178","type":"journal-article","created":{"date-parts":[[2022,10,7]],"date-time":"2022-10-07T16:03:43Z","timestamp":1665158623000},"page":"2050-2056","source":"Crossref","is-referenced-by-count":6,"title":["Trading-off privacy and utility: the Washington State experience assessing the performance of a public health digital exposure notification system for coronavirus disease 2019"],"prefix":"10.1093","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4118-8882","authenticated-orcid":false,"given":"Courtney D","family":"Segal","sequence":"first","affiliation":[{"name":"Department of Health Systems and Population Health, University of Washington , Seattle, Washington DC, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1053-7501","authenticated-orcid":false,"given":"William B","family":"Lober","sequence":"additional","affiliation":[{"name":"Department of Biobehavioral Nursing and Health Informatics, University of Washington , Seattle, Washington DC, USA"},{"name":"Clinical Informatics Research Group, University of Washington , Seattle, Washington DC, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3863-7774","authenticated-orcid":false,"given":"Debra","family":"Revere","sequence":"additional","affiliation":[{"name":"Department of Health Systems and Population Health, University of Washington , Seattle, Washington DC, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daniel","family":"Lorigan","sequence":"additional","affiliation":[{"name":"Clinical Informatics Research Group, University of Washington , Seattle, Washington DC, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5452-9362","authenticated-orcid":false,"given":"Bryant T","family":"Karras","sequence":"additional","affiliation":[{"name":"Washington State Department of Health , Olympia, Washington DC, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1974-8196","authenticated-orcid":false,"given":"Janet G","family":"Baseman","sequence":"additional","affiliation":[{"name":"Department of Epidemiology, University of Washington , Seattle, Washington DC, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2022,10,7]]},"reference":[{"issue":"5","key":"2022111607593885200_ocac178-B1","doi-asserted-by":"crossref","first-page":"435","DOI":"10.1097\/00124784-200409000-00011","article-title":"Getting what you pay for: public health spending and the performance of essential public health services","volume":"10","author":"Mays","year":"2004","journal-title":"J Public Health Manag Pract"},{"key":"2022111607593885200_ocac178-B2","author":"Centers for Disease Control"},{"key":"2022111607593885200_ocac178-B3","author":"World Health Organization"},{"issue":"23","key":"2022111607593885200_ocac178-B4","doi-asserted-by":"crossref","first-page":"2352","DOI":"10.1056\/NEJMoa032111","article-title":"Public health measures to control the spread of the severe acute respiratory syndrome during the outbreak in Toronto","volume":"350","author":"Svoboda","year":"2004","journal-title":"N Engl J Med"},{"issue":"24","key":"2022111607593885200_ocac178-B5","doi-asserted-by":"crossref","first-page":"E653","DOI":"10.1503\/cmaj.200922","article-title":"Digital contact tracing for COVID-19","volume":"192","author":"Kleinman","year":"2020","journal-title":"CMAJ"},{"issue":"3","key":"2022111607593885200_ocac178-B6","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1007\/s10676-020-09566-8","article-title":"Digital contact tracing and exposure notification: ethical guidance for trustworthy pandemic management","volume":"23","author":"Ranisch","year":"2021","journal-title":"Ethics Inf Technol"},{"key":"2022111607593885200_ocac178-B7","year":"2020"},{"key":"2022111607593885200_ocac178-B8","author":"Troncoso"},{"key":"2022111607593885200_ocac178-B9","author":"Lovett","year":"2020"},{"key":"2022111607593885200_ocac178-B10","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1155\/2017\/1452415","article-title":"Informatics metrics and measures for a smart public health systems approach: information science perspective","volume":"2017","author":"Carney","year":"2017","journal-title":"Comput Math Methods Med"},{"issue":"6","key":"2022111607593885200_ocac178-B11","doi-asserted-by":"crossref","first-page":"535","DOI":"10.1136\/jamia.2001.0080535","article-title":"A national agenda for public health informatics: summarized recommendations from the 2001 AMIA Spring Congress","volume":"8","author":"Yasnoff","year":"2001","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2022111607593885200_ocac178-B12","article-title":"WA Notify: the planning and implementation of a Bluetooth exposure notification tool for COVID-19 pandemic response in Washington State","volume":"13","author":"Chen","year":"2021","journal-title":"Online J Public Health Inform"},{"key":"2022111607593885200_ocac178-B13","author":"World Health Organization"},{"key":"2022111607593885200_ocac178-B14","doi-asserted-by":"crossref","first-page":"64","DOI":"10.3389\/fpubh.2019.00064","article-title":"RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review","volume":"7","author":"Glasgow","year":"2019","journal-title":"Front Public Health"},{"key":"2022111607593885200_ocac178-B15","author":"Exposure notifications"},{"key":"2022111607593885200_ocac178-B16","author":"Exposure Notification"},{"key":"2022111607593885200_ocac178-B17","author":"Analytics in Exposure Notifications Express: FAQ"},{"key":"2022111607593885200_ocac178-B18","author":"Link to Linux Public Risk Score Webpage"},{"key":"2022111607593885200_ocac178-B19","doi-asserted-by":"crossref","first-page":"003335492211103","DOI":"10.1177\/00333549221110301","article-title":"Engagement in protective behaviors by digital exposure notification users during the COVID-19 pandemic, Washington State, January\u2013June 2021","author":"Baseman","year":"2022","journal-title":"Public Health Rep"},{"key":"2022111607593885200_ocac178-B20","author":"Revere","year":"2022"},{"key":"2022111607593885200_ocac178-B21"},{"key":"2022111607593885200_ocac178-B22","doi-asserted-by":"crossref","first-page":"w20457","DOI":"10.4414\/smw.2020.20457","article-title":"Early evidence of effectiveness of digital contact tracing for SARS-CoV-2 in Switzerland","volume":"150","author":"Salath\u00e9","year":"2020","journal-title":"Swiss Med Wkly"},{"key":"2022111607593885200_ocac178-B23","first-page":"519","author":"Sun"},{"key":"2022111607593885200_ocac178-B24","author":"Abueg"},{"key":"2022111607593885200_ocac178-B25","doi-asserted-by":"crossref","first-page":"408","DOI":"10.1038\/s41586-021-03606-z","article-title":"The epidemiological impact of the NHS COVID-19 app","author":"Wymant","year":"2021","journal-title":"Nature"},{"issue":"6491","key":"2022111607593885200_ocac178-B26","doi-asserted-by":"publisher","DOI":"10.1126\/science.abb6936","article-title":"Quantifying SARS-CoV-2 transmission suggests epidemic control with digital contact tracing","volume":"368","author":"Ferretti","year":"2020","journal-title":"Science"},{"issue":"4","key":"2022111607593885200_ocac178-B27","doi-asserted-by":"crossref","first-page":"e218184","DOI":"10.1001\/jamanetworkopen.2021.8184","article-title":"A data-driven simulation of the exposure notification cascade for digital contact tracing of SARS-CoV-2 in Zurich, Switzerland","volume":"4","author":"Menges","year":"2021","journal-title":"JAMA Netw Open"},{"issue":"4","key":"2022111607593885200_ocac178-B28","doi-asserted-by":"crossref","first-page":"e19359","DOI":"10.2196\/19359","article-title":"Covid-19 contact tracing and data protection can go together","volume":"8","author":"Abeler","year":"2020","journal-title":"JMIR Mhealth Uhealth"},{"issue":"7","key":"2022111607593885200_ocac178-B29","doi-asserted-by":"crossref","first-page":"e342","DOI":"10.1016\/S2589-7500(20)30133-3","article-title":"The need for privacy with public digital contact tracing during the COVID-19 pandemic","volume":"2","author":"Bengio","year":"2020","journal-title":"Lancet Digit Health"},{"issue":"11","key":"2022111607593885200_ocac178-B30","doi-asserted-by":"crossref","first-page":"2076","DOI":"10.2105\/AJPH.2009.190959","article-title":"Reaiming RE-AIM: using the model to plan, implement, and evaluate the effects of environmental change approaches to enhancing population health","volume":"100","author":"King","year":"2010","journal-title":"Am J Public Health"},{"issue":"3","key":"2022111607593885200_ocac178-B31","doi-asserted-by":"crossref","first-page":"361","DOI":"10.1038\/s41591-021-01236-6","article-title":"Time to evaluate COVID-19 contact-tracing apps","volume":"27","author":"Colizza","year":"2021","journal-title":"Nat Med"},{"key":"2022111607593885200_ocac178-B32","doi-asserted-by":"crossref","first-page":"29","DOI":"10.4414\/smw.2020.20324","article-title":"A research agenda for digital proximity tracing apps. Author(s): ETH Library A research agenda for digital proximity tracing apps","volume":"150","author":"von Wyl","year":"2020","journal-title":"Swiss Med Wkly"},{"key":"2022111607593885200_ocac178-B33","author":"Panovska-Griffiths"},{"issue":"2","key":"2022111607593885200_ocac178-B34","doi-asserted-by":"crossref","first-page":"246","DOI":"10.3390\/electronics11020246","article-title":"Understanding users\u2019 behavior towards applications privacy policies","volume":"11","author":"Ullah","year":"2022","journal-title":"Electronics"},{"issue":"1","key":"2022111607593885200_ocac178-B35","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1007\/s43621-022-00074-x","article-title":"How policymakers and other leaders can build a more sustainable post-COVID-19 normal","volume":"3","author":"Bragge","year":"2022","journal-title":"Discov Sustain"},{"key":"2022111607593885200_ocac178-B36","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1007\/978-3-030-41215-9_9","volume-title":"Public Health Informatics and Information Systems.","author":"Hulkower","year":"2020"},{"issue":"6","key":"2022111607593885200_ocac178-B37","doi-asserted-by":"crossref","first-page":"e23250","DOI":"10.2196\/23250","article-title":"State of the art in adoption of contact tracing apps and recommendations regarding privacy protection and public health: systematic review","volume":"9","author":"Kolasa","year":"2021","journal-title":"JMIR Mhealth Uhealth"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/12\/2050\/47027806\/ocac178.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/12\/2050\/47027806\/ocac178.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,21]],"date-time":"2023-02-21T00:02:07Z","timestamp":1676937727000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/29\/12\/2050\/6752071"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,7]]},"references-count":37,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2022,10,7]]},"published-print":{"date-parts":[[2022,11,14]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocac178","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,12,1]]},"published":{"date-parts":[[2022,10,7]]}}}