{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,8]],"date-time":"2026-04-08T16:24:58Z","timestamp":1775665498467,"version":"3.50.1"},"reference-count":56,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,3,13]],"date-time":"2020-03-13T00:00:00Z","timestamp":1584057600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,3,13]],"date-time":"2020-03-13T00:00:00Z","timestamp":1584057600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This manuscript is focused on the use of connected sensor technologies, including wearables and other biosensors, for a wide range of health services, such as collecting digital endpoints in clinical trials and remotely monitoring patients in clinical care. The adoption of these technologies poses five risks that currently exceed our abilities to evaluate and secure these products: (1) validation, (2) security practices, (3) data rights and governance, (4) utility and usability; and (5) economic feasibility. In this manuscript we conduct a landscape analysis of emerging evaluation frameworks developed to better manage these risks, broadly in digital health. We then propose a framework specifically for connected sensor technologies. We provide a pragmatic guide for how to put this evaluation framework into practice, taking lessons from concepts in drug and nutrition labels to craft a connected sensor technology label.<\/jats:p>","DOI":"10.1038\/s41746-020-0237-3","type":"journal-article","created":{"date-parts":[[2020,3,13]],"date-time":"2020-03-13T11:03:25Z","timestamp":1584097405000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":66,"title":["Modernizing and designing evaluation frameworks for connected sensor technologies in medicine"],"prefix":"10.1038","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5379-3540","authenticated-orcid":false,"given":"Andrea","family":"Coravos","sequence":"first","affiliation":[]},{"given":"Megan","family":"Doerr","sequence":"additional","affiliation":[]},{"given":"Jennifer","family":"Goldsack","sequence":"additional","affiliation":[]},{"given":"Christine","family":"Manta","sequence":"additional","affiliation":[]},{"given":"Mark","family":"Shervey","sequence":"additional","affiliation":[]},{"given":"Beau","family":"Woods","sequence":"additional","affiliation":[]},{"given":"William A.","family":"Wood","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,3,13]]},"reference":[{"key":"237_CR1","unstructured":"Hailu, R. Fitbits and other wearables may not accurately track heart rates in people of color. https:\/\/www.statnews.com\/2019\/07\/24\/fitbit-accuracy-dark-skin\/ (2019)."},{"key":"237_CR2","unstructured":"Yazdi, D. The Apple Watch 4 is an iffy atrial fibrillation detector in those under age 55. https:\/\/www.statnews.com\/2019\/01\/08\/apple-watch-iffy-atrial-fibrillation-detector\/ (2019)."},{"key":"237_CR3","unstructured":"Brandom, R. UK hospitals hit with massive ransomware attack. https:\/\/www.theverge.com\/2017\/5\/12\/15630354\/nhs-hospitals-ransomware-hack-wannacry-bitcoin (2017)."},{"key":"237_CR4","unstructured":"Apple. ECG app and irregular heart rhythm notification available today on Apple Watch. https:\/\/www.apple.com\/newsroom\/2018\/12\/ecg-app-and-irregular-heart-rhythm-notification-available-today-on-apple-watch\/ (2018)."},{"key":"237_CR5","doi-asserted-by":"publisher","first-page":"956","DOI":"10.1056\/NEJMra1806949","volume":"381","author":"I Sim","year":"2019","unstructured":"Sim, I. Mobile devices and health. N. Engl. J. Med. 381, 956\u2013968. https:\/\/doi.org\/10.1056\/NEJMra1806949 (2019).","journal-title":"N. Engl. J. Med."},{"key":"237_CR6","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-019-0090-4","volume":"2","author":"A Coravos","year":"2019","unstructured":"Coravos, A., Khozin, S. & Mandl, K. D. Developing and adopting safe and effective digital biomarkers to improve patient outcomes. npj Digital Med. 2, 14 (2019).","journal-title":"npj Digital Med."},{"key":"237_CR7","unstructured":"Wang, T., Azad, T. & Rajan, R. The emerging influence of digital biomarkers on healthcare. https:\/\/rockhealth.com\/reports\/the-emerging-influence-of-digital-biomarkers-on-healthcare\/ (n.d.)."},{"key":"237_CR8","unstructured":"US Department of Health and Human Services, Food and Drug Administration. Device\u2013not a device. https:\/\/www.fda.gov\/medical-devices\/classify-your-medical-device\/device-not-device (2018)."},{"key":"237_CR9","unstructured":"Liu, S. Number of connected wearable devices worldwide from 2016 to 2022 (in millions). https:\/\/www.statista.com\/statistics\/487291\/global-connected-wearable-devices\/ (2019)."},{"key":"237_CR10","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1002\/cpt.1441","volume":"106","author":"S Khozin","year":"2019","unstructured":"Khozin, S. & Coravos, A. Decentralized trials in the age of real\u2010world evidence and inclusivity in clinical investigations. Clin. Pharmacol. Therapeutics 106, 25\u201327 (2019).","journal-title":"Clin. Pharmacol. Therapeutics"},{"key":"237_CR11","doi-asserted-by":"publisher","unstructured":"Dodge, H. H. et al. Use of high-frequency in-home monitoring data may reduce sample sizes needed in clinical trials. https:\/\/doi.org\/10.1371\/journal.pone.0138095 (2015).","DOI":"10.1371\/journal.pone.0138095"},{"key":"237_CR12","doi-asserted-by":"crossref","unstructured":"Goldsack, J. C. et al. Verification, Analytical Validation, and Clinical Validation (V3): The Foundation of Determining Fit-for-Purpose for Biometric Monitoring Technologies (BioMeTs). NPJ Digital Medicine, In Press. \u201cIN SAME SPECIAL ISSUE.\u201d (2020).","DOI":"10.1038\/s41746-020-0260-4"},{"key":"237_CR13","doi-asserted-by":"crossref","unstructured":"Huckvale, K., Torous, J. & Larsen, M. E. Assessment of the data sharing and privacy practices of smartphone apps for depression and smoking cessation. JAMA Netw. Open 2, e192542. https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/fullarticle\/2730782 (2019).","DOI":"10.1001\/jamanetworkopen.2019.2542"},{"key":"237_CR14","doi-asserted-by":"crossref","unstructured":"Coravos, A. et al. Digital medicine: a primer on measurement. Digital Biomark. 3, 31\u201371. https:\/\/www.karger.com\/Article\/FullText\/500413 (2019).","DOI":"10.1159\/000500413"},{"key":"237_CR15","unstructured":"Matwyshyn, A. M. The \u2018internet of bodies\u2019 is here. Are courts and regulators ready? https:\/\/www.wsj.com\/articles\/the-internet-of-bodies-is-here-are-courts-and-regulators-ready-1542039566 (2018)."},{"key":"237_CR16","unstructured":"FDA-NIH Biomarker Working Group. BEST (Biomarkers, EndpointS, and other Tools) Resource. Food and Drug Administration (US). https:\/\/www.ncbi.nlm.nih.gov\/books\/NBK326791 (2016)."},{"key":"237_CR17","unstructured":"Nielsen, J. Usability 101: Introduction to usability. https:\/\/www.nngroup.com\/articles\/usability-101-introduction-to-usability\/ (2012)."},{"key":"237_CR18","unstructured":"Elezea. Utility is more important than usability. https:\/\/elezea.com\/2016\/01\/utility-is-more-important-than-usability\/ (2016)."},{"key":"237_CR19","unstructured":"Cambridge. Cambridge Business English Dictionary. (Cambridge University Press, Cambridge, England. https:\/\/www.cambridge.org\/us\/cambridgeenglish\/catalog\/business-professional-and-vocational\/cambridge-business-english-dictionary\/ (2011)."},{"key":"237_CR20","unstructured":"US Department of Health and Human Services, Food and Drug Administration. Statement on new steps to advance digital health policies that encourage innovation and enable efficient and modern regulatory oversight. https:\/\/www.fda.gov\/news-events\/press-announcements\/statement-new-steps-advance-digital-health-policies-encourage-innovation-and-enable-efficient-and (2019)."},{"key":"237_CR21","unstructured":"US Department of Health and Human Services, Food and Drug Administration. Digital health software precertification (Pre-Cert) program. https:\/\/www.fda.gov\/medical-devices\/digital-health\/digital-health-software-precertification-pre-cert-program (2019)."},{"key":"237_CR22","unstructured":"Li, V. Senior Medical Devise Regulation Expert. Changes to existing medical software policies resulting from Section 3060 of the 21st Century Cures Act. http:\/\/www.fdalawblog.net\/2019\/10\/changes-to-existing-medical-software-policies-resulting-from-section-3060-of-the-21st-century-cures-act\/ (2019)."},{"key":"237_CR23","unstructured":"US Department of Health and Human Services, Food and Drug Administration. FDA warns JUUL Labs for marketing unauthorized modified risk tobacco products, including in outreach to youth. https:\/\/www.fda.gov\/news-events\/press-announcements\/fda-warns-juul-labs-marketing-unauthorized-modified-risk-tobacco-products-including-outreach-youth (2019)."},{"key":"237_CR24","unstructured":"US Department of Health and Human Services, Food and Drug Administration. Software as a Medical Device (SaMD). https:\/\/www.fda.gov\/medical-devices\/digital-health\/software-medical-device-samd (2018)."},{"key":"237_CR25","unstructured":"Anonymous. Reproducibility in machine learning for health. OpenReview.net. https:\/\/openreview.net\/forum?id=HylgS2IpLN (2019)."},{"key":"237_CR26","unstructured":"Gebru, T. et al. Datasheets for datasets. https:\/\/arxiv.org\/pdf\/1803.09010.pdf (2019)."},{"key":"237_CR27","unstructured":"Ross, H. & Bassoff, N. W. The \u201cDataset nutrition label project\u201d Tackles dataset health and standards. https:\/\/cyber.harvard.edu\/story\/2019-01\/dataset-nutrition-label-project-tackles-dataset-health-and-standards (2019)."},{"key":"237_CR28","unstructured":"Clinical Trials Transformation Initiative. CTTI unveils new database of feasibility studies on mobile technologies in clinical research. https:\/\/www.ctti-clinicaltrials.org\/news\/ctti-unveils-new-database-feasibility-studies-mobile-technologies-clinical-research (2019)."},{"key":"237_CR29","unstructured":"Goldsack, J., Chasse, R. A. & Wood, W. A. Digital endpoints library can aid clinical trials for new medicines. https:\/\/www.statnews.com\/2019\/11\/06\/digital-endpoints-library-clinical-trials-drug-development\/ (2019)."},{"key":"237_CR30","unstructured":"Digital Medicine Society (DiMe). DiMe\u2019s Library of Digital Endpoints. https:\/\/www.dimesociety.org\/index.php\/knowledge-center\/library-of-digital-endpoints (2019)."},{"key":"237_CR31","doi-asserted-by":"publisher","unstructured":"Beam, A. L., Manrai, A. K. & Ghassemi, M. Challenges to the reproducibility of machine learning models in health care. JAMA. https:\/\/doi.org\/10.1001\/jama.2019.20866.https:\/\/jamanetwork.com\/journals\/jama\/fullarticle\/2758612 (2020).","DOI":"10.1001\/jama.2019.20866."},{"key":"237_CR32","unstructured":"Coravos, A., Chen, I., Gordhanas, A. & Stern A. D. We should treat algorithms like prescription drugs. https:\/\/qz.com\/1540594\/treating-algorithms-like-prescription-drugs-could-reduce-ai-bias\/ (2019)."},{"key":"237_CR33","doi-asserted-by":"crossref","unstructured":"Woods, B., Coravos, A. & Corman, J. D. The case for a hippocratic oath for connected medical devices: Viewpoint. J. Med. Internet Res. 21, e12568. https:\/\/www.jmir.org\/2019\/3\/e12568\/ (2019).","DOI":"10.2196\/12568"},{"key":"237_CR34","unstructured":"US Department of Health and Human Services, Food and Drug Administration. Content of premarket submissions for management of cybersecurity in medical devices. https:\/\/www.fda.gov\/regulatory-information\/search-fda-guidance-documents\/content-premarket-submissions-management-cybersecurity-medical-devices (2018)."},{"key":"237_CR35","unstructured":"US Department of Health and Human Services, Food and Drug Administration. Postmarket management of cybersecurity in medical devices. https:\/\/www.fda.gov\/regulatory-information\/search-fda-guidance-documents\/postmarket-management-cybersecurity-medical-devices (2016)."},{"key":"237_CR36","unstructured":"Wikipedia. HITRUST. https:\/\/en.wikipedia.org\/wiki\/HITRUST (2019)."},{"key":"237_CR37","unstructured":"Sage Bionetworks. Elements of Informed Consent. https:\/\/sagebionetworks.org\/tools_resources\/elements-of-informed-consent\/ (2019)."},{"key":"237_CR38","unstructured":"PrivacyToolkit. A toolkit for biomedical researchers of best practices for privacy in digital studies. http:\/\/designmanual.sagebionetworks.org\/privacy_toolkit.html (n.d.)"},{"key":"237_CR39","unstructured":"HHS.gov. Section 508. https:\/\/www.hhs.gov\/web\/section-508\/additional-resources\/index.html (2019)."},{"key":"237_CR40","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1111\/cts.12602","volume":"12","author":"ES Izmailova","year":"2019","unstructured":"Izmailova, E. S. et al. Evaluation of wearable digital devices in a phase I clinical trial. Clin. Transl. Sci. 12, 247\u2013259. https:\/\/doi.org\/10.1111\/cts.12602 (2019).","journal-title":"Clin. Transl. Sci."},{"key":"237_CR41","unstructured":"Eldeas, A. The Internet of Things and cybersecurity: A Q&A with Sen. Edward J. Markey (D-MA) and Rep. Ted Lieu (D-CA). https:\/\/www.aei.org\/technology-and-innovation\/the-internet-of-things-and-cybersecurity-a-qa-with-sen-edward-j-markey-d-ma-and-rep-ted-lieu-d-ca\/ (2018)."},{"key":"237_CR42","unstructured":"National Telecommunications and Information Administration (NTIA). Communicating IoT Device Security Update Capability to Improve Transparency for Consumers. https:\/\/www.ntia.doc.gov\/files\/ntia\/publications\/draft_communicating_iot_security_update_capability_-_jul_14_2017_-_ntia_multistakeholder_process.pdf (2017)."},{"key":"237_CR43","unstructured":"Harris Interactive. Consumer Internet of Things Security Labelling Survey Research Findings. https:\/\/assets.publishing.service.gov.uk\/government\/uploads\/system\/uploads\/attachment_data\/file\/798543\/Harris_Interactive_Consumer_IoT_Security_Labelling_Survey_Report.pdf (n.d.)."},{"key":"237_CR44","unstructured":"Emami-Naeini, P., Agarwal, Y. & Cranor, L.F. Privacy and security nutrition label for smart devices. Privacy Forecast. https:\/\/privacy.shorensteincenter.org\/nutrition-labels (2019)."},{"key":"237_CR45","unstructured":"Shen, Y. & Vervier, P. A. Why we need a security and privacy \u201cNutrition Label\u201d for IoT devices. https:\/\/www.symantec.com\/blogs\/expert-perspectives\/why-we-need-security-and-privacy-nutrition-label-iot-devices (2019)."},{"key":"237_CR46","unstructured":"DataLabel.org https:\/\/www.datalabel.org\/ (2019)."},{"key":"237_CR47","doi-asserted-by":"crossref","unstructured":"Gropper, A. Patient Privacy Rights Information Governance Label. SSRN. https:\/\/ssrn.com\/abstract=3439701 (2019).","DOI":"10.2139\/ssrn.3439701"},{"key":"237_CR48","unstructured":"Peterson, M., Creating A. Cybersecurity \u201cEnergy Star\u201d. https:\/\/www.publicknowledge.org\/blog\/creating-a-cybersecurity-energy-star\/ (2018)."},{"key":"237_CR49","unstructured":"Digital Medicine Society (DiMe). Digital medicine society now accepting members. https:\/\/www.prnewswire.com\/news-releases\/digital-medicine-society-now-accepting-members-300848402.html (2019)."},{"key":"237_CR50","unstructured":"Goldsack, J. Laying the foundation: Defining digital medicine. https:\/\/medium.com\/digital-medicine-society-dime\/laying-the-foundation-defining-digital-medicine-49ab7b6ab6ef (2019)."},{"key":"237_CR51","unstructured":"NIST. Considerations for a core IoT cybersecurity capabilities baseline. https:\/\/www.nist.gov\/sites\/default\/files\/documents\/2019\/02\/01\/final_core_iot_cybersecurity_capabilities_baseline_considerations.pdf (2019)."},{"key":"237_CR52","unstructured":"National Telecommunications and Information Administration. Multistakeholder process: cybersecurity vulnerabilities. https:\/\/www.ntia.doc.gov\/other-publication\/2016\/multistakeholder-process-cybersecurity-vulnerabilities (2016)."},{"key":"237_CR53","unstructured":"Wikipedia. Bug bounty program. https:\/\/en.wikipedia.org\/wiki\/Bug_bounty_program (2019)."},{"key":"237_CR54","unstructured":"Fahs, G. et al. Building resilient medical technology supply chains with a software bill of materials. NPJ Digital Med., \u201cIN SAME SPECIAL ISSUE.\u201d (2020)."},{"key":"237_CR55","unstructured":"Fields, H. 508 compliance: Making your website more accessible. https:\/\/www.webdevelopmentgroup.com\/2017\/09\/508-compliance-making-websites-accessible-for-people-with-disabilities\/ (2017)."},{"key":"237_CR56","unstructured":"US Department of Health and Human Services, Food and Drug Administration. Guidance on Exculpatory Language in Informed Consent. https:\/\/www.fda.gov\/media\/81521\/download (2011)."}],"updated-by":[{"DOI":"10.1038\/s41746-020-0263-1","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2020,4,2]],"date-time":"2020-04-02T00:00:00Z","timestamp":1585785600000}}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0237-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0237-3","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0237-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,7]],"date-time":"2022-12-07T02:02:22Z","timestamp":1670378542000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0237-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,13]]},"references-count":56,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2020,12]]}},"alternative-id":["237"],"URL":"https:\/\/doi.org\/10.1038\/s41746-020-0237-3","relation":{"correction":[{"id-type":"doi","id":"10.1038\/s41746-020-0263-1","asserted-by":"object"}]},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,3,13]]},"assertion":[{"value":"21 October 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 February 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 March 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 February 2021","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Correction","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"A Correction to this paper has been published:","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"https:\/\/doi.org\/10.1038\/s41746-020-0263-1","URL":"https:\/\/doi.org\/10.1038\/s41746-020-0263-1","order":7,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 April 2020","order":8,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Correction","order":9,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"A Correction to this paper has been published:","order":10,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"https:\/\/doi.org\/10.1038\/s41746-020-0263-1","URL":"https:\/\/doi.org\/10.1038\/s41746-020-0263-1","order":11,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"J.G. is an employee of HealthMode. M.D. is a co-inventor of the Cleveland Clinic\u2019s MyLegacy intellectual property portfolio, now licensed to Family Care Path, Inc.; as part of this license, she is entitled to a share in both royalties and returns on equity. W.W., B.W. are advisors and equity-holders, and A.C., C.M., and M.S. are employees of Elektra Labs, a healthcare-security company working in digital medicine. W.W. is also an advisor for Koneksa Health, a consultant for Best Doctors, and has received research funding within the last 12 months from Pfizer and Genentech. The other authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"37"}}