{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:34:53Z","timestamp":1760243693180,"version":"build-2065373602"},"reference-count":52,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2022,9,27]],"date-time":"2022-09-27T00:00:00Z","timestamp":1664236800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Despite the many successes of artificial intelligence in healthcare applications where human\u2013machine teaming is an intrinsic characteristic of the environment, there is little work that proposes methods for adapting quantitative health data-features with human expertise insights. A method for incorporating qualitative expert perspectives in machine learning training data is proposed. The method implements an entropy-based consensus construct that minimizes the challenges of qualitative-scale data such that they can be combined with quantitative measures in a critical clinical event (CCE) vector. Specifically, the CCE vector minimizes the effects where (a) the sample size is too small, (b) the data may not be normally distributed, or (c) The data are from Likert scales, which are ordinal, so parametric statistics cannot be used. The incorporation of human perspectives in machine learning training data provides encoding of human considerations in the subsequent machine learning model. This encoding provides a basis for increasing explainability, understandability, and ultimately trust in AI-based clinical decision support system (CDSS), thereby improving human\u2013machine teaming concerns. A discussion of applying the CCE vector in a CDSS regime and implications for machine learning are also presented.<\/jats:p>","DOI":"10.3390\/e24101369","type":"journal-article","created":{"date-parts":[[2022,9,27]],"date-time":"2022-09-27T23:12:12Z","timestamp":1664320332000},"page":"1369","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Providing Care: Intrinsic Human\u2013Machine Teams and Data"],"prefix":"10.3390","volume":"24","author":[{"given":"Stephen","family":"Russell","sequence":"first","affiliation":[{"name":"Department of Research, Opportunities and Innovation in Data Science, Jackson Health System, Miami, FL 33136, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ashwin","family":"Kumar","sequence":"additional","affiliation":[{"name":"Department of Research, Opportunities and Innovation in Data Science, Jackson Health System, Miami, FL 33136, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,9,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1038\/s41591-021-01614-0","article-title":"AI in Health and Medicine","volume":"28","author":"Rajpurkar","year":"2022","journal-title":"Nat. Med."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"103902","DOI":"10.1016\/j.robot.2021.103902","article-title":"Robotics and Artificial Intelligence in Healthcare during COVID-19 Pandemic: A Systematic Review","volume":"146","author":"Sarker","year":"2021","journal-title":"Robot. Auton. Syst."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"9999504","DOI":"10.1155\/2021\/9999504","article-title":"Internet of Things and Robotics in Transforming Current-Day Healthcare Services","volume":"2021","author":"Pradhan","year":"2021","journal-title":"J. Healthc. Eng."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Kolpashchikov, D., Gerget, O., and Meshcheryakov, R. (2022). Robotics in Healthcare. Handbook of Artificial Intelligence in Healthcare, Springer.","DOI":"10.1007\/978-3-030-83620-7_12"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Bakshi, G., Kumar, A., and Puranik, A.N. (2022). Adoption of Robotics Technology in Healthcare Sector. Advances in Communication, Devices and Networking, Springer.","DOI":"10.1007\/978-981-16-2911-2_42"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Kulkov, I. (2021). Next-Generation Business Models for Artificial Intelligence Start-Ups in the Healthcare Industry. Int. J. Entrep. Behav. Res., 27.","DOI":"10.1108\/IJEBR-04-2021-0304"},{"key":"ref_7","unstructured":"Daugherty, P., Carrel-Billiard, M., and Biltz, M. (2019). Accenture Technology Vision 2019, Accenture."},{"key":"ref_8","unstructured":"Collier, M., Fu, R., Yin, L., and Christiansen, P. (2017). Artificial Intelligence: Healthcare\u2019s New Nervous System, Accenture."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Park, S.Y., Kuo, P.-Y., Barbarin, A., Kaziunas, E., Chow, A., Singh, K., Wilcox, L., and Lasecki, W.S. (2019, January 9\u201313). Identifying Challenges and Opportunities in Human-AI Collaboration in Healthcare. Proceedings of the Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing, Austin, TX, USA.","DOI":"10.1145\/3311957.3359433"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Mesk\u00f3, B., Het\u00e9nyi, G., and Gy Horffy, Z. (2018). Will Artificial Intelligence Solve the Human Resource Crisis in Healthcare?. BMC Health Serv. Res., 18.","DOI":"10.1186\/s12913-018-3359-4"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Russell, S., Jalaian, B., and Moskowitz, I.S. (2021). Re-Orienting towards the Science of the Artificial: Engineering AI Systems. Systems Engineering and Artificial Intelligence, Springer Nature, Switzerland AG.","DOI":"10.1007\/978-3-030-77283-3_8"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"100347","DOI":"10.1016\/j.patter.2021.100347","article-title":"Addressing Bias in Big Data and AI for Health Care: A Call for Open Science","volume":"2","author":"Norori","year":"2021","journal-title":"Patterns"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Lyons, J.B., Wynne, K.T., Mahoney, S., and Roebke, M.A. (2019). Trust and Human-Machine Teaming: A Qualitative Study. Artificial Intelligence for the Internet of Everything, Elsevier.","DOI":"10.1016\/B978-0-12-817636-8.00006-5"},{"key":"ref_14","first-page":"610","article-title":"The Utility of Explainable Ai in Ad Hoc Human-Machine Teaming","volume":"34","author":"Paleja","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"4","DOI":"10.21037\/jmai-21-25","article-title":"Factors Influencing Trust in Medical Artificial Intelligence for Healthcare Professionals: A Narrative Review","volume":"5","author":"Tucci","year":"2022","journal-title":"J. Med. Artif. Intell."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Lai, Y., Kankanhalli, A., and Ong, D. (2021, January 5\u20138). Human-AI Collaboration in Healthcare: A Review and Research Agenda. Proceedings of the Hawaii International Conference on System Sciences 2021, Maui, HI, USA.","DOI":"10.24251\/HICSS.2021.046"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1016\/j.ijhcs.2011.08.002","article-title":"Workarounds in the Use of IS in Healthcare: A Case Study of an Electronic Medication Administration System","volume":"70","author":"Yang","year":"2012","journal-title":"Int. J. Hum. Comput. Stud."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"e15154","DOI":"10.2196\/15154","article-title":"Artificial Intelligence and Human Trust in Healthcare: Focus on Clinicians","volume":"22","author":"Asan","year":"2020","journal-title":"J. Med. Internet Res."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Maadi, M., Akbarzadeh Khorshidi, H., and Aickelin, U. (2021). A Review on Human\u2013AI Interaction in Machine Learning and Insights for Medical Applications. Int. J. Environ. Res. Public Health, 18.","DOI":"10.3390\/ijerph18042121"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Wang, D., Churchill, E., Maes, P., Fan, X., Shneiderman, B., Shi, Y., and Wang, Q. (2020, January 25\u201330). From Human-Human Collaboration to Human-AI Collaboration: Designing AI Systems That Can Work Together with People. Proceedings of the Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA.","DOI":"10.1145\/3334480.3381069"},{"key":"ref_21","unstructured":"Mucha, T., Halminen, O., Tenhunen, H., and Sepp\u00e4l\u00e4, T. (May, January 28). Commercial Adoption of AI in the Healthcare Sector: An Exploratory Analysis of S&P500 Companies. Proceedings of the MIE, Geneva, Switzerland."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"6","DOI":"10.12688\/f1000research.26997.2","article-title":"AI in Healthcare: A Narrative Review","volume":"10","author":"Haataja","year":"2021","journal-title":"F1000Research"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1186\/s12967-019-02204-y","article-title":"Perceptions of Artificial Intelligence in Healthcare: Findings from a Qualitative Survey Study among Actors in France","volume":"18","author":"Brian","year":"2020","journal-title":"J. Transl. Med."},{"key":"ref_24","unstructured":"(2022, August 16). Health IT Analytics Arguing the Pros and Cons of Artificial Intelligence in Healthcare. Available online: https:\/\/healthitanalytics.com\/news\/arguing-the-pros-and-cons-of-artificial-intelligence-in-healthcare."},{"key":"ref_25","unstructured":"Davenport, T.H., and Bean, R. (2022, August 16). Clinical AI Gets the Headlines, but Administrative AI May Be a Better Bet. Available online: https:\/\/sloanreview.mit.edu\/article\/clinical-ai-gets-the-headlines-but-administrative-ai-may-be-a-better-bet\/."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"153316","DOI":"10.1109\/ACCESS.2021.3127881","article-title":"A Systematic Review of Human-Computer Interaction and Explainable Artificial Intelligence in Healthcare with Artificial Intelligence Techniques","volume":"9","author":"Nazar","year":"2021","journal-title":"IEEE Access"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Pawar, U., O\u2019Shea, D., Rea, S., and O\u2019Reilly, R. (2020, January 15\u201319). Explainable Ai in Healthcare. Proceedings of the 2020 International Conference on Cyber Situational Awareness, Data Analytics and Assessment (CyberSA), Dublin, Ireland.","DOI":"10.1109\/CyberSA49311.2020.9139655"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"727856","DOI":"10.3389\/fdata.2021.727856","article-title":"Editorial: ML and AI Safety, Effectiveness and Explainability in Healthcare","volume":"4","author":"Benrimoh","year":"2021","journal-title":"Front. Big Data"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"e1379","DOI":"10.1002\/widm.1379","article-title":"Interpretability of Machine Learning-Based Prediction Models in Healthcare","volume":"10","author":"Stiglic","year":"2020","journal-title":"Wiley Interdiscip. Rev. Data Min. Knowl. Discov."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"59","DOI":"10.7326\/M19-2548","article-title":"Should Health Care Demand Interpretable Artificial Intelligence or Accept \u201cBlack Box\u201d Medicine?","volume":"172","author":"Wang","year":"2020","journal-title":"Ann. Intern. Med."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1016\/j.pec.2013.04.006","article-title":"Using Design Science and Artificial Intelligence to Improve Health Communication: ChronologyMD Case Example","volume":"92","author":"Neuhauser","year":"2013","journal-title":"Patient Educ. Couns."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"16094069211045476","DOI":"10.1177\/16094069211045476","article-title":"Without Qualitative Health Data, Precision Health Will Be Imprecise","volume":"20","author":"Myroniuk","year":"2021","journal-title":"Int. J. Qual. Methods"},{"key":"ref_33","first-page":"2","article-title":"The Human-Machine Interface: Inviting Your Computer Into Your Patient-Clinician Relationship","volume":"5","author":"Janisse","year":"2001","journal-title":"Perm. J."},{"key":"ref_34","first-page":"6","article-title":"Another Look at Likert Scales","volume":"31","author":"Willits","year":"2016","journal-title":"J. Rural Soc. Sci."},{"key":"ref_35","first-page":"64","article-title":"Likert Scales and Data Analyses","volume":"40","author":"Allen","year":"2007","journal-title":"Qual. Prog."},{"key":"ref_36","first-page":"331","article-title":"The Likert Scale Revisited","volume":"39","author":"Albaum","year":"1997","journal-title":"Mark. Res. Soc. J."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1093\/swr\/27.1.45","article-title":"Phrase Completions: An Alternative to Likert Scales","volume":"27","author":"Hodge","year":"2003","journal-title":"Soc. Work Res."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1016\/j.eujim.2015.07.002","article-title":"Delphi Methodology in Health Research: How to Do It?","volume":"7","author":"Trevelyan","year":"2015","journal-title":"Eur. J. Integr. Med."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Laka, M., Milazzo, A., and Merlin, T. (2021). Factors That Impact the Adoption of Clinical Decision Support Systems (CDSS) in Healthcare Settings for Evidence-Based Decision Making. Int. J. Environ. Res. Public Health, 18.","DOI":"10.3390\/ijerph18041901"},{"key":"ref_40","unstructured":"Agency for Healthcare Research and Quality (AHRQ) (2019). Clinical Decision Support Systems, Agency for Healthcare Research and Quality (AHRQ)."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1319","DOI":"10.1016\/j.lpm.2019.09.060","article-title":"Expert consensus statement on blood pressure measurement from the French Society of Hypertension, an affiliate of the French Society of Cardiology","volume":"48","author":"Denolle","year":"2019","journal-title":"Presse Medicale"},{"key":"ref_42","unstructured":"Wierman, M.J., and Tastle, W.J. (2005, January 26\u201328). Consensus and Dissention: Theory and Properties. Proceedings of the NAFIPS 2005-2005 Annual Meeting of the North American Fuzzy Information Processing Society, Detroit, MI, USA."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1002\/j.1538-7305.1948.tb01338.x","article-title":"A Mathematical Theory of Communication","volume":"27","author":"Shannon","year":"1948","journal-title":"Bell Syst. Tech. J."},{"key":"ref_44","unstructured":"Tastle, J.M., and Tastle, W.J. (2006). Extending the Consensus Measure: Analyzing Ordinal Data with Respect to Extrema. Syst. Educ. J., 4, Available online: http:\/\/isedj.org\/4\/72\/."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"487","DOI":"10.3758\/BF03192803","article-title":"An Information Theoretic Measure for the Evaluation of Ordinal Scale Data","volume":"38","author":"Tastle","year":"2006","journal-title":"Behav. Res. Methods"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Santoyo-S\u00e1nchez, G., Merino-Soto, C., Flores-Hern\u00e1ndez, S., Pelcastre-Villafuerte, B.E., and Reyes-Morales, H. (2022). Content Validity of a Scale Designed to Measure the Access of Older Adults to Outpatient Health Services. Int. J. Environ. Res. Public Health, 19.","DOI":"10.3390\/ijerph191610102"},{"key":"ref_47","unstructured":"Bahaitham, H.A. (2011). A Framework for Quantifying Sustainability of Lean Implementation in Healthcare Organizations. [Ph.D. Thesis, College of Engineering and Computer Science, University of Central Florida]."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Engle, K.M., and Forgionne, G.A. (2009, January 19\u201324). Rescaling Non-Metric Data to Metric Data Using Multi-Dimensional Scaling. Proceedings of the International Conference on Internationalization, Design and Global Development, San Diego, CA, USA.","DOI":"10.1007\/978-3-642-02767-3_27"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1038\/s41746-021-00385-9","article-title":"Do as AI Say: Susceptibility in Deployment of Clinical Decision-Aids","volume":"4","author":"Gaube","year":"2021","journal-title":"NPJ Digit. Med."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1038\/s41746-022-00597-7","article-title":"Human\u2013Machine Teaming Is Key to AI Adoption: Clinicians\u2019 Experiences with a Deployed Machine Learning System","volume":"5","author":"Henry","year":"2022","journal-title":"NPJ Digit. Med."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"777","DOI":"10.1037\/rev0000325","article-title":"On the Role of Similarity in Mental Accounting and Hedonic Editing","volume":"129","author":"Evers","year":"2022","journal-title":"Psychol. Rev."},{"key":"ref_52","first-page":"741","article-title":"Similarity-Based Classification: Concepts and Algorithms","volume":"10","author":"Chen","year":"2009","journal-title":"J. Mach. Learn. Res."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/10\/1369\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:40:22Z","timestamp":1760143222000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/10\/1369"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,27]]},"references-count":52,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2022,10]]}},"alternative-id":["e24101369"],"URL":"https:\/\/doi.org\/10.3390\/e24101369","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2022,9,27]]}}}