{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T01:49:50Z","timestamp":1776131390175,"version":"3.50.1"},"reference-count":57,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,6,8]],"date-time":"2023-06-08T00:00:00Z","timestamp":1686182400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,6,8]],"date-time":"2023-06-08T00:00:00Z","timestamp":1686182400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"FCT - Foundation for Science and Technology, I.P. - MEDI-VALUE project","award":["PTDC\/EGE-OGE\/29699\/2017"],"award-info":[{"award-number":["PTDC\/EGE-OGE\/29699\/2017"]}]},{"name":"FCT - Foundation for Science and Technology, I.P. - MEDI-VALUE project","award":["PTDC\/EGE-OGE\/29699\/2017"],"award-info":[{"award-number":["PTDC\/EGE-OGE\/29699\/2017"]}]},{"name":"FCT - Foundation for Science and Technology, I.P. - MEDI-VALUE project","award":["PTDC\/EGE-OGE\/29699\/2017"],"award-info":[{"award-number":["PTDC\/EGE-OGE\/29699\/2017"]}]},{"name":"FCT - Foundation for Science and Technology, I.P. - MEDI-VALUE project","award":["PTDC\/EGE-OGE\/29699\/2017"],"award-info":[{"award-number":["PTDC\/EGE-OGE\/29699\/2017"]}]},{"name":"FCT - Foundation for Science and Technology, I.P. - MEDI-VALUE project","award":["PTDC\/EGE-OGE\/29699\/2017"],"award-info":[{"award-number":["PTDC\/EGE-OGE\/29699\/2017"]}]},{"name":"FCT - Foundation for Science and Technology, I.P. - CEG-IST","award":["UIDB\/00097\/2020"],"award-info":[{"award-number":["UIDB\/00097\/2020"]}]},{"name":"FCT - Foundation for Science and Technology, I.P. - CEG-IST","award":["UIDB\/00097\/2020"],"award-info":[{"award-number":["UIDB\/00097\/2020"]}]},{"name":"FCT - Foundation for Science and Technology, I.P. - CEG-IST","award":["UIDB\/00097\/2020"],"award-info":[{"award-number":["UIDB\/00097\/2020"]}]},{"name":"FCT - Foundation for Science and Technology, I.P. - CEG-IST","award":["UIDB\/00097\/2020"],"award-info":[{"award-number":["UIDB\/00097\/2020"]}]},{"name":"FCT - Foundation for Science and Technology, I.P. - Individual Doctoral Fellowship","award":["2020.05289.BD"],"award-info":[{"award-number":["2020.05289.BD"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Health Serv Res"],"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>Implementation and uptake of health technology assessment for evaluating medical devices require including aspects that different stakeholders consider relevant, beyond cost and effectiveness. However, the involvement of stakeholders in sharing their views still needs to be improved.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Objective<\/jats:title>\n                <jats:p>This article explores the relevance of distinct value aspects for evaluating different types of medical devices according to stakeholders' views.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>Thirty-four value aspects collected through literature review and expert validation were the input for a 2-round Web-Delphi process. In the Web-Delphi, a panel of participants from five stakeholders\u2019 groups (healthcare professionals, buyers and policymakers, academics, industry, and patients and citizens) judged the relevance of each aspect, by assigning a relevance-level (\u2018Critical\u2019, \u2018Fundamental\u2019, \u2018Complementary\u2019, or \u2018Irrelevant\u2019), for two types of medical devices separately: \u2018Implantable\u2019 and \u2018In vitro tests based on biomarkers\u2019. Opinions were analysed at the panel and group level, and similarities across devices were identified.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>One hundred thirty-four participants completed the process. No aspects were considered \u2018Irrelevant\u2019, neither for the panel nor for stakeholder groups, in both types of devices. The panel considered effectiveness and safety-related aspects \u2018Critical\u2019 (e.g., \u2018Adverse events for the patient\u2019), and costs-related aspects \u2018Fundamental\u2019 (e.g., \u2018Cost of the medical device\u2019). Several additional aspects not included in existing frameworks\u2019 literature, e.g., related to environmental impact and devices\u2019 usage by the healthcare professional, were deemed as relevant by the panel. A moderate to substantial agreement across and within groups was observed.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusion<\/jats:title>\n                <jats:p>Different stakeholders agree on the relevance of including multiple aspects in medical devices\u2019 evaluation. This study produces key information to inform the development of frameworks for valuing medical devices, and to guide evidence collection.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12913-023-09550-0","type":"journal-article","created":{"date-parts":[[2023,6,8]],"date-time":"2023-06-08T08:01:43Z","timestamp":1686211303000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Which value aspects are relevant for the evaluation of medical devices? Exploring stakeholders\u2019 views through a Web-Delphi process"],"prefix":"10.1186","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9513-2476","authenticated-orcid":false,"given":"Liliana","family":"Freitas","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3863-1070","authenticated-orcid":false,"given":"Ana C. L.","family":"Vieira","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4053-9214","authenticated-orcid":false,"given":"M\u00f3nica D.","family":"Oliveira","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4085-875X","authenticated-orcid":false,"given":"Helena","family":"Monteiro","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7906-379X","authenticated-orcid":false,"given":"Carlos A.","family":"Bana e Costa","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,6,8]]},"reference":[{"issue":"4","key":"9550_CR1","doi-asserted-by":"publisher","first-page":"404","DOI":"10.1111\/j.1524-4733.2008.00476_2.x","volume":"12","author":"RS Taylor","year":"2009","unstructured":"Taylor RS, Iglesias CP. Assessing the clinical and cost-effectiveness of medical devices and drugs: are they that different? Value Health. 2009;12(4):404\u20136.","journal-title":"Value Health"},{"key":"9550_CR2","doi-asserted-by":"publisher","first-page":"891","DOI":"10.1007\/s10198-019-01052-3","volume":"20","author":"MD Oliveira","year":"2019","unstructured":"Oliveira MD, Mataloto I, Kanavos P. Multi-criteria decision analysis for health technology assessment: addressing methodological challenges to improve the state of the art. Eur J Health Econ. 2019;20:891\u2013918.","journal-title":"Eur J Health Econ"},{"issue":"3","key":"9550_CR3","doi-asserted-by":"publisher","first-page":"215","DOI":"10.1016\/j.healthpol.2016.08.010","volume":"121","author":"S Fuchs","year":"2017","unstructured":"Fuchs S, Olberg B, Panteli D, Perleth M, Busse R. HTA of medical devices: Challenges and ideas for the future from a European perspective. Health Policy. 2017;121(3):215\u201329.","journal-title":"Health Policy"},{"key":"9550_CR4","doi-asserted-by":"publisher","first-page":"278","DOI":"10.3389\/fmed.2019.00278","volume":"6","author":"M Bl\u00fcher","year":"2019","unstructured":"Bl\u00fcher M, Saunders SJ, Mittard V, Torrejon Torres R, Davis JA, Saunders R. Critical Review of European Health-Economic Guidelines for the Health Technology Assessment of Medical Devices. Front Med. 2019;6:278.","journal-title":"Front Med"},{"issue":"6","key":"9550_CR5","doi-asserted-by":"publisher","first-page":"711","DOI":"10.1586\/erp.12.73","volume":"12","author":"JS Rotter","year":"2012","unstructured":"Rotter JS, Foerster D, Bridges JF. The changing role of economic evaluation in valuing medical technologies. Expert Rev Pharmacoecon Outcomes Res. 2012;12(6):711\u201323.","journal-title":"Expert Rev Pharmacoecon Outcomes Res"},{"key":"9550_CR6","volume-title":"Methods for the Economic Evaluation of Health Care Programmes","author":"MF Drummond","year":"2015","unstructured":"Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the Economic Evaluation of Health Care Programmes. 4th ed. Oxford: Oxford University Press; 2015.","edition":"4"},{"issue":"3","key":"9550_CR7","doi-asserted-by":"publisher","first-page":"154","DOI":"10.1017\/S0266462315000185","volume":"31","author":"O Ciani","year":"2015","unstructured":"Ciani O, Wilcher B, Blankart CR, Hatz M, Rupel VP, Erker RS, Varabyova Y, Taylor RS. Health technology assessment of medical devices: a survey of non-European union agencies. Int J Technol Assess Health Care. 2015;31(3):154\u201365.","journal-title":"Int J Technol Assess Health Care"},{"issue":"10","key":"9550_CR8","doi-asserted-by":"publisher","first-page":"993","DOI":"10.1080\/17434440.2020.1825074","volume":"17","author":"R Tarricone","year":"2020","unstructured":"Tarricone R, Ciani O, Torbica A, Brouwer W, Chaloutsos G, Drummond MF, Martelli N, Persson U, Leidl R, Levin L, et al. Lifecycle evidence requirements for high-risk implantable medical devices: a European perspective. Expert Rev Med Devices. 2020;17(10):993\u20131006.","journal-title":"Expert Rev Med Devices"},{"issue":"S1","key":"9550_CR9","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1002\/hec.3468","volume":"26","author":"R Tarricone","year":"2017","unstructured":"Tarricone R, Torbica A, Drummond M. Key Recommendations from the MedtecHTA Project. Health Econ. 2017;26(S1):145\u201352.","journal-title":"Health Econ"},{"issue":"4","key":"9550_CR10","doi-asserted-by":"publisher","first-page":"402","DOI":"10.1111\/j.1524-4733.2008.00476_1.x","volume":"12","author":"M Drummond","year":"2009","unstructured":"Drummond M, Griffin A, Tarricone R. Economic evaluation for devices and drugs\u2013same or different? Value Health. 2009;12(4):402\u20134.","journal-title":"Value Health"},{"key":"9550_CR11","volume-title":"Health technology assessment of medical devices","author":"World Health Organization","year":"2011","unstructured":"World Health Organization. Health technology assessment of medical devices. Geneva: World Health Organization; 2011."},{"key":"9550_CR12","unstructured":"WHO. Medical devices. [https:\/\/www.who.int\/health-topics\/medical-devices#tab=tab_1]. Accessed 08 June 2022."},{"key":"9550_CR13","unstructured":"Regulation (EU) 2017\/745 of the European parliament and of the council of 5 April 2017 on medical devices, amending directive 2001\/83\/EC, regulation (EC) no 178\/2002 and regulation (EC) No 1223\/2009 and repealing council directives 90\/385\/EEC and 93\/42\/EEC [2017] OJ L117\/1."},{"key":"9550_CR14","unstructured":"Regulation (EU) 2017\/746 of the European parliament and of the council of 5 April 2017 on in vitro diagnostic medical devices and repealing directive 98\/79\/EC and commission decision 2010\/227\/EU [2017] OJ L117\/176."},{"key":"9550_CR15","unstructured":"Regulation (EU) 2021\/2282 of the European parliament and of the council of 15 december 2021 on health technology assessment and amending directive 2011\/24\/EU [2021] OJ L458\/1."},{"issue":"4","key":"9550_CR16","doi-asserted-by":"publisher","first-page":"309","DOI":"10.1016\/j.zefq.2015.06.011","volume":"109","author":"P Schnell-Inderst","year":"2015","unstructured":"Schnell-Inderst P, Mayer J, Lauterberg J, Hunger T, Arvandi M, Conrads-Frank A, Nachtnebel A, Wild C, Siebert U. Health technology assessment of medical devices: What is different? An overview of three European projects. Z Evid Fortbild Qual Gesundhwes. 2015;109(4):309\u201318.","journal-title":"Z Evid Fortbild Qual Gesundhwes"},{"issue":"4","key":"9550_CR17","doi-asserted-by":"publisher","first-page":"367","DOI":"10.1007\/s40273-022-01130-0","volume":"40","author":"M Gomes","year":"2022","unstructured":"Gomes M, Murray E, Raftery J. Economic Evaluation of Digital Health Interventions: Methodological Issues and Recommendations for Practice. Pharmacoeconomics. 2022;40(4):367\u201378.","journal-title":"Pharmacoeconomics"},{"issue":"Suppl 1","key":"9550_CR18","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1002\/hec.3479","volume":"26","author":"O Ciani","year":"2017","unstructured":"Ciani O, Wilcher B, van Giessen A, Taylor RS. Linking the Regulatory and Reimbursement Processes for Medical Devices: The Need for Integrated Assessments. Health Econ. 2017;26(Suppl 1):13\u201329.","journal-title":"Health Econ"},{"issue":"2","key":"9550_CR19","doi-asserted-by":"publisher","first-page":"256","DOI":"10.1016\/j.jval.2016.11.019","volume":"20","author":"R Baltussen","year":"2017","unstructured":"Baltussen R, Jansen MPM, Bijlmakers L, Grutters J, Kluytmans A, Reuzel RP, Tummers M, van der Wilt GJ. Value assessment frameworks for HTA agencies: The organization of evidence-informed deliberative processes. Value Health. 2017;20(2):256\u201360.","journal-title":"Value Health"},{"issue":"10","key":"9550_CR20","doi-asserted-by":"publisher","first-page":"1055","DOI":"10.1007\/s40273-020-00935-1","volume":"38","author":"B Grigore","year":"2020","unstructured":"Grigore B, Ciani O, Dams F, Federici C, de Groot S, M\u00f6llenkamp M, Rabbe S, Shatrov K, Zemplenyi A, Taylor RS. Surrogate endpoints in health technology assessment: an international review of methodological guidelines. Pharmacoeconomics. 2020;38(10):1055\u201370.","journal-title":"Pharmacoeconomics"},{"issue":"5","key":"9550_CR21","doi-asserted-by":"publisher","first-page":"602","DOI":"10.1016\/j.healthpol.2021.03.003","volume":"125","author":"R Tarricone","year":"2021","unstructured":"Tarricone R, Amatucci F, Armeni P, Banks H, Borsoi L, Callea G, Ciani O, Costa F, Federici C, Torbica A, et al. Establishing a national HTA program for medical devices in Italy: Overhauling a fragmented system to ensure value and equal access to new medical technologies. Health Policy. 2021;125(5):602\u20138.","journal-title":"Health Policy"},{"issue":"2","key":"9550_CR22","doi-asserted-by":"publisher","first-page":"302","DOI":"10.1016\/j.jval.2021.07.005","volume":"25","author":"M Zhang","year":"2022","unstructured":"Zhang M, Bao Y, Lang Y, Fu S, Kimber M, Levine M, Xie F. What Is Value in Health and Healthcare? A Systematic Literature Review of Value Assessment Frameworks. Value Health. 2022;25(2):302\u201317.","journal-title":"Value Health"},{"issue":"2","key":"9550_CR23","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1007\/s40258-015-0200-7","volume":"14","author":"GA van Voorn","year":"2016","unstructured":"van Voorn GA, Vemer P, Hamerlijnck D, Ramos IC, Teunissen GJ, Al M, Feenstra TL. The missing stakeholder group: why patients should be involved in health economic modelling. Appl Health Econ Health Policy. 2016;14(2):129\u201333.","journal-title":"Appl Health Econ Health Policy"},{"issue":"5","key":"9550_CR24","doi-asserted-by":"publisher","first-page":"435","DOI":"10.1007\/s40273-015-0370-z","volume":"34","author":"A Angelis","year":"2016","unstructured":"Angelis A, Kanavos P. Value-based assessment of new medical technologies: towards a robust methodological framework for the application of multiple criteria decision analysis in the context of health technology assessment. Pharmacoeconomics. 2016;34(5):435\u201346.","journal-title":"Pharmacoeconomics"},{"key":"9550_CR25","doi-asserted-by":"publisher","first-page":"S1","DOI":"10.1111\/j.1524-4733.2009.00550.x","volume":"12","author":"JC O'Donnell","year":"2009","unstructured":"O\u2019Donnell JC, Pham SV, Pashos CL, Miller DW, Smith MD. Health technology assessment: lessons learned from around the world - an overview. Value Health. 2009;12:S1\u20135.","journal-title":"Value Health"},{"issue":"1","key":"9550_CR26","first-page":"6","volume":"2","author":"D Mueller","year":"2017","unstructured":"Mueller D, Tivey D, Croce D. Health-technology assessment: Its role in strengthening health systems in developing countries. South Afr J Public Health. 2017;2(1):6\u201311.","journal-title":"South Afr J Public Health"},{"key":"9550_CR27","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1186\/s12962-021-00261-z","volume":"19","author":"D Mueller","year":"2021","unstructured":"Mueller D, Pattinson RC, Hlongwane TM, Busse R, Panteli D. Portable continuous wave Doppler ultrasound for primary healthcare in South Africa: can the EUnetHTA Core Model guide evaluation before technology adoption? Cost Eff Resour Alloc. 2021;19:8.","journal-title":"Cost Eff Resour Alloc"},{"key":"9550_CR28","doi-asserted-by":"crossref","unstructured":"Domingo C, Fernandez M, Garin N, Milara J, Moran I, Muerza I, Pacheco A, Teruel C, Bentley R, Subiran R et al. Determining what represents value in the treatment of refractory or unexplained chronic cough from the perspective of key stakeholders in Spain using multi-criteria decision analysis. Appl Health Econ Health Policy. 2023;21(1):119\u2013310.","DOI":"10.1007\/s40258-022-00770-9"},{"issue":"1","key":"9550_CR29","doi-asserted-by":"publisher","first-page":"276","DOI":"10.1186\/s12913-022-07657-4","volume":"22","author":"R Ruseckaite","year":"2022","unstructured":"Ruseckaite R, Maharaj AD, Dean J, Krysinska K, Ackerman IN, Brennan AL, Busija L, Carter H, Earnest A. Forrest CB et al: Preliminary development of recommendations for the inclusion of patient-reported outcome measures in clinical quality registries. BMC Health Serv Res. 2022;22(1):276.","journal-title":"BMC Health Serv Res"},{"issue":"1","key":"9550_CR30","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1007\/s40258-021-00683-z","volume":"20","author":"E Bell","year":"2022","unstructured":"Bell E, Neri M, Steuten L. Towards a broader assessment of value in vaccines: the BRAVE way forward. Appl Health Econ Health Policy. 2022;20(1):105\u201317.","journal-title":"Appl Health Econ Health Policy"},{"issue":"1","key":"9550_CR31","doi-asserted-by":"publisher","first-page":"475","DOI":"10.1016\/j.ejor.2022.05.037","volume":"305","author":"CA Bana e Costa","year":"2023","unstructured":"Bana e Costa CA, Oliveira MD, Vieira ACL, Freitas L, Rodrigues TC, Bana e Costa J, Freitas \u00c2, Santana P. Collaborative development of composite indices from qualitative value judgements: The EURO-HEALTHY Population Health Index model. Eur J Oper Res. 2023;305(1):475\u201392.","journal-title":"Eur J Oper Res"},{"key":"9550_CR32","doi-asserted-by":"publisher","first-page":"102047","DOI":"10.1016\/j.omega.2019.03.005","volume":"94","author":"ACL Vieira","year":"2020","unstructured":"Vieira ACL, Oliveira MD, Bana e Costa CA. Enhancing knowledge construction processes within multicriteria decision analysis: The collaborative value modelling framework. Omega. 2020;94:102047.","journal-title":"Omega"},{"issue":"9","key":"9550_CR33","doi-asserted-by":"publisher","first-page":"1712","DOI":"10.1016\/j.techfore.2010.09.011","volume":"78","author":"HA Linstone","year":"2011","unstructured":"Linstone HA, Turoff M. Delphi: A brief look backward and forward. Technol Forecast Soc Change. 2011;78(9):1712\u20139.","journal-title":"Technol Forecast Soc Change"},{"key":"9550_CR34","unstructured":"MEDI-VALUE - Developing HTA tools to consensualise MEDIcal devices\u2019 VALUE through multicriteria decision analysis. [https:\/\/medivalue.tecnico.ulisboa.pt\/]. Accessed 11 Feb 2020."},{"key":"9550_CR35","volume-title":"The Delphi Method: Techniques and Applications","author":"HA Linstone","year":"2002","unstructured":"Linstone HA, Turoff A. The Delphi Method: Techniques and Applications. Addison-Wesley Publishing Company, Advanced Book Program; 2002."},{"key":"9550_CR36","unstructured":"Decision Eyes. Welphi. [https:\/\/www.welphi.com\/en\/Home.html]. Accessed 09 Mar 2022."},{"issue":"3","key":"9550_CR37","doi-asserted-by":"publisher","first-page":"227","DOI":"10.1016\/S0305-0483(02)00029-4","volume":"30","author":"CA Bana e Costa","year":"2002","unstructured":"Bana e Costa CA, Corr\u00eaa EC, De Corte JM, Vansnick JC. Facilitating bid evaluation in public call for tenders: a socio-technical approach. Omega. 2002;30(3):227\u201342.","journal-title":"Omega"},{"issue":"2","key":"9550_CR38","doi-asserted-by":"publisher","first-page":"604","DOI":"10.1016\/j.ejor.2019.02.039","volume":"277","author":"M Marttunen","year":"2019","unstructured":"Marttunen M, Haag F, Belton V, Mustajoki J, Lienert J. Methods to inform the development of concise objectives hierarchies in multi-criteria decision analysis. Eur J Oper Res. 2019;277(2):604\u201320.","journal-title":"Eur J Oper Res"},{"key":"9550_CR39","doi-asserted-by":"publisher","unstructured":"Oliveira MD, Vieira ACL, Dimitrovova K, Angelis A, Kanavos P, Bana e Costa CA. Multi-criteria evaluation framework - Advancing knowledge and MCDA tools to assist HTA agencies in evaluating medicines on a common basis, Report 7.2 of IMPACT HTA. 2021. https:\/\/doi.org\/10.5281\/zenodo.7471696.","DOI":"10.5281\/zenodo.7471696"},{"key":"9550_CR40","volume-title":"Discovering Statistics Using R","author":"A Field","year":"2012","unstructured":"Field A, Miles J, Field Z. Discovering Statistics Using R. SAGE Publications; 2012."},{"key":"9550_CR41","volume-title":"Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters","author":"KL Gwet","year":"2014","unstructured":"Gwet KL. Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. 4th ed. USA: Advanced Analytics, LLC; 2014.","edition":"4"},{"key":"9550_CR42","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1186\/1471-2288-13-61","volume":"13","author":"N Wongpakaran","year":"2013","unstructured":"Wongpakaran N, Wongpakaran T, Wedding D, Gwet KL. A comparison of Cohen\u2019s Kappa and Gwet\u2019s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Med Res Methodol. 2013;13:61.","journal-title":"BMC Med Res Methodol"},{"key":"9550_CR43","doi-asserted-by":"publisher","DOI":"10.1093\/med\/9780199685219.001.0001","volume-title":"Health Measurement Scales: A Practical Guide to Their Development and Use","author":"DL Streiner","year":"2015","unstructured":"Streiner DL, Norman GR, Cairney J. Health Measurement Scales: A Practical Guide to Their Development and Use. 5th ed. Oxford: Oxford University Press; 2015.","edition":"5"},{"issue":"1","key":"9550_CR44","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","volume":"33","author":"JR Landis","year":"1977","unstructured":"Landis JR, Koch GG. The Measurement of Observer Agreement for Categorical Data. Biometrics. 1977;33(1):159\u201374.","journal-title":"Biometrics"},{"issue":"4","key":"9550_CR45","first-page":"871","volume":"18","author":"D Klein","year":"2018","unstructured":"Klein D. Implementing a general framework for assessing interrater agreement in Stata. Stand Genomic Sci. 2018;18(4):871\u2013901.","journal-title":"Stand Genomic Sci"},{"key":"9550_CR46","doi-asserted-by":"publisher","first-page":"413","DOI":"10.1007\/s10198-022-01482-6","volume":"24","author":"DJ Mott","year":"2023","unstructured":"Mott DJ, Ternent L, Vale L. Do preferences differ based on respondent experience of a health issue and its treatment? A case study using a public health intervention. Eur J Health Econ. 2023;24:413\u201323.","journal-title":"Eur J Health Econ."},{"issue":"7","key":"9550_CR47","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0219905","volume":"14","author":"M Trapero-Bertran","year":"2019","unstructured":"Trapero-Bertran M, Rodr\u00edguez-Mart\u00edn B, L\u00f3pez-Bastida J. What attributes should be included in a discrete choice experiment related to health technologies? A systematic literature review. PLoS ONE. 2019;14(7): e0219905.","journal-title":"PLoS ONE"},{"issue":"3","key":"9550_CR48","doi-asserted-by":"publisher","first-page":"034701","DOI":"10.1115\/1.4053928","volume":"16","author":"NC Onwudiwe","year":"2022","unstructured":"Onwudiwe NC, Charter R, Gingles B, Abrishami P, Alder H, Bahkai A, Civic D, Klie\u00df MK, Lessard C, Zema CL. Generating Appropriate and Reliable Evidence for Value Assessment of Medical Devices: An ISPOR Medical Devices and Diagnostics Special Interest Group Report. J Med Device. 2022;16(3):034701\u201310.","journal-title":"J Med Device"},{"issue":"2","key":"9550_CR49","doi-asserted-by":"publisher","first-page":"244","DOI":"10.1016\/j.jval.2016.12.010","volume":"20","author":"FB Kristensen","year":"2017","unstructured":"Kristensen FB, Lampe K, Wild C, Cerbo M, Goettsch W, Becla L. The HTA Core Model\u00ae\u201410 Years of Developing an International Framework to Share Multidimensional Value Assessment. Value Health. 2017;20(2):244\u201350.","journal-title":"Value Health"},{"issue":"2","key":"9550_CR50","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1016\/j.healthpol.2018.12.003","volume":"123","author":"O Efthymiadou","year":"2019","unstructured":"Efthymiadou O, Mossman J, Kanavos P. Health related quality of life aspects not captured by EQ-5D-5L: Results from an international survey of patients. Health Policy. 2019;123(2):159\u201365.","journal-title":"Health Policy"},{"key":"9550_CR51","volume-title":"How can the impact of health technology assessments be enhanced?, vol. EUR\/07\/5065810","author":"C Sorenson","year":"2008","unstructured":"Sorenson C, Drummond M, Kristensen FB, Busse R. How can the impact of health technology assessments be enhanced?, vol. EUR\/07\/5065810. Copenhagen: WHO Regional Office for Europe; 2008."},{"issue":"6","key":"9550_CR52","doi-asserted-by":"publisher","first-page":"913","DOI":"10.1007\/s10198-022-01458-6","volume":"23","author":"M Drummond","year":"2022","unstructured":"Drummond M, Tarricone R, Torbica A. European union regulation of health technology assessment: what is required for it to succeed? Eur J Health Econ. 2022;23(6):913\u20135.","journal-title":"Eur J Health Econ"},{"issue":"12","key":"9550_CR53","doi-asserted-by":"publisher","first-page":"1417","DOI":"10.1007\/s40273-019-00846-w","volume":"37","author":"S Ghabri","year":"2019","unstructured":"Ghabri S, Josselin J-M, Le Maux B. Could or Should We Use MCDA in the French HTA Process? Pharmacoeconomics. 2019;37(12):1417\u20139.","journal-title":"Pharmacoeconomics"},{"key":"9550_CR54","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1016\/j.techfore.2019.07.002","volume":"147","author":"I Belton","year":"2019","unstructured":"Belton I, MacDonald A, Wright G, Hamlin I. Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process. Technol Forecast Soc Change. 2019;147:72\u201382.","journal-title":"Technol Forecast Soc Change"},{"key":"9550_CR55","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1186\/1471-2288-5-37","volume":"5","author":"RB Akins","year":"2005","unstructured":"Akins RB, Tolson H, Cole BR. Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Med Res Methodol. 2005;5:37.","journal-title":"BMC Med Res Methodol"},{"key":"9550_CR56","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1016\/j.techfore.2016.05.020","volume":"109","author":"M Makkonen","year":"2016","unstructured":"Makkonen M, Hujala T, Uusivuori J. Policy experts\u2019 propensity to change their opinion along Delphi rounds. Technol Forecast Soc Change. 2016;109:61\u20138.","journal-title":"Technol Forecast Soc Change"},{"key":"9550_CR57","doi-asserted-by":"publisher","first-page":"120897","DOI":"10.1016\/j.techfore.2021.120897","volume":"170","author":"I Belton","year":"2021","unstructured":"Belton I, Wright G, Sissons A, Bolger F, Crawford MM, Hamlin I, Taylor Browne L\u016bka C, Vasilichi A. Delphi with feedback of rationales: How large can a Delphi group be such that participants are not overloaded, de-motivated, or disengaged? Technol Forecast Soc Change. 2021;170:120897.","journal-title":"Technol Forecast Soc Change"}],"container-title":["BMC Health Services Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12913-023-09550-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12913-023-09550-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12913-023-09550-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,8]],"date-time":"2023-06-08T08:04:36Z","timestamp":1686211476000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmchealthservres.biomedcentral.com\/articles\/10.1186\/s12913-023-09550-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,8]]},"references-count":57,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["9550"],"URL":"https:\/\/doi.org\/10.1186\/s12913-023-09550-0","relation":{},"ISSN":["1472-6963"],"issn-type":[{"value":"1472-6963","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,8]]},"assertion":[{"value":"15 December 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 May 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 June 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"All methods were carried out in accordance with relevant guidelines and regulations. This study was approved by the Ethics Committee of Instituto Superior T\u00e9cnico, University of Lisbon (reference no. 14\/2019 (CE-IST)). Informed consent was obtained from all individual participants in the study.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"593"}}