{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,3]],"date-time":"2026-10-03T01:52:53Z","timestamp":1790992373417,"version":"4.1.0"},"reference-count":19,"publisher":"Wiley","license":[{"start":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T00:00:00Z","timestamp":1790899200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"},{"start":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T00:00:00Z","timestamp":1790899200000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/doi.wiley.com\/10.1002\/tdm_license_1.1"}],"content-domain":{"domain":["asmepublications.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Medical Education"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Assessment in health professions education has evolved from individual assessment instruments towards integrated systems of assessment that support learning and high\u2010stakes decision\u2010making. While considerable attention has been devoted to the design, validity and reliability of individual assessment methods, far less attention has been paid to evaluating assessment systems themselves. This is an important gap, as assessment systems may contain multiple high\u2010quality assessment tools yet still fail to function as coherent and effective wholes.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Purpose<\/jats:title>\n                    <jats:p>This paper proposes a conceptual framework for evaluating systems of assessment in health professions education. Drawing on systems thinking and complexity theory, we argue that contemporary assessment systems, such as programmatic assessment, can be understood as complex adaptive systems and therefore require evaluation approaches that extend beyond the evaluation of individual assessment components.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Approach<\/jats:title>\n                    <jats:p>We trace the evolution of assessment thinking\u2014from individual assessments to systems of assessment\u2014and consider these systems through the lens of complex adaptive systems. We then combine the Logic Model and the Context\u2013Input\u2013Process\u2013Product (CIPP) evaluation framework to connect programme theory with evaluation and examine both intended and unintended consequences.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Framework<\/jats:title>\n                    <jats:p>The integrated framework comprises six components: Planning, Resource Design, Implementation, Short\u2010term Results, Outcomes and Long\u2010term Impact. Together, these components provide a structure for evaluating how assessment systems are designed, implemented, experienced and improved over time. A worked example illustrates how system\u2010level evaluation identifies issues that are not apparent when individual assessment tools are evaluated in isolation.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Evaluating assessment systems requires approaches that examine interactions, implementation processes, educational impact and broader consequences. The proposed framework offers a structured theory\u2010informed approach to support continuous improvement, quality assurance and the development of validity arguments for contemporary systems of assessment. Future research should explore the framework's feasibility, utility and impact across diverse health professions education contexts.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1111\/medu.70329","type":"journal-article","created":{"date-parts":[[2026,10,3]],"date-time":"2026-10-03T01:01:34Z","timestamp":1790989294000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["The whole is greater than the sum of its parts: Evaluating systems of assessment as complex adaptive systems"],"prefix":"10.1111","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6912-5886","authenticated-orcid":false,"given":"Phil","family":"Hider","sequence":"first","affiliation":[{"name":"Faculty of Medicine University of Otago  Christchurch New Zealand"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4924-4888","authenticated-orcid":false,"given":"Dario","family":"Torre","sequence":"additional","affiliation":[{"name":"University of Turin Medical School  Turin Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6103-8075","authenticated-orcid":false,"given":"Sylvia","family":"Heeneman","sequence":"additional","affiliation":[{"name":"Department of Pathology, Faculty of Health, Medicine and Health Sciences, School of Health Professions Education Maastricht University  Maastricht The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4080-4164","authenticated-orcid":false,"given":"Tim J.","family":"Wilkinson","sequence":"additional","affiliation":[{"name":"Faculty of Medicine University of Otago  Christchurch New Zealand"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2026,10,2]]},"reference":[{"key":"e_1_2_14_2_1","doi-asserted-by":"publisher","DOI":"10.3109\/0142159X.2011.551559"},{"key":"e_1_2_14_3_1","volume-title":"Assessing Health Professional Education: Workshop Summary","author":"Global Forum on Innovation in Health Professional Education","year":"2014"},{"key":"e_1_2_14_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10459-020-10003-0"},{"key":"e_1_2_14_5_1","doi-asserted-by":"publisher","DOI":"10.1080\/0142159X.2018.1500016"},{"key":"e_1_2_14_6_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-3992.1998.tb00616.x"},{"key":"e_1_2_14_7_1","doi-asserted-by":"publisher","DOI":"10.1097\/ACM.0000000000005251"},{"key":"e_1_2_14_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/S40037-018-0485-Y"},{"key":"e_1_2_14_9_1","doi-asserted-by":"publisher","DOI":"10.1097\/ACM.0000000000001114"},{"issue":"1","key":"e_1_2_14_10_1","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1111\/j.1745-3992.2002.tb00082.x","article-title":"Strategies for examining the conequences of assessment and acoountability programs","volume":"21","author":"Lane S","year":"2002","journal-title":"Educ Meas Issues Pract"},{"issue":"2","key":"e_1_2_14_11_1","doi-asserted-by":"crossref","first-page":"112","DOI":"10.3102\/0013189X19826958","article-title":"Education as a complex system: conceptual and methodological implications","volume":"48","author":"Jacobson M","year":"2019","journal-title":"Educ Res"},{"key":"e_1_2_14_12_1","volume-title":"The Fifth Discipline: The Art and Practice of the Learning Organization","author":"Senge PM","year":"1992"},{"key":"e_1_2_14_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11424-006-0001-z"},{"key":"e_1_2_14_14_1","volume-title":"Logic Model Development Guide","author":"Foundation. WKK","year":"2004"},{"key":"e_1_2_14_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-010-0309-4_4"},{"key":"e_1_2_14_16_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-3992.1983.tb00688.x"},{"key":"e_1_2_14_17_1","volume-title":"Clinical Education for the Health Professions","author":"Lindemann I","year":"2023"},{"key":"e_1_2_14_18_1","doi-asserted-by":"publisher","DOI":"10.1046\/j.1365-2923.2003.01594.x"},{"key":"e_1_2_14_19_1","volume-title":"Thinking in Systems: A Primer","author":"Meadows DH","year":"2008"},{"key":"e_1_2_14_20_1","doi-asserted-by":"publisher","DOI":"10.1111\/jedm.12007"}],"container-title":["Medical Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/asmepublications.onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/medu.70329","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/asmepublications.onlinelibrary.wiley.com\/doi\/full-xml\/10.1111\/medu.70329","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/asmepublications.onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/medu.70329","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,3]],"date-time":"2026-10-03T01:01:35Z","timestamp":1790989295000},"score":1,"resource":{"primary":{"URL":"https:\/\/asmepublications.onlinelibrary.wiley.com\/doi\/10.1111\/medu.70329"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,2]]},"references-count":19,"alternative-id":["10.1111\/medu.70329"],"URL":"https:\/\/doi.org\/10.1111\/medu.70329","archive":["Portico"],"relation":{},"ISSN":["0308-0110","1365-2923"],"issn-type":[{"value":"0308-0110","type":"print"},{"value":"1365-2923","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,2]]},"assertion":[{"value":"2026-02-16","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-20","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-10-02","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"medu.70329"}}