{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T10:18:43Z","timestamp":1777889923838,"version":"3.51.4"},"reference-count":49,"publisher":"SAGE Publications","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["SW"],"published-print":{"date-parts":[[2021,3,9]]},"abstract":"<jats:p>Successful exams require a balance of easy, medium, and difficult questions. Question difficulty is generally either estimated by an expert or determined after an exam is taken. The latter provides no utility for the generation of new questions and the former is expensive both in terms of time and cost. Additionally, it is not known whether expert prediction is indeed a good proxy for estimating question difficulty. In this paper, we analyse and compare two ontology-based measures for difficulty prediction of multiple choice questions, as well as comparing each measure with expert prediction (by 15 experts) against the exam performance of 12 residents over a corpus of 231 medical case-based questions that are in multiple choice format. We find one ontology-based measure (relation strength indicativeness) to be of comparable performance (accuracy\u00a0= 47%) to expert prediction (average accuracy\u00a0= 49%).<\/jats:p>","DOI":"10.3233\/sw-200390","type":"journal-article","created":{"date-parts":[[2020,10,20]],"date-time":"2020-10-20T11:03:35Z","timestamp":1603191815000},"page":"449-465","source":"Crossref","is-referenced-by-count":5,"title":["A comparative study of methods for a priori prediction of MCQ difficulty"],"prefix":"10.1177","volume":"12","author":[{"given":"Ghader","family":"Kurdi","sequence":"first","affiliation":[{"name":"Department of Computer Science, The University of Manchester, Oxford Road, Manchester, M139PL, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jared","family":"Leo","sequence":"additional","affiliation":[{"name":"Department of Computer Science, The University of Manchester, Oxford Road, Manchester, M139PL, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nicolas","family":"Matentzoglu","sequence":"additional","affiliation":[{"name":"Department of Computer Science, The University of Manchester, Oxford Road, Manchester, M139PL, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bijan","family":"Parsia","sequence":"additional","affiliation":[{"name":"Department of Computer Science, The University of Manchester, Oxford Road, Manchester, M139PL, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Uli","family":"Sattler","sequence":"additional","affiliation":[{"name":"Department of Computer Science, The University of Manchester, Oxford Road, Manchester, M139PL, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sophie","family":"Forge","sequence":"additional","affiliation":[{"name":"Elsevier, 1600 John F. Kennedy Boulevard, Philadelphia, PA 19103, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gina","family":"Donato","sequence":"additional","affiliation":[{"name":"Elsevier, 1600 John F. Kennedy Boulevard, Philadelphia, PA 19103, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Will","family":"Dowling","sequence":"additional","affiliation":[{"name":"Elsevier, 1600 John F. Kennedy Boulevard, Philadelphia, PA 19103, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","reference":[{"issue":"3","key":"10.3233\/SW-200390_ref2","first-page":"26","article-title":"Automatic generation of multiple choice questions using surface-based semantic relations","volume":"6","author":"Afzal","year":"2015","journal-title":"International Journal of Computational Linguistics (IJCL)"},{"key":"10.3233\/SW-200390_ref3","doi-asserted-by":"crossref","unstructured":"M.\u00a0Al-Yahya, Ontology-based multiple choice question generation, The Scientific World Journal 2014 (2014).","DOI":"10.1155\/2014\/274949"},{"key":"10.3233\/SW-200390_ref5","doi-asserted-by":"crossref","unstructured":"T.\u00a0Alsubait, B.\u00a0Parsia and U.\u00a0Sattler, Generating multiple choice questions from ontologies: Lessons learnt, in: OWLED, 2014, pp.\u00a073\u201384.","DOI":"10.1007\/978-3-319-17966-7_7"},{"key":"10.3233\/SW-200390_ref6","doi-asserted-by":"publisher","DOI":"10.1109\/FIE.2015.7344299"},{"key":"10.3233\/SW-200390_ref7","unstructured":"B.S.\u00a0Bloom, M.D.\u00a0Engelhart, E.J.\u00a0Furst, W.H.\u00a0Hill and D.R.\u00a0Krathwohl, Taxonomy of Educational Objectives, Handbook I: The Cognitive Domain, Vol.\u00a019, David McKay Co Inc, New York, 1956."},{"issue":"2","key":"10.3233\/SW-200390_ref9","doi-asserted-by":"publisher","first-page":"543","DOI":"10.1148\/rg.262055145","article-title":"Writing multiple-choice questions for continuing medical education activities and self-assessment modules","volume":"26","author":"Collins","year":"2006","journal-title":"RadioGraphics"},{"issue":"1","key":"10.3233\/SW-200390_ref10","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1016\/S1322-7696(08)60478-3","article-title":"Design, format, validity and reliability of multiple choice questions for use in nursing research and education","volume":"12","author":"Considine","year":"2005","journal-title":"Collegian"},{"issue":"3","key":"10.3233\/SW-200390_ref11","doi-asserted-by":"publisher","first-page":"346","DOI":"10.1080\/02671522.2012.673005","article-title":"Modelling question difficulty in an a level physics examination","volume":"28","author":"Crisp","year":"2013","journal-title":"Research Papers in Education"},{"key":"10.3233\/SW-200390_ref12","unstructured":"L.\u00a0Crocker and J.\u00a0Algina, Introduction to Classical and Modern Test Theory, ERIC, 1986."},{"key":"10.3233\/SW-200390_ref13","unstructured":"M.\u00a0Cubric and M.\u00a0Tosic, Towards automatic generation of e-assessment using semantic web technologies, International Journal of e-Assessment 1(1) (2011)."},{"key":"10.3233\/SW-200390_ref14","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-011-4886-3_40"},{"issue":"3","key":"10.3233\/SW-200390_ref15","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1023\/A:1021112514626","article-title":"Threats to the validity of locally developed multiple-choice tests in medical education: Construct-irrelevant variance and construct underrepresentation","volume":"7","author":"Downing","year":"2002","journal-title":"Advances in Health Sciences Education"},{"key":"10.3233\/SW-200390_ref16","unstructured":"M.K.\u00a0Enright and K.\u00a0Sheehan, Modeling the difficulty of quantitative reasoning items: Implications for item generation, in: Item Generation for Test Development, S.H.\u00a0Irvine and P.C.\u00a0Kyllonen, eds, Routledge, 2002, pp.\u00a0129\u2013157."},{"key":"10.3233\/SW-200390_ref17","doi-asserted-by":"publisher","DOI":"10.1186\/1472-6920-14-232"},{"key":"10.3233\/SW-200390_ref18","unstructured":"K.\u00a0Gautam, I.\u00a0Gupta and K.\u00a0Chandramouli, Conceptual extraction of questions from Wikipedia, in: ET LACNEM, 2013."},{"issue":"8","key":"10.3233\/SW-200390_ref20","doi-asserted-by":"publisher","first-page":"490","DOI":"10.3928\/01484834-20170712-08","article-title":"Flaws of multiple choice questions in teacher-constructed nursing examinations: A pilot descriptive study","volume":"56","author":"Hijji","year":"2017","journal-title":"Journal of Nursing Education"},{"issue":"2","key":"10.3233\/SW-200390_ref21","first-page":"142","article-title":"Analysis of one-best MCQs: The difficulty index, discrimination index and distractor efficiency","volume":"62","author":"Hingorjo","year":"2012","journal-title":"JPMA-Journal of the Pakistan Medical Association"},{"issue":"1","key":"10.3233\/SW-200390_ref22","doi-asserted-by":"publisher","first-page":"11","DOI":"10.3233\/SW-2011-0025","article-title":"The OWL API: A Java API for OWL ontologies","volume":"2","author":"Horridge","year":"2011","journal-title":"Semantic Web"},{"key":"10.3233\/SW-200390_ref23","doi-asserted-by":"publisher","DOI":"10.1109\/SWSTE.2014.16"},{"key":"10.3233\/SW-200390_ref24","first-page":"547","article-title":"\u00c9tude comparative de la distribution florale dans une portion des Alpes et des Jura","volume":"37","author":"Jaccard","year":"1901","journal-title":"Bull Soc Vaudoise Sci Nat"},{"issue":"4","key":"10.3233\/SW-200390_ref25","doi-asserted-by":"publisher","first-page":"396","DOI":"10.1152\/advan.00062.2011","article-title":"Are faculty predictions or item taxonomies useful for estimating the outcome of multiple-choice examinations?","volume":"35","author":"Kibble","year":"2011","journal-title":"Advances in Physiology Education"},{"key":"10.3233\/SW-200390_ref26","doi-asserted-by":"publisher","DOI":"10.1109\/CogInfoCom.2013.6719278"},{"key":"10.3233\/SW-200390_ref28","doi-asserted-by":"crossref","unstructured":"G.\u00a0Kurdi, J.\u00a0Leo, B.\u00a0Parsia, U.\u00a0Sattler and S.\u00a0Al-Emari, A systematic review of automatic question generation for educational purposes, International Journal of Artificial Intelligence in Education (2019), in press.","DOI":"10.1007\/s40593-019-00186-y"},{"issue":"1","key":"10.3233\/SW-200390_ref29","first-page":"85","article-title":"Problem complexity: A measure of problem difficulty in algebra by using computer","volume":"28","author":"Lee","year":"2000","journal-title":"Education Journal"},{"key":"10.3233\/SW-200390_ref30","doi-asserted-by":"publisher","DOI":"10.1007\/s40593-018-00172-w"},{"issue":"3","key":"10.3233\/SW-200390_ref31","doi-asserted-by":"publisher","first-page":"251","DOI":"10.1109\/TLT.2012.5","article-title":"Using Wikipedia and conceptual graph structures to generate questions for academic writing support","volume":"5","author":"Liu","year":"2012","journal-title":"IEEE Transactions on Learning Technologies"},{"issue":"8","key":"10.3233\/SW-200390_ref32","doi-asserted-by":"publisher","first-page":"919","DOI":"10.1080\/02602938.2011.586991","article-title":"Peer review improves the quality of MCQ examinations","volume":"37","author":"Malau-Aduli","year":"2012","journal-title":"Assessment & Evaluation in Higher Education"},{"issue":"1","key":"10.3233\/SW-200390_ref33","doi-asserted-by":"publisher","first-page":"25","DOI":"10.3928\/0148-4834-20010101-07","article-title":"Assessment of multiple-choice questions in selected test banks accompanying text books used in nursing education","volume":"40","author":"Masters","year":"2001","journal-title":"Journal of Nursing Education"},{"key":"10.3233\/SW-200390_ref34","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevSTPER.7.010110"},{"key":"10.3233\/SW-200390_ref35","unstructured":"M.\u00a0Mukhopadhyay, K.\u00a0Bhowmick, S.\u00a0Chakraborty, D.\u00a0Roy, P.K.\u00a0Sen and I.\u00a0Chakraborty, Evaluation of MCQs for judgement of higher levels of cognitive learning, Gomal Journal of Medical Sciences 8(2) (2010)."},{"issue":"7","key":"10.3233\/SW-200390_ref36","first-page":"1891","article-title":"Assessment of functional and nonfunctional distracter in an item analysis","volume":"3","author":"Namdeo","year":"2016","journal-title":"International Journal of Contemporary Medical Research"},{"issue":"2","key":"10.3233\/SW-200390_ref37","doi-asserted-by":"publisher","first-page":"52","DOI":"10.1097\/NND.0b013e318286c2f1","article-title":"Assessment of item-writing flaws in multiple-choice questions","volume":"29","author":"Nedeau-Cayo","year":"2013","journal-title":"Journal for Nurses in Professional Development"},{"key":"10.3233\/SW-200390_ref38","doi-asserted-by":"publisher","DOI":"10.1186\/s13104-016-2202-4"},{"key":"10.3233\/SW-200390_ref39","unstructured":"A.\u00a0Papasalouros, K.\u00a0Kanaris and K.\u00a0Kotis, Automatic generation of multiple choice questions from domain ontologies, in: IADIS International Conference E-Learning, 2008, pp.\u00a0427\u2013434."},{"key":"10.3233\/SW-200390_ref40","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-33245-1_7"},{"key":"10.3233\/SW-200390_ref41","doi-asserted-by":"crossref","unstructured":"M.C.\u00a0Rodr\u00edguez-D\u00edez, M.\u00a0Alegre, N.\u00a0D\u00edez, L.\u00a0Arbea and M.\u00a0Ferrer, Technical flaws in multiple-choice questions in the access exam to medical specialties (\u201cexamen MIR\u201d), in: Spain (2009\u20132013), BMC Medical Education, Vol.\u00a016, 2016, p.\u00a047.","DOI":"10.1186\/s12909-016-0559-7"},{"key":"10.3233\/SW-200390_ref42","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-016-0773-3"},{"issue":"11","key":"10.3233\/SW-200390_ref43","first-page":"1103","article-title":"Item analysis of published MCQs","volume":"35","author":"Sarin","year":"1998","journal-title":"Indian pediatrics"},{"issue":"2","key":"10.3233\/SW-200390_ref44","doi-asserted-by":"publisher","first-page":"150","DOI":"10.1046\/j.1365-2923.1998.00193.x","article-title":"An investigation of the difficulty of computer-based case simulations","volume":"32","author":"Scheuneman","year":"1998","journal-title":"Medical Education"},{"issue":"4","key":"10.3233\/SW-200390_ref45","doi-asserted-by":"publisher","first-page":"348","DOI":"10.1046\/j.1365-2923.2001.00771.x","article-title":"Do short cases elicit different thinking processes than factual knowledge questions do?","volume":"35","author":"Schuwirth","year":"2001","journal-title":"Medical Education"},{"key":"10.3233\/SW-200390_ref46","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-45062-4_104"},{"issue":"5","key":"10.3233\/SW-200390_ref47","doi-asserted-by":"publisher","first-page":"721","DOI":"10.1080\/02602938.2016.1164830","article-title":"Assessing scientific reasoning: A comprehensive evaluation of item features that affect item difficulty","volume":"41","author":"Stiller","year":"2016","journal-title":"Assessment & Evaluation in Higher Education"},{"key":"10.3233\/SW-200390_ref48","doi-asserted-by":"publisher","DOI":"10.1186\/1472-6920-9-40"},{"issue":"2","key":"10.3233\/SW-200390_ref49","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1016\/j.edurev.2006.05.001","article-title":"Teachers\u2019 and students\u2019 perceptions of assessments: A review and a study into the ability and accuracy of estimating the difficulty levels of assessment items","volume":"1","author":"van de Watering","year":"2006","journal-title":"Educational Research Review"},{"key":"10.3233\/SW-200390_ref50","doi-asserted-by":"publisher","first-page":"40","DOI":"10.1016\/j.websem.2015.05.005","article-title":"A novel approach to generate MCQs from domain ontology: Considering DL semantics and open-world assumption","volume":"34","author":"Vinu","year":"2015","journal-title":"Web Semantics: Science, Services and Agents on the World Wide Web"},{"key":"10.3233\/SW-200390_ref51","unstructured":"E.V.\u00a0Vinu and P.S.\u00a0Kumar, Difficulty-level modeling of ontology-based factual questions, Semantic Web Journal (2018), in press."},{"issue":"3","key":"10.3233\/SW-200390_ref52","doi-asserted-by":"publisher","first-page":"238","DOI":"10.1080\/01421590802155597","article-title":"Quality assurance of item writing: During the introduction of multiple choice questions in medicine for high stakes examinations","volume":"31","author":"Ware","year":"2009","journal-title":"Medical Teacher"},{"key":"10.3233\/SW-200390_ref53","doi-asserted-by":"publisher","DOI":"10.1002\/0471264385.wei1003"},{"key":"10.3233\/SW-200390_ref54","unstructured":"S.\u00a0Williams, Generating mathematical word problems, in: The Association for the Advancement of Artificial Intelligence AAAI Fall Symposium: Question Generation, 2011, pp.\u00a061\u201364."}],"container-title":["Semantic Web"],"original-title":[],"link":[{"URL":"https:\/\/content.iospress.com\/download?id=10.3233\/SW-200390","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T05:24:58Z","timestamp":1777613098000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/full\/10.3233\/SW-200390"}},"subtitle":[],"editor":[{"given":"Dagmar","family":"Gromann","sequence":"additional","affiliation":[{"name":"University of Vienna, Austria"}],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2021,3,9]]},"references-count":49,"journal-issue":{"issue":"3"},"URL":"https:\/\/doi.org\/10.3233\/sw-200390","relation":{},"ISSN":["2210-4968","1570-0844"],"issn-type":[{"value":"2210-4968","type":"electronic"},{"value":"1570-0844","type":"print"}],"subject":[],"published":{"date-parts":[[2021,3,9]]}}}