{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T12:05:07Z","timestamp":1775131507875,"version":"3.50.1"},"reference-count":60,"publisher":"Emerald","issue":"1","license":[{"start":{"date-parts":[[2019,3,11]],"date-time":"2019-03-11T00:00:00Z","timestamp":1552262400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["ITSE"],"published-print":{"date-parts":[[2019,3,11]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-subheading\">Purpose<\/jats:title><jats:p>This paper aims to examine whether multiple choice questions (MCQs) can be answered correctly without knowing the answer and whether constructed response questions (CRQs) offer more reliable assessment.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Design\/methodology\/approach<\/jats:title><jats:p>The paper presents a critical review of existing research on MCQs, then reports on an experimental study where two objective tests (using MCQs and CRQs) were set for an introductory undergraduate course. To maximise completion, tests were kept short; consequently, differences between individuals\u2019 scores across both tests are examined rather than overall averages and pass rates.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Findings<\/jats:title><jats:p>Most students who excelled in the MCQ test did not do so in the CRQ test. Students could do well without necessarily understanding the principles being tested.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Research limitations\/implications<\/jats:title><jats:p>Conclusions are limited by the small number of questions in each test and by delivery of the tests at different times. This meant that statistical average data would be too coarse to use, and that some students took one test but not the other. Conclusions concerning CRQs are limited to disciplines where numerical answers or short and constrained text answers are appropriate.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Practical implications<\/jats:title><jats:p>MCQs, while useful in formative assessment, are best avoided for summative assessments. Where appropriate, CRQs should be used instead.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Social implications<\/jats:title><jats:p>MCQs are commonplace as summative assessments in education and training. Increasing the use of CRQs in place of MCQs should increase the reliability of tests, including those administered in safety-critical areas.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Originality\/value<\/jats:title><jats:p>While others have recommended that MCQs should not be used (Hinchliffe 2014, Srivastava<jats:italic>et al.<\/jats:italic>, 2004) because they are vulnerable to guessing, this paper presents an experimental study designed to demonstrate whether this hypothesis is correct.<\/jats:p><\/jats:sec>","DOI":"10.1108\/itse-09-2018-0071","type":"journal-article","created":{"date-parts":[[2019,2,6]],"date-time":"2019-02-06T08:57:51Z","timestamp":1549443471000},"page":"59-73","source":"Crossref","is-referenced-by-count":27,"title":["Multiple choice questions: answering correctly and knowing the answer"],"prefix":"10.1108","volume":"16","author":[{"given":"Peter","family":"McKenna","sequence":"first","affiliation":[]}],"member":"140","reference":[{"key":"key2020092418093925400_ref001","doi-asserted-by":"crossref","first-page":"96","DOI":"10.4103\/1319-1683.71992","article-title":"A practical discussion to avoid common pitfalls when constructing multiple choice questions items","volume":"17","year":"2010","journal-title":"Journal of Family and Community Medicine"},{"key":"key2020092418093925400_ref002","unstructured":"Australian Mathematics Trust (2017), \u201cAustralian mathematics competition\u201d, available at: www.amt.edu.au\/mathematics\/amc\/amc-scoring-system\/ (accessed April 28 2018)."},{"key":"key2020092418093925400_ref04a","volume-title":"Language Testing in Practice and Palmer","year":"1996"},{"key":"key2020092418093925400_ref003","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/s11299-005-0001-z","article-title":"Scoring and keying multiple choice tests: a case study in irrationality","volume":"4","year":"2005","journal-title":"Mind and Society"},{"key":"key2020092418093925400_ref004","first-page":"21","article-title":"Using MCQ-based assessment to achieve validity, reliability and manageability in introductory level large class assessment","year":"2010","journal-title":"HE Monitor 10 Teaching and Learning beyond Formal Access: Assessment through the Looking Glass"},{"issue":"2","key":"key2020092418093925400_ref005","doi-asserted-by":"crossref","first-page":"229","DOI":"10.5406\/amerjpsyc.128.2.0229","article-title":"Can multiple-choice testing induce desirable difficulties? Evidence from the laboratory and the classroom","volume":"128","year":"2015","journal-title":"The American Journal of Psychology"},{"key":"key2020092418093925400_ref006","first-page":"2","article-title":"Negatively-Marked MCQ assessments that reward partial knowledge do not introduce gender bias yet increase student performance and satisfaction and reduce anxiety","volume":"8","year":"2013","journal-title":"Plos One"},{"key":"key2020092418093925400_ref007","volume-title":"The Challenge of Problem-Based Learning","year":"2013"},{"key":"key2020092418093925400_ref008","volume-title":"Blueprint for Computer-assisted Assessment","year":"2004"},{"issue":"1","key":"key2020092418093925400_ref009","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1080\/02602930020022273","article-title":"Quantifying the effects of chance in multiple choice and true\/false tests: question selection and guessing of answers","volume":"26","year":"2001","journal-title":"Assessment and Evaluation in Higher Education"},{"issue":"9","key":"key2020092418093925400_ref010","doi-asserted-by":"crossref","first-page":"805","DOI":"10.1046\/j.1365-2923.2002.01299.x","article-title":"Misinformation, partial knowledge and guessing in true\/false tests","volume":"36","year":"2002","journal-title":"Medical Education"},{"key":"key2020092418093925400_ref011","article-title":"Alternative marking schemes for on-line multiple choice tests","volume-title":"in 7th Annual Conference on the Teaching of Computing","year":"1999"},{"issue":"2","key":"key2020092418093925400_ref012","doi-asserted-by":"crossref","first-page":"218","DOI":"10.1080\/02602938.2014.902192","article-title":"Reducing the need for guesswork in multiple-choice tests","volume":"40","year":"2015","journal-title":"Assessment and Evaluation in Higher Education"},{"key":"key2020092418093925400_ref013","unstructured":"CAA (2018), \u201cTheoretical knowledge examinations\u201d, available at: www.caa.co.uk\/General-aviation\/Pilot-licences\/EASA-requirements\/General\/Theoretical-knowledge-examinations\/ (accessed 13 April 2018)."},{"issue":"7","key":"key2020092418093925400_ref014","doi-asserted-by":"crossref","first-page":"1194","DOI":"10.1021\/ed500465q","article-title":"Multiple-Choice exams and guessing: results from a one-year study of general chemistry tests designed to discourage guessing","volume":"92","year":"2015","journal-title":"Journal of Chemical Education"},{"issue":"4","key":"key2020092418093925400_ref015","doi-asserted-by":"crossref","first-page":"957","DOI":"10.2307\/1061503","article-title":"Are Multiple-Choice exams easier for economics students? A comparison of multiple-choice and \u2018equivalent\u2019 constructed-response exam questions","volume":"68","year":"2002","journal-title":"Southern Economic Journal"},{"key":"key2020092418093925400_ref016","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1007\/978-94-017-1386-3_21","article-title":"Assertion-Reason assessment in formative and summative tests: results from two graduate case studies","volume-title":"Educational Innovation in Economics and Business","year":"2004"},{"issue":"1","key":"key2020092418093925400_ref017","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1016\/S1322-7696(08)60478-3","article-title":"Design, format, validity and reliability of multiple choice questions for use in nursing research and education","volume":"12","year":"2005","journal-title":"Collegian"},{"key":"key2020092418093925400_ref018","doi-asserted-by":"crossref","first-page":"398","DOI":"10.1016\/j.sbspro.2014.03.432","article-title":"Comparison between three-and four-option multiple choice questions,\u201d proceedings of the international conference on current trends in ELTA","volume":"98","year":"2014","journal-title":"Procedia - Social and Behavioral Sciences"},{"issue":"1","key":"key2020092418093925400_ref019","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1016\/j.cptl.2016.08.036","article-title":"How-to-guide for writing multiple choice questions for the pharmacy instructor","volume":"9","year":"2017","journal-title":"Currents in Pharmacy Teaching and Learning"},{"issue":"3","key":"key2020092418093925400_ref020","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1119\/1.1466554","article-title":"Marking sense of students\u2019 answers to multiple-choice questions","volume":"40","year":"2002","journal-title":"The Physics Teacher"},{"issue":"5","key":"key2020092418093925400_ref021","doi-asserted-by":"crossref","first-page":"415","DOI":"10.1016\/j.jmp.2010.06.001","article-title":"Optimal correction for guessing in multiple-choice tests","volume":"54","year":"2010","journal-title":"Journal of Mathematical Psychology"},{"issue":"6","key":"key2020092418093925400_ref056","doi-asserted-by":"crossref","first-page":"703","DOI":"10.1080\/0260293042000227245","article-title":"Using assessment to support higher level learning: the multiple choice item development assignment","volume":"29","year":"2004","journal-title":"Assessment and Evaluation in Higher Education"},{"issue":"6","key":"key2020092418093925400_ref022","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1017\/S0955603600002336","article-title":"Extended matching items (EMIs): solving the conundrum","volume":"27","year":"2003","journal-title":"Psychiatric Bulletin"},{"issue":"10","key":"key2020092418093925400_ref023","doi-asserted-by":"crossref","first-page":"1310","DOI":"10.1021\/ed400247d","article-title":"Polytomous versus dichotomous scoring on multiple-choice examinations: development of a rubric for rating partial credit","volume":"90","year":"2013","journal-title":"Journal of Chemical Education"},{"key":"key2020092418093925400_ref024","article-title":"Developing and validating multiple-choice test items","year":"2004"},{"issue":"1","key":"key2020092418093925400_ref025","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3108\/beej.2003.02000007","article-title":"Multiple-choice questions \u2013 a reprieve","volume":"2","year":"2003","journal-title":"BioScience Education"},{"key":"key2020092418093925400_ref026","unstructured":"Hinchliffe, J. (2014), \u201cCQ university scraps multiple choice exams in an australian first\u201d, available at: www.abc.net.au\/news\/2014-09-23\/cqu-scraps-multiple-choice-exams-in-an-australian-first\/5763226 (accessed 28 July 2017), ABC News, 23 September 2014."},{"issue":"1","key":"key2020092418093925400_ref027","first-page":"20","article-title":"The incidence of clueing in multiple choice testbank questions in accounting: Some evidence from Australia","volume":"10","year":"2016","journal-title":"The E \u2013 Journal of Business Education and Scholarship of Teaching"},{"key":"key2020092418093925400_ref028","article-title":"Designing with the mind in mind: simple guide to understanding user interface design rules","year":"2014"},{"key":"key2020092418093925400_ref029","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1016\/j.sbspro.2011.02.035","article-title":"Multiple choice and constructed response tests: do test format and scoring matter?","volume":"12","year":"2011","journal-title":"Procedia - Social and Behavioral Sciences"},{"key":"key2020092418093925400_ref033a","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1207\/s15324818ame1004_4","article-title":"Combining multiple-choice and constructed-response test scores: an economist\u2019s view","volume":"10","year":"1997","journal-title":"Applied Measurment in Education"},{"issue":"8","key":"key2020092418093925400_ref030","first-page":"926","article-title":"Assessment of the relation between students\u2019 gender and their scores on selecting confidence choices in Confidence-Based exams","volume":"11","year":"2012","journal-title":"Iranian Journal of Medical Education"},{"issue":"1","key":"key2020092418093925400_ref031","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1111\/j.1540-4609.2009.00243.x","article-title":"Why is performance on multiple-choice tests and constructed-response tests not more closely related? Theory and an empirical test","volume":"8","year":"2010","journal-title":"Decision Sciences Journal of Innovative Education"},{"issue":"3","key":"key2020092418093925400_ref032","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1016\/j.stueduc.2013.07.001","article-title":"Scoring methods for multiple choice assessment in higher education \u2013 is it still a matter of number-right scoring or negative marking?","volume":"39","year":"2013","journal-title":"Studies in Educational Evaluation"},{"issue":"3","key":"key2020092418093925400_ref033","first-page":"269","article-title":"The negative suggestion effect of the false statement in the true-false test","volume":"2","year":"1934","journal-title":"The Journal of Experimental Education Educational Measurements"},{"issue":"2","key":"key2020092418093925400_ref034","doi-asserted-by":"crossref","first-page":"261","DOI":"10.3758\/BF03193404","article-title":"Paradoxical effects of testing: repeated retrieval attempts enhance the likelihood of later accurate and false recall","volume":"34","year":"2006","journal-title":"Memory and Cognition"},{"key":"key2020092418093925400_ref035","first-page":"98","article-title":"Using constructed-response and multiple-choice questions in undergraduate examinations","volume":"2","year":"2010","journal-title":"World Journal of Management"},{"key":"key2020092418093925400_ref036","unstructured":"McElvaney, E.J., Fermelis, J. and Strawbridge, H. (2012), \u201cAssessment method difference: comparisons between international and domestic students within a first year undergraduate management course\u201d, International Review of Business Research Papers 8, pp. 205-214."},{"issue":"8","key":"key2020092418093925400_ref037","doi-asserted-by":"crossref","first-page":"919","DOI":"10.1080\/02602938.2011.586991","article-title":"Peer review improves the quality of MCQ examinations","volume":"37","year":"2012","journal-title":"Assessment and Evaluation in Higher Education"},{"key":"key2020092418093925400_ref038","doi-asserted-by":"crossref","first-page":"25","DOI":"10.3928\/0148-4834-20010101-07","article-title":"Assessment of multiple-choice questions in selected test banks accompanying test books used in nursing education","volume":"40","year":"2001","journal-title":"Journal of Nursing Education"},{"issue":"2","key":"key2020092418093925400_ref039","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1177\/000494419904300202","article-title":"Multiple-choice versus Short-response items: differences in omit behaviour","volume":"43","year":"1999","journal-title":"Australian Journal of Education"},{"key":"key2020092418093925400_ref040","unstructured":"Moore, H. (2014), \u201cComment on \u201cDoes the student a) know the answer, or are they b) guessing?\u201d [The conversation]\u201d, available at: https:\/\/theconversation.com\/does-the-student-a-know-the-answer-or-are-they-b-guessing-31893, (accessed 19 April 2018)."},{"key":"key2020092418093925400_ref041","doi-asserted-by":"crossref","first-page":"17","DOI":"10.3928\/0148-4834-20010101-06","article-title":"Writing multiple-choice test items that promote and measure critical thinking","volume":"40","year":"2001","journal-title":"Journal of Nursing Education"},{"key":"key2020092418093925400_ref042","unstructured":"Newcastle University (2015, 2018), \u201cNumbas\u201d, available at: www.numbas.org.uk\/"},{"issue":"1","key":"key2020092418093925400_ref043","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1080\/03098770601167922","article-title":"E-assessment by design: using multiple\u2010choice tests to good effect","volume":"31","year":"2007","journal-title":"Journal of Further and Higher Education"},{"issue":"2","key":"key2020092418093925400_ref044","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1027\/\/1015-5759.15.2.143","article-title":"The effect of instructions on multiple-choice test scores","volume":"15","year":"1999","journal-title":"European Journal of Psychological Assessment"},{"issue":"1","key":"key2020092418093925400_ref045","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1037\/h0070067","article-title":"The negative suggestion effect of true-false examination questions","volume":"17","year":"1926","journal-title":"Journal of Educational Psychology"},{"issue":"5","key":"key2020092418093925400_ref046","first-page":"1155","article-title":"The positive and negative consequences of multiple-choice testing","volume":"31","year":"2005","journal-title":"Journal of Experimental Psychology"},{"key":"key2020092418093925400_ref047","unstructured":"Roper, M. (2008, 2014), \u201cScientists calculate odd ways to die\u201d, available at: www.mirror.co.uk\/news\/weird-news\/scientists-calculate-odd-ways-die-282884 (accessed 8 August 2016, 30 May 2008, updated 3 April 2014)."},{"key":"key2020092418093925400_ref048","unstructured":"RWAP Services (2019), \u201cThe Chances of Winning The UK National Lottery\u201d, available at: http:\/\/playlotto.org.uk\/lottery\/uklottery_odds.html (accessed 8 August 2016)."},{"issue":"1","key":"key2020092418093925400_ref049","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1111\/j.1540-4609.2005.00053.x","article-title":"Multiple-Choice tests and student understanding: what is the connection?","volume":"3","year":"2005","journal-title":"Decision Sciences - Journal of Innovative Education"},{"key":"key2020092418093925400_ref050","first-page":"246","article-title":"Why MCQ","volume":"66","year":"2004","journal-title":"Indian Journal of Surgery"},{"issue":"6","key":"key2020092418093925400_ref051","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1016\/j.nepr.2006.07.002","article-title":"The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments","volume":"6","year":"2006","journal-title":"Nurse Education in Practice"},{"issue":"3","key":"key2020092418093925400_ref052","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1177\/014662167700100304","article-title":"On the equivalence of constructed-response and multiple-choice tests","volume":"1","year":"1977","journal-title":"Applied Psychological Measurement"},{"issue":"5","key":"key2020092418093925400_ref056a","doi-asserted-by":"crossref","first-page":"571","DOI":"10.4103\/0253-7613.190757","article-title":"Comparison between three option, four option and five option multiple choice question tests for quality parameters: a randomized study","volume":"48","year":"2016","journal-title":"Indian Journal of Pharmacology"},{"issue":"2","key":"key2020092418093925400_ref053","first-page":"103","article-title":"Combining multiple-choice and constructed-response test scores: toward a marxist theory of test construction","volume":"6","year":"2012","journal-title":"Applied Measurement in Education"},{"issue":"3","key":"key2020092418093925400_ref054","doi-asserted-by":"crossref","first-page":"287","DOI":"10.1080\/02602930500352857","article-title":"Assertion\u2010reason multiple\u2010choice testing as a tool for deep learning: a qualitative analysis","volume":"31","year":"2006","journal-title":"Assessment and Evaluation in Higher Education"},{"key":"key2020092418093925400_ref055","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1016\/j.nedt.2011.09.011","article-title":"Implications of multiple-choice testing in nursing education","volume":"32","year":"2012","journal-title":"Nurse Education Today"},{"key":"key2020092418093925400_ref057","first-page":"637","article-title":"The effect of four formats of multiple-choice questions on the listening comprehension of EFL learners","volume-title":"Proceedings of the International Conference on Current Trends in ELT. Procedia - Social and Behavioural Sciences","year":"2014"}],"container-title":["Interactive Technology and Smart Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/ITSE-09-2018-0071\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/ITSE-09-2018-0071\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T21:56:58Z","timestamp":1753394218000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/itse\/article\/16\/1\/59-73\/185845"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,3,11]]},"references-count":60,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,3,11]]}},"alternative-id":["10.1108\/ITSE-09-2018-0071"],"URL":"https:\/\/doi.org\/10.1108\/itse-09-2018-0071","relation":{},"ISSN":["1741-5659","1741-5659"],"issn-type":[{"value":"1741-5659","type":"print"},{"value":"1741-5659","type":"print"}],"subject":[],"published":{"date-parts":[[2019,3,11]]}}}