{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,26]],"date-time":"2026-02-26T16:45:44Z","timestamp":1772124344718,"version":"3.50.1"},"reference-count":26,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2023,5,13]],"date-time":"2023-05-13T00:00:00Z","timestamp":1683936000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Union","award":["15HLT04 NeuroMet"],"award-info":[{"award-number":["15HLT04 NeuroMet"]}]},{"name":"European Union","award":["18HLT09 NeuroMet2"],"award-info":[{"award-number":["18HLT09 NeuroMet2"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>The justification for making a measurement can be sought in asking what decisions are based on measurement, such as in assessing the compliance of a quality characteristic of an entity in relation to a specification limit, SL. The relative performance of testing devices and classification algorithms used in assessing compliance is often evaluated using the venerable and ever popular receiver operating characteristic (ROC). However, the ROC tool has potentially all the limitations of classic test theory (CTT) such as the non-linearity, effects of ordinality and confounding task difficulty and instrument ability. These limitations, inherent and often unacknowledged when using the ROC tool, are tackled here for the first time with a modernised approach combining measurement system analysis (MSA) and item response theory (IRT), using data from pregnancy testing as an example. The new method of assessing device ability from separate Rasch IRT regressions for each axis of ROC curves is found to perform significantly better, with correlation coefficients with traditional area-under-curve metrics of at least 0.92 which exceeds that of linearised ROC plots, such as Linacre\u2019s, and is recommended to replace other approaches for device assessment. The resulting improved measurement quality of each ROC curve achieved with this original approach should enable more reliable decision-making in conformity assessment in many scenarios, including machine learning, where its use as a metric for assessing classification algorithms has become almost indispensable.<\/jats:p>","DOI":"10.3390\/a16050253","type":"journal-article","created":{"date-parts":[[2023,5,15]],"date-time":"2023-05-15T02:02:11Z","timestamp":1684116131000},"page":"253","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Modernising Receiver Operating Characteristic (ROC) Curves"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4349-500X","authenticated-orcid":false,"given":"Leslie R.","family":"Pendrill","sequence":"first","affiliation":[{"name":"Measurement Science and Technology, Division Safety and Transport, RISE Research Institutes of Sweden, 41258 Gothenburg, Sweden"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3700-3921","authenticated-orcid":false,"given":"Jeanette","family":"Melin","sequence":"additional","affiliation":[{"name":"Measurement Science and Technology, Division Safety and Transport, RISE Research Institutes of Sweden, 41258 Gothenburg, Sweden"},{"name":"Department of Leadership, Demand and Control, Swedish Defence University, 65340 Karlstad, Sweden"}]},{"given":"Anne","family":"Stavelin","sequence":"additional","affiliation":[{"name":"Noklus, The Norwegian Organisation for Quality Improvement of Laboratory Examinations, Boks 6165, 5892 Bergen, Norway"}]},{"given":"Gunnar","family":"Nordin","sequence":"additional","affiliation":[{"name":"Equalis, Box 977, SE75109 Uppsala, Sweden"}]}],"member":"1968","published-online":{"date-parts":[[2023,5,13]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"S206","DOI":"10.1088\/0026-1394\/51\/4\/S206","article-title":"Using Measurement Uncertainty in Decision-Making and Conformity Assessment","volume":"51","author":"Pendrill","year":"2014","journal-title":"Metrologia"},{"key":"ref_2","first-page":"22","article-title":"Evaluation of Measurement Data: The Role of Measurement Uncertainty in Conformity Assessment","volume":"35","author":"Hibbert","year":"2013","journal-title":"Chem. Int.\u2014Newsmag. IUPAC"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1109\/TIT.1954.1057460","article-title":"The Theory of Signal Detectability","volume":"4","author":"Peterson","year":"1954","journal-title":"Trans. IRE Prof. Group Inf. Theory"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Pendrill, L. (2019). Quality Assured Measurement: Unification across Social and Physical Sciences, Springer International Publishing.","DOI":"10.1007\/978-3-030-28695-8"},{"key":"ref_5","unstructured":"Mosteller, F., and Tukey, J. (1977). Data Analysis and Regression: A Second Course in Statistics, Pearson. [1st ed.]."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1042\/EBC20150012","article-title":"Lateral Flow Assays","volume":"60","author":"Koczula","year":"2016","journal-title":"Essays Biochem."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"662","DOI":"10.3109\/00365510902968756","article-title":"How to Deal with Semi-Quantitative Tests? Application of an Ordinal Scale Model to Measurements of Urine Glucose","volume":"69","author":"Petersen","year":"2009","journal-title":"Scand. J. Clin. Lab. Investig."},{"key":"ref_8","first-page":"020601","article-title":"Current Status of the Lateral Flow Immunoassay for the Detection of SARS-CoV-2 in Nasopharyngeal Swabs","volume":"31","author":"Dodig","year":"2021","journal-title":"Biochem. Med."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"531","DOI":"10.3233\/IDA-130592","article-title":"ROC Analysis of Classifiers in Machine Learning: A Survey","volume":"17","author":"Majnik","year":"2013","journal-title":"Intell. Data Anal."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"2305","DOI":"10.1093\/clinchem\/33.12.2305a","article-title":"Cumulative Distribution Analysis Graphs\u2014An Alternative to ROC Curves","volume":"33","author":"Krouwer","year":"1987","journal-title":"Clin. Chem."},{"key":"ref_11","unstructured":"Birdsall, T. (1973). The Theory of Signal Detectability: ROC Curves and Their Character, University of Michigan Library."},{"key":"ref_12","first-page":"317","article-title":"Evaluating a Screening Test","volume":"7","author":"Linacre","year":"1994","journal-title":"Rasch Meas. Trans."},{"key":"ref_13","first-page":"180","article-title":"Comparing Rasch Analyses Probability Estimates to Sensitivity, Specificity and Likelihood Ratios When Examining the Utility of Medical Diagnostic Tests","volume":"6","author":"Cipriani","year":"2005","journal-title":"J. Appl. Meas."},{"key":"ref_14","first-page":"271","article-title":"Embedding Measurement within Existing Computerized Data Systems: Scaling Clinical Laboratory and Medical Records Heart Failure Data to Predict ICU Admission","volume":"11","author":"Fisher","year":"2010","journal-title":"J. Appl. Meas."},{"key":"ref_15","first-page":"729","article-title":"Multiple Regression via Measurement","volume":"14","author":"Wright","year":"2000","journal-title":"Rasch Meas. Trans."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"777","DOI":"10.1515\/cclm-2022-1256","article-title":"Tutorial: Statistical Methods for the Meta-Analysis of Diagnostic Test Accuracy Studies","volume":"61","author":"Schlattmann","year":"2023","journal-title":"Clin. Chem. Lab. Med."},{"key":"ref_17","unstructured":"E11 Committee (2022). Guide for Measurement Systems Analysis (MSA), ASTM International."},{"key":"ref_18","unstructured":"Rasch, G. (1960). Studies in Mathematical Psychology: I. Probabilistic Models for Some Intelligence and Attainment Tests, Nielsen & Lydiche."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"012012","DOI":"10.1088\/1742-6596\/772\/1\/012012","article-title":"Ability Evaluation by Binary Tests: Problems, Challenges & Recent Advances","volume":"772","author":"Bashkansky","year":"2016","journal-title":"J. Phys. Conf. Ser."},{"key":"ref_20","unstructured":"Montgomery, D.C. (1996). Introduction to Statistical Quality Control, Wiley. [3rd ed.]."},{"key":"ref_21","unstructured":"Statology (2023, May 10). Zach How to Create a ROC Curve in Excel (Step-by-Step). Available online: https:\/\/www.statology.org\/roc-curve-excel\/."},{"key":"ref_22","first-page":"111","article-title":"Evidence Based Emergency Medicine; Part 5 Receiver Operating Characteristic Curve and Area under the Curve","volume":"4","author":"Safari","year":"2016","journal-title":"Emergency"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1002\/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3","article-title":"Index for Rating Diagnostic Tests","volume":"3","author":"Youden","year":"1950","journal-title":"Cancer"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Fisher, W.P., William, P., and Cano, S.J. (2023). Person-Centered Outcome Metrology: Principles and Applications for High Stakes Decision Making, Springer International Publishing.","DOI":"10.1007\/978-3-031-07465-3"},{"key":"ref_25","unstructured":"Rasch, G. (1961). Psychology, Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 4: Contributions to Biology and Problems of Medicine, Berkeley, CA, USA, 20 June\u201330 July 1960, University of California Press."},{"key":"ref_26","first-page":"100","article-title":"Examining Rating Scales Using Rasch and Mokken Models for Rater-Mediated Assessments","volume":"15","author":"Wind","year":"2014","journal-title":"J. Appl. Meas."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/16\/5\/253\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:34:19Z","timestamp":1760124859000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/16\/5\/253"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,13]]},"references-count":26,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2023,5]]}},"alternative-id":["a16050253"],"URL":"https:\/\/doi.org\/10.3390\/a16050253","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,13]]}}}