{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T08:12:40Z","timestamp":1789027960753,"version":"build-2803163510"},"reference-count":33,"publisher":"JMIR Publications Inc.","issue":"6","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"\n Background<\/jats:title>\n A large Midwestern state commissioned a virtual driving test (VDT) to assess driving skills preparedness before the on-road examination (ORE). Since July 2017, a pilot deployment of the VDT in state licensing centers (VDT pilot) has collected both VDT and ORE data from new license applicants with the aim of creating a scoring algorithm that could predict those who were underprepared.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n Leveraging data collected from the VDT pilot, this study aimed to develop and conduct an initial evaluation of a novel machine learning (ML)\u2013based classifier using limited domain knowledge and minimal feature engineering to reliably predict applicant pass\/fail on the ORE. Such methods, if proven useful, could be applicable to the classification of other time series data collected within medical and other settings.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We analyzed an initial dataset that comprised 4308 drivers who completed both the VDT and the ORE, in which 1096 (25.4%) drivers went on to fail the ORE. We studied 2 different approaches to constructing feature sets to use as input to ML algorithms: the standard method of reducing the time series data to a set of manually defined variables that summarize driving behavior and a novel approach using time series clustering. We then fed these representations into different ML algorithms to compare their ability to predict a driver\u2019s ORE outcome (pass\/fail).<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n The new method using time series clustering performed similarly compared with the standard method in terms of overall accuracy for predicting pass or fail outcome (76.1% vs 76.2%) and area under the curve (0.656 vs 0.682). However, the time series clustering slightly outperformed the standard method in differentially predicting failure on the ORE. The novel clustering method yielded a risk ratio for failure of 3.07 (95% CI 2.75-3.43), whereas the standard variables method yielded a risk ratio for failure of 2.68 (95% CI 2.41-2.99). In addition, the time series clustering method with logistic regression produced the lowest ratio of false alarms (those who were predicted to fail but went on to pass the ORE; 27.2%).<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Our results provide initial evidence that the clustering method is useful for feature construction in classification tasks involving time series data when resources are limited to create multiple, domain-relevant variables.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/13995","type":"journal-article","created":{"date-parts":[[2019,12,16]],"date-time":"2019-12-16T02:02:12Z","timestamp":1576461732000},"page":"e13995","source":"Crossref","is-referenced-by-count":7,"title":["Simulator Pre-Screening of Underprepared Drivers Prior to Licensing On-Road Examination: Clustering of Virtual Driving Test Time Series Data"],"prefix":"10.2196","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0395-9300","authenticated-orcid":false,"given":"David","family":"Grethlein","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6632-1403","authenticated-orcid":false,"given":"Flaura Koplin","family":"Winston","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0466-4272","authenticated-orcid":false,"given":"Elizabeth","family":"Walshe","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3452-019X","authenticated-orcid":false,"given":"Sean","family":"Tanner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7745-6821","authenticated-orcid":false,"given":"Venk","family":"Kandadai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9616-2981","authenticated-orcid":false,"given":"Santiago","family":"Onta\u00f1\u00f3n","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2020,6,18]]},"reference":[{"key":"ref1","unstructured":"Mini\u00f1oAMCenters for Disease Control and Prevention20102019-01-19Mortality Among Teenagers Aged 12-19 Years: United States, 1999-2006https:\/\/www.cdc.gov\/nchs\/products\/databriefs\/db37.htm"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.1016\/s0001-4575(02)00006-4"},{"key":"ref3","unstructured":"CurryAEPfeifferMRDurbinDRElliottMRKimKHAAA Foundation for Traffic Safety20142019-03-06Young Driver Crash Rates in New Jersey by Driving Experience, Age, and License Phasehttps:\/\/aaafoundation.org\/young-driver-crash-rates-new-jersey-driving-experience-age-license-phase\/"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.1016\/j.jsr.2014.05.005"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.1016\/j.aap.2010.10.019"},{"key":"ref6","doi-asserted-by":"publisher","DOI":"10.1016\/j.aap.2014.07.016"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.1016\/s0001-4575(02)00100-8"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.2943"},{"key":"ref9","unstructured":"WalsheEAOppenheimerNKandadaiVWinstonFK2011-2018 Annual Meeting Compendium of Papers Archive20192019-03-12Comparison of Virtual Driving Test Performance and On-Road Examination for Licensure Performance: A Replication Studyhttp:\/\/amonline.trb.org\/68387-trb-1.4353651\/t0016-1.4367835\/1346-1.4368021\/19-01210-1.4362689\/19-01210-1.4368040?qr=1"},{"key":"ref10","unstructured":"Diagnostic Driving20182019-03-12How Does Ready-Assess Work?http:\/\/www.diagnosticdriving.com\/how-readyassess-works"},{"key":"ref11","unstructured":"The Orange County Independent20172019-03-12Drivers License Test Ends in Spectacular Crash at Costa Mesa DMVhttp:\/\/theorangecountyindependent.com\/2017\/01\/drivers-license-test-ends-in-horrendous-crash-at-costa-mesa-dmv\/"},{"key":"ref12","unstructured":"OsborneMABC News20182019-03-12Teen Taking Driver's Exam Crashes Into Test Officehttps:\/\/abcnews.go.com\/US\/teen-taking-drivers-exam-crashes-test-office\/story?id=53998272"},{"key":"ref13","unstructured":"HooperBUnited Press International20182019-03-12Student Driver Crashes Into DMV During Road Testhttps:\/\/www.upi.com\/Student-driver-crashes-into-DMV-during-road-test\/6151522848041\/"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.1136\/injuryprev-2014-041480"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.3141\/2321-10"},{"key":"ref16","unstructured":"CrashStats20192019-03-06National Motor Vehicle Crash Causation Survey: Report to Congresshttps:\/\/crashstats.nhtsa.dot.gov\/Api\/Public\/ViewPublication\/811059"},{"key":"ref17","unstructured":"Diagnostic Driving2019-03-06Ready-Assess for Driver's Licensing Centershttp:\/\/www.diagnosticdriving.com\/for-drivers-license-programs"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.2147\/IJGM.S19639"},{"key":"ref19","unstructured":"ten HoltGAReindersMJHendriksEAIntelligent Systems20072019-02-20Multi-Dimensional Dynamic Time Warping for Gesture Recognitionhttp:\/\/msp.ewi.tudelft.nl\/sites\/default\/files\/DTW-vASCI.pdf"},{"key":"ref20","unstructured":"RangsitPSSemantic Scholar2019-02-20Generalized Dynamic Time Warping for Multiple Multidimensional Sequence Alignmenthttps:\/\/pdfs.semanticscholar.org\/1b7f\/0f25361d8dfcfea6d4e72a64f80b662272ef.pdf"},{"key":"ref21","unstructured":"SchubertERousseeuwPJArXiv20182019-02-20Faster k-Medoids Clustering: Improving the PAM, CLARA, and CLARANS Algorithmshttps:\/\/arxiv.org\/abs\/1810.05691"},{"key":"ref22","unstructured":"Weka-SourceForge2019-02-20Class SimpleLogistichttp:\/\/weka.sourceforge.net\/doc.dev\/weka\/classifiers\/functions\/SimpleLogistic.html"},{"key":"ref23","unstructured":"Weka-SourceForge2019-02-20Class LibSVMhttp:\/\/weka.sourceforge.net\/doc.stable\/weka\/classifiers\/functions\/LibSVM.html"},{"key":"ref24","doi-asserted-by":"publisher","DOI":"10.5555\/1046920.1194907"},{"key":"ref25","unstructured":"HsuCWChangCCLinCJNational Taiwan University: Department of Computer Science20162019-02-21A Practical Guide to Support Vector Classificationhttps:\/\/www.csie.ntu.edu.tw\/~cjlin\/papers\/guide\/guide.pdf"},{"key":"ref26","unstructured":"TapeTGThe Darwin Web Server2019-03-12The Area Under an ROC Curvehttp:\/\/gim.unmc.edu\/dxtests\/ROC3.htm"},{"key":"ref27","doi-asserted-by":"publisher","DOI":"10.1109\/icdm.2003.1250910"},{"key":"ref28","unstructured":"SteinGUniversity of Central Florida20092019-02-21Falconet: Force-Feedback Approach For Learning From Coaching And Observation Using Natural And Experiential Traininghttps:\/\/stars.library.ucf.edu\/etd\/3988\/"},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611972771.37"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/n16-1172"},{"key":"ref31","doi-asserted-by":"publisher","DOI":"10.1016\/j.jadohealth.2012.02.012"},{"key":"ref32","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btg119"},{"key":"ref33","doi-asserted-by":"publisher","DOI":"10.1109\/jsen.2015.2443380"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2020\/6\/e13995","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:03:40Z","timestamp":1782842620000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2020\/6\/e13995"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,18]]},"references-count":33,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2020]]}},"URL":"https:\/\/doi.org\/10.2196\/13995","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.13995","asserted-by":"object"}]},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6,18]]}}}