{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T02:02:13Z","timestamp":1787018533510,"version":"build-2736575974"},"reference-count":88,"publisher":"SAGE Publications","issue":"5","license":[{"start":{"date-parts":[[2020,9,28]],"date-time":"2020-09-28T00:00:00Z","timestamp":1601251200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"name":"Next Generation Advanced Procedures Initiative"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Hum Factors"],"published-print":{"date-parts":[[2023,8]]},"abstract":"<jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>The goal of this study is to assess machine learning for predicting procedure performance from operator and procedure characteristics.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Procedures are vital for the performance and safety of high-risk industries. Current procedure design guidelines are insufficient because they rely on subjective assessments and qualitative analyses that struggle to integrate and quantify the diversity of factors that influence procedure performance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Method<\/jats:title>\n                    <jats:p>We used data from a 25-participant study with four procedures, conducted on a high-fidelity oil extraction simulation to develop logistic regression (LR), random forest (RF), and decision tree (DT) algorithms that predict procedure step performance from operator, step, readability, and natural language processing-based features. Features were filtered using the Boruta approach. The algorithms were trained and optimized with a repeated 10-fold cross-validation. After training, inference was performed using variable importance and partial dependence plots.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The RF, DT, and LR algorithms with all features had an area under the receiver operating characteristic curve (AUC) of 0.78, 0.77, and 0.75, respectively, and significantly outperformed the LR with only operator features (LROP), with an AUC of 0.61. The most important features were experience, familiarity, total words, and character-based metrics. The partial dependence plots showed that steps with fewer words, abbreviations, and characters were correlated with correct step performance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Machine learning algorithms are a promising approach for predicting step-level procedure performance, with acknowledged limitations on interpolating to nonobserved data, and may help guide procedure design after validation with additional data on further tasks.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Application<\/jats:title>\n                    <jats:p>After validation, the inferences from these models can be used to generate procedure design alternatives.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1177\/0018720820958588","type":"journal-article","created":{"date-parts":[[2020,9,29]],"date-time":"2020-09-29T01:48:55Z","timestamp":1601344135000},"page":"701-717","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":2,"title":["Predicting Procedure Step Performance From Operator and Text Features: A Critical First Step Toward Machine Learning-Driven Procedure Design"],"prefix":"10.1177","volume":"65","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7827-8828","authenticated-orcid":false,"given":"Anthony D.","family":"McDonald","sequence":"first","affiliation":[{"name":"Texas A&M University, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6885-477X","authenticated-orcid":false,"given":"Nilesh","family":"Ade","sequence":"additional","affiliation":[{"name":"Texas A&M University, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3679-9171","authenticated-orcid":false,"given":"S. Camille","family":"Peres","sequence":"additional","affiliation":[{"name":"Texas A&M University, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2020,9,28]]},"reference":[{"key":"bibr1-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1016\/j.ergon.2020.102930"},{"key":"bibr2-0018720820958588","first-page":"490","volume":"26","author":"Anderson J","year":"1983","journal-title":"Journal of Reading"},{"key":"bibr3-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.89.4.369"},{"key":"bibr4-0018720820958588","volume-title":"Lesen-Verstehen-Lernen-Schreiben [Read-Understand-Learn-Write]","author":"Bamberger R.","year":"1984"},{"key":"bibr5-0018720820958588","unstructured":"Baron R. (2009). Failure to follow procedures: Deviations are a significant factor in maintenance errors. Federal Aviation Administration. https:\/\/www.faa.gov\/about\/initiatives\/maintenance_hf\/library\/documents\/media\/roi\/failure_to_follow_procedures_deviations_are_a_significant_factor_in_maintenance_errors.pdf"},{"key":"bibr6-0018720820958588","unstructured":"Bates S., Holroyd J. (2012). Human factors that lead to non-compliance with standard operating procedures (Research Report RR 919). Health and Safety Executive Laboratory."},{"key":"bibr7-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1016\/S0951-8320(00)00051-X"},{"key":"bibr8-0018720820958588","doi-asserted-by":"publisher","DOI":"10.21105\/joss.00774"},{"key":"bibr9-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1016\/j.aap.2008.03.012"},{"key":"bibr10-0018720820958588","unstructured":"Bj\u00f6rnsson C. H. (1968).\n                      L\u00e4sbarhet\n                      [\n                      Readability\n                      ]\n                      . Liber.\n                      (6th ed.)."},{"key":"bibr11-0018720820958588","first-page":"143","volume-title":"Oxford handbook of cognitive engineering","author":"Boot W.","year":"2011"},{"key":"bibr12-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010933404324"},{"key":"bibr14-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1016\/j.jlp.2010.05.008"},{"key":"bibr13-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1177\/154193120404802006"},{"key":"bibr15-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1016\/S0097-8485(01)00094-8"},{"key":"bibr16-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1207\/s15327043hup0401_1"},{"key":"bibr17-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1016\/j.ssci.2016.06.008"},{"key":"bibr18-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1518\/hfes.45.3.408.27248"},{"key":"bibr19-0018720820958588","unstructured":"Caylor J. S., Sticht T. G. (1973).\n                      Development of a simple readability index for job reading material\n                      . Department of the Army. Retrieved from ERIC database (ED076707)."},{"key":"bibr20-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2015.04.059"},{"key":"bibr21-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-35488-8_13"},{"key":"bibr22-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1080\/001401398186603"},{"key":"bibr23-0018720820958588","first-page":"155","volume-title":"Verbal learning research and the technology of written instruction","author":"Coleman E. B","year":"1971"},{"key":"bibr24-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1037\/h0076540"},{"key":"bibr25-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1016\/S0003-6870(03)00031-0"},{"key":"bibr26-0018720820958588","doi-asserted-by":"publisher","DOI":"10.2307\/2531595"},{"key":"bibr27-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2017.07.032"},{"key":"bibr28-0018720820958588","unstructured":"DuBay W. (2008). The principles of readability.\n                      Costa Mesa: Impact Information, 949\n                      , 77."},{"key":"bibr29-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1037\/h0062427"},{"key":"bibr71-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1037\/h0057532"},{"key":"bibr30-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1013203451"},{"key":"bibr31-0018720820958588","first-page":"234","volume-title":"Ein Beispiel mathematischer [An example of math] Stilanalyse","volume":"1","author":"Fucks W","year":"1955"},{"key":"bibr32-0018720820958588","doi-asserted-by":"publisher","DOI":"10.28945\/469"},{"key":"bibr33-0018720820958588","volume-title":"The technique of clear writing","author":"Gunning R","year":"1952"},{"key":"bibr34-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1016\/j.ssci.2012.05.011"},{"key":"bibr35-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1016\/j.ssci.2012.05.013"},{"key":"bibr36-0018720820958588","volume-title":"Improving compliance with safety procedures, reducing industrial violations","author":"HFRG","year":"1995"},{"key":"bibr37-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1115847109"},{"key":"bibr38-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1080\/10508410701749365"},{"key":"bibr39-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1198\/106186006X133933"},{"key":"bibr40-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v8i1.14550"},{"key":"bibr41-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4614-7138-7"},{"key":"bibr42-0018720820958588","doi-asserted-by":"crossref","unstructured":"Jockers M. L. (2015). Syuzhet: Extract sentiment and plot arcs from text. https:\/\/github.com\/mjockers\/syuzhet","DOI":"10.32614\/CRAN.package.syuzhet"},{"key":"bibr43-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1177\/1541931218621194"},{"key":"bibr44-0018720820958588","doi-asserted-by":"publisher","DOI":"10.21236\/ADA006655"},{"key":"bibr45-0018720820958588","doi-asserted-by":"publisher","DOI":"10.2307\/747086"},{"key":"bibr46-0018720820958588","unstructured":"Kuhn M. (2020). Caret: classification and regression training. https:\/\/cran.r-project.org\/package=caret"},{"key":"bibr47-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4614-6849-3"},{"key":"bibr48-0018720820958588","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v036.i11"},{"key":"bibr49-0018720820958588","volume-title":"Erreur humaine, fiabilite humaine dans Ie travail [Human error, human reliability in work]","author":"Leplat J","year":"1985"},{"key":"bibr50-0018720820958588","unstructured":"Liaw A., Wiener M (2002). Classification and regression by randomforest. R News, 2, 18-22."},{"key":"bibr51-0018720820958588","doi-asserted-by":"publisher","DOI":"10.2200\/S00416ED1V01Y201204HLT016"},{"key":"bibr52-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1097\/JTO.0b013e3181ec173d"},{"key":"bibr53-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1177\/0018720819856454"},{"key":"bibr54-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1177\/0018720813515272"},{"key":"bibr55-0018720820958588","first-page":"639","volume":"12","author":"McLaughlin G. H","year":"1969","journal-title":"Journal of Reading"},{"key":"bibr56-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2105-10-213"},{"key":"bibr57-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1177\/1541931218621362"},{"key":"bibr58-0018720820958588","volume-title":"[Conference session]. Proceedings of the Human Factors and Ergonomics Society 2018 Annual Meeting, Philadelphia","author":"Neville T. J.","year":"2018"},{"key":"bibr59-0018720820958588","volume-title":"[Conference session]. CEUR Workshop Proceedings (Vol. 718, pp. 93\u201398)","author":"Nielsen F. \u00c5","year":"2011"},{"key":"bibr60-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1016\/j.psep.2012.11.003"},{"key":"bibr61-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-12-803806-2.00017-0"},{"key":"bibr62-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1016\/j.anucene.2015.05.012"},{"key":"bibr63-0018720820958588","doi-asserted-by":"publisher","DOI":"10.4043\/27019-MS"},{"key":"bibr64-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1016\/j.jlp.2016.08.003"},{"key":"bibr65-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1016\/j.jlp.2020.104264"},{"key":"bibr66-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1016\/j.ssci.2014.09.005"},{"key":"bibr67-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1037\/h0043254"},{"key":"bibr68-0018720820958588","unstructured":"R Core Team. (2014). R: A language and environment for statistical computing. http:\/\/www.r-project.org\/"},{"key":"bibr69-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1109\/TSMC.1983.6313160"},{"key":"bibr70-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1093\/oxfordhb\/9780199757183.013.0008"},{"key":"bibr72-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1126\/science.aat2663"},{"key":"bibr73-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1016\/j.psep.2017.09.010"},{"key":"bibr74-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1520-6564(199822)8:3<265::AID-HFM5>3.0.CO;2-1"},{"key":"bibr75-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1016\/j.ssci.2016.08.013"},{"key":"bibr76-0018720820958588","volume-title":"Automated readability index. AMRL-TR66-22","author":"Smith E. A.","year":"1967"},{"key":"bibr77-0018720820958588","volume-title":"A qualitative analysis of media language","author":"Solomon N. W","year":"2006"},{"key":"bibr78-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1109\/ICSEM.2010.14"},{"key":"bibr79-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1086\/458513"},{"key":"bibr80-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1145\/357442.357445"},{"key":"bibr81-0018720820958588","first-page":"231","volume":"16","author":"Tr\u00e4nkle U.","year":"1984","journal-title":"Journal of Developmental Psychology and Educational Psychology"},{"key":"bibr82-0018720820958588","unstructured":"UK Health and Safety Executive. (2015). Hse human factors briefing note no 4: procedures. Retrieved May 9, 2018, http:\/\/www.hse.gov.uk\/humanfactors\/topics\/04procedures.pdf"},{"key":"bibr83-0018720820958588","unstructured":"US Chemical Safety and Hazard Investigation Board. (2007). Investigation report, refinery explosion and fire, bp Texas City. https:\/\/www.csb.gov\/file.aspx?DocumentId=5596"},{"key":"bibr84-0018720820958588","unstructured":"Vicente K. J., Burns C. M. (1995). A field study of operator cognitive monitoring at Pickering nuclear generating station (Report No. CEL 95-04): Cognitive Engineering Laboratory."},{"key":"bibr85-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2017.1319839"},{"key":"bibr86-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1201\/9781410607775.ch28"},{"key":"bibr87-0018720820958588","doi-asserted-by":"publisher","DOI":"10.1109\/ACII.2013.72"},{"key":"bibr88-0018720820958588","first-page":"1","volume":"1903","author":"Zhao Q.","year":"2019","journal-title":"Journal of Business & Economic Statistics"}],"container-title":["Human Factors: The Journal of the Human Factors and Ergonomics Society"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0018720820958588","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/0018720820958588","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0018720820958588","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T07:52:40Z","timestamp":1777449160000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/0018720820958588"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,28]]},"references-count":88,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2023,8]]}},"alternative-id":["10.1177\/0018720820958588"],"URL":"https:\/\/doi.org\/10.1177\/0018720820958588","relation":{},"ISSN":["0018-7208","1547-8181"],"issn-type":[{"value":"0018-7208","type":"print"},{"value":"1547-8181","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,9,28]]}}}