{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T14:32:32Z","timestamp":1791469952418,"version":"4.3.4"},"reference-count":34,"publisher":"Oxford University Press (OUP)","issue":"6","funder":[{"DOI":"10.13039\/100000030","name":"Centers for Disease Control and Prevention","doi-asserted-by":"publisher","award":["PA\u00a0#96043"],"award-info":[{"award-number":["PA\u00a0#96043"]}],"id":[{"id":"10.13039\/100000030","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000030","name":"Centers for Disease Control and Prevention","doi-asserted-by":"publisher","award":["PA\u00a0#02081"],"award-info":[{"award-number":["PA\u00a0#02081"]}],"id":[{"id":"10.13039\/100000030","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000030","name":"Centers for Disease Control and Prevention","doi-asserted-by":"publisher","award":["FOA #DD09-001"],"award-info":[{"award-number":["FOA #DD09-001"]}],"id":[{"id":"10.13039\/100000030","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000030","name":"Centers for Disease Control and Prevention","doi-asserted-by":"publisher","award":["FOA #DD13-003"],"award-info":[{"award-number":["FOA #DD13-003"]}],"id":[{"id":"10.13039\/100000030","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000030","name":"Centers for Disease Control and Prevention","doi-asserted-by":"publisher","award":["NOFO #DD18-001"],"award-info":[{"award-number":["NOFO #DD18-001"]}],"id":[{"id":"10.13039\/100000030","id-type":"DOI","asserted-by":"publisher"}]},{"name":"National Birth Defects Prevention Study"},{"name":"Birth Defects Study To Evaluate Pregnancy exposureS"},{"DOI":"10.13039\/100000125","name":"National Institute for Occupational Safety and Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000125","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,7,3]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Introduction<\/jats:title>\n                  <jats:p>When it is not possible to capture direct measures of occupational exposure or conduct biomonitoring, retrospective exposure assessment methods are often used. Among the common retrospective assessment methods, assigning exposure estimates by multiple expert rater review of detailed job descriptions is typically the most valid, but also the most time-consuming and expensive. Development of screening protocols to prioritize a subset of jobs for expert rater review can reduce the exposure assessment cost and time requirement, but there is often little data with which to evaluate different screening approaches. We used existing job-by-job exposure assessment data (assigned by consensus between multiple expert raters) from a large, population-based study of women to create and test screening algorithms for polycyclic aromatic hydrocarbons (PAHs) that would be suitable for use in other population-based studies.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>We evaluated three approaches to creating a screening algorithm: a machine-learning algorithm, a set of a priori decision rules created by experts based on features (such as keywords) found in the job description, and a hybrid algorithm incorporating both sets of criteria. All coded jobs held by mothers of infants participating in National Birth Defects Prevention Study (NBDPS) (n = 35,424) were used in developing or testing the screening algorithms. The job narrative fields considered for all approaches included job title, type of product made by the company, main activities or duties, and chemicals or substances handled. Each screening approach was evaluated against the consensus rating of two or more expert raters.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The machine-learning algorithm considered over 30,000 keywords and industry\/occupation codes (separate and in combination). Overall, the hybrid method had a similar sensitivity (87.1%) as the expert decision rules (85.5%) but was higher than the machine-learning algorithm (67.7%). Specificity was best in the machine-learning algorithm (98.1%), compared to the expert decision rules (89.2%) and hybrid approach (89.1%). Using different probability cutoffs in the hybrid approach resulted in improvements in sensitivity (24\u201330%), without the loss of much specificity (7\u201318%).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>Both expert decision rules and the machine-learning algorithm performed reasonably well in identifying the majority of jobs with potential exposure to PAHs. The hybrid screening approach demonstrated that by reviewing approximately 20% of the total jobs, it could identify 87% of all jobs exposed to PAHs; sensitivity could be further increased, albeit with a decrease in specificity, by adjusting the algorithm. The resulting screening algorithm could be applied to other population-based studies of women. The process of developing the algorithm also provides a useful illustration of the strengths and potential pitfalls of these approaches to developing exposure assessment algorithms.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/annweh\/wxab002","type":"journal-article","created":{"date-parts":[[2021,1,7]],"date-time":"2021-01-07T12:18:19Z","timestamp":1610021899000},"page":"682-693","source":"Crossref","is-referenced-by-count":4,"title":["Testing and Validating Semi-automated Approaches to the Occupational Exposure Assessment of Polycyclic Aromatic Hydrocarbons"],"prefix":"10.1093","volume":"65","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2076-721X","authenticated-orcid":false,"given":"Albeliz","family":"Santiago-Col\u00f3n","sequence":"first","affiliation":[{"name":"Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health, Cincinnati, OH, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carissa M","family":"Rocheleau","sequence":"additional","affiliation":[{"name":"Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health, Cincinnati, OH, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephen","family":"Bertke","sequence":"additional","affiliation":[{"name":"Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health, Cincinnati, OH, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Annette","family":"Christianson","sequence":"additional","affiliation":[{"name":"Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health, Cincinnati, OH, USA"},{"name":"Department of Environmental and Public Health Sciences, University of Cincinnati, Cincinnati, OH, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Devon T","family":"Collins","sequence":"additional","affiliation":[{"name":"Department of Epidemiology, University of Kentucky, College of Public Health, Lexington, KY, USA"},{"name":"Inova Fairfax Medical Campus, Falls Church, VA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emma","family":"Trester-Wilson","sequence":"additional","affiliation":[{"name":"Department of Epidemiology, University of Kentucky, College of Public Health, Lexington, KY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wayne","family":"Sanderson","sequence":"additional","affiliation":[{"name":"Department of Epidemiology, University of Kentucky, College of Public Health, Lexington, KY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Martha A","family":"Waters","sequence":"additional","affiliation":[{"name":"Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health, Cincinnati, OH, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennita","family":"Reefhuis","sequence":"additional","affiliation":[{"name":"Centers for Disease Control and Prevention, National Center on Birth Defects and Developmental Disabilities, Atlanta, GA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"name":"National Birth Defects Prevention Study","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2021,4,23]]},"reference":[{"key":"2022021211312518600_CIT0001","volume-title":"Toxicological profile for polycyclic aromatic hydrocarbons (PAHs). Atlanta, GA: US Department of Health and Human Services, Public Health Service","author":"Agency for Toxic Substances and Disease Registry","year":"1995"},{"key":"2022021211312518600_CIT0002","volume-title":"Introduction to machine learning","author":"Alpaydin"},{"key":"2022021211312518600_CIT0003","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1016\/j.jsr.2012.10.012","article-title":"Development and evaluation of a Na\u00efve Bayesian model for coding causation of workers\u2019 compensation claims","volume":"43","author":"Bertke","year":"2012","journal-title":"J Safety Res"},{"key":"2022021211312518600_CIT0004","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1016\/j.aap.2015.12.006","article-title":"Comparison of methods for auto-coding causation of injury narratives","volume":"88","author":"Bertke","year":"2016","journal-title":"Accid Anal Prev"},{"key":"2022021211312518600_CIT0005","doi-asserted-by":"crossref","first-page":"S121","DOI":"10.1093\/ije\/22.Supplement_2.S121","article-title":"Standardization in the retrospective evaluation by experts of occupational exposure to organic solvents in a population-based case-control study","volume":"22","author":"Clavel","year":"1993","journal-title":"Int J Epidemiol"},{"key":"2022021211312518600_CIT0006","doi-asserted-by":"crossref","first-page":"292","DOI":"10.1080\/15459624.2013.862591","article-title":"Retrospective assessment of exposure to chemicals for a microelectronics and business machine manufacturing facility","volume":"11","author":"Fleming","year":"2014","journal-title":"J Occup Environ Hyg"},{"key":"2022021211312518600_CIT0007","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1093\/annweh\/wxy084","article-title":"Inter-rater agreement between exposure assessment using automatic algorithms and using experts","volume":"63","author":"Florath","year":"2019","journal-title":"Ann Work Expo Health"},{"key":"2022021211312518600_CIT0008","first-page":"612","article-title":"Systematically extracting metal- and solvent-related occupational information from free-text responses to lifetime occupational history questionnaires","volume":"58","author":"Friesen","year":"2014","journal-title":"Ann Occup Hyg"},{"key":"2022021211312518600_CIT0009","first-page":"470","article-title":"Comparison of algorithm-based estimates of occupational diesel exhaust exposure to those of multiple independent raters in a population-based case-control study","volume":"57","author":"Friesen","year":"2013","journal-title":"Ann Occup Hyg"},{"key":"2022021211312518600_CIT0010","first-page":"455","article-title":"Using hierarchical cluster models to systematically identify groups of jobs with similar occupational questionnaire response patterns to assist rule-based expert exposure assessment in population-based studies","volume":"59","author":"Friesen","year":"2015","journal-title":"Ann Occup Hyg"},{"key":"2022021211312518600_CIT0011","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1093\/annhyg\/mev095","article-title":"Combining decision rules from classification tree models and expert assessment to estimate occupational exposure to diesel exhaust for a case-control study","volume":"60","author":"Friesen","year":"2016","journal-title":"Ann Occup Hyg"},{"key":"2022021211312518600_CIT0012","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1093\/occmed\/kqy126","article-title":"OccIDEAS \u2013 occupational exposure assessment in community-based studies","volume":"69","author":"Fritschi","year":"2019","journal-title":"Occup Med"},{"key":"2022021211312518600_CIT0013","doi-asserted-by":"crossref","first-page":"957023","DOI":"10.1155\/2009\/957023","article-title":"OccIDEAS: retrospective occupational exposure assessment in community-based studies made easier","volume":"2009","author":"Fritschi","year":"2009","journal-title":"J Environ Public Health"},{"key":"2022021211312518600_CIT0014","doi-asserted-by":"crossref","first-page":"521","DOI":"10.1093\/oxfordjournals.aje.a008959","article-title":"Self-assessed versus expert-assessed occupational exposures","volume":"144","author":"Fritschi","year":"1996","journal-title":"Am J Epidemiol"},{"key":"2022021211312518600_CIT0015","doi-asserted-by":"crossref","first-page":"729","DOI":"10.1080\/15459624.2015.1043049","article-title":"Statistical modeling of occupational exposure to polycyclic aromatic hydrocarbons using OSHA data","volume":"12","author":"Lee","year":"2015","journal-title":"J Occup Environ Hyg"},{"key":"2022021211312518600_CIT0016","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1136\/ip.2008.021337","article-title":"Bayesian methods: a useful tool for classifying injury narratives into cause groups","volume":"15","author":"Lehto","year":"2009","journal-title":"Inj Prev"},{"key":"2022021211312518600_CIT0017","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1016\/j.aap.2015.06.014","article-title":"A practical tool for public health surveillance: Semi-automated coding of short injury narratives from large administrative databases using Na\u00efve Bayes algorithms","volume":"84","author":"Marucci-Wellman","year":"2015","journal-title":"Accid Anal Prev"},{"key":"2022021211312518600_CIT0018","volume-title":"Automated coding of worker injury narratives","author":"Measure","year":"2014"},{"key":"2022021211312518600_CIT0019","volume-title":"OSHA annotated PELs Table Z-1","author":"Occupational Safety and Health Administration (OSHA)","year":"2019"},{"key":"2022021211312518600_CIT0020","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1136\/oemed-2013-101699","article-title":"Rule-based exposure assessment versus case-by-case expert assessment using the same information in a community-based study","volume":"71","author":"Peters","year":"2014","journal-title":"Occup Environ Med"},{"key":"2022021211312518600_CIT0021","doi-asserted-by":"crossref","first-page":"752","DOI":"10.1136\/oemed-2011-100524","article-title":"Comparison of two expert-based assessments of diesel exhaust exposure in a case-control study: programmable decision rules versus expert review of individual jobs","volume":"69","author":"Pronk","year":"2012","journal-title":"Occup Environ Med"},{"key":"2022021211312518600_CIT0022","doi-asserted-by":"crossref","first-page":"656","DOI":"10.1002\/bdra.23384","article-title":"The national birth defects prevention study: a review of the methods","volume":"103","author":"Reefhuis","year":"2015","journal-title":"Birth Defects Res A Clin Mol Teratol"},{"key":"2022021211312518600_CIT0023","first-page":"858","article-title":"Cost-efficient design of occupational exposure assessment strategies\u2013a review","volume":"54","author":"Rezagholi","year":"2010","journal-title":"Ann Occup Hyg"},{"key":"2022021211312518600_CIT0024","doi-asserted-by":"crossref","first-page":"718","DOI":"10.1080\/15459624.2011.627293","article-title":"Inter-rater reliability of assessed prenatal maternal occupational exposures to solvents, polycyclic aromatic hydrocarbons, and heavy metals","volume":"8","author":"Rocheleau","year":"2011","journal-title":"J Occup Environ Hyg"},{"key":"2022021211312518600_CIT0025","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1186\/s12940-019-0451-0","article-title":"A hybrid expert approach for retrospective assessment of occupational exposures in a population-based case-control study of cancer","volume":"18","author":"Sauv\u00e9","year":"2019","journal-title":"Environ Health"},{"key":"2022021211312518600_CIT0026","doi-asserted-by":"crossref","first-page":"575","DOI":"10.1136\/oem.59.9.575","article-title":"Occupational exposure assessment in case-control studies: opportunities for improvement","volume":"59","author":"Teschke","year":"2002","journal-title":"Occup Environ Med"},{"key":"2022021211312518600_CIT0027","doi-asserted-by":"crossref","first-page":"733","DOI":"10.1002\/bdra.23373","article-title":"Next steps for birth defects research and prevention: The birth defects study to evaluate pregnancy exposures (BD-STEPS)","volume":"103","author":"Tinker","year":"2015","journal-title":"Birth Defects Res A Clin Mol Teratol"},{"key":"2022021211312518600_CIT0028","volume-title":"Polycyclic aromatic hydrocarbons (PAHs","author":"U.S. Environmental Protection Agency","year":"2008"},{"key":"2022021211312518600_CIT0029","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1016\/j.aap.2015.03.018","article-title":"Machine learning approaches to analysing textual injury surveillance data: a systematic review","volume":"79","author":"Vallmuur","year":"2015","journal-title":"Accid Anal Prev"},{"key":"2022021211312518600_CIT0030","first-page":"324","article-title":"Comparison of ordinal and nominal classification trees to predict ordinal expert-based occupational exposure estimates in a case-control study","volume":"59","author":"Wheeler","year":"2015","journal-title":"Ann Occup Hyg"},{"key":"2022021211312518600_CIT0031","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1136\/oemed-2012-100918","article-title":"Inside the black box: starting to uncover the underlying decision rules used in a one-by-one expert assessment of occupational exposure in case-control studies","volume":"70","author":"Wheeler","year":"2013","journal-title":"Occup Environ Med"},{"key":"2022021211312518600_CIT0032","doi-asserted-by":"crossref","DOI":"10.1093\/acprof:oso\/9780198509851.001.0001","volume-title":"Principles of exposure measurement in epidemiology","author":"White","year":"2008"},{"key":"2022021211312518600_CIT0033","doi-asserted-by":"crossref","first-page":"i15","DOI":"10.1136\/ip.7.suppl_1.i15","article-title":"Use of narrative analysis for comparisons of the causes of fatal accidents in three countries: New Zealand, Australia, and the United States","volume":"7","author":"Williamson","year":"2001","journal-title":"Inj Prev"},{"key":"2022021211312518600_CIT0034","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1093\/phr\/116.S1.32","article-title":"The National Birth Defects Prevention Study","volume":"116","author":"Yoon","year":"2001","journal-title":"Public Health Rep"}],"container-title":["Annals of Work Exposures and Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/annweh\/article-pdf\/65\/6\/682\/41819258\/wxab002.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/annweh\/article-pdf\/65\/6\/682\/41819258\/wxab002.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,2,12]],"date-time":"2022-02-12T11:32:08Z","timestamp":1644665528000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/annweh\/article\/65\/6\/682\/6247063"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,23]]},"references-count":34,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2021,4,23]]},"published-print":{"date-parts":[[2021,7,3]]}},"URL":"https:\/\/doi.org\/10.1093\/annweh\/wxab002","relation":{},"ISSN":["2398-7308","2398-7316"],"issn-type":[{"value":"2398-7308","type":"print"},{"value":"2398-7316","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,7,1]]},"published":{"date-parts":[[2021,4,23]]}}}