{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,29]],"date-time":"2026-10-29T23:34:42Z","timestamp":1793316882737,"version":"4.5.0"},"reference-count":17,"publisher":"Oxford University Press (OUP)","issue":"11","license":[{"start":{"date-parts":[[2023,9,21]],"date-time":"2023-09-21T00:00:00Z","timestamp":1695254400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,11,2]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Background<\/jats:title>\n                  <jats:p>Measuring parathyroid hormone-related peptide (PTHrP) helps diagnose the humoral hypercalcemia of malignancy, but is often ordered for patients with low pretest probability, resulting in poor test utilization. Manual review of results to identify inappropriate PTHrP orders is a cumbersome process.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>Using a dataset of 1330 patients from a single institute, we developed a machine learning (ML) model to predict abnormal PTHrP results. We then evaluated the performance of the model on two external datasets. Different strategies (model transporting, retraining, rebuilding, and fine-tuning) were investigated to improve model generalizability. Maximum mean discrepancy (MMD) was adopted to quantify the shift of data distributions across different datasets.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The model achieved an area under the receiver operating characteristic curve (AUROC) of 0.936, and a specificity of 0.842 at 0.900 sensitivity in the development cohort. Directly transporting this model to two external datasets resulted in a deterioration of AUROC to 0.838 and 0.737, with the latter having a larger MMD corresponding to a greater data shift compared to the original dataset. Model rebuilding using site-specific data improved AUROC to 0.891 and 0.837 on the two sites, respectively. When external data is insufficient for retraining, a fine-tuning strategy also improved model utility.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>ML offers promise to improve PTHrP test utilization while relieving the burden of manual review. Transporting a ready-made model to external datasets may lead to performance deterioration due to data distribution shift. Model retraining or rebuilding could improve generalizability when there are enough data, and model fine-tuning may be favorable when site-specific data is limited.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/clinchem\/hvad141","type":"journal-article","created":{"date-parts":[[2023,9,22]],"date-time":"2023-09-22T20:47:17Z","timestamp":1695415637000},"page":"1260-1269","source":"Crossref","is-referenced-by-count":20,"title":["Generalizability of a Machine Learning Model for Improving Utilization of Parathyroid Hormone-Related Peptide Testing across Multiple Clinical Centers"],"prefix":"10.1093","volume":"69","author":[{"given":"He S","family":"Yang","sequence":"first","affiliation":[{"name":"Department of Pathology and Laboratory Medicine, Weill Cornell Medicine , New York, NY , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weishen","family":"Pan","sequence":"additional","affiliation":[{"name":"Department of Population Health Sciences, Weill Cornell Medicine , New York, NY , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yingheng","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Cornell University , Ithaca, NY , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mark A","family":"Zaydman","sequence":"additional","affiliation":[{"name":"Department of Pathology and Immunology, Washington University School of Medicine , St. Louis, MO , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5873-351X","authenticated-orcid":false,"given":"Nicholas C","family":"Spies","sequence":"additional","affiliation":[{"name":"Department of Pathology and Immunology, Washington University School of Medicine , St. Louis, MO , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9742-5982","authenticated-orcid":false,"given":"Zhen","family":"Zhao","sequence":"additional","affiliation":[{"name":"Department of Pathology and Laboratory Medicine, Weill Cornell Medicine , New York, NY , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Theresa A","family":"Guise","sequence":"additional","affiliation":[{"name":"Department of Endocrine Neoplasia and Hormonal Disorders, Division of Internal Medicine, The University of Texas, MD Anderson , Houston, TX , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1433-9860","authenticated-orcid":false,"given":"Qing H","family":"Meng","sequence":"additional","affiliation":[{"name":"Department of Laboratory Medicine, The University of Texas MD Anderson Cancer Center , Houston, TX , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fei","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Population Health Sciences, Weill Cornell Medicine , New York, NY , United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2023,9,21]]},"reference":[{"key":"2024091007343772600_hvad141-B1","doi-asserted-by":"crossref","first-page":"675","DOI":"10.1024\/1661-8157\/a003888","article-title":"[Hypercalcemia\u2014diagnosis and management]","volume":"111","author":"Koster","year":"2022","journal-title":"Praxis (Bern 1994)"},{"key":"2024091007343772600_hvad141-B2","doi-asserted-by":"crossref","first-page":"107","DOI":"10.3109\/10408363.2014.970266","article-title":"Laboratory approaches for the diagnosis and assessment of hypercalcemia","volume":"52","author":"Meng","year":"2015","journal-title":"Crit Rev Clin Lab Sci"},{"key":"2024091007343772600_hvad141-B3","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1016\/j.clinbiochem.2022.04.005","article-title":"Development of a pthrp chemiluminescent immunoassay to assess humoral hypercalcemia of malignancy","volume":"105\u2013106","author":"Ashrafzadeh-Kian","year":"2022","journal-title":"Clin Biochem"},{"key":"2024091007343772600_hvad141-B4","doi-asserted-by":"crossref","first-page":"2024","DOI":"10.1210\/jc.2014-4250","article-title":"PTHrP-mediated hypercalcemia: causes and survival in 138 patients","volume":"100","author":"Donovan","year":"2015","journal-title":"J Clin Endocrinol Metab"},{"key":"2024091007343772600_hvad141-B5","doi-asserted-by":"crossref","first-page":"146","DOI":"10.1016\/j.cca.2008.12.040","article-title":"The clinical utility of parathyroid hormone-related peptide in the assessment of hypercalcemia","volume":"402","author":"Fritchie","year":"2009","journal-title":"Clin Chim Acta"},{"key":"2024091007343772600_hvad141-B6","doi-asserted-by":"crossref","first-page":"826","DOI":"10.5858\/arpa.2021-0635-RA","article-title":"Building the model: challenges and considerations of developing and implementing machine learning tools for clinical laboratory medicine practice","volume":"147","author":"Yang","year":"2023;","journal-title":"Arch Pathol Lab Med"},{"key":"2024091007343772600_hvad141-B7","doi-asserted-by":"crossref","first-page":"1640","DOI":"10.1093\/jalm\/jfab075","article-title":"Rise of the machines: artificial intelligence and the clinical laboratory","volume":"6","author":"Haymond","year":"2021","journal-title":"J Appl Lab Med"},{"key":"2024091007343772600_hvad141-B8","author":"Michaud"},{"key":"2024091007343772600_hvad141-B9","doi-asserted-by":"crossref","first-page":"392","DOI":"10.1093\/clinchem\/hvab272","article-title":"How can we ensure reproducibility and clinical translation of machine learning applications in laboratory medicine?","volume":"68","author":"Haymond","year":"2022","journal-title":"Clin Chem"},{"key":"2024091007343772600_hvad141-B10","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1038\/s41746-022-00614-9","article-title":"Machine learning generalizability across healthcare settings: insights from multi-site COVID-19 screening","volume":"5","author":"Yang","year":"2022","journal-title":"NPJ Digit Med"},{"key":"2024091007343772600_hvad141-B11","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1111\/j.2517-6161.1995.tb02031.x","article-title":"Controlling the false discovery rate: A practical and powerful approach to multiple testing","volume":"57","author":"Benjamini","year":"1995","journal-title":"J R Stat Soc Series B Stat Methodol"},{"key":"2024091007343772600_hvad141-B12","article-title":"A unified approach to interpreting model predictions","author":"Lundberg","year":"2017","journal-title":"Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS 2017)"},{"key":"2024091007343772600_hvad141-B13","doi-asserted-by":"crossref","first-page":"5668","DOI":"10.1038\/s41467-020-19551-w","article-title":"Cross-site transportability of an explainable artificial intelligence model for acute kidney injury prediction","volume":"11","author":"Song","year":"2020","journal-title":"Nat Commun"},{"key":"2024091007343772600_hvad141-B14","author":"Smola"},{"key":"2024091007343772600_hvad141-B15","doi-asserted-by":"crossref","first-page":"770","DOI":"10.1055\/a-1049-0647","article-title":"Diagnosis, pathophysiology and management of hypercalcemia in malignancy: A review of the literature","volume":"51","author":"Asonitis","year":"2019","journal-title":"Horm Metab Res"},{"key":"2024091007343772600_hvad141-B16","author":"Kushnir"},{"key":"2024091007343772600_hvad141-B17","doi-asserted-by":"crossref","first-page":"690","DOI":"10.1093\/clinchem\/hvad055","article-title":"Machine learning in laboratory medicine: recommendations of the IFCC working group","volume":"69","author":"Master","year":"2023","journal-title":"Clin Chem"}],"container-title":["Clinical Chemistry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/clinchem\/article-pdf\/69\/11\/1260\/59067477\/hvad141.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/clinchem\/article-pdf\/69\/11\/1260\/59067477\/hvad141.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,10]],"date-time":"2024-09-10T09:22:47Z","timestamp":1725960167000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/clinchem\/article\/69\/11\/1260\/7280268"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,21]]},"references-count":17,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2023,9,21]]},"published-print":{"date-parts":[[2023,11,2]]}},"URL":"https:\/\/doi.org\/10.1093\/clinchem\/hvad141","relation":{},"ISSN":["0009-9147","1530-8561"],"issn-type":[{"value":"0009-9147","type":"print"},{"value":"1530-8561","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,11]]},"published":{"date-parts":[[2023,9,21]]}}}