{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,5]],"date-time":"2026-11-05T21:06:14Z","timestamp":1793912774996,"version":"4.8.0"},"reference-count":35,"publisher":"Springer Science and Business Media LLC","issue":"11","license":[{"start":{"date-parts":[[2025,5,16]],"date-time":"2025-05-16T00:00:00Z","timestamp":1747353600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,5,16]],"date-time":"2025-05-16T00:00:00Z","timestamp":1747353600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Eur Radiol"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>To evaluate how different test set sampling strategies\u2014random selection and balanced sampling\u2014affect the performance of artificial intelligence (AI) models in pediatric wrist fracture detection using radiographs, aiming to highlight the need for standardization in test set design.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and methods<\/jats:title>\n                    <jats:p>This retrospective study utilized the open-sourced GRAZPEDWRI-DX dataset of 6091 pediatric wrist radiographs. Two test sets, each containing 4588 images, were constructed: one using a balanced approach based on case difficulty, projection type, and fracture presence and the other a random selection. EfficientNet and YOLOv11 models were trained and validated on 18,762 radiographs and tested on both sets. Binary classification and object detection tasks were evaluated using metrics such as precision, recall, F1 score, AP50, and AP50-95. Statistical comparisons between test sets were performed using nonparametric tests.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Performance metrics significantly decreased in the balanced test set with more challenging cases. For example, the precision for YOLOv11 models decreased from 0.95 in the random set to 0.83 in the balanced set. Similar trends were observed for recall, accuracy, and F1 score, indicating that models trained on easy-to-recognize cases performed poorly on more complex ones. These results were consistent across all model variants tested.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>AI models for pediatric wrist fracture detection exhibit reduced performance when tested on balanced datasets containing more difficult cases, compared to randomly selected cases. This highlights the importance of constructing representative and standardized test sets that account for clinical complexity to ensure robust AI performance in real-world settings.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Key Points<\/jats:title>\n                    <jats:p>\n                      <jats:bold>\n                        <jats:italic>Question<\/jats:italic>\n                      <\/jats:bold>\n                      <jats:italic>Do different sampling strategies based on samples\u2019 complexity have an influence in deep learning models\u2019 performance in fracture detection?<\/jats:italic>\n                    <\/jats:p>\n                    <jats:p>\n                      <jats:bold>\n                        <jats:italic>Findings<\/jats:italic>\n                      <\/jats:bold>\n                      <jats:italic>AI performance in pediatric wrist fracture detection significantly drops when tested on balanced datasets with more challenging cases, compared to randomly selected cases.<\/jats:italic>\n                    <\/jats:p>\n                    <jats:p>\n                      <jats:bold>\n                        <jats:italic>Clinical relevance<\/jats:italic>\n                      <\/jats:bold>\n                      <jats:italic>Without standardized and validated test datasets for AI that reflect clinical complexities, performance metrics may be overestimated, limiting the utility of AI in real-world settings.<\/jats:italic>\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Graphical Abstract<\/jats:title>\n                  <\/jats:sec>","DOI":"10.1007\/s00330-025-11669-z","type":"journal-article","created":{"date-parts":[[2025,5,16]],"date-time":"2025-05-16T12:57:31Z","timestamp":1747400251000},"page":"6853-6864","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Impact of test set composition on AI performance in pediatric wrist fracture detection in X-rays"],"prefix":"10.1007","volume":"35","author":[{"given":"Tristan","family":"Till","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mario","family":"Scherkl","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-7274-9403","authenticated-orcid":false,"given":"Nikolaus","family":"Stranger","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Georg","family":"Singer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Saskia","family":"Hankel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christina","family":"Flucher","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Franko","family":"Hr\u017ei\u0107","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ivan","family":"\u0160tajduhar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sebastian","family":"Tschauner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,5,16]]},"reference":[{"key":"11669_CR1","doi-asserted-by":"publisher","first-page":"9638","DOI":"10.1007\/s00330-021-08035-0","volume":"31","author":"EJ Kempen","year":"2021","unstructured":"Kempen EJ, Post M, Mannil M et al (2021) Performance of machine learning algorithms for glioma segmentation of brain MRI: a systematic literature review and metaanalysis. Eur Radiol 31:9638\u20139653. https:\/\/doi.org\/10.1007\/s00330-021-08035-0","journal-title":"Eur Radiol"},{"key":"11669_CR2","doi-asserted-by":"publisher","unstructured":"Freeman K, Geppert J, Stinton C et al (2021) Use of artificial intelligence for image analysis in breast cancer screening programmes: systematic review of test accuracy. BMJ 1872. https:\/\/doi.org\/10.1136\/bmj.n1872","DOI":"10.1136\/bmj.n1872"},{"key":"11669_CR3","doi-asserted-by":"publisher","first-page":"487","DOI":"10.1148\/radiol.2019192515","volume":"294","author":"DA Bluemke","year":"2020","unstructured":"Bluemke DA, Moy L, Bredella MA et al (2020) Assessing radiology research on artificial intelligence: a brief guide for authors, reviewers, and readers\u2014from the radiology editorial board. Radiology 294:487\u2013489. https:\/\/doi.org\/10.1148\/radiol.2019192515","journal-title":"Radiology"},{"key":"11669_CR4","doi-asserted-by":"publisher","unstructured":"Armato SG, Drukker K, Hadjiiski L (2023) AI in medical imaging grand challenges: translation from competition to research benefit and patient care. Br J Radiol 96. https:\/\/doi.org\/10.1259\/bjr.20221152","DOI":"10.1259\/bjr.20221152"},{"key":"11669_CR5","unstructured":"US Food and Drug Administration et al (2024) Artificial intelligence & medical products: how CBER, CDER, CDRH, and OCP are Working Together. US Food and Drug Administration"},{"key":"11669_CR6","unstructured":"European Medicines Agency (2023) Reflection paper on the use of artificial intelligence (AI) in the medicinal product lifecycle. Draft Reflection Paper, EMA\/CHMP\/CVMP\/83833\/2023. https:\/\/www.ema.europa.eu"},{"key":"11669_CR7","doi-asserted-by":"publisher","first-page":"627","DOI":"10.1148\/radiol.210937","volume":"302","author":"A Guermazi","year":"2022","unstructured":"Guermazi A, Tannoury C, Kompel AJ et al (2022) Improving radiographic fracture recognition performance and efficiency using artificial intelligence. Radiology 302:627\u2013636. https:\/\/doi.org\/10.1148\/radiol.210937","journal-title":"Radiology"},{"key":"11669_CR8","doi-asserted-by":"publisher","first-page":"2215","DOI":"10.1007\/s00247-022-05496-3","volume":"52","author":"T Nguyen","year":"2022","unstructured":"Nguyen T, Maarek R, Hermann A-L et al (2022) Assessment of an artificial intelligence aid for the detection of appendicular skeletal fractures in children and young adults by senior and junior radiologists. Pediatr Radiol 52:2215\u20132226. https:\/\/doi.org\/10.1007\/s00247-02205496-3","journal-title":"Pediatr Radiol"},{"key":"11669_CR9","doi-asserted-by":"publisher","first-page":"1675","DOI":"10.1007\/s00247-023-05621-w","volume":"53","author":"I Gasmi","year":"2023","unstructured":"Gasmi I, Calinghen A, Parienti J-J, Belloy F, Fohlen A, Pelage J-P (2023) Comparison of diagnostic performance of a deep learning algorithm, emergency physicians, junior radiologists and senior radiologists in the detection of appendicular fractures in children. Pediatr Radiol 53:1675\u20131684. https:\/\/doi.org\/10.1007\/s00247-023-05621-w","journal-title":"Pediatr Radiol"},{"key":"11669_CR10","doi-asserted-by":"publisher","unstructured":"Jacques T, Cardot N, Ventre J, Demondion X, Cotten A (2023) Commercially available AI algorithm improves radiologists\u2019 sensitivity for wrist and hand fracture detection on X-ray, compared to a ct-based ground truth. Eur Radiol. https:\/\/doi.org\/10.1007\/s00330-023-10380-1","DOI":"10.1007\/s00330-023-10380-1"},{"key":"11669_CR11","doi-asserted-by":"crossref","unstructured":"Till T, Tschauner S, Singer G, Lichtenegger K, Till H (2023) Development and optimization of AI algorithms for wrist fracture detection in children using a freely available dataset. Front Pediatr 11. 10.3389\/fped. 2023.1291804","DOI":"10.3389\/fped.2023.1291804"},{"key":"11669_CR12","doi-asserted-by":"publisher","unstructured":"Ju R-Y, Cai W (2023) Fracture detection in pediatric wrist trauma X-ray images using YOLOv8 algorithm. Sci Rep 13. https:\/\/doi.org\/10.1038\/s41598-023-47460-7","DOI":"10.1038\/s41598-023-47460-7"},{"key":"11669_CR13","doi-asserted-by":"publisher","unstructured":"Janisch M, Apfaltrer G, Hr\u017ei\u0107 F et al (2022) Pediatric radius torus fractures in x-rays\u2014how computer vision could render lateral projections obsolete. Front Pediatr 10. https:\/\/doi.org\/10.3389\/fped.2022.1005099","DOI":"10.3389\/fped.2022.1005099"},{"key":"11669_CR14","doi-asserted-by":"publisher","unstructured":"Oka K, Shiode R, Yoshii Y, Tanaka H, Iwahashi T, Murase T (2021) Artificial intelligence to diagnosis distal radius fracture using biplane plain X-rays. J Orthopaed Surg Res 16. https:\/\/doi.org\/10.1186\/s13018-021-02845-0","DOI":"10.1186\/s13018-021-02845-0"},{"key":"11669_CR15","doi-asserted-by":"publisher","unstructured":"Raisuddin AM, Vaattovaara E, Nevalainen M et al (2021) Critical evaluation of deep neural networks for wrist fracture detection. Sci Rep 11. https:\/\/doi.org\/10.1038\/s41598-021-85570-2","DOI":"10.1038\/s41598-021-85570-2"},{"key":"11669_CR16","doi-asserted-by":"publisher","first-page":"2939","DOI":"10.3390\/math10162939","volume":"10","author":"F Hr\u017ei\u0107","year":"2022","unstructured":"Hr\u017ei\u0107 F, Tschauner S, Sorantin E, \u0160tajduhar I (2022) Fracture recognition in paediatric wrist radiographs: An object detection approach. Mathematics 10:2939. https:\/\/doi.org\/10.3390\/math10162939","journal-title":"Mathematics"},{"key":"11669_CR17","doi-asserted-by":"publisher","first-page":"2118","DOI":"10.1016\/j.acra.2023.06.016","volume":"30","author":"V Bousson","year":"2023","unstructured":"Bousson V, Attan\u00e9 G, Benoist N et al (2023) Artificial intelligence for detecting acute fractures in patients admitted to an emergency department: Real-life performance of three commercial algorithms. Acad Radiol 30:2118\u20132139. https:\/\/doi.org\/10.1016\/j.acra.2023.06.016","journal-title":"Acad Radiol"},{"key":"11669_CR18","doi-asserted-by":"publisher","first-page":"1989","DOI":"10.1016\/j.acra.2023.10.042","volume":"31","author":"T Fu","year":"2024","unstructured":"Fu T, Viswanathan V, Attia A et al (2024) Assessing the potential of a deep learning tool to improve fracture detection by radiologists and emergency physicians on extremity radiographs. Acad Radiol 31:1989\u20131999. https:\/\/doi.org\/10.1016\/j.acra.2023.10.042","journal-title":"Acad Radiol"},{"key":"11669_CR19","doi-asserted-by":"publisher","first-page":"104679","DOI":"10.1016\/j.ijmedinf.2021.104679","volume":"159","author":"EM Nwanosike","year":"2022","unstructured":"Nwanosike EM, Conway BR, Merchant HA, Hasan SS (2022) Potential applications and performance of machine learning techniques and algorithms in clinical practice: a systematic review. Int J Med Inform 159:104679. https:\/\/doi.org\/10.1016\/j.ijmedinf.2021.104679","journal-title":"Int J Med Inform"},{"key":"11669_CR20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1749603.1749605","volume":"42","author":"BCM Fung","year":"2010","unstructured":"Fung BCM, Wang K, Chen R, Yu PS (2010) Privacy-preserving data publishing: a survey of recent developments. ACM Comput Surv 42:1\u201353. https:\/\/doi.org\/10.1145\/1749603.1749605","journal-title":"ACM Comput Surv"},{"key":"11669_CR21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0224365","volume":"14","author":"A Vabalas","year":"2019","unstructured":"Vabalas A, Gowen E, Poliakoff E, Casson AJ (2019) Machine learning algorithm validation with a limited sample size. PLoS One 14:1\u201320. https:\/\/doi.org\/10.1371\/journal.pone.0224365","journal-title":"PLoS One"},{"key":"11669_CR22","unstructured":"Pecher B, Srba I, Bielikova M, Vanschoren J (2024) Automatic combination of sample selection strategies for few-shot learning. http:\/\/arxiv.org\/abs\/2402.03038"},{"key":"11669_CR23","doi-asserted-by":"publisher","unstructured":"Nagy E, Janisch M, Hr\u017ei\u0107 F, Sorantin E, Tschauner S (2022) A pediatric wrist trauma x-ray dataset (GRAZPEDWRI-DX) for machine learning. Sci Data 9. https:\/\/doi.org\/10.1038\/s41597-022-01328-z","DOI":"10.1038\/s41597-022-01328-z"},{"key":"11669_CR24","doi-asserted-by":"publisher","first-page":"106902","DOI":"10.1016\/j.engappai.2023.106902","volume":"126","author":"R Raza","year":"2023","unstructured":"Raza R, Zulfiqar F, Khan MO et al (2023) Lung-effnet: lung cancer classification using efficientnet from CT-scan images. Eng Appl Artif Intell 126:106902. https:\/\/doi.org\/10.1016\/j.engappai.2023.106902","journal-title":"Eng Appl Artif Intell"},{"key":"11669_CR25","doi-asserted-by":"publisher","first-page":"100034","DOI":"10.1016\/j.neuri.2021.100034","volume":"2","author":"K Ali","year":"2022","unstructured":"Ali K, Shaikh ZA, Khan AA, Laghari AA (2022) Multiclass skin cancer classification using efficientnets\u2013a first step towards preventing skin cancer. Neurosci Inform 2:100034. https:\/\/doi.org\/10.1016\/j.neuri.2021.100034","journal-title":"Neurosci Inform"},{"key":"11669_CR26","doi-asserted-by":"publisher","DOI":"10.1186\/s12880-024-01404-3","volume":"24","author":"M Latha","year":"2024","unstructured":"Latha M, Kumar PS, Chandrika RR, Mahesh T, Kumar VV, Guluwadi S (2024) Revolutionizing breast ultrasound diagnostics with efficientnet-b7 and explainable ai. BMC Med Imaging 24:230","journal-title":"BMC Med Imaging"},{"key":"11669_CR27","unstructured":"Tan M, Le Q (2019) Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning. pp. 6105\u20136114. PMLR."},{"key":"11669_CR28","unstructured":"Howard J et al (2018) fastai. GitHub"},{"key":"11669_CR29","doi-asserted-by":"publisher","unstructured":"Ju R-Y, Chien C-T, Lin C-M, Chiang J-S (2024) Global context modeling in YOLOv8 for pediatric wrist fracture detection. International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS), Kaohsiung, Taiwan, pp. 1\u20135. https:\/\/doi.org\/10.1109\/ISPACS62486.2024.10869064","DOI":"10.1109\/ISPACS62486.2024.10869064"},{"key":"11669_CR30","unstructured":"Jocher G, Chaurasia A, Qiu J (2024) Ultralytics YOLO. https:\/\/github.com\/ultralytics\/ultralytics"},{"key":"11669_CR31","unstructured":"Jocher G et al (2024) Ultralytics: YOLO Performance Metrics\u2014docs.ultralytics.com. https:\/\/docs.ultralytics.com\/guides\/yolo-performance-metrics\/#introduction. Accessed 29-Aug-2024"},{"key":"11669_CR32","doi-asserted-by":"publisher","unstructured":"Tian J, Jin Q, Wang Y, Yang J, Zhang S, Sun D (2024) Performance analysis of deep learning-based object detection algorithms on coco benchmark: a comparative study. J Eng Appl Sci 71. https:\/\/doi.org\/10.1186\/s44147-024-00411-z","DOI":"10.1186\/s44147-024-00411-z"},{"key":"11669_CR33","doi-asserted-by":"publisher","unstructured":"Rezatofighi H, Tsoi N, Gwak J, Sadeghian A, Reid I, Savarese S (2019) Generalized intersection over union: a metric and a loss for bounding box regression. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA, USA: IEEE. pp. 658\u2013666. https:\/\/doi.org\/10.1109\/CVPR.2019.00075","DOI":"10.1109\/CVPR.2019.00075"},{"key":"11669_CR34","doi-asserted-by":"publisher","first-page":"136","DOI":"10.1007\/s00247-023-05822-3","volume":"54","author":"I Altmann-Schneider","year":"2023","unstructured":"Altmann-Schneider I, Kellenberger CJ, Pistorius S-M et al (2023) Artificial intelligence-based detection of paediatric appendicular skeletal fractures: performance and limitations for common fracture types and locations. Pediatr Radiol 54:136\u2013145. https:\/\/doi.org\/10.1007\/s00247-023-05822-3","journal-title":"Pediatr Radiol"},{"key":"11669_CR35","doi-asserted-by":"publisher","unstructured":"Karimi-Rouzbahani H, Bagheri N, Ebrahimpour R (2017) Invariant object recognition is a personalized selection of invariant features in humans, not simply explained by hierarchical feed-forward vision models. Sci Rep 7. https:\/\/doi.org\/10.1038\/s41598-017-13756-8","DOI":"10.1038\/s41598-017-13756-8"}],"container-title":["European Radiology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00330-025-11669-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00330-025-11669-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00330-025-11669-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,28]],"date-time":"2025-10-28T05:02:27Z","timestamp":1761627747000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00330-025-11669-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,16]]},"references-count":35,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2025,11]]}},"alternative-id":["11669"],"URL":"https:\/\/doi.org\/10.1007\/s00330-025-11669-z","relation":{},"ISSN":["1432-1084"],"issn-type":[{"value":"1432-1084","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,16]]},"assertion":[{"value":"1 November 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 February 2025","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 April 2025","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 May 2025","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Compliance with ethical standards"}},{"value":"The scientific guarantor of this publication is Nikolaus Stranger.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Guarantor"}},{"value":"The authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"One of the authors has significant statistical expertise.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Statistics and biometry"}},{"value":"Written informed consent was waived by the Institutional Review Board.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed consent"}},{"value":"Institutional Review Board approval was obtained.","order":6,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}},{"value":"This study used the open-sourced GRAZPEDWRI-DX dataset of 6091 pediatric wrist radiographs.","order":7,"name":"Ethics","group":{"name":"EthicsHeading","label":"Study subjects or cohorts overlap"}},{"order":8,"name":"Ethics","group":{"name":"EthicsHeading","label":"Methodology"}}]}}