{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,19]],"date-time":"2026-08-19T17:15:49Z","timestamp":1787159749715,"version":"build-2736575974"},"reference-count":36,"publisher":"Wiley","issue":"3","license":[{"start":{"date-parts":[[2025,1,24]],"date-time":"2025-01-24T00:00:00Z","timestamp":1737676800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001786","name":"University of Adelaide","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001786","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100015539","name":"Australian Government","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100015539","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["obgyn.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Ultrasound in Obstet &amp;amp; Gyne"],"published-print":{"date-parts":[[2025,3]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>The development of valuable artificial intelligence (AI) tools to assist with ultrasound diagnosis depends on algorithms developed using high\u2010quality data. This study aimed to test the intra\u2010 and interobserver agreement of a proposed image\u2010quality scoring system to quantify the quality of gynecological transvaginal ultrasound (TVS) images, which could be used in clinical practice and AI tool development.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>A proposed scoring system to quantify TVS image quality was created following a review of the literature. This system involved a score of 1\u20134 (2\u2009=\u2009poor, 3\u2009=\u2009suboptimal  and 4\u2009=\u2009optimal image quality) assigned by a rater for individual ultrasound images. If the image was deemed inaccurate, it was assigned a score of 1, corresponding to \u2018reject\u2019. Six professionals, including two radiologists, two sonographers and two sonologists, reviewed 150 images (50 images of the uterus and 100 images of the ovaries) obtained from 50 women, assigning each image a score of 1\u20134. The review of all images was repeated a second time by each rater after a period of at least 1\u2009week. Mean scores were calculated for each rater. Overall interobserver agreement was assessed using intraclass correlation coefficient (ICC), and interobserver agreement between paired professionals and intraobserver agreement for all professionals were assessed using weighted Cohen's kappa and ICC.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      Poor levels of interobserver agreement were obtained between the six raters for all 150 images (ICC, 0.480 (95%\u2009CI, 0.363\u20130.586)), as well as for assessment of the uterine images only (ICC, 0.359 (95%\u2009CI, 0.204\u20130.523)). Moderate agreement was achieved for the ovarian images (ICC, 0.531 (95%\u2009CI, 0.417\u20130.636)). Agreement between the paired sonographers and sonologists was poor for all images (ICC, 0.336 (95%\u2009CI, \u22120.078 to 0.619) and 0.425 (95%\u2009CI, 0.014\u20130.665), respectively), as well as when images were grouped into uterine images (ICC, 0.253 (95%\u2009CI, \u22120.097 to 0.577) and 0.299 (95%\u2009CI, \u22120.094 to 0.606), respectively) and ovarian images (ICC, 0.400 (95%\u2009CI, \u22120.043 to 0.669) and 0.469 (95%\u2009CI, 0.088\u20130.689), respectively). Agreement between the paired radiologists was moderate for all images (ICC, 0.600 (95%\u2009CI, 0.487\u20130.693)) and for their assessment of uterine images (ICC, 0.538 (95%\u2009CI, 0.311\u20130.707)) and ovarian images (ICC, 0.621 (95%\u2009CI, 0.483\u20130.728)). Weak\u2010to\u2010moderate intraobserver agreement was seen for each of the raters  with weighted Cohen's kappa ranging from 0.533 to 0.718 for all images and from 0.467 to 0.751 for ovarian images. Similarly, for all raters, the ICC indicated moderate\u2010to\u2010good intraobserver agreement for all images overall (ICC ranged from 0.636 to 0.825) and for ovarian images (ICC ranged from 0.596 to 0.862). Slightly better intraobserver agreement was seen for uterine images, with weighted Cohen's kappa ranging from 0.568 to 0.808 indicating weak\u2010to\u2010strong agreement, and ICC ranging from 0.546 to 0.893 indicating moderate\u2010to\u2010good agreement. All measures were statistically significant (\n                      <jats:italic>P<\/jats:italic>\n                      \u2009&lt;\u20090.001).\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>\n                      The proposed image quality scoring system was shown to have poor\u2010to\u2010moderate interobserver agreement and mostly weak\u2010to\u2010moderate levels of intraobserver agreement. More refinement of the scoring system may be needed to improve agreement, although it remains unclear whether quantification of image quality can be achieved, given the highly subjective nature of ultrasound interpretation. Although some AI systems can tolerate labeling noise, most will favor clean (high\u2010quality) data. As such, innovative data\u2010labeling strategies are needed. \u00a9 2025 The Author(s).\n                      <jats:italic>Ultrasound in Obstetrics &amp; Gynecology<\/jats:italic>\n                      published by John Wiley &amp; Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.\n                    <\/jats:p>\n                  <\/jats:sec>","DOI":"10.1002\/uog.29178","type":"journal-article","created":{"date-parts":[[2025,1,24]],"date-time":"2025-01-24T16:00:01Z","timestamp":1737734401000},"page":"364-371","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Intra\u2010 and interobserver agreement of proposed objective transvaginal ultrasound image\u2010quality scoring system for use in artificial intelligence algorithm development"],"prefix":"10.1002","volume":"65","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7094-3950","authenticated-orcid":false,"given":"A.","family":"Deslandes","sequence":"first","affiliation":[{"name":"Robinson Research Institute University of Adelaide  Adelaide Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"J. C.","family":"Avery","sequence":"additional","affiliation":[{"name":"Robinson Research Institute University of Adelaide  Adelaide Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"H.\u2010T.","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Computer and Mathematical Sciences University of Adelaide  Adelaide Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5538-6906","authenticated-orcid":false,"given":"M.","family":"Leonardi","sequence":"additional","affiliation":[{"name":"Robinson Research Institute University of Adelaide  Adelaide Australia"},{"name":"Department of Obstetrics and Gynecology McMaster University  Hamilton ON Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"S.","family":"Knox","sequence":"additional","affiliation":[{"name":"Benson Radiology  Adelaide Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"G.","family":"Lo","sequence":"additional","affiliation":[{"name":"Sir Charles Gairdner Hospital  Nedlands Western Australia Australia"},{"name":"Curtin University  Bentley Western Australia Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"R.","family":"O'Hara","sequence":"additional","affiliation":[{"name":"Robinson Research Institute University of Adelaide  Adelaide Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"G.","family":"Condous","sequence":"additional","affiliation":[{"name":"Robinson Research Institute University of Adelaide  Adelaide Australia"},{"name":"Endometriosis Ultrasound &amp; Advanced Endosurgery Unit, Sydney Medical School Nepean University of Sydney, Nepean Hospital  Sydney Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"M. L.","family":"Hull","sequence":"additional","affiliation":[{"name":"Robinson Research Institute University of Adelaide  Adelaide Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"name":"Collaborators","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2025,1,24]]},"reference":[{"key":"e_1_2_7_2_1","doi-asserted-by":"publisher","DOI":"10.3390\/biomedicines9070720"},{"key":"e_1_2_7_3_1","doi-asserted-by":"publisher","DOI":"10.1002\/jum.15765"},{"key":"e_1_2_7_4_1","doi-asserted-by":"publisher","DOI":"10.1002\/uog.22122"},{"key":"e_1_2_7_5_1","doi-asserted-by":"publisher","DOI":"10.1097\/TA.0000000000000639"},{"key":"e_1_2_7_6_1","doi-asserted-by":"publisher","DOI":"10.7863\/ultra.15.07083"},{"key":"e_1_2_7_7_1","doi-asserted-by":"publisher","DOI":"10.1002\/uog.3913"},{"key":"e_1_2_7_8_1","doi-asserted-by":"publisher","DOI":"10.1590\/0100-3984.2017.0021"},{"key":"e_1_2_7_9_1","doi-asserted-by":"publisher","DOI":"10.7863\/jum.2011.30.12.1649"},{"key":"e_1_2_7_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jemermed.2015.01.010"},{"key":"e_1_2_7_11_1","doi-asserted-by":"publisher","DOI":"10.1002\/uog.19101"},{"key":"e_1_2_7_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejogrb.2008.12.003"},{"key":"e_1_2_7_13_1","doi-asserted-by":"publisher","DOI":"10.1002\/uog.2665"},{"key":"e_1_2_7_14_1","doi-asserted-by":"publisher","DOI":"10.37015\/AUDT.2021.210003"},{"key":"e_1_2_7_15_1","unstructured":"LiuRB BlaivasM MooreC et al.Emergency Ultrasound Standard Reporting Guidelines. Published online2018https:\/\/www.acep.org\/globalassets\/uploads\/uploaded\u2010files\/acep\/clinical\u2010and\u2010practice\u2010management\/policy\u2010statements\/information\u2010papers\/emergency\u2010ultrasound\u2010standard\u2010reporting\u2010guidelines\u2010\u2010\u20102018.pdf"},{"key":"e_1_2_7_16_1","doi-asserted-by":"publisher","DOI":"10.1097\/AOG.0000000000001820"},{"key":"e_1_2_7_17_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0057687"},{"key":"e_1_2_7_18_1","doi-asserted-by":"publisher","DOI":"10.1002\/jum.16095"},{"key":"e_1_2_7_19_1","doi-asserted-by":"publisher","DOI":"10.1002\/uog.13198"},{"key":"e_1_2_7_20_1","doi-asserted-by":"publisher","DOI":"10.3390\/app13063693"},{"key":"e_1_2_7_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA48891.2023.10160435"},{"key":"e_1_2_7_22_1","unstructured":"LuoH LiuH LiK ZhangB.Automatic quality assessment for 2D fetal sonographic standard plane based on multi\u2010task learning. Published online December 11 2019."},{"key":"e_1_2_7_23_1","doi-asserted-by":"publisher","DOI":"10.1155\/2021\/9938367"},{"key":"e_1_2_7_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10554-020-02046-6"},{"key":"e_1_2_7_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICIP42928.2021.9506431"},{"key":"e_1_2_7_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcm.2016.02.012"},{"key":"e_1_2_7_27_1","doi-asserted-by":"publisher","DOI":"10.11613\/BM.2012.031"},{"key":"e_1_2_7_28_1","unstructured":"ArifinW.wnarifin.github.io > Sample size calculator. Sample Size Calculator.2024Accessed June 8 2023.https:\/\/wnarifin.github.io\/ssc\/ssicc.html"},{"key":"e_1_2_7_29_1","doi-asserted-by":"publisher","DOI":"10.1111\/coa.12772"},{"key":"e_1_2_7_30_1","doi-asserted-by":"publisher","DOI":"10.1111\/medu.12621"},{"key":"e_1_2_7_31_1","doi-asserted-by":"publisher","DOI":"10.1002\/uog.13376"},{"key":"e_1_2_7_32_1","doi-asserted-by":"publisher","DOI":"10.1002\/uog.26661"},{"key":"e_1_2_7_33_1","doi-asserted-by":"publisher","DOI":"10.1002\/uog.6323"},{"key":"e_1_2_7_34_1","doi-asserted-by":"publisher","DOI":"10.1046\/j.1469-0705.1998.12060398.x"},{"key":"e_1_2_7_35_1","doi-asserted-by":"publisher","DOI":"10.1111\/1471-0528.12315"},{"key":"e_1_2_7_36_1","doi-asserted-by":"crossref","unstructured":"WangH ButlerD ZhangY et al.Human\u2010AI Collaborative Multi\u2010modal Multi\u2010rater Learning for Endometriosis Diagnosis. Published online September 5 2024.","DOI":"10.1088\/1361-6560\/ad997e"},{"key":"e_1_2_7_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISBI45749.2020.9098666"}],"container-title":["Ultrasound in Obstetrics &amp; Gynecology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/obgyn.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/uog.29178","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,13]],"date-time":"2025-12-13T16:08:17Z","timestamp":1765642097000},"score":1,"resource":{"primary":{"URL":"https:\/\/obgyn.onlinelibrary.wiley.com\/doi\/10.1002\/uog.29178"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,24]]},"references-count":36,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,3]]}},"alternative-id":["10.1002\/uog.29178"],"URL":"https:\/\/doi.org\/10.1002\/uog.29178","archive":["Portico"],"relation":{},"ISSN":["0960-7692","1469-0705"],"issn-type":[{"value":"0960-7692","type":"print"},{"value":"1469-0705","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,24]]},"assertion":[{"value":"2024-07-02","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-23","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-01-24","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}