{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T18:44:10Z","timestamp":1785437050137,"version":"3.56.0"},"reference-count":40,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2026,6,29]],"date-time":"2026-06-29T00:00:00Z","timestamp":1782691200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003648","name":"National Medical Center","doi-asserted-by":"crossref","award":["I_2024_0130001"],"award-info":[{"award-number":["I_2024_0130001"]}],"id":[{"id":"10.13039\/501100003648","id-type":"DOI","asserted-by":"crossref"}]},{"award":["I_2024_0130001"],"award-info":[{"award-number":["I_2024_0130001"]}],"id":[{"id":"https:\/\/ror.org\/04pqpfz42","id-type":"ROR","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Bioengineering"],"abstract":"<jats:p>Early diagnosis of Diabetic Retinopathy (DR) is critical for preventing irreversible vision loss, but precise lesion annotation by ophthalmologists is the dominant cost in building any clinical-grade DR detection model. The structural problem in real hospital settings is not labeling cost per se, but expert availability: ophthalmologists\u2019 time is bounded by clinical duties, so the active-learning (AL) cycle can iterate only a handful of times in practice. We frame this constraint explicitly and ask which AL designs work best under a tight expert budget. We propose Virtuous Cycle, a Human-in-the-Loop (HITL) pipeline that integrates (i) a YOLOv8x-based object detector for microaneurysms, hemorrhages, and exudates, (ii) four AL sampling strategies (Average Confidence, Random, Hybrid-Diversity, Monte Carlo Dropout), and (iii) an in-hospital annotation platform (Diavision Studio) in which clinicians refine AI pre-labels rather than draw from scratch. We evaluate Virtuous Cycle on a real-world fundus dataset from the National Medical Center (NMC) across eight AL rounds, expanding the labeled pool from 81 images (R0) to 481 images (R8) within the actual expert-time budget of two ophthalmologists. Across three independent random seeds, random sampling dominates at cold start (mean mAP@50 0.14\u21920.25 over R0\u2013R1), whereas Hybrid-Diversity converges to the highest mAP@50, Precision, and Recall by R7 (431 images; mAP@50 0.40, Precision 0.55, Recall 0.41), with MC Dropout close behind; by R8, the labeled pool is exhausted and all strategies converge to the same final model. A clinician crossover analysis of 36 paired clinical images, controlling for per-clinician speed bias and per-image difficulty bias, shows no statistically significant difference in overall per-image labeling time between AI-assisted and manual annotation (p=0.52), but a statistically significant increase in confirmed lesion detections under AI assistance (p=0.0058), driven predominantly (84\u2013100% of the net increase) by microaneurysms, the lesion type most prone to being missed unaided. The results indicate that, under expert-budget constraints, AL strategy choice should be staged: random sampling for cold start, uncertainty-and-diversity sampling once the model has matured, and that AI assistance trades a modest, lesion-burden-dependent time cost for a measurable gain in the sensitivity of microaneurysm detection.<\/jats:p>","DOI":"10.3390\/bioengineering13070762","type":"journal-article","created":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T05:58:04Z","timestamp":1782799084000},"page":"762","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Active Learning Under Expert-Budget Constraints: A Human-in-the-Loop Pipeline for Diabetic Retinopathy Lesion Detection"],"prefix":"10.3390","volume":"13","author":[{"given":"Hyeok","family":"Kim","sequence":"first","affiliation":[{"name":"Department of Industrial Engineering, Seoul National University of Science and Technology, 232 Gongneung-ro, Nowon-gu, Seoul 01811, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Seok-Min","family":"Chang","sequence":"additional","affiliation":[{"name":"Department of Industrial Engineering, Seoul National University of Science and Technology, 232 Gongneung-ro, Nowon-gu, Seoul 01811, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3325-7545","authenticated-orcid":false,"given":"Bo-Young","family":"Lim","sequence":"additional","affiliation":[{"name":"Department of Data Science, Seoul National University of Science and Technology, 232 Gongneung-ro, Nowon-gu, Seoul 01811, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Soo Young","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, National Medical Center, Seoul 04564, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-1672-5178","authenticated-orcid":false,"given":"Ho-Gil","family":"Jung","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, National Medical Center, Seoul 04564, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,6,29]]},"reference":[{"key":"ref_1","unstructured":"World Health Organization (2023). Diabetes Country Profiles 2023; Report, World Health Organization."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"556","DOI":"10.2337\/dc11-1909","article-title":"Global prevalence and major risk factors of diabetic retinopathy","volume":"35","author":"Yau","year":"2012","journal-title":"Diabetes Care"},{"key":"ref_3","unstructured":"IDF Diabetes Atlas 10th Edition Scientific Committee (2021). IDF Diabetes Atlas, International Diabetes Federation. [10th ed.]."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"e31","DOI":"10.3346\/jkms.2026.41.e31","article-title":"Nationwide Trends and Future Projections of Diabetes and Diabetic Retinopathy Prevalence in Korea: Korean National Health and Nutrition Examination Survey Study","volume":"41","author":"Kim","year":"2026","journal-title":"J. Korean Med. Sci."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"421","DOI":"10.1038\/s41467-023-44676-z","article-title":"Autonomous artificial intelligence increases screening and follow-up for diabetic retinopathy in youth: The ACCESS randomized control trial","volume":"15","author":"Wolf","year":"2024","journal-title":"Nat. Commun."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"584","DOI":"10.1038\/s41591-023-02702-z","article-title":"A deep learning system for predicting time to progression of diabetic retinopathy","volume":"30","author":"Dai","year":"2024","journal-title":"Nat. Med."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"100330","DOI":"10.1016\/j.xops.2023.100330","article-title":"AI-Human Hybrid Workflow Enhances Teleophthalmology for the Detection of Diabetic Retinopathy","volume":"3","author":"Dow","year":"2023","journal-title":"Ophthalmol. Sci."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Xu, X., Zhang, M., Huang, S., Li, X., Kui, X., and Liu, J. (2024). The application of artificial intelligence in diabetic retinopathy: Progress and prospects. Front. Cell Dev. Biol., 12.","DOI":"10.3389\/fcell.2024.1473176"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"102830","DOI":"10.1016\/j.artmed.2024.102830","article-title":"Trustworthy clinical AI solutions: A unified review of uncertainty quantification in Deep Learning models for medical image analysis","volume":"150","author":"Lambert","year":"2024","journal-title":"Artif. Intell. Med."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"101693","DOI":"10.1016\/j.media.2020.101693","article-title":"Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation","volume":"63","author":"Tajbakhsh","year":"2020","journal-title":"Med. Image Anal."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"280","DOI":"10.1016\/j.media.2019.03.009","article-title":"Not-so-supervised: A survey of semi-supervised, multi-instance, and transfer learning in medical image analysis","volume":"54","author":"Cheplygina","year":"2019","journal-title":"Med. Image Anal."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"103201","DOI":"10.1016\/j.media.2024.103201","article-title":"A comprehensive survey on deep active learning in medical image analysis","volume":"95","author":"Wang","year":"2024","journal-title":"Med. Image Anal."},{"key":"ref_13","unstructured":"Settles, B. (2009). Active Learning Literature Survey, University of Wisconsin\u2013Madison, Computer Sciences Department. Computer Sciences Technical Report 1648."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"102062","DOI":"10.1016\/j.media.2021.102062","article-title":"A survey on active learning and human-in-the-loop deep learning for medical image analysis","volume":"71","author":"Budd","year":"2021","journal-title":"Med. Image Anal."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"V\u00e1squez-Venegas, C., Sotomayor, C.G., Ramos, B., Casta\u00f1eda, V., Pereira, G., Cabrera-Vives, G., and H\u00e4rtel, S. (2024). Human-in-the-Loop\u2014A Deep Learning Strategy in Combination with a Patient-Specific Gaussian Mixture Model Leads to the Fast Characterization of Volumetric Ground-Glass Opacity and Consolidation in the Computed Tomography Scans of COVID-19 Patients. J. Clin. Med., 13.","DOI":"10.3390\/jcm13175231"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Yalcinkaya, D.M., Youssef, K., Heydari, B., Simonetti, O., Dharmakumar, R., Raman, S., and Sharif, B. (2023). Temporal Uncertainty Localization to Enable Human-in-the-loop Analysis of Dynamic Contrast-enhanced Cardiac MRI Datasets. Proceedings of the Medical Image Computing and Computer Assisted Intervention\u2014MICCAI 2023; Lecture Notes in Computer Science, Springer.","DOI":"10.1007\/978-3-031-43898-1_44"},{"key":"ref_17","unstructured":"Sener, O., and Savarese, S. (May, January 30). Active Learning for Convolutional Neural Networks: A Core-Set Approach. Proceedings of the International Conference on Learning Representations (ICLR), Vancouver, BC, Canada."},{"key":"ref_18","unstructured":"Kirsch, A., van Amersfoort, J., and Gal, Y. (2019, January 8\u201314). BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Yoo, D., and Kweon, I. (2019, January 16\u201320). Learning Loss for Active Learning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00018"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1038\/s41746-018-0040-6","article-title":"Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices","volume":"1","author":"Lavin","year":"2018","journal-title":"npj Digit. Med."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"723","DOI":"10.1136\/bjophthalmol-2020-316594","article-title":"Prospective evaluation of an artificial intelligence-enabled algorithm for automated diabetic retinopathy screening of 30 000 patients","volume":"105","author":"Heydon","year":"2021","journal-title":"Br. J. Ophthalmol."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"064001","DOI":"10.1117\/1.JMI.9.6.064001","article-title":"Efficient labeling of retinal fundus photographs using deep active learning","volume":"9","author":"Paul","year":"2022","journal-title":"J. Med. Imaging"},{"key":"ref_23","first-page":"48","article-title":"Adaptive Curriculum Query Strategy for Active Learning in Medical Image Classification","volume":"Volume 15011","author":"Ma","year":"2024","journal-title":"Proceedings of the Medical Image Computing and Computer Assisted Intervention\u2014MICCAI 2024; Lecture Notes in Computer Science"},{"key":"ref_24","unstructured":"Jocher, G., Chaurasia, A., and Qiu, J. (2026, June 25). YOLOv8 by Ultralytics. Software, AGPL-3.0 License. Available online: https:\/\/github.com\/ultralytics\/ultralytics."},{"key":"ref_25","unstructured":"Gal, Y., and Ghahramani, Z. (2016, January 20\u201322). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA."},{"key":"ref_26","unstructured":"Kendall, A., and Gal, Y. (2017, January 4\u20139). What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"2402","DOI":"10.1001\/jama.2016.17216","article-title":"Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs","volume":"316","author":"Gulshan","year":"2016","journal-title":"JAMA"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"2211","DOI":"10.1001\/jama.2017.18152","article-title":"Development and Validation of a Deep Learning System for Diabetic Retinopathy and Related Eye Diseases Using Retinal Images from Multiethnic Populations with Diabetes","volume":"318","author":"Ting","year":"2017","journal-title":"JAMA"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"e35","DOI":"10.1016\/S2589-7500(19)30004-4","article-title":"Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: A clinical validation study","volume":"1","author":"Bellemo","year":"2019","journal-title":"Lancet Digit. Health"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"987","DOI":"10.1001\/jamaophthalmol.2019.2004","article-title":"Performance of a Deep-Learning Algorithm vs Manual Grading for Detecting Diabetic Retinopathy in India","volume":"137","author":"Gulshan","year":"2019","journal-title":"JAMA Ophthalmol."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1264","DOI":"10.1016\/j.ophtha.2018.01.034","article-title":"Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy","volume":"125","author":"Krause","year":"2018","journal-title":"Ophthalmology"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"231","DOI":"10.5566\/ias.1155","article-title":"Feedback on a publicly distributed image database: The Messidor database","volume":"33","author":"Zhang","year":"2014","journal-title":"Image Anal. Stereol."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"101561","DOI":"10.1016\/j.media.2019.101561","article-title":"IDRiD: Diabetic Retinopathy\u2014Segmentation and Grading Challenge","volume":"59","author":"Porwal","year":"2020","journal-title":"Med. Image Anal."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"511","DOI":"10.1016\/j.ins.2019.06.011","article-title":"Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening","volume":"501","author":"Li","year":"2019","journal-title":"Inf. Sci."},{"key":"ref_35","unstructured":"Houlsby, N., Husz\u00e1r, F., Ghahramani, Z., and Lengyel, M. (2011). Bayesian Active Learning for Classification and Preference Learning. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"102958","DOI":"10.1016\/j.media.2023.102958","article-title":"Active learning for medical image segmentation with stochastic batches","volume":"90","author":"Gaillochet","year":"2023","journal-title":"Med. Image Anal."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Yuan, T., Wan, F., Fu, M., Liu, J., Xu, S., Ji, X., and Ye, Q. (2021, January 20\u201325). Multiple Instance Active Learning for Object Detection. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00529"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Yu, W., Zhu, S., Yang, T., and Chen, C. (2022, January 19\u201320). Consistency-based Active Learning for Object Detection. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, New Orleans, LA, USA.","DOI":"10.1109\/CVPRW56347.2022.00440"},{"key":"ref_39","unstructured":"Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017, January 4\u20139). Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Lewis, D.D., and Gale, W.A. (1994). A Sequential Algorithm for Training Text Classifiers. SIGIR \u201994, Springer.","DOI":"10.1007\/978-1-4471-2099-5_1"}],"container-title":["Bioengineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2306-5354\/13\/7\/762\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T04:10:42Z","timestamp":1783743042000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2306-5354\/13\/7\/762"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,29]]},"references-count":40,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2026,7]]}},"alternative-id":["bioengineering13070762"],"URL":"https:\/\/doi.org\/10.3390\/bioengineering13070762","relation":{},"ISSN":["2306-5354"],"issn-type":[{"value":"2306-5354","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,29]]}}}