{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,24]],"date-time":"2026-01-24T19:13:42Z","timestamp":1769282022961,"version":"3.49.0"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2022,6,23]],"date-time":"2022-06-23T00:00:00Z","timestamp":1655942400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,6,23]],"date-time":"2022-06-23T00:00:00Z","timestamp":1655942400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001809","name":"national natural science foundation of china","doi-asserted-by":"publisher","award":["71671032"],"award-info":[{"award-number":["71671032"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Complex Intell. Syst."],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Cloud healthcare system (CHS) can provide the telemedicine services, which is helpful to cope with the difficulty of patients getting medical service in the traditional medical systems. However, resource scheduling in CHS has to face with a great of challenges since managing the trade-off of efficiency and quality becomes complicated due to the uncertainty of patient choice behavior. Motivated by this, a resource scheduling problem with multi-stations queueing network in CHS is studied in this paper. A Markov decision model with uncertainty is developed to optimize the match process of patients and scarce resources with the objective of minimizing the total medical costs that consist of three conflicting sub-costs, i.e., medical costs, waiting time costs and the penalty costs caused by unmuting choice behavior of patients. For solving the proposed model, a three-stage dynamic scheduling method is designed, in which an improved Q-learning algorithm is employed to achieve the optimal schedule. Numerical experimental results show that this Q-learning-based scheduling algorithm outperforms two traditional scheduling algorithms significantly, as well as the balance of the three conflicting sub-costs is kept and the service efficiency is improved.<\/jats:p>","DOI":"10.1007\/s40747-022-00776-9","type":"journal-article","created":{"date-parts":[[2022,6,23]],"date-time":"2022-06-23T07:02:47Z","timestamp":1655967767000},"page":"4603-4618","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Optimal scheduling in cloud healthcare system using Q-learning algorithm"],"prefix":"10.1007","volume":"8","author":[{"given":"Yafei","family":"Li","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8954-0876","authenticated-orcid":false,"given":"Hongfeng","family":"Wang","sequence":"additional","affiliation":[]},{"given":"Na","family":"Wang","sequence":"additional","affiliation":[]},{"given":"Tianhong","family":"Zhang","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,6,23]]},"reference":[{"key":"776_CR1","first-page":"18","volume":"2","author":"Z Salehahmadi","year":"2013","unstructured":"Salehahmadi Z, Hajialiasghari F (2013) Telemedicine in Iran: chances and challenges. World J Plast Surg 2:18\u201325","journal-title":"World J Plast Surg"},{"key":"776_CR2","doi-asserted-by":"publisher","first-page":"164","DOI":"10.1016\/j.jss.2017.06.085","volume":"220","author":"JS Jue","year":"2017","unstructured":"Jue JS, Spector SA, Spector SA (2017) Telemedicine broadening access to care for complex cases. J Surg Res 220:164\u2013170","journal-title":"J Surg Res"},{"key":"776_CR3","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1016\/j.urpr.2017.01.001","volume":"5","author":"BG Sherwood","year":"2018","unstructured":"Sherwood BG, Han Y, Nepple KG, Erickson BA (2018) Evaluating the effectiveness, efficiency and safety of telemedicine for urological care in the male prisoner population. Urol Pract 5:44\u201351","journal-title":"Urol Pract"},{"key":"776_CR4","doi-asserted-by":"publisher","first-page":"211","DOI":"10.5858\/135.2.211","volume":"135","author":"CL Hitchcock","year":"2011","unstructured":"Hitchcock CL (2011) The future of telepathology for the developing world. Arch Pathol Lab Med 135:211\u2013214","journal-title":"Arch Pathol Lab Med"},{"key":"776_CR5","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1016\/j.urpr.2017.01.001","volume":"5","author":"BG Sherwood","year":"2017","unstructured":"Sherwood BG, Han Y, Nepple KG, Erickson BA (2017) Evaluating the effectiveness, efficiency and safety of telemedicine for urological care in the male prisoner population. Urol Pract 5:44\u201351","journal-title":"Urol Pract"},{"key":"776_CR6","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1109\/TII.2018.2824815","volume":"15","author":"S Roy","year":"2019","unstructured":"Roy S, Das AK, Chatterjee S, Kumar N, Chattopadhyay S (2019) Provably secure fine-grained data access control over multiple cloud servers in mobile cloud computing based healthcare applications. IEEE Trans Ind Inf 15:457\u2013468","journal-title":"IEEE Trans Ind Inf"},{"key":"776_CR7","doi-asserted-by":"crossref","unstructured":"Idrees M, Iqbal W, Bazaz SA (2013) Real-time doctor-patient assignment in a telemedicine system. In: 2013 16th Int Multi Top Conf INMIC, 2013, pp 55\u201360","DOI":"10.1109\/INMIC.2013.6731324"},{"issue":"7","key":"776_CR8","doi-asserted-by":"publisher","first-page":"132","DOI":"10.1007\/s10916-020-01596-5","volume":"44","author":"BA Jnr","year":"2020","unstructured":"Jnr BA (2020) Use of telemedicine and virtual care for remote treatment in response to COVID-19 pandemic. J MED SYST 44(7):132","journal-title":"J MED SYST"},{"issue":"1","key":"776_CR9","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1109\/TITB.2004.842410","volume":"9","author":"A Pal","year":"2005","unstructured":"Pal A, Mbarika V, Cobb-Payton F, Datta P, Mccoy S (2005) Telemedicine diffusion in a developing country: the case of india (march 2004). IEEE Trans Inf Technol Biomed 9(1):59\u201365","journal-title":"IEEE Trans Inf Technol Biomed"},{"issue":"2","key":"776_CR10","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1109\/EMR.2016.2568959","volume":"44","author":"S Kumar","year":"2016","unstructured":"Kumar S, Southard PB, White M (2016) Telemedicine: determining \u201ccritical to quality\u201d characteristics for a healthcare service system design based on a survey of physical rehabilitation providers. IEEE Eng Manag Rev 44(2):41\u201355","journal-title":"IEEE Eng Manag Rev"},{"key":"776_CR11","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1016\/j.artmed.2018.02.001","volume":"85","author":"X Li","year":"2018","unstructured":"Li X, Wang J, Fung RYK (2018) Approximate dynamic programming approaches for appointment scheduling with patient preferences. Artif Intell Med 85:16\u201325","journal-title":"Artif Intell Med"},{"issue":"5","key":"776_CR12","doi-asserted-by":"publisher","first-page":"1975","DOI":"10.1287\/mnsc.2016.2704","volume":"64","author":"N Liu","year":"2017","unstructured":"Liu N, Finkelstein SR, Kruk ME, Rosenthal D (2017) When waiting to see a doctor is less irritating: understanding patient preferences and choice behavior in appointment scheduling. Manag Sci 64(5):1975\u20131996","journal-title":"Manag Sci"},{"key":"776_CR13","doi-asserted-by":"publisher","first-page":"166","DOI":"10.1016\/j.omega.2018.06.009","volume":"85","author":"AK Dogru","year":"2019","unstructured":"Dogru AK, Melouk SH (2019) Adaptive appointment scheduling for patient-centered medical homes. Omega 85:166\u2013181","journal-title":"Omega"},{"key":"776_CR14","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1016\/j.cor.2014.04.007","volume":"50","author":"BP Berg","year":"2014","unstructured":"Berg BP, Denton BT, Ayca Erdogan S et al (2014) Optimal booking and scheduling in outpatient procedure centers. Comput Oper Res 50:24\u201337","journal-title":"Comput Oper Res"},{"issue":"3","key":"776_CR15","doi-asserted-by":"publisher","first-page":"458","DOI":"10.1111\/poms.12800","volume":"27","author":"D Wang","year":"2017","unstructured":"Wang D, Morrice DJ, Muthuraman K et al (2017) Coordinated scheduling for a multi-server network in outpatient pre-operative care. Prod Oper Manag 27(3):458\u2013479","journal-title":"Prod Oper Manag"},{"issue":"2","key":"776_CR16","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1007\/s11134-011-9220-7","volume":"68","author":"R Yang","year":"2011","unstructured":"Yang R, Bhulai S, Van Der Mei R (2011) Optimal resource allocation for multiqueue systems with a shared server pool. Queueing Syst 68(2):133\u2013163","journal-title":"Queueing Syst"},{"issue":"7","key":"776_CR17","doi-asserted-by":"publisher","first-page":"132","DOI":"10.1007\/s10916-020-01596-5","volume":"44","author":"B Anthony Jnr","year":"2020","unstructured":"Anthony Jnr B (2020) Use of telemedicine and virtual care for remote treatment in response to COVID-19 pandemic. J Med Syst 44(7):132","journal-title":"J Med Syst"},{"key":"776_CR18","first-page":"861","volume":"14","author":"CR Doarn","year":"2008","unstructured":"Doarn CR, Merrell RC (2008) A roadmap for telemedicine: barriers yet to overcome. Telemed J e-health Off J Am Telemed Assoc 14:861\u2013862","journal-title":"Telemed J e-health Off J Am Telemed Assoc"},{"key":"776_CR19","doi-asserted-by":"publisher","first-page":"365","DOI":"10.1258\/jtt.2012.120005","volume":"18","author":"F Garc\u00eda-Lizana","year":"2012","unstructured":"Garc\u00eda-Lizana F, Giorgo F (2012) The future of e-health, including telemedicine and telecare, in the European Union: from stakeholders\u2019 views to evidence-based decisions. J Telemed Telecare 18:365\u2013366","journal-title":"J Telemed Telecare"},{"key":"776_CR20","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1017\/S026646231000005X","volume":"26","author":"P Whitten","year":"2010","unstructured":"Whitten P, Holtz B, Nguyen L (2010) Keys to a successful and sustainable telemedicine program. Int J Technol Assess Health Care 26:211\u2013216","journal-title":"Int J Technol Assess Health Care"},{"key":"776_CR21","doi-asserted-by":"publisher","first-page":"473","DOI":"10.1089\/tmj.2015.0166","volume":"22","author":"LD O\u2019Gorman","year":"2016","unstructured":"O\u2019Gorman LD, Hogenbirk JC, Warry W (2016) Clinical telemedicine utilization in Ontario over the Ontario telemedicine network. Telemed e-Health 22:473\u2013479","journal-title":"Telemed e-Health"},{"key":"776_CR22","doi-asserted-by":"publisher","first-page":"2571","DOI":"10.1016\/j.ophtha.2016.08.021","volume":"123","author":"HV Nguyen","year":"2016","unstructured":"Nguyen HV, Tan GSW, Tapp RJ et al (2016) Cost-effectiveness of a National Telemedicine Diabetic Retinopathy Screening Program in Singapore. Ophthalmology 123:2571\u20132580","journal-title":"Ophthalmology"},{"issue":"11","key":"776_CR23","doi-asserted-by":"publisher","first-page":"4967","DOI":"10.1287\/mnsc.2017.2905","volume":"64","author":"S Saghafian","year":"2018","unstructured":"Saghafian S, Hopp WJ, Iravani S (2018) Workload management in telemedical physician triage and other knowledge-based service systems. Manag Sci 64(11):4967\u20135460","journal-title":"Manag Sci"},{"key":"776_CR24","doi-asserted-by":"publisher","first-page":"1267","DOI":"10.1080\/0740817X.2015.1011355","volume":"47","author":"SA Erdogan","year":"2015","unstructured":"Erdogan SA, Gose A, Denton BT (2015) Online appointment sequencing and scheduling. IIE Trans Inst Ind Eng 47:1267\u20131286","journal-title":"IIE Trans Inst Ind Eng"},{"issue":"2","key":"776_CR25","doi-asserted-by":"publisher","DOI":"10.2196\/11330","volume":"21","author":"A Buvik","year":"2019","unstructured":"Buvik A, Bergmo TS, Bugge E et al (2019) Cost-effectiveness of telemedicine in remote orthopedic consultations: randomized controlled trial. J Med Internet Res 21(2):e11330","journal-title":"J Med Internet Res"},{"key":"776_CR26","doi-asserted-by":"publisher","first-page":"283","DOI":"10.1177\/1357633X0200800507","volume":"8","author":"A Ohinmaa","year":"2002","unstructured":"Ohinmaa A, Vuolio S, Haukipuro K, Winblad I (2002) A cost-minimization analysis of orthopaedic consultations using videoconferencing in comparison with conventional consulting. J Telemed Telecare 8:283\u2013289","journal-title":"J Telemed Telecare"},{"key":"776_CR27","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1471-2458-12-1121","volume":"12","author":"L Tang","year":"2012","unstructured":"Tang L (2012) The patient\u2019s anxiety before seeing a doctor and her\/his hospital choice behavior in China. BMC Public Health 12:1","journal-title":"BMC Public Health"},{"issue":"C","key":"776_CR28","doi-asserted-by":"publisher","first-page":"166","DOI":"10.1016\/j.omega.2018.06.009","volume":"85","author":"AK Dogru","year":"2019","unstructured":"Dogru AK, Melouk SH (2019) Adaptive appointment scheduling for patient-centered medical homes. Omega 85(C):166\u2013181","journal-title":"Omega"},{"issue":"4","key":"776_CR29","doi-asserted-by":"publisher","first-page":"780","DOI":"10.1287\/msom.2018.0724","volume":"21","author":"JY Liu","year":"2019","unstructured":"Liu JY, Xie JG, Yang KK, Zheng AC (2019) Effects of rescheduling on patient no-show behavior in outpatient clinics. M&SOM 21(4):780\u2013797","journal-title":"M&SOM"},{"key":"776_CR30","doi-asserted-by":"publisher","first-page":"576","DOI":"10.1287\/opre.1080.0542","volume":"56","author":"D Gupta","year":"2008","unstructured":"Gupta D, Wang L (2008) Revenue management for a primary-care clinic in the presence of patient choice. Oper Res 56:576\u2013592","journal-title":"Oper Res"},{"key":"776_CR31","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1109\/17.293383","volume":"41","author":"H Aytug","year":"1994","unstructured":"Aytug H, Bhattacharyya S, Koehler GJ, Snowdon JL (1994) A review of machine learning in scheduling. IEEE Trans Eng Manag 41:165\u2013171","journal-title":"IEEE Trans Eng Manag"},{"key":"776_CR32","doi-asserted-by":"crossref","unstructured":"Keddis N, Javed B, Igna G, Zoitl A (2015) Optimizing schedules for adaptable manufacturing systems. In: 2015 IEEE 20th Conference on emerging technologies & factory automation (ETFA). pp 1\u20138","DOI":"10.1109\/ETFA.2015.7301452"},{"key":"776_CR33","doi-asserted-by":"publisher","first-page":"604","DOI":"10.1016\/j.cie.2018.03.039","volume":"125","author":"Y-R Shiue","year":"2018","unstructured":"Shiue Y-R, Lee K-C, Su C-T (2018) Real-time scheduling for a smart factory using a reinforcement learning approach. Comput Ind Eng 125:604\u2013614","journal-title":"Comput Ind Eng"},{"key":"776_CR34","doi-asserted-by":"publisher","first-page":"417","DOI":"10.1007\/s10845-018-1454-3","volume":"31","author":"YF Wang","year":"2020","unstructured":"Wang YF (2020) Adaptive job shop scheduling strategy based on weighted Q-learning algorithm. J Intell Manuf 31:417\u2013432","journal-title":"J Intell Manuf"}],"container-title":["Complex &amp; Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-022-00776-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s40747-022-00776-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-022-00776-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,27]],"date-time":"2022-10-27T12:28:19Z","timestamp":1666873699000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s40747-022-00776-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,23]]},"references-count":34,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["776"],"URL":"https:\/\/doi.org\/10.1007\/s40747-022-00776-9","relation":{},"ISSN":["2199-4536","2198-6053"],"issn-type":[{"value":"2199-4536","type":"print"},{"value":"2198-6053","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,6,23]]},"assertion":[{"value":"30 June 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 May 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 June 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}