{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,29]],"date-time":"2026-05-29T11:19:41Z","timestamp":1780053581537,"version":"3.54.0"},"reference-count":28,"publisher":"American Institute of Aeronautics and Astronautics (AIAA)","issue":"9","license":[{"start":{"date-parts":[[2026,5,25]],"date-time":"2026-05-25T00:00:00Z","timestamp":1779667200000},"content-version":"am","delay-in-days":266,"URL":"https:\/\/www.aiaa.org\/userlicenses\/1.0\/#CompEndUserLicense"}],"funder":[{"DOI":"10.13039\/100017437","name":"NASA Headquarters","doi-asserted-by":"publisher","award":["80NSSC23K1182"],"award-info":[{"award-number":["80NSSC23K1182"]}],"id":[{"id":"10.13039\/100017437","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006602","name":"Air Force Research Laboratory","doi-asserted-by":"publisher","award":["FA9453-22-2-0050"],"award-info":[{"award-number":["FA9453-22-2-0050"]}],"id":[{"id":"10.13039\/100006602","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["arc.aiaa.org"],"crossmark-restriction":true},"short-container-title":["Journal of Aerospace Information Systems"],"published-print":{"date-parts":[[2025,9]]},"abstract":"<jats:p> Advances in request scheduling algorithms are necessary to fully utilize the capabilities of agile Earth-observing satellites. While typical current approaches preplan sequences of image collections for a static set of requests, onboard autonomous methods that select requests to fulfill on-the-fly can adapt to opportunistic events and reduce the burden on operators. Prior approaches that utilized reinforcement learning for autonomy considered fixed-duration decision intervals, which limited performance over high-density requests and oversimplified mission operations. In this paper, reinforcement learning is used to derive policies that make decisions at variable intervals to improve performance by scheduling requests in a mission-like manner. The resulting policies perform competitively against optimal global schedules in a flight-like evaluation environment. This is achieved by training on a new semi-Markov decision process (sMDP) formulation of the problem, leveraging step-duration information when learning. Ablation studies over new sMDP-specific training algorithm modifications and observation spaces are performed to learn a highly performant policy. Benefits of the method, including the implicit ability to account for resource constraints, are demonstrated. Ultimately, variable-decision-interval reinforcement-learning-based policies are established as a viable method for autonomous Earth-observing satellite scheduling under mission-realistic conditions, providing a closed-loop and computationally inexpensive alternative to traditional preplanning methods. <\/jats:p>","DOI":"10.2514\/1.i011649","type":"journal-article","created":{"date-parts":[[2025,5,26]],"date-time":"2025-05-26T01:39:56Z","timestamp":1748223596000},"page":"789-799","update-policy":"https:\/\/doi.org\/10.2514\/aiaa_crossmarkpolicy","source":"Crossref","is-referenced-by-count":3,"title":["Learning Policies for Autonomous Earth-Observing Satellite Scheduling over Semi-Markov Decision Processes"],"prefix":"10.2514","volume":"22","author":[{"given":"Mark Andrew","family":"Stephenson","sequence":"first","affiliation":[{"name":"University of Colorado Boulder"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lorenzzo Quevedo","family":"Mantovani","sequence":"additional","affiliation":[{"name":"University of Colorado Boulder"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0002-6035","authenticated-orcid":false,"given":"Hanspeter","family":"Schaub","sequence":"additional","affiliation":[{"name":"University of Colorado Boulder"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1387","reference":[{"key":"r1","doi-asserted-by":"publisher","DOI":"10.1016\/j.actaastro.2011.12.014"},{"key":"r2","doi-asserted-by":"publisher","DOI":"10.1109\/JSYST.2020.2997050"},{"key":"r3","doi-asserted-by":"publisher","DOI":"10.1023\/A:1018920709696"},{"key":"r5","unstructured":"ApplegateD. L.BixbyR. E.Chvat\u00e1lV.CookW. J., The Traveling Salesman Problem: A Computational Study, Princeton Univ. Press, Princeton, NJ, 2006, pp.\u00a01\u201358."},{"key":"r6","doi-asserted-by":"publisher","DOI":"10.1016\/j.cor.2019.05.030"},{"key":"r7","doi-asserted-by":"publisher","DOI":"10.2514\/1.I010620"},{"key":"r8","doi-asserted-by":"publisher","DOI":"10.1016\/j.asr.2017.11.010"},{"key":"r9","doi-asserted-by":"publisher","DOI":"10.2514\/1.I010775"},{"key":"r10","doi-asserted-by":"publisher","DOI":"10.1016\/j.cie.2021.107292"},{"key":"r11","doi-asserted-by":"publisher","DOI":"10.2514\/1.A36097"},{"key":"r13","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2015.03.011"},{"key":"r14","doi-asserted-by":"publisher","DOI":"10.2514\/6.2021-1469"},{"key":"r15","doi-asserted-by":"publisher","DOI":"10.2514\/1.I011233"},{"key":"r16","first-page":"47","volume-title":"Adaptive Computation and Machine Learning","author":"Sutton R. S.","year":"2018","edition":"2"},{"key":"r21","doi-asserted-by":"publisher","DOI":"10.2514\/1.A35169"},{"key":"r22","doi-asserted-by":"publisher","DOI":"10.1109\/TAES.2023.3251307"},{"key":"r25","doi-asserted-by":"publisher","DOI":"10.2514\/1.I010754"},{"key":"r26","doi-asserted-by":"publisher","DOI":"10.3390\/math11194059"},{"key":"r27","volume-title":"ASCEND","author":"Garcia Buzzi P.","year":"2021"},{"key":"r28","doi-asserted-by":"publisher","DOI":"10.1016\/j.ast.2021.107224"},{"key":"r29","first-page":"393","volume-title":"Advances in Neural Information Processing Systems","volume":"7","author":"Bradtke S.","year":"1994"},{"key":"r30","doi-asserted-by":"publisher","DOI":"10.1016\/S0004-3702(99)00052-1"},{"key":"r31","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2018.2848264"},{"key":"r32","doi-asserted-by":"publisher","DOI":"10.2514\/1.A34931"},{"key":"r33","doi-asserted-by":"publisher","DOI":"10.1016\/j.actaastro.2018.03.022"},{"key":"r34","unstructured":"TowersM.TerryJ. K.KwiatkowskiA.BalisJ. U.de ColaG.DeleuT.Goul\u00e3oM.KallinterisA.A.K. G.KrimmelM.et\u00a0al. \u201cGymnasium,\u201d Oct.\u00a02023."},{"key":"r35","doi-asserted-by":"publisher","DOI":"10.2514\/1.I010762"},{"issue":"11","key":"r36","first-page":"S02731177230052","volume":"73","author":"Brandonisio A.","year":"2023","journal-title":"Advances in Space Research"}],"container-title":["Journal of Aerospace Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/arc.aiaa.org\/doi\/am-pdf\/10.2514\/1.I011649","content-type":"application\/pdf","content-version":"am","intended-application":"unspecified"},{"URL":"https:\/\/arc.aiaa.org\/doi\/pdf\/10.2514\/1.I011649","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/arc.aiaa.org\/doi\/pdf\/10.2514\/1.I011649","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,25]],"date-time":"2025-08-25T09:01:13Z","timestamp":1756112473000},"score":1,"resource":{"primary":{"URL":"https:\/\/arc.aiaa.org\/doi\/10.2514\/1.I011649"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9]]},"references-count":28,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2025,9]]}},"alternative-id":["10.2514\/1.I011649"],"URL":"https:\/\/doi.org\/10.2514\/1.i011649","relation":{},"ISSN":["1940-3151","2327-3097"],"issn-type":[{"value":"1940-3151","type":"print"},{"value":"2327-3097","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9]]},"assertion":[{"value":"2025-02-20","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-04-17","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-25","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}