{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T14:02:04Z","timestamp":1791295324604,"version":"4.1.0"},"reference-count":41,"publisher":"American Chemical Society (ACS)","issue":"39","license":[{"start":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T00:00:00Z","timestamp":1790208000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100015197","name":"Renewable Bioproducts Institute","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100015197","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,10,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Anaerobic digestion (AD) is a cornerstone of energy recovery and decarbonization at wastewater resource recovery facilities (WRRFs), yet full-scale digesters are often operated conservatively due to delayed process feedback, nonlinear stability constraints, and limited observability. This study presents a novel reinforcement learning (RL)-enabled supervisory control framework for full-scale AD that integrates a calibrated mechanistic digital twin model (DTM) with offline policy learning and counterfactual uncertainty evaluation. Specifically, the DTM was calibrated to reproduce control-relevant digester behavior. Using 2.7 years of simulated operation, a batch-constrained Q (BCQ)-reinforcement learning policy was trained to maximize biogas production while penalizing instability associated with pH excursions and elevated volatile fatty acids (VFAs)\/alkalinity (ALK) ratios. Counterfactual evaluation against historical, greedy, and heuristic controls used 1 year unseen DTM simulation and 1 year of full-scale operational data. The RL policy achieved comparable mean daily biogas production (+0.35%) in simulated testing. Under the 1 year operational data, it achieved plant-equivalent mean production of 9802 m3\/day, corresponding to a modest potential cumulative improvement (+2.7%, 93\u202f405 m3\/year) subject to substantial uncertainty P(\u0394 &amp;gt; 0) = 0.524. These results indicate that offline-trained RL, combined with a mechanistic digital twin and counterfactual evaluation, provides a feasible and interpretable framework for developing and assessing supervisory AD control without live-plant experimentation while showing that the resulting benefit must be interpreted under explicit model-plant uncertainty.<\/jats:p>","DOI":"10.1021\/acs.est.6c05822","type":"journal-article","created":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T07:02:11Z","timestamp":1790233331000},"page":"28064-28078","source":"Crossref","is-referenced-by-count":0,"title":["Reinforcement-Learning-Enabled\nSupervisory Control\nof a Full-Scale Anaerobic Digestion Facility Using a Calibrated Digital\nTwin"],"prefix":"10.1021","volume":"60","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3110-6257","authenticated-orcid":true,"given":"Ahmed I.","family":"Yunus","sequence":"first","affiliation":[{"name":"Georgia Institute of Technology , , , ,","place":["Atlanta, Georgia, United States, 30318"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Srinivas","family":"Jalla","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology , , , ,","place":["Atlanta, Georgia, United States, 30318"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9791-1043","authenticated-orcid":true,"suffix":"III","given":"Joe F.","family":"Bozeman","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology , , , ,","place":["Atlanta, Georgia, United States, 30318"]},{"name":"Georgia Institute of Technology , , , ,","place":["Atlanta, Georgia, United States, 30332"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9519-2302","authenticated-orcid":true,"given":"Yongsheng","family":"Chen","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology , , , ,","place":["Atlanta, Georgia, United States, 30318"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"316","published-online":{"date-parts":[[2026,9,24]]},"reference":[{"key":"2026100609064651700_cit1","doi-asserted-by":"publisher","first-page":"107980","DOI":"10.1016\/j.psep.2025.107980","article-title":"Per- and Polyfluoroalkyl\nSubstances (PFAS) in Resource\nRecovery: Transforming Challenges into Opportunities for Sustainable\nNutrients and Biosolids Management","volume":"203","author":"Yunus","year":"2025","journal-title":"Process\nSaf. Environ. Prot."},{"key":"2026100609064651700_cit2","doi-asserted-by":"publisher","first-page":"112875","DOI":"10.1016\/j.jenvman.2021.112875","article-title":"Biogas Maximization Using Data-Driven Modelling with\nUncertainty Analysis and Genetic Algorithm for Municipal Wastewater\nAnaerobic Digestion","volume":"293","author":"Asadi","year":"2021","journal-title":"J. Environ. Manage."},{"issue":"2","key":"2026100609064651700_cit3","doi-asserted-by":"publisher","first-page":"2614","DOI":"10.1007\/s10668-020-00689-y","article-title":"Analysis\nof the Economic Viability of the Use of Biogas Produced in Wastewater\nTreatment Plants to Generate Electrical Energy","volume":"23","author":"Campello","year":"2021","journal-title":"Environ. Dev. Sustain."},{"issue":"2","key":"2026100609064651700_cit4","doi-asserted-by":"publisher","first-page":"03125006","DOI":"10.1061\/JOEEDU.EEENG-8235","article-title":"Wastewater Sludge as\nEnergy Resource: Technoeconomic and Environmental Implications","volume":"152","author":"Mittapalli","year":"2026","journal-title":"J. Environ. Eng."},{"issue":"1","key":"2026100609064651700_cit5","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1038\/s44172-024-00183-7","article-title":"Reinforcement Learning-Based Control for Waste Biorefining Processes\nunder Uncertainty","volume":"3","author":"Gao","year":"2024","journal-title":"Commun. Eng."},{"issue":"1","key":"2026100609064651700_cit6","doi-asserted-by":"publisher","first-page":"8564","DOI":"10.1080\/15567036.2020.1767728","article-title":"Modeling and\nOptimization of Biomethane Production from Solid-State Anaerobic Co-Digestion\nof Organic Fraction Municipal Solid Waste and Other Co-Substrates","volume":"46","author":"Saghouri","year":"2024","journal-title":"Energy Sources Part Recovery Util. Environ. Eff."},{"key":"2026100609064651700_cit7","doi-asserted-by":"publisher","first-page":"138976","DOI":"10.1016\/j.chemosphere.2023.138976","article-title":"Prediction of Biogas Production of Industrial Scale\nAnaerobic Digestion Plant by Machine Learning Algorithms","volume":"335","author":"Yildirim","year":"2023","journal-title":"Chemosphere"},{"key":"2026100609064651700_cit8","doi-asserted-by":"publisher","first-page":"498","DOI":"10.1016\/j.biortech.2016.10.015","article-title":"Anaerobic Co-Digestion: A Critical\nReview of Mathematical Modelling for Performance Optimization","volume":"222","author":"Xie","year":"2016","journal-title":"Bioresour. Technol."},{"issue":"4","key":"2026100609064651700_cit9","doi-asserted-by":"publisher","first-page":"771","DOI":"10.1007\/s11157-019-09515-y","article-title":"Biogas Production from\nAnaerobic Co-Digestion of Waste Activated Sludge: Co-Substrates and\nInfluencing Parameters","volume":"18","author":"Yang","year":"2019","journal-title":"Rev. Environ. Sci. Biotechnol."},{"key":"2026100609064651700_cit10","doi-asserted-by":"publisher","first-page":"108043","DOI":"10.1016\/j.biombioe.2025.108043","article-title":"From Process Fundamentals to Engineering\nPerspectives: A Technical Review on in-Situ Biogas Upgrading via the\nHydrogenotrophic Methanogenesis Pathway","volume":"200","author":"Abdelrahman","year":"2025","journal-title":"Biomass\nBioenergy"},{"issue":"38","key":"2026100609064651700_cit11","doi-asserted-by":"publisher","first-page":"12990","DOI":"10.1021\/acssuschemeng.1c04612","article-title":"Tree-Based Automated Machine Learning\nto Predict Biogas Production for Anaerobic Co-Digestion of Organic\nWaste","volume":"9","author":"Wang","year":"2021","journal-title":"ACS Sustain. Chem. Eng."},{"key":"2026100609064651700_cit12","doi-asserted-by":"publisher","first-page":"122495","DOI":"10.1016\/j.biortech.2019.122495","article-title":"Prediction of Anaerobic\nDigestion Performance and Identification of Critical Operational Parameters\nUsing Machine Learning Algorithms","volume":"298","author":"Wang","year":"2020","journal-title":"Bioresour.\nTechnol."},{"issue":"8","key":"2026100609064651700_cit13","doi-asserted-by":"publisher","first-page":"5284","DOI":"10.1109\/TSMC.2021.3122802","article-title":"Reinforcement-Learning-Based Tracking\nControl of Waste Water Treatment Process Under Realistic System Conditions\nand Control Performance Requirements","volume":"52","author":"Yang","year":"2022","journal-title":"IEEE Trans.\nSyst. Man Cybern. Syst."},{"key":"2026100609064651700_cit14","doi-asserted-by":"publisher","first-page":"121145","DOI":"10.1016\/j.watres.2024.121145","article-title":"Deep Reinforcement\nLearning Challenges and Opportunities\nfor Urban Water Systems","volume":"253","author":"Negm","year":"2024","journal-title":"Water Res."},{"issue":"1","key":"2026100609064651700_cit15","doi-asserted-by":"publisher","first-page":"4814","DOI":"10.1038\/s41598-025-88337-1","article-title":"Boosting Biogas Production through\nInnovative Data-Driven Modeling and Optimization Methods at NJWTP","volume":"15","author":"Duan","year":"2025","journal-title":"Sci. Rep."},{"key":"2026100609064651700_cit16","doi-asserted-by":"publisher","first-page":"134140","DOI":"10.1016\/j.jclepro.2022.134140","article-title":"Data-Driven\nModel for\nFeedstock Blending Optimization of Anaerobic Co-Digestion by BMP Maximization","volume":"375","author":"Moretta","year":"2022","journal-title":"J. Clean. Prod."},{"issue":"3","key":"2026100609064651700_cit17","doi-asserted-by":"publisher","first-page":"660","DOI":"10.1021\/acsestengg.3c00435","article-title":"Feature Engineering and Supervised Machine Learning\nto Forecast Biogas Production during Municipal Anaerobic Co-Digestion","volume":"4","author":"Schroer","year":"2024","journal-title":"ACS EST Eng."},{"key":"2026100609064651700_cit18","doi-asserted-by":"publisher","first-page":"130665","DOI":"10.1016\/j.biortech.2024.130665","article-title":"Machine Learning\nfor High Solid Anaerobic Digestion: Performance Prediction and Optimization","volume":"400","author":"Ganeshan","year":"2024","journal-title":"Bioresour. Technol."},{"key":"2026100609064651700_cit19","doi-asserted-by":"publisher","first-page":"125883","DOI":"10.1016\/j.energy.2022.125883","article-title":"Comparison\nof Machine Learning Methods for Predicting the Methane Production\nfrom Anaerobic Digestion of Lignocellulosic Biomass","volume":"263","author":"Wang","year":"2023","journal-title":"Energy"},{"issue":"5","key":"2026100609064651700_cit20","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1016\/j.eng.2024.04.012","article-title":"Digital Twins for Wastewater\nTreatment: A Technical Review","volume":"36","author":"Wang","year":"2024","journal-title":"Engineering"},{"issue":"7","key":"2026100609064651700_cit21","doi-asserted-by":"publisher","DOI":"10.1002\/wer.11074","article-title":"Automated Data Transfer\nfor Digital Twin Applications: Two Case Studies","volume":"96","author":"Molin","year":"2024","journal-title":"Water Environ. Res."},{"key":"2026100609064651700_cit22","doi-asserted-by":"publisher","first-page":"arXiv:2511.16297","DOI":"10.48550\/arXiv.2511.16297","article-title":"Optimizing Operation Recipes with Reinforcement Learning for Safe\nand Interpretable Control of Chemical Processes","author":"Brandner","year":"2025","journal-title":"arXiv"},{"key":"2026100609064651700_cit23","doi-asserted-by":"publisher","first-page":"arXiv:2408.13566","DOI":"10.48550\/arXiv.2408.13566","article-title":"Control-Informed\nReinforcement Learning\nfor Chemical Processes","author":"Bloor","year":"2024","journal-title":"arXiv"},{"key":"2026100609064651700_cit24","doi-asserted-by":"publisher","first-page":"113326","DOI":"10.1016\/j.engappai.2025.113326","article-title":"Reinforcement\nLearning Meets Bioprocess Control through Behavior Cloning: Real-World\nDeployment in an Industrial Photobioreactor","volume":"164","author":"Gil","year":"2026","journal-title":"Eng. Appl. Artif. Intell."},{"key":"2026100609064651700_cit25","doi-asserted-by":"publisher","first-page":"126433","DOI":"10.1016\/j.biortech.2021.126433","article-title":"Application\nof Machine Learning in Anaerobic Digestion: Perspectives and Challenges","volume":"345","author":"Andrade Cruz","year":"2022","journal-title":"Bioresour. Technol."},{"key":"2026100609064651700_cit26","doi-asserted-by":"publisher","first-page":"130665","DOI":"10.1016\/j.biortech.2024.130665","article-title":"Machine Learning\nfor High Solid Anaerobic Digestion: Performance Prediction and Optimization","volume":"400","author":"Ganeshan","year":"2024","journal-title":"Bioresour. Technol."},{"key":"2026100609064651700_cit27","doi-asserted-by":"publisher","first-page":"117182","DOI":"10.1016\/j.watres.2021.117182","article-title":"Predicting\nthe Performance of Anaerobic Digestion Using Machine Learning Algorithms\nand Genomic Data","volume":"199","author":"Long","year":"2021","journal-title":"Water Res."},{"issue":"1","key":"2026100609064651700_cit28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41545-025-00440-y","article-title":"Quantum\nMachine Learning Regression Optimisation for Full-Scale Sewage Sludge\nAnaerobic Digestion","volume":"8","author":"Mohamed","year":"2025","journal-title":"Npj Clean Water"},{"key":"2026100609064651700_cit29","doi-asserted-by":"publisher","first-page":"108822","DOI":"10.1016\/j.rineng.2025.108822","article-title":"Leveraging Wastewater\nResource Recovery to Reduce Landfill Organic Loading: Process Modeling,\nEnvironmental, and Economic Impact Assessment","volume":"29","author":"Yunus","year":"2026","journal-title":"Results Eng."},{"key":"2026100609064651700_cit30","doi-asserted-by":"publisher","first-page":"109515","DOI":"10.1016\/j.compchemeng.2025.109515","article-title":"A Survey and Tutorial\nof Reinforcement Learning Methods in Process Systems Engineering","volume":"206","author":"Bloor","year":"2026","journal-title":"Comput. Chem. Eng."},{"key":"2026100609064651700_cit31","unstructured":"Bartee, J. F.; Macharia, M. A.; Noll, P. D.; Tay, M. E.\n          Model Predictive\nControl of Fermentation Temperature in Biofuel Production. U.S. Patent 7,831,318 B2, Nov 9, 2010 (accessed 2025\u201303\u201306)."},{"key":"2026100609064651700_cit32","unstructured":"Rodriguez, J.; Ahmed, W.\n          Automatic Start-up\nof Anaerobic\nDigestion Reactors Using Model Predictive Control and Practically\nFeasible Sets of Measurements. WO Patent 2020261151 A1, Dec 30, 2020 (accessed 2025\u201303\u201306)."},{"key":"2026100609064651700_cit33","doi-asserted-by":"publisher","first-page":"755398","DOI":"10.3389\/fmicb.2021.755398","article-title":"Modeling the\nPerformance of Full-Scale\nAnaerobic Biochemical System Treating Deinking Pulp Wastewater Based\non Modified Anaerobic Digestion Model No. 1","volume":"12","author":"Huang","year":"2021","journal-title":"Front. Microbiol."},{"key":"2026100609064651700_cit34","doi-asserted-by":"publisher","first-page":"124745","DOI":"10.1016\/j.biortech.2021.124745","article-title":"Biochemical Methane\nPotential Prediction for Mixed Feedstocks of Straw and Manure in Anaerobic\nCo-Digestion","volume":"326","author":"Yang","year":"2021","journal-title":"Bioresour. Technol."},{"key":"2026100609064651700_cit35","doi-asserted-by":"publisher","first-page":"121495","DOI":"10.1016\/j.watres.2024.121495","article-title":"Microbial\nCommunity Assembly in Engineered Bioreactors","volume":"255","author":"Smith","year":"2024","journal-title":"Water Res."},{"key":"2026100609064651700_cit36","doi-asserted-by":"publisher","first-page":"136537","DOI":"10.1016\/j.chemosphere.2022.136537","article-title":"Two-Stage\nAnaerobic Membrane Bioreactor for Co-Treatment of Food Waste and Kitchen\nWastewater for Biogas Production and Nutrients Recovery","volume":"309","author":"Le","year":"2022","journal-title":"Chemosphere"},{"issue":"19","key":"2026100609064651700_cit37","doi-asserted-by":"publisher","first-page":"12741","DOI":"10.1021\/acs.est.1c01339","article-title":"Machine Learning:\nNew Ideas and Tools in Environmental Science and Engineering","volume":"55","author":"Zhong","year":"2021","journal-title":"Environ. Sci. Technol."},{"issue":"48","key":"2026100609064651700_cit38","doi-asserted-by":"publisher","first-page":"19078","DOI":"10.1021\/acs.est.3c05291","article-title":"Anaerobic Digestion\nas a Core Technology in Addressing the Global Sanitation Crisis: Challenges\nand Opportunities","volume":"57","author":"Almansa","year":"2023","journal-title":"Environ. Sci. Technol."},{"key":"2026100609064651700_cit39","doi-asserted-by":"publisher","first-page":"114285","DOI":"10.1016\/j.jenvman.2021.114285","article-title":"Out of Sight, out of\nMind: Understanding the Sanitation Crisis in Global South Cities","volume":"306","author":"Beard","year":"2022","journal-title":"J. Environ. Manage."},{"key":"2026100609064651700_cit40","unstructured":"U.S. Bureau of Labor Statistics.\n          .  Water and Wastewater Treatment Plant and\nSystem Operators: Occupational Outlook Handbook; U.S. Bureau of Labor Statistics: Washington, D.C., 2024; https:\/\/www.bls.gov\/ooh\/production\/water-and-wastewater-treatment-plant-and-system-operators.htm (accessed 2024\u201306\u201308)."},{"key":"2026100609064651700_cit41","doi-asserted-by":"publisher","DOI":"10.2166\/9781780401164","volume-title":"Guidelines for Using Activated Sludge\nModels","author":"Rieger","year":"2012"}],"container-title":["Environmental Science &amp; Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pubs.acs.org\/doi\/pdf\/10.1021\/acs.est.6c05822","content-type":"application\/pdf","content-version":"vor","intended-application":"unspecified"},{"URL":"https:\/\/pubs.acs.org\/esthag\/article-pdf\/60\/39\/28064\/68632942\/acs.est.6c05822.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/pubs.acs.org\/esthag\/article-pdf\/60\/39\/28064\/68632942\/acs.est.6c05822.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T13:30:26Z","timestamp":1791293426000},"score":1,"resource":{"primary":{"URL":"https:\/\/pubs.acs.org\/esthag\/article\/60\/39\/28064\/5437338\/Reinforcement-Learning-Enabled-Supervisory-Control"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,24]]},"references-count":41,"journal-issue":{"issue":"39","published-online":{"date-parts":[[2026,9,24]]},"published-print":{"date-parts":[[2026,10,6]]}},"URL":"https:\/\/doi.org\/10.1021\/acs.est.6c05822","relation":{},"ISSN":["0013-936X","1520-5851"],"issn-type":[{"value":"0013-936X","type":"print"},{"value":"1520-5851","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,24]]}}}