{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,27]],"date-time":"2026-09-27T02:01:46Z","timestamp":1790474506294,"version":"4.1.0"},"reference-count":56,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2025,6,11]],"date-time":"2025-06-11T00:00:00Z","timestamp":1749600000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["82222060"],"award-info":[{"award-number":["82222060"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["82430103"],"award-info":[{"award-number":["82430103"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["82473572"],"award-info":[{"award-number":["82473572"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["81930090"],"award-info":[{"award-number":["81930090"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["82073487"],"award-info":[{"award-number":["82073487"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["81602790"],"award-info":[{"award-number":["81602790"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Biology"],"abstract":"<jats:p>Single-cell RNA sequencing now profiles whole transcriptomes for hundreds of thousands of cells, yet existing trajectory-inference tools rarely pinpoint where and when fate decisions are made. We present single-cell reinforcement learning (scRL), an actor\u2013critic framework that recasts differentiation as a sequential decision process on an interpretable latent manifold derived with Latent Dirichlet Allocation. The critic learns state-value functions that quantify fate intensity for each cell, while the actor traces optimal developmental routes across the manifold. Benchmarks on hematopoiesis, mouse endocrinogenesis, acute myeloid leukemia, and gene-knockout and irradiation datasets show that scRL surpasses fifteen state-of-the-art methods in five independent evaluation dimensions, recovering early decision states that precede overt lineage commitment and revealing regulators such as Dapp1. Beyond fate decisions, the same framework produces competitive measures of lineage-contribution intensity without requiring ground-truth probabilities, providing a unified and extensible approach for decoding developmental logic from single-cell data.<\/jats:p>","DOI":"10.3390\/biology14060679","type":"journal-article","created":{"date-parts":[[2025,6,11]],"date-time":"2025-06-11T11:18:40Z","timestamp":1749640720000},"page":"679","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["scRL: Utilizing Reinforcement Learning to Evaluate Fate Decisions in Single-Cell Data"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-8329-0108","authenticated-orcid":false,"given":"Zeyu","family":"Fu","sequence":"first","affiliation":[{"name":"State Key Laboratory of Trauma and Chemical Poisoning, Institute of Combined Injury, Chongqing Engineering Research Center for Nanomedicine, College of Preventive Medicine, Army Medical University, Chongqing 400038, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-2328-8774","authenticated-orcid":false,"given":"Chunlin","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Rehabilitation Medicine, The First Affiliated Hospital, Sun Yat-sen University, Guangzhou 510080, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Song","family":"Wang","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Trauma and Chemical Poisoning, Institute of Combined Injury, Chongqing Engineering Research Center for Nanomedicine, College of Preventive Medicine, Army Medical University, Chongqing 400038, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Junping","family":"Wang","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Trauma and Chemical Poisoning, Institute of Combined Injury, Chongqing Engineering Research Center for Nanomedicine, College of Preventive Medicine, Army Medical University, Chongqing 400038, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3362-2909","authenticated-orcid":false,"given":"Shilei","family":"Chen","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Trauma and Chemical Poisoning, Institute of Combined Injury, Chongqing Engineering Research Center for Nanomedicine, College of Preventive Medicine, Army Medical University, Chongqing 400038, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,6,11]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1728","DOI":"10.1101\/gr.275430.121","article-title":"Applications of single-cell genomics and computational strategies to study common disease and population-level variation","volume":"31","author":"Auerbach","year":"2021","journal-title":"Genome Res."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"e91","DOI":"10.1093\/nar\/gkac452","article-title":"Detecting critical transition signals from single-cell transcriptomes to infer lineage-determining transcription factors","volume":"50","author":"Yang","year":"2022","journal-title":"Nucleic Acids Res."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"7286","DOI":"10.1038\/s41467-023-42841-y","article-title":"A statistical framework for differential pseudotime analysis with multiple single-cell RNA-seq samples","volume":"14","author":"Hou","year":"2023","journal-title":"Nat. Commun."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"L\u00e4hnemann, D., K\u00f6ster, J., Szczurek, E., McCarthy, D.J., Hicks, S.C., Robinson, M.D., Vallejos, C.A., Campbell, K.R., Beerenwinkel, N., and Mahfouz, A. (2020). Eleven grand challenges in single-cell data science. Genome Biol., 21.","DOI":"10.1186\/s13059-020-1926-6"},{"key":"ref_5","unstructured":"Van der Maaten, L., Postma, E., and Van den Herik, J. (2009). Dimensionality Reduction: A Comparative Review, Tilburg University. Technical Report TiCC TR 2009-005."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"e86","DOI":"10.1093\/nar\/gkac412","article-title":"Inference of cell state transitions and cell fate plasticity from single-cell with MARGARET","volume":"50","author":"Pandey","year":"2022","journal-title":"Nucleic Acids Res."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"105499","DOI":"10.1016\/j.isci.2022.105499","article-title":"Quantitative landscapes reveal trajectories of cell-state transitions associated with drug resistance in melanoma","volume":"25","author":"Pillai","year":"2022","journal-title":"iScience"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Sun, S., Zhu, J., Ma, Y., and Zhou, X. (2019). Accuracy, robustness and scalability of dimensionality reduction methods for single-cell RNA-seq analysis. Genome Biol., 20.","DOI":"10.1186\/s13059-019-1898-6"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1836","DOI":"10.1038\/s41467-023-37478-w","article-title":"Comparative analysis of dimension reduction methods for cytometry by time-of-flight data","volume":"14","author":"Wang","year":"2023","journal-title":"Nat. Commun."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"861","DOI":"10.21105\/joss.00861","article-title":"UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction","volume":"3","author":"McInnes","year":"2018","journal-title":"J. Open Source Softw."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Wolf, F.A., Hamey, F.K., Plass, M., Solana, J., Dahlin, J.S., G\u00f6ttgens, B., Rajewsky, N., Simon, L., and Theis, F.J. (2019). PAGA: Graph abstraction reconciles clustering with trajectory inference through a topology preserving map of single cells. Genome Biol., 20.","DOI":"10.1186\/s13059-019-1663-x"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"496","DOI":"10.1038\/s41586-019-0969-x","article-title":"The single-cell transcriptional landscape of mammalian organogenesis","volume":"566","author":"Cao","year":"2019","journal-title":"Nature"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Street, K., Risso, D., Fletcher, R.B., Das, D., Ngai, J., Yosef, N., Purdom, E., and Dudoit, S. (2018). Slingshot: Cell lineage and pseudotime inference for single-cell transcriptomics. BMC Genom., 19.","DOI":"10.1186\/s12864-018-4772-0"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"5972","DOI":"10.1038\/s41467-022-33681-3","article-title":"Alignment of single-cell trajectory trees with CAPITAL","volume":"13","author":"Sugihara","year":"2022","journal-title":"Nat. Commun."},{"key":"ref_15","unstructured":"Granja, J.M., Ananthakrishnan, B., Zhang, C.A., Camacho, J., Santos, P.M., Wu, M.O., Dal Molin, A., Dhanasekaran, P., and Fischbeck, N.P. (2020). Deciphering the regulatory landscape of fetal and adult human epicardial ATMs at single-cell resolution. bioRxiv, bioRxiv:2020.04.27.999235."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"393","DOI":"10.1146\/annurev-statistics-040522-115238","article-title":"Manifold Learning: What, How, and Why","volume":"11","year":"2024","journal-title":"Annu. Rev. Stat. Its Appl."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1373","DOI":"10.1162\/089976603321780317","article-title":"Laplacian eigenmaps for dimensionality reduction and data representation","volume":"15","author":"Belkin","year":"2003","journal-title":"Neural Comput."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"2319","DOI":"10.1126\/science.290.5500.2319","article-title":"A global geometric framework for nonlinear dimensionality reduction","volume":"290","author":"Tenenbaum","year":"2000","journal-title":"Science"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"2323","DOI":"10.1126\/science.290.5500.2323","article-title":"Nonlinear dimensionality reduction by locally linear embedding","volume":"290","author":"Roweis","year":"2000","journal-title":"Science"},{"key":"ref_20","first-page":"2579","article-title":"Visualizing data using t-SNE","volume":"9","author":"Hinton","year":"2008","journal-title":"J. Mach. Learn. Res."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1016\/j.coisb.2017.12.008","article-title":"Manifold learning-based methods for analyzing single-cell RNA-sequencing data","volume":"7","author":"Moon","year":"2018","journal-title":"Curr. Opin. Syst. Biol."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Vasighizaker, A., Danda, S., and Rueda, L. (2022). Discovering cell types using manifold learning and enhanced visualization of single-cell RNA-Seq data. Sci. Rep., 12.","DOI":"10.1038\/s41598-021-03613-0"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"100547","DOI":"10.1016\/j.crmeth.2023.100547","article-title":"An introduction to representation learning for single-cell data analysis","volume":"3","author":"Gunawan","year":"2023","journal-title":"Cell Rep. Methods"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Nguyen, N.D., and Wang, D. (2019). ManiNetCluster: A novel manifold learning approach to reveal the functional links between gene networks. BMC Genom., 20.","DOI":"10.1186\/s12864-019-6329-2"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Huang, J.X., Zeng, H., Wang, L., Hill, S.L., and Zhao, F. (2021). Manifold learning analysis suggests strategies to align gene expression and electrophysiology measurements of neuronal diversity in the mouse brain. Commun. Biol., 4.","DOI":"10.1038\/s42003-021-02807-6"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Verma, A., and Engelhardt, B.E. (2020). A robust nonlinear low-dimensional manifold for single cell RNA-seq data. BMC Bioinform., 21.","DOI":"10.1186\/s12859-020-03625-z"},{"key":"ref_27","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Kempka, M., Wydmuch, M., Runc, G., Toczek, J., and Ja\u015bkowski, W. (2016, January 20\u201323). ViZDoom: A Doom-based AI research platform for visual reinforcement learning. Proceedings of the 2016 IEEE Conference on Computational Intelligence and Games (CIG), Santorini, Greece.","DOI":"10.1109\/CIG.2016.7860433"},{"key":"ref_29","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing Atari with Deep Reinforcement Learning. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Brahmbhatt, S., and Hays, J. (2017, January 21\u201326). DeepNav: Learning to Navigate Large Cities. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.329"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"693","DOI":"10.1038\/nrg.2016.98","article-title":"Transition states and cell fate decisions in epigenetic landscapes","volume":"17","author":"Moris","year":"2016","journal-title":"Nat. Rev. Genet."},{"key":"ref_32","first-page":"261","article-title":"Reinforcement learning in continuous action spaces: A case study in the game of Simulated Curling","volume":"62","author":"Yu","year":"2018","journal-title":"J. Artif. Intell. Res."},{"key":"ref_33","first-page":"30","article-title":"Reinforcement learning approaches to decision-making in biological systems: An overview and future directions","volume":"451","author":"Peng","year":"2018","journal-title":"J. Theor. Biol."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"928","DOI":"10.1016\/j.cell.2019.01.006","article-title":"Optimal-transport analysis of single-cell gene expression identifies developmental trajectories in reprogramming","volume":"176","author":"Schiebinger","year":"2019","journal-title":"Cell"},{"key":"ref_36","first-page":"1677","article-title":"Exploring cell identity as a stochastic process","volume":"12","author":"Anger","year":"2021","journal-title":"Nat. Commun."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"637","DOI":"10.1038\/nbt.3569","article-title":"Wishbone identifies bifurcating developmental trajectories from single-cell data","volume":"34","author":"Setty","year":"2016","journal-title":"Nat. Biotechnol."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"451","DOI":"10.1038\/s41587-019-0068-4","article-title":"Characterization of cell fate probabilities in single-cell data with Palantir","volume":"37","author":"Setty","year":"2019","journal-title":"Nat. Biotechnol."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"2989","DOI":"10.1093\/bioinformatics\/btv325","article-title":"Diffusion maps for high-dimensional single-cell analysis of differentiation data","volume":"31","author":"Haghverdi","year":"2015","journal-title":"Bioinformatics"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"979","DOI":"10.1038\/nmeth.4402","article-title":"Reversed graph embedding resolves complex single-cell trajectories","volume":"14","author":"Qiu","year":"2017","journal-title":"Nat. Methods"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1038\/nmeth.4662","article-title":"FateID infers cell fate bias in multipotent progenitors from single-cell RNA-seq data","volume":"15","author":"Herman","year":"2018","journal-title":"Nat. Methods"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1038\/s41592-021-01346-6","article-title":"CellRank for directed single-cell fate mapping","volume":"19","author":"Lange","year":"2022","journal-title":"Nat. Methods"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"2395","DOI":"10.1038\/s41467-019-10291-0","article-title":"A comprehensive single cell transcriptional landscape of human hematopoietic progenitors","volume":"10","author":"Pellin","year":"2019","journal-title":"Nat. Commun."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1285","DOI":"10.1038\/s41467-023-36969-0","article-title":"Longitudinal single-cell profiling of chemotherapy response in acute myeloid leukemia","volume":"14","author":"Naldini","year":"2023","journal-title":"Nat. Commun."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1038\/s42256-019-0025-4","article-title":"Reinforcement learning in artificial and biological systems","volume":"1","author":"Neftci","year":"2019","journal-title":"Nat. Mach. Intell."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"2063","DOI":"10.1109\/TNNLS.2018.2790388","article-title":"Applications of Deep Learning and Reinforcement Learning to Biological Data","volume":"29","author":"Mahmud","year":"2018","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1016\/j.cels.2020.11.008","article-title":"Benchmarking computational doublet-detection methods for single-cell RNA sequencing data","volume":"12","author":"Xi","year":"2021","journal-title":"Cell Syst."},{"key":"ref_48","first-page":"68","article-title":"Data analysis guidelines for single-cell RNA-seq in biomedical studies and clinical applications","volume":"9","author":"Su","year":"2022","journal-title":"Mil. Med Res."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Chari, T., and Pachter, L. (2023). The specious art of single-cell genomics. PLoS Comput. Biol., 19.","DOI":"10.1371\/journal.pcbi.1011288"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"1482","DOI":"10.1038\/s41587-019-0336-3","article-title":"Visualizing structure and transitions in high-dimensional biological data","volume":"37","author":"Moon","year":"2019","journal-title":"Nat. Biotechnol."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"547","DOI":"10.1038\/s41587-019-0071-9","article-title":"A comparison of single-cell trajectory inference methods","volume":"37","author":"Saelens","year":"2019","journal-title":"Nat. Biotechnol."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"E2467","DOI":"10.1073\/pnas.1714723115","article-title":"Fundamental limits on dynamic inference from single-cell snapshots","volume":"115","author":"Weinreb","year":"2018","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"619","DOI":"10.1038\/s41587-020-00803-5","article-title":"Quantifying the effect of experimental perturbations at single-cell resolution","volume":"39","author":"Burkhardt","year":"2021","journal-title":"Nat. Biotechnol."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Li, Q. (2023). scTour: A deep learning architecture for robust inference and accurate prediction of cellular dynamics. Genome Biol., 24.","DOI":"10.1186\/s13059-023-02988-9"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"5528","DOI":"10.1038\/s41467-021-25773-3","article-title":"Generalized and scalable trajectory inference in single-cell omics data with VIA","volume":"12","author":"Stassen","year":"2021","journal-title":"Nat. Commun."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Fang, M., Gorin, G., and Pachter, L. (2025). Trajectory inference from single-cell genomics data with a process time model. PLoS Comput. Biol., 21.","DOI":"10.1371\/journal.pcbi.1012752"}],"container-title":["Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-7737\/14\/6\/679\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:50:15Z","timestamp":1760032215000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-7737\/14\/6\/679"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,11]]},"references-count":56,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2025,6]]}},"alternative-id":["biology14060679"],"URL":"https:\/\/doi.org\/10.3390\/biology14060679","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2024.07.04.602019","asserted-by":"object"}]},"ISSN":["2079-7737"],"issn-type":[{"value":"2079-7737","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,11]]}}}