{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"bioRxiv"}],"indexed":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T16:30:07Z","timestamp":1785429007685,"version":"3.56.0"},"posted":{"date-parts":[[2026,4,9]]},"group-title":"Bioinformatics","reference-count":56,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2026,4,9]],"date-time":"2026-04-09T00:00:00Z","timestamp":1775692800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"award":["R01 AI183668"],"award-info":[{"award-number":["R01 AI183668"]}],"id":[{"id":"https:\/\/ror.org\/043z4tv69","id-type":"ROR","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2026,4,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                <jats:p>\n                  The generation of Metagenome Assembled Genomes (MAGs) has become a standard and basic step in the analysis of metagenomic data. This multi-step process, which includes assembly, binning, refinement, and quality control, has many alternative approaches, algorithms, and parameters. Determining the ideal approach for a given ecosystem and study, or highlighting algorithmic gaps in need of additional research and development, requires rigorous benchmarking. We present MAG-E (\n                  <jats:bold>MAG<\/jats:bold>\n                  pipeline\n                  <jats:bold>E<\/jats:bold>\n                  valuator), a generalizable and expandable framework for end-to-end evaluation of entire MAG pipelines: from assembly, through binning, to quality control and filtering. MAG-E relies on simulations that are built to match an ecosystem of interest and provide a ground truth for accurate evaluation. To demonstrate the capabilities of MAG-E, we benchmark two assemblers, six binning algorithms, three binning modes, and three quality control and refinement methods in the context of the human gut microbiome. Our findings offer multiple insights into optimal MAG generation in this context. We find that metaSPAdes consistently outperforms MEGAHIT in terms of recall (completeness), and that COMEBin overall outperforms alternative binning algorithms, but has lower precision than SemiBin2. While multi-sample binning results in higher precision, as previously shown, single-sample binning has higher recall and leads to better overall performance with modern binners. Binning refinement, which combines bins from multiple different algorithms, leads to reduced performance. We further show that CheckM2 systematically overestimates completeness and underestimates contamination, and that this is partially ameliorated when using GUNC. Finally, we analyze performance at the contig level, and demonstrate that binning algorithms systematically underperform for prophages and fail to bin contigs that are shared between genomes. Overall, MAG-E offers deep insights into successes and gaps in this important analytic process.\n                <\/jats:p>","DOI":"10.64898\/2026.04.06.712906","type":"posted-content","created":{"date-parts":[[2026,4,9]],"date-time":"2026-04-09T17:25:31Z","timestamp":1775755531000},"source":"Crossref","is-referenced-by-count":0,"title":["End-to-end evaluation of pipelines for metagenome-assembled genomes reveals hidden performance gaps"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4697-6079","authenticated-orcid":false,"given":"Izaak","family":"Coleman","sequence":"first","affiliation":[{"name":"Program for Mathematical Genomics, Department of Systems Biology, Columbia University Irving Medical Center, New York, NY"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiong","family":"Ma","sequence":"additional","affiliation":[{"name":"Program for Mathematical Genomics, Department of Systems Biology, Columbia University Irving Medical Center, New York, NY"},{"name":"Department of Biostatistics, Mailman School of Public Health, Columbia University, New York, NY"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9603-6275","authenticated-orcid":false,"given":"Gordon","family":"Qian","sequence":"additional","affiliation":[{"name":"Program for Mathematical Genomics, Department of Systems Biology, Columbia University Irving Medical Center, New York, NY"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yusheng","family":"Jiang","sequence":"additional","affiliation":[{"name":"Program for Mathematical Genomics, Department of Systems Biology, Columbia University Irving Medical Center, New York, NY"},{"name":"Department of Computer Science, Columbia University, New York, NY"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2085-126X","authenticated-orcid":false,"given":"Aya","family":"Brown Kav","sequence":"additional","affiliation":[{"name":"Program for Mathematical Genomics, Department of Systems Biology, Columbia University Irving Medical Center, New York, NY"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0609-0858","authenticated-orcid":false,"given":"Tal","family":"Korem","sequence":"additional","affiliation":[{"name":"Program for Mathematical Genomics, Department of Systems Biology, Columbia University Irving Medical Center, New York, NY"},{"name":"Department of Obstetrics and Gynecology, Columbia University Irving Medical Center, New York, NY"},{"name":"Herbert Irving Comprehensive Cancer Center, Columbia University Irving Medical Center, New York, NY"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"54368","reference":[{"key":"2026041112450699000_2026.04.06.712906v1.1","doi-asserted-by":"publisher","DOI":"10.1038\/s41587-020-0603-3"},{"key":"2026041112450699000_2026.04.06.712906v1.2","doi-asserted-by":"crossref","first-page":"2185","DOI":"10.1038\/s41564-024-01751-5","article-title":"A multi-kingdom collection of 33,804 reference genomes for the human vaginal microbiome","volume":"9","year":"2024","journal-title":"Nat. Microbiol"},{"key":"2026041112450699000_2026.04.06.712906v1.3","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-023-40719-7"},{"key":"2026041112450699000_2026.04.06.712906v1.4","doi-asserted-by":"crossref","first-page":"9959","DOI":"10.1038\/s41467-025-64950-6","article-title":"Integration of metagenome-assembled genomes with clinical isolates expands the genomic landscape of gut-associated Klebsiella pneumoniae","volume":"16","year":"2025","journal-title":"Nat. Commun"},{"key":"2026041112450699000_2026.04.06.712906v1.5","doi-asserted-by":"publisher","DOI":"10.1016\/j.cub.2015.01.014"},{"key":"2026041112450699000_2026.04.06.712906v1.6","doi-asserted-by":"crossref","first-page":"1533","DOI":"10.1038\/s41564-017-0012-7","article-title":"Recovery of nearly 8,000 metagenome-assembled genomes substantially expands the tree of life","volume":"2","year":"2017","journal-title":"Nat. Microbiol"},{"key":"2026041112450699000_2026.04.06.712906v1.7","unstructured":"Brixi, G. et al. Genome modeling and design across all domains of life with Evo 2. Genomics (2025)."},{"key":"2026041112450699000_2026.04.06.712906v1.8","doi-asserted-by":"publisher","DOI":"10.7717\/peerj.7359"},{"key":"2026041112450699000_2026.04.06.712906v1.9","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btv638"},{"key":"2026041112450699000_2026.04.06.712906v1.10","doi-asserted-by":"publisher","DOI":"10.1038\/nmeth.3103"},{"key":"2026041112450699000_2026.04.06.712906v1.11","doi-asserted-by":"publisher","DOI":"10.1038\/s41587-020-00777-4"},{"key":"2026041112450699000_2026.04.06.712906v1.12","doi-asserted-by":"crossref","first-page":"585","DOI":"10.1038\/s41467-023-44290-z","article-title":"Effective binning of metagenomic contigs using contrastive multi-view representation learning","volume":"15","year":"2024","journal-title":"Nat. Commun"},{"key":"2026041112450699000_2026.04.06.712906v1.13","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btad209"},{"key":"2026041112450699000_2026.04.06.712906v1.14","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-019-0965"},{"key":"2026041112450699000_2026.04.06.712906v1.15","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-019-1058-x"},{"key":"2026041112450699000_2026.04.06.712906v1.16","doi-asserted-by":"publisher","DOI":"10.1016\/J.CELL.2019.01.001\/ATTACHMENT\/31D7FB71-DCA8-4659-BC04-FB25BCC69750\/MMC7.XLSX"},{"key":"2026041112450699000_2026.04.06.712906v1.17","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-023-01934-8"},{"key":"2026041112450699000_2026.04.06.712906v1.18","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-023-01940-w"},{"key":"2026041112450699000_2026.04.06.712906v1.19","doi-asserted-by":"publisher","DOI":"10.7717\/peerj.10119"},{"key":"2026041112450699000_2026.04.06.712906v1.20","doi-asserted-by":"crossref","first-page":"4379","DOI":"10.1038\/s41467-020-18164-7","article-title":"Widespread transfer of mobile antibiotic resistance genes within individual gut microbiomes revealed through bacterial Hi-C","volume":"11","year":"2020","journal-title":"Nat. Commun"},{"key":"2026041112450699000_2026.04.06.712906v1.21","doi-asserted-by":"publisher","DOI":"10.1038\/nmeth.4458"},{"key":"2026041112450699000_2026.04.06.712906v1.22","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-022-01431-4"},{"key":"2026041112450699000_2026.04.06.712906v1.23","doi-asserted-by":"crossref","first-page":"2865","DOI":"10.1038\/s41467-025-57957-6","article-title":"Benchmarking metagenomic binning tools on real datasets across sequencing platforms and binning modes","volume":"16","year":"2025","journal-title":"Nat. Commun"},{"key":"2026041112450699000_2026.04.06.712906v1.24","first-page":"e000436","article-title":"Metagenome-assembled genome binning methods with short reads disproportionately fail for plasmids and genomic Islands","volume":"6","year":"2020","journal-title":"Microb. Genom"},{"key":"2026041112450699000_2026.04.06.712906v1.25","doi-asserted-by":"crossref","unstructured":"Shaw, J. & Yu, Y. W . Rapid species-level metagenome profiling and containment estimation with sylph. Nat. Biotechnol. 1\u201312 (2024).","DOI":"10.1038\/s41587-024-02412-y"},{"key":"2026041112450699000_2026.04.06.712906v1.26","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/bty630"},{"key":"2026041112450699000_2026.04.06.712906v1.27","doi-asserted-by":"publisher","DOI":"10.1006\/jmbi.1990.9999"},{"key":"2026041112450699000_2026.04.06.712906v1.28","doi-asserted-by":"publisher","DOI":"10.1186\/s40168-022-01410-z"},{"key":"2026041112450699000_2026.04.06.712906v1.29","doi-asserted-by":"publisher","DOI":"10.1016\/j.cell.2015.11.001"},{"key":"2026041112450699000_2026.04.06.712906v1.30","doi-asserted-by":"publisher","DOI":"10.1186\/s40168-019-0633-6"},{"key":"2026041112450699000_2026.04.06.712906v1.31","doi-asserted-by":"publisher","DOI":"10.1186\/s13059-016-0997-x"},{"key":"2026041112450699000_2026.04.06.712906v1.32","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btv033"},{"key":"2026041112450699000_2026.04.06.712906v1.33","doi-asserted-by":"publisher","DOI":"10.1101\/gr.213959.116"},{"key":"2026041112450699000_2026.04.06.712906v1.34","doi-asserted-by":"crossref","first-page":"836","DOI":"10.1038\/s41564-018-0171-1","article-title":"Recovery of genomes from metagenomes via a dereplication, aggregation and scoring strategy","volume":"3","year":"2018","journal-title":"Nat. Microbiol"},{"key":"2026041112450699000_2026.04.06.712906v1.35","doi-asserted-by":"publisher","DOI":"10.1101\/2024.05.06.592671"},{"key":"2026041112450699000_2026.04.06.712906v1.36","doi-asserted-by":"publisher","DOI":"10.1038\/s41587-023-01953-y"},{"key":"2026041112450699000_2026.04.06.712906v1.37","doi-asserted-by":"publisher","DOI":"10.1038\/nbt.3893"},{"key":"2026041112450699000_2026.04.06.712906v1.38","doi-asserted-by":"publisher","DOI":"10.1186\/s13059-021-02393-0"},{"key":"2026041112450699000_2026.04.06.712906v1.39","doi-asserted-by":"crossref","first-page":"1245","DOI":"10.1038\/s42256-024-00908-5","article-title":"A multi-modal deep language model for contaminant removal from metagenome-assembled genomes","volume":"6","year":"2024","journal-title":"Nat. Mach. Intell"},{"key":"2026041112450699000_2026.04.06.712906v1.40","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gkac294"},{"key":"2026041112450699000_2026.04.06.712906v1.41","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-024-02823-z"},{"key":"2026041112450699000_2026.04.06.712906v1.42","doi-asserted-by":"publisher","DOI":"10.1186\/s12866-023-02765-5"},{"key":"2026041112450699000_2026.04.06.712906v1.43","doi-asserted-by":"crossref","first-page":"2393791","DOI":"10.1080\/19490976.2024.2393791","article-title":"MRGM: an enhanced catalog of mouse gut microbial genomes substantially broadening taxonomic and functional landscapes","volume":"16","year":"2024","journal-title":"Gut Microbes"},{"key":"2026041112450699000_2026.04.06.712906v1.44","doi-asserted-by":"publisher","DOI":"10.1186\/s12859-020-03667-3"},{"key":"2026041112450699000_2026.04.06.712906v1.45","doi-asserted-by":"publisher","DOI":"10.1093\/gigascience\/giy069"},{"key":"2026041112450699000_2026.04.06.712906v1.46","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gkae1038"},{"key":"2026041112450699000_2026.04.06.712906v1.47","doi-asserted-by":"publisher","DOI":"10.1038\/ismej.2017.126"},{"key":"2026041112450699000_2026.04.06.712906v1.48","doi-asserted-by":"publisher","DOI":"10.1038\/nmeth.1923"},{"key":"2026041112450699000_2026.04.06.712906v1.49","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btp352"},{"key":"2026041112450699000_2026.04.06.712906v1.50","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btu170"},{"key":"2026041112450699000_2026.04.06.712906v1.51","doi-asserted-by":"publisher","DOI":"10.1038\/s41587-023-01688-w"},{"key":"2026041112450699000_2026.04.06.712906v1.52","doi-asserted-by":"crossref","first-page":"3139","DOI":"10.21105\/joss.03139","article-title":"performance: An R Package for Assessment, Comparison and Testing of Statistical Models","volume":"6","year":"2021","journal-title":"Journal of Open Source Software"},{"key":"2026041112450699000_2026.04.06.712906v1.53","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v067.i0"},{"key":"2026041112450699000_2026.04.06.712906v1.54","unstructured":"Lenth, R. V. & Piaskowski, J. Emmeans: Estimated Marginal Means, Aka Least-Squares Means. https:\/\/rvlenth.github.io\/emmeans\/ (2025)."},{"key":"2026041112450699000_2026.04.06.712906v1.55","unstructured":"R Core Team. R: A Language and Environment for Statistical Computing. https:\/\/www.R-project.org\/ (2021)."},{"key":"2026041112450699000_2026.04.06.712906v1.56","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-019-1065-y"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.64898\/2026.04.06.712906","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,11]],"date-time":"2026-04-11T19:45:16Z","timestamp":1775936716000},"score":1,"resource":{"primary":{"URL":"http:\/\/biorxiv.org\/lookup\/doi\/10.64898\/2026.04.06.712906"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,9]]},"references-count":56,"URL":"https:\/\/doi.org\/10.64898\/2026.04.06.712906","relation":{},"subject":[],"published":{"date-parts":[[2026,4,9]]},"subtype":"preprint"}}