{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,12]],"date-time":"2025-11-12T09:10:00Z","timestamp":1762938600552,"version":"3.45.0"},"reference-count":77,"publisher":"IOP Publishing","issue":"4","license":[{"start":{"date-parts":[[2025,11,12]],"date-time":"2025-11-12T00:00:00Z","timestamp":1762905600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2025,11,12]],"date-time":"2025-11-12T00:00:00Z","timestamp":1762905600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2025,12,30]]},"abstract":"Abstract<\/jats:title>\n \n Unsupervised pre-training on vast amounts of graph data is critical in real-world applications wherein labeled data is limited, such as molecule properties prediction or materials science. Existing approaches pre-train models for specific graph domains, neglecting the inherent connections within networks. This limits their ability to transfer knowledge to various supervised tasks. In this work, we propose a novel pre-training strategy on graphs that focuses on modeling their multi-resolution structural information, allowing us to capture global information of the whole graph while preserving local structures around its nodes. We extend the work of Graph\n Wave<\/jats:bold>\n let\n P<\/jats:bold>\n ositional\n E<\/jats:bold>\n ncoding (WavePE) from Ngo\n et al<\/jats:italic>\n (2023\n J. Chem. Phys.<\/jats:italic>\n 159<\/jats:bold>\n 034109) by pretraining a\n H<\/jats:bold>\n igh-\n O<\/jats:bold>\n rder\n P<\/jats:bold>\n ermutation-\n E<\/jats:bold>\n quivariant Autoencoder (HOPE-WavePE) to reconstruct node connectivities from their multi-resolution wavelet signals. Since our approach relies solely on the graph structure, it is domain-agnostic and adaptable to datasets from various domains, therefore paving the way for developing general graph structure encoders and graph foundation models. We theoretically demonstrate that for\n k<\/jats:italic>\n given resolutions, the width required for the autoencoder to learn arbitrarily long-range information is\n \n \n \n <\/jats:tex-math>\n \n \n O<\/mml:mi>\n \n (<\/mml:mo>\n \n n<\/mml:mi>\n \n 1<\/mml:mn>\n \n \/<\/mml:mo>\n <\/mml:mrow>\n k<\/mml:mi>\n <\/mml:mrow>\n <\/mml:msup>\n \n r<\/mml:mi>\n \n 1<\/mml:mn>\n +<\/mml:mo>\n 1<\/mml:mn>\n \n \/<\/mml:mo>\n <\/mml:mrow>\n k<\/mml:mi>\n <\/mml:mrow>\n <\/mml:msup>\n \n \u03f5<\/mml:mi>\n \n \u2212<\/mml:mo>\n 1<\/mml:mn>\n \n \/<\/mml:mo>\n <\/mml:mrow>\n k<\/mml:mi>\n <\/mml:mrow>\n <\/mml:msup>\n )<\/mml:mo>\n <\/mml:mrow>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:inline-formula>\n where\n \n \n \n <\/jats:tex-math>\n \n \n n<\/mml:mi>\n ,<\/mml:mo>\n r<\/mml:mi>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:inline-formula>\n denote the number of nodes and the rank of normalized Laplacian, respectively, and\n \u03b5<\/jats:italic>\n is the error tolerance defined by the Frobenius norm. We also evaluate HOPE-WavePE on graph-level prediction tasks of different areas and show its superiority compared to other methods. Our source code is publicly available at\n https:\/\/github.com\/HySonLab\/WaveletPE<\/jats:ext-link>\n .\n <\/jats:p>","DOI":"10.1088\/2632-2153\/ae1acd","type":"journal-article","created":{"date-parts":[[2025,11,3]],"date-time":"2025-11-03T22:53:10Z","timestamp":1762210390000},"page":"045043","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Range-aware graph positional encoding via high-order pretraining: theory and practice"],"prefix":"10.1088","volume":"6","author":[{"given":"Viet","family":"Anh Nguyen","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nhat","family":"Khang Ngo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5092-3757","authenticated-orcid":true,"given":"Truong-Son","family":"Hy","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"266","published-online":{"date-parts":[[2025,11,12]]},"reference":[{"article-title":"How powerful are graph neural networks?","year":"2019","author":"Xu","key":"mlstae1acdbib1","type":"conference-proceedings"},{"key":"mlstae1acdbib2","first-page":"pp 3438","type":"conference-proceedings","article-title":"Measuring and relieving the over-smoothing problem for graph neural networks from the topological view","volume":"vol 34","author":"Chen","year":"2020"},{"article-title":"Understanding over-squashing and bottlenecks on graphs via curvature","year":"2022","author":"Topping","key":"mlstae1acdbib3","type":"conference-proceedings"},{"key":"mlstae1acdbib4","type":"conference-proceedings","article-title":"Graph transformer networks","volume":"vol 32","author":"Yun","year":"2019"},{"key":"mlstae1acdbib5","first-page":"pp 21618","type":"conference-proceedings","article-title":"Rethinking graph transformers with spectral attention","volume":"vol 34","author":"Kreuzer","year":"2021"},{"article-title":"A generalization of transformer networks to graphs","year":"2020a","author":"Dwivedi","key":"mlstae1acdbib6","type":"preprint"},{"article-title":"Do transformers really perform badly for graph representation?","year":"2021a","author":"Ying","key":"mlstae1acdbib7","type":"conference-proceedings"},{"article-title":"On the connection between mpnn and graph transformer","year":"2023","author":"Cai","key":"mlstae1acdbib8","type":"preprint"},{"key":"mlstae1acdbib9","first-page":"pp 1263","type":"conference-proceedings","article-title":"Neural message passing for quantum chemistry","volume":"vol 70","author":"Gilmer","year":"2017"},{"article-title":"Graph neural networks with learnable structural and positional representations","year":"2022a","author":"Dwivedi","key":"mlstae1acdbib10","type":"conference-proceedings"},{"article-title":"Graph inductive biases in transformers without message passing","year":"2023","author":"Ma","key":"mlstae1acdbib11","type":"conference-proceedings"},{"key":"mlstae1acdbib12","doi-asserted-by":"publisher","DOI":"10.1063\/5.0152833","type":"journal-article","article-title":"Multiresolution graph transformers and wavelet positional encoding for learning long-range and hierarchical structures","volume":"159","author":"Ngo","year":"2023","journal-title":"J. Chem. Phys."},{"article-title":"Graph wavelet neural network","year":"2019b","author":"Xu","key":"mlstae1acdbib13","type":"conference-proceedings"},{"key":"mlstae1acdbib14","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1016\/j.acha.2010.04.005","type":"journal-article","article-title":"Wavelets on graphs via spectral graph theory","volume":"30","author":"Hammond","year":"2011","journal-title":"Appl. Comput. Harmon. Anal."},{"key":"mlstae1acdbib15","doi-asserted-by":"publisher","first-page":"pp 1320","DOI":"10.1145\/3219819.3220025)","type":"conference-proceedings","article-title":"Learning structural node embeddings via diffusion wavelets","author":"Donnat","year":"2018"},{"key":"mlstae1acdbib16","doi-asserted-by":"publisher","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","type":"journal-article","article-title":"A survey on transfer learning","volume":"22","author":"Pan","year":"2010","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"mlstae1acdbib17","first-page":"pp 2712","type":"conference-proceedings","article-title":"Using pre-training can improve model robustness and uncertainty","volume":"vol 97)","author":"Hendrycks","year":"2019"},{"key":"mlstae1acdbib18","doi-asserted-by":"publisher","first-page":"2544","DOI":"10.1038\/s41467-018-04948-5","type":"journal-article","article-title":"Prioritizing network communities","volume":"9","author":"Zitnik","year":"2018","journal-title":"Nat. Commun."},{"article-title":"Sign and basis invariant networks for spectral graph representation learning","year":"2022","author":"Lim","key":"mlstae1acdbib19","type":"other"},{"article-title":"Equivariant and stable positional encoding for more powerful graph neural networks","year":"2022a","author":"Wang","key":"mlstae1acdbib20","type":"conference-proceedings"},{"key":"mlstae1acdbib21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.5555\/3648699.3648742","type":"journal-article","article-title":"Benchmarking graph neural networks","volume":"24","author":"Dwivedi","year":"2023","journal-title":"J. Mach.: Learn. Res."},{"article-title":"A generalization of transformer networks to graphs","year":"2020b","author":"Dwivedi","key":"mlstae1acdbib22","type":"preprint"},{"key":"mlstae1acdbib23","type":"journal-article","article-title":"Self-attention in colors: Another take on encoding graph structure in transformers","author":"Menegaux","year":"2023","journal-title":"Trans. Mach. Learn. Res."},{"key":"mlstae1acdbib24","first-page":"pp 14582","type":"conference-proceedings","article-title":"Pure transformers are powerful graph learners","volume":"vol 35","author":"Kim","year":"2022"},{"article-title":"Recipe for a general, powerful, scalable graph transformer","year":"2022","author":"Ramp\u00e1\u0161ek","key":"mlstae1acdbib25","type":"preprint"},{"article-title":"Graphit: encoding graph structure in transformers","year":"2021","author":"Mialon","key":"mlstae1acdbib26","type":"preprint"},{"key":"mlstae1acdbib27","first-page":"p 33","type":"conference-proceedings","article-title":"Distance encoding: design provably more powerful neural networks for graph representation learning","author":"Li","year":"2020"},{"key":"mlstae1acdbib28","first-page":"pp 28877","type":"conference-proceedings","article-title":"Do transformers really perform badly for graph representation?","volume":"vol 34","author":"Ying","year":"2021b"},{"article-title":"Graph positional and structural encoder","year":"2023","author":"Liu","key":"mlstae1acdbib29","type":"preprint"},{"key":"mlstae1acdbib30","first-page":"pp 1877","type":"conference-proceedings","article-title":"Language models are few-shot learners","volume":"vol 33","author":"Brown","year":"2020"},{"article-title":"Gemini: a family of highly capable multimodal models","year":"2023","author":"Team","key":"mlstae1acdbib31","type":"preprint"},{"key":"mlstae1acdbib32","first-page":"pp 9729","type":"conference-proceedings","article-title":"Momentum contrast for unsupervised visual representation learning","author":"He","year":"2020"},{"key":"mlstae1acdbib33","first-page":"pp 8748","type":"conference-proceedings","article-title":"Learning transferable visual models from natural language supervision","author":"Radford","year":"2021"},{"key":"mlstae1acdbib34","first-page":"pp 12121","type":"conference-proceedings","article-title":"Graph contrastive learning automated","volume":"vol 139","author":"You","year":"2021"},{"key":"mlstae1acdbib35","first-page":"pp 11548","type":"conference-proceedings","article-title":"Self-supervised graph-level representation learning with local and global structure","volume":"vol 139","author":"Xu","year":"2021a"},{"article-title":"An empirical study of graph contrastive learning","year":"2021","author":"Zhu","key":"mlstae1acdbib36","type":"preprint"},{"key":"mlstae1acdbib37","first-page":"pp 5812","type":"conference-proceedings","article-title":"Graph contrastive learning with augmentations","volume":"vol 33","author":"You","year":"2020a"},{"key":"mlstae1acdbib38","first-page":"pp 222","type":"conference-proceedings","article-title":"Sub-graph contrast for scalable self-supervised graph representation learning","author":"Jiao","year":"2020"},{"key":"mlstae1acdbib39","doi-asserted-by":"publisher","DOI":"10.1088\/2632-2153\/acc0d8","type":"journal-article","article-title":"Multiresolution equivariant graph variational autoencoder","volume":"4","author":"Hy","year":"2023","journal-title":"Mach. Learn.: Sci. Technol."},{"article-title":"The general theory of permutation equivarant neural networks and higher order graph variational encoders","year":"2020","author":"Thiede","key":"mlstae1acdbib40","type":"preprint"},{"key":"mlstae1acdbib41","doi-asserted-by":"publisher","first-page":"513","DOI":"10.1039\/C7SC02664A","type":"journal-article","article-title":"Moleculenet: a benchmark for molecular machine learning","volume":"9","author":"Wu","year":"2018","journal-title":"Chem. Sci."},{"article-title":"Long range graph benchmark","year":"2022b","author":"Dwivedi","key":"mlstae1acdbib42","type":"conference-proceedings"},{"article-title":"On the stability of expressive positional encodings for graphs","year":"2024","author":"Huang","key":"mlstae1acdbib43","type":"preprint"},{"article-title":"Equivariant and stable positional encoding for more powerful graph neural networks","year":"2022","author":"Wang","key":"mlstae1acdbib44","type":"preprint"},{"key":"mlstae1acdbib45","doi-asserted-by":"publisher","first-page":"889","DOI":"10.1007\/s00211-019-01097-9","type":"journal-article","article-title":"Explicit error estimates for spline approximation of arbitrary smoothness in isogeometric analysis","volume":"144","author":"Sande","year":"2020","journal-title":"Numer. Math."},{"key":"mlstae1acdbib46","first-page":"pp 12096","type":"conference-proceedings","article-title":"Does GNN pretraining help molecular representation?","volume":"vol 35","author":"Sun","year":"2022"},{"key":"mlstae1acdbib47","doi-asserted-by":"publisher","first-page":"3370","DOI":"10.1021\/acs.jcim.9b00237","type":"journal-article","article-title":"Analyzing learned molecular representations for property prediction","volume":"59","author":"Yang","year":"2019","journal-title":"J. Chem. Inf. Modeling"},{"key":"mlstae1acdbib48","doi-asserted-by":"publisher","first-page":"8749","DOI":"10.1021\/acs.jmedchem.9b00959","type":"journal-article","article-title":"Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism","volume":"63","author":"Xiong","year":"2019","journal-title":"J. Med. Chem."},{"article-title":"Strategies for pre-training graph neural networks","year":"2020","author":"Hu","key":"mlstae1acdbib49","type":"conference-proceedings"},{"key":"mlstae1acdbib50","first-page":"pp 5812","type":"conference-proceedings","article-title":"Graph contrastive learning with augmentations","volume":"vol 33","author":"You","year":"2020"},{"key":"mlstae1acdbib51","first-page":"pp 11548","type":"conference-proceedings","article-title":"Self-supervised graph-level representation learning with local and global structure","volume":"vol 139","author":"Xu","year":"2021"},{"article-title":"Self-supervised graph transformer on large-scale molecular data","year":"2020","author":"Rong","key":"mlstae1acdbib52","type":"conference-proceedings"},{"key":"mlstae1acdbib53","first-page":"pp 68028","type":"conference-proceedings","article-title":"Evaluating self-supervised learning for molecular graph embeddings","volume":"vol 36","author":"Wang","year":"2023"},{"key":"mlstae1acdbib54","type":"conference-proceedings","article-title":"N-gram graph: simple unsupervised representation for graphs, with applications to molecules","volume":"vol 32","author":"Liu","year":"2019"},{"article-title":"Tudataset: a collection of benchmark datasets for learning with graphs","year":"2020","author":"Morris","key":"mlstae1acdbib55","type":"preprint"},{"article-title":"Invariant and equivariant graph networks","year":"2019","author":"Maron","key":"mlstae1acdbib56","type":"conference-proceedings"},{"key":"mlstae1acdbib57","first-page":"pp 129","type":"book","article-title":"On graph kernels: Hardness results and efficient alternatives","author":"G\u00e4rtner","year":"2003"},{"key":"mlstae1acdbib58","first-page":"pp 488","type":"conference-proceedings","article-title":"Efficient graphlet kernels for large graph comparison","volume":"vol 5","author":"Shervashidze","year":"2009"},{"key":"mlstae1acdbib59","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1007\/s10994-015-5517-9","type":"journal-article","article-title":"Propagation kernels: efficient graph kernels from propagated information","volume":"102","author":"Neumann","year":"2014","journal-title":"Mach. Learn."},{"key":"mlstae1acdbib60","first-page":"2539","type":"journal-article","article-title":"Weisfeiler-lehman graph kernels","volume":"12","author":"Shervashidze","year":"2011","journal-title":"J. Mach.: Learn. Res."},{"article-title":"Diffusion-convolutional neural networks","year":"2016","author":"Atwood","key":"mlstae1acdbib61","type":"other"},{"key":"mlstae1acdbib62","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v32i1.11782","type":"conference-proceedings","article-title":"An end-to-end deep learning architecture for graph classification","author":"Zhang","year":"2018"},{"key":"mlstae1acdbib63","type":"conference-proceedings","article-title":"Provably powerful graph networks","volume":"vol 32","author":"Maron","year":"2019"},{"article-title":"Natural graph networks","year":"2020","author":"de Haan","key":"mlstae1acdbib64","type":"other"},{"key":"mlstae1acdbib65","doi-asserted-by":"publisher","first-page":"657","DOI":"10.1109\/TPAMI.2022.3154319","type":"journal-article","article-title":"Improving graph neural network expressivity via subgraph isomorphism counting","volume":"45","author":"Bouritsas","year":"2023","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"article-title":"Weisfeiler and lehman go topological: message passing simplicial networks","year":"2021","author":"Bodnar","key":"mlstae1acdbib66","type":"other"},{"article-title":"Weisfeiler and lehman go cellular: Cw networks","year":"2022","author":"Bodnar","key":"mlstae1acdbib67","type":"other"},{"article-title":"Automatic chemical design using a data-driven continuous representation of molecules","year":"2016","author":"G\u00f3mez-Bombarelli","key":"mlstae1acdbib68","type":"preprint"},{"article-title":"Graph Attention Networks","year":"2018","author":"Veli\u010dkovi\u0107","key":"mlstae1acdbib69","type":"conference-proceedings"},{"article-title":"Residual gated graph convnets","year":"2017","author":"Bresson","key":"mlstae1acdbib70","type":"preprint"},{"article-title":"A generalization of vit\/mlp-mixer to graphs","year":"2023","author":"He","key":"mlstae1acdbib71","type":"other"},{"article-title":"Semi-supervised classification with graph convolutional networks","year":"2016","author":"Kipf","key":"mlstae1acdbib72","type":"preprint"},{"article-title":"Drew: dynamically rewired message passing with delay","year":"2023","author":"Gutteridge","key":"mlstae1acdbib73","type":"other"},{"article-title":"Enhancing graph transformers with hierarchical distance structural encoding","year":"2024","author":"Luo","key":"mlstae1acdbib74","type":"preprint"},{"key":"mlstae1acdbib75","first-page":"1","type":"journal-article","article-title":"Benchmarking graph neural networks","volume":"24","author":"Dwivedi","year":"2023","journal-title":"J. Mach.: Learn. Res."},{"article-title":"Pygsp: graph signal processing in python","year":"2017","author":"Defferrard","key":"mlstae1acdbib76","type":"other"},{"key":"mlstae1acdbib77","doi-asserted-by":"publisher","first-page":"pp 1548","DOI":"10.24963\/ijcai.2021\/214)","type":"conference-proceedings","article-title":"Masked label prediction: unified message passing model for semi-supervised classification","author":"Shi","year":"2021"}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1acd","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1acd\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1acd","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1acd\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1acd\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1acd\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1acd\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1acd\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,12]],"date-time":"2025-11-12T09:05:09Z","timestamp":1762938309000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1acd"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,12]]},"references-count":77,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2025,11,12]]},"published-print":{"date-parts":[[2025,12,30]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/ae1acd","relation":{},"ISSN":["2632-2153"],"issn-type":[{"type":"electronic","value":"2632-2153"}],"subject":[],"published":{"date-parts":[[2025,11,12]]},"assertion":[{"value":"Range-aware graph positional encoding via high-order pretraining: theory and practice","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2025 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2025-03-12","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2025-11-03","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2025-11-12","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}