{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,4]],"date-time":"2026-06-04T17:26:38Z","timestamp":1780593998253,"version":"3.54.1"},"reference-count":46,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,7,24]],"date-time":"2024-07-24T00:00:00Z","timestamp":1721779200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,7,24]],"date-time":"2024-07-24T00:00:00Z","timestamp":1721779200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["IIS-2133650"],"award-info":[{"award-number":["IIS-2133650"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Cheminform"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Pretrained Graph Neural Networks have been widely adopted for various molecular property prediction tasks. Despite their ability to encode structural and relational features of molecules, traditional fine-tuning of such pretrained GNNs on the target task can lead to poor generalization. To address this, we explore the adaptation of pretrained GNNs to the target task by jointly training them with multiple auxiliary tasks. This could enable the GNNs to learn both general and task-specific features, which may benefit the target task. However, a major challenge is to determine the relatedness of auxiliary tasks with the target task. To address this, we investigate multiple strategies to measure the relevance of auxiliary tasks and integrate such tasks by adaptively combining task gradients or by learning task weights via bi-level optimization. Additionally, we propose a novel gradient surgery-based approach, Rotation of Conflicting Gradients (<jats:inline-formula><jats:alternatives><jats:tex-math>$$\\mathop {\\texttt{RCGrad}}\\limits$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mi>RCGrad<\/mml:mi>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula>), that learns to align conflicting auxiliary task gradients through rotation. Our experiments with state-of-the-art pretrained GNNs demonstrate the efficacy of our proposed methods, with improvements of up to 7.7% over fine-tuning. This suggests that incorporating auxiliary tasks along with target task fine-tuning can be an effective way to improve the generalizability of pretrained GNNs for molecular property prediction.<\/jats:p><jats:p><jats:bold>Scientific contribution<\/jats:bold><\/jats:p><jats:p>We introduce a novel framework for adapting pretrained GNNs to molecular tasks using auxiliary learning to address the critical issue of negative transfer. Leveraging novel gradient surgery techniques such as <jats:inline-formula><jats:alternatives><jats:tex-math>$$\\mathop {\\texttt{RCGrad}}\\limits$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mi>RCGrad<\/mml:mi>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula>, the proposed adaptation framework represents a significant departure from the dominant pretraining fine-tuning approach for molecular GNNs. Our contributions are significant for drug discovery research, especially for tasks with limited data, filling a notable gap in the efficient adaptation of pretrained models for molecular GNNs.<\/jats:p>","DOI":"10.1186\/s13321-024-00880-7","type":"journal-article","created":{"date-parts":[[2024,7,24]],"date-time":"2024-07-24T12:05:31Z","timestamp":1721822731000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Enhancing molecular property prediction with auxiliary learning and task-specific adaptation"],"prefix":"10.1186","volume":"16","author":[{"given":"Vishal","family":"Dey","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xia","family":"Ning","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,7,24]]},"reference":[{"key":"880_CR1","unstructured":"(2022) Rdkit: Open-source cheminformatics. http:\/\/www.rdkit.org, accessed on May 2023"},{"key":"880_CR2","first-page":"3634","volume":"162","author":"H Chen","year":"2022","unstructured":"Chen H, Wang X, Guan C et al (2022) Auxiliary learning with joint task and data scheduling. Proc 39 th Int Machine Learn Proc Machine Learn Res 162:3634\u20133647","journal-title":"Proc 39 th Int Machine Learn Proc Machine Learn Res"},{"key":"880_CR3","unstructured":"Chen S, Crammer K, He H, et\u00a0al (2021) Weighted training for cross-task learning. In: International Conference on Learning Representations"},{"key":"880_CR4","unstructured":"Chen Z, Badrinarayanan V, Lee CY, et\u00a0al (2018) Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. International conference on machine learning, PMLR, pp 794\u2013803"},{"issue":"1","key":"880_CR5","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13321-020-00460-5","volume":"12","author":"L David","year":"2020","unstructured":"David L, Thakkar A, Mercado R et al (2020) Molecular representations in ai-driven drug discovery: a review and practical guide. J Cheminform 12(1):1\u201322","journal-title":"J Cheminform"},{"key":"880_CR6","unstructured":"Dery LM, Dauphin Y, Grangier D (2021a) Auxiliary task update decomposition: The good, the bad and the neutral. arXiv preprint arXiv:2108.11346"},{"key":"880_CR7","unstructured":"Dery LM, Michel P, Talwalkar A, et\u00a0al (2021b) Should we be pre-training? an argument for end-task aware training as an alternative. In: International Conference on Learning Representations"},{"key":"880_CR8","unstructured":"Dery LM, Michel P, Khodak M, et\u00a0al (2022) Aang: Automating auxiliary learning. In: The Eleventh International Conference on Learning Representations"},{"key":"880_CR9","unstructured":"Du Y, Czarnecki WM, Jayakumar SM, et\u00a0al (2018) Adapting auxiliary losses using gradient similarity. arXiv preprint arXiv:1812.02224"},{"key":"880_CR10","doi-asserted-by":"publisher","first-page":"681","DOI":"10.1007\/s11023-020-09548-1","volume":"30","author":"L Floridi","year":"2020","unstructured":"Floridi L, Chiriatti M (2020) Gpt-3: Its nature, scope, limits, and consequences. Minds Machines 30:681\u2013694","journal-title":"Minds Machines"},{"key":"880_CR11","first-page":"15421","volume":"34","author":"J Gasteiger","year":"2021","unstructured":"Gasteiger J, Yeshwanth C, G\u00fcnnemann S (2021) Directional message passing on molecular graphs via synthetic coordinates. Adv Neural Inform Process Syst 34:15421\u201315433","journal-title":"Adv Neural Inform Process Syst"},{"key":"880_CR12","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2023\/744","author":"Z Guo","year":"2023","unstructured":"Guo Z, Guo K, Nan B et al (2023) Graph-based molecular representation learning In of the Thirty-Second International Joint Artificial Intelligence, IJCAI-23. Int Joint Conf Artif Intell Organ. https:\/\/doi.org\/10.24963\/ijcai.2023\/744","journal-title":"Int Joint Conf Artif Intell Organ"},{"key":"880_CR13","doi-asserted-by":"crossref","unstructured":"Gururangan S, Marasovi\u0107 A, Swayamdipta S, et\u00a0al (2020) Don\u2019t stop pretraining: Adapt language models to domains and tasks. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp 8342\u20138360","DOI":"10.18653\/v1\/2020.acl-main.740"},{"key":"880_CR14","unstructured":"Hamilton WL, Ying Z, Leskovec J (2017) Inductive Representation Learning on Large Graphs. In: NIPS, pp 1024\u20131034"},{"key":"880_CR15","first-page":"2205","volume":"2022","author":"Y He","year":"2022","unstructured":"He Y, Feng X, Cheng C et al (2022) Metabalance: improving multi-task recommendations via adapting gradient magnitudes of auxiliary tasks. Proc ACM Web Conf 2022:2205\u20132215","journal-title":"Proc ACM Web Conf"},{"key":"880_CR16","first-page":"2790","volume":"97","author":"N Houlsby","year":"2019","unstructured":"Houlsby N, Giurgiu A, Jastrzebski S et al (2019) Parameter-efficient transfer learningfor NLP. Proc 36th Int Conf machine Learn Res 97:2790\u20132799","journal-title":"Proc 36th Int Conf machine Learn Res"},{"key":"880_CR17","unstructured":"Hu W, Liu B, Gomes J, et\u00a0al (2019) Strategies for pre-training graph neural networks. In: International Conference on Learning Representations"},{"key":"880_CR18","doi-asserted-by":"crossref","unstructured":"Hu Z, Dong Y, Wang K, et\u00a0al (2020) Gpt-gnn: Generative pre-training of graph neural networks. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp 1857\u20131867","DOI":"10.1145\/3394486.3403237"},{"key":"880_CR19","first-page":"1296","volume":"2021","author":"Y Lee","year":"2021","unstructured":"Lee Y (2021) Improving end-to-end task-oriented dialog system with a simple auxiliary task. Findings Assoc Computat Linguistics: EMNLP 2021:1296\u20131303","journal-title":"Findings Assoc Computat Linguistics: EMNLP"},{"key":"880_CR20","unstructured":"Liebel L, K\u00f6rner M (2018) Auxiliary tasks in multi-task learning. arXiv preprint arXiv:1805.06334"},{"key":"880_CR21","unstructured":"Liu S, Davison A, Johns E (2019a) Self-supervised generalisation with meta auxiliary learning. Advances in Neural Information Processing Systems 32"},{"key":"880_CR22","unstructured":"Liu S, Wang H, Liu W, et\u00a0al (2021) Pre-training molecular graph representation with 3d geometry. In: International Conference on Learning Representations"},{"key":"880_CR23","unstructured":"Liu Y, Ott M, Goyal N, et\u00a0al (2019b) Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692"},{"key":"880_CR24","unstructured":"Lorraine J, Vicol P, Duvenaud D (2020) Optimizing millions of hyperparameters by implicit differentiation. In: International conference on artificial intelligence and statistics, PMLR, pp 1540\u20131552"},{"key":"880_CR25","unstructured":"Navon A, Achituve I, Maron H, et\u00a0al (2020) Auxiliary learning by implicit differentiation. In: International Conference on Learning Representations"},{"key":"880_CR26","doi-asserted-by":"crossref","unstructured":"Nediyanchath A, Paramasivam P, Yenigalla P (2020) Multi-head attention for speech emotion recognition with auxiliary learning of gender recognition. ICASSP 2020\u20132020 IEEE International Conference on Acoustics. Speech and Signal Processing (ICASSP), IEEE, pp 7179\u20137183","DOI":"10.1109\/ICASSP40776.2020.9054073"},{"key":"880_CR27","unstructured":"Radford A, Narasimhan K, Salimans T, et\u00a0al (2018) Improving language understanding by generative pre-training"},{"key":"880_CR28","first-page":"12559","volume":"33","author":"Y Rong","year":"2020","unstructured":"Rong Y, Bian Y, Xu T et al (2020) Self-supervised graph transformer on large-scale molecular data. Adv Neural Inform Process Syst 33:12559\u201312571","journal-title":"Adv Neural Inform Process Syst"},{"key":"880_CR29","unstructured":"Rosenstein MT, Marx Z, Kaelbling LP, et\u00a0al (2005) To transfer or not to transfer. In: NIPS 2005 workshop on transfer learning"},{"key":"880_CR30","unstructured":"Ruder S (2017) An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098"},{"key":"880_CR31","unstructured":"Shamsian A, Navon A, Glazer N, et\u00a0al (2023) Auxiliary learning as an asymmetric bargaining game. arXiv preprint arXiv:2301.13501"},{"key":"880_CR32","first-page":"7148","volume":"33","author":"B Shi","year":"2020","unstructured":"Shi B, Hoffman J, Saenko K et al (2020) Auxiliary task reweighting for minimum-data learning. Adv Neural Inform Process Syst 33:7148\u20137160","journal-title":"Adv Neural Inform Process Syst"},{"key":"880_CR33","unstructured":"Sun FY, Hoffmann J, Verma V, et\u00a0al (2019) Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv:1908.01000"},{"key":"880_CR34","first-page":"12096","volume":"35","author":"R Sun","year":"2022","unstructured":"Sun R, Dai H, Yu AW (2022) Does gnn pretraining help molecular representation? Adv Neural Inform Process Syst 35:12096\u201312109","journal-title":"Adv Neural Inform Process Syst"},{"key":"880_CR35","unstructured":"Trinh T, Dai A, Luong T, et\u00a0al (2018) Learning longer-term dependencies in rnns with auxiliary losses. In: International Conference on Machine Learning, PMLR, pp 4965\u20134974"},{"key":"880_CR36","unstructured":"Wang H, Liu S, Kaddour J, et\u00a0al (2022a) Evaluating self-supervised learned molecular graphs. In: ICML 2022 2nd AI for Science Workshop"},{"issue":"3","key":"880_CR37","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1038\/s42256-022-00447-x","volume":"4","author":"Y Wang","year":"2022","unstructured":"Wang Y, Wang J, Cao Z et al (2022) Molecular contrastive learning of representations via graph neural networks. Nature Machine Intell 4(3):279\u2013287","journal-title":"Nature Machine Intell"},{"key":"880_CR38","unstructured":"Wei J, Tay Y, Bommasani R, et\u00a0al (2022) Emergent abilities of large language models. arXiv preprint arXiv:2206.07682"},{"key":"880_CR39","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.ddtec.2020.11.009","volume":"37","author":"O Wieder","year":"2020","unstructured":"Wieder O, Kohlbacher S, Kuenemann M et al (2020) A compact review of molecular property prediction with graph neural networks. Drug Discov Today Technol 37:1\u201312","journal-title":"Drug Discov Today Technol"},{"issue":"2","key":"880_CR40","doi-asserted-by":"publisher","first-page":"513","DOI":"10.1039\/C7SC02664A","volume":"9","author":"Z Wu","year":"2018","unstructured":"Wu Z, Ramsundar B, Feinberg EN et al (2018) Moleculenet: a benchmark for molecular machine learning. Chem Sci 9(2):513\u2013530","journal-title":"Chem Sci"},{"key":"880_CR41","doi-asserted-by":"crossref","unstructured":"Xia J, Zheng J, Tan C, et\u00a0al (2022a) Towards effective and generalizable fine-tuning for pre-trained molecular graph models. bioRxiv pp 2022\u201302","DOI":"10.1101\/2022.02.03.479055"},{"key":"880_CR42","unstructured":"Xia J, Zhu Y, Du Y, et\u00a0al (2022b) Pre-training graph neural networks for molecular representations: retrospect and prospect. In: ICML 2022 2nd AI for Science Workshop"},{"key":"880_CR43","unstructured":"Xuhong L, Grandvalet Y, Davoine F (2018) Explicit inductive bias for transfer learning with convolutional networks. In: International Conference on Machine Learning, PMLR, pp 2825\u20132834"},{"key":"880_CR44","doi-asserted-by":"crossref","unstructured":"Yang E, Nair S, Chandradevan R, et\u00a0al (2022) C3: Continued pretraining with contrastive weak supervision for cross language ad-hoc retrieval. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp 2507\u20132512","DOI":"10.1145\/3477495.3531886"},{"key":"880_CR45","first-page":"5824","volume":"33","author":"T Yu","year":"2020","unstructured":"Yu T, Kumar S, Gupta A et al (2020) Gradient surgery for multi-task learning. Adv Neural Inform Process Syst 33:5824\u20135836","journal-title":"Adv Neural Inform Process Syst"},{"key":"880_CR46","doi-asserted-by":"publisher","unstructured":"Zhang J, Xiao X, Huang LK, et\u00a0al (2022) Fine-tuning graph neural networks via graph topology induced optimal transport. In: Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22. International Joint Conferences on Artificial Intelligence Organization, pp 3730\u20133736, https:\/\/doi.org\/10.24963\/ijcai.2022\/518","DOI":"10.24963\/ijcai.2022\/518"}],"container-title":["Journal of Cheminformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13321-024-00880-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13321-024-00880-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13321-024-00880-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,24]],"date-time":"2024-07-24T12:10:05Z","timestamp":1721823005000},"score":1,"resource":{"primary":{"URL":"https:\/\/jcheminf.biomedcentral.com\/articles\/10.1186\/s13321-024-00880-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,24]]},"references-count":46,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["880"],"URL":"https:\/\/doi.org\/10.1186\/s13321-024-00880-7","relation":{},"ISSN":["1758-2946"],"issn-type":[{"value":"1758-2946","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,24]]},"assertion":[{"value":"18 April 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 July 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 July 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"85"}}