{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,18]],"date-time":"2025-10-18T05:11:02Z","timestamp":1760764262070,"version":"build-2065373602"},"reference-count":34,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2025,10,15]],"date-time":"2025-10-15T00:00:00Z","timestamp":1760486400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Key Research and Development Program of China","award":["2023YFE0119600"],"award-info":[{"award-number":["2023YFE0119600"]}]},{"DOI":"10.13039\/501100002886","name":"CNPC 14th Five-Year R&D Project","doi-asserted-by":"publisher","award":["2023DJ8406"],"award-info":[{"award-number":["2023DJ8406"]}],"id":[{"id":"10.13039\/501100002886","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Retrieval-augmented generation (RAG) has established a new search paradigm, in which large language models integrate external resources to compensate for their inherent knowledge limitations. However, limited context awareness reduces the performance of large language models in RAG tasks. Existing solutions incur additional time and memory overhead and depend on specific positional encodings. In this paper, we propose Attention Head Detection and Reweighting (ADR), a lightweight and general framework. Specifically, we employ a recognition task to identify RAG-suppressing heads that limit the model\u2019s context awareness. We then reweight their outputs with learned coefficients to mitigate the influence of these RAG-suppressing heads. After training, the weights are fixed during inference, introducing no additional time overhead and remaining agnostic to the choice of positional embedding. Experiments on PetroAI further demonstrate that ADR enhances the context awareness of fine-tuned models.<\/jats:p>","DOI":"10.3390\/info16100900","type":"journal-article","created":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T11:39:34Z","timestamp":1760701174000},"page":"900","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["ADR: Attention Head Detection and Reweighting Enhance RAG Performance in a Positional-Encoding-Free Paradigm"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-7311-9322","authenticated-orcid":false,"given":"Mingwei","family":"Wang","sequence":"first","affiliation":[{"name":"AI Research Center, Research Institute of Petroleum Exploration and Development, PetroChina, Beijing 100083, China"},{"name":"Artificial Intelligence Technology R & D Center for Exploration and Development, China National Petroleum Corporation, Beijing 100083, China"}]},{"given":"Xiaobo","family":"Li","sequence":"additional","affiliation":[{"name":"AI Research Center, Research Institute of Petroleum Exploration and Development, PetroChina, Beijing 100083, China"},{"name":"Artificial Intelligence Technology R & D Center for Exploration and Development, China National Petroleum Corporation, Beijing 100083, China"},{"name":"National Key Laboratory for Multi-Resources Collaborative Green Production of Continental Shale Oil, Daqing 163712, China"}]},{"given":"Qian","family":"Zeng","sequence":"additional","affiliation":[{"name":"AI Research Center, Research Institute of Petroleum Exploration and Development, PetroChina, Beijing 100083, China"},{"name":"Artificial Intelligence Technology R & D Center for Exploration and Development, China National Petroleum Corporation, Beijing 100083, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-0520-2871","authenticated-orcid":false,"given":"Xingbang","family":"Liu","sequence":"additional","affiliation":[{"name":"AI Research Center, Research Institute of Petroleum Exploration and Development, PetroChina, Beijing 100083, China"},{"name":"Artificial Intelligence Technology R & D Center for Exploration and Development, China National Petroleum Corporation, Beijing 100083, China"}]},{"given":"Minghao","family":"Yang","sequence":"additional","affiliation":[{"name":"AI Research Center, Research Institute of Petroleum Exploration and Development, PetroChina, Beijing 100083, China"},{"name":"Artificial Intelligence Technology R & D Center for Exploration and Development, China National Petroleum Corporation, Beijing 100083, China"}]},{"given":"Zhichen","family":"Jia","sequence":"additional","affiliation":[{"name":"AI Research Center, Research Institute of Petroleum Exploration and Development, PetroChina, Beijing 100083, China"},{"name":"Artificial Intelligence Technology R & D Center for Exploration and Development, China National Petroleum Corporation, Beijing 100083, China"}]}],"member":"1968","published-online":{"date-parts":[[2025,10,15]]},"reference":[{"key":"ref_1","unstructured":"Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., and Dong, Z. (2023). A survey of large language models. arXiv."},{"key":"ref_2","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","volume":"33","author":"Lewis","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1162\/tacl_a_00638","article-title":"Lost in the middle: How language models use long contexts","volume":"12","author":"Liu","year":"2024","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_4","first-page":"79573","article-title":"Mixture of In-context experts enhance LLMs\u2019 long context awareness","volume":"37","author":"Lin","year":"2024","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_5","unstructured":"Zhang, Z., Chen, R., Liu, S., Yao, Z., Ruwase, O., Chen, B., Wu, X., and Wang, Z. (2024, January 9\u201315). Found in the middle: How language models use long contexts better via plug-and-play positional encoding. Proceedings of the Thirty-eighth Annual Conference on Neural Information Processing Systems, Vancouver, BC, Canada."},{"key":"ref_6","unstructured":"Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, J. (2023). Progress measures for grokking via mechanistic interpretability. arXiv."},{"key":"ref_7","unstructured":"Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. (2022). Interpretability in the wild: A circuit for indirect object identification in gpt-2 small. arXiv."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Yu, Q., Merullo, J., and Pavlick, E. (2023, January 6\u201310). Characterizing mechanisms for factual recall in language models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore.","DOI":"10.18653\/v1\/2023.emnlp-main.615"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Geva, M., Schuster, R., Berant, J., and Levy, O. (2021, January 7\u201311). Transformer feed-forward layers are key-value memories. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic.","DOI":"10.18653\/v1\/2021.emnlp-main.446"},{"key":"ref_10","first-page":"17359","article-title":"Locating and editing factual associations in gpt","volume":"35","author":"Meng","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"127063","DOI":"10.1016\/j.neucom.2023.127063","article-title":"Roformer: Enhanced transformer with rotary position embedding","volume":"568","author":"Su","year":"2024","journal-title":"Neurocomputing"},{"key":"ref_12","unstructured":"Press, O., Smith, N.A., and Lewis, M. (2021). Train short, test long: Attention with linear biases enables input length extrapolation. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Chen, Y., Lv, A., Lin, T.E., Chen, C., Wu, Y., Huang, F., Li, Y.B., and Yan, R. (2024, January 11\u201316). Fortify the shortest stave in attention: Enhancing context awareness of large language models for effective tool use. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand.","DOI":"10.18653\/v1\/2024.acl-long.601"},{"key":"ref_14","unstructured":"Zhang, Q., Singh, C., Liu, L., Liu, X., Yu, B., Gao, J., and Zhao, T. (2023, January 1\u20135). Tell your model where to attend: Post-hoc attention steering for LLMs. Proceedings of the Twelfth International Conference on Learning Representations, Kigali, Rwanda."},{"key":"ref_15","unstructured":"Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. (2017, January 24\u201326). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. Proceedings of the International Conference on Learning Representations, Toulon, France."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Li, W., Zhang, Y., Luo, G., Yu, D., and Ji, R. (August, January 27). Training long-context LLMs efficiently via chunk-wise optimization. Proceedings of the Findings of the Association for Computational Linguistics: ACL 2025, Vienna, Austria.","DOI":"10.18653\/v1\/2025.findings-acl.138"},{"key":"ref_17","unstructured":"Li, M., Xu, L.H., Tan, Q., Cao, T., and Liu, Y. (2025). Sculptor: Empowering LLMs with Cognitive Agency via Active Context Management. arXiv."},{"key":"ref_18","unstructured":"Behrouz, A., Li, Z., Kacham, P., Daliri, M., Deng, Y., Zhong, P., Razaviyayn, M., and Mirrokni, V. (2025). Atlas: Learning to optimally memorize the context at test time. arXiv."},{"key":"ref_19","unstructured":"Yang, M.H., Li, X.B., Zeng, Q., and Li, X. (2024). The technical practice of large language models in the upstream business of oil and gas. China CIO News, 61\u201365."},{"key":"ref_20","first-page":"107","article-title":"The Application and Challenges of Large Artificial Intelligence Models in the Field of Oil and Gas Exploration and Development","volume":"43","author":"Yang","year":"2024","journal-title":"Pet. Sci. Technol. Forum."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Wei, Q., Sun, H., Xu, Y., Pang, Z., and Gao, F. (2024). Exploring the application of large language models based AI agents in leakage detection of natural gas valve chambers. Energies, 17.","DOI":"10.3390\/en17225633"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Eckroth, J., Gipson, M., Boden, J., Hough, L., Elliott, J., and Quintana, J. (2023, January 16\u201318). Answering natural language questions with OpenAI\u2019s GPT in the petroleum industry. Proceedings of the SPE Annual Technical Conference and Exhibition?, San Antonio, TX, USA.","DOI":"10.2118\/214888-MS"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Gong, Z., Lv, A., Guan, J., Yan, J., Wu, W., Zhang, H., Huang, M., Zhao, D., and Yan, R. (2024, January 12\u201316). Mixture-of-modules: Reinventing transformers as dynamic assemblies of modules. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, FL, USA.","DOI":"10.18653\/v1\/2024.emnlp-main.1164"},{"key":"ref_24","unstructured":"Merullo, J., Eickhoff, C., and Pavlick, E. (2024, January 7\u201311). Circuit component reuse across tasks in transformer language models. Proceedings of the Twelfth International Conference on Learning Representations, Vienna, Austria."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"e00024.001","DOI":"10.23915\/distill.00024.001","article-title":"Zoom in: An introduction to circuits","volume":"5","author":"Olah","year":"2020","journal-title":"Distill"},{"key":"ref_26","unstructured":"Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv."},{"key":"ref_27","unstructured":"Cao, H., Wu, Y., Cai, Y., Zhao, X., and Ou, Z. (2025). Improving End-to-End Training of Retrieval-Augmented Generation Models via Joint Stochastic Approximation. arXiv."},{"key":"ref_28","unstructured":"Shi, Z., Yan, L., Sun, W., Feng, Y., Ren, P., Ma, X., Wang, S., Yin, D., de Rijke, M., and Ren, Z. (2025). Direct retrieval-augmented optimization: Synergizing knowledge selection and language models. arXiv."},{"key":"ref_29","unstructured":"Yang, P., Li, X., Hu, Z., Wang, J., Yin, J., Wang, H., He, L., Yang, S., Wang, S., and Huang, Y. (2025). HeteRAG: A Heterogeneous Retrieval-augmented Generation Framework with Decoupled Knowledge Representations. arXiv."},{"key":"ref_30","unstructured":"Cong, Y., Akash, P.S., Wang, C., and Chang, K.C.C. (2024). Query optimization for parametric knowledge refinement in retrieval-augmented large language models. arXiv."},{"key":"ref_31","unstructured":"Wang, L., Chen, H., Yang, N., Huang, X., Dou, Z., and Wei, F. (2025). Chain-of-Retrieval Augmented Generation. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Ho, X., Nguyen, A.K.D., Sugawara, S., and Aizawa, A. (2020, January 8\u201313). Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain.","DOI":"10.18653\/v1\/2020.coling-main.580"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"539","DOI":"10.1162\/tacl_a_00475","article-title":"\u266b MuSiQue: Multihop questions via single-hop question composition","volume":"10","author":"Trivedi","year":"2022","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N.A., and Gardner, M. (2021, January 6\u201311). A dataset of information-seeking questions and answers anchored in research papers. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online.","DOI":"10.18653\/v1\/2021.naacl-main.365"}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/10\/900\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,18]],"date-time":"2025-10-18T04:32:35Z","timestamp":1760761955000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/10\/900"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,15]]},"references-count":34,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2025,10]]}},"alternative-id":["info16100900"],"URL":"https:\/\/doi.org\/10.3390\/info16100900","relation":{},"ISSN":["2078-2489"],"issn-type":[{"type":"electronic","value":"2078-2489"}],"subject":[],"published":{"date-parts":[[2025,10,15]]}}}