{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T16:15:26Z","timestamp":1786724126574,"version":"build-2736575974"},"reference-count":63,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2025,9,3]],"date-time":"2025-09-03T00:00:00Z","timestamp":1756857600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2105564, 2236987, 2346133, 2350189, 2402383"],"award-info":[{"award-number":["2105564, 2236987, 2346133, 2350189, 2402383"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100016682","name":"VMware","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100016682","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2025,9,3]]},"abstract":"<jats:p>High-quality environment lighting is essential for creating immersive mobile augmented reality (AR) experiences. However, achieving visually coherent estimation for mobile AR is challenging due to several key limitations in AR device sensing capabilities, including low camera FoV and limited pixel dynamic ranges. Recent advancements in generative AI, which can generate high-quality images from different types of prompts, including texts and images, present a potential solution for high-quality lighting estimation. Still, to effectively use generative image diffusion models, we must address two key limitations of content quality and slow inference. In this work, we design and implement a generative lighting estimation system called CleAR that can produce high-quality, diverse environment maps in the format of 360\u00b0 HDR images. Specifically, we design a two-step generation pipeline guided by AR environment context data to ensure the output aligns with the physical environment's visual context and color appearance. To improve the estimation robustness under different lighting conditions, we design a real-time refinement component to adjust lighting estimation results on AR devices. To train and test our generative models, we curate a large-scale environment lighting estimation dataset with diverse lighting conditions. Through a combination of quantitative and qualitative evaluations, we show that CleAR outperforms state-of-the-art lighting estimation methods on both estimation accuracy, latency, and robustness, and is rated by 31 participants as producing better renderings for most virtual objects. For example, CleAR achieves 51% to 56% accuracy improvement on virtual object renderings across objects of three distinctive types of materials and reflective properties. CleAR produces lighting estimates of comparable or better quality in just 3.2 seconds---over 110X faster than state-of-the-art methods. Moreover, CleAR supports real-time refinement of lighting estimation results, ensuring robust and timely updates for AR applications.<\/jats:p>","DOI":"10.1145\/3749535","type":"journal-article","created":{"date-parts":[[2025,9,3]],"date-time":"2025-09-03T17:15:45Z","timestamp":1756919745000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["CleAR: Robust Context-Guided Generative Lighting Estimation for Mobile Augmented Reality"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1044-4732","authenticated-orcid":false,"given":"Yiqin","family":"Zhao","sequence":"first","affiliation":[{"name":"Worcester Polytechnic Institute, Worcester, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8855-036X","authenticated-orcid":false,"given":"Mallesham","family":"Dasari","sequence":"additional","affiliation":[{"name":"Northeastern University, Boston, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0060-2266","authenticated-orcid":false,"given":"Tian","family":"Guo","sequence":"additional","affiliation":[{"name":"Worcester Polytechnic Institute, Worcester, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,9,3]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2024. Poly Haven. https:\/\/polyhaven.com\/. Accessed: 2024-05-01."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00147"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01115"},{"key":"e_1_2_1_4_1","volume-title":"Retrieved","author":"Apple Inc","year":"2022","unstructured":"Inc Apple. 2022. Augmented Reality - Apple Developer. Retrieved Oct 14, 2022 from https:\/\/developer.apple.com\/augmented-reality\/"},{"key":"e_1_2_1_5_1","volume-title":"Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). https:\/\/openreview.net\/forum?id=tjZjv_qh_CE","author":"Baruch Gilad","year":"2021","unstructured":"Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman. 2021. ARKitScenes - A Diverse Real-World Dataset for 3D Indoor Scene Understanding Using Mobile RGB-D Data. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). https:\/\/openreview.net\/forum?id=tjZjv_qh_CE"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3561972"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.00741"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-006-0002-3"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/3DV.2017.00081"},{"key":"e_1_2_1_10_1","unstructured":"Guanling Chen and David Kotz. 2000. A survey of context-aware mobile computing research. (2000)."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3550454.3555447"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ausmj.2020.04.004"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2023.3261988"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1185657.1185686"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3596711.3596779"},{"key":"e_1_2_1_16_1","volume-title":"Rafal Mantiuk, and Marta Mrak.","author":"Dufaux Frederic","year":"2016","unstructured":"Frederic Dufaux, Patrick Le Callet, Rafal Mantiuk, and Marta Mrak. 2016. High dynamic range video: from acquisition, to display and applications. Academic Press."},{"key":"e_1_2_1_17_1","volume-title":"Depth map prediction from a single image using a multi-scale deep network. Advances in neural information processing systems 27","author":"Eigen David","year":"2014","unstructured":"David Eigen, Christian Puhrsch, and Rob Fergus. 2014. Depth map prediction from a single image using a multi-scale deep network. Advances in neural information processing systems 27 (2014)."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01268"},{"key":"e_1_2_1_19_1","volume-title":"Learning to Predict Indoor Illumination from a Single Image. ACM Transactions on Graphics","author":"Gardner Marc-Andr\u00e9","year":"2017","unstructured":"Marc-Andr\u00e9 Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagn\u00e9, and Jean-Fran\u00e7ois Lalonde. 2017. Learning to Predict Indoor Illumination from a Single Image. ACM Transactions on Graphics (2017)."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00727"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3422622"},{"key":"e_1_2_1_22_1","volume-title":"Archives of the game developers conference","author":"Green Robin","unstructured":"Robin Green. 2003. Spherical harmonic lighting: The gritty details. In Archives of the game developers conference, Vol. 56. 4."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448103"},{"key":"e_1_2_1_24_1","volume-title":"Denoising diffusion probabilistic models. Advances in neural information processing systems 33","author":"Ho Jonathan","year":"2020","unstructured":"Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in neural information processing systems 33 (2020), 6840--6851."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00145"},{"key":"e_1_2_1_26_1","volume-title":"Understanding color models: a review. ARPN Journal of science and technology 2, 3","author":"Ibraheem Noor A","year":"2012","unstructured":"Noor A Ibraheem, Mokhtar M Hasan, Rafiqul Z Khan, and Pramod K Mishra. 2012. Understanding color models: a review. ARPN Journal of science and technology 2, 3 (2012), 265--275."},{"key":"e_1_2_1_27_1","volume-title":"Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114","author":"Kingma Diederik P","year":"2013","unstructured":"Diederik P Kingma. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3474085.3475413"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306307.3328173"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3595921"},{"key":"e_1_2_1_31_1","volume-title":"Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784","author":"Mirza Mehdi","year":"2014","unstructured":"Mehdi Mirza. 2014. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)."},{"key":"e_1_2_1_32_1","doi-asserted-by":"crossref","unstructured":"Hiroki Noguchi and Toshihiko Sakaguchi. 1999. Effect of illuminance and color temperature on lowering of physiological activity. Applied human science 18 4(1999) 117--123.","DOI":"10.2114\/jpa.18.117"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1080\/15502724.2014.839020"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459872"},{"key":"e_1_2_1_35_1","doi-asserted-by":"crossref","unstructured":"Pakkapon Phongthawee Worameth Chinchuthakun Nontaphat Sinsunthithet Amit Raj Varun Jampani Pramook Khungurn and Supasorn Suwajanakorn. 2023. DiffusionLight: Light Probes for Free by Painting a Chrome Ball. In ArXiv.","DOI":"10.1109\/CVPR52733.2024.00018"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3307334.3326098"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jretconser.2019.03.004"},{"key":"e_1_2_1_38_1","volume-title":"International conference on machine learning. PMLR, 1060--1069","author":"Reed Scott","year":"2016","unstructured":"Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. 2016. Generative adversarial text to image synthesis. In International conference on machine learning. PMLR, 1060--1069."},{"key":"e_1_2_1_39_1","doi-asserted-by":"crossref","unstructured":"Robin Rombach Andreas Blattmann Dominik Lorenz Patrick Esser and Bj\u00f6rn Ommer. 2021. High-Resolution Image Synthesis with Latent Diffusion Models. arXiv:2112.10752 [cs.CV]","DOI":"10.1109\/CVPR52688.2022.01042"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jobe.2022.103998"},{"key":"e_1_2_1_41_1","unstructured":"Maximilian Seitzer. 2020. pytorch-fid: FID Score for PyTorch. https:\/\/github.com\/mseitzer\/pytorch-fid. Version 0.3.0."},{"key":"e_1_2_1_42_1","doi-asserted-by":"crossref","unstructured":"Gowri Somanath and Daniel Kurz. 2021. HDR Environment Map Estimation for Real-Time Augmented Reality. https:\/\/arxiv.org\/pdf\/2011.10687.pdf","DOI":"10.1109\/CVPR46437.2021.01114"},{"key":"e_1_2_1_43_1","volume-title":"Neural Illumination: Lighting Prediction for Indoor Environments. CVPR","author":"Song Shuran","year":"2019","unstructured":"Shuran Song and Thomas Funkhouser. 2019. Neural Illumination: Lighting Prediction for Indoor Environments. CVPR (2019)."},{"key":"e_1_2_1_44_1","volume-title":"Digest of Papers. Second International Symposium on Wearable Computers (Cat. No. 98EX215)","author":"Starner Thad","unstructured":"Thad Starner, Bernt Schiele, and Alex Pentland. [n. d.]. Visual contextual awareness in wearable computing. In Digest of Papers. Second International Symposium on Wearable Computers (Cat. No. 98EX215)."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISMAR-Adjunct51615.2020.00072"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.array.2022.100222"},{"key":"e_1_2_1_47_1","doi-asserted-by":"crossref","unstructured":"Daniel Vlasic Pieter Peers Ilya Baran Paul Debevec Jovan Popovi\u0107 Szymon Rusinkiewicz and Wojciech Matusik. 2009. Dynamic shape capture using multi-view photometric stereo. In ACM SIGGRAPH Asia 2009 papers. 1--11.","DOI":"10.1145\/1661412.1618520"},{"key":"e_1_2_1_48_1","volume-title":"StyleLight: HDR Panorama Generation for Lighting Estimation and Editing. In European Conference on Computer Vision (ECCV).","author":"Wang Guangcong","year":"2022","unstructured":"Guangcong Wang, Yinuo Yang, Chen Change Loy, and Ziwei Liu. 2022. StyleLight: HDR Panorama Generation for Lighting Estimation and Editing. In European Conference on Computer Vision (ECCV)."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3379337.3415815"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401132.1401170"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01228-1_26"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3554729"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/311535.311559"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.00355"},{"key":"e_1_2_1_55_1","volume-title":"Unipc: A unified predictor-corrector framework for fast sampling of diffusion models. Advances in Neural Information Processing Systems 36","author":"Zhao Wenliang","year":"2024","unstructured":"Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. 2024. Unipc: A unified predictor-corrector framework for fast sampling of diffusion models. Advances in Neural Information Processing Systems 36 (2024)."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3572864.3580337"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58592-1_40"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458864.3467886"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3638550.3643617"},{"key":"e_1_2_1_60_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3550291","article-title":"LITAR: Visually Coherent Lighting for Mobile Augmented Reality","volume":"6","author":"Zhao Yiqin","year":"2022","unstructured":"Yiqin Zhao, Chongyang Ma, Haibin Huang, and Tian Guo. 2022. LITAR: Visually Coherent Lighting for Mobile Augmented Reality. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 3 (2022), 1--29.","journal-title":"Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3503161.3548386"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58545-7_30"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.544"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3749535","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3749535","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,25]],"date-time":"2025-09-25T16:30:49Z","timestamp":1758817849000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3749535"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,3]]},"references-count":63,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,9,3]]}},"alternative-id":["10.1145\/3749535"],"URL":"https:\/\/doi.org\/10.1145\/3749535","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,3]]},"assertion":[{"value":"2025-09-03","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}