{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T16:26:38Z","timestamp":1779380798241,"version":"3.53.1"},"reference-count":47,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2024,11,19]],"date-time":"2024-11-19T00:00:00Z","timestamp":1731974400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2024,12,19]]},"abstract":"<jats:p>\n            Augmented reality (AR) mirrors are novel displays that have great potential for commercial applications such as virtual apparel try-on. Typically the camera is placed beside the display, leading to distorted perspectives during user interaction. In this paper, we present a novel approach to address this problem by placing the camera behind a transparent display, thereby providing users with a perspective-aligned experience. Simply placing the camera behind the display can compromise image quality due to optical effects. We meticulously analyze the image formation process, and present an image restoration algorithm that benefits from physics-based data synthesis and network design. Our method significantly improves image quality and outperforms existing methods especially on the underexplored wire and backscatter artifacts. We then carefully design a full AR mirror system including display and camera selection, real-time processing pipeline, and mechanical design. Our user study demonstrates that the system is exceptionally well-received by users, highlighting its advantages over existing camera configurations not only as an AR mirror, but also for video conferencing. Our work represents a step forward in the development of AR mirrors, with potential applications in retail, cosmetics, fashion,\n            <jats:italic>etc.<\/jats:italic>\n            The image restoration dataset and code are available at https:\/\/perspective-armirror.github.io\/.\n          <\/jats:p>","DOI":"10.1145\/3687995","type":"journal-article","created":{"date-parts":[[2024,11,19]],"date-time":"2024-11-19T15:46:04Z","timestamp":1732031164000},"page":"1-11","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Perspective-Aligned AR Mirror with Under-Display Camera"],"prefix":"10.1145","volume":"43","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5266-3808","authenticated-orcid":false,"given":"Jian","family":"Wang","sequence":"first","affiliation":[{"name":"Snap Inc., New York, United States of America"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0092-9744","authenticated-orcid":false,"given":"Sizhuo","family":"Ma","sequence":"additional","affiliation":[{"name":"Snap Inc., New York, United States of America"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-3330-0189","authenticated-orcid":false,"given":"Karl","family":"Bayer","sequence":"additional","affiliation":[{"name":"Snap Inc., New York, United States of America"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7147-9125","authenticated-orcid":false,"given":"Yi","family":"Zhang","sequence":"additional","affiliation":[{"name":"Snap Inc., New York, United States of America"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0657-1595","authenticated-orcid":false,"given":"Peihao","family":"Wang","sequence":"additional","affiliation":[{"name":"Snap Inc., New York, United States of America"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0838-7858","authenticated-orcid":false,"given":"Bing","family":"Zhou","sequence":"additional","affiliation":[{"name":"Snap Inc., New York, United States of America"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6452-6998","authenticated-orcid":false,"given":"Shree","family":"Nayar","sequence":"additional","affiliation":[{"name":"Columbia University, New York, United States of America"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1533-2169","authenticated-orcid":false,"given":"Gurunandan","family":"Krishnan","sequence":"additional","affiliation":[{"name":"Snap Inc., New York, United States of America"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,11,19]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"UDCSIT: A Real-World Dataset for Under-Display Cameras. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.","author":"Ahn Kyusu","year":"2023","unstructured":"Kyusu Ahn, Byeonghyun Ko, HyunGyu Lee, Chanwoo Park, and Jaejin Lee. 2023. UDCSIT: A Real-World Dataset for Under-Display Cameras. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2501988.2502045"},{"key":"e_1_2_1_3_1","volume-title":"2012 IEEE Virtual Reality Workshops (VRW). IEEE, 115--116","author":"Blum Tobias","year":"2012","unstructured":"Tobias Blum, Valerie Kleeberger, Christoph Bichlmeier, and Nassir Navab. 2012. mirracle: An augmented reality magic mirror system for anatomy education. In 2012 IEEE Virtual Reality Workshops (VRW). IEEE, 115--116."},{"key":"e_1_2_1_4_1","volume-title":"Exploring non-reversing magic mirrors for screen-based augmented reality systems. In 2017 IEEE virtual reality (VR)","author":"Bork Felix","unstructured":"Felix Bork, Roghayeh Barmaki, Ulrich Eck, Pascal Fallavolita, Bernhard Fuerst, and Nassir Navab. 2017. Exploring non-reversing magic mirrors for screen-based augmented reality systems. In 2017 IEEE virtual reality (VR). IEEE, 373--374."},{"key":"e_1_2_1_5_1","volume-title":"The benefits of an augmented reality magic mirror system for integrated radiology teaching in gross anatomy. Anatomical sciences education 12, 6","author":"Bork Felix","year":"2019","unstructured":"Felix Bork, Leonard Stratmann, Stefan Enssle, Ulrich Eck, Nassir Navab, Jens Waschke, and Daniela Kugelmann. 2019. The benefits of an augmented reality magic mirror system for integrated radiology teaching in gross anatomy. Anatomical sciences education 12, 6 (2019), 585--598."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-20071-7_2"},{"key":"e_1_2_1_7_1","volume-title":"Humans perceive flicker artifacts at 500 Hz. Scientific reports 5, 1","author":"Davis James","year":"2015","unstructured":"James Davis, Yi-Hsuan Hsieh, and Hung-Chi Lee. 2015. Humans perceive flicker artifacts at 500 Hz. Scientific reports 5, 1 (2015), 7861."},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 5013--5022","author":"Feng Ruicheng","year":"2023","unstructured":"Ruicheng Feng, Chongyi Li, Huaijin Chen, Shuai Li, Jinwei Gu, and Chen Change Loy. 2023. Generating Aligned Pseudo-Supervision from Non-Aligned Data for Image Restoration in Under-Display Camera. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 5013--5022."},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 662--671","author":"Feng Ruicheng","year":"2021","unstructured":"Ruicheng Feng, Chongyi Li, Huaijin Chen, Shuai Li, Chen Change Loy, and Jinwei Gu. 2021. Removing diffraction image artifacts in under-display camera via dynamic skip connection network. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 662--671."},{"key":"e_1_2_1_10_1","volume-title":"European Conference on Computer Vision. Springer, 60--77","author":"Feng Ruicheng","year":"2022","unstructured":"Ruicheng Feng, Chongyi Li, Shangchen Zhou, Wenxiu Sun, Qingpeng Zhu, Jun Jiang, Qingyu Yang, Chen Change Loy, Jinwei Gu, Yurui Zhu, et al. 2022. Mipi 2022 challenge on under-display camera image restoration: Methods and results. In European Conference on Computer Vision. Springer, 60--77."},{"key":"e_1_2_1_11_1","volume-title":"Multiple view geometry in computer vision","author":"Hartley Richard","unstructured":"Richard Hartley and Andrew Zisserman. 2003. Multiple view geometry in computer vision. Cambridge university press."},{"key":"e_1_2_1_12_1","volume-title":"Single image haze removal using dark channel prior","author":"He Kaiming","year":"2010","unstructured":"Kaiming He, Jian Sun, and Xiaoou Tang. 2010. Single image haze removal using dark channel prior. IEEE transactions on pattern analysis and machine intelligence 33, 12 (2010), 2341--2353."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300630"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3592433"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 1950--1959","author":"Koh Jaihyun","year":"2022","unstructured":"Jaihyun Koh, Jangho Lee, and Sungroh Yoon. 2022. BNUDC: a two-branched deep neural network for restoring images from under-display cameras. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 1950--1959."},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 2073--2082","author":"Kwon Kinam","year":"2021","unstructured":"Kinam Kwon, Eunhee Kang, Sangwon Lee, Su-Jin Lee, Hyong-Euk Lee, ByungIn Yoo, and Jae-Joon Han. 2021. Controllable image restoration for under-display camera in smartphones. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 2073--2082."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3478513.3480490"},{"key":"e_1_2_1_18_1","volume-title":"Transparent OLED Displays. https:\/\/www.lg.com\/us\/business\/digital-signage\/oled-signage\/transparent-oled-displays. [Online","author":"LG.","year":"2024","unstructured":"LG. 2019. Transparent OLED Displays. https:\/\/www.lg.com\/us\/business\/digital-signage\/oled-signage\/transparent-oled-displays. [Online; accessed 1-Jan.-2024]."},{"key":"e_1_2_1_19_1","volume-title":"SID Symposium Digest of Technical Papers","volume":"52","author":"Lim Sehoon","year":"2021","unstructured":"Sehoon Lim, Luming Liang, Yatao Zhong, Neil Emerton, Tim Large, and Steven Bathiche. 2021. 18-3: Free Viewpoint Teleconferencing Using Cameras Behind Screen. In SID Symposium Digest of Technical Papers, Vol. 52. Wiley Online Library, 218--221."},{"key":"e_1_2_1_20_1","volume-title":"European Conference on Computer Vision. Springer, 113--129","author":"Liu Xina","year":"2022","unstructured":"Xina Liu, Jinfan Hu, Xiangyu Chen, and Chao Dong. 2022. UDC-UNet: Under-Display Camera Image Restoration via U-shape Dynamic Network. In European Conference on Computer Vision. Springer, 113--129."},{"key":"e_1_2_1_21_1","volume-title":"Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308.09713","author":"Luiten Jonathon","year":"2023","unstructured":"Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. 2023. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308.09713 (2023)."},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 10050--10060","author":"Luo Katie","year":"2021","unstructured":"Katie Luo, Guandao Yang, Wenqi Xian, Harald Haraldsson, Bharath Hariharan, and Serge Belongie. 2021. Stay positive: Non-negative image synthesis for augmented reality. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 10050--10060."},{"key":"e_1_2_1_23_1","volume-title":"2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 277--278","author":"Meng Ma","year":"2013","unstructured":"Ma Meng, Pascal Fallavollita, Tobias Blum, Ulrich Eck, Christian Sandor, Simon Weidert, Jens Waschke, and Nassir Navab. 2013. Kinect for interactive AR anatomy learning. In 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 277--278."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3503250"},{"key":"e_1_2_1_25_1","volume-title":"Motorola Edge X30. https:\/\/www.gsmarena.com\/motorola_edge_x3011262.php. [Online","year":"2024","unstructured":"Motorola. 2021. Motorola Edge X30. https:\/\/www.gsmarena.com\/motorola_edge_x3011262.php. [Online; accessed 1-Jan.-2024]."},{"key":"e_1_2_1_26_1","volume-title":"Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG) 41, 4","author":"M\u00fcller Thomas","year":"2022","unstructured":"Thomas M\u00fcller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG) 41, 4 (2022), 1--15."},{"key":"e_1_2_1_27_1","volume-title":"NVIDIA Broadcast App. https:\/\/www.nvidia.com\/en-us\/geforce\/broadcasting\/broadcast-app\/. [Online","author":"NVIDIA.","year":"2024","unstructured":"NVIDIA. 2020. NVIDIA Broadcast App. https:\/\/www.nvidia.com\/en-us\/geforce\/broadcasting\/broadcast-app\/. [Online; accessed 17-May.-2024]."},{"key":"e_1_2_1_28_1","volume-title":"Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32","author":"Paszke Adam","year":"2019","unstructured":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019)."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/JDT.2016.2594815"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the 2016 CHI conference on human factors in computing systems. 6058--6063","author":"Saakes Daniel","year":"2016","unstructured":"Daniel Saakes, Hui-Shyong Yeo, Seung-Tak Noh, Gyeol Han, and Woontack Woo. 2016. Mirror mirror: An on-body t-shirt design system. In Proceedings of the 2016 CHI conference on human factors in computing systems. 6058--6063."},{"key":"e_1_2_1_31_1","unstructured":"Samsung. 2023. A look at the Under Display Camera (UDC) on the Galaxy Z Fold4 and Fold5. https:\/\/www.samsung.com\/latin_en\/support\/mobile-devices\/a-look-at-the-under-display-camera-udc-on-the-galaxy-z-fold4-and-fold5\/. [Online; accessed 1-Jan.-2024]."},{"key":"e_1_2_1_32_1","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1109\/TIP.2022.3227503","article-title":"Adapool: Exponential adaptive pooling for information-retaining downsampling","volume":"32","author":"Stergiou Alexandros","year":"2022","unstructured":"Alexandros Stergiou and Ronald Poppe. 2022. Adapool: Exponential adaptive pooling for information-retaining downsampling. IEEE Transactions on Image Processing 32 (2022), 251--266.","journal-title":"IEEE Transactions on Image Processing"},{"key":"e_1_2_1_33_1","volume-title":"Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23--28, 2020, Proceedings, Part V 16","author":"Sundar Varun","year":"2020","unstructured":"Varun Sundar, Sumanth Hegde, Divya Kothandaraman, and Kaushik Mitra. 2020. Deep atrous guided filter for image restoration in under display cameras. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23--28, 2020, Proceedings, Part V 16. Springer, 379--397."},{"key":"e_1_2_1_34_1","volume-title":"Chao Dong, and Chen Change Loy.","author":"Wang Xintao","year":"2018","unstructured":"Xintao Wang, Ke Yu, Kelvin CK Chan, Chao Dong, and Chen Change Loy. 2018a. BasicSR: Open source image and video restoration toolbox. GitHub (2018)."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00070"},{"key":"e_1_2_1_36_1","first-page":"8520","article-title":"Physics-based noise modeling for extreme low-light photography","volume":"44","author":"Wei Kaixuan","year":"2021","unstructured":"Kaixuan Wei, Ying Fu, Yinqiang Zheng, and Jiaolong Yang. 2021. Physics-based noise modeling for extreme low-light photography. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 11 (2021), 8520--8537.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.02134"},{"key":"e_1_2_1_38_1","volume-title":"https:\/\/en.wikipedia.org\/wiki\/Xiaomi_MIX_4. [Online","author":"Xiaomi MIX","year":"2024","unstructured":"Xiaomi. 2021. Xiaomi MIX 4. https:\/\/en.wikipedia.org\/wiki\/Xiaomi_MIX_4. [Online; accessed 1-Jan.-2024]."},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision. 10637--10645","author":"Yang Anqi","year":"2023","unstructured":"Anqi Yang, Eunhee Kang, Hyong-Euk Lee, and Aswin C Sankaranarayanan. 2023. Designing Phase Masks for Under-Display Cameras. In Proceedings of the IEEE\/CVF International Conference on Computer Vision. 10637--10645."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2021.3075978"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00564"},{"key":"e_1_2_1_42_1","volume-title":"ZERO10 is a fashion AR try-on company. https:\/\/zero10.ar\/. [Online","year":"2024","unstructured":"ZERO10. 2024. ZERO10 is a fashion AR try-on company. https:\/\/zero10.ar\/. [Online; accessed 24-Jan.-2024]."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00537"},{"key":"e_1_2_1_44_1","volume-title":"UDC 2020 challenge on image restoration of under-display camera: Methods and results. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23--28, 2020, Proceedings, Part V 16","author":"Zhou Yuqian","year":"2020","unstructured":"Yuqian Zhou, Michael Kwan, Kyle Tolentino, Neil Emerton, Sehoon Lim, Tim Large, Lijiang Fu, Zhihong Pan, Baopu Li, Qirui Yang, et al. 2020. UDC 2020 challenge on image restoration of under-display camera: Methods and results. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23--28, 2020, Proceedings, Part V 16. Springer, 337--351."},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 9179--9188","author":"Zhou Yuqian","year":"2021","unstructured":"Yuqian Zhou, David Ren, Neil Emerton, Sehoon Lim, and Timothy Large. 2021. Image restoration for under-display camera. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 9179--9188."},{"key":"e_1_2_1_46_1","volume-title":"Computer Vision-ECCV 2022 Workshops: Tel Aviv, Israel, October 23--27, 2022, Proceedings, Part V. Springer, 130--146","author":"Zhu Yurui","year":"2023","unstructured":"Yurui Zhu, Xi Wang, Xueyang Fu, and Xiaowei Hu. 2023. Enhanced Coarse-to-Fine Network for Image Restoration from Under-Display Cameras. In Computer Vision-ECCV 2022 Workshops: Tel Aviv, Israel, October 23--27, 2022, Proceedings, Part V. Springer, 130--146."},{"key":"e_1_2_1_47_1","volume-title":"ZTE Axon 20 5G: Path To New Vision The World's First Under-Display Camera Smartphone. https:\/\/ztedevices.com\/en-gl\/zte-axon-20-5g\/. [Online","author":"ZTE.","year":"2024","unstructured":"ZTE. 2020. ZTE Axon 20 5G: Path To New Vision The World's First Under-Display Camera Smartphone. https:\/\/ztedevices.com\/en-gl\/zte-axon-20-5g\/. [Online; accessed 1-Jan.-2024]."}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3687995","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3687995","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:09:58Z","timestamp":1750295398000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3687995"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,19]]},"references-count":47,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,12,19]]}},"alternative-id":["10.1145\/3687995"],"URL":"https:\/\/doi.org\/10.1145\/3687995","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,19]]},"assertion":[{"value":"2024-11-19","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}