{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,4]],"date-time":"2026-06-04T13:20:51Z","timestamp":1780579251467,"version":"3.54.1"},"reference-count":37,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2021,5,5]],"date-time":"2021-05-05T00:00:00Z","timestamp":1620172800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Brown Institute for Media Innovation and Nvidia"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2021,6,30]]},"abstract":"<jats:p>We present a system that converts annotated broadcast video of tennis matches into interactively controllable video sprites that behave and appear like professional tennis players. Our approach is based on controllable video textures and utilizes domain knowledge of the cyclic structure of tennis rallies to place clip transitions and accept control inputs at key decision-making moments of point play. Most importantly, we use points from the video collection to model a player\u2019s court positioning and shot selection decisions during points. We use these behavioral models to select video clips that reflect actions the real-life player is likely to take in a given match-play situation, yielding sprites that behave realistically at the macro level of full points, not just individual tennis motions. Our system can generate novel points between professional tennis players that resemble Wimbledon broadcasts, enabling new experiences, such as the creation of matchups between players that have not competed in real life or interactive control of players in the Wimbledon final. According to expert tennis players, the rallies generated using our approach are significantly more realistic in terms of player behavior than video sprite methods that only consider the quality of motion transitions during video synthesis.<\/jats:p>\n          <jats:p>The supplementary material\/video are available at our https:\/\/cs.stanford.edu\/~haotianz\/research\/vid2player\/ project website.<\/jats:p>","DOI":"10.1145\/3448978","type":"journal-article","created":{"date-parts":[[2021,5,6]],"date-time":"2021-05-06T05:25:11Z","timestamp":1620278711000},"page":"1-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":24,"title":["Vid2Player: Controllable Video Sprites That Behave and Appear Like Professional Tennis Players"],"prefix":"10.1145","volume":"40","author":[{"given":"Haotian","family":"Zhang","sequence":"first","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cristobal","family":"Sciutto","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maneesh","family":"Agrawala","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kayvon","family":"Fatahalian","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,5,5]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00282"},{"key":"e_1_2_2_2_1","volume-title":"The OpenCV library. Dr. Dobb\u2019s J. Softw. Tools","author":"Bradski G.","year":"2000","unstructured":"G. Bradski . 2000. The OpenCV library. Dr. Dobb\u2019s J. Softw. Tools ( 2000 ). G. Bradski. 2000. The OpenCV library. Dr. Dobb\u2019s J. Softw. Tools (2000)."},{"key":"e_1_2_2_3_1","unstructured":"H. Brody R. Cross and C. Lindsey. 2004. The Physics and Technology of Tennis. Racquet Tech Publishing.  H. Brody R. Cross and C. Lindsey. 2004. The Physics and Technology of Tennis. Racquet Tech Publishing."},{"key":"e_1_2_2_4_1","volume-title":"Proceedings of the IEEE International Conference on Computer Vision (ICCV\u201919)","author":"Chan Caroline","unstructured":"Caroline Chan , Shiry Ginosar , Tinghui Zhou , and Alexei A. Efros . 2019. Everybody dance now . In Proceedings of the IEEE International Conference on Computer Vision (ICCV\u201919) . Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A. Efros. 2019. Everybody dance now. In Proceedings of the IEEE International Conference on Computer Vision (ICCV\u201919)."},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00012"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00916"},{"key":"e_1_2_2_7_1","volume-title":"Proceedings of the 9th IEEE International Conference on Computer Vision (ICCV\u201903)","author":"Efros Alexei A.","year":"2003","unstructured":"Alexei A. Efros , Alexander C. Berg , Greg Mori , and Jitendra Malik . 2003 . Recognizing action at a distance . In Proceedings of the 9th IEEE International Conference on Computer Vision (ICCV\u201903) . IEEE Computer Society, 726. Alexei A. Efros, Alexander C. Berg, Greg Mori, and Jitendra Malik. 2003. Recognizing action at a distance. In Proceedings of the 9th IEEE International Conference on Computer Vision (ICCV\u201903). IEEE Computer Society, 726."},{"key":"e_1_2_2_8_1","volume-title":"Wolfgang Effelsberg et\u00a0al","author":"Farin Dirk","year":"2003","unstructured":"Dirk Farin , Susanne Krabbe , Wolfgang Effelsberg et\u00a0al . 2003 . Robust camera calibration for sport videos using court models. In Storage and Retrieval Methods and Applications for Multimedia 2004, Vol. 5307 . International Society for Optics and Photonics , 80\u201391. Dirk Farin, Susanne Krabbe, Wolfgang Effelsberg et\u00a0al. 2003. Robust camera calibration for sport videos using court models. In Storage and Retrieval Methods and Applications for Multimedia 2004, Vol. 5307. International Society for Optics and Photonics, 80\u201391."},{"key":"e_1_2_2_9_1","first-page":"1785","article-title":"Memory augmented deep generative models for forecasting the next shot location in tennis","volume":"32","author":"Fernando Tharindu","year":"2019","unstructured":"Tharindu Fernando , Simon Denman , Sridha Sridharan , and Clinton Fookes . 2019 . Memory augmented deep generative models for forecasting the next shot location in tennis . IEEE Trans. Knowl. Data Eng. 32 , 9 (2019), 1785 \u2013 1797 . DOI:https:\/\/doi.org\/10.1109\/TKDE.2019.2911507 10.1109\/TKDE.2019.2911507 Tharindu Fernando, Simon Denman, Sridha Sridharan, and Clinton Fookes. 2019. Memory augmented deep generative models for forecasting the next shot location in tennis. IEEE Trans. Knowl. Data Eng. 32, 9 (2019), 1785\u20131797. DOI:https:\/\/doi.org\/10.1109\/TKDE.2019.2911507","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"e_1_2_2_10_1","volume-title":"Proceedings of the Symposium on Interactive 3D Graphics and Games (I3D\u201909)","author":"Flagg Matthew","unstructured":"Matthew Flagg , Atsushi Nakazawa , Qiushuang Zhang , Sing Bing Kang , Young Kee Ryu , Irfan Essa , and James M. Rehg . 2009. Human video textures . In Proceedings of the Symposium on Interactive 3D Graphics and Games (I3D\u201909) . Association for Computing Machinery, New York, NY, 199\u2013206. Matthew Flagg, Atsushi Nakazawa, Qiushuang Zhang, Sing Bing Kang, Young Kee Ryu, Irfan Essa, and James M. Rehg. 2009. Human video textures. In Proceedings of the Symposium on Interactive 3D Graphics and Games (I3D\u201909). Association for Computing Machinery, New York, NY, 199\u2013206."},{"key":"e_1_2_2_11_1","unstructured":"Oran Gafni Lior Wolf and Yaniv Taigman. 2019. Vid2Game: Controllable characters extracted from real-world videos. Retrieved from https:\/\/arXiv:1904.08379.  Oran Gafni Lior Wolf and Yaniv Taigman. 2019. Vid2Game: Controllable characters extracted from real-world videos. Retrieved from https:\/\/arXiv:1904.08379."},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00044"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00762"},{"key":"e_1_2_2_14_1","volume-title":"Mask R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV\u201917)","author":"He Kaiming","year":"2017","unstructured":"Kaiming He , Georgia Gkioxari , Piotr Doll\u00e1r , and Ross Girshick . 2017 . Mask R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV\u201917) . IEEE, 2980\u20132988. Kaiming He, Georgia Gkioxari, Piotr Doll\u00e1r, and Ross Girshick. 2017. Mask R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV\u201917). IEEE, 2980\u20132988."},{"key":"e_1_2_2_15_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201917)","author":"Isola P.","year":"2017","unstructured":"P. Isola , J. Zhu , T. Zhou , and A. A. Efros . 2017. Image-to-image translation with conditional adversarial networks . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201917) . 5967\u20135976. DOI:https:\/\/doi.org\/10.1109\/CVPR. 2017 .632 10.1109\/CVPR.2017.632 P. Isola, J. Zhu, T. Zhou, and A. A. Efros. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201917). 5967\u20135976. DOI:https:\/\/doi.org\/10.1109\/CVPR.2017.632"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201283"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/566570.566605"},{"key":"e_1_2_2_18_1","first-page":"6","article-title":"Tennis real play","volume":"14","author":"Lai J.","year":"2012","unstructured":"J. Lai , C. Chen , P. Wu , C. Kao , M. Hu , and S. Chien . 2012 . Tennis real play . IEEE Trans. Multimedia 14 , 6 (Dec. 2012), 1602\u20131617. DOI:https:\/\/doi.org\/10.1109\/TMM.2012.2197190 10.1109\/TMM.2012.2197190 J. Lai, C. Chen, P. Wu, C. Kao, M. Hu, and S. Chien. 2012. Tennis real play. IEEE Trans. Multimedia 14, 6 (Dec. 2012), 1602\u20131617. DOI:https:\/\/doi.org\/10.1109\/TMM.2012.2197190","journal-title":"IEEE Trans. Multimedia"},{"key":"e_1_2_2_19_1","volume-title":"Proceedings of the MIT Sloan Sports Analytics Conference (MITSSAC\u201917)","author":"Le H.","unstructured":"H. Le , P. Carr , Y. Yue , and P. Lucey . 2017. Data-driven ghosting using deep imitation learning . In Proceedings of the MIT Sloan Sports Analytics Conference (MITSSAC\u201917) . Boston, MA. H. Le, P. Carr, Y. Yue, and P. Lucey. 2017. Data-driven ghosting using deep imitation learning. In Proceedings of the MIT Sloan Sports Analytics Conference (MITSSAC\u201917). Boston, MA."},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/566654.566607"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3333002"},{"key":"e_1_2_2_22_1","volume-title":"Proceedings of the International Conference on Visual Information Engineering (VIE\u201903)","author":"Owens N.","unstructured":"N. Owens , C. Harris , and C. Stennett . 2003. Hawk-eye tennis system . In Proceedings of the International Conference on Visual Information Engineering (VIE\u201903) . 182\u2013185. N. Owens, C. Harris, and C. Stennett. 2003. Hawk-eye tennis system. In Proceedings of the International Conference on Visual Information Engineering (VIE\u201903). 182\u2013185."},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2786984.2786995"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098051"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1141911.1141920"},{"key":"e_1_2_2_26_1","volume-title":"Proceedings of the ACM SIGGRAPH\/Eurographics Symposium on Computer Animation (SCA\u201902)","author":"Sch\u00f6dl Arno","unstructured":"Arno Sch\u00f6dl and Irfan A. Essa . 2002. Controlled animation of video sprites . In Proceedings of the ACM SIGGRAPH\/Eurographics Symposium on Computer Animation (SCA\u201902) . Association for Computing Machinery, 121\u2013127. Arno Sch\u00f6dl and Irfan A. Essa. 2002. Controlled animation of video sprites. In Proceedings of the ACM SIGGRAPH\/Eurographics Symposium on Computer Animation (SCA\u201902). Association for Computing Machinery, 121\u2013127."},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/344779.345012"},{"key":"e_1_2_2_28_1","unstructured":"Second Spectrum Inc.2020. Second Spectrum Corporate Website. Retrieved from https:\/\/www.secondspectrum.com.  Second Spectrum Inc.2020. Second Spectrum Corporate Website. Retrieved from https:\/\/www.secondspectrum.com."},{"key":"e_1_2_2_29_1","volume-title":"Density Estimation for Statistics and Data Analysis","author":"Silverman Bernard W","unstructured":"Bernard W Silverman . 2018. Density Estimation for Statistics and Data Analysis . Routledge . Bernard W Silverman. 2018. Density Estimation for Statistics and Data Analysis. Routledge."},{"key":"e_1_2_2_30_1","volume-title":"Absolute Tennis: The Best and Next Way to Play the Game","author":"Smith Marty","year":"2017","unstructured":"Marty Smith . 2017 . Absolute Tennis: The Best and Next Way to Play the Game . New Chapter Press . Marty Smith. 2017. Absolute Tennis: The Best and Next Way to Play the Game. New Chapter Press."},{"key":"e_1_2_2_31_1","unstructured":"StatsPerform Inc.2020. SportVU 2.0: Real-Time Optical Tracking. Retrieved from https:\/\/www.statsperform.com\/team-performance\/football\/optical-tracking.  StatsPerform Inc.2020. SportVU 2.0: Real-Time Optical Tracking. Retrieved from https:\/\/www.statsperform.com\/team-performance\/football\/optical-tracking."},{"key":"e_1_2_2_32_1","volume-title":"Advances in Neural Information Processing Systems","author":"Wang Ting-Chun","unstructured":"Ting-Chun Wang , Ming-Yu Liu , Jun-Yan Zhu , Guilin Liu , Andrew Tao , Jan Kautz , and Bryan Catanzaro . 2018a. Video-to-video synthesis . In Advances in Neural Information Processing Systems . MIT Press , 1144\u20131156. Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. 2018a. Video-to-video synthesis. In Advances in Neural Information Processing Systems. MIT Press, 1144\u20131156."},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00917"},{"key":"e_1_2_2_34_1","volume-title":"Proceedings of the MIT Sloan Sports Analytics Conference (MITSSAC\u201916)","author":"Wei X.","unstructured":"X. Wei , P. Lucey , S. Morgan , M. Reid , and S. Sridharan . 2016. The thin edge of the wedge: Accurately predicting shot outcomes in tennis using style and context priors . In Proceedings of the MIT Sloan Sports Analytics Conference (MITSSAC\u201916) . Boston, MA. X. Wei, P. Lucey, S. Morgan, M. Reid, and S. Sridharan. 2016. The thin edge of the wedge: Accurately predicting shot outcomes in tennis using style and context priors. In Proceedings of the MIT Sloan Sports Analytics Conference (MITSSAC\u201916). Boston, MA."},{"key":"e_1_2_2_35_1","first-page":"11","article-title":"Forecasting the next shot location in tennis using fine-grained spatiotemporal tracking data","volume":"28","author":"Wei X.","year":"2016","unstructured":"X. Wei , P. Lucey , S. Morgan , and S. Sridharan . 2016 . Forecasting the next shot location in tennis using fine-grained spatiotemporal tracking data . IEEE Trans. Knowl. Data Eng. 28 , 11 (Nov. 2016), 2988\u20132997. X. Wei, P. Lucey, S. Morgan, and S. Sridharan. 2016. Forecasting the next shot location in tennis using fine-grained spatiotemporal tracking data. IEEE Trans. Knowl. Data Eng. 28, 11 (Nov. 2016), 2988\u20132997.","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"e_1_2_2_36_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201917)","author":"Xu N.","year":"2017","unstructured":"N. Xu , B. Price , S. Cohen , and T. Huang . 2017. Deep image matting . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201917) . 311\u2013320. DOI:https:\/\/doi.org\/10.1109\/CVPR. 2017 .41 10.1109\/CVPR.2017.41 N. Xu, B. Price, S. Cohen, and T. Huang. 2017. Deep image matting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201917). 311\u2013320. DOI:https:\/\/doi.org\/10.1109\/CVPR.2017.41"},{"key":"e_1_2_2_37_1","volume-title":"Proceedings of the IEEE International Conference on Computer Vision (ICCV\u201917)","author":"Zhu J.","unstructured":"J. Zhu , T. Park , P. Isola , and A. A. Efros . 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks . In Proceedings of the IEEE International Conference on Computer Vision (ICCV\u201917) . 2242\u20132251. J. Zhu, T. Park, P. Isola, and A. A. Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV\u201917). 2242\u20132251."}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3448978","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3448978","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:47:55Z","timestamp":1750193275000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3448978"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,5]]},"references-count":37,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,6,30]]}},"alternative-id":["10.1145\/3448978"],"URL":"https:\/\/doi.org\/10.1145\/3448978","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,5]]},"assertion":[{"value":"2020-08-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-02-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-05-05","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}