{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,27]],"date-time":"2026-09-27T04:55:06Z","timestamp":1790484906335,"version":"4.1.0"},"reference-count":46,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,5,23]],"date-time":"2022-05-23T00:00:00Z","timestamp":1653264000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,5,23]],"date-time":"2022-05-23T00:00:00Z","timestamp":1653264000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    In multiagent worlds, several decision-making individuals interact while adhering to the dynamics constraints imposed by the environment. These interactions, combined with the potential stochasticity of the agents\u2019 dynamic behaviors, make such systems complex and interesting to study from a decision-making perspective. Significant research has been conducted on learning models for forward-direction estimation of agent behaviors, for example, pedestrian predictions used for collision-avoidance in self-driving cars. In many settings, only sporadic observations of agents may be available in a given trajectory sequence. In football, subsets of players may come in and out of view of broadcast video footage, while unobserved players continue to interact off-screen. In this paper, we study the problem of multiagent time-series imputation in the context of human football play, where available past and future observations of subsets of agents are used to estimate missing observations for other agents. Our approach, called the\n                    <jats:italic>Graph Imputer<\/jats:italic>\n                    , uses past and future information in combination with graph networks and variational autoencoders to enable learning of a distribution of imputed trajectories. We demonstrate our approach on multiagent settings involving players that are partially-observable, using the Graph Imputer to predict the behaviors of off-screen players. To quantitatively evaluate the approach, we conduct experiments on football matches with ground truth trajectory data, using a camera module to simulate the off-screen player state estimation setting. We subsequently use our approach for downstream football analytics under partial observability using the well-established framework of pitch control, which traditionally relies on fully observed data. We illustrate that our method outperforms several state-of-the-art approaches, including those hand-crafted for football, across all considered metrics.\n                  <\/jats:p>","DOI":"10.1038\/s41598-022-12547-0","type":"journal-article","created":{"date-parts":[[2022,5,23]],"date-time":"2022-05-23T08:03:57Z","timestamp":1653293037000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":27,"title":["Multiagent off-screen behavior prediction in football"],"prefix":"10.1038","volume":"12","author":[{"given":"Shayegan","family":"Omidshafiei","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Hennes","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marta","family":"Garnelo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhe","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adria","family":"Recasens","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eugene","family":"Tarassov","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yi","family":"Yang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Romuald","family":"Elie","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jerome T.","family":"Connor","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paul","family":"Muller","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Natalie","family":"Mackraz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kris","family":"Cao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pol","family":"Moreno","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pablo","family":"Sprechmann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Demis","family":"Hassabis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ian","family":"Graham","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"William","family":"Spearman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicolas","family":"Heess","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Karl","family":"Tuyls","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,5,23]]},"reference":[{"key":"12547_CR1","unstructured":"Foerster, J.\u00a0N., Farquhar, G., Afouras, T., Nardelli, N., & Whiteson, S. Counterfactual multi-agent policy gradients. In Proc. Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th Innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2\u20137, 2018 (eds. McIlraith, S.\u00a0A. & Weinberger, K.\u00a0Q.) 2974\u20132982 (AAAI Press, 2018)."},{"key":"12547_CR2","unstructured":"Brown, N., Lerer, A., Gross, S. & Sandholm, T. Deep counterfactual regret minimization. In Proc. 36th International Conference on Machine Learning, ICML 2019, 9\u201315 June 2019, Long Beach, California, USA, Volume 97 of Proc. Machine Learning Research (eds. Chaudhuri, K. & Salakhutdinov, R.) 793\u2013802 (PMLR, 2019)."},{"key":"12547_CR3","unstructured":"Sun, C., Karlsson, P., Wu, J., Tenenbaum, J.\u00a0B. & Murphy, K. Predicting the present and future states of multi-agent systems from partially-observed visual data. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=r1xdH3CcKX (2019)."},{"key":"12547_CR4","volume-title":"Modelling Financial Time Series. Number 6578 in World Scientific Books","author":"SJ Taylor","year":"2007","unstructured":"Taylor, S. J. Modelling Financial Time Series. Number 6578 in World Scientific Books (World Scientific Publishing, 2007)."},{"key":"12547_CR5","doi-asserted-by":"crossref","unstructured":"Sezer, O.\u00a0B., Gudelek, M.\u00a0U. & \u00d6zbayoglu, A.\u00a0M. Financial time series forecasting with deep learning: A systematic literature review: 2005\u20132019. CoRR. http:\/\/arxiv.org\/abs\/1911.13288 (2019).","DOI":"10.1016\/j.asoc.2020.106181"},{"issue":"4","key":"12547_CR6","doi-asserted-by":"publisher","first-page":"309","DOI":"10.1016\/S0305-0483(01)00026-3","volume":"29","author":"FE Tay","year":"2001","unstructured":"Tay, F. E. & Cao, L. Application of support vector machines in financial time series forecasting. Omega 29(4), 309\u2013317 (2001).","journal-title":"Omega"},{"key":"12547_CR7","doi-asserted-by":"crossref","unstructured":"Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L. & Savarese, S. Social LSTM: Human trajectory prediction in crowded spaces. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016).","DOI":"10.1109\/CVPR.2016.110"},{"key":"12547_CR8","doi-asserted-by":"crossref","unstructured":"Sakata, N., Kinoshita, Y. & Kato, Y. Predicting a pedestrian trajectory using seq2seq for mobile robot navigation. In IECON 2018\u201444th Annual Conference of the IEEE Industrial Electronics Society 4300\u20134305 (2018).","DOI":"10.1109\/IECON.2018.8591519"},{"issue":"8","key":"12547_CR9","doi-asserted-by":"publisher","first-page":"895","DOI":"10.1177\/0278364920917446","volume":"39","author":"A Rudenko","year":"2020","unstructured":"Rudenko, A. et al. Human motion trajectory prediction: A survey. Int. J. Robot. Res. 39(8), 895\u2013935 (2020).","journal-title":"Int. J. Robot. Res."},{"key":"12547_CR10","unstructured":"Le, H.\u00a0M., Yue, Y., Carr, P. & Lucey, P. Coordinated multi-agent imitation learning. In International Conference on Machine Learning 1995\u20132003 (PMLR, 2017)."},{"key":"12547_CR11","unstructured":"Le, H.\u00a0M., Carr, P., Yue, Y. & Lucey, P. Data-driven ghosting using deep imitation learning. MIT Sloan Sports Analytics Conference (SSAC) (2017)."},{"key":"12547_CR12","doi-asserted-by":"crossref","unstructured":"Yeh, R.\u00a0A., Schwing, A.\u00a0G., Huang, J. & Murphy, K. Diverse generation for multi-agent sports games. In Proc. IEEE\/CVF Conference on Computer Vision and Pattern Recognition 4610\u20134619 (2019).","DOI":"10.1109\/CVPR.2019.00474"},{"key":"12547_CR13","doi-asserted-by":"crossref","unstructured":"Hauri, S., Djuric, N., Radosavljevic, V. & Vucetic, S. Multi-modal trajectory prediction of NBA players. arXiv (2020).","DOI":"10.1109\/WACV48630.2021.00168"},{"key":"12547_CR14","unstructured":"Spearman, W., Basye, A., Dick, G., Hotovy, R. & Pop, P. Physics-based modeling of pass probabilities in soccer. In Proc. 11th MIT Sloan Sports Analytics Conference (2017)."},{"key":"12547_CR15","volume-title":"Advances in Neural Information Processing Systems","author":"Y Hoshen","year":"2017","unstructured":"Hoshen, Y. Vain: Attentional multi-agent predictive modeling. In Advances in Neural Information Processing Systems Vol. 30 (eds Guyon, I. et al.) (Curran Associates, Inc., 2017)."},{"key":"12547_CR16","unstructured":"Alcorn, M.\u00a0A. & Nguyen, A. baller2vec: A multi-entity transformer for multi-agent spatiotemporal modeling. Preprint at http:\/\/arXiv.org\/2102.03291 (2021)."},{"key":"12547_CR17","doi-asserted-by":"crossref","unstructured":"Su, S., Hong, J.\u00a0P., Shi, J. & Park, H.\u00a0S. Predicting behaviors of basketball players from first person videos. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21\u201326, 2017 1206\u20131215 (IEEE Computer Society, 2017).","DOI":"10.1109\/CVPR.2017.133"},{"key":"12547_CR18","doi-asserted-by":"crossref","unstructured":"Suda, S., Makino, Y. & Shinoda, H. Prediction of volleyball trajectory using skeletal motions of setter player. In Proc. 10th Augmented Human International Conference 2019, New York, NY, USA (Association for Computing Machinery, 2019).","DOI":"10.1145\/3311823.3311844"},{"key":"12547_CR19","unstructured":"Li, J., Yang, F., Tomizuka, M. & Choi, C. Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning. In Proc. Neural Information Processing Systems (NeurIPS) (2020)."},{"issue":"1","key":"12547_CR20","doi-asserted-by":"publisher","first-page":"207","DOI":"10.32614\/RJ-2017-009","volume":"9","author":"S Moritz","year":"2017","unstructured":"Moritz, S. & Bartz-Beielstein, T. Imputets: Time series missing value imputation in r. R J. 9(1), 207 (2017).","journal-title":"R J."},{"key":"12547_CR21","unstructured":"Silva, I., Moody, G., Scott, D.\u00a0J., Celi, L.\u00a0A. & Mark, R.\u00a0G. Predicting in-hospital mortality of icu patients: The physionet\/computing in cardiology challenge 2012. In 2012 Computing in Cardiology 245\u2013248 (IEEE, 2012)."},{"key":"12547_CR22","unstructured":"Fedus, W., Goodfellow, I. & Dai, A.\u00a0M. MaskGAN: Better text generation via filling in the _. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=ByOExmWAb (2018)."},{"key":"12547_CR23","volume-title":"Advances in Neural Information Processing Systems","author":"J Chung","year":"2015","unstructured":"Chung, J. et al. A recurrent latent variable model for sequential data. In Advances in Neural Information Processing Systems Vol. 28 (eds Cortes, C. et al.) (Curran Associates, Inc., 2015)."},{"key":"12547_CR24","unstructured":"Battaglia, P. et al. Relational inductive biases, deep learning, and graph networks. arXiv. http:\/\/arxiv.org\/abs\/1806.01261 (2018)."},{"key":"12547_CR25","doi-asserted-by":"crossref","unstructured":"Bialkowski, A., Lucey, P., Carr, P., Yue, Y., Sridharan, S. & Matthews, I. Large-scale analysis of soccer matches using spatiotemporal tracking data. In 2014 IEEE International Conference on Data Mining 725\u2013730 (IEEE, 2014).","DOI":"10.1109\/ICDM.2014.133"},{"key":"12547_CR26","unstructured":"Spearman, W. Beyond expected goals. In Proc. 12th MIT Sloan Sports Analytics Conference 1\u201317 (2018)."},{"key":"12547_CR27","doi-asserted-by":"crossref","unstructured":"Zhu, H., Claramunt, F.\u00a0M., Brito, B. & Alonso-Mora, J. Learning interaction-aware trajectory predictions for decentralized multi-robot motion planning in dynamic environments. arXiv (2021).","DOI":"10.1109\/LRA.2021.3061073"},{"key":"12547_CR28","doi-asserted-by":"publisher","first-page":"7833","DOI":"10.1109\/JSEN.2019.2923982","volume":"21","author":"Z Han","year":"2019","unstructured":"Han, Z., Zhao, J., Leung, H., Ma, K. F. & Wang, W. A review of deep learning models for time series prediction. IEEE Sens. J. 21, 7833 (2019).","journal-title":"IEEE Sens. J."},{"key":"12547_CR29","doi-asserted-by":"crossref","unstructured":"Sun, H., Zhao, Z. & He, Z. Reciprocal learning networks for human trajectory prediction. In Proc. IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020).","DOI":"10.1109\/CVPR42600.2020.00744"},{"key":"12547_CR30","unstructured":"Alcorn, M.\u00a0A. & Nguyen, A. baller2vec++: A look-ahead multi-entity transformer for modeling coordinated agents. Preprint at http:\/\/arXiv.org\/2104.11980 (2021)."},{"key":"12547_CR31","unstructured":"Yang, F., Chen, L., Zhou, F., Gao, Y. & Cao, W. Relational state-space model for stochastic multi-object systems. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=B1lGU64tDr (2020)."},{"key":"12547_CR32","unstructured":"Shang, W., Espeholt, L., Raichuk, A. & Salimans, T. Agent-centric representations for multi-agent reinforcement learning. Preprint at http:\/\/arXiv.org\/2104.09402 (2021)."},{"key":"12547_CR33","doi-asserted-by":"crossref","unstructured":"Mohamed, A., Qian, K., Elhoseiny, M. & Claudel, C. Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction. In Proc. IEEE\/CVF Conference on Computer Vision and Pattern Recognition 14424\u201314432 (2020).","DOI":"10.1109\/CVPR42600.2020.01443"},{"key":"12547_CR34","doi-asserted-by":"crossref","unstructured":"Salzmann, T., Ivanovic, B., Chakravarty, P. & Pavone, M. Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data. In Computer Vision\u2013ECCV 2020: 16th European Conference, Glasgow, UK, August 23\u201328, 2020, Proc., Part XVIII 16 683\u2013700 (Springer, 2020).","DOI":"10.1007\/978-3-030-58523-5_40"},{"key":"12547_CR35","doi-asserted-by":"crossref","unstructured":"Mangalam, K. et al. It is not the journey but the destination: Endpoint conditioned trajectory prediction. In European Conference on Computer Vision 759\u2013776 (Springer, 2020).","DOI":"10.1007\/978-3-030-58536-5_45"},{"key":"12547_CR36","unstructured":"Kipf, T., Fetaya, E., Wang, K.-C., Welling, M. & Zemel, R. Neural relational inference for interacting systems. In International Conference on Machine Learning 2688\u20132697 (PMLR, 2018)."},{"key":"12547_CR37","doi-asserted-by":"crossref","unstructured":"Graber, C. & Schwing, A.\u00a0G. Dynamic neural relational inference. In Proc. IEEE\/CVF Conference on Computer Vision and Pattern Recognition 8513\u20138522 (2020).","DOI":"10.1109\/CVPR42600.2020.00854"},{"issue":"1","key":"12547_CR38","first-page":"1","volume":"8","author":"Z Che","year":"2018","unstructured":"Che, Z., Purushotham, S., Cho, K., Sontag, D. & Liu, Y. Recurrent neural networks for multivariate time series with missing values. Sci. Rep. 8(1), 1\u201312 (2018).","journal-title":"Sci. Rep."},{"key":"12547_CR39","unstructured":"Liu, Y., Yu, R., Zheng, S., Zhan, E. & Yue, Y. Naomi: Non-autoregressive multiresolution sequence imputation. Advances in Neural Information Processing Systems 32 (NIPS 2019) (2019)."},{"key":"12547_CR40","unstructured":"Luo, Y., Cai, X., Zhang, Y., Xu, J. & Yuan, X. Multivariate time series imputation with generative adversarial networks. In Proc. 32nd International Conference on Neural Information Processing Systems 1603\u20131614 (2018)."},{"key":"12547_CR41","first-page":"3094","volume-title":"E2gan: End-to-End Generative Adversarial Network for Multivariate Time Series Imputation","author":"Y Luo","year":"2019","unstructured":"Luo, Y., Zhang, Y., Cai, X. & Yuan, X. E2gan: End-to-End Generative Adversarial Network for Multivariate Time Series Imputation 3094\u20133100 (AAAI Press, 2019)."},{"key":"12547_CR42","volume-title":"Advances in Neural Information Processing Systems","author":"W Cao","year":"2018","unstructured":"Cao, W. et al. Brits: Bidirectional recurrent imputation for time series. In Advances in Neural Information Processing Systems Vol. 31 (eds Bengio, S. et al.) (Curran Associates, Inc., 2018)."},{"key":"12547_CR43","unstructured":"Yoon, J., Zame, W.\u00a0R. & van\u00a0der Schaar, M. Estimating missing data in temporal data streams using multi-directional recurrent neural networks. CoRR. http:\/\/arxiv.org\/abs\/1711.08742 (2017)."},{"key":"12547_CR44","unstructured":"Higgins, I. et al. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations (2017)."},{"key":"12547_CR45","unstructured":"Kingma, D.\u00a0P. & Ba, J. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7\u20139, 2015, Conference Track Proceedings (2015)."},{"key":"12547_CR46","unstructured":"Nair, V. & Hinton, G.\u00a0E. Rectified linear units improve restricted Boltzmann machines. In ICML 807\u2013814 (2010)."}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-12547-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-12547-0","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-12547-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,24]],"date-time":"2022-11-24T22:48:37Z","timestamp":1669330117000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-12547-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,23]]},"references-count":46,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["12547"],"URL":"https:\/\/doi.org\/10.1038\/s41598-022-12547-0","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,5,23]]},"assertion":[{"value":"24 January 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 May 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 May 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"8638"}}