{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T18:29:09Z","timestamp":1777487349395,"version":"3.51.4"},"reference-count":45,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2022,7,1]],"date-time":"2022-07-01T00:00:00Z","timestamp":1656633600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["866435"],"award-info":[{"award-number":["866435"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2022,7]]},"abstract":"<jats:p>Time-lapse image sequences offer visually compelling insights into dynamic processes that are too slow to observe in real time. However, playing a long time-lapse sequence back as a video often results in distracting flicker due to random effects, such as weather, as well as cyclic effects, such as the day-night cycle. We introduce the problem of disentangling time-lapse sequences in a way that allows separate, after-the-fact control of overall trends, cyclic effects, and random effects in the images, and describe a technique based on data-driven generative models that achieves this goal. This enables us to \"re-render\" the sequences in ways that would not be possible with the input images alone. For example, we can stabilize a long sequence to focus on plant growth over many months, under selectable, consistent weather.<\/jats:p>\n          <jats:p>Our approach is based on Generative Adversarial Networks (GAN) that are conditioned with the time coordinate of the time-lapse sequence. Our architecture and training procedure are designed so that the networks learn to model random variations, such as weather, using the GAN's latent space, and to disentangle overall trends and cyclic variations by feeding the conditioning time label to the model using Fourier features with specific frequencies.<\/jats:p>\n          <jats:p>We show that our models are robust to defects in the training data, enabling us to amend some of the practical difficulties in capturing long time-lapse sequences, such as temporary occlusions, uneven frame spacing, and missing frames.<\/jats:p>","DOI":"10.1145\/3528223.3530170","type":"journal-article","created":{"date-parts":[[2022,7,22]],"date-time":"2022-07-22T21:06:27Z","timestamp":1658523987000},"page":"1-13","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Disentangling random and cyclic effects in time-lapse sequences"],"prefix":"10.1145","volume":"41","author":[{"given":"Erik","family":"H\u00e4rk\u00f6nen","sequence":"first","affiliation":[{"name":"Aalto University, Finland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Miika","family":"Aittala","sequence":"additional","affiliation":[{"name":"NVIDIA, Finland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tuomas","family":"Kynk\u00e4\u00e4nniemi","sequence":"additional","affiliation":[{"name":"Aalto University, Finland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Samuli","family":"Laine","sequence":"additional","affiliation":[{"name":"NVIDIA, Finland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Timo","family":"Aila","sequence":"additional","affiliation":[{"name":"NVIDIA, Finland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jaakko","family":"Lehtinen","sequence":"additional","affiliation":[{"name":"Aalto University &amp; NVIDIA, Finland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,7,22]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00751"},{"key":"e_1_2_2_2_1","volume-title":"Proc. ICLR.","author":"Brock A.","year":"2019","unstructured":"Brock, A., Donahue, J., and Simonyan, K. (2019). Large scale gan training for high fidelity natural image synthesis. In Proc. ICLR."},{"key":"e_1_2_2_3_1","volume-title":"Proc. CVPR.","author":"Choi Y.","unstructured":"Choi, Y., Uh, Y., Yoo, J., and Ha, J.-W. (2020). Stargan v2: Diverse image synthesis for multiple domains. In Proc. CVPR."},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00386"},{"key":"e_1_2_2_5_1","volume-title":"Efficient video generation on complex datasets. CoRR, abs\/1907.06571","author":"Clark A.","year":"2019","unstructured":"Clark, A., Donahue, J., and Simonyan, K. (2019). Efficient video generation on complex datasets. CoRR, abs\/1907.06571."},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00581"},{"key":"e_1_2_2_7_1","volume-title":"Proc. International Conference on Computational Creativity.","author":"Colton S.","year":"2021","unstructured":"Colton, S. and Ferrer, B. P. (2021). Ganlapse generative photography. In Proc. International Conference on Computational Creativity."},{"key":"e_1_2_2_8_1","volume-title":"Proc. ICLR.","author":"Dinh L.","year":"2017","unstructured":"Dinh, L., Sohl-Dickstein, J., and Bengio, S. (2017). Density estimation using Real NVP. In Proc. ICLR."},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356523"},{"key":"e_1_2_2_10_1","volume-title":"Proc. NIPS.","author":"Goodfellow I.","year":"2014","unstructured":"Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative Adversarial Networks. In Proc. NIPS."},{"key":"e_1_2_2_11_1","volume-title":"Proc. NeurIPS.","author":"H\u00e4rk\u00f6nen E.","year":"2020","unstructured":"H\u00e4rk\u00f6nen, E., Hertzmann, A., Lehtinen, J., and Paris, S. (2020). GANSpace: Discovering interpretable GAN controls. In Proc. NeurIPS."},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.5555\/3295222.3295408"},{"key":"e_1_2_2_13_1","volume-title":"Proc. NeurIPS.","author":"Ho J.","year":"2020","unstructured":"Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. In Proc. NeurIPS."},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-41404-7_44"},{"key":"e_1_2_2_15_1","unstructured":"Huang X. Mallya A. Wang T.-C. and Liu M.-Y. (2021). Multimodal conditional image synthesis with product-of-experts GANs. CoRR abs\/2112.05130."},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1653771.1653789"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2007.383258"},{"key":"e_1_2_2_18_1","volume-title":"Stylefusion: A generative model for disentangling spatial segments. CoRR, abs\/2107.07437","author":"Kafri O.","year":"2021","unstructured":"Kafri, O., Patashnik, O., Alaluf, Y., and Cohen-Or, D. (2021). Stylefusion: A generative model for disentangling spatial segments. CoRR, abs\/2107.07437."},{"key":"e_1_2_2_19_1","volume-title":"Proc. TOG.","author":"Karacan L.","year":"2019","unstructured":"Karacan, L., Akata, Z., Erdem, A., and Erdem, E. (2019). Manipulating attributes of natural scenes via hallucination. In Proc. TOG."},{"key":"e_1_2_2_20_1","volume-title":"Proc. NeurIPS.","author":"Karras T.","unstructured":"Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., and Aila, T. (2020a). Training generative adversarial networks with limited data. In Proc. NeurIPS."},{"key":"e_1_2_2_21_1","volume-title":"Proc. NeurIPS.","author":"Karras T.","year":"2021","unstructured":"Karras, T., Aittala, M., Laine, S., H\u00e4rk\u00f6nen, E., Hellsten, J., Lehtinen, J., and Aila, T. (2021). Alias-free generative adversarial networks. In Proc. NeurIPS."},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00453"},{"key":"e_1_2_2_23_1","volume-title":"Proc. CVPR.","author":"Karras T.","unstructured":"Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. (2020b). Analyzing and improving the image quality of StyleGAN. In Proc. CVPR."},{"key":"e_1_2_2_24_1","volume-title":"Proc. ICLR.","author":"Kim J.","year":"2020","unstructured":"Kim, J., Kim, M., Kang, H., and Lee, K. (2020). U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation. In Proc. ICLR."},{"key":"e_1_2_2_25_1","volume-title":"Proc. NeurIPS.","author":"Kingma D. P.","year":"2018","unstructured":"Kingma, D. P. and Dhariwal, P. (2018). Glow: Generative flow with invertible 1x1 convolutions. In Proc. NeurIPS."},{"key":"e_1_2_2_26_1","volume-title":"Proc. ICLR.","author":"Kingma D. P.","year":"2014","unstructured":"Kingma, D. P. and Welling, M. (2014). Auto-encoding variational bayes. In Proc. ICLR."},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58592-1_16"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766903"},{"key":"e_1_2_2_29_1","volume-title":"Conditional generative adversarial nets. CoRR, abs\/1411.1784","author":"Mirza M.","year":"2014","unstructured":"Mirza, M. and Osindero, S. (2014). Conditional generative adversarial nets. CoRR, abs\/1411.1784."},{"key":"e_1_2_2_30_1","volume-title":"Proc. ICLR.","author":"Miyato T.","year":"2018","unstructured":"Miyato, T. and Koyama, M. (2018). cgans with projection discriminator. In Proc. ICLR."},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00150"},{"key":"e_1_2_2_32_1","volume-title":"Proc. CVPR.","author":"Park T.","unstructured":"Park, T., Liu, M.-Y., Wang, T., and Zhu, J.-Y. (2019). Semantic image synthesis with spatially-adaptive normalization. In Proc. CVPR."},{"key":"e_1_2_2_33_1","volume-title":"Proc. NeurIPS.","author":"Park T.","year":"2020","unstructured":"Park, T., Zhu, J.-Y., Wang, O., Lu, J., Shechtman, E., Efros, A. A., and Zhang, R. (2020). Swapping autoencoder for deep image manipulation. In Proc. NeurIPS."},{"key":"e_1_2_2_34_1","volume-title":"Proc. ICML.","author":"Sohl-Dickstein J.","year":"2015","unstructured":"Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., and Ganguli, S. (2015). Deep unsupervised learning using nonequilibrium thermodynamics. In Proc. ICML."},{"key":"e_1_2_2_35_1","volume-title":"Proc. NeurIPS.","author":"Song Y.","year":"2019","unstructured":"Song, Y. and Ermon, S. (2019). Generative modeling by estimating gradients of the data distribution. In Proc. NeurIPS."},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00881"},{"key":"e_1_2_2_37_1","volume-title":"Proc. NeurIPS.","author":"Tancik M.","year":"2020","unstructured":"Tancik, M., Srinivasan, P. P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J. T., and Ng, R. (2020). Fourier features let networks learn high frequency functions in low dimensional domains. In Proc. NeurIPS."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00165"},{"key":"e_1_2_2_39_1","volume-title":"Proc. ICML.","author":"van den Oord A.","unstructured":"van den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. (2016a). Pixel recurrent neural networks. In Proc. ICML."},{"key":"e_1_2_2_40_1","volume-title":"Proc. NIPS.","author":"van den Oord A.","unstructured":"van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., and Kavukcuoglu, K. (2016b). Conditional image generation with PixelCNN decoders. In Proc. NIPS."},{"key":"e_1_2_2_41_1","volume-title":"Proc. NeurIPS.","author":"Vaswani A.","year":"2017","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. In Proc. NeurIPS."},{"key":"e_1_2_2_42_1","volume-title":"Proc. NeurIPS.","author":"Wang T.-C.","year":"2019","unstructured":"Wang, T.-C., Liu, M.-Y., Tao, A., Liu, G., Kautz, J., and Catanzaro, B. (2019). Few-shot video-to-video synthesis. In Proc. NeurIPS."},{"key":"e_1_2_2_43_1","volume-title":"Proc. NeurIPS.","author":"Wang T.-C.","year":"2018","unstructured":"Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Liu, G., Tao, A., Kautz, J., and Catanzaro, B. (2018). Video-to-video synthesis. In Proc. NeurIPS."},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00251"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.244"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3528223.3530170","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3528223.3530170","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:02:49Z","timestamp":1750186969000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3528223.3530170"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7]]},"references-count":45,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,7]]}},"alternative-id":["10.1145\/3528223.3530170"],"URL":"https:\/\/doi.org\/10.1145\/3528223.3530170","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7]]},"assertion":[{"value":"2022-07-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}