{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:19:10Z","timestamp":1750220350376,"version":"3.41.0"},"reference-count":80,"publisher":"Association for Computing Machinery (ACM)","issue":"3s","license":[{"start":{"date-parts":[[2021,10,31]],"date-time":"2021-10-31T00:00:00Z","timestamp":1635638400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2021,10,31]]},"abstract":"<jats:p>\n            Generative adversarial networks (GANs) nowadays are capable of producing images of incredible realism. Two concerns raised are whether the state-of-the-art GAN\u2019s learned distribution still suffers from mode collapse and what to do if so. Existing diversity tests of samples from GANs are usually conducted qualitatively on a small scale and\/or depend on the access to original training data as well as the trained model parameters. This article explores GAN\n            <jats:italic>intra-mode collapse<\/jats:italic>\n            and calibrates that in a novel\n            <jats:italic>black-box<\/jats:italic>\n            setting: access to neither training data nor the trained model parameters is assumed. The new setting is practically demanded yet rarely explored and significantly more challenging. As a first stab, we devise a set of statistical tools based on sampling that can visualize, quantify, and rectify\n            <jats:italic>intra-mode collapse<\/jats:italic>\n            . We demonstrate the effectiveness of our proposed diagnosis and calibration techniques, via extensive simulations and experiments, on unconditional GAN image generation (e.g., face and vehicle). Our study reveals that the\n            <jats:italic>intra-mode collapse<\/jats:italic>\n            is still a prevailing problem in state-of-the-art GANs and the mode collapse is diagnosable and calibratable in\n            <jats:italic>black-box<\/jats:italic>\n            settings. Our codes are available at\n            <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"url\" xlink:href=\"https:\/\/github.com\/VITA-Group\/BlackBoxGANCollapse\">https:\/\/github.com\/VITA-Group\/BlackBoxGANCollapse<\/jats:ext-link>\n            .\n          <\/jats:p>","DOI":"10.1145\/3472768","type":"journal-article","created":{"date-parts":[[2021,12,23]],"date-time":"2021-12-23T12:35:10Z","timestamp":1640262910000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Black-Box Diagnosis and Calibration on GAN Intra-Mode Collapse: A Pilot Study"],"prefix":"10.1145","volume":"17","author":[{"given":"Zhenyu","family":"Wu","sequence":"first","affiliation":[{"name":"Texas A&amp;M University, College Station, TX, USA"}]},{"given":"Zhaowen","family":"Wang","sequence":"additional","affiliation":[{"name":"Adobe Research, San Jose, CA, USA"}]},{"given":"Ye","family":"Yuan","sequence":"additional","affiliation":[{"name":"Texas A&amp;M University, College Station, TX, USA"}]},{"given":"Jianming","family":"Zhang","sequence":"additional","affiliation":[{"name":"Adobe Research, San Jose, CA, USA"}]},{"given":"Zhangyang","family":"Wang","sequence":"additional","affiliation":[{"name":"The University of Texas at Austin, Austin, TX, USA"}]},{"given":"Hailin","family":"Jin","sequence":"additional","affiliation":[{"name":"Adobe Research, San Jose, CA, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,12,23]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"[n.d.]. 265 Bird Species. Retrieved from https:\/\/www.kaggle.com\/gpiosenka\/100-bird-species."},{"key":"e_1_3_2_3_2","unstructured":"[n.d.]. Flowers-17 & Flowers-102. Retrieved from https:\/\/www.robots.ox.ac.uk\/ vgg\/data\/flowers\/."},{"key":"e_1_3_2_4_2","unstructured":"[n.d.]. Flowers Recognition. Retrieved from https:\/\/www.kaggle.com\/alxmamaev\/flowers-recognition."},{"key":"e_1_3_2_5_2","unstructured":"[n.d.]. Flowers Recognition. Retrieved from https:\/\/public.roboflow.com\/classification\/flowers\/1."},{"key":"e_1_3_2_6_2","unstructured":"[n.d.]. PlantCLEF 2021: Cross-domain Plant Identification. Retrieved from https:\/\/www.imageclef.org\/PlantCLEF2021."},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314243"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.5555\/3305381.3305404"},{"key":"e_1_3_2_9_2","volume-title":"Proceedings of the International Conference on Learning Representations (ICLR\u201918)","author":"Arora Sanjeev","year":"2018","unstructured":"Sanjeev Arora, Andrej Risteski, and Yi Zhang. 2018. Do GANs learn the distribution? some theory and empirics. In Proceedings of the International Conference on Learning Representations (ICLR\u201918)."},{"key":"e_1_3_2_10_2","article-title":"Do gans actually learn the distribution? an empirical study","author":"Arora Sanjeev","year":"2017","unstructured":"Sanjeev Arora and Yi Zhang. 2017. Do gans actually learn the distribution? an empirical study. arXiv preprint arXiv:1706.08224.","journal-title":"arXiv preprint arXiv:1706.08224"},{"key":"e_1_3_2_11_2","article-title":"Discriminator rejection sampling","author":"Azadi Samaneh","year":"2018","unstructured":"Samaneh Azadi, Catherine Olsson, Trevor Darrell, Ian Goodfellow, and Augustus Odena. 2018. Discriminator rejection sampling. arXiv preprint arXiv:1810.06758.","journal-title":"arXiv preprint arXiv:1810.06758"},{"key":"e_1_3_2_12_2","article-title":"A note on the inception score","author":"Barratt Shane","year":"2018","unstructured":"Shane Barratt and Rishi Sharma. 2018. A note on the inception score. arXiv preprint arXiv:1801.01973.","journal-title":"arXiv preprint arXiv:1801.01973"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00460"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2018.10.009"},{"key":"e_1_3_2_15_2","article-title":"GANsfer learning: Combining labelled and unlabelled data for GAN based data augmentation","author":"Bowles Christopher","year":"2018","unstructured":"Christopher Bowles, Roger Gunn, Alexander Hammers, and Daniel Rueckert. 2018. GANsfer learning: Combining labelled and unlabelled data for GAN based data augmentation. arXiv preprint arXiv:1811.10669.","journal-title":"arXiv preprint arXiv:1811.10669"},{"key":"e_1_3_2_16_2","article-title":"Large scale gan training for high fidelity natural image synthesis","author":"Brock Andrew","year":"2018","unstructured":"Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096.","journal-title":"arXiv preprint arXiv:1809.11096"},{"key":"e_1_3_2_17_2","article-title":"Mode regularized generative adversarial networks","author":"Che Tong","year":"2016","unstructured":"Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li. 2016. Mode regularized generative adversarial networks. arXiv preprint arXiv:1612.02136.","journal-title":"arXiv preprint arXiv:1612.02136"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00482"},{"key":"e_1_3_2_19_2","article-title":"RetinaFace: Single-stage dense face localisation in the wild","author":"Deng Jiankang","year":"2019","unstructured":"Jiankang Deng, Jia Guo, Zhou Yuxiang, Jinke Yu, Irene Kotsia, and Stefanos Zafeiriou. 2019. RetinaFace: Single-stage dense face localisation in the wild. arXiv preprint arXiv:1905.00641.","journal-title":"arXiv preprint arXiv:1905.00641"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-018-1134-y"},{"key":"e_1_3_2_21_2","article-title":"Generative multi-adversarial networks","author":"Durugkar Ishan","year":"2016","unstructured":"Ishan Durugkar, Ian Gemp, and Sridhar Mahadevan. 2016. Generative multi-adversarial networks. arXiv preprint arXiv:1611.01673.","journal-title":"arXiv preprint arXiv:1611.01673"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2010.12.066"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00888"},{"key":"e_1_3_2_24_2","volume-title":"Conference and Labs of the Evaluation Forum (CLEF\u201916)","author":"Go\u00ebau Herv\u00e9","year":"2016","unstructured":"Herv\u00e9 Go\u00ebau, Pierre Bonnet, and Alexis Joly. 2016. Plant identification in an open-world (lifeclef 2016). In Conference and Labs of the Evaluation Forum (CLEF\u201916)."},{"key":"e_1_3_2_25_2","volume-title":"Conference and Labs of the Evaluation Forum (CLEF\u201917)","author":"Go\u00ebau Herv\u00e9","year":"2017","unstructured":"Herv\u00e9 Go\u00ebau, Pierre Bonnet, and Alexis Joly. 2017. Plant identification based on noisy web data: The amazing performance of deep learning (LifeCLEF 2017). In Conference and Labs of the Evaluation Forum (CLEF\u201917)."},{"key":"e_1_3_2_26_2","volume-title":"Conference and labs of the Evaluation Forum (CLEF\u201920)","author":"Go\u00ebau Herv\u00e9","year":"2020","unstructured":"Herv\u00e9 Go\u00ebau, Pierre Bonnet, and Alexis Joly. 2020. Overview of lifeclef plant identification task 2020. In Conference and labs of the Evaluation Forum (CLEF\u201920)."},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00332"},{"key":"e_1_3_2_28_2","article-title":"NIPS 2016 tutorial: Generative adversarial networks","author":"Goodfellow Ian","year":"2016","unstructured":"Ian Goodfellow. 2016. NIPS 2016 tutorial: Generative adversarial networks. arXiv preprint arXiv:1701.00160.","journal-title":"arXiv preprint arXiv:1701.00160"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.5555\/2969033.2969125"},{"key":"e_1_3_2_30_2","volume-title":"Proceedings of the British Machine Vision Conference (BMVC\u201918)","author":"Guo Jia","year":"2018","unstructured":"Jia Guo, Jiankang Deng, Niannan Xue, and Stefanos Zafeiriou. 2018. Stacked dense U-nets with dual transformers for robust face alignment. In Proceedings of the British Machine Vision Conference (BMVC\u201918)."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.5555\/3295222.3295408"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300830"},{"key":"e_1_3_2_33_2","article-title":"An introduction to image synthesis with generative adversarial nets","author":"Huang He","year":"2018","unstructured":"He Huang, Philip S. Yu, and Changhu Wang. 2018. An introduction to image synthesis with generative adversarial nets. arXiv:1803.04469. Retrieved from https:\/\/arxiv.org\/abs\/1803.04469.","journal-title":"arXiv:1803.04469"},{"issue":"3","key":"e_1_3_2_34_2","article-title":"Transgan: Two transformers can make one strong gan","volume":"1","author":"Jiang Yifan","year":"2021","unstructured":"Yifan Jiang, Shiyu Chang, and Zhangyang Wang. 2021. Transgan: Two transformers can make one strong gan. arXiv preprint arXiv:2102.07074 1, 3 (2021).","journal-title":"arXiv preprint arXiv:2102.07074"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2021.3051462"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00782"},{"key":"e_1_3_2_37_2","article-title":"Progressive growing of gans for improved quality, stability, and variation","author":"Karras Tero","year":"2017","unstructured":"Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196.","journal-title":"arXiv preprint arXiv:1710.10196"},{"key":"e_1_3_2_38_2","first-page":"4401","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Karras Tero","year":"2018","unstructured":"Tero Karras, Samuli Laine, and Timo Aila. 2018. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 4401\u20134410."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00813"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00897"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.5555\/3326943.3327081"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.5555\/3157096.3157149"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2019.8683590"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICME.2018.8486589"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.304"},{"key":"e_1_3_2_46_2","unstructured":"Jian Mei and Hao Dong. 2020. The DongNiao international birds 10000 dataset. (unpublished)."},{"key":"e_1_3_2_47_2","article-title":"Unrolled Generative Adversarial Networks","author":"Metz Luke","year":"2016","unstructured":"Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. 2016. Unrolled Generative Adversarial Networks. arXiv preprint arXiv:1611.02163.","journal-title":"arXiv preprint arXiv:1611.02163"},{"key":"e_1_3_2_48_2","volume-title":"Proceedings of the International Conference on Learning Representations (ICLR\u201918)","author":"Miyato Takeru","year":"2018","unstructured":"Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018. Spectral normalization for generative adversarial networks. In Proceedings of the International Conference on Learning Representations (ICLR\u201918)."},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICVGIP.2008.47"},{"key":"e_1_3_2_50_2","article-title":"Unsupervised representation learning with deep convolutional generative adversarial networks","author":"Radford Alec","year":"2015","unstructured":"Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434.","journal-title":"arXiv preprint arXiv:1511.06434"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.5555\/3157096.3157346"},{"key":"e_1_3_2_52_2","volume-title":"Proceedings of the International Conference on Machine Learning","volume":"4480","author":"Santurkar Shibani","year":"2017","unstructured":"Shibani Santurkar, Ludwig Schmidt, and Aleksander M\u0105dry. 2017. A classification-based study of covariate shift in gan distributions. In Proceedings of the International Conference on Machine Learning. PMLR, 4480\u20134489."},{"key":"e_1_3_2_53_2","article-title":"Fairness gan","author":"Sattigeri Prasanna","year":"2018","unstructured":"Prasanna Sattigeri, Samuel C. Hoffman, Vijil Chenthamarakshan, and Kush R. Varshney. 2018. Fairness gan. arXiv preprint arXiv:1805.09910.","journal-title":"arXiv preprint arXiv:1805.09910"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.7717\/peerj-cs.197"},{"key":"e_1_3_2_55_2","unstructured":"Marco Seeland Michael Rzanny Nedal Alaqraa Jana W\u00e4ldchen and Patrick M\u00e4der. 2017. Jena Flowers 30 Dataset. DOI: https:\/\/doi.org\/10.7910\/DVN\/QDHYST"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.5555\/3294996.3295090"},{"key":"e_1_3_2_57_2","article-title":"Towards principled unsupervised learning","author":"Sutskever Ilya","year":"2015","unstructured":"Ilya Sutskever, Rafal Jozefowicz, Karol Gregor, Danilo Rezende, Tim Lillicrap, and Oriol Vinyals. 2015. Towards principled unsupervised learning. arXiv preprint arXiv:1511.06440.","journal-title":"arXiv preprint arXiv:1511.06440"},{"key":"e_1_3_2_58_2","volume-title":"Proceedings of the International Conference on Machine Learning (ICML\u201919)","author":"Tan Mingxing","year":"2019","unstructured":"Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the International Conference on Machine Learning (ICML\u201919). PMLR."},{"key":"e_1_3_2_59_2","first-page":"6345","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Turner Ryan","year":"2018","unstructured":"Ryan Turner, Jane Hung, Yunus Saatci, and Jason Yosinski. 2018. Metropolis-hastings generative adversarial networks. In Proceedings of the International Conference on Machine Learning. PMLR, 6345\u20136353."},{"key":"e_1_3_2_60_2","volume-title":"Proceedings of the Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops\u201919)","author":"Uplavikar Pritish M.","year":"2019","unstructured":"Pritish M. Uplavikar, Zhenyu Wu, and Zhangyang Wang. 2019. All-in-one underwater image enhancement using domain-adversarial learning. In Proceedings of the Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops\u201919)."},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298658"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58580-8_43"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58548-8_4"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01153"},{"key":"e_1_3_2_65_2","volume-title":"Caltech-UCSD Birds 200","author":"Welinder P.","year":"2010","unstructured":"P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. 2010. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001. California Institute of Technology."},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3394171.3413555"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00129"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2020.3026709"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01270-0_37"},{"key":"e_1_3_2_70_2","article-title":"Differentially private generative adversarial network","author":"Xie Liyang","year":"2018","unstructured":"Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. 2018. Differentially private generative adversarial network. arXiv preprint arXiv:1802.06739.","journal-title":"arXiv preprint arXiv:1802.06739"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2018.8622525"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2021.3055211"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58555-6_36"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00454"},{"key":"e_1_3_2_75_2","article-title":"Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop","author":"Yu Fisher","year":"2015","unstructured":"Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. 2015. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365.","journal-title":"arXiv preprint arXiv:1506.03365"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2018.2856256"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2019.8683197"},{"key":"e_1_3_2_78_2","article-title":"Attributes guided feature learning for vehicle re-identification","author":"Zheng Aihua","year":"2019","unstructured":"Aihua Zheng, Xianmin Lin, Chenglong Li, Ran He, and Jin Tang. 2019. Attributes guided feature learning for vehicle re-identification. arXiv preprint arXiv:1905.08997.","journal-title":"arXiv preprint arXiv:1905.08997"},{"key":"e_1_3_2_79_2","unstructured":"Yuli Zheng Zhenyu Wu Ye Yuan Tianlong Chen and Zhangyang Wang. 2020. PCAL: A privacy-preserving intelligent credit risk modeling framework based on adversarial learning. (unpublished)."},{"key":"e_1_3_2_80_2","unstructured":"Zhiming Zhou Weinan Zhang and Jun Wang. 2017. Inception score label smoothing gradient vanishing and-log (d (x)) alternative. arXiv preprint arXiv:1708 ."},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i07.7016"}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3472768","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3472768","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:17:09Z","timestamp":1750191429000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3472768"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,31]]},"references-count":80,"journal-issue":{"issue":"3s","published-print":{"date-parts":[[2021,10,31]]}},"alternative-id":["10.1145\/3472768"],"URL":"https:\/\/doi.org\/10.1145\/3472768","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"type":"print","value":"1551-6857"},{"type":"electronic","value":"1551-6865"}],"subject":[],"published":{"date-parts":[[2021,10,31]]},"assertion":[{"value":"2020-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-12-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}