{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,4,1]],"date-time":"2025-04-01T05:44:44Z","timestamp":1743486284586,"version":"3.37.3"},"reference-count":56,"publisher":"Springer Science and Business Media LLC","issue":"12","license":[{"start":{"date-parts":[[2022,9,9]],"date-time":"2022-09-09T00:00:00Z","timestamp":1662681600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,9,9]],"date-time":"2022-09-09T00:00:00Z","timestamp":1662681600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In this paper, we focus on unsupervised representation learning for clustering of images. Recent advances in deep clustering and unsupervised representation learning are based on the idea that different views of an input image (generated through data augmentation techniques) must be close in the representation space (exemplar consistency), and\/or similar images must have similar cluster assignments (population consistency). We define an additional notion of consistency,<jats:italic>consensus consistency<\/jats:italic>, which ensures that representations are learned to induce similar partitions for variations in the representation space, different clustering algorithms or different initializations of a single clustering algorithm. We define a clustering loss by executing variations in the representation space and seamlessly integrate all three consistencies (consensus, exemplar and population) into an end-to-end learning framework. The proposed algorithm, consensus clustering using unsupervised representation learning (ConCURL), improves upon the clustering performance of state-of-the-art methods on four out of five image datasets. Furthermore, we extend the evaluation procedure for clustering to reflect the challenges encountered in real-world clustering tasks, such as maintaining clustering performance in cases with distribution shifts. We also perform a detailed ablation study for a deeper understanding of the proposed algorithm. The code and the trained models are available at<jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/JayanthRR\/ConCURL_NCE\">https:\/\/github.com\/JayanthRR\/ConCURL_NCE<\/jats:ext-link>.<\/jats:p>","DOI":"10.1007\/s10994-022-06194-9","type":"journal-article","created":{"date-parts":[[2022,9,9]],"date-time":"2022-09-09T21:03:00Z","timestamp":1662757380000},"page":"4601-4638","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Representation learning for clustering via building consensus"],"prefix":"10.1007","volume":"111","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7292-8436","authenticated-orcid":false,"given":"Aniket Anand","family":"Deshmukh","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jayanth Reddy","family":"Regatti","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eren","family":"Manavoglu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Urun","family":"Dogan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,9,9]]},"reference":[{"key":"6194_CR1","unstructured":"Asano, Y.M. , Rupprecht, C., Vedaldi, A. (2019). Self-labelling via simultaneous clustering and representation learning. arXiv preprint arXiv:1911.05371."},{"key":"6194_CR2","doi-asserted-by":"crossref","unstructured":"Bengio, Y. , Lamblin, P. , Popovici, D. , Larochelle, H., Montreal, U. (2007). Greedy layer-wise training of deep networks . NeurIPS19, 153-160.","DOI":"10.7551\/mitpress\/7503.003.0024"},{"key":"6194_CR3","unstructured":"Cai, D. , He, X. , Wang, X. , Bao, H., Han, J. (2009). Locality Preserving Nonnegative Matrix Factorization. Ijcai (pp. 1010\u20131015). http:\/\/ijcai.org\/Proceedings\/09\/Papers\/171.pdf"},{"key":"6194_CR4","doi-asserted-by":"crossref","unstructured":"Caron, M. , Bojanowski, P. , Joulin, A., Douze, M. (2018a). Deep clustering for unsupervised learning of visual features. In: Proceedings of the European Conference on Computer Vision (eccv) (pp. 132\u2013149).","DOI":"10.1007\/978-3-030-01264-9_9"},{"key":"6194_CR5","doi-asserted-by":"crossref","unstructured":"Caron, M., Bojanowski, P., Joulin, A., & Douze, M. (2018). Deep clustering for unsupervised learning of visual features. Eccv,11218, 139\u2013156.","DOI":"10.1007\/978-3-030-01264-9_9"},{"key":"6194_CR6","unstructured":"Caron, M. , Misra, I. , Mairal, J. , Goyal, P. , Bojanowski, P., Joulin, A. (2020). Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09882."},{"key":"6194_CR7","unstructured":"Chang, J. , Guo, Y. , Wang, L. , Meng, G. , Xiang, S., Pan, C. (2019). Deep discriminative clustering analysis."},{"key":"6194_CR8","doi-asserted-by":"crossref","unstructured":"Chang, J. , Wang, L. , Meng, G. , Xiang, S., Pan, C. (2017). Deep adaptive image clustering. The IEEE International Conference on Computer Vision (iccv).","DOI":"10.1109\/ICCV.2017.626"},{"key":"6194_CR9","doi-asserted-by":"publisher","unstructured":"Chang, J. , Wang, L. , Meng, G. , Xiang, S., Pan, C. (2017). Deep adaptive image clustering. Iccv (pp. 5880-5888). https:\/\/doi.org\/10.1109\/ICCV.2017.626","DOI":"10.1109\/ICCV.2017.626"},{"key":"6194_CR10","unstructured":"Chen, T. , Kornblith, S. , Norouzi, M., Hinton, G. (2020). A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709."},{"key":"6194_CR11","unstructured":"Cuturi, M. (2013). Sinkhorn distances: Lightspeed computation of optimal transport. Advances in Neural Information Processing Systems (pp. 2292\u20132300)."},{"key":"6194_CR12","doi-asserted-by":"crossref","unstructured":"Deng, J. , Dong, W. , Socher, R. , Li, L.J. , Li, K., Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition (pp. 248\u2013255).","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"6194_CR13","unstructured":"Fern, X.Z., & Brodley, C.E. (2003). Random projection for high dimensional data clustering: A cluster ensemble approach. In: Proceedings of the 20th International Conference on Machine Learning (icml-03) (pp. 186\u2013193)."},{"issue":"11","key":"6194_CR14","doi-asserted-by":"publisher","first-page":"1875","DOI":"10.1109\/TPAMI.2006.227","volume":"28","author":"P Franti","year":"2006","unstructured":"Franti, P., Virmajoki, O., & Hautamaki, V. (2006). Fast agglomerative clustering using a k-nearest neighbor graph. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(11), 1875\u20131881.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"issue":"6","key":"6194_CR15","doi-asserted-by":"publisher","first-page":"835","DOI":"10.1109\/TPAMI.2005.113","volume":"27","author":"AL Fred","year":"2005","unstructured":"Fred, A. L., & Jain, A. K. (2005). Combining multiple clusterings using evidence accumulation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(6), 835\u2013850.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"issue":"5814","key":"6194_CR16","doi-asserted-by":"publisher","first-page":"972","DOI":"10.1126\/science.1136800","volume":"315","author":"BJ Frey","year":"2007","unstructured":"Frey, B. J., & Dueck, D. (2007). Clustering by passing messages between data points. Science, 315(5814), 972\u2013976.","journal-title":"Science"},{"issue":"4","key":"6194_CR17","first-page":"305","volume":"1","author":"J Ghosh","year":"2011","unstructured":"Ghosh, J., & Acharya, A. (2011). Cluster ensembles. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 1(4), 305\u2013315.","journal-title":"Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery"},{"key":"6194_CR18","unstructured":"Grill, J. B. , Strub, F. , Altch\u00e9, F. , Tallec, C. , Richemond, P.H. , Buchatskaya, E. & others (2020). Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733."},{"key":"6194_CR19","doi-asserted-by":"crossref","unstructured":"Haeusser, P. Plapp, J. , Golkov, V. , Aljalbout, E., Cremers, D. (2019). Associative deep clustering: Training a classification network with no labels. T.\u00a0Brox, A.\u00a0Bruhn, & M.\u00a0Fritz (eds), Pattern recognition (pp. 18\u201332). Cham, Springer International Publishing.","DOI":"10.1007\/978-3-030-12939-2_2"},{"key":"6194_CR20","doi-asserted-by":"crossref","unstructured":"He, K. , Fan, H. , Wu, Y. , Xie, S., Girshick, R. (2019). Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722.","DOI":"10.1109\/CVPR42600.2020.00975"},{"key":"6194_CR21","doi-asserted-by":"crossref","unstructured":"He, K. , Fan, H. , Wu, Y. , Xie, S., Girshick, R. (2020). Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (pp. 9729\u20139738).","DOI":"10.1109\/CVPR42600.2020.00975"},{"key":"6194_CR22","unstructured":"Hsu, K. , Levine, S., Finn, C. (2018). Unsupervised learning via meta-learning. arXiv preprint arXiv:1810.02334."},{"key":"6194_CR23","doi-asserted-by":"crossref","unstructured":"Huang, J. , Gong, S., Zhu, X. (2020a). Deep semantic clustering by partition confidence maximisation. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (pp. 8849\u20138858).","DOI":"10.1109\/CVPR42600.2020.00887"},{"key":"6194_CR24","doi-asserted-by":"crossref","unstructured":"Huang, J. , Gong, S., Zhu, X. (2020b). Deep semantic clustering by partition confidence maximisation. Cvpr.","DOI":"10.1109\/CVPR42600.2020.00887"},{"key":"6194_CR25","unstructured":"Jain, A.K., & Dubes, R.C. (1988). Algorithms for clustering data. Prentice-Hall, Inc."},{"issue":"3","key":"6194_CR26","doi-asserted-by":"publisher","first-page":"264","DOI":"10.1145\/331499.331504","volume":"31","author":"AK Jain","year":"1999","unstructured":"Jain, A. K., Murty, M. N., & Flynn, P. J. (1999). Data clustering: A review. ACM Computing Surveys (CSUR), 31(3), 264\u2013323.","journal-title":"ACM Computing Surveys (CSUR)"},{"key":"6194_CR27","doi-asserted-by":"crossref","unstructured":"Ji, X. , Henriques, J.F., Vedaldi, A. (2019a). Invariant information clustering for unsupervised image classification and segmentation. In: Proceedings of the IEEE International Conference on Computer Vision (pp. 9865\u20139874).","DOI":"10.1109\/ICCV.2019.00996"},{"key":"6194_CR28","doi-asserted-by":"crossref","unstructured":"Ji, X., Henriques, J.F., Vedaldi, A. (2019b). Invariant information clustering for unsupervised image classification and segmentation. Iccv.","DOI":"10.1109\/ICCV.2019.00996"},{"issue":"1\u20132","key":"6194_CR29","doi-asserted-by":"publisher","first-page":"83","DOI":"10.1002\/nav.3800020109","volume":"2","author":"HW Kuhn","year":"1955","unstructured":"Kuhn, H. W. (1955). The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1\u20132), 83\u201397.","journal-title":"Naval Research Logistics Quarterly"},{"issue":"4","key":"6194_CR30","doi-asserted-by":"publisher","first-page":"253","DOI":"10.1002\/nav.3800030404","volume":"3","author":"HW Kuhn","year":"1956","unstructured":"Kuhn, H. W. (1956). Variants of the hungarian method for assignment problems. Naval Research Logistics Quarterly, 3(4), 253\u2013258.","journal-title":"Naval Research Logistics Quarterly"},{"issue":"7553","key":"6194_CR31","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436\u2013444.","journal-title":"Nature"},{"key":"6194_CR32","doi-asserted-by":"crossref","unstructured":"Li, Y. , Hu, P. , Liu, Z. , Peng, D. , Zhou, J.T., Peng, X. (2021). Contrastive clustering. Aaai.","DOI":"10.1609\/aaai.v35i10.17037"},{"key":"6194_CR33","unstructured":"Macqueen, J. (1967). Some methods for classification and analysis of multivariate observations. In 5-th berkeley Symposium on Mathematical Statistics and Probability (pp. 281\u2013297)."},{"issue":"2","key":"6194_CR34","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1016\/S0933-3657(98)00069-4","volume":"16","author":"F Masulli","year":"1999","unstructured":"Masulli, F., & Schenone, A. (1999). A fuzzy clustering based segmentation system as support to diagnosis in medical imaging. Artificial Intelligence in Medicine, 16(2), 129\u2013147.","journal-title":"Artificial Intelligence in Medicine"},{"key":"6194_CR35","unstructured":"Ng, A.Y. , Jordan, M.I., Weiss, Y. (2002). On spectral clustering: Analysis and an algorithm. T.G.\u00a0Dietterich, S.\u00a0Becker, & Z.\u00a0Ghahramani (Eds.), NeurIPS Neurips (pp. 849\u2013856). MIT Press. http:\/\/papers.nips.cc\/paper\/2092-on-spectral-clustering-analysis-and-an-algorithm.pdf"},{"key":"6194_CR36","doi-asserted-by":"crossref","unstructured":"Niu, C. , Shan, H., Wang, G. (2021). Spice: Semantic pseudo-labeling for image clustering. arXiv preprint arXiv:2103.09382.","DOI":"10.1109\/TIP.2022.3221290"},{"key":"6194_CR37","doi-asserted-by":"crossref","unstructured":"Niu, C. , Zhang, J. , Wang, G., Liang, J. (2020a). Gatcluster: Self-supervised gaussian-attention network for image clustering. pp. 735\u2013751.","DOI":"10.1007\/978-3-030-58595-2_44"},{"key":"6194_CR38","doi-asserted-by":"crossref","unstructured":"Niu, C. , Zhang, J. , Wang, G., Liang, J. (2020b). Gatcluster: Self-supervised gaussian-attention network for image clustering. Eccv (pp. 735\u2013751).","DOI":"10.1007\/978-3-030-58595-2_44"},{"key":"6194_CR39","doi-asserted-by":"crossref","unstructured":"Regatti, J.R. , Deshmukh, A.A. , Manavoglu, E., Dogan, U. (2021). Consensus clustering with unsupervised representation learning. In: International Joint Conference on Neural Networks (IJCNN) arXiv preprint arXiv:2010.01245.","DOI":"10.1109\/IJCNN52387.2021.9533714"},{"key":"6194_CR40","doi-asserted-by":"crossref","unstructured":"Schops, T. , Schonberger, J.L. , Galliani, S. , Sattler, T. , Schindler, K. , Pollefeys, M., Geiger, A. (2017). A multi-view stereo benchmark with high-resolution images and multi-camera videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3260\u20133269).","DOI":"10.1109\/CVPR.2017.272"},{"key":"6194_CR41","unstructured":"Shah, S.A., & Koltun, V. (2018). Deep continuous clustering. arXiv preprint arXiv:1803.01449."},{"issue":"1","key":"6194_CR42","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s40537-019-0197-0","volume":"6","author":"C Shorten","year":"2019","unstructured":"Shorten, C., & Khoshgoftaar, T. M. (2019). A survey on image data augmentation for deep learning. Journal of Big Data, 6(1), 1\u201348.","journal-title":"Journal of Big Data"},{"key":"6194_CR43","first-page":"583","volume":"3","author":"A Strehl","year":"2002","unstructured":"Strehl, A., & Ghosh, J. (2002). Cluster ensembles\u2013a knowledge reuse framework for combining multiple partitions. Journal of Machine Learning Research, 3, 583\u2013617.","journal-title":"Journal of Machine Learning Research"},{"key":"6194_CR44","unstructured":"Tao, Y. , Takagi, K., Nakata, K. (2021). Clustering-friendly representation learn-ing via instance discrimination and feature decorrelation. arXiv preprint arXiv:2106.00131."},{"key":"6194_CR45","unstructured":"Tian, Y. , Sun, C. , Poole, B. , Krishnan, D. , Schmid, C., Isola, P. (2020). What makes for good views for contrastive learning. arXiv preprint arXiv:2005.10243."},{"key":"6194_CR46","doi-asserted-by":"crossref","unstructured":"Van\u00a0Gansbeke, W. , Vandenhende, S. , Georgoulis, S. , Proesmans, M. & Van\u00a0Gool, L. (2020). Scan: Learning to classify images without labels.","DOI":"10.1007\/978-3-030-58607-2_16"},{"issue":"12","key":"6194_CR47","first-page":"3371","volume":"11","author":"P Vincent","year":"2010","unstructured":"Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., & Manzagol, P. A. (2010). Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 11(12), 3371\u20133408.","journal-title":"Journal of Machine Learning Research"},{"key":"6194_CR48","doi-asserted-by":"crossref","unstructured":"Wu, J. , Long, K. , Wang, F. , Qian, C. , Li, C. , Lin, Z., Zha, H. (2019a). Deep comprehensive correlation mining for image clustering. In: Proceedings of the IEEE International Conference on Computer Vision (pp. 8150\u20138159).","DOI":"10.1109\/ICCV.2019.00824"},{"key":"6194_CR49","doi-asserted-by":"crossref","unstructured":"Wu, J. , Long, K. , Wang, F. , Qian, C. , Li, C. , Lin, Z., Zha, H. (2019b). Deep comprehensive correlation mining for image clustering. Iccv.","DOI":"10.1109\/ICCV.2019.00824"},{"key":"6194_CR50","doi-asserted-by":"crossref","unstructured":"Wu, Z. , Xiong, Y. , Yu, S.X., Lin, D. (2018). Unsupervised feature learning via non-parametric instance discrimination. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3733\u20133742).","DOI":"10.1109\/CVPR.2018.00393"},{"key":"6194_CR51","unstructured":"Xie, J. , Girshick, R., Farhadi, A. (2016a). Unsupervised deep embedding for clustering analysis. In: International Conference on Machine Learning (pp. 478\u2013487)."},{"key":"6194_CR52","unstructured":"Xie, J. , Girshick, R., Farhadi, A. (2016b). Unsupervised deep embedding for clustering analysis. Icml (pp. 478\u2013487). JMLR.org. http:\/\/dl.acm.org\/citation.cfm?id=3045390.3045442"},{"issue":"3","key":"6194_CR53","doi-asserted-by":"publisher","first-page":"645","DOI":"10.1109\/TNN.2005.845141","volume":"16","author":"R Xu","year":"2005","unstructured":"Xu, R., & Wunsch, D. (2005). Survey of clustering algorithms. IEEE Transactions on Neural Networks, 16(3), 645\u2013678.","journal-title":"IEEE Transactions on Neural Networks"},{"key":"6194_CR54","doi-asserted-by":"crossref","unstructured":"Yang, J. , Parikh, D., Batra, D. (2016). Joint unsupervised learning of deep representations and image clusters. Cvpr.","DOI":"10.1109\/CVPR.2016.556"},{"key":"6194_CR55","doi-asserted-by":"crossref","unstructured":"Zeiler, M.D. , Krishnan, D. , Taylor, G.W., Fergus, R. (2010). Deconvolutional networks. In: Computer Vision and Pattern Recognition.","DOI":"10.1109\/CVPR.2010.5539957"},{"key":"6194_CR56","doi-asserted-by":"crossref","unstructured":"Zhuang, C. , Zhai, A.L., Yamins, D. (2019). Local aggregation for unsupervised learning of visual embeddings. In: Proceedings of the IEEE International Conference on Computer Vision (pp. 6002\u20136012).","DOI":"10.1109\/ICCV.2019.00610"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-022-06194-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-022-06194-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-022-06194-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,26]],"date-time":"2023-11-26T20:06:34Z","timestamp":1701029194000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-022-06194-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,9]]},"references-count":56,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["6194"],"URL":"https:\/\/doi.org\/10.1007\/s10994-022-06194-9","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"type":"print","value":"0885-6125"},{"type":"electronic","value":"1573-0565"}],"subject":[],"published":{"date-parts":[[2022,9,9]]},"assertion":[{"value":"8 September 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 February 2022","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 April 2022","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 September 2022","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Authors are from Microsoft and The Ohio State University. There are no conflict of interests to disclose.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}},{"value":"Not applicable","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}},{"value":"Yes","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"Yes","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}}]}}