{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T07:41:20Z","timestamp":1760254880831,"version":"build-2065373602"},"reference-count":40,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2018,5,3]],"date-time":"2018-05-03T00:00:00Z","timestamp":1525305600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Analyzing huge amounts of data becomes essential in the era of Big Data, where databases are populated with hundreds of Gigabytes that must be processed to extract knowledge. Hence, classical algorithms must be adapted towards distributed computing methodologies that leverage the underlying computational power of these platforms. Here, a parallel, scalable, and optimized design for self-organized maps (SOM) is proposed in order to analyze massive data gathered by the spectrophotometric sensor of the European Space Agency (ESA) Gaia spacecraft, although it could be extrapolated to other domains. The performance comparison between the sequential implementation and the distributed ones based on Apache Hadoop and Apache Spark is an important part of the work, as well as the detailed analysis of the proposed optimizations. Finally, a domain-specific visualization tool to explore astronomical SOMs is presented.<\/jats:p>","DOI":"10.3390\/s18051419","type":"journal-article","created":{"date-parts":[[2018,5,4]],"date-time":"2018-05-04T03:08:21Z","timestamp":1525403301000},"page":"1419","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Distributed Fast Self-Organized Maps for Massive Spectrophotometric Data Analysis \u2020"],"prefix":"10.3390","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4693-7555","authenticated-orcid":false,"given":"Carlos","family":"Dafonte","sequence":"first","affiliation":[{"name":"CITIC\u2014Department of Computer Science, University of A Coru\u00f1a, Campus de Elvi\u00f1a s\/n, 15071 A Coru\u00f1a, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7133-6623","authenticated-orcid":false,"given":"Daniel","family":"Garabato","sequence":"additional","affiliation":[{"name":"CITIC\u2014Department of Computer Science, University of A Coru\u00f1a, Campus de Elvi\u00f1a s\/n, 15071 A Coru\u00f1a, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6786-2620","authenticated-orcid":false,"given":"Marco A.","family":"\u00c1lvarez","sequence":"additional","affiliation":[{"name":"CITIC\u2014Department of Computer Science, University of A Coru\u00f1a, Campus de Elvi\u00f1a s\/n, 15071 A Coru\u00f1a, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7711-5581","authenticated-orcid":false,"given":"Minia","family":"Manteiga","sequence":"additional","affiliation":[{"name":"CITIC\u2014Astronomy and Astrophysics, University of A Coru\u00f1a, Campus de Elvi\u00f1a s\/n, 15071 A Coru\u00f1a, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2018,5,3]]},"reference":[{"key":"ref_1","unstructured":"Karau, H., Konwinski, A., Wendell, P., and Zaharia, M. (2015). Learning Spark: Lightning-Fast Big Data Analytics, O\u2019Reilly Media, Inc.. [1st ed.]."},{"key":"ref_2","unstructured":"White, T. (2015). Hadoop: The Definitive Guide, O\u2019Reilly Media Inc."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"28","DOI":"10.3847\/1538-3881\/aa7567","article-title":"Sloan digital sky survey IV: Mapping the Milky Way, nearby galaxies, and the distant universe","volume":"154","author":"Blanton","year":"2017","journal-title":"Astron. J."},{"key":"ref_4","unstructured":"Gaia Collaboration, Prusti, T., de Bruijne, J.H.J., Brown, A.G.A., Vallenari, A., Babusiaux, C., Bailer-Jones, C.A.L., Bastian, U., Biermann, M., and Evans, D.W. (2016). The Gaia mission. Astron. Astrophys., 595, A1."},{"key":"ref_5","unstructured":"LSST Science Collaboration, Abell, P.A., Allison, J., Anderson, S.F., Andrew, J.R., Angel, J.R.P., Armus, L., Arnett, D., Asztalos, S.J., and Axelrod, T.S. (arXiv, 2009). LSST Science Book, Version 2.0, arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"A48","DOI":"10.1051\/0004-6361\/201015441","article-title":"Gaia broad band photometry","volume":"523","author":"Jordi","year":"2010","journal-title":"Astron. Astrophys."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1007\/s10509-012-1019-4","article-title":"Science performance of Gaia, ESA\u2019s space-astrometry mission","volume":"341","year":"2012","journal-title":"Astrophys. Space Sci."},{"key":"ref_8","unstructured":"Gaia Collaboration, Brown, A.G.A., Vallenari, A., Prusti, T., de Bruijne, J.H.J., Babusiaux, C., and Bailer-Jones, C.A.L. (arXiv, 2018). Gaia Data Release 2. Summary of the contents and survey properties, arXiv."},{"key":"ref_9","unstructured":"Gaia Collaboration, Brown, A.G.A., Vallenari, A., Prusti, T., de Bruijne, J.H.J., Babusiaux, C., and Bailer-Jones, C.A.L. (2016). Gaia Data Release 1. Summary of the astrometric, photometric, and survey properties. Astron. Astrophys., 595, A2."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"A74","DOI":"10.1051\/0004-6361\/201322344","article-title":"The Gaia astrophysical parameters inference system (Apsis). Pre-launch description","volume":"559","author":"Andrae","year":"2013","journal-title":"Astron. Astrophys."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"3245","DOI":"10.1088\/0004-6256\/137\/2\/3245","article-title":"Starmind: A fuzzy logic knowledge-based system for the automated classification of stars in the MK system","volume":"137","author":"Manteiga","year":"2009","journal-title":"Astron. J."},{"key":"ref_12","first-page":"204","article-title":"HSC: A multi-resolution clustering strategy in Self-Organizing Maps applied to astronomical observations","volume":"12","author":"Dafonte","year":"2012","journal-title":"ASOC Elsevier"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Sarro, L.M., Eyer, L., O\u2019Mullane, W., and De Ridder, J. (2012). The discrete source classifier in Gaia-apsis. Astrostatistics and Data Mining, Springer.","DOI":"10.1007\/978-1-4614-3323-1"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1007\/BF00994018","article-title":"Support-Vector Networks","volume":"20","author":"Cortes","year":"1995","journal-title":"Mach. Learn."},{"key":"ref_15","unstructured":"Jolliffe, I. (2002). Principal Component Analysis, Springer."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1111\/j.1469-1809.1936.tb02137.x","article-title":"The use of multiple measurements in taxonomic problems","volume":"7","author":"Fisher","year":"1936","journal-title":"Ann. Eugen."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1109\/TNN.2005.845141","article-title":"Survey of clustering algorithms","volume":"16","author":"Xu","year":"2005","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1007\/BF00337288","article-title":"Self-organized formation of topologically correct feature maps","volume":"43","author":"Kohonen","year":"1982","journal-title":"Biol. Cybern."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Kohonen, T. (2001). Self-Organizing Maps, Springer.","DOI":"10.1007\/978-3-642-56927-2"},{"key":"ref_20","first-page":"1530","article-title":"SOM ensemble for unsupervised outlier analysis. Application to outlier identification in the Gaia astronomical survey","volume":"40","author":"Fustes","year":"2013","journal-title":"ESWA"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"A7","DOI":"10.1051\/0004-6361\/201321445","article-title":"An approach to the analysis of SDSS spectroscopic outliers based on self-organizing maps: Designing the outlier analysis software package for the next Gaia survey","volume":"559","author":"Fustes","year":"2013","journal-title":"Astron. Astrophys."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2633","DOI":"10.1111\/j.1365-2966.2011.19913.x","article-title":"Unsupervised self-organized mapping: A versatile empirical tool for object selection, classification and redshift estimation in large surveys","volume":"419","author":"Geach","year":"2012","journal-title":"MNRAS"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1088\/0004-637X\/727\/1\/48","article-title":"Structure in the Three-dimensional galaxy distribution. I. Methods and example results","volume":"727","author":"Way","year":"2011","journal-title":"Astrophys. J."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"274","DOI":"10.1086\/664796","article-title":"Can self-organizing maps accurately predict photometric redshifts?","volume":"124","author":"Way","year":"2012","journal-title":"Publ. Astron. Soc. Pac."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"A117","DOI":"10.1051\/0004-6361\/201629710","article-title":"Gaia eclipsing binary and multiple systems. Supervised classification and self-organizing maps","volume":"603","author":"Barblan","year":"2017","journal-title":"Astron. Astrophys."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"2634","DOI":"10.1093\/mnras\/stw2881","article-title":"Transit shapes and self-organizing maps as a tool for ranking planetary candidates: Application to Kepler and K2","volume":"465","author":"Armstrong","year":"2017","journal-title":"MNRAS"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Valette, V., and Amsif, K. (2012). CNES Gaia Data Processing Centre: A Complex Operation Plan, American Institute of Aeronautics and Astronautics.","DOI":"10.2514\/6.2012-1291264"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Brunet, P., Montmorry, A., and Frezouls, B. (2012). Big data challenges, an insight into the GAIA Hadoop solution, American Institute of Aeronautics and Astronautics.","DOI":"10.2514\/6.2012-1275512"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"A42","DOI":"10.1051\/0004-6361\/201117125","article-title":"A semi-empirical library of galaxy spectra for Gaia classification based on SDSS data and P\u00c9GASE models","volume":"537","author":"Tsalmantza","year":"2012","journal-title":"Astron. Astrophys."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"415","DOI":"10.1007\/978-3-642-11250-8_106","article-title":"GUMS & GOG: Simulating the universe for Gaia","volume":"Volume 14","author":"Isasi","year":"2010","journal-title":"Highlights of Spanish Astrophysics V"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"A119","DOI":"10.1051\/0004-6361\/201423636","article-title":"Overview and stellar statistics of the expected Gaia Catalogue using the Gaia Object Generator","volume":"566","author":"Luri","year":"2014","journal-title":"Astron. Astrophys."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1086\/167900","article-title":"The relationship between infrared, optical, and ultraviolet extinction","volume":"345","author":"Cardelli","year":"1989","journal-title":"Astrophys. J."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Garabato, D., Dafonte, C., Manteiga, M., Fustes, D., \u00c1lvarez, M.A., and Arcay, B. (July, January 30). A distributed learning algorithm for Self-Organizing Maps intended for outlier analysis in the GAIA\u2014ESA mission. Proceedings of the 2015 Conference of the International Fuzzy Systems Association and the European Society for Fuzzy Logic and Technology, Gij\u00f3n, Spain.","DOI":"10.2991\/ifsa-eusflat-15.2015.126"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1007\/BF02289263","article-title":"Who belongs in the family?","volume":"18","author":"Thorndike","year":"1953","journal-title":"Psychometrika"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"789","DOI":"10.1016\/0167-8191(96)00024-5","article-title":"A high-performance, portable implementation of the MPI message passing interface standard","volume":"22","author":"Lusk","year":"1996","journal-title":"Parallel Comput."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1145\/1327452.1327492","article-title":"MapReduce: Simplified data processing on large clusters","volume":"51","author":"Dean","year":"2008","journal-title":"Commun. ACM"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"\u00c1lvarez, M.A., Dafonte, C., Garabato, D., and Manteiga, M. (2016). Analysis and knowledge discovery by means of self-organizing maps for Gaia data releases. Neural Information Processing, Proceedings of the 23rd International Conference on Neural Information Processing ICONIP, Kyoto, Japan, 16\u201321 October 2016, Springer International Publishing.","DOI":"10.1007\/978-3-319-46681-1_17"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1051\/aas:2000332","article-title":"The SIMBAD astronomical database: The CDS reference database for astronomical objects","volume":"143","author":"Wenger","year":"2000","journal-title":"Astron. Astrophys. Suppl. Ser."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1016\/j.ascom.2014.12.007","article-title":"SAMP, the simple application messaging protocol: Letting applications talk to each other","volume":"11","author":"Taylor","year":"2015","journal-title":"Astron. Comput."},{"key":"ref_40","unstructured":"Sanders, J., and Kandrot, E. (2010). CUDA by Example: An Introduction to General-Purpose GPU Programming, Addison-Wesley Professional. [1st ed.]."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/18\/5\/1419\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T15:03:11Z","timestamp":1760194991000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/18\/5\/1419"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,5,3]]},"references-count":40,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2018,5]]}},"alternative-id":["s18051419"],"URL":"https:\/\/doi.org\/10.3390\/s18051419","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2018,5,3]]}}}