{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T22:43:30Z","timestamp":1777675410977,"version":"3.51.4"},"reference-count":36,"publisher":"SAGE Publications","issue":"1","license":[{"start":{"date-parts":[[2017,4,6]],"date-time":"2017-04-06T00:00:00Z","timestamp":1491436800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of High Performance Computing Applications"],"published-print":{"date-parts":[[2018,1]]},"abstract":"<jats:p>Task-based programming has proven to be a suitable model for high-performance computing (HPC) applications. Different implementations have been good demonstrators of this fact and have promoted the acceptance of task-based programming in the OpenMP standard. Furthermore, in recent years, Apache Spark has gained wide popularity in business and research environments as a programming model for addressing emerging big data problems. COMP Superscalar (COMPSs) is a task-based environment that tackles distributed computing (including Clouds) and is a good alternative for a task-based programming model for big data applications. This article describes why we consider that task-based programming models are a good approach for big data applications. The article includes a comparison of Spark and COMPSs in terms of architecture, programming model, and performance. It focuses on the differences that both frameworks have in structural terms, on their programmability interface, and in terms of their efficiency by means of three widely known benchmarking kernels: Wordcount, Kmeans, and Terasort. These kernels enable the evaluation of the more important functionalities of both programming models and analyze different work flows and conditions. The main results achieved from this comparison are (1) COMPSs is able to extract the inherent parallelism from the user code with minimal coding effort as opposed to Spark, which requires the existing algorithms to be adapted and rewritten by explicitly using their predefined functions, (2) it is an improvement in terms of performance when compared with Spark, and (3) COMPSs has shown to scale better than Spark in most cases. Finally, we discuss the advantages and disadvantages of both frameworks, highlighting the differences that make them unique, thereby helping to choose the right framework for each particular objective.<\/jats:p>","DOI":"10.1177\/1094342017701278","type":"journal-article","created":{"date-parts":[[2017,4,6]],"date-time":"2017-04-06T06:46:09Z","timestamp":1491461169000},"page":"45-60","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":26,"title":["Task-based programming in COMPSs to converge from HPC to big data"],"prefix":"10.1177","volume":"32","author":[{"given":"Javier","family":"Conejero","sequence":"first","affiliation":[{"name":"Barcelona Supercomputing Center (BSC), Barcelona, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sandra","family":"Corella","sequence":"additional","affiliation":[{"name":"Barcelona Supercomputing Center (BSC), Barcelona, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rosa M","family":"Badia","sequence":"additional","affiliation":[{"name":"Barcelona Supercomputing Center (BSC), Barcelona, Spain"},{"name":"Institut d\u2019Investigaci\u00f3 en Intel-lig\u00e8ncia Artificial \u2013 Consejo Superior de Investigaciones Cientificas (IIIA \u2013 CSIC), Barcelona, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jesus","family":"Labarta","sequence":"additional","affiliation":[{"name":"Barcelona Supercomputing Center (BSC), Barcelona, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2017,4,6]]},"reference":[{"key":"bibr1-1094342017701278","unstructured":"Amazon Enterprise Cloud Computing (2016) Available at: http:\/\/aws.amazon.com\/enterprise\/ (accessed 27 November 2016)."},{"key":"bibr2-1094342017701278","unstructured":"Amazon Simple Storage Service (S3) (2016) Available at: https:\/\/aws.amazon.com\/s3\/ (accessed 20 December 2016)."},{"key":"bibr3-1094342017701278","unstructured":"Apache Hadoop (2016) Available at: http:\/\/hadoop.apache.org\/ (accessed 15 November 2016)."},{"key":"bibr4-1094342017701278","unstructured":"Apache Spark \u2013 Lightning-fast cluster computing (2016) Available at: http:\/\/spark.apache.org\/ (accessed 15 November 2016)."},{"key":"bibr5-1094342017701278","doi-asserted-by":"publisher","DOI":"10.1016\/j.softx.2015.10.004"},{"key":"bibr6-1094342017701278","unstructured":"COMP Superscalar (2016) Available at: http:\/\/compss.bsc.es\/ (accessed 17 November 2016)."},{"key":"bibr7-1094342017701278","first-page":"95","volume-title":"Proceedings of the 10th Doctoral Symposium in Informatics Engineering","author":"Da Silva Morais T","year":"2015"},{"key":"bibr8-1094342017701278","unstructured":"Decorators for Functions and Methods (2016) Available at: https:\/\/www.python.org\/dev\/peps\/pep-0318\/ (accessed 17 November 2016)."},{"key":"bibr9-1094342017701278","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2016.7840622"},{"key":"bibr10-1094342017701278","unstructured":"ehigss Spark Terasort (2016) Available at: https:\/\/github.com\/ehiggs\/spark-terasort (accessed 25 November 2016)."},{"key":"bibr11-1094342017701278","unstructured":"Examples Apache Spark (2016) Available at: http:\/\/spark.apache.org\/examples.html (accessed 22 November 2016)."},{"key":"bibr12-1094342017701278","unstructured":"Extrae (2016) Available at: https:\/\/tools.bsc.es\/extrae (accessed 19 December 2016)."},{"key":"bibr13-1094342017701278","first-page":"3","volume-title":"Big Data Benchmarking \u2013 6th International Workshop, WBDB 2015","author":"Fox GC","year":"2015"},{"key":"bibr14-1094342017701278","doi-asserted-by":"publisher","DOI":"10.5120\/19788-0531"},{"key":"bibr15-1094342017701278","unstructured":"Google Cloud Platform \u2013 Cloud Computing, Hosting Services & APIs (2016) Available at: http:\/\/cloud.google.com\/ (accessed 4 December 2016)."},{"key":"bibr16-1094342017701278","first-page":"295","volume-title":"Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation, NSDI\u201911","author":"Hindman B","year":"2011"},{"key":"bibr17-1094342017701278","unstructured":"JNI APIs and Developer Guides (2016) Available at: https:\/\/docs.oracle.com\/javase\/8\/docs\/technotes\/guides\/jni\/ (accessed 19 December 2016)."},{"key":"bibr18-1094342017701278","volume-title":"Java Native Interface: Programmer\u2019s Guide and Reference","author":"Liang S","year":"1999"},{"key":"bibr19-1094342017701278","unstructured":"MareNostrum BSC-CNS (2016) Available at: https:\/\/www.bsc.es\/innovation-and-services\/supercomputers-and-facilities\/marenostrum (accessed 22 November 2016)."},{"key":"bibr20-1094342017701278","doi-asserted-by":"publisher","DOI":"10.1109\/SERVICES.2013.59"},{"issue":"1","key":"bibr21-1094342017701278","first-page":"1235","volume":"17","author":"Meng X","year":"2016","journal-title":"Journal of Machine Learning Research"},{"key":"bibr22-1094342017701278","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2012.76"},{"key":"bibr23-1094342017701278","unstructured":"OCCI \u2013 Open Cloud Computing Interface (2016) Available at: http:\/\/occi-wg.org\/ (accessed 27 November 2016)."},{"key":"bibr24-1094342017701278","unstructured":"OpenNebula \u2013 Flexible Enterprise Cloud Made Simple (2016) Available at: http:\/\/opennebula.org\/ (accessed 15 November 2016)."},{"key":"bibr25-1094342017701278","unstructured":"OpenStack \u2013 Open Source Cloud Computing Software (2016) Available at: http:\/\/openstack.org\/ (accessed 27 November 2016)."},{"key":"bibr26-1094342017701278","unstructured":"Paraver: a flexible performance analysis tool. Available at: https:\/\/tools.bsc.es\/paraver, (accessed 19 December 2016)."},{"key":"bibr27-1094342017701278","unstructured":"Pan S (2016) The performance comparison of hadoop and spark. Master\u2019s Thesis, St. Cloud State University, USA. Culminating Projects in Computer Science and Information Technology. Paper 7."},{"key":"bibr28-1094342017701278","unstructured":"Pillet V, Labarta J, Cortes T, (1995) Paraver: a tool to visualize and analyze parallel code. Transputer and Occam Developments. 17\u201332, April 1995. Available at: http:\/\/www.bsc.es\/ (accessed 17\u201332 April 2012)."},{"key":"bibr29-1094342017701278","doi-asserted-by":"publisher","DOI":"10.1145\/2699414"},{"key":"bibr30-1094342017701278","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2015.07.286"},{"key":"bibr31-1094342017701278","doi-asserted-by":"publisher","DOI":"10.14778\/2831360.2831365"},{"key":"bibr32-1094342017701278","doi-asserted-by":"publisher","DOI":"10.1109\/MSST.2010.5496972"},{"key":"bibr33-1094342017701278","doi-asserted-by":"publisher","DOI":"10.1177\/1094342015594678"},{"key":"bibr34-1094342017701278","unstructured":"UCI Machine Learning Repository: HIGGS Data Set (2016) Available at: http:\/\/archive.ics.uci.edu\/ml\/datasets\/HIGGS (accessed 4 December 2016)."},{"key":"bibr35-1094342017701278","doi-asserted-by":"publisher","DOI":"10.1145\/2523616.2523633"},{"key":"bibr36-1094342017701278","volume-title":"Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing, HotCloud\u201910","author":"Zaharia M","year":"2010"}],"container-title":["The International Journal of High Performance Computing Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1094342017701278","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/1094342017701278","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1094342017701278","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T08:15:29Z","timestamp":1777450529000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/1094342017701278"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,4,6]]},"references-count":36,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2018,1]]}},"alternative-id":["10.1177\/1094342017701278"],"URL":"https:\/\/doi.org\/10.1177\/1094342017701278","relation":{},"ISSN":["1094-3420","1741-2846"],"issn-type":[{"value":"1094-3420","type":"print"},{"value":"1741-2846","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,4,6]]}}}