{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T22:43:06Z","timestamp":1777675386159,"version":"3.51.4"},"reference-count":60,"publisher":"SAGE Publications","issue":"4","license":[{"start":{"date-parts":[[2016,6,1]],"date-time":"2016-06-01T00:00:00Z","timestamp":1464739200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"name":"VERCE","award":["EU FP7 RI 283543"],"award-info":[{"award-number":["EU FP7 RI 283543"]}]},{"name":"Terracorrelator","award":["NERC NE\/L012979\/1"],"award-info":[{"award-number":["NERC NE\/L012979\/1"]}]},{"name":"Gordon and Betty Moore Foundation and the National Science Foundation and major contributions from Open Cloud Consortium"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of High Performance Computing Applications"],"published-print":{"date-parts":[[2017,7]]},"abstract":"<jats:p>This paper presents dispel4py, a new Python framework for describing abstract stream-based workflows for distributed data-intensive applications. These combine the familiarity of Python programming with the scalability of workflows. Data streaming is used to gain performance, rapid prototyping and applicability to live observations. dispel4py enables scientists to focus on their scientific goals, avoiding distracting details and retaining flexibility over the computing infrastructure they use. The implementation, therefore, has to map dispel4py abstract workflows optimally onto target platforms chosen dynamically. We present four dispel4py mappings: Apache Storm, message-passing interface (MPI), multi-threading and sequential, showing two major benefits: a) smooth transitions from local development on a laptop to scalable execution for production work, and b) scalable enactment on significantly different distributed computing infrastructures. Three application domains are reported and measurements on multiple infrastructures show the optimisations achieved; they have provided demanding real applications and helped us develop effective training. The dispel4py.org is an open-source project to which we invite participation. The effective mapping of dispel4py onto multiple target infrastructures demonstrates exploitation of data-intensive and high-performance computing (HPC) architectures and consistent scalability.<\/jats:p>","DOI":"10.1177\/1094342016649766","type":"journal-article","created":{"date-parts":[[2016,6,2]],"date-time":"2016-06-02T20:57:37Z","timestamp":1464901057000},"page":"316-334","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":12,"title":["<tt>dispel4py<\/tt>\n                    : A Python framework for data-intensive scientific computing"],"prefix":"10.1177","volume":"31","author":[{"given":"Rosa","family":"Filguiera","sequence":"first","affiliation":[{"name":"University of Edinburgh, School of Informatics, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Amrey","family":"Krause","sequence":"additional","affiliation":[{"name":"University of Edinburgh, EPCC, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Malcolm","family":"Atkinson","sequence":"additional","affiliation":[{"name":"University of Edinburgh, School of Informatics, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Iraklis","family":"Klampanos","sequence":"additional","affiliation":[{"name":"National Centre for Scientific Research Demokritos, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alexander","family":"Moreno","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology, School of Computer Science, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2016,6,1]]},"reference":[{"key":"bibr1-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1145\/1833398.1833406"},{"key":"bibr2-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1007\/s00530-007-0082-0"},{"key":"bibr3-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1007\/s00453-010-9453-6"},{"key":"bibr4-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1109\/BDCloud.2014.63"},{"key":"bibr5-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1002\/pmic.201400391"},{"key":"bibr6-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1002\/9781118540343.ch4"},{"key":"bibr7-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1002\/9781118540343.ch1"},{"key":"bibr8-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1002\/9781118540343"},{"key":"bibr9-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1109\/eScience.2015.38"},{"key":"bibr10-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1007\/s10619-012-7105-3"},{"key":"bibr11-1094342016649766","volume-title":"Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC\u201910)","author":"Baccianella S","year":"2010"},{"key":"bibr12-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2009.8"},{"key":"bibr13-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1016\/j.ecoinf.2009.08.008"},{"key":"bibr14-1094342016649766","doi-asserted-by":"crossref","unstructured":"Beisken S, Meinl T, Wiswedel B, (2013) Knime-cdk: Workflow-driven cheminformatics. BMC Bioinformatics 14(1): 257 (online). Available at: http:\/\/www.biomedcentral.com\/1471-2105\/14\/257","DOI":"10.1186\/1471-2105-14-257"},{"key":"bibr15-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1016\/j.websem.2015.01.003"},{"key":"bibr16-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-246X.2007.03374.x"},{"key":"bibr17-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-84628-757-2_3"},{"key":"bibr18-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1117\/12.856486"},{"key":"bibr19-1094342016649766","doi-asserted-by":"crossref","unstructured":"Berthold MR, Cebron N, Dill F, (2009) Knime \u2013 The konstanz information miner. SIGKDD Explorations 11: 26\u201331 (online). Available at: http:\/\/doi.acm.org\/10.1145\/1656274.1656280","DOI":"10.1145\/1656274.1656280"},{"key":"bibr20-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1785\/gssrl.81.3.530"},{"key":"bibr21-1094342016649766","volume-title":"Galaxy: A Web-Based Genome Analysis Tool for Experimentalists","author":"Blankenberg D","year":"2010"},{"key":"bibr22-1094342016649766","unstructured":"Bode MF, Cruz MJ, Molster FJ (2008) The ASTRONET infrastructure roadmap. ASTRONET (online). Available at: http:\/\/books.google.co.uk\/books?id=Y5tQPgAACAAJ"},{"key":"bibr23-1094342016649766","first-page":"18","author":"Buil-Aranda C","year":"2013","journal-title":"Web Semantics"},{"key":"bibr24-1094342016649766","doi-asserted-by":"crossref","unstructured":"Carpene M, Klampanos I, Leong S, (2013) Towards addressing CPU-intensive seismological applications in Europe. In: Kunkel J, Ludwig T, Meuer H (eds) Supercomputing (vol. 7905, LNCS). Berlin\/Heidelberg: Springer, pp. 55\u201366 (online). Available at: http:\/\/dx.doi.org\/10.1007\/978-3-642-38750-05","DOI":"10.1007\/978-3-642-38750-0_5"},{"key":"bibr25-1094342016649766","unstructured":"Cho H, Lee J, Kim S (2013) Enhancing lexicon-based review classification by merging and revising sentiment dictionaries. In: Sixth International Joint Conference on Natural Language Processing (IJCNLP 2013), Nagoya, Japan, 14\u201318 October 2013, pp. 463\u2013470. Asian Federation of Natural Language Processing\/ACL (online). Available at: http:\/\/aclweb.org\/anthology\/I\/I13\/I13-1053.pdf"},{"key":"bibr26-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1002\/cpe.992"},{"key":"bibr27-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2009.22"},{"key":"bibr28-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2008.06.010."},{"key":"bibr29-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2014.10.008"},{"key":"bibr30-1094342016649766","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/9780262014397.001.0001"},{"key":"bibr31-1094342016649766","doi-asserted-by":"crossref","unstructured":"Earle PS, Wald DJ, Jaiswal KS, (2009) Prompt Assessment of Global Earthquakes for Response (PAGER): A System for Rapidly Determining the Impact of Earthquakes Worldwide (Technical Report). US Geological Survey.","DOI":"10.3133\/ofr20091131"},{"key":"bibr32-1094342016649766","doi-asserted-by":"crossref","unstructured":"Falt Z, Bedn\u00e1rek D, Kruli\u0161 M, (2014) Bobolang: A language for parallel streaming applications. In: Proceedings of the 23rd international symposium on high-performance parallel and distributed computing (HPDC\u201914). New York: ACM, pp. 311\u2013314 (online). Available at: http:\/\/doi.acm.org\/10.1145\/2600212.2600711","DOI":"10.1145\/2600212.2600711"},{"key":"bibr33-1094342016649766","author":"Filgueira R","year":"2015","journal-title":"Proceedings of the IEEE e-Science 2015"},{"key":"bibr34-1094342016649766","author":"Filgueira R","year":"2014","journal-title":"Euro-Par"},{"key":"bibr35-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-84628-757-2_9"},{"key":"bibr36-1094342016649766","doi-asserted-by":"crossref","unstructured":"Garijo D, Corcho O, Gil Y, (2014) Workflow reuse in practice: A study of neuroimaging pipeline users. In:e-Science. IEEE Computer Society, pp. 239\u2013246 (online). Available at: http:\/\/dblp.uni-trier.de\/db\/conf\/eScience\/eScience2014.html#GarijoCGBHHJTT14","DOI":"10.1109\/eScience.2014.33"},{"key":"bibr37-1094342016649766","doi-asserted-by":"crossref","unstructured":"Gesing S, Atkinson M, Filgueira R, (2014) Workflows in a dashboard: A new generation of usability. In: Proceedings of the WORKS\u201914. Piscataway, NJ: IEEE Press, 2014, pp. 82\u201393 (online). Available: http:\/\/dx.doi.org\/10.1109\/WORKS.2014.6","DOI":"10.1109\/WORKS.2014.6"},{"key":"bibr38-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpdc.2013.04.006"},{"key":"bibr39-1094342016649766","volume-title":"The Fourth Paradigm: Data-Intensive Scientific Discovery","author":"Hey AJG","year":"2009"},{"key":"bibr40-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1002\/9781118540343.ch22"},{"key":"bibr41-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-84628-757-2_10"},{"key":"bibr42-1094342016649766","doi-asserted-by":"crossref","unstructured":"Medhat W, Hassan A, Korashy H (2014) Sentiment analysis algorithms and applications: A survey. Ain Shams Engineering Journal 5(4): 1093\u20131113 (online). Available at: http:\/\/www.sciencedirect.com\/science\/article\/pii\/S2090447914000550","DOI":"10.1016\/j.asej.2014.04.011"},{"key":"bibr43-1094342016649766","first-page":"165","volume":"8","author":"MPI Forum","year":"1994","journal-title":"International Journal of Supercomputer Applications"},{"key":"bibr44-1094342016649766","volume-title":"AFINN","author":"Nielsen FA","year":"2011"},{"key":"bibr45-1094342016649766","volume-title":"Proceedings of the International Conference on Language Resources and Evaluation (LREC 2010)","author":"Pak A","year":"2010"},{"key":"bibr46-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1016\/j.cageo.2014.12.006"},{"key":"bibr47-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1007\/s10723-013-9267-2"},{"key":"bibr48-1094342016649766","unstructured":"Rosner M, Tapias D (eds) (2010) European Language Resources Association (online). Available at: http:\/\/www.lrec-conf.org\/proceedings\/lrec2010\/summaries\/385.html"},{"key":"bibr49-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1016\/j.ascom.2014.09.002"},{"key":"bibr50-1094342016649766","volume-title":"ADASS Conference","author":"Rynge M","year":"2013"},{"key":"bibr51-1094342016649766","volume-title":"Beautiful Data: The Stories behind Elegant Data Solutions","author":"Segaran T","year":"2009"},{"key":"bibr52-1094342016649766","volume-title":"Scientific Data Management: Challenges, Technology and Deployment","author":"Shoshani A","year":"2010"},{"key":"bibr53-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1109\/ADVCOMP.2009.14"},{"key":"bibr54-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1145\/2457317.2457369"},{"key":"bibr55-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2013.19"},{"key":"bibr56-1094342016649766","author":"Vahi K","year":"2013","journal-title":"Workshop on Big Data and Science: Infrastructure and Services"},{"key":"bibr57-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1038\/scientificamerican0805-32"},{"key":"bibr58-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1016\/j.parco.2011.05.005"},{"key":"bibr59-1094342016649766","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gkt328"},{"key":"bibr60-1094342016649766","doi-asserted-by":"crossref","unstructured":"Wozniak JM, Armstrong TG, Maheshwari K, (2013) Turbine: A distributed-memory dataflow engine for high performance many-task applications. Fundamenta Informaticae 128(3): 337\u2013366 (online). Available at: http:\/\/dx.doi.org\/10.3233\/FI-2013-949","DOI":"10.3233\/FI-2013-949"}],"container-title":["The International Journal of High Performance Computing Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1094342016649766","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/1094342016649766","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1094342016649766","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T08:15:25Z","timestamp":1777450525000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/1094342016649766"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,6,1]]},"references-count":60,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2017,7]]}},"alternative-id":["10.1177\/1094342016649766"],"URL":"https:\/\/doi.org\/10.1177\/1094342016649766","relation":{},"ISSN":["1094-3420","1741-2846"],"issn-type":[{"value":"1094-3420","type":"print"},{"value":"1741-2846","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,6,1]]}}}