{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:11:05Z","timestamp":1750219865478,"version":"3.41.0"},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2023,5,26]],"date-time":"2023-05-26T00:00:00Z","timestamp":1685059200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2046236; 1838222; 1924694; 1954644; 1751392"],"award-info":[{"award-number":["2046236; 1838222; 1924694; 1954644; 1751392"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Manag. Data"],"published-print":{"date-parts":[[2023,5,26]]},"abstract":"<jats:p>The popularity of JSON as a data interchange format resulted in big amounts of datasets available for processing. Users would like to analyze this data using SQL queries but existing distributed systems limit their users to only two specific formats, JSONLine and GeoJSON. The complexity of JSON schema makes it challenging to parse arbitrary files in a modern distributed system while producing records with unified schema that can be processed with SQL. To address these challenges, this paper introduces dsJSON, a state-of-the-art distributed JSON processor that overcomes limitations in existing systems and scales to big and complex data. dsJSON introduces the projection tree, a novel data structure that applies selective parsing of nested attributes to produce records that are ready for SQL processors. The key objective of the projection tree is to parse a big JSON file in parallel to produce records with a unified schema that can be processed with SQL. dsJSON is integrated into SparkSQL which enables users to run arbitrary SQL queries on complex JSON files. It also pushes projection and filter down into the parser for full integration between the parser and the processor. Experiments on up-to two terabytes of real data show that dsJSON performs several times faster than existing systems. It can also efficiently parse extremely large files not supported by existing distributed parsers<\/jats:p>","DOI":"10.1145\/3588957","type":"journal-article","created":{"date-parts":[[2023,5,30]],"date-time":"2023-05-30T17:42:05Z","timestamp":1685468525000},"page":"1-25","source":"Crossref","is-referenced-by-count":0,"title":["dsJSON: A Distributed SQL JSON Processor"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9202-6894","authenticated-orcid":false,"given":"Majid","family":"Saeedan","sequence":"first","affiliation":[{"name":"University of California, Riverside, Riverside, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6584-1455","authenticated-orcid":false,"given":"Ahmed","family":"Eldawy","sequence":"additional","affiliation":[{"name":"University of California, Riverside, Riverside, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2616-4241","authenticated-orcid":false,"given":"Zhijia","family":"Zhao","sequence":"additional","affiliation":[{"name":"University of California, Riverside, Riverside, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,5,30]]},"reference":[{"key":"e_1_2_2_1_1","unstructured":"Json encoder and decoder. Available at https:\/\/docs.python.org\/3\/library\/json.html."},{"key":"e_1_2_2_2_1","unstructured":"MongoDB. Available at https:\/\/www.mongodb.com."},{"key":"e_1_2_2_3_1","first-page":"11","article-title":"A unified engine for big data processing","volume":"59","author":"Apache","year":"2016","unstructured":"Apache spark: A unified engine for big data processing. Commun. ACM 59, 11 (Oct. 2016), 56--65.","journal-title":"Commun. ACM"},{"key":"e_1_2_2_4_1","unstructured":"Bestbuy developer api 2021. Retrieved from https:\/\/bestbuyapis.github.io\/api-documentation\/."},{"key":"e_1_2_2_5_1","unstructured":"Jackson 2021. Available at https:\/\/github.com\/FasterXML\/jackson."},{"key":"e_1_2_2_6_1","unstructured":"Jayway JsonPath 2021. Available at https:\/\/github.com\/json-path\/JsonPath."},{"key":"e_1_2_2_7_1","unstructured":"RapidJSON 2021. Available at https:\/\/rapidjson.org\/."},{"key":"e_1_2_2_8_1","unstructured":"Wikipedia json dumps 2021. Retrieved from https:\/\/dumps.wikimedia.org\/wikidatawiki\/latest\/."},{"key":"e_1_2_2_9_1","first-page":"14","article-title":"Asterixdb: A scalable, open source bdms","volume":"7","author":"Altwaijry S. A. Y. A. H.","year":"2014","unstructured":"Altwaijry, S. A. Y. A. H., Behm, A., Carey, V. B. Y. B. M., Cheelangi, I. C. M., Faraaz, K., Heilbron, E. G. R. G. Z., Vernica, P. P. V. T. R., Wen, J., and Westmann, T. Asterixdb: A scalable, open source bdms. Proceedings of the VLDB Endowment 7, 14 (2014).","journal-title":"Proceedings of the VLDB Endowment"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2742797"},{"key":"e_1_2_2_11_1","volume-title":"Human-in-the-Loop Schema Inference for Massive JSON Datasets. In EDBT 2020 - 23nd International Conference on Extending Database Technology","author":"Baazizi M.-A.","year":"2020","unstructured":"Baazizi, M.-A., Berti, C., Colazzo, D., Ghelli, G., and Sartiani, C. Human-in-the-Loop Schema Inference for Massive JSON Datasets. In EDBT 2020 - 23nd International Conference on Extending Database Technology (Copenhagen, Denmark, Mar. 2020), OpenProceedings.org, pp. 635--638."},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-018-0532-7"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.scico.2015.09.002"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.14778\/3402755.3402761"},{"key":"e_1_2_2_15_1","volume-title":"Imdb review dataset","author":"Biswas E.","year":"2021","unstructured":"Biswas, E. Imdb review dataset, 2021. Retrieved from https:\/\/www.kaggle.com\/dsv\/1836923."},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.14778\/3137765.3137782"},{"key":"e_1_2_2_17_1","volume-title":"Xml path language (xpath)","author":"Clark J.","year":"1999","unstructured":"Clark, J., DeRose, S., et al. Xml path language (xpath), 1999."},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-65847-2_16"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2882924"},{"key":"e_1_2_2_20_1","volume-title":"JSON Tiles: Fast Analytics on Semi-Structured Data","author":"Durner D.","year":"2021","unstructured":"Durner, D., Leis, V., and Neumann, T. JSON Tiles: Fast Analytics on Semi-Structured Data. Association for Computing Machinery, New York, NY, USA, 2021, p. 445--458."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3481897"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3319898"},{"key":"e_1_2_2_23_1","volume-title":"Feb.","author":"Goessner S.","year":"2007","unstructured":"Goessner, S. JSONPath - XPath for JSON, Feb. 2007. Available at https:\/\/goessner.net\/articles\/JsonPath\/."},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3345252.3345285"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.14778\/3436905.3436926"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3297858.3304008"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3155284.3018772"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3503222.3507719"},{"key":"e_1_2_2_29_1","unstructured":"JSON 2021. Available at https:\/\/www.json.org\/."},{"key":"e_1_2_2_30_1","unstructured":"Documentation for the json lines text file format 2021. Available at https:\/\/jsonlines.org."},{"key":"e_1_2_2_31_1","volume-title":"Technologie und Web (BTW 2015)","author":"Klettke M.","year":"2015","unstructured":"Klettke, M., St\u00f6rl, U., and Scherzinger, S. Schema extraction and structural outlier detection for json-based nosql data stores. Datenbanksysteme f\u00fcr Business, Technologie und Web (BTW 2015) (2015)."},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-019-00578-5"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.14778\/3115404.3115416"},{"key":"e_1_2_2_34_1","first-page":"223","volume-title":"Proceedings of the 7th IEEE\/ACM International Conference on Grid Computing (USA, 2006), GRID '06, IEEE Computer Society","author":"Lu W.","unstructured":"Lu, W., Chiu, K., and Pan, Y. A parallel approach to xml parsing. In Proceedings of the 7th IEEE\/ACM International Conference on Grid Computing (USA, 2006), GRID '06, IEEE Computer Society, p. 223--230."},{"key":"e_1_2_2_35_1","first-page":"1","article-title":"Mllib: Machine learning in apache spark","volume":"17","author":"Meng X.","year":"2016","unstructured":"Meng, X., Bradley, J., Yavuz, B., Sparks, E., Venkataraman, S., Liu, D., Freeman, J., Tsai, D., Amde, M., Owen, S., et al. Mllib: Machine learning in apache spark. The Journal of Machine Learning Research 17, 1 (2016), 1235--1241.","journal-title":"The Journal of Machine Learning Research"},{"key":"e_1_2_2_36_1","volume-title":"Computer generated building footprints in all 50 us states","author":"Microsoft","year":"2020","unstructured":"Microsoft. Computer generated building footprints in all 50 us states., 2020. Retrieved from UCR-STAR https:\/\/star.cs.ucr.edu\/?MSBuildings&d."},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.14778\/3236187.3236207"},{"key":"e_1_2_2_38_1","first-page":"576","volume-title":"EDBT","author":"Pavlopoulou C.","year":"2018","unstructured":"Pavlopoulou, C., Carman Jr, E. P., Westmann, T., Carey, M. J., and Tsotras, V. J. A parallel and scalable processor for json data. In EDBT (2018), pp. 576--587."},{"key":"e_1_2_2_39_1","volume-title":"Native JSON Benchmark","author":"Yip M.","year":"2021","unstructured":"Yip, M. Native JSON Benchmark, 2021. Available at https:\/\/github.com\/miloyip\/nativejson-benchmark."},{"key":"e_1_2_2_40_1","volume-title":"Openstreetmap all map points","author":"Zhang Y.","year":"2021","unstructured":"Zhang, Y., and Eldawy, A. Openstreetmap all map points, 2021. Retrieved from UCR-STAR https:\/\/star.cs.ucr.edu\/?osm21\/all_nodes&d."}],"container-title":["Proceedings of the ACM on Management of Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3588957","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3588957","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3588957","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:47:38Z","timestamp":1750178858000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3588957"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,26]]},"references-count":40,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,5,26]]}},"alternative-id":["10.1145\/3588957"],"URL":"https:\/\/doi.org\/10.1145\/3588957","relation":{},"ISSN":["2836-6573"],"issn-type":[{"type":"electronic","value":"2836-6573"}],"subject":[],"published":{"date-parts":[[2023,5,26]]}}}