{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,28]],"date-time":"2026-01-28T04:19:02Z","timestamp":1769573942231,"version":"3.49.0"},"reference-count":85,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2024,4,12]],"date-time":"2024-04-12T00:00:00Z","timestamp":1712880000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,4,12]],"date-time":"2024-04-12T00:00:00Z","timestamp":1712880000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Swiss Federal Institute of Technology Zurich"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["The VLDB Journal"],"published-print":{"date-parts":[[2024,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Modern machine learning (ML) systems commonly use stochastic gradient descent (SGD) to train ML models. However, SGD relies on random data order to converge, which usually requires a full data shuffle. For in-DB ML systems and deep learning systems with large datasets stored on<jats:italic>block-addressable secondary storage<\/jats:italic>such as HDD and SSD, this full data shuffle leads to low I\/O performance\u2014the data shuffling time can be even longer than the training itself, due to massive random data accesses. To balance the convergence rate of SGD (which favors data randomness) and its I\/O performance (which favors sequential access), previous work has proposed several data shuffling strategies. In this paper, we first perform an empirical study on existing data shuffling strategies, showing that these strategies suffer from either low performance or low convergence rate. To solve this problem, we propose a simple but novel<jats:italic>two-level<\/jats:italic>data shuffling strategy named , which can<jats:italic>avoid<\/jats:italic>a full data shuffle while maintaining<jats:italic>comparable<\/jats:italic>convergence rate of SGD as if a full shuffle were performed. We further theoretically analyze the convergence behavior of and empirically evaluate its efficacy in both in-DB ML and deep learning systems. For in-DB ML systems, we integrate into PostgreSQL by introducing three new<jats:italic>physical<\/jats:italic>operators with optimizations. For deep learning systems, we extend single-process to multi-process for the parallel\/distributed environment and integrate it into PyTorch. Our evaluation shows that can achieve comparable convergence rate with the full-shuffle-based SGD for both linear models and deep learning models. For in-DB ML with linear models, is 1.6<jats:inline-formula><jats:alternatives><jats:tex-math>$$\\times $$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\"><mml:mo>\u00d7<\/mml:mo><\/mml:math><\/jats:alternatives><\/jats:inline-formula><jats:inline-formula><jats:alternatives><jats:tex-math>$$-$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\"><mml:mo>-<\/mml:mo><\/mml:math><\/jats:alternatives><\/jats:inline-formula>12.8<jats:inline-formula><jats:alternatives><jats:tex-math>$$\\times $$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\"><mml:mo>\u00d7<\/mml:mo><\/mml:math><\/jats:alternatives><\/jats:inline-formula>faster than two state-of-the-art systems, Apache MADlib and Bismarck, on both HDD and SSD. For deep learning models on ImageNet, is 1.5<jats:inline-formula><jats:alternatives><jats:tex-math>$$\\times $$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\"><mml:mo>\u00d7<\/mml:mo><\/mml:math><\/jats:alternatives><\/jats:inline-formula>faster than PyTorch with full data shuffle.<\/jats:p>","DOI":"10.1007\/s00778-024-00845-0","type":"journal-article","created":{"date-parts":[[2024,4,12]],"date-time":"2024-04-12T13:01:45Z","timestamp":1712926905000},"page":"1231-1255","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Stochastic gradient descent without full data shuffle: with applications to in-database machine learning and deep learning systems"],"prefix":"10.1007","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-2013-4241","authenticated-orcid":false,"given":"Lijie","family":"Xu","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shuang","family":"Qiu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Binhang","family":"Yuan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiawei","family":"Jiang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Cedric","family":"Renggli","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shaoduo","family":"Gan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kaan","family":"Kara","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guoliang","family":"Li","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ji","family":"Liu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wentao","family":"Wu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jieping","family":"Ye","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ce","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,4,12]]},"reference":[{"key":"845_CR1","unstructured":"Buffer Manager of PostgreSQL. https:\/\/www.interdb.jp\/pg\/pgsql08.html (2022)"},{"key":"845_CR2","unstructured":"Abadi, M., Barham, P., Chen, J., et\u00a0al.: Tensorflow: A system for large-scale machine learning. In: 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, pp. 265\u2013283. USENIX Association (2016)"},{"key":"845_CR3","unstructured":"Paszke, A., Gross, S., Massa, F., et\u00a0al.: Pytorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, pp. 8024\u20138035 (2019)"},{"issue":"12","key":"845_CR4","doi-asserted-by":"publisher","first-page":"1700","DOI":"10.14778\/2367502.2367510","volume":"5","author":"JM Hellerstein","year":"2012","unstructured":"Hellerstein, J.M., R\u00e9, C., Schoppmann, F., Wang, D.Z., Fratkin, E., Gorajek, A., Ng, K.S., Welton, C., Feng, X., Li, K., Kumar, A.: The madlib analytics library or MAD skills, the SQL. Proc. VLDB Endow. 5(12), 1700\u20131711 (2012)","journal-title":"Proc. VLDB Endow."},{"key":"845_CR5","doi-asserted-by":"crossref","unstructured":"Feng, X., Kumar, A., Recht, B., R\u00e9, C.: Towards a unified architecture for in-rdbms analytics. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, SIGMOD \u201912, pp. 325\u2013336 (2012)","DOI":"10.1145\/2213836.2213874"},{"key":"845_CR6","doi-asserted-by":"crossref","unstructured":"Schleich, M., Olteanu, D., Ciucanu, R.: Learning linear regression models over factorized joins. In: Proceedings of the 2016 International Conference on Management of Data, SIGMOD Conference 2016, pp. 3\u201318. ACM (2016)","DOI":"10.1145\/2882903.2882939"},{"key":"845_CR7","doi-asserted-by":"crossref","unstructured":"Kumar, A., Naughton, J.F., Patel, J.M.: Learning generalized linear models over normalized data. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pp. 1969\u20131984. ACM (2015)","DOI":"10.1145\/2723372.2723713"},{"issue":"13","key":"845_CR8","doi-asserted-by":"publisher","first-page":"1573","DOI":"10.14778\/3007263.3007312","volume":"9","author":"D Olteanu","year":"2016","unstructured":"Olteanu, D., Schleich, M.: F: regression models over factorized views. Proc. VLDB Endow. 9(13), 1573\u20131576 (2016)","journal-title":"Proc. VLDB Endow."},{"issue":"11","key":"845_CR9","doi-asserted-by":"publisher","first-page":"1214","DOI":"10.14778\/3137628.3137633","volume":"10","author":"L Chen","year":"2017","unstructured":"Chen, L., Kumar, A., Naughton, J.F., Patel, J.M.: Towards linear algebra over normalized data. Proc. VLDB Endow. 10(11), 1214\u20131225 (2017)","journal-title":"Proc. VLDB Endow."},{"issue":"5","key":"845_CR10","doi-asserted-by":"publisher","first-page":"337","DOI":"10.14778\/2535573.2488340","volume":"6","author":"S Rendle","year":"2013","unstructured":"Rendle, S.: Scaling factorization machines to relational data. Proc. VLDB Endow. 6(5), 337\u2013348 (2013)","journal-title":"Proc. VLDB Endow."},{"key":"845_CR11","doi-asserted-by":"crossref","unstructured":"Khamis, M.A., Ngo, H.Q., Nguyen, X., Olteanu, D., Schleich, M.: In-database learning with sparse tensors. In: Proceedings of the 37th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems, pp. 325\u2013340. ACM (2018)","DOI":"10.1145\/3196959.3196960"},{"issue":"8","key":"845_CR12","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1145\/3405470","volume":"63","author":"S Luo","year":"2020","unstructured":"Luo, S., Gao, Z.J., Gubanov, M.N., Perez, L.L., Jankov, D., Jermaine, C.M.: Scalable linear algebra on a relational database system. Commun. ACM 63(8), 93\u2013101 (2020)","journal-title":"Commun. ACM"},{"key":"845_CR13","unstructured":"Apache MADlib: big data machine learning in SQL. http:\/\/madlib.apache.org\/"},{"key":"845_CR14","doi-asserted-by":"crossref","unstructured":"Zhang, C., R\u00e9, C.: Dimmwitted: a study of main-memory statistical analytics. Proc. VLDB Endow. 7(12) (2014)","DOI":"10.14778\/2732977.2733001"},{"issue":"4","key":"845_CR15","doi-asserted-by":"publisher","first-page":"348","DOI":"10.14778\/3297753.3297756","volume":"12","author":"K Kara","year":"2018","unstructured":"Kara, K., Eguro, K., Zhang, C., Alonso, G.: Columnml: column-store machine learning with on-the-fly data transformation. Proc. VLDB Endow. 12(4), 348\u2013361 (2018)","journal-title":"Proc. VLDB Endow."},{"key":"845_CR16","unstructured":"Petersen, T.K.: Inside the lustre file system. SEAGATE Technology paper (2015)"},{"key":"845_CR17","unstructured":"Sliding-Window Shuffle in TensorFlow. https:\/\/www.tensorflow.org\/api_docs\/python\/tf\/data\/Dataset"},{"key":"845_CR18","unstructured":"PostgreSQL. https:\/\/www.postgresql.org\/"},{"issue":"1","key":"845_CR19","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1109\/69.273032","volume":"6","author":"G Graefe","year":"1994","unstructured":"Graefe, G.: Volcano\u2014an extensible and parallel query evaluation system. IEEE Trans. Knowl. Data Eng. 6(1), 120\u2013135 (1994)","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"845_CR20","doi-asserted-by":"crossref","unstructured":"Xu, L., Qiu, S., Yuan, B., Jiang, J., Renggli, C., Gan, S., Kara, K., Li, G., Liu, J., Wu, W., Ye, J., Zhang, C.: In-database machine learning with corgipile: stochastic gradient descent without full data shuffle. In: SIGMOD \u201922: International Conference on Management of Data, Philadelphia, PA, USA, June 12\u201317, 2022, pp. 1286\u20131300. ACM (2022)","DOI":"10.1145\/3514221.3526150"},{"key":"845_CR21","unstructured":"ImageNet Dataset. https:\/\/www.image-net.org\/"},{"key":"845_CR22","doi-asserted-by":"crossref","unstructured":"Bottou, L.: Stochastic gradient descent tricks. In: Neural Networks: Tricks of the Trade, pp. 421\u2013436. Springer (2012)","DOI":"10.1007\/978-3-642-35289-8_25"},{"key":"845_CR23","first-page":"5957","volume":"33","author":"CM De Sa","year":"2020","unstructured":"De Sa, C.M.: Random reshuffling is not always better. Adv. Neural Inf. Process. Syst. 33, 5957\u20135967 (2020)","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"845_CR24","unstructured":"Yun, C., Sra, S., Jadbabaie, A.: Open problem: Can single-shuffle SGD be better than reshuffling SGD and gd? In: Conference on Learning Theory, COLT 2021, Proceedings of Machine Learning Research, vol. 134, pp. 4653\u20134658. PMLR (2021)"},{"issue":"7","key":"845_CR25","doi-asserted-by":"publisher","first-page":"1228","DOI":"10.14778\/3450980.3450991","volume":"14","author":"D Jankov","year":"2021","unstructured":"Jankov, D., Yuan, B., Luo, S., Jermaine, C.: Distributed numerical and machine learning computations via two-phase execution of aggregated join trees. Proc. VLDB Endow. 14(7), 1228\u20131240 (2021)","journal-title":"Proc. VLDB Endow."},{"key":"845_CR26","doi-asserted-by":"crossref","unstructured":"Luo, S., Jankov, D., Yuan, B., Jermaine, C.: Automatic optimization of matrix implementations for distributed machine learning and linear algebra. In: Proceedings of the 2021 International Conference on Management of Data, pp. 1222\u20131234 (2021)","DOI":"10.1145\/3448016.3457317"},{"issue":"8","key":"845_CR27","doi-asserted-by":"publisher","first-page":"1338","DOI":"10.14778\/3457390.3457399","volume":"14","author":"B Yuan","year":"2021","unstructured":"Yuan, B., Jankov, D., Zou, J., Tang, Y., Bourgeois, D., Jermaine, C.: Tensor relational algebra for distributed machine learning system design. Proc. VLDB Endow. 14(8), 1338\u20131350 (2021)","journal-title":"Proc. VLDB Endow."},{"key":"845_CR28","doi-asserted-by":"crossref","unstructured":"Jasny, M., Ziegler, T., Kraska, T., R\u00f6hm, U., Binnig, C.: DB4ML\u2014an in-memory database kernel with machine learning support. In: Proceedings of the 2020 International Conference on Management of Data, SIGMOD Conference 2020, pp. 159\u2013173. ACM (2020)","DOI":"10.1145\/3318464.3380575"},{"key":"845_CR29","unstructured":"Ruder, S.: An overview of gradient descent optimization algorithms. CoRR abs\/1609.04747 (2016)"},{"key":"845_CR30","unstructured":"The optimization algorithms in PyTorch. https:\/\/pytorch.org\/docs\/stable\/optim.html"},{"key":"845_CR31","unstructured":"The optimization algorithms in TensorFlow. https:\/\/www.tensorflow.org\/api_docs\/python\/tf\/keras\/optimizers"},{"key":"845_CR32","unstructured":"Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: 3rd International Conference on Learning Representations, ICLR 2015 (2015)"},{"key":"845_CR33","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, pp. 770\u2013778 (2016)","DOI":"10.1109\/CVPR.2016.90"},{"key":"845_CR34","unstructured":"LIBSVM Data. https:\/\/www.csie.ntu.edu.tw\/~cjlin\/libsvmtools\/datasets\/"},{"key":"845_CR35","unstructured":"CIFAR-10 Dataset. http:\/\/www.cs.toronto.edu\/~kriz\/cifar.html"},{"key":"845_CR36","unstructured":"Arpaci-Dusseau, R.H., Arpaci-Dusseau, A.C.: Operating Systems: Three Easy Pieces, 1.00 edn. Arpaci-Dusseau Books (2018)"},{"key":"845_CR37","unstructured":"PyTorch\u2019s shuffling strategy. https:\/\/pytorch.org\/data\/main\/generated\/torchdata.datapipes.iter.Shuffler.html (2024)"},{"key":"845_CR38","unstructured":"Bottou, L.: Curiously fast convergence of some stochastic gradient descent algorithms. In: Proceedings of the Symposium on Learning and Data Science, Paris, vol.\u00a08, pp. 2624\u20132633 (2009)"},{"key":"845_CR39","doi-asserted-by":"crossref","unstructured":"G\u00fcrb\u00fczbalaban, M., Ozdaglar, A., Parrilo, P.A.: Why random reshuffling beats stochastic gradient descent. Math. Program. pp. 1\u201336 (2019)","DOI":"10.1007\/s10107-019-01440-w"},{"key":"845_CR40","unstructured":"HaoChen, J.Z., Sra, S.: Random shuffling beats SGD after finite epochs. In: Proceedings of the 36th International Conference on Machine Learning, ICML 2019. In: Proceedings of Machine Learning Research, vol.\u00a097, pp. 2624\u20132633. PMLR (2019)"},{"key":"845_CR41","unstructured":"Shamir, O.: Without-replacement sampling for stochastic gradient methods. In: Advances in Neural Information Processing Systems, pp. 46\u201354 (2016)"},{"issue":"1","key":"845_CR42","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1007\/s10107-019-01440-w","volume":"186","author":"M G\u00fcrb\u00fczbalaban","year":"2021","unstructured":"G\u00fcrb\u00fczbalaban, M., Ozdaglar, A.E., Parrilo, P.A.: Why random reshuffling beats stochastic gradient descent. Math. Program. 186(1), 49\u201384 (2021)","journal-title":"Math. Program."},{"issue":"2","key":"845_CR43","doi-asserted-by":"publisher","first-page":"474","DOI":"10.1109\/TSP.2018.2878551","volume":"67","author":"B Ying","year":"2019","unstructured":"Ying, B., Yuan, K., Vlaski, S., Sayed, A.H.: Stochastic learning under random reshuffling with constant step-sizes. IEEE Trans. Signal Process. 67(2), 474\u2013489 (2019)","journal-title":"IEEE Trans. Signal Process."},{"issue":"2","key":"845_CR44","doi-asserted-by":"publisher","first-page":"223","DOI":"10.1137\/16M1080173","volume":"60","author":"L Bottou","year":"2018","unstructured":"Bottou, L., Curtis, F.E., Nocedal, J.: Optimization methods for large-scale machine learning. SIAM Rev. 60(2), 223\u2013311 (2018)","journal-title":"SIAM Rev."},{"issue":"1","key":"845_CR45","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1561\/1900000062","volume":"9","author":"J Liu","year":"2020","unstructured":"Liu, J., Zhang, C.: Distributed learning systems with first-order methods. Found. Trends Databases 9(1), 1\u2013100 (2020)","journal-title":"Found. Trends Databases"},{"key":"845_CR46","doi-asserted-by":"publisher","unstructured":"Xu, L., Qiu, S., Yuan, B., Jiang, J., Renggli, C., Gan, S., Kara, K., Li, G., Liu, J., Wu, W., Ye, J., Zhang, C.: Stochastic gradient descent without full data shuffle. CoRR abs\/2206.05830 (2022). https:\/\/doi.org\/10.48550\/arXiv.2206.05830","DOI":"10.48550\/arXiv.2206.05830"},{"key":"845_CR47","unstructured":"Getting Started With Distributed Data Parallel. https:\/\/pytorch.org\/tutorials\/intermediate\/ddp_tutorial.html"},{"key":"845_CR48","unstructured":"Hadoop HDFS Architecture. https:\/\/hadoop.apache.org\/docs\/current\/hadoop-project-dist\/hadoop-hdfs\/HdfsDesign.html"},{"key":"845_CR49","unstructured":"Amazon Elastic Block Store (EBS). https:\/\/aws.amazon.com\/ebs"},{"key":"845_CR50","unstructured":"The Lustre file system. https:\/\/www.lustre.org\/"},{"key":"845_CR51","unstructured":"ETH Euler Cluster. https:\/\/scicomp.ethz.ch\/wiki\/Euler"},{"key":"845_CR52","unstructured":"Lustre reads\/writes data in blocks. https:\/\/scicomp.ethz.ch\/wiki\/Conda"},{"key":"845_CR53","unstructured":"The TFRecord format for storing a sequence of binary records. https:\/\/www.tensorflow.org\/tutorials\/load_data\/tfrecord"},{"key":"845_CR54","unstructured":"TFRecord format for PyTorch. https:\/\/github.com\/vahidk\/tfrecord"},{"key":"845_CR55","unstructured":"MADlib-Deep Learning. https:\/\/madlib.apache.org\/docs\/latest\/group__grp__keras.html (2024)"},{"issue":"2","key":"845_CR56","doi-asserted-by":"publisher","first-page":"64","DOI":"10.1145\/2812802","volume":"59","author":"B Thomee","year":"2016","unstructured":"Thomee, B., Shamma, D.A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., Li, L.: YFCC100M: the new data in multimedia research. Commun. ACM 59(2), 64\u201373 (2016)","journal-title":"Commun. ACM"},{"key":"845_CR57","unstructured":"ImageNet training in PyTorch. https:\/\/github.com\/pytorch\/examples\/tree\/main\/imagenet"},{"key":"845_CR58","unstructured":"PostgreSQL TOAST. https:\/\/www.postgresql.org\/docs\/9.5\/storage-toast.html"},{"key":"845_CR59","unstructured":"Epsilon Dataset. https:\/\/www.k4all.org\/project\/large-scale-learning-challenge\/"},{"key":"845_CR60","unstructured":"Bandwidth Bench. https:\/\/github.com\/JerryLead\/bandwidth-bench (2024)"},{"key":"845_CR61","unstructured":"Best practices on Lustre parallel file systems. https:\/\/scicomp.ethz.ch\/wiki\/Best_practices_on_Lustre_parallel_file_systems"},{"key":"845_CR62","doi-asserted-by":"crossref","unstructured":"Bottou, L.: Large-scale machine learning with stochastic gradient descent. In: 19th International Conference on Computational Statistics, COMPSTAT 2010, pp. 177\u2013186 (2010)","DOI":"10.1007\/978-3-7908-2604-3_16"},{"key":"845_CR63","unstructured":"Moulines, E., Bach, F.R.: Non-asymptotic analysis of stochastic approximation algorithms for machine learning. In: Advances in Neural Information Processing Systems, pp. 451\u2013459 (2011)"},{"issue":"4","key":"845_CR64","doi-asserted-by":"publisher","first-page":"2341","DOI":"10.1137\/120880811","volume":"23","author":"S Ghadimi","year":"2013","unstructured":"Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM J. Optim. 23(4), 2341\u20132368 (2013)","journal-title":"SIAM J. Optim."},{"issue":"4","key":"845_CR65","first-page":"643","volume":"3","author":"BT Polyak","year":"1963","unstructured":"Polyak, B.T.: Gradient methods for minimizing functionals. Zhurnal Vychislitel\u2019noi Matematiki i Matematicheskoi Fiziki 3(4), 643\u2013653 (1963)","journal-title":"Zhurnal Vychislitel\u2019noi Matematiki i Matematicheskoi Fiziki"},{"key":"845_CR66","doi-asserted-by":"crossref","unstructured":"Haas, P.J., Koenig, C.: A bi-level Bernoulli scheme for database sampling. In: Proceedings of the ACM SIGMOD International Conference on Management of Data, 2004, pp. 275\u2013286. ACM (2004)","DOI":"10.1145\/1007568.1007601"},{"key":"845_CR67","doi-asserted-by":"crossref","unstructured":"Cheng, Y., Zhao, W., Rusu, F.: Bi-level online aggregation on raw data. In: Proceedings of the 29th International Conference on Scientific and Statistical Database Management, Chicago, IL, USA, June 27\u201329, 2017, pp. 10:1\u201310:12. ACM (2017)","DOI":"10.1145\/3085504.3085514"},{"key":"845_CR68","volume-title":"Predictive Analysis with SAP","author":"J MacGregor","year":"2013","unstructured":"MacGregor, J.: Predictive Analysis with SAP. Galileo Press, Bonn (2013)"},{"key":"845_CR69","unstructured":"Oracle R Enterprise Versions of R Models. https:\/\/docs.oracle.com\/cd\/E11882_01\/doc.112\/e36761\/orelm.htm"},{"key":"845_CR70","unstructured":"The Greenplum MADlib extension. https:\/\/greenplum.docs.pivotal.io\/6-19\/analytics\/madlib.html"},{"key":"845_CR71","doi-asserted-by":"crossref","unstructured":"Fard, A., Le, A., Larionov, G., Dhillon, W., Bear, C.: Vertica-ml: distributed machine learning in vertica database. In: Proceedings of the 2020 International Conference on Management of Data, SIGMOD Conference 2020, pp. 755\u2013768. ACM (2020)","DOI":"10.1145\/3318464.3386137"},{"key":"845_CR72","unstructured":"Google BigQuery ML. https:\/\/cloud.google.com\/bigquery-ml\/docs\/introduction"},{"key":"845_CR73","unstructured":"Microsoft SQL Server Machine Learning Services. https:\/\/docs.microsoft.com\/en-us\/sql\/machine-learning\/sql-server-machine-learning-services?view=sql-server-ver15"},{"issue":"4","key":"845_CR74","first-page":"1","volume":"1","author":"T Chen","year":"2015","unstructured":"Chen, T., He, T., Benesty, M., Khotilovich, V., Tang, Y., Cho, H., et al.: Xgboost: extreme gradient boosting. R Package Version 0.4\u20132 1(4), 1\u20134 (2015)","journal-title":"R Package Version 0.4\u20132"},{"key":"845_CR75","first-page":"3146","volume":"30","author":"G Ke","year":"2017","unstructured":"Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., Liu, T.Y.: Lightgbm: a highly efficient gradient boosting decision tree. Adv. Neural Inf. Process. Syst. 30, 3146\u20133154 (2017)","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"845_CR76","first-page":"34:1","volume":"17","author":"X Meng","year":"2016","unstructured":"Meng, X., Bradley, J.K., Yavuz, B., et al.: Mllib: machine learning in apache spark. J. Mach. Learn. Res. 17, 34:1-34:7 (2016)","journal-title":"J. Mach. Learn. Res."},{"key":"845_CR77","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Jiang, J., Wu, W., Zhang, C., Yu, L., Cui, B.: Mllib*: fast training of glms using spark mllib. In: 35th IEEE International Conference on Data Engineering, ICDE 2019, pp. 1778\u20131789. IEEE (2019)","DOI":"10.1109\/ICDE.2019.00194"},{"key":"845_CR78","doi-asserted-by":"crossref","unstructured":"Cai, Z., Vagena, Z., Perez, L.L., Arumugam, S., Haas, P.J., Jermaine, C.M.: Simulation of database-valued Markov chains using simsql. In: Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2013, pp. 637\u2013648. ACM (2013)","DOI":"10.1145\/2463676.2465283"},{"key":"845_CR79","first-page":"1223","volume":"25","author":"J Dean","year":"2012","unstructured":"Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P., Yang, K., et al.: Large scale distributed deep networks. Adv. Neural Inf. Process. Syst. 25, 1223\u20131231 (2012)","journal-title":"Adv. Neural Inf. Process. Syst."},{"issue":"2","key":"845_CR80","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1109\/TBDATA.2015.2472014","volume":"1","author":"EP Xing","year":"2015","unstructured":"Xing, E.P., Ho, Q., Dai, W., Kim, J.K., Wei, J., Lee, S., Zheng, X., Xie, P., Kumar, A., Yu, Y.: Petuum: a new platform for distributed machine learning on big data. IEEE Trans. Big Data 1(2), 49\u201367 (2015)","journal-title":"IEEE Trans. Big Data"},{"key":"845_CR81","doi-asserted-by":"crossref","unstructured":"Jiang, J., Cui, B., Zhang, C., Yu, L.: Heterogeneity-aware distributed parameter servers. In: Proceedings of the 2017 ACM International Conference on Management of Data, pp. 463\u2013478 (2017)","DOI":"10.1145\/3035918.3035933"},{"key":"845_CR82","unstructured":"Lian, X., Zhang, C., Zhang, H., Hsieh, C.J., Zhang, W., Liu, J.: Can decentralized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 5336\u20135346 (2017)"},{"key":"845_CR83","unstructured":"Tang, H., Lian, X., Yan, M., Zhang, C., Liu, J.: D$${}^{\\text{2}}$$: Decentralized training over decentralized data. In: Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Proceedings of Machine Learning Research, vol.\u00a080, pp. 4855\u20134863. PMLR (2018)"},{"issue":"10","key":"845_CR84","doi-asserted-by":"publisher","first-page":"1769","DOI":"10.14778\/3467861.3467867","volume":"14","author":"Y Zhang","year":"2021","unstructured":"Zhang, Y., Mcquillan, F., Jayaram, N., Kak, N., Khanna, E., Kislal, O., Valdano, D., Kumar, A.: Distributed deep learning on data systems: a comparative analysis of approaches. Proc. VLDB Endow. 14(10), 1769\u20131782 (2021)","journal-title":"Proc. VLDB Endow."},{"key":"845_CR85","doi-asserted-by":"crossref","unstructured":"Nakandala, S., Zhang, Y., Kumar, A.: Cerebro: a data system for optimized deep learning model selection. Proc. VLDB Endow. 13(11), 2159\u20132173 (2020)","DOI":"10.14778\/3407790.3407816"}],"container-title":["The VLDB Journal"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00778-024-00845-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00778-024-00845-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00778-024-00845-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,16]],"date-time":"2024-11-16T04:16:22Z","timestamp":1731730582000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00778-024-00845-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,12]]},"references-count":85,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,9]]}},"alternative-id":["845"],"URL":"https:\/\/doi.org\/10.1007\/s00778-024-00845-0","relation":{},"ISSN":["1066-8888","0949-877X"],"issn-type":[{"value":"1066-8888","type":"print"},{"value":"0949-877X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,12]]},"assertion":[{"value":"19 July 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 February 2024","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 February 2024","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 April 2024","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}