{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,2,9]],"date-time":"2024-02-09T09:43:48Z","timestamp":1707471828512},"reference-count":43,"publisher":"Association for Computing Machinery (ACM)","issue":"10","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2017,6]]},"abstract":"<jats:p>Existing data analytics systems have approached predictive model training exclusively from a data-parallel perspective. Data examples are partitioned to multiple workers and training is executed concurrently over different partitions, under various synchronization policies that emphasize speedup or convergence. Since models with millions and even billions of features become increasingly common nowadays, model management becomes an equally important task for effective training. In this paper, we present a general framework for parallelizing stochastic optimization algorithms over massive models that cannot fit in memory. We extend the lock-free HOGWILD!-family of algorithms to disk-resident models by vertically partitioning the model offline and asynchronously updating the resulting partitions online. Unlike HOGWILD!, concurrent requests to the common model are minimized by a preemptive push-based sharing mechanism that reduces the number of disk accesses. Experimental results on real and synthetic datasets show that the proposed framework achieves improved convergence over HOGWILD! and is the only solution scalable to massive models.<\/jats:p>","DOI":"10.14778\/3115404.3115405","type":"journal-article","created":{"date-parts":[[2017,9,7]],"date-time":"2017-09-07T13:35:53Z","timestamp":1504791353000},"page":"986-997","source":"Crossref","is-referenced-by-count":6,"title":["Scalable asynchronous gradient descent optimization for out-of-core models"],"prefix":"10.14778","volume":"10","author":[{"given":"Chengjie","family":"Qin","sequence":"first","affiliation":[{"name":"GraphSQL, Inc."}]},{"given":"Martin","family":"Torres","sequence":"additional","affiliation":[{"name":"University of California Merced"}]},{"given":"Florin","family":"Rusu","sequence":"additional","affiliation":[{"name":"University of California Merced"}]}],"member":"320","published-online":{"date-parts":[[2017,6]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.5555\/2627435.2638571"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463676.2465283"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1365815.1365816"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2213836.2213936"},{"key":"e_1_2_1_5_1","volume-title":"OSDI","author":"Chilimbi T.","year":"2014"},{"key":"e_1_2_1_6_1","unstructured":"D. Crankshaw P. Bailis et al. The Missing Piece in Complex Analytics: Low Latency Scalable Model Management and Serving with Velox. CoRR abs\/1409.3809 2014.  D. Crankshaw P. Bailis et al. The Missing Piece in Complex Analytics: Low Latency Scalable Model Management and Serving with Velox. CoRR abs\/1409.3809 2014."},{"key":"e_1_2_1_7_1","unstructured":"A. Crotty etal Tupleware: Redefining Modern Analytics. CoRR abs\/1406.6667 2014.  A. Crotty et al. Tupleware: Redefining Modern Analytics. CoRR abs\/1406.6667 2014."},{"key":"e_1_2_1_8_1","volume-title":"Large Scale Distributed Deep Networks. In NIPS","author":"Dean J.","year":"2012"},{"key":"e_1_2_1_9_1","volume-title":"NIPS","author":"Duchi J.","year":"2013"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2342356.2342360"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2213836.2213874"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2011.5767930"},{"key":"e_1_2_1_13_1","volume-title":"Correlation-Based Feature Selection for Discrete and Numeric Class Machine Learning. In ICML","author":"Hall M.","year":"2000"},{"key":"e_1_2_1_14_1","volume-title":"Cache-Conscious Storage Technique. In VLDB","author":"Hankins R.","year":"2003"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.14778\/2367502.2367510"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.14778\/2536336.2536338"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2935694.2935698"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2723713"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2882952"},{"key":"e_1_2_1_20_1","volume-title":"On Model Parallelization and Scheduling Strategies for Distributed Machine Learning. In NIPS","author":"Lee S.","year":"2015"},{"key":"e_1_2_1_21_1","volume-title":"Parameter Server for Distributed Machine Learning. In Big Learning NIPS Workshop","author":"Li M.","year":"2013"},{"key":"e_1_2_1_22_1","volume-title":"Scaling Distributed Machine Learning with the Parameter Server. In OSDI","author":"Li M.","year":"2014"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2592798.2592820"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2005.66"},{"key":"e_1_2_1_25_1","volume-title":"An Asynchronous Parallel Stochastic Coordinate Descent Algorithm. In ICML","author":"Liu J.","year":"2014"},{"key":"e_1_2_1_26_1","volume-title":"GraphLab: A New Parallel Framework for Machine Learning. In UAI","author":"Low Y.","year":"2010"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2488200"},{"key":"e_1_2_1_28_1","volume-title":"NIPS","author":"Niu F.","year":"2011"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/s002360050048"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2016.2545664"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2799562.2799563"},{"key":"e_1_2_1_32_1","unstructured":"C. Qin and F. Rusu. Dot-Product Join: An Array-Relation Join Operator for Big Model Analytics. CoRR abs\/1602.08845 2016.  C. Qin and F. Rusu. Dot-Product Join: An Array-Relation Join Operator for Big Model Analytics. CoRR abs\/1602.08845 2016."},{"key":"e_1_2_1_33_1","volume-title":"Factoring Non-Negative Matrices with Linear Programs. In NIPS","author":"Recht B.","year":"2012"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2882948"},{"key":"e_1_2_1_35_1","volume-title":"High Performance Parallel Stochastic Gradient Descent in Shared Memory. In IPDPS","author":"Sallinen S.","year":"2016"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2882939"},{"key":"e_1_2_1_37_1","volume-title":"MLI: An API for Distributed Machine Learning. In ICDM","author":"Sparks E.","year":"2013"},{"key":"e_1_2_1_38_1","volume-title":"OptiML: An Implicitly Parallel Domain-Specific Language for Machine Learning. In ICML","author":"Sujeeth A.","year":"2011"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3147.3165"},{"key":"e_1_2_1_40_1","unstructured":"P. Xinghao M. Lam et al. CYCLADES: Conflict-free Asynchronous Machine Learning. CoRR abs\/1605.09721 2016.  P. Xinghao M. Lam et al. CYCLADES: Conflict-free Asynchronous Machine Learning. CoRR abs\/1605.09721 2016."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2877204"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2791347.2791369"},{"key":"e_1_2_1_43_1","volume-title":"Parallelized Stochastic Gradient Descent. In NIPS","author":"Zinkevich M.","year":"2010"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3115404.3115405","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T09:49:11Z","timestamp":1672220951000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3115404.3115405"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,6]]},"references-count":43,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2017,6]]}},"alternative-id":["10.14778\/3115404.3115405"],"URL":"https:\/\/doi.org\/10.14778\/3115404.3115405","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2017,6]]}}}