{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T06:47:02Z","timestamp":1782802022076,"version":"3.54.5"},"reference-count":65,"publisher":"Association for Computing Machinery (ACM)","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:p>\n            In modern Machine Learning, model training is an iterative, experimental process that can consume enormous computation resources and developer time. To aid in that process, experienced model developers log and visualize program variables during training runs. Exhaustive logging of all variables is infeasible, so developers are left to choose between slowing down training via extensive\n            <jats:italic>conservative<\/jats:italic>\n            logging, or letting training run fast via minimalist\n            <jats:italic>optimistic<\/jats:italic>\n            logging that may omit key information. As a compromise, optimistic logging can be accompanied by program checkpoints; this allows developers to add log statements post-hoc, and \"replay\" desired log statements from checkpoint---a process we refer to as\n            <jats:italic>hindsight<\/jats:italic>\n            logging. Unfortunately, hindsight logging raises tricky problems in data management and software engineering. Done poorly, hindsight logging can waste resources and generate technical debt embodied in multiple variants of training code. In this paper, we present methodologies for efficient and effective logging practices for model training, with a focus on techniques for hindsight logging. Our goal is for experienced model developers to learn and adopt these practices. To make this easier, we provide an open-source suite of tools for Fast Low-Overhead Recovery (flor) that embodies our design across three tasks: (i) efficient background logging in Python, (ii) adaptive periodic checkpointing, and (iii) an instrumentation library that codifies hindsight logging for efficient and automatic record-replay of model-training. Model developers can use each flor tool separately as they see fit, or they can use flor in hands-free mode, entrusting it to instrument their code end-to-end for efficient record-replay. Our solutions leverage techniques from physiological transaction logs and recovery in database systems. Evaluations on modern ML benchmarks demonstrate that flor can produce fast checkpointing with small user-specifiable overheads (e.g. 7%), and still provide hindsight log replay times orders of magnitude faster than restarting training from scratch.\n          <\/jats:p>","DOI":"10.14778\/3436905.3436925","type":"journal-article","created":{"date-parts":[[2021,2,22]],"date-time":"2021-02-22T17:23:50Z","timestamp":1614014630000},"page":"682-693","source":"Crossref","is-referenced-by-count":5,"title":["Hindsight logging for model training"],"prefix":"10.14778","volume":"14","author":[{"given":"Rolando","family":"Garcia","sequence":"first","affiliation":[{"name":"UC Berkeley"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eric","family":"Liu","sequence":"additional","affiliation":[{"name":"UC Berkeley"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vikram","family":"Sreekanti","sequence":"additional","affiliation":[{"name":"UC Berkeley"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bobby","family":"Yan","sequence":"additional","affiliation":[{"name":"UC Berkeley"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anusha","family":"Dandamudi","sequence":"additional","affiliation":[{"name":"UC Berkeley"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joseph E.","family":"Gonzalez","sequence":"additional","affiliation":[{"name":"UC Berkeley"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joseph M.","family":"Hellerstein","sequence":"additional","affiliation":[{"name":"UC Berkeley"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Koushik","family":"Sen","sequence":"additional","affiliation":[{"name":"UC Berkeley"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,2,22]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2020. Google PAIR. https:\/\/research.google\/teams\/brain\/pair\/.  2020. Google PAIR. https:\/\/research.google\/teams\/brain\/pair\/."},{"key":"e_1_2_1_2_1","unstructured":"2020. PyTorch Documentation. pytorch.org\/docs.  2020. PyTorch Documentation. pytorch.org\/docs."},{"key":"e_1_2_1_3_1","unstructured":"2020. Weights & Biases. wandb.com.  2020. Weights & Biases. wandb.com."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1629575.1629594"},{"key":"e_1_2_1_5_1","volume-title":"Concrete problems in AI safety. arXiv preprint arXiv:1606.06565","author":"Amodei Dario","year":"2016"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1297081.1297091"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3373376.3378458"},{"key":"e_1_2_1_8_1","volume-title":"Aggressive type inference. language 1050","author":"Aycock John","year":"2000"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098021"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3133875"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.5555\/2480856"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.5555\/3291168.3291171"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.5555\/2999134.2999271"},{"key":"e_1_2_1_14_1","volume-title":"KDD CMI Workshop","volume":"114","author":"Garcia Rolando","year":"2018"},{"key":"e_1_2_1_15_1","unstructured":"Google. 2020. TensorBoard. tensorflow.org\/tensorboard.  Google. 2020. TensorBoard. tensorflow.org\/tensorboard."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.5555\/573304"},{"key":"e_1_2_1_17_1","volume-title":"Ground: A Data Context Service.. In CIDR.","author":"Hellerstein Joseph M","year":"2017"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1142\/S0218488598000094"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.5555\/1862659.1862665"},{"key":"e_1_2_1_20_1","volume-title":"Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146","author":"Howard Jeremy","year":"2018"},{"key":"e_1_2_1_21_1","volume-title":"OrpheusDB: bolt-on versioning for relational databases. arXiv preprint arXiv:1703.02475","author":"Huang Silu","year":"2017"},{"key":"e_1_2_1_22_1","volume-title":"SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and &lt","author":"Iandola Forrest N","year":"2016"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/69.755613"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2833157.2833162"},{"key":"e_1_2_1_25_1","volume-title":"Demystifying a dark art: Understanding real-world machine learning model development. arXiv preprint arXiv:2005.01520","author":"Lee Angela","year":"2020"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.5555\/3291168.3291213"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3386126"},{"key":"e_1_2_1_28_1","volume-title":"Low Overhead Materialization with Flor. Master's thesis. EECS Department","author":"Liu Eric","year":"2020"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2598831"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2009.56"},{"key":"e_1_2_1_31_1","volume-title":"Dying relu and initialization: Theory and numerical examples. arXiv preprint arXiv:1903.06733","author":"Lu Lu","year":"2019"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.5555\/3295222.3295230"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.14778\/2947618.2947619"},{"key":"e_1_2_1_34_1","unstructured":"Peter Mattson Christine Cheng Cody Coleman Greg Diamos Paulius Micikevicius David Patterson Hanlin Tang Gu-Yeon Wei Peter Bailis Victor Bittorf etal 2019. Mlperf training benchmark. arXiv preprint arXiv:1910.01500 (2019).  Peter Mattson Christine Cheng Cody Coleman Greg Diamos Paulius Micikevicius David Patterson Hanlin Tang Gu-Yeon Wei Peter Bailis Victor Bittorf et al. 2019. Mlperf training benchmark. arXiv preprint arXiv:1910.01500 (2019)."},{"key":"e_1_2_1_35_1","first-page":"26","article-title":"ProvDB: Provenance-enabled Lifecycle Management of Collaborative Data Analysis Workflows","volume":"41","author":"Miao Hui","year":"2018","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2017.192"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/128765.128770"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2749436"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/SCAM.2015.7335406"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.23915\/distill.00010"},{"key":"e_1_2_1_41_1","unstructured":"Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury Gregory Chanan Trevor Killeen Zeming Lin Natalia Gimelshein Luca Antiga etal 2019. PyTorch: An imperative style high-performance deep learning library. In Advances in Neural Information Processing Systems. 8024--8035.  Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury Gregory Chanan Trevor Killeen Zeming Lin Natalia Gimelshein Luca Antiga et al. 2019. PyTorch: An imperative style high-performance deep learning library. In Advances in Neural Information Processing Systems. 8024--8035."},{"key":"e_1_2_1_42_1","volume-title":"Sysml: The new frontier of machine learning systems. arXiv preprint arXiv:1904.03257","author":"Ratner Alexander","year":"2019"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2598838"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11491"},{"key":"e_1_2_1_46_1","volume-title":"Machine Learning Systems Workshop at NIPS. 27--29","author":"Schelter Sebastian","year":"2017"},{"key":"e_1_2_1_47_1","volume-title":"Machine Learning: The High Interest Credit Card of Technical Debt. SE4ML: Software Engineering for Machine Learning (NIPS Workshop)","author":"Sculley D.","year":"2014"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.74"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2491411.2491447"},{"key":"e_1_2_1_50_1","first-page":"219","article-title":"Compiler detection of function call side effects","volume":"18","author":"Spuler David A","year":"1994","journal-title":"Informatica"},{"key":"e_1_2_1_51_1","volume-title":"David Patterson, Michael W Mahoney, Randy Katz, Anthony D Joseph, Michael Jordan, Joseph M Hellerstein, Joseph E Gonzalez, et al.","author":"Stoica Ion","year":"2017"},{"key":"e_1_2_1_52_1","doi-asserted-by":"crossref","volume-title":"The design of the Postgres storage system","author":"Stonebraker Michael","DOI":"10.21236\/ADA187244"},{"key":"e_1_2_1_53_1","volume-title":"Energy and policy considerations for deep learning in NLP. arXiv preprint arXiv:1906.02243","author":"Strubell Emma","year":"2019"},{"key":"e_1_2_1_54_1","unstructured":"RS Sutton. 2019. The Bitter Lesson. incompleteideas.net\/IncIdeas\/BitterLesson.html.  RS Sutton. 2019. The Bitter Lesson. incompleteideas.net\/IncIdeas\/BitterLesson.html."},{"key":"e_1_2_1_55_1","volume-title":"Opening the black box-data driven visualization of neural networks","author":"Tzeng F-Y"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3196934"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939502.2939516"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/2775052.2661101"},{"key":"e_1_2_1_59_1","volume-title":"GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461","author":"Wang Alex","year":"2018"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.5555\/378243"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.14778\/3297753.3297763"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.5555\/2387880.2387909"},{"key":"e_1_2_1_63_1","volume-title":"Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, et al.","author":"Zaharia Matei","year":"2018"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/2877204"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.5555\/2685048.2685085"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3436905.3436925","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T10:30:42Z","timestamp":1672223442000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3436905.3436925"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12]]},"references-count":65,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["10.14778\/3436905.3436925"],"URL":"https:\/\/doi.org\/10.14778\/3436905.3436925","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2020,12]]}}}