{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T17:38:59Z","timestamp":1740159539519,"version":"3.37.3"},"reference-count":44,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,11,28]],"date-time":"2023-11-28T00:00:00Z","timestamp":1701129600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,11,28]],"date-time":"2023-11-28T00:00:00Z","timestamp":1701129600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100016135","name":"Universit\u00e4t Passau","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100016135","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Datenbank Spektrum"],"published-print":{"date-parts":[[2024,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Despite algorithmic advancements in the field of machine learning, a\u00a0need for improvement in the infrastructure supporting machine learning development and research has become increasingly apparent. Machine learning experiments usually tend to be more ad-hoc in nature, and results are communicated most often in the form of a\u00a0publication. Experimental details are often omitted due to size or time constraints, or simply because the complexity in terms of technical setup or parametrization became intractable. Even access to code bases, disregard important properties of the environment and experimental setup, like for example random generators or computing infrastructure. At the same time, tracking and communicating an often inherently exploratory scientific process is a\u00a0task with considerable effort. We explored different venues to tackle these issues from a\u00a0data science engineering point of view. The efforts resulted in PyPads, a\u00a0framework providing an infrastructure to extend experimental setups with logging, communication and analysis features in a\u00a0mostly non-intrusive way. PyPads can be extended to different Python-based frameworks, utilizing community driven, descriptive metadata in an effort to harmonize library specific logs in an ontology. Meanwhile, we also try to emphasize similarities to practices in software engineering, which have turned out to be essential in practical applications.<\/jats:p>","DOI":"10.1007\/s13222-023-00459-w","type":"journal-article","created":{"date-parts":[[2023,11,28]],"date-time":"2023-11-28T16:02:35Z","timestamp":1701187355000},"page":"53-62","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["PyPads"],"prefix":"10.1007","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3532-354X","authenticated-orcid":false,"given":"Thomas","family":"Wei\u00dfgerber","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1785-8367","authenticated-orcid":false,"given":"Mehdi Ben","family":"Amor","sequence":"additional","affiliation":[]},{"given":"Christofer","family":"Fellicious","sequence":"additional","affiliation":[]},{"given":"Michael","family":"Granitzer","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,11,28]]},"reference":[{"issue":"2","key":"459_CR1","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1007\/s13222-019-00317-8","volume":"19","author":"M Pawlik","year":"2019","unstructured":"Pawlik M, H\u00fctter T, Kocher D, Mann W, Augsten N (2019) Datenbank Spektrum 19(2):107. https:\/\/doi.org\/10.1007\/s13222-019-00317-8","journal-title":"Datenbank Spektrum"},{"issue":"2","key":"459_CR2","doi-asserted-by":"publisher","first-page":"117","DOI":"10.1007\/s13222-019-00316-9","volume":"19","author":"J Risch","year":"2019","unstructured":"Risch J, Krestel R (2019) Datenbank Spektrum 19(2):117. https:\/\/doi.org\/10.1007\/s13222-019-00316-9","journal-title":"Datenbank Spektrum"},{"key":"459_CR3","volume-title":"Adversarial examples in the physical world","author":"A Kurakin","year":"2016","unstructured":"Kurakin A, Goodfellow I, Bengio S et al (2016) Adversarial examples in the physical world"},{"issue":"6","key":"459_CR4","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1109\/5992.881708","volume":"2","author":"M Schwab","year":"2000","unstructured":"Schwab M, Karrenbach N, Claerbout J (2000) Comput Sci Eng 2(6):61","journal-title":"Comput Sci Eng"},{"issue":"2","key":"459_CR5","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1007\/s10994-011-5277-0","volume":"87","author":"J Vanschoren","year":"2012","unstructured":"Vanschoren J, Blockeel H, Pfahringer B, Holmes G (2012) Mach Learn 87(2):127","journal-title":"Mach Learn"},{"issue":"Oct","key":"459_CR6","first-page":"2443","volume":"8","author":"S Sonnenburg","year":"2007","unstructured":"Sonnenburg S, Braun ML, Ong CS, Bengio S, Bottou L, Holmes G, LeCun Y, M\u00c3\u017eller KR, Pereira F, Rasmussen CE et al (2007) J\u00a0Mach Learn Res 8(Oct):2443","journal-title":"J Mach Learn Res"},{"issue":"7604","key":"459_CR7","doi-asserted-by":"publisher","first-page":"452","DOI":"10.1038\/533452a","volume":"533","author":"M Baker","year":"2016","unstructured":"Baker M (2016) Nat News 533(7604):452","journal-title":"Nat News"},{"unstructured":"Olorisade BK, Brereton P, Andras P (2017) ICML 2017 RML Workshop: Reproducibility in Machine Learning","key":"459_CR8"},{"issue":"7450","key":"459_CR9","doi-asserted-by":"publisher","first-page":"433","DOI":"10.1038\/497433a","volume":"497","author":"CG Begley","year":"2013","unstructured":"Begley CG (2013) Nature 497(7450):433","journal-title":"Nature"},{"key":"459_CR10","doi-asserted-by":"publisher","DOI":"10.1126\/science.359.6377.725","volume-title":"Artificial intelligence faces reproducibility crisis","author":"M Hutson","year":"2018","unstructured":"Hutson M (2018) Artificial intelligence faces reproducibility crisis"},{"doi-asserted-by":"crossref","unstructured":"Wendlinger L, Stier J, Granitzer M (2021) Genetic Programming. EuroGP 2021. In: Hu T, Louren\u00e7o N, Medvet E (eds) Genetic Programming. EuroGP 2021. Lecture Notes in Computer Science, Springer, Cham, vol 12691, pp 162\u2013178","key":"459_CR11","DOI":"10.1007\/978-3-030-72812-0_11"},{"key":"459_CR12","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2009.19","author":"V Stodden","year":"2009","unstructured":"Stodden V (2009) Ann Intern Med. https:\/\/doi.org\/10.1109\/MCSE.2009.19","journal-title":"Ann Intern Med"},{"key":"459_CR13","doi-asserted-by":"publisher","DOI":"10.1198\/106186007X178663","author":"R Gentleman","year":"2007","unstructured":"Gentleman R (2007) D. Temple Lang. J\u00a0Comput Graph Stat. https:\/\/doi.org\/10.1198\/106186007X178663","journal-title":"J\u00a0Comput Graph Stat"},{"issue":"3","key":"459_CR14","doi-asserted-by":"publisher","first-page":"2687","DOI":"10.1007\/s11192-020-03587-2","volume":"124","author":"J Homolak","year":"2020","unstructured":"Homolak J, Kodvanj I, Virag D (2020) Scientometrics 124(3):2687. https:\/\/doi.org\/10.1007\/s11192-020-03587-2","journal-title":"Scientometrics"},{"key":"459_CR15","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1007\/978-3-319-00026-8_2","volume-title":"Opening science","author":"B Fecher","year":"2014","unstructured":"Fecher B, Friesike S (2014) Opening science, pp 17\u201347"},{"issue":"6242","key":"459_CR16","doi-asserted-by":"publisher","first-page":"1422","DOI":"10.1126\/science.aab2374","volume":"348","author":"BA Nosek","year":"2015","unstructured":"Nosek BA, Alter G, Banks GC, Borsboom D, Bowman SD, Breckler SJ, Buck S, Chambers CD, Chin G, Christensen G, Contestabile M, Dafoe A, Eich E, Freese J, Glennerster R, Goroff D, Green DP, Hesse B, Humphreys M, Ishiyama J, Karlan D, Kraut A, Lupia A, Mabry P, Madon T, Malhotra N, Mayo-Wilson E, McNutt M, Miguel E, Paluck EL, Simonsohn U, Soderberg C, Spellman BA, Turitto J, VandenBos G, Vazire S, Wagenmakers EJ, Wilson R, Yarkoni T (2015) https:\/\/science.sciencemag.org\/content\/348\/6242\/1422. Science 348(6242):1422. https:\/\/doi.org\/10.1126\/science.aab2374","journal-title":"Science"},{"issue":"6168","key":"459_CR17","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1126\/science.1250475","volume":"343","author":"M McNutt","year":"2014","unstructured":"McNutt M (2014) Science 343(6168):229. https:\/\/doi.org\/10.1126\/science.1250475 (https:\/\/science.sciencemag.org\/content\/343\/6168\/229)","journal-title":"Science"},{"key":"459_CR18","doi-asserted-by":"publisher","first-page":"343","DOI":"10.1201\/9781315373461-13","volume-title":"Implementing reproducible research","author":"ML Braun","year":"2018","unstructured":"Braun ML, Ong CS (2018) Implementing reproducible research. Chapman and Hall\/CRC, pp 343\u2013365"},{"issue":"4","key":"459_CR19","first-page":"8","volume":"17","author":"B Marwick","year":"2017","unstructured":"Marwick B, d\u2019Alpoim Guedes J, Barton CM, Bates LA, Baxter M, Bevan A, Bollwerk EA, Bocinsky RK, Brughmans T, Carter AK et al (2017) SAA archaeological. Record 17(4):8","journal-title":"Record"},{"issue":"1","key":"459_CR20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1093\/joc\/jqz052","volume":"71","author":"T Dienlin","year":"2020","unstructured":"Dienlin T, Johannes N, Bowman ND, Masur PK, Engesser S, K\u00fcmpel AS, Lukito J, Bier LM, Zhang R, Johnson BK, Huskey R, Schneider FM, Breuer J, Parry DA, Vermeulen I, Fisher JT, Banks J, Weber R, Ellis DA, Smits T, Ivory JD, Trepte S, McEwan B, Rinke EM, Neubaum G, Winter S, Carpenter CJ, Kr\u00e4mer N, Utz S, Unkel J, Wang X, Davidson BI, Kim N, Won AS, Domahidi E, Lewis NA, de Vreese C (2020) J\u00a0Commun 71(1):1. https:\/\/doi.org\/10.1093\/joc\/jqz052","journal-title":"J Commun"},{"key":"459_CR21","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.4870627","author":"T Wei\u00dfgerber","year":"2019","unstructured":"Wei\u00dfgerber T, Fellicious C, Granitzer M (2019) PADRE: Platform for mAchine learning and Data science REproducibility. Open Sci Process Model Mach Learn (OSPMML). https:\/\/doi.org\/10.5281\/zenodo.4870627","journal-title":"Open Sci Process Model Mach Learn (OSPMML)"},{"unstructured":"Heumos L, Ehmele P, Menden K, Cuellar LK, Miller E, Lemke S, Gabernet G, Nahnsen S (2021) CoRR abs\/2104.07651. https:\/\/arxiv.org\/abs\/2104.07651. Accessed 19.4.2021","key":"459_CR22"},{"unstructured":"Nagarajan P, Warnell G, Stone P (2018) CoRR abs\/1809.05676. http:\/\/arxiv.org\/abs\/1809.05676. Accessed 5.10.2018","key":"459_CR23"},{"key":"459_CR24","volume-title":"LOD","author":"C Fellicious","year":"2020","unstructured":"Fellicious C, Wei\u00dfgerber T, Granitzer M (2020) LOD"},{"key":"459_CR25","doi-asserted-by":"publisher","first-page":"325","DOI":"10.1007\/978-3-030-20290-3","volume-title":"A\u00a0beginners guide to python 3 programming","author":"J Hunt","year":"2019","unstructured":"Hunt J (2019) A\u00a0beginners guide to python 3 programming. Springer, pp 325\u2013336"},{"key":"459_CR26","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.4697245","volume-title":"Pypads: bootstrapping community-driven open","author":"T Wei\u00dfgerber","year":"2021","unstructured":"Wei\u00dfgerber T, BenAmor M, Fellicious C, Granitzer M (2021) Pypads: bootstrapping community-driven open. Science for Machine Learning https:\/\/doi.org\/10.5281\/zenodo.4697245"},{"key":"459_CR27","doi-asserted-by":"publisher","first-page":"593","DOI":"10.1007\/978-1-4842-3420-4_13","volume-title":"XML Mapping Files","author":"M Keith","year":"2018","unstructured":"Keith M, Schincariol M, Nardone M (2018) XML Mapping Files. Apress, Berkeley, pp\u00a0593\u2013654 https:\/\/doi.org\/10.1007\/978-1-4842-3420-4_13"},{"unstructured":"Publio GC, Esteves D, Lawrynowicz A, Panov P, Soldatova LN, Soru T, Vanschoren J, Zafar H (2018) CoRR abs\/1807.05351.  http:\/\/arxiv.org\/abs\/1807.05351. Accessed 2018-8-13","key":"459_CR28"},{"unstructured":"MLFlow. Mlflow \u2013 a\u00a0platform for the machine learning lifecycle | mlflow. https:\/\/mlflow.org\/. Accessed 2019-12-17","key":"459_CR29"},{"issue":"4","key":"459_CR30","first-page":"39","volume":"41","author":"M Zaharia","year":"2018","unstructured":"Zaharia M, Chen A, Davidson A, Ghodsi A, Hong SA, Konwinski A, Murching S, Nykodym T, Ogilvie P, Parkhe M et al (2018) IEEE Data Eng Bull 41(4):39","journal-title":"IEEE Data Eng Bull"},{"key":"459_CR31","volume-title":"J.\u00a0Schmidhuber, proceedings of the AutoML. International machine learning society","author":"K Greff","year":"2015","unstructured":"Greff K (2015) J.\u00a0Schmidhuber, proceedings of the AutoML. International machine learning society"},{"unstructured":"W&B. Weigths & biases. https:\/\/docs.wandb.ai\/. Accessed 2021-08-02","key":"459_CR32"},{"key":"459_CR33","first-page":"06034","volume":"214","author":"T \u0160imko","year":"2019","unstructured":"\u0160imko T, Heinrich L, Hirvonsalo H, Kousidis D, Rodr\u00edguez D (2019) EPJ web of conferences. EDP Sci 214:6034","journal-title":"EDP Sci"},{"unstructured":"dvc. Data version control \u00b7 dvc. https:\/\/dvc.org\/. Accessed 2020-03-16","key":"459_CR34"},{"issue":"2","key":"459_CR35","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1145\/2641190.2641198","volume":"15","author":"J Vanschoren","year":"2014","unstructured":"Vanschoren J, Van Rijn JN, Bischl B, Torgo L (2014) ACM SIGKDD. Explor Newsl 15(2):49","journal-title":"Explor Newsl"},{"key":"459_CR36","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1201\/9781315373461-3","volume":"2018","author":"AP Davison","year":"2018","unstructured":"Davison\u00a0AP, Mattioni\u00a0M, Samarkanov\u00a0D, Tele\u0144czuk\u00a0B (2018) Sumatra: A Toolkit for Reproducible Research. Implementing Reproducible Research [Internet] 2018:57\u201378. https:\/\/doi.org\/10.1201\/9781315373461-3","journal-title":"Implementing Reproducible Research [Internet]"},{"unstructured":"Google. Kubeflow|kubeflow. https:\/\/www.kubeflow.org\/docs\/about\/kubeflow\/. Accessed 2019-11-16","key":"459_CR37"},{"key":"459_CR38","doi-asserted-by":"publisher","first-page":"2085","DOI":"10.1145\/2882903.2899401","volume-title":"ACM SIGMOD International Conference on Management of Data, SIGMOD 2016","author":"F Chirigati","year":"2016","unstructured":"Chirigati F, Rampin R, Shasha D, Freire J (2016) SIGMOD 2016 - proceedings of the 2016 international conference on management of data (association for computing machinery, 2016), proceedings of the ACM SIGMOD international conference on management of data. In: ACM SIGMOD International Conference on Management of Data, SIGMOD 2016, pp 2085\u20132088 https:\/\/doi.org\/10.1145\/2882903.2899401"},{"key":"459_CR39","doi-asserted-by":"publisher","DOI":"10.1145\/2618137.2618138","volume-title":"Proceedings of the 1st ACM SIGPLAN Workshop on Reproducible Research Methodologies and New Publication Models in Computer Engineering","author":"Y Janin","year":"2014","unstructured":"Janin Y, Vincent C, Duraffort R (2014) Proceedings of the 1st ACM SIGPLAN Workshop on Reproducible Research Methodologies and New Publication Models in Computer Engineering. Association for Computing Machinery, New York, NY, USA https:\/\/doi.org\/10.1145\/2618137.2618138"},{"unstructured":"Pham Q, Malik T, Foster I. 2013. Using provenance for repeatability. In Proceedings of the 5th USENIX Workshop on the Theory and Practice of Provenance (TaPP \u201913). USENIX Association, USA, Article 2, 1\u20134.","key":"459_CR40"},{"issue":"4","key":"459_CR41","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1109\/MCSE.2012.36","volume":"14","author":"P Guo","year":"2012","unstructured":"Guo P (2012) Comput Sci Eng 14(4):32","journal-title":"Comput Sci Eng"},{"key":"459_CR42","series-title":"EDBT","doi-asserted-by":"publisher","first-page":"765","DOI":"10.1145\/2452376.2452475","volume-title":"in proceedings of the 16th international conference on extending database technology","author":"M Huq","year":"2013","unstructured":"Huq M, Apers P, Wombacher A (2013) in proceedings of the 16th international conference on extending database technology. EDBT, vol 2013. Association for Computing Machinery (ACM), United States, pp 765\u2013768 https:\/\/doi.org\/10.1145\/2452376.2452475"},{"doi-asserted-by":"crossref","unstructured":"Murta L, Braganholo V, Chirigati F, Koop D, Freire J (2015) Provenance and annotation of data and processes. In: Lud\u00e4scher B, Plale B (eds) Provenance and Annotation of Data and Processes, Springer, Cham, pp 71\u201383","key":"459_CR43","DOI":"10.1007\/978-3-319-16462-5_6"},{"key":"459_CR44","doi-asserted-by":"publisher","DOI":"10.1145\/3311955","author":"JF Pimentel","year":"2019","unstructured":"Pimentel JF, Freire J, Murta L, Braganholo V (2019) ACM Comput Surv. https:\/\/doi.org\/10.1145\/3311955","journal-title":"ACM Comput Surv"}],"container-title":["Datenbank-Spektrum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13222-023-00459-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s13222-023-00459-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13222-023-00459-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,27]],"date-time":"2024-03-27T10:27:30Z","timestamp":1711535250000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s13222-023-00459-w"}},"subtitle":["Transparent Machine Learning Experiment Tracking"],"short-title":[],"issued":{"date-parts":[[2023,11,28]]},"references-count":44,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,3]]}},"alternative-id":["459"],"URL":"https:\/\/doi.org\/10.1007\/s13222-023-00459-w","relation":{},"ISSN":["1618-2162","1610-1995"],"issn-type":[{"type":"print","value":"1618-2162"},{"type":"electronic","value":"1610-1995"}],"subject":[],"published":{"date-parts":[[2023,11,28]]},"assertion":[{"value":"31 May 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 October 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 November 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}