{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,12]],"date-time":"2026-02-12T17:05:40Z","timestamp":1770915940837,"version":"3.50.1"},"reference-count":38,"publisher":"IOP Publishing","issue":"4","license":[{"start":{"date-parts":[[2025,11,4]],"date-time":"2025-11-04T00:00:00Z","timestamp":1762214400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2025,11,4]],"date-time":"2025-11-04T00:00:00Z","timestamp":1762214400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/501100001711","name":"Schweizerischer Nationalfonds zur F\u00f6rderung der Wissenschaftlichen Forschung","doi-asserted-by":"crossref","award":["TMAG-2_209263"],"award-info":[{"award-number":["TMAG-2_209263"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000001","name":"U.S. National Science Foundation","doi-asserted-by":"crossref","award":["2411204"],"award-info":[{"award-number":["2411204"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100003246","name":"Dutch Research Council","doi-asserted-by":"crossref","award":["VI.Veni.202.004"],"award-info":[{"award-number":["VI.Veni.202.004"]}],"id":[{"id":"10.13039\/501100003246","id-type":"DOI","asserted-by":"crossref"}]},{"name":"CERN openlab"}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2025,12,30]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Ensuring data integrity via data quality monitoring (DQM) is critical in large-scale particle physics experiments. This traditionally relies on labor-intensive manual inspection or static machine learning models, which are adequate for stable operating conditions. However, the current era of major detector upgrades at the large hadron collider (LHC) challenges this paradigm. These upgrades are followed by prolonged commissioning periods with frequent and unpredictable changes in detector conditions, a regime for which static models are ill-suited. To address this, we reframe DQM as a dynamic decision-making problem and introduce a human-in-the-loop reinforcement learning (RLHL) framework. Our proximal policy optimization agent learns both to classify data and to strategically decide when to query human experts, optimizing the automation-oversight balance. On synthetic data, the system rapidly adapts to abrupt condition changes and successfully learns from noisy labels. In a simulated online regime, the agent minimizes human intervention by requesting it mainly when its uncertainty is high. A preliminary study on a real offline dataset from the LHCb experiment demonstrates that our synthetic approach is a reasonable proxy for real-world scenarios. The algorithm generalizes effectively with only superficial hyperparameter tuning, robustly identifying anomalies even when trained on augmented data. This work presents a scalable, adaptive solution for semi-autonomous DQM.<\/jats:p>","DOI":"10.1088\/2632-2153\/ae1563","type":"journal-article","created":{"date-parts":[[2025,10,21]],"date-time":"2025-10-21T07:47:18Z","timestamp":1761032838000},"page":"045032","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Human-in-the-loop reinforcement learning for data quality monitoring in particle physics experiments"],"prefix":"10.1088","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-8373-1282","authenticated-orcid":true,"given":"Olivia","family":"Jullian Parra","sequence":"first","affiliation":[]},{"given":"Juli\u00e1n","family":"Garc\u00eda Pardi\u00f1as","sequence":"additional","affiliation":[]},{"given":"Lorenzo","family":"Del Pianta P\u00e9rez","sequence":"additional","affiliation":[]},{"given":"Maximilian","family":"Janisch","sequence":"additional","affiliation":[]},{"given":"Suzanne","family":"Klaver","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7508-7233","authenticated-orcid":true,"given":"Thomas","family":"Leh\u00e9ricy","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5033-0580","authenticated-orcid":true,"given":"Nicola","family":"Serra","sequence":"additional","affiliation":[]}],"member":"266","published-online":{"date-parts":[[2025,11,4]]},"reference":[{"key":"mlstae1563bib1","doi-asserted-by":"publisher","DOI":"10.1088\/1748-0221\/3\/08\/S08003","type":"journal-article","article-title":"The ATLAS experiment at the CERN large hadron collider","volume":"3","author":"Aad","year":"2008","journal-title":"JINST"},{"key":"mlstae1563bib2","doi-asserted-by":"publisher","DOI":"10.1088\/1748-0221\/19\/05\/P05065","type":"journal-article","article-title":"The LHCb Upgrade I","volume":"19","author":"Aaij","year":"2024","journal-title":"JINST"},{"key":"mlstae1563bib3","doi-asserted-by":"publisher","DOI":"10.1088\/1748-0221\/ 3\/08\/S08002","type":"journal-article","article-title":"The ALICE experiment at the CERN LHC","volume":"3","author":"Aamodt","year":"2008","journal-title":"JINST"},{"key":"mlstae1563bib4","article-title":"Tensorflow: large-scale machine learning on heterogeneous distributed systems","author":"Abadi","year":"2016","type":"preprint"},{"key":"mlstae1563bib5","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1007\/s41781-024-00118-z","type":"journal-article","article-title":"Autoencoder-based anomaly detection system for online data quality monitoring of the CMS electromagnetic calorimeter","volume":"8","author":"Abadjiev","year":"2024","journal-title":"Comput. Softw. Big Sci."},{"key":"mlstae1563bib6","article-title":"Proximal policy optimization. OpenAI spinning up","author":"Achiam","year":"2018","type":"other"},{"key":"mlstae1563bib7","doi-asserted-by":"publisher","DOI":"10.1088\/1742-6596\/898\/9\/092027","type":"journal-article","article-title":"LHCb data quality monitoring","volume":"898","author":"Adinolfi","year":"2017a","journal-title":"J. Phys.: Conf. Ser."},{"key":"mlstae1563bib8","doi-asserted-by":"publisher","DOI":"10.1088\/1748-0221\/3\/08\/S08005","type":"journal-article","article-title":"The LHCb Detector at the LHC","volume":"3","author":"Alves","year":"2008","journal-title":"JINST"},{"key":"mlstae1563bib9","doi-asserted-by":"publisher","first-page":"124017","DOI":"10.1109\/ACCESS.2022.3224023","type":"journal-article","article-title":"Deep reinforcement learning for anomaly detection: a systematic review","volume":"10","author":"Arshad","year":"2022","journal-title":"IEEE Access"},{"key":"mlstae1563bib10","doi-asserted-by":"publisher","first-page":"9679","DOI":"10.3390\/s23249679","type":"journal-article","article-title":"Spatio-temporal anomaly detection with graph networks for data quality monitoring of the hadron calorimeter","volume":"23","author":"Asres","year":"2023","journal-title":"Sensors"},{"key":"mlstae1563bib11","article-title":"Minimal pytorch implementation of proximal policy optimization","author":"Barhate","year":"2018","type":"other"},{"key":"mlstae1563bib12","article-title":"Openai gym","author":"Brockman","year":"2016","type":"preprint"},{"key":"mlstae1563bib13","doi-asserted-by":"publisher","DOI":"10.1088\/1748-0221\/ 3\/08\/S08004","type":"journal-article","article-title":"The CMS Experiment at the CERN LHC","volume":"3","author":"Chatrchyan","year":"2008","journal-title":"JINST"},{"key":"mlstae1563bib14","doi-asserted-by":"publisher","author":"LHCb collaboration","year":"2013","DOI":"10.7483\/OPENDATA.LHCb.HKJW.TWSZ","type":"report"},{"key":"mlstae1563bib15","doi-asserted-by":"publisher","first-page":"236","DOI":"10.22323\/1.350.0236","type":"journal-article","article-title":"Using machine learning techniques for data quality monitoring in CMS and ALICE experiments","volume":"LHCP2019","author":"Deja","year":"2019","journal-title":"PoS"},{"key":"mlstae1563bib16","first-page":"p WEBO04","type":"conference-proceedings","article-title":"Enhancement of the S-DALINAC control system with machine learning methods","author":"Hanten","year":"2019"},{"key":"mlstae1563bib17","article-title":"Model-free and Bayesian ensembling model-based deep reinforcement learning for particle accelerator control demonstrated on the FERMI FEL","author":"Hirlaender","year":"2020","type":"preprint"},{"key":"mlstae1563bib18","first-page":"p THL038","type":"conference-proceedings","article-title":"Ultra fast reinforcement learning demonstrated at CERN AWAKE","volume":"vol IPAC2023","author":"Hirlaender","year":"2023"},{"key":"mlstae1563bib19","doi-asserted-by":"crossref","DOI":"10.2172\/1825276","type":"conference-proceedings","article-title":"Developing robust digital twins and reinforcement learning for accelerator control systems at the fermilab booster","author":"Kafkes","year":"2021"},{"key":"mlstae1563bib20","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevAccelBeams.23.124801","type":"journal-article","article-title":"Sample-efficient reinforcement learning for CERN accelerator control","volume":"23","author":"Kain","year":"2020","journal-title":"Phys. Rev. Accel. Beams"},{"key":"mlstae1563bib21","article-title":"A survey of reinforcement learning from human feedback","author":"Kaufmann","year":"2023","type":"preprint"},{"key":"mlstae1563bib22","article-title":"Adam: a method for stochastic optimization","author":"Kingma","year":"2014","type":"preprint"},{"key":"mlstae1563bib23","first-page":"pp 1196","type":"conference-proceedings","article-title":"Learning with noisy labels","author":"Natarajan","year":"2013"},{"key":"mlstae1563bib24","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.17019894)","type":"other","article-title":"Human in the loop rl for DQM","author":"Olivia Jullian Parra","year":"2025"},{"key":"mlstae1563bib25","article-title":"Autonomous control of a particle accelerator using deep reinforcement learning","author":"Pang","year":"2020","type":"preprint"},{"key":"mlstae1563bib26","article-title":"Pytorch: an imperative style, high-performance deep learning library","author":"Paszke","year":"2019","type":"preprint"},{"key":"mlstae1563bib27","doi-asserted-by":"crossref","DOI":"10.1051\/epjconf\/201921406008","type":"other","article-title":"Anomaly detection using deep autoencoders for the assessment of the quality of the data acquired by the CMS experiment","author":"Pol","year":"2019a"},{"key":"mlstae1563bib28","article-title":"Trigger rate anomaly detection with conditional variational autoencoders at the CMS experiment","author":"Pol","year":"2019b","type":"conference-proceedings"},{"key":"mlstae1563bib29","first-page":"pp 115","author":"Pol","year":"2022","type":"book"},{"key":"mlstae1563bib30","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/s41781-018-0020-1","type":"journal-article","article-title":"Detector monitoring with artificial neural networks at the CMS experiment at the CERN Large Hadron Collider","volume":"3","author":"Pol","year":"2019c","journal-title":"Comput. Softw. Big Sci."},{"key":"mlstae1563bib31","doi-asserted-by":"publisher","first-page":"239","DOI":"10.1080\/00401706.1959.10489860","type":"journal-article","article-title":"Control chart tests based on geometric moving averages","volume":"1","author":"Roberts","year":"1959","journal-title":"Technometrics"},{"key":"mlstae1563bib32","author":"Schulman","year":"2017","type":"preprint"},{"key":"mlstae1563bib33","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s40537-019-0197-0","type":"journal-article","article-title":"A survey on image data augmentation for deep learning","volume":"6","author":"Shorten","year":"2019","journal-title":"J. Big Data"},{"key":"mlstae1563bib34","first-page":"p WEA099","type":"conference-proceedings","article-title":"Reinforcement control for LEBT and RFQ of linear accelerators","volume":"vol IPAC2023","author":"Su","year":"2023"},{"key":"mlstae1563bib35","first-page":"pp 4653","type":"conference-proceedings","article-title":"Time series data augmentation for deep learning: a survey","author":"Wen","year":"2021"},{"key":"mlstae1563bib36","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1023\/A:1022672621406","type":"journal-article","article-title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","volume":"8","author":"Williams","year":"1992","journal-title":"Mach. Learn."},{"key":"mlstae1563bib37","article-title":"Beyond PID controllers: PPO with neuralized PID policy for proton beam intensity control in Mu2e","author":"Xu","year":"2023","type":"conference-proceedings"},{"key":"mlstae1563bib38","article-title":"High-Luminosity Large Hadron Collider (HL-LHC): technical design report","author":"Zurbano Fernandez","year":"2020","type":"other"}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1563","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1563\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1563","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1563\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1563\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1563\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1563\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1563\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,4]],"date-time":"2025-11-04T09:16:49Z","timestamp":1762247809000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ae1563"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,4]]},"references-count":38,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2025,11,4]]},"published-print":{"date-parts":[[2025,12,30]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/ae1563","relation":{},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,4]]},"assertion":[{"value":"Human-in-the-loop reinforcement learning for data quality monitoring in particle physics experiments","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2025 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2025-06-13","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2025-10-20","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2025-11-04","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}