{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,4]],"date-time":"2026-06-04T23:03:53Z","timestamp":1780614233711,"version":"3.54.1"},"reference-count":65,"publisher":"IOP Publishing","issue":"3","license":[{"start":{"date-parts":[[2024,9,17]],"date-time":"2024-09-17T00:00:00Z","timestamp":1726531200000},"content-version":"vor","delay-in-days":16,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2024,9,17]],"date-time":"2024-09-17T00:00:00Z","timestamp":1726531200000},"content-version":"tdm","delay-in-days":16,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/100006208","name":"High Energy Physics","doi-asserted-by":"crossref","award":["DE-AC02-76SF00515"],"award-info":[{"award-number":["DE-AC02-76SF00515"]}],"id":[{"id":"10.13039\/100006208","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001742","name":"United States-Israel Binational Science Foundation","doi-asserted-by":"crossref","award":["2022641"],"award-info":[{"award-number":["2022641"]}],"id":[{"id":"10.13039\/501100001742","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"crossref","award":["EXC-2094-390783311"],"award-info":[{"award-number":["EXC-2094-390783311"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"crossref"}]},{"name":"SNSF Sinergia","award":["200020_212127"],"award-info":[{"award-number":["200020_212127"]}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2024,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>We propose <jats:italic>masked particle modeling<\/jats:italic> (MPM) as a self-supervised method for learning generic, transferable, and reusable representations on unordered sets of inputs for use in high energy physics (HEP) scientific data. This work provides a novel scheme to perform masked modeling based pre-training to learn permutation invariant functions on sets. More generally, this work provides a step towards building large foundation models for HEP that can be generically pre-trained with self-supervised learning and later fine-tuned for a variety of down-stream tasks. In MPM, particles in a set are masked and the training objective is to recover their identity, as defined by a discretized token representation of a pre-trained vector quantized variational autoencoder. We study the efficacy of the method in samples of high energy jets at collider physics experiments, including studies on the impact of discretization, permutation invariance, and ordering. We also study the fine-tuning capability of the model, showing that it can be adapted to tasks such as supervised and weakly supervised jet classification, and that the model can transfer efficiently with small fine-tuning data sets to new classes and new data domains.<\/jats:p>","DOI":"10.1088\/2632-2153\/ad64a8","type":"journal-article","created":{"date-parts":[[2024,7,17]],"date-time":"2024-07-17T22:59:16Z","timestamp":1721257156000},"page":"035074","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":23,"title":["Masked particle modeling on sets: towards self-supervised high energy physics foundation models"],"prefix":"10.1088","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8535-6687","authenticated-orcid":true,"given":"Tobias","family":"Golling","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4048-7584","authenticated-orcid":true,"given":"Lukas","family":"Heinrich","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3386-6869","authenticated-orcid":true,"given":"Michael","family":"Kagan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2999-6150","authenticated-orcid":true,"given":"Samuel","family":"Klein","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1406-1413","authenticated-orcid":true,"given":"Matthew","family":"Leigh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5480-5099","authenticated-orcid":true,"given":"Margarita","family":"Osadchy","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5987-4648","authenticated-orcid":false,"given":"John","family":"Andrew Raine","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"266","published-online":{"date-parts":[[2024,9,17]]},"reference":[{"key":"mlstad64a8bib1","article-title":"On the opportunities and risks of foundation models","author":"Bommasani","year":"2022"},{"key":"mlstad64a8bib2","article-title":"Bart: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","author":"Lewis","year":"2019"},{"key":"mlstad64a8bib3","article-title":"BERT: pre-training of deep bidirectional transformers for language understanding","author":"Devlin","year":"2019"},{"key":"mlstad64a8bib4","article-title":"Gpt-4 technical report","author":"OpenAI","year":"2023"},{"key":"mlstad64a8bib5","first-page":"pp 1877","article-title":"Language models are few-shot learners","volume":"vol 33","author":"Brown","year":"2020"},{"key":"mlstad64a8bib6","article-title":"An image is worth 16x16 words: transformers for image recognition at scale","author":"Dosovitskiy","year":"2021"},{"key":"mlstad64a8bib7","first-page":"pp 9630","article-title":"Emerging properties in self-supervised vision transformers","author":"Caron","year":"2021"},{"key":"mlstad64a8bib8","article-title":"BEiT: BERT pre-training of image transformers","author":"Bao","year":"2022"},{"key":"mlstad64a8bib9","article-title":"Zero-shot text-to-image generation","author":"Ramesh","year":"2021"},{"key":"mlstad64a8bib10","article-title":"Flamingo: a visual language model for few-shot learning","author":"Alayrac","year":"2022"},{"key":"mlstad64a8bib11","first-page":"pp 9640","article-title":"An empirical study of training self-supervised vision transformers","author":"Chen","year":"2021"},{"key":"mlstad64a8bib12","first-page":"pp 26010","article-title":"Position prediction as an effective pretraining strategy","author":"Zhai","year":"2022"},{"key":"mlstad64a8bib13","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2016239118","article-title":"Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences","volume":"118","author":"Rives","year":"2021","journal-title":"Proc. Natl Acad. Sci."},{"key":"mlstad64a8bib14","doi-asserted-by":"publisher","first-page":"1256","DOI":"10.1038\/s42256-022-00580-7","article-title":"Large-scale chemical language representations capture molecular structure and properties","volume":"4","author":"Ross","year":"2022","journal-title":"Nat. Mach. Intell."},{"key":"mlstad64a8bib15","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1038\/s43588-023-00399-1","article-title":"Large language model for molecular chemistry","volume":"3","author":"Pan","year":"2023","journal-title":"Nat. Commun."},{"key":"mlstad64a8bib16","article-title":"AstroCLIP: cross-modal pre-training for astronomical foundation models","author":"Lanusse","year":"2023"},{"key":"mlstad64a8bib17","article-title":"Towards galaxy foundation models with hybrid contrastive learning","author":"Walmsley","year":"2022"},{"key":"mlstad64a8bib18","doi-asserted-by":"publisher","first-page":"188","DOI":"10.21468\/SciPostPhys.12.6.188","article-title":"Symmetries, safety and self-supervision","volume":"12","author":"Dillon","year":"2022","journal-title":"SciPost Phys."},{"key":"mlstad64a8bib19","article-title":"Anomalies, representations, and self-supervision","author":"Dillon","year":"2023"},{"key":"mlstad64a8bib20","doi-asserted-by":"publisher","DOI":"10.1088\/1748-0221\/17\/08\/P08024","article-title":"A method to challenge symmetries in data with self-supervised learning","volume":"17","author":"Tombs","year":"2022","journal-title":"J. Instrum."},{"key":"mlstad64a8bib21","article-title":"Pre-training strategy using real particle collision data for event classification in collider physics","author":"Kishimoto","year":"2023"},{"key":"mlstad64a8bib22","article-title":"Particle transformer for jet tagging","author":"Qu","year":"2022"},{"key":"mlstad64a8bib23","doi-asserted-by":"publisher","DOI":"10.1088\/2632-2153\/ac07f6","article-title":"Point cloud transformers applied to collider physics","volume":"2","author":"Mikuni","year":"2021","journal-title":"Mach. Learn.: Sci. Technol."},{"key":"mlstad64a8bib24","article-title":"Point cloud generation using transformer encoders and normalising flows","author":"Ach","year":"2022"},{"key":"mlstad64a8bib25","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevD.107.076017","article-title":"Evaluating generative models in high energy physics","volume":"107","author":"Kansal","year":"2023","journal-title":"Phys. Rev. D"},{"key":"mlstad64a8bib26","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevD.105.112008","article-title":"Permutationless many-jet event reconstruction with symmetry preserving attention networks","volume":"105","author":"Fenton","year":"2022","journal-title":"Phys. Rev. D"},{"key":"mlstad64a8bib27","article-title":"Transformer neural networks for identifying boosted Higgs Bosons decaying into bb\u00af and cc\u00af in ATLAS","author":"ATLAS Collaboration","year":"2023"},{"key":"mlstad64a8bib28","article-title":"Differentiable vertex fitting for jet flavour tagging","author":"Smith","year":"2023"},{"key":"mlstad64a8bib29","doi-asserted-by":"crossref","DOI":"10.22323\/1.453.0001","article-title":"Equivariant transformer is all you need","author":"Tomiya","year":"2023"},{"key":"mlstad64a8bib30","article-title":"Attention to mean-fields for particle cloud generation","author":"Ach","year":"2023"},{"key":"mlstad64a8bib31","article-title":"\u03bd2-flows: fast and improved neutrino reconstruction in multi-neutrino final states with conditional normalizing flows","author":"Raine","year":"2023"},{"key":"mlstad64a8bib32","doi-asserted-by":"publisher","first-page":"JHEP06(2023)184","DOI":"10.1007\/JHEP06(2023)184","article-title":"Learning the language of qcd jets with transformers","author":"Finke","year":"2023","journal-title":"J. High Energy Phys."},{"key":"mlstad64a8bib33","article-title":"Jet diffusion versus jetgpt \u2013 modern networks for the LHC","author":"Butter","year":"2023"},{"key":"mlstad64a8bib34","article-title":"Finetuning foundation models for joint analysis optimization","author":"Vigl","year":"2024"},{"key":"mlstad64a8bib35","article-title":"Neural discrete representation learning","author":"van","year":"2017"},{"key":"mlstad64a8bib36","first-page":"pp 281","article-title":"Some methods for classification and analysis of multivariate observations","volume":"vol 1","author":"MacQueen","year":"1967"},{"key":"mlstad64a8bib37","first-page":"pp 1027","article-title":"K-means++: the advantages of careful seeding","author":"Arthur","year":"2007"},{"key":"mlstad64a8bib38","first-page":"pp 108","article-title":"API design for machine learning software: experiences from the scikit-learn project","author":"Buitinck","year":"2013"},{"key":"mlstad64a8bib39","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.6619768)","article-title":"JetClass: a large-scale dataset for deep learning in jet physics","author":"Qu","year":"2022"},{"key":"mlstad64a8bib40","article-title":"RODEM Jet Datasets","author":"Zoch","year":"2024"},{"key":"mlstad64a8bib41","doi-asserted-by":"publisher","first-page":"JHEP07(2014)079","DOI":"10.1007\/JHEP07(2014)079","article-title":"The automated computation of tree-level and next-to-leading order differential cross sections and their matching to parton shower simulations","author":"Alwall","year":"2014","journal-title":"J. High Energy Phys."},{"key":"mlstad64a8bib42","doi-asserted-by":"publisher","first-page":"852","DOI":"10.1016\/j.cpc.2008.01.036","article-title":"A brief introduction to PYTHIA 8.1","volume":"178","author":"Sj\u00f6strand","year":"2008","journal-title":"Comput. Phys. Commun."},{"key":"mlstad64a8bib43","doi-asserted-by":"publisher","first-page":"JHEP03(2013)015","DOI":"10.1007\/JHEP03(2013)015","article-title":"Automatic spin-entangled decays of heavy resonances in Monte Carlo simulations","author":"Artoisenet","year":"2013","journal-title":"J. High Energy Phys."},{"key":"mlstad64a8bib44","doi-asserted-by":"publisher","first-page":"JHEP02(2014)057","DOI":"10.1007\/JHEP02(2014)057","article-title":"DELPHES 3: a modular framework for fast simulation of a generic collider experiment","author":"(The DELPHES 3 collaboration)","year":"2014","journal-title":"J. High Energy Phys."},{"key":"mlstad64a8bib45","doi-asserted-by":"publisher","first-page":"JHEP04(2008)063","DOI":"10.1088\/1126-6708\/2008\/04\/063","article-title":"The anti-kt jet clustering algorithm","author":"Cacciari","year":"2008","journal-title":"J. High Energy Phys."},{"key":"mlstad64a8bib46","article-title":"Normformer: improved transformer pretraining with extra normalization","author":"Shleifer","year":"2021"},{"key":"mlstad64a8bib47","article-title":"Adam: a method for stochastic optimization","author":"Kingma","year":"2014"},{"key":"mlstad64a8bib48","article-title":"Decoupled weight decay regularization","author":"Loshchilov","year":"2017"},{"key":"mlstad64a8bib49","article-title":"Straightening out the straight-through estimator: overcoming optimization challenges in vector quantized networks","author":"Huh","year":"2023"},{"key":"mlstad64a8bib50","article-title":"Convolutional sequence to sequence learning","author":"Gehring","year":"2017"},{"key":"mlstad64a8bib51","article-title":"Attention is all you need","author":"Vaswani","year":"2017"},{"key":"mlstad64a8bib52","doi-asserted-by":"publisher","first-page":"JHEP10(2017)174","DOI":"10.1007\/JHEP10(2017)174","article-title":"Classification without labels: learning from mixed samples in high energy physics","author":"Metodiev","year":"2017","journal-title":"J. High Energy Phys."},{"key":"mlstad64a8bib53","doi-asserted-by":"publisher","DOI":"10.3389\/fdata.2023.899345","article-title":"Curtains for your sliding window: constructing unobserved regions by transforming adjacent intervals","volume":"6","author":"Raine","year":"2023","journal-title":"Front. Big Data"},{"key":"mlstad64a8bib54","article-title":"Classifying anomalies through outer density estimation (CATHODE)","author":"Hallin","year":"2021"},{"key":"mlstad64a8bib55","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.125.131801","article-title":"Dijet resonance search with weak supervision using s = 13 tev p p collisions in the atlas detector","volume":"125","author":"Aad","year":"2020","journal-title":"Phys. Rev. Lett."},{"key":"mlstad64a8bib56","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevD.101.095004","article-title":"Simulation assisted likelihood-free anomaly detection","volume":"101","author":"Andreassen","year":"2020","journal-title":"Phys. Rev. D"},{"key":"mlstad64a8bib57","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevD.107.096025","article-title":"Flow-enhanced transportation for anomaly detection","volume":"107","author":"Golling","year":"2023","journal-title":"Phys. Rev. D"},{"key":"mlstad64a8bib58","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevD.99.014038","article-title":"Extending the search for new resonances with machine learning","volume":"99","author":"Collins","year":"2019","journal-title":"Phys. Rev. D"},{"key":"mlstad64a8bib59","doi-asserted-by":"publisher","first-page":"508","DOI":"10.1140\/epjc\/s10052-022-10454-2","article-title":"Data-directed search for new physics based on symmetries of the sm","volume":"82","author":"Birman","year":"2022","journal-title":"Eur. Phys. J. C"},{"key":"mlstad64a8bib60","article-title":"Full phase space resonant anomaly detection","author":"Buhmann","year":"2023"},{"key":"mlstad64a8bib61","article-title":"Improving new physics searches with diffusion models for event observables and jet constituents","author":"Sengupta","year":"2023"},{"key":"mlstad64a8bib62","doi-asserted-by":"crossref","DOI":"10.1103\/PhysRevD.108.092008","article-title":"Learning to isolate muons in data","author":"Witkowski","year":"2023"},{"key":"mlstad64a8bib63","article-title":"Estimating or propagating gradients through stochastic neurons for conditional computation","author":"Bengio","year":"2013"},{"key":"mlstad64a8bib64","article-title":"vqtorch: PyTorch package for vector quantization","author":"Minyoung","year":"2022"},{"key":"mlstad64a8bib65","first-page":"2579","article-title":"Visualizing data using t-SNE","volume":"9","author":"van der Maaten","year":"2008","journal-title":"J. Mach. Learn. Res."}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad64a8","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad64a8\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad64a8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad64a8\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad64a8\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad64a8\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad64a8\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad64a8\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,19]],"date-time":"2024-09-19T08:50:47Z","timestamp":1726735847000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ad64a8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,1]]},"references-count":65,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,9,17]]},"published-print":{"date-parts":[[2024,9,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/ad64a8","relation":{},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,1]]},"assertion":[{"value":"Masked particle modeling on sets: towards self-supervised high energy physics foundation models","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2024 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2024-03-19","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-07-17","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-09-17","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}