{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T15:41:35Z","timestamp":1774539695806,"version":"3.50.1"},"reference-count":39,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,12,12]],"date-time":"2023-12-12T00:00:00Z","timestamp":1702339200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Comput. Neurosci."],"abstract":"<jats:p>Burst patterns, characterized by their temporal heterogeneity, have been observed across a wide range of domains, encompassing event sequences from neuronal firing to various facets of human activities. Recent research on predicting event sequences leveraged a Transformer based on the Hawkes process, incorporating a self-attention mechanism to capture long-term temporal dependencies. To effectively handle bursty temporal patterns, we propose a Burst and Memory-aware Transformer (BMT) model, designed to explicitly address temporal heterogeneity. The BMT model embeds the burstiness and memory coefficient into the self-attention module, enhancing the learning process with insights derived from the bursty patterns. Furthermore, we employed a novel loss function designed to optimize the burstiness and memory coefficient values, as well as their corresponding discretized one-hot vectors, both individually and jointly. Numerical experiments conducted on diverse synthetic and real-world datasets demonstrated the outstanding performance of the BMT model in terms of accurately predicting event times and intensity functions compared to existing models and control groups. In particular, the BMT model exhibits remarkable performance for temporally heterogeneous data, such as those with power-law inter-event time distributions. Our findings suggest that the incorporation of burst-related parameters assists the Transformer in comprehending heterogeneous event sequences, leading to an enhanced predictive performance.<\/jats:p>","DOI":"10.3389\/fncom.2023.1292842","type":"journal-article","created":{"date-parts":[[2023,12,12]],"date-time":"2023-12-12T05:10:42Z","timestamp":1702357842000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Burst and Memory-aware Transformer: capturing temporal heterogeneity"],"prefix":"10.3389","volume":"17","author":[{"given":"Byounghwa","family":"Lee","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jung-Hoon","family":"Lee","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sungyup","family":"Lee","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Cheol Ho","family":"Kim","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2023,12,12]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1038\/nature03459","article-title":"The origin of bursts and heavy tails in human dynamics","volume":"435","author":"Barabasi","year":"2005","journal-title":"Nature"},{"key":"B2","doi-asserted-by":"publisher","first-page":"42","DOI":"10.3389\/fncom.2016.00042","article-title":"Burst firing enhances neural output correlation","volume":"10","author":"Chan","year":"2016","journal-title":"Front. Comput. Neurosci"},{"key":"B3","doi-asserted-by":"publisher","first-page":"661","DOI":"10.1137\/070710111","article-title":"Power-law distributions in empirical data","volume":"51","author":"Clauset","year":"2009","journal-title":"SIAM Rev"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.92.108501","article-title":"Long-term clustering, scaling, and universality in the temporal occurrence of earthquakes","volume":"92","author":"Corral","year":"2004","journal-title":"Phys. Rev. Lett"},{"key":"B5","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1111\/j.2517-6161.1955.tb00188.x","article-title":"Some statistical methods connected with series of events","volume":"17","author":"Cox","year":"1955","journal-title":"J. R. Stat. Soc. Ser"},{"key":"B6","volume-title":"An Introduction to the Theory of Point Processes: Volume II: General Theory and Structure","author":"Daley","year":"2008"},{"key":"B7","doi-asserted-by":"publisher","first-page":"5","DOI":"10.3389\/fncom.2018.00005","article-title":"Propensity for bistability of bursting and silence in the leech heart interneuron","volume":"12","author":"Dashevskiy","year":"2018","journal-title":"Front. Comput. Neurosci"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.96.051102","article-title":"Universality in solar flare and earthquake occurrence","volume":"96","author":"de Arcangelis","year":"2006","journal-title":"Phys. Rev. Lett"},{"key":"B9","doi-asserted-by":"crossref","first-page":"1555","DOI":"10.1145\/2939672.2939875","article-title":"\u201cRecurrent marked temporal point processes: Embedding event history to vector,\u201d","volume-title":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Du","year":"2016"},{"key":"B10","doi-asserted-by":"publisher","DOI":"10.1209\/0295-5075\/81\/48002","article-title":"Burstiness and memory in complex systems","volume":"81","author":"Goh","year":"2008","journal-title":"Europhys. Lett"},{"key":"B11","doi-asserted-by":"publisher","first-page":"83","DOI":"10.1093\/biomet\/58.1.83","article-title":"Spectra of some self-exciting and mutually exciting point processes","volume":"58","author":"Hawkes","year":"1971","journal-title":"Biometrika"},{"key":"B12","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput"},{"key":"B13","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1016\/0304-4149(79)90008-5","article-title":"A self-correcting point process","volume":"8","author":"Isham","year":"1979","journal-title":"Stochast. Process. Appl"},{"key":"B14","doi-asserted-by":"publisher","first-page":"022307","DOI":"10.1103\/PhysRevE.100.022307","article-title":"Copula-based algorithm for generating bursty time series","volume":"100","author":"Jo","year":"2019","journal-title":"Phys. Rev. E"},{"key":"B15","doi-asserted-by":"publisher","first-page":"062131","DOI":"10.1103\/PhysRevE.87.062131","article-title":"Contextual analysis framework for bursty dynamics","volume":"87","author":"Jo","year":"2013","journal-title":"Phys. Rev. E"},{"key":"B16","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-68540-3","volume-title":"Bursty Human Dynamics","author":"Karsai","year":"2018"},{"key":"B17","doi-asserted-by":"publisher","first-page":"144","DOI":"10.1016\/j.biosystems.2010.06.002","article-title":"A power-law distribution of inter-spike intervals in renal sympathetic nerve activity in salt-sensitive hypertension-induced chronic heart failure","volume":"101","author":"Kemuriyama","year":"2010","journal-title":"BioSystems"},{"key":"B18","doi-asserted-by":"publisher","first-page":"032311","DOI":"10.1103\/PhysRevE.94.032311","article-title":"Measuring burstiness for finite event sequences","volume":"94","author":"Kim","year":"2016","journal-title":"Phys. Rev. E"},{"key":"B19","doi-asserted-by":"crossref","DOI":"10.1093\/oso\/9780198536932.001.0001","volume-title":"Poisson Processes, Vol. 3","author":"Kingman","year":"1992"},{"key":"B20","unstructured":"LeskovecJ.\n            KrevlA.\n          Ann Arbor, MISnap Datasets: Stanford Large Network Dataset Collection2014"},{"key":"B21","doi-asserted-by":"publisher","first-page":"1748","DOI":"10.1016\/j.ijforecast.2021.03.012","article-title":"Temporal fusion transformers for interpretable multi-horizon time series forecasting","volume":"37","author":"Lim","year":"2021","journal-title":"Int. J. Forecast"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.3389\/fncom.2023.1120566","article-title":"An n400 identification method based on the combination of soft-dtw and transformer","volume":"17","author":"Ma","year":"2023","journal-title":"Front. Comput. Neurosci"},{"key":"B23","first-page":"6754","article-title":"The neural hawkes process: a neurally self-modulating multivariate point process","volume":"30","author":"Mei","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B24","doi-asserted-by":"publisher","first-page":"81","DOI":"10.3389\/fncom.2016.00081","article-title":"Burst firing in the electrosensory system of gymnotiform weakly electric fish: mechanisms and functional roles","volume":"10","author":"Metzen","year":"2016","journal-title":"Front. Comput. Neurosci"},{"key":"B25","volume-title":"An Introduction to Copulas","author":"Nelsen","year":"2006"},{"key":"B26","first-page":"2120","article-title":"Fully neural network based model for general temporal point processes","volume":"32","author":"Omi","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1806.00221","article-title":"Lecture notes: temporal point processes and the conditional intensity function","author":"Rasmussen","year":"2018","journal-title":"arXiv"},{"key":"B28","volume-title":"Stochastic Processes for Insurance and Finance","author":"Rolski","year":"2009"},{"key":"B29","doi-asserted-by":"publisher","first-page":"139","DOI":"10.3389\/fncom.2016.00139","article-title":"Serial spike time correlations affect probability distribution of joint spike events","volume":"10","author":"Shahi","year":"2016","journal-title":"Front. Comput. Neurosci"},{"key":"B30","article-title":"\u201cIntensity-free learning of temporal point processes,\u201d","volume-title":"International Conference on Learning Representations","author":"Shchur","year":"2019"},{"key":"B31","first-page":"5998","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B32","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2023\/759","article-title":"Transformers in time series: a survey","author":"Wen","year":"2022","journal-title":"arXiv"},{"key":"B33","doi-asserted-by":"publisher","first-page":"448","DOI":"10.1086\/306492","article-title":"The waiting-time distribution of solar flare hard x-ray bursts","volume":"509","author":"Wheatland","year":"1998","journal-title":"Astrophys. J"},{"key":"B34","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v31i1.10724","article-title":"\u201cModeling the intensity function of point process via recurrent neural networks,\u201d","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Xiao","year":"2017"},{"key":"B35","doi-asserted-by":"publisher","first-page":"42","DOI":"10.3389\/fncom.2011.00042","article-title":"Elemental spiking neuron model for reproducing diverse firing patterns and predicting precise firing times","volume":"5","author":"Yamauchi","year":"2011","journal-title":"Front. Comput. Neurosci"},{"key":"B36","doi-asserted-by":"publisher","first-page":"48","DOI":"10.3389\/fncom.2018.00048","article-title":"Neural coding with bursts\u2014current state and future perspectives","volume":"12","author":"Zeldenrust","year":"2018","journal-title":"Front. Comput. Neurosci"},{"key":"B37","first-page":"11183","article-title":"\u201cSelf-attentive hawkes process,\u201d","volume-title":"International Conference on Machine Learning","author":"Zhang","year":"2020"},{"key":"B38","doi-asserted-by":"crossref","first-page":"1513","DOI":"10.1145\/2783258.2783401","article-title":"\u201cSeismic: a self-exciting point process model for predicting tweet popularity,\u201d","volume-title":"Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Zhao","year":"2015"},{"key":"B39","first-page":"11692","article-title":"\u201cTransformer hawkes process,\u201d","volume-title":"International Conference on Machine Learning","author":"Zuo","year":"2020"}],"container-title":["Frontiers in Computational Neuroscience"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fncom.2023.1292842\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,12]],"date-time":"2023-12-12T05:10:49Z","timestamp":1702357849000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fncom.2023.1292842\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,12]]},"references-count":39,"alternative-id":["10.3389\/fncom.2023.1292842"],"URL":"https:\/\/doi.org\/10.3389\/fncom.2023.1292842","relation":{},"ISSN":["1662-5188"],"issn-type":[{"value":"1662-5188","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,12]]},"article-number":"1292842"}}