{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T13:08:18Z","timestamp":1740143298555,"version":"3.37.3"},"reference-count":71,"publisher":"Public Library of Science (PLoS)","issue":"12","license":[{"start":{"date-parts":[[2024,12,30]],"date-time":"2024-12-30T00:00:00Z","timestamp":1735516800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Spike train modeling across large neural populations is a powerful tool for understanding how neurons code information in a coordinated manner. Recent studies have employed marked point processes in neural population modeling. The marked point process is a stochastic process that generates a sequence of events with marks. Spike train models based on such processes use the waveform features of spikes as marks and express the generative structure of the unsorted spikes without applying spike sorting. In such modeling, the goal is to estimate the joint mark intensity that describes how observed covariates or hidden states (e.g., animal behaviors, animal internal states, and experimental conditions) influence unsorted spikes. A major issue with this approach is that existing joint mark intensity models are not designed to capture high-dimensional and highly nonlinear observations. To address this limitation, we propose a new joint mark intensity model based on a variational autoencoder, capable of representing the dependency structure of unsorted spikes on observed covariates or hidden states in a data-driven manner. Our model defines the joint mark intensity as a latent variable model, where a neural network decoder transforms a shared latent variable into states and marks. With our model, we derive a new log-likelihood lower bound by exploiting the variational evidence lower bound and upper bound (e.g., the <jats:italic>\u03c7<\/jats:italic> upper bound) and use this new lower bound for parameter estimation. To demonstrate the strength of this approach, we integrate our model into a state space model with a nonlinear embedding to capture the hidden state dynamics underlying the observed covariates and unsorted spikes. This enables us to reconstruct covariates from unsorted spikes, known as neural decoding. Our model achieves superior performance in prediction and decoding tasks for synthetic data and the spiking activities of place cells.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1012620","type":"journal-article","created":{"date-parts":[[2024,12,30]],"date-time":"2024-12-30T18:49:31Z","timestamp":1735584571000},"page":"e1012620","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["Marked point process variational autoencoder with applications to unsorted spiking activities"],"prefix":"10.1371","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-9494-2627","authenticated-orcid":true,"given":"Ryohei","family":"Shibue","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tomoharu","family":"Iwata","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"340","published-online":{"date-parts":[[2024,12,30]]},"reference":[{"issue":"7679","key":"pcbi.1012620.ref001","doi-asserted-by":"crossref","first-page":"232","DOI":"10.1038\/nature24636","article-title":"Fully integrated silicon probes for high-density recording of neural activity","volume":"551","author":"JJ Jun","year":"2017","journal-title":"Nature"},{"issue":"6539","key":"pcbi.1012620.ref002","doi-asserted-by":"crossref","first-page":"eabf4588","DOI":"10.1126\/science.abf4588","article-title":"Neuropixels 2.0: a miniaturized high-density probe for stable, long-term brain recordings","volume":"372","author":"NA Steinmetz","year":"2021","journal-title":"Science"},{"issue":"2","key":"pcbi.1012620.ref003","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1023\/A:1003403601725","article-title":"Space-time point-process models for earthquake occurrences","volume":"50","author":"Y Ogata","year":"1998","journal-title":"Ann Inst Stat Math"},{"key":"pcbi.1012620.ref004","unstructured":"Linderman S, Adams R. Discovering latent network structure in point process data. In: Proc 31st Int Conf Mach Learn. vol. 32; 2014."},{"volume-title":"KDD","year":"2013","author":"T Iwata","key":"pcbi.1012620.ref005"},{"volume-title":"Adv Neural Inf Process Syst","year":"2016","author":"Y Wang","key":"pcbi.1012620.ref006"},{"issue":"493","key":"pcbi.1012620.ref007","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1198\/jasa.2011.ap09546","article-title":"Self-exciting point process modeling of crime","volume":"106","author":"GO Mohler","year":"2011","journal-title":"J Am Stat Assoc"},{"key":"pcbi.1012620.ref008","unstructured":"Chen Z, Kloosterman F, Layton S, Wilson MA. Transductive neural decoding for unsorted neuronal spikes of rat hippocampus. In: Conf IEEE Eng Med Biol Soc; 2012."},{"issue":"1","key":"pcbi.1012620.ref009","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1152\/jn.01046.2012","article-title":"Bayesian decoding using unsorted spikes in the rat hippocampus","volume":"111","author":"F Kloosterman","year":"2014","journal-title":"J Neurophysiol"},{"issue":"7","key":"pcbi.1012620.ref010","doi-asserted-by":"crossref","first-page":"1438","DOI":"10.1162\/NECO_a_00744","article-title":"Clusterless decoding of position from multiunit activity using a marked point process filter","volume":"27","author":"X Deng","year":"2015","journal-title":"Neural Comput"},{"key":"pcbi.1012620.ref011","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.knosys.2015.09.013","article-title":"Kernel density compression for real-time Bayesian encoding\/decoding of unsorted hippocampal spikes","volume":"94","author":"D Sodkomkham","year":"2016","journal-title":"Knowl Based Syst"},{"issue":"5","key":"pcbi.1012620.ref012","doi-asserted-by":"crossref","first-page":"2902","DOI":"10.1152\/jn.00818.2016","article-title":"Firing rate estimation using infinite mixture models and its application to neural decoding","volume":"118","author":"R Shibue","year":"2017","journal-title":"J Neurophysiol"},{"key":"pcbi.1012620.ref013","article-title":"Marked point process filter for clusterless and adaptive encoding-decoding of multiunit activity","author":"K Arai","year":"2018","journal-title":"bioRxiv"},{"issue":"2","key":"pcbi.1012620.ref014","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1007\/s10827-018-0698-4","article-title":"A common goodness-of-fit framework for neural population models using marked point process time-rescaling","volume":"45","author":"L Tao","year":"2018","journal-title":"J Comput Neurosci"},{"key":"pcbi.1012620.ref015","doi-asserted-by":"crossref","first-page":"109006","DOI":"10.1016\/j.jneumeth.2020.109006","article-title":"Real-time point process filter for multidimensional decoding problems using mixture models","volume":"348","author":"MR Rezaei","year":"2021","journal-title":"J Neurosci Methods"},{"issue":"18","key":"pcbi.1012620.ref016","doi-asserted-by":"crossref","first-page":"7411","DOI":"10.1523\/JNEUROSCI.18-18-07411.1998","article-title":"A statistical paradigm for neural spike train decoding applied to position prediction from ensemble firing patterns of rat hippocampal place cells","volume":"18","author":"EN Brown","year":"1998","journal-title":"J Neurosci"},{"key":"pcbi.1012620.ref017","doi-asserted-by":"crossref","first-page":"106","DOI":"10.1016\/j.brainresbull.2015.04.007","article-title":"Past, present and future of spike sorting techniques","volume":"119","author":"HG Rey","year":"2015","journal-title":"Brain Res Bull"},{"issue":"5","key":"pcbi.1012620.ref018","doi-asserted-by":"crossref","first-page":"056005","DOI":"10.1088\/1741-2560\/11\/5\/056005","article-title":"To sort or not to sort: the impact of spike-sorting on neural decoding performance","volume":"11","author":"S Todorova","year":"2014","journal-title":"J Neural Eng"},{"issue":"10","key":"pcbi.1012620.ref019","doi-asserted-by":"crossref","first-page":"2521","DOI":"10.1016\/j.celrep.2017.02.038","article-title":"Unsupervised spike sorting for large-scale, high-density multielectrode arrays","volume":"18","author":"G Hilgen","year":"2017","journal-title":"Cell Rep"},{"volume-title":"Adv Neural Inf Process Syst","year":"2023","author":"Y Zhang","key":"pcbi.1012620.ref020"},{"volume-title":"Adv Neural Inf Process Syst","year":"2021","author":"J Boussard","key":"pcbi.1012620.ref021"},{"key":"pcbi.1012620.ref022","article-title":"YASS: Yet Another Spike Sorter applied to large-scale multi-electrode array recordings in primate retina","author":"J Lee","year":"2020","journal-title":"bioRxiv"},{"key":"pcbi.1012620.ref023","unstructured":"Kingma DP, Welling M. Auto-encoding variational Bayes. In: Int Conf Learn Represent; 2014."},{"volume-title":"Adv Neural Inf Process Syst","year":"2016","author":"Y Li","key":"pcbi.1012620.ref024"},{"volume-title":"Adv Neural Inf Process Syst","year":"2017","author":"AB Dieng","key":"pcbi.1012620.ref025"},{"volume-title":"Adv Neural Inf Process Syst","year":"2018","author":"D Wang","key":"pcbi.1012620.ref026"},{"key":"pcbi.1012620.ref027","unstructured":"Chen L, Tao C, Zhang R, Henao R, Duke LC. Variational inference and model selection with generalized evidence bounds. In: Proc 35th Int Conf Mach Learn; 2018."},{"key":"pcbi.1012620.ref028","article-title":"f-divergence variational inference","author":"N Wan","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"pcbi.1012620.ref029","unstructured":"Zoltowski D, Pillow J, Linderman S. A general recurrent state space framework for modeling neural dynamics during decision-making. In: Proc 37th Int Conf Mach Learn; 2020."},{"volume-title":"Adv Neural Inf Process Syst","year":"2011","author":"JH Macke","key":"pcbi.1012620.ref030"},{"volume-title":"Adv Neural Inf Process Syst","year":"2014","author":"EW Archer","key":"pcbi.1012620.ref031"},{"volume-title":"Adv Neural Inf Process Syst","year":"2016","author":"Y Gao","key":"pcbi.1012620.ref032"},{"volume-title":"Adv Neural Inf Process Syst","year":"2020","author":"J Glaser","key":"pcbi.1012620.ref033"},{"key":"pcbi.1012620.ref034","unstructured":"Archer E, Park IM, Buesing L, Cunningham J, Paninski L. Black box variational inference for state space models. arXiv. 2015;."},{"key":"pcbi.1012620.ref035","article-title":"Composing graphical models with neural networks for structured representations and fast inference","author":"MJ Johnson","year":"2016","journal-title":"Adv Neural Inf Process Syst"},{"volume-title":"An introduction to the theory of point processes","year":"2003","author":"DJ Daley","key":"pcbi.1012620.ref036"},{"issue":"5","key":"pcbi.1012620.ref037","doi-asserted-by":"crossref","DOI":"10.3390\/e20050383","article-title":"On f-divergences: integral representations, local behavior, and inequalities","volume":"20","author":"I Sason","year":"2018","journal-title":"Entropy"},{"key":"pcbi.1012620.ref038","unstructured":"Burda Y, Grosse R, Salakhutdinov R. Importance weighted autoencoders. In: Int Conf Learn Represent; 2015."},{"volume-title":"Adv Neural Inf Process Syst","year":"2020","author":"R Lopez","key":"pcbi.1012620.ref039"},{"volume-title":"Adv Neural Inf Process Syst","year":"2017","author":"V Kuleshov","key":"pcbi.1012620.ref040"},{"key":"pcbi.1012620.ref041","unstructured":"Nowozin S. Debiasing evidence approximations: on importance-weighted autoencoders and jackknife variational inference. In: Int Conf Learn Represent; 2018."},{"key":"pcbi.1012620.ref042","unstructured":"Tucker G, Lawson D, Gu S, Maddison CJ. Doubly reparameterized gradient estimators for Monte Carlo objectives. In: Int Conf Learn Represent; 2019."},{"issue":"5","key":"pcbi.1012620.ref043","doi-asserted-by":"crossref","first-page":"971","DOI":"10.1162\/089976604773135069","article-title":"Dynamic analysis of neural encoding by point process adaptive filtering","volume":"16","author":"UT Eden","year":"2004","journal-title":"Neural Comput"},{"issue":"489","key":"pcbi.1012620.ref044","doi-asserted-by":"crossref","first-page":"170","DOI":"10.1198\/jasa.2009.tm08326","article-title":"Approximate methods for state-space models","volume":"105","author":"S Koyama","year":"2010","journal-title":"J Am Stat Assoc"},{"issue":"1-2","key":"pcbi.1012620.ref045","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1007\/s10827-009-0179-x","article-title":"A new look at state-space models for neural data","volume":"29","author":"L Paninski","year":"2010","journal-title":"J Comput Neurosci"},{"issue":"1-2","key":"pcbi.1012620.ref046","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1007\/s10827-009-0150-x","article-title":"Efficient computation of the maximum a posteriori path and parameter estimation in integrate-and-fire and more general state-space models","volume":"29","author":"S Koyama","year":"2010","journal-title":"J Comput Neurosci"},{"issue":"2","key":"pcbi.1012620.ref047","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1007\/BF01585997","article-title":"A Gauss-Newton method for convex composite optimization","volume":"71","author":"JV Burke","year":"1995","journal-title":"Math Program"},{"volume-title":"Adv Neural Inf Process Syst","year":"2014","author":"DP Kingma","key":"pcbi.1012620.ref048"},{"volume-title":"Adv Neural Inf Process Syst","year":"2015","author":"K Sohn","key":"pcbi.1012620.ref049"},{"volume-title":"KDD","year":"2016","author":"N Du","key":"pcbi.1012620.ref050"},{"key":"pcbi.1012620.ref051","doi-asserted-by":"crossref","unstructured":"Cho K, van Merri\u00ebnboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Proc Conf Empir Methods Nat Lang Process; 2014.","DOI":"10.3115\/v1\/D14-1179"},{"issue":"4","key":"pcbi.1012620.ref052","doi-asserted-by":"crossref","DOI":"10.1523\/ENEURO.0506-19.2020","article-title":"Machine learning for neural decoding","volume":"7","author":"JI Glaser","year":"2020","journal-title":"eNeuro"},{"key":"pcbi.1012620.ref053","unstructured":"Grosmark AD, Long J, Buzs\u00e1ki G. Recordings from hippocampal area CA1, PRE, during and POST novel spatial learning; 2016. CRCNS.org. http:\/\/dx.doi.org\/10.6080\/K0862DC5."},{"issue":"6280","key":"pcbi.1012620.ref054","doi-asserted-by":"crossref","first-page":"1440","DOI":"10.1126\/science.aad1935","article-title":"Diversity in neural firing dynamics supports both rigid and learned hippocampal sequences","volume":"351","author":"AD Grosmark","year":"2016","journal-title":"Science"},{"issue":"1","key":"pcbi.1012620.ref055","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1016\/0006-8993(71)90358-1","article-title":"The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat","volume":"34","author":"J O\u2019Keefe","year":"1971","journal-title":"Brain Res"},{"issue":"2","key":"pcbi.1012620.ref056","doi-asserted-by":"crossref","first-page":"292","DOI":"10.1016\/j.neuron.2019.05.003","article-title":"Accurate estimation of neural population dynamics without spike sorting","volume":"103","author":"EM Trautmann","year":"2019","journal-title":"Neuron"},{"key":"pcbi.1012620.ref057","article-title":"Reproducibility of in vivo electrophysiological measurements in mice","volume":"13","author":"IBL","year":"2024","journal-title":"Elife"},{"volume-title":"Adv Neural Inf Process Syst","year":"2016","author":"M Pachitariu","key":"pcbi.1012620.ref058"},{"key":"pcbi.1012620.ref059","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1016\/j.conb.2018.01.009","article-title":"Challenges and opportunities for large-scale electrophysiology with Neuropixels probes","volume":"50","author":"NA Steinmetz","year":"2018","journal-title":"Curr Opin Neurobiol"},{"volume-title":"Adv Neural Inf Process Syst","year":"2017","author":"H Mei","key":"pcbi.1012620.ref060"},{"key":"pcbi.1012620.ref061","doi-asserted-by":"crossref","unstructured":"Xiao S, Yan J, Yang X, Zha H, Chu SM. Modeling the intensity function of point process via recurrent neural networks. In: Proc AAAI Conf Artif Intell; 2017.","DOI":"10.1609\/aaai.v31i1.10724"},{"volume-title":"Adv Neural Inf Process Syst","year":"2019","author":"T Omi","key":"pcbi.1012620.ref062"},{"key":"pcbi.1012620.ref063","doi-asserted-by":"crossref","unstructured":"Mehrasa N, Jyothi AA, Durand T, He J, Sigal L, Mori G. A variational auto-encoder model for stochastic point processes. In: Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit; 2019.","DOI":"10.1109\/CVPR.2019.00328"},{"key":"pcbi.1012620.ref064","doi-asserted-by":"crossref","unstructured":"Shchur O, Bilo\u0161 M, G\u00fcnnemann S. Intensity-free learning of temporal point processes. In: Int Conf Learn Represent; 2020.","DOI":"10.24963\/ijcai.2021\/623"},{"volume-title":"Adv Neural Inf Process Syst","year":"2020","author":"O Shchur","key":"pcbi.1012620.ref065"},{"volume-title":"Adv Neural Inf Process Syst","year":"2018","author":"RTQ Chen","key":"pcbi.1012620.ref066"},{"volume-title":"Adv Neural Inf Process Syst","year":"2019","author":"J Jia","key":"pcbi.1012620.ref067"},{"key":"pcbi.1012620.ref068","unstructured":"Kim TD, Luo TZ, Pillow JW, Brody CD. Inferring latent dynamics underlying neural population activity via neural differential equations. In: Proc 38th Int Conf Mach Learn; 2021."},{"issue":"5","key":"pcbi.1012620.ref069","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1073\/pnas.17.5.315","article-title":"Hamiltonian systems and transformation in hilbert space","volume":"17","author":"BO Koopman","year":"1931","journal-title":"Proc Natl Acad Sci U S A"},{"issue":"3","key":"pcbi.1012620.ref070","doi-asserted-by":"crossref","first-page":"583","DOI":"10.1587\/transinf.E93.D.583","article-title":"Least-squares conditional density estimation","volume":"93","author":"M Sugiyama","year":"2010","journal-title":"IEICE Trans Inf Syst"},{"volume-title":"Adv Neural Inf Process Syst","year":"2016","author":"T Salimans","key":"pcbi.1012620.ref071"}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1012620","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,30]],"date-time":"2024-12-30T18:49:47Z","timestamp":1735584587000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1012620"}},"subtitle":[],"editor":[{"given":"Joshua","family":"Glaser","sequence":"first","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2024,12,30]]},"references-count":71,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2024,12,30]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1012620","relation":{},"ISSN":["1553-7358"],"issn-type":[{"type":"electronic","value":"1553-7358"}],"subject":[],"published":{"date-parts":[[2024,12,30]]}}}