{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,14]],"date-time":"2026-07-14T04:00:20Z","timestamp":1784001620976,"version":"3.55.0"},"reference-count":18,"publisher":"Springer Science and Business Media LLC","issue":"20","license":[{"start":{"date-parts":[[2023,7,26]],"date-time":"2023-07-26T00:00:00Z","timestamp":1690329600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,7,26]],"date-time":"2023-07-26T00:00:00Z","timestamp":1690329600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Appl Intell"],"published-print":{"date-parts":[[2023,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Despite recent breakthroughs in the domain of implicit generative models, the task of evaluating these models remains a challenging task. With no single metric to assess overall performance, various existing metrics only offer partial information. This issue is further compounded for unintuitive data types such as time series, where manual inspection is infeasible. This deficiency hinders the confident application of modern implicit generative models on time series data. To alleviate this problem, we propose two new metrics, the InceptionTime Score (ITS) and the Fr\u00e9chet InceptionTime Distance (FITD), to assess the quality of class-conditional generative models on time series data. We conduct extensive experiments on 80 different datasets to study the discriminative capabilities of proposed metrics alongside two existing evaluation metrics: Train on Synthetic Test on Real (TSTR) and Train on Real Test on Synthetic (TRTS). Our evaluations reveal that the proposed assessment evaluation metrics, i.e., ITS and FITD in combination with TSTR, can accurately assess class-conditional generative model performance and detect common issues in implicit generative models. Our findings suggest that the proposed evaluation framework can be a valuable tool for confidently applying modern implicit generative models in time series analysis.<\/jats:p>","DOI":"10.1007\/s10489-023-04644-y","type":"journal-article","created":{"date-parts":[[2023,7,26]],"date-time":"2023-07-26T08:02:23Z","timestamp":1690358543000},"page":"24530-24550","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Quantifying quality of class-conditional generative models in time series domain"],"prefix":"10.1007","volume":"53","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7370-9369","authenticated-orcid":false,"given":"Alireza","family":"Koochali","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maria","family":"Walch","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sankrutyayan","family":"Thota","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter","family":"Schichtel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andreas","family":"Dengel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sheraz","family":"Ahmed","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,7,26]]},"reference":[{"key":"4644_CR1","unstructured":"Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial Nets. In: Adv Neural Inf Process Syst 27. Curran Associates Inc. pp 2672\u20132681"},{"key":"4644_CR2","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1016\/j.cviu.2018.10.009","volume":"179","author":"A Borji","year":"2019","unstructured":"Borji A (2019) Pros and cons of gan evaluation measures. Comput Vis Image Underst 179:41\u201365","journal-title":"Comput Vis Image Underst"},{"key":"4644_CR3","doi-asserted-by":"publisher","first-page":"103329","DOI":"10.1016\/j.cviu.2021.103329","volume":"215","author":"A Borji","year":"2022","unstructured":"Borji A (2022) Pros and cons of gan evaluation measures: New developments. Comput Vis Image Underst 215:103329","journal-title":"Comput Vis Image Underst"},{"key":"4644_CR4","unstructured":"Esteban C, Hyland SL, R\u00e4tsch G (2017) Real-valued (medical) time series generation with recurrent conditional gans. arXiv preprint arXiv:1706.02633"},{"key":"4644_CR5","doi-asserted-by":"publisher","unstructured":"Wiese M, Bai L, Wood B, Buehler H (2019) Deep hedging: learning to simulate equity option markets. https:\/\/doi.org\/10.2139\/ssrn.3470756","DOI":"10.2139\/ssrn.3470756"},{"key":"4644_CR6","doi-asserted-by":"crossref","unstructured":"Ni H, Szpruch L, Wiese M, Liao S, Xiao B (2020) Conditional sig-wasserstein gans for time series generation. arXiv preprint arXiv:2006.05421","DOI":"10.2139\/ssrn.3623086"},{"key":"4644_CR7","doi-asserted-by":"publisher","unstructured":"Gatta F, Giampaolo F, Prezioso E, Mei G, Cuomo S, Piccialli F (2022) Neural networks generative models for time series. J King Saud Univ-Comput Inf Sci 34:7920\u20137939. https:\/\/doi.org\/10.1016\/j.jksuci.2022.07.010","DOI":"10.1016\/j.jksuci.2022.07.010"},{"issue":"1","key":"4644_CR8","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1214\/aoms\/1177729694","volume":"22","author":"S Kullback","year":"1951","unstructured":"Kullback S, Leibler RA (1951) On information and sufficiency. Ann Math Stat 22(1):79\u201386","journal-title":"Ann Math Stat"},{"key":"4644_CR9","unstructured":"Yoon J, Jarrett D, Van\u00a0der Schaar M (2019) Time-series generative adversarial networks. in: Adv Neural Inf Process Syst 32: Annu Conf Neural Inf Pocess Syst 2019 NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada pp 5509\u20135519"},{"key":"4644_CR10","doi-asserted-by":"crossref","unstructured":"Shmelkov K, Schmid C, Alahari K (2018) How good is my gan? In: Proceedings of the European conference on computer vision (ECCV). pp 213\u2013229","DOI":"10.1007\/978-3-030-01216-8_14"},{"key":"4644_CR11","unstructured":"Ravuri SV, Vinyals O (2019) Classification accuracy score for conditional generative models. in: Adv Neural Inf Process Syst 32: Annu Conf Neural Inf Process Syst 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada. pp 12247\u201312258"},{"key":"4644_CR12","doi-asserted-by":"crossref","unstructured":"Li X, Metsis V, Wang H, Ngu AHH (2022) Tts-gan: A transformer-based time-series generative adversarial network. In: Artificial Intelligence in Medicine: 20th International Conference on Artificial Intelligence in Medicine, AIME 2022, Halifax, NS, Canada, June 14\u201317, 2022, Proceedings. Springer, pp 133\u2013143","DOI":"10.1007\/978-3-031-09342-5_13"},{"issue":"1","key":"4644_CR13","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1109\/18.61115","volume":"37","author":"J Lin","year":"1991","unstructured":"Lin J (1991) Divergence measures based on the shannon entropy. IEEE Trans Inf Theory 37(1):145\u2013151","journal-title":"IEEE Trans Inf Theory"},{"key":"4644_CR14","unstructured":"Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S (2017) GANs trained by a two time-scale update rule converge to a local nash equilibrium. In: Adv Neural Inf Process Syst 30: Annu Conf Neural Inf Process Syst 2017, December 4-9, 2017, Long Beach, CA, USA. pp 6626\u20136637"},{"key":"4644_CR15","unstructured":"Smith KE, Smith AO (2020) Conditional gan for timeseries generation. arXiv preprint arXiv:2006.16477"},{"key":"4644_CR16","doi-asserted-by":"crossref","unstructured":"Ismail Fawaz H, Lucas B, Forestier G, Pelletier C, Schmidt DF, Weber J, Webb GI, Idoumghar L, Muller PA, Petitjean F (2020) Inceptiontime: Finding alexnet for time series classification. Data Min Knowl Discov 34(6):1936\u20131962","DOI":"10.1007\/s10618-020-00710-y"},{"key":"4644_CR17","unstructured":"Chen Y, Keogh E, Hu B, Begum N, Bagnall A, Mueen A, Batista G (2015) The UCR time series classification archive. https:\/\/www.cs.ucr.edu\/~eamonn\/time_series_data\/"},{"key":"4644_CR18","unstructured":"Arora S, Ge R, Liang Y, Ma T, Zhang Y (2017) Generalization and equilibrium in generative adversarial nets (gans). In: International Conference on Machine Learning. PMLR, pp 224\u2013232"}],"container-title":["Applied Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10489-023-04644-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10489-023-04644-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10489-023-04644-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,21]],"date-time":"2023-10-21T16:26:59Z","timestamp":1697905619000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10489-023-04644-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,26]]},"references-count":18,"journal-issue":{"issue":"20","published-print":{"date-parts":[[2023,10]]}},"alternative-id":["4644"],"URL":"https:\/\/doi.org\/10.1007\/s10489-023-04644-y","relation":{},"ISSN":["0924-669X","1573-7497"],"issn-type":[{"value":"0924-669X","type":"print"},{"value":"1573-7497","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,26]]},"assertion":[{"value":"15 April 2023","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 July 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no relevant financial or non-financial interests to disclose.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}},{"value":"All authors agreed on the content of paper for publication.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}}]}}