{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,27]],"date-time":"2025-08-27T10:10:21Z","timestamp":1756289421990,"version":"3.44.0"},"reference-count":50,"publisher":"BMJ","issue":"1","funder":[{"DOI":"10.13039\/100011272","name":"FP7 Health","doi-asserted-by":"publisher","award":["grant agreement No: 603098"],"award-info":[{"award-number":["grant agreement No: 603098"]}],"id":[{"id":"10.13039\/100011272","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["Evid Based Mental Health"],"accepted":{"date-parts":[[2020,1,10]]},"published-print":{"date-parts":[[2020,2]]},"abstract":"<jats:p>\n                  <jats:bold>Background<\/jats:bold> Self-reported client assessments during online treatments enable the development of statistical models for the prediction of client improvement and symptom development. Evaluation of these models is mandatory to ensure their validity.<\/jats:p>\n               <jats:p>\n                  <jats:bold>Methods<\/jats:bold> For this purpose, we suggest besides a model evaluation based on study data the use of a simulation analysis. The simulation analysis provides insight into the model performance and enables to analyse reasons for a low predictive accuracy. In this study, we evaluate a temporal causal model (TCM) and show that it does not provide reliable predictions of clients\u2019 future mood levels.<\/jats:p>\n               <jats:p>\n                  <jats:bold>Results<\/jats:bold> Based on the simulation analysis we investigate the potential reasons for the low predictive performance, for example, noisy measurements and sampling frequency. We conclude that the analysed TCM in its current form is not sufficient to describe the underlying psychological processes.<\/jats:p>\n               <jats:p>\n                  <jats:bold>Conclusions<\/jats:bold> The results demonstrate the importance of model evaluation and the benefit of a simulation analysis. The current manuscript provides practical guidance for conducting model evaluation including simulation analysis.<\/jats:p>","DOI":"10.1136\/ebmental-2019-300135","type":"journal-article","created":{"date-parts":[[2020,2,11]],"date-time":"2020-02-11T11:35:20Z","timestamp":1581420920000},"page":"27-33","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":3,"title":["Evaluation of a temporal causal model for predicting the mood of clients in an online therapy"],"prefix":"10.1136","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1437-6127","authenticated-orcid":false,"given":"Dennis","family":"Becker","sequence":"first","affiliation":[{"name":"Institute of Information Systems, Leuphana University of L\u00fcneburg, Luneburg, Germany"}]},{"given":"Vincent","family":"Bremer","sequence":"additional","affiliation":[{"name":"Institute of Information Systems, Leuphana University of L\u00fcneburg, Luneburg, Germany"}]},{"given":"Burkhardt","family":"Funk","sequence":"additional","affiliation":[{"name":"Institute of Information Systems, Leuphana University of L\u00fcneburg, Luneburg, Germany"}]},{"given":"Mark","family":"Hoogendoorn","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands"}]},{"given":"Artur","family":"Rocha","sequence":"additional","affiliation":[{"name":"Centre for Information Systems and Computer Graphics, INESC TEC, Porto, Portugal"}]},{"given":"Heleen","family":"Riper","sequence":"additional","affiliation":[{"name":"Department of Clinical, Neuro- & Developmental Psychology, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands"}]}],"member":"239","published-online":{"date-parts":[[2020,2,11]]},"reference":[{"key":"2025082702385336000_23.1.27.1","doi-asserted-by":"publisher","DOI":"10.1192\/bjp.bp.108.054429"},{"key":"2025082702385336000_23.1.27.2","first-page":"24","article-title":"The outcome Questionnaire-45","volume":"2","author":"Lambert","year":"2012","journal-title":"Integrating Science and Practice"},{"key":"2025082702385336000_23.1.27.3","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1002\/da.22646","article-title":"Sensing behavioral symptoms of mental health and delivering personalized interventions using mobile technologies","volume":"34","author":"Aung","year":"2017","journal-title":"Depress Anxiety"},{"key":"2025082702385336000_23.1.27.4","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1146\/annurev-clinpsy-032816-044949","article-title":"Personal sensing: understanding mental health using ubiquitous sensors and machine learning","volume":"13","author":"Mohr","year":"2017","journal-title":"Annu Rev Clin Psychol"},{"key":"2025082702385336000_23.1.27.5","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.128.6.934"},{"key":"2025082702385336000_23.1.27.6","doi-asserted-by":"publisher","DOI":"10.1207\/S15324796ABM2602_06"},{"key":"2025082702385336000_23.1.27.7","doi-asserted-by":"publisher","DOI":"10.1016\/j.cpr.2012.05.007"},{"key":"2025082702385336000_23.1.27.8","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2016.02029"},{"key":"2025082702385336000_23.1.27.9","doi-asserted-by":"crossref","first-page":"496","DOI":"10.1177\/1073191116638735","article-title":"Enhancing the Personalization of psychotherapy with dynamic assessment and modeling","volume":"23","author":"Fisher","year":"2016","journal-title":"Assessment"},{"key":"2025082702385336000_23.1.27.10","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0162755"},{"key":"2025082702385336000_23.1.27.11","doi-asserted-by":"crossref","DOI":"10.1136\/bmj.b375","article-title":"Prognosis and prognostic research: what, why, and how?","volume":"338","author":"Moons","year":"2009","journal-title":"BMJ"},{"key":"2025082702385336000_23.1.27.12","doi-asserted-by":"crossref","first-page":"887","DOI":"10.1038\/ijo.2015.214","article-title":"The importance of prediction model validation and assessment in obesity and nutrition research","volume":"40","author":"Ivanescu","year":"2016","journal-title":"Int J Obes"},{"key":"2025082702385336000_23.1.27.13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s41512-018-0033-6","article-title":"Evaluating the impact of prediction models: lessons learned, challenges, and recommendations","volume":"2","author":"Kappen","year":"2018","journal-title":"Diagnostic and Prognostic Research"},{"key":"2025082702385336000_23.1.27.14","doi-asserted-by":"crossref","DOI":"10.1136\/bmj.b605","article-title":"Prognosis and prognostic research: validating a prognostic model","volume":"338","author":"Altman","year":"2009","journal-title":"BMJ"},{"key":"2025082702385336000_23.1.27.15","doi-asserted-by":"crossref","DOI":"10.1136\/bmj.b606","article-title":"Prognosis and prognostic research: application and impact of prognostic models in clinical practice","volume":"338","author":"Moons","year":"2009","journal-title":"BMJ"},{"key":"2025082702385336000_23.1.27.16","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pmed.1001381","article-title":"Prognosis research strategy (progress) 3: prognostic model research","volume":"10","author":"Steyerberg","year":"2013","journal-title":"PLoS Med"},{"key":"2025082702385336000_23.1.27.17","doi-asserted-by":"publisher","DOI":"10.1093\/eurheartj\/ehu207"},{"key":"2025082702385336000_23.1.27.18","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pmed.1001221","article-title":"Reporting and methods in clinical prediction research: a systematic review","volume":"9","author":"Bouwmeester","year":"2012","journal-title":"PLoS Med"},{"key":"2025082702385336000_23.1.27.19","first-page":"361","article-title":"Altaf Hussain Abro MK. validation of a computational model for mood and social integration","volume":"2016","year":"2016","journal-title":"Lecture Notes in Computer Science"},{"key":"2025082702385336000_23.1.27.20","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1016\/j.bica.2016.02.002","article-title":"Dynamic modeling based on a temporal\u2013causal network modeling approach","volume":"16","author":"Treur","year":"2016","journal-title":"Biologically Inspired Cognitive Architectures"},{"key":"2025082702385336000_23.1.27.21","first-page":"27","volume-title":"Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics","volume":"vol. 9875","author":"EFdM","year":"2016"},{"key":"2025082702385336000_23.1.27.22","unstructured":"Treur J , Ziabari SSM . Computational collective intelligence. vol. 11056. Springer International Publishing, 2018."},{"key":"2025082702385336000_23.1.27.23","unstructured":"Franke A , Hosain RW . A Temporal-Causal model for spread of messages in disasters 2017;10449."},{"key":"2025082702385336000_23.1.27.24","doi-asserted-by":"crossref","DOI":"10.1186\/s13063-016-1511-1","article-title":"European comparative effectiveness research on blended depression treatment versus treatment-as-usual (E-COMPARED): study protocol for a randomized controlled, non-inferiority trial in eight European countries","volume":"17","author":"Kleiboer","year":"2016","journal-title":"Trials"},{"key":"2025082702385336000_23.1.27.25","first-page":"347","article-title":"Assessment of temporal predictive models for health care using a formal method","volume":"2017","author":"van Breda","year":"2016","journal-title":"Computers in Biology and Medicine"},{"key":"2025082702385336000_23.1.27.26","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1016\/j.invent.2017.10.001","article-title":"Predicting short term mood developments among depressed patients using adherence and ecological momentary assessment data","volume":"12","author":"Mikus","year":"2018","journal-title":"Internet Interventions"},{"key":"2025082702385336000_23.1.27.27","unstructured":"Jaques N , Taylor S , Sano A , et al . Predicting Tomorrow\u2019s Mood, Health, and Stress Level using Personalized Multitask Learning and Domain Adaptation 2017."},{"key":"2025082702385336000_23.1.27.28","doi-asserted-by":"crossref","unstructured":"Constantinides M , Busk J , Matic A , et al . Personalized versus Generic Mood Prediction Models in Bipolar Disorder. In: Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers - UbiComp \u201918. New York, New York, USA: ACM Press, 2018: 1700\u20137.","DOI":"10.1145\/3267305.3267536"},{"key":"2025082702385336000_23.1.27.29","unstructured":"Sudman S , Bradburn NM , Schwarz N . Thinking about answers: the application of cognitive processes to survey methodology. San Francisco, CA, US: Jossey-Bass, 1996."},{"key":"2025082702385336000_23.1.27.30","unstructured":"Clark HH , Schober MF . Asking questions and influencing answers. in: questions about questions: inquiries into the cognitive bases of surveys. New York, NY, US: Russell Sage Foundation, 1992: 15\u201348."},{"key":"2025082702385336000_23.1.27.31","doi-asserted-by":"publisher","DOI":"10.1037\/a0026733"},{"key":"2025082702385336000_23.1.27.32","doi-asserted-by":"publisher","DOI":"10.1093\/ntr\/ntt154"},{"key":"2025082702385336000_23.1.27.33","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1093\/icvts\/ivy102","article-title":"Statistical primer: how to deal with missing data in scientific research?\u2020","volume":"27","author":"Papageorgiou","year":"2018","journal-title":"Interact Cardiovasc Thorac Surg"},{"key":"2025082702385336000_23.1.27.34","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2648.2008.04959.x"},{"key":"2025082702385336000_23.1.27.35","doi-asserted-by":"publisher","DOI":"10.1192\/bjp.bp.115.169094"},{"key":"2025082702385336000_23.1.27.36","doi-asserted-by":"publisher","DOI":"10.1097\/00006842-200205000-00005"},{"key":"2025082702385336000_23.1.27.37","doi-asserted-by":"publisher","DOI":"10.1037\/0021-843X.109.1.11"},{"key":"2025082702385336000_23.1.27.38","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.67.6.1101"},{"key":"2025082702385336000_23.1.27.39","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1037\/a0015416","article-title":"Depression and everyday social activity","volume":"56","author":"Steger","year":"2010","journal-title":"J Couns Psychol"},{"key":"2025082702385336000_23.1.27.40","doi-asserted-by":"publisher","DOI":"10.1037\/a0017216"},{"key":"2025082702385336000_23.1.27.41","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1177\/0898264316673511","article-title":"Social isolation, depression, and psychological distress among older adults","volume":"30","author":"Taylor","year":"2018","journal-title":"J Aging Health"},{"key":"2025082702385336000_23.1.27.42","first-page":"43","volume-title":"Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics","volume":"9545","author":"Spanakis","year":"2016"},{"key":"2025082702385336000_23.1.27.43","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1155\/2019\/3481624","article-title":"Heterogeneity matters: predicting self-esteem in online interventions based on ecological Momentary assessment data","volume":"2019","author":"Bremer","year":"2019","journal-title":"Depress Res Treat"},{"key":"2025082702385336000_23.1.27.44","doi-asserted-by":"publisher","DOI":"10.1109\/4235.996017"},{"key":"2025082702385336000_23.1.27.45","doi-asserted-by":"crossref","unstructured":"Parducci A . Category judgment: a range-frequency model. US: American Psychological Association, 1965.","DOI":"10.1037\/h0022602"},{"key":"2025082702385336000_23.1.27.46","doi-asserted-by":"crossref","unstructured":"Daamen DDL , de Bie SE . Serial context effects in survey interviews. in: context effects in social and psychological research. New York, NY: Springer New York 1992:97\u2013113.","DOI":"10.1007\/978-1-4612-2848-6_8"},{"key":"2025082702385336000_23.1.27.47","doi-asserted-by":"publisher","DOI":"10.1037\/0096-3445.134.1.3"},{"key":"2025082702385336000_23.1.27.48","doi-asserted-by":"crossref","first-page":"578","DOI":"10.1207\/s15328007sem1204_4","article-title":"Issues in evaluating model fit with missing data","volume":"12","author":"Davey","year":"2005","journal-title":"Struct Equ Modeling: A Multidisciplin J"},{"key":"2025082702385336000_23.1.27.49","doi-asserted-by":"crossref","first-page":"609","DOI":"10.1111\/add.14503","article-title":"Compliance with ecological momentary assessment protocols in substance users: a meta\u2010analysis","volume":"114","author":"Jones","year":"2019","journal-title":"Addiction"},{"key":"2025082702385336000_23.1.27.50","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.b2393"}],"container-title":["Evidence Based Mental Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/ebmental-2019-300135","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,27]],"date-time":"2025-08-27T09:39:08Z","timestamp":1756287548000},"score":1,"resource":{"primary":{"URL":"https:\/\/mentalhealth.bmj.com\/lookup\/doi\/10.1136\/ebmental-2019-300135"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,2]]},"references-count":50,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2020,2,11]]},"published-print":{"date-parts":[[2020,2]]}},"alternative-id":["10.1136\/ebmental-2019-300135"],"URL":"https:\/\/doi.org\/10.1136\/ebmental-2019-300135","relation":{},"ISSN":["1362-0347","1468-960X"],"issn-type":[{"type":"print","value":"1362-0347"},{"type":"electronic","value":"1468-960X"}],"subject":[],"published":{"date-parts":[[2020,2]]}}}