{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T23:11:17Z","timestamp":1740179477264,"version":"3.37.3"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T00:00:00Z","timestamp":1660694400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T00:00:00Z","timestamp":1660694400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"crossref","award":["DP160101122"],"award-info":[{"award-number":["DP160101122"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001786","name":"The University of Adelaide","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001786","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Comput Soc Sc"],"published-print":{"date-parts":[[2022,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The field of conflict forecasting has matured greatly over the last decade. Advances in machine learning have allowed researchers to forecast rare political and social events in near real time. Yet the maturity of the field has led to a proliferation of diverse platforms for forecasting, divergent results across forecasts, and an explosion of forecasting methodologies. While the field has done much to establish some baseline results, true, consensual benchmarks against which future forecasts may be evaluated remain elusive, and thus, agreed upon empirical results are still rare. The aim of this work is to address these concerns and provide the field of conflict forecasting with a standardized analysis pipeline to evaluate future forecasts of political violence. We aim to open the black box of the conflict forecasting pipeline and provide empirical evidence on how modeling decisions along all steps of the pipeline affect end results. In this way, we empirically demonstrate best practices that conflict forecasting researchers may utilize in future endeavors. We employ forecasts of targeted mass killings and genocides to support our methodological claims.<\/jats:p>","DOI":"10.1007\/s42001-022-00176-6","type":"journal-article","created":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T16:07:40Z","timestamp":1660752460000},"page":"1489-1510","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["From academia to policy makers: a methodology for real-time forecasting of infrequent events"],"prefix":"10.1007","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6730-0302","authenticated-orcid":false,"given":"Alfred","family":"Krzywicki","sequence":"first","affiliation":[]},{"given":"David","family":"Muchlinski","sequence":"additional","affiliation":[]},{"given":"Benjamin E.","family":"Goldsmith","sequence":"additional","affiliation":[]},{"given":"Arcot","family":"Sowmya","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,8,17]]},"reference":[{"issue":"2","key":"176_CR1","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1177\/0022343317691330","volume":"54","author":"H Hegre","year":"2017","unstructured":"Hegre, H., Metternich, N. W., Nyg\u00e5rd, H. M., & Wucherpfennig, J. (2017). Introduction: Forecasting in peace research. Journal of Peace Research, 54(2), 113\u2013124.","journal-title":"Journal of Peace Research"},{"key":"176_CR2","doi-asserted-by":"crossref","unstructured":"D\u2019Orazio, V., Honaker, J., Prasady, R., and Shoemate, M.: Modeling and forecasting armed conflict: AutoML with human-guided machine learning, In 2019 IEEE International Conference on Big Data (Big Data), pp. 4714\u20134723.","DOI":"10.1109\/BigData47090.2019.9005963"},{"issue":"4","key":"176_CR3","doi-asserted-by":"publisher","first-page":"437","DOI":"10.1177\/0022343313484167","volume":"50","author":"BE Goldsmith","year":"2013","unstructured":"Goldsmith, B. E., Butcher, C. R., Semenovich, D., & Sowmya, A. (2013). Forecasting the onset of genocide and politicide: Annual out-of-sample forecasts on a global dataset, 1988\u20132003. Journal of Peace Research, 50(4), 437\u2013452.","journal-title":"Journal of Peace Research"},{"key":"176_CR4","unstructured":"https:\/\/earlywarningproject.ushmm.org\/. Accessed 12 Aug 2022"},{"issue":"1","key":"176_CR5","doi-asserted-by":"publisher","first-page":"190","DOI":"10.1111\/j.1540-5907.2009.00426.x","volume":"54","author":"JA Goldstone","year":"2010","unstructured":"Goldstone, J. A., Bates, R. H., Epstein, D. L., Gurr, T. R., Lustik, M. B., Marshall, M. G., Ulfelder, J., & Woodward, M. (2010). A global model for forecasting political instability. American Journal of Political Science, 54(1), 190\u2013208.","journal-title":"American Journal of Political Science"},{"issue":"3","key":"176_CR6","doi-asserted-by":"publisher","first-page":"205316801455751","DOI":"10.1177\/2053168014557511","volume":"1","author":"A Beger","year":"2014","unstructured":"Beger, A., Dorff, C. L., & Ward, M. D. (2014). Ensemble forecasting of irregular leadership change. Research & Politics, 1(3), 2053168014557511.","journal-title":"Research & Politics"},{"key":"176_CR7","doi-asserted-by":"publisher","DOI":"10.1016\/j.pdisas.2020.100069","volume":"6","author":"M Halkia","year":"2020","unstructured":"Halkia, M., Ferri, S., Schellens, M. K., Papazoglou, M., & Thomakos, D. (2020). The global conflict risk index: A quantitative tool for policy support on conflict prevention. Progress in Disaster Science, 6, 100069.","journal-title":"Progress in Disaster Science"},{"issue":"1","key":"176_CR8","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1080\/14623528.2017.1379631","volume":"20","author":"BE Goldsmith","year":"2018","unstructured":"Goldsmith, B. E., & Butcher, C. (2018). Genocide forecasting: Past accuracy and new forecasts to 2020. Journal of Genocide Research, 20(1), 90\u2013107.","journal-title":"Journal of Genocide Research"},{"issue":"3","key":"176_CR9","doi-asserted-by":"publisher","first-page":"289","DOI":"10.1214\/10-STS330","volume":"25","author":"G Shmueli","year":"2010","unstructured":"Shmueli, G. (2010). To explain or to predict? Statistical science, 25(3), 289\u2013310.","journal-title":"Statistical science"},{"key":"176_CR10","doi-asserted-by":"publisher","DOI":"10.1177\/0022002719896405","author":"C Butcher","year":"2020","unstructured":"Butcher, C., Goldsmith, B. E., Nanlohy, S., Sowmya, A., & Muchlinski, D. (2020). Introducing the targeted mass killing data set for the study and forecasting of mass atrocities. Journal of Conflict Resolution. https:\/\/doi.org\/10.1177\/0022002719896405","journal-title":"Journal of Conflict Resolution"},{"key":"176_CR11","doi-asserted-by":"publisher","DOI":"10.1093\/pan\/mpv024","author":"D Muchlinski","year":"2016","unstructured":"Muchlinski, D., Siroky, D., He, J., & Kocher, M. (2016). Comparing random forest with logistic regression for predicting class-imbalanced civil war onset data. Political Analysis. https:\/\/doi.org\/10.1093\/pan\/mpv024","journal-title":"Political Analysis"},{"key":"176_CR12","first-page":"305","volume-title":"The best writing on mathematics","author":"A Gelman","year":"2016","unstructured":"Gelman, A., & Loken, E. (2016). The statistical crisis in science. In M. Pitici (Ed.), The best writing on mathematics (pp. 305\u2013318). Princeton University Press."},{"issue":"2","key":"176_CR13","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1177\/0022343316682065","volume":"54","author":"M Colaresi","year":"2017","unstructured":"Colaresi, M., & Mahmood, Z. (2017). Do the robot: Lessons from machine learning to improve conflict forecasting. Journal of Peace Research, 54(2), 193\u2013214.","journal-title":"Journal of Peace Research"},{"key":"176_CR14","unstructured":"Koyejo, O., Natarajan, N., Ravikumar, P., and Dhillon, I.S. (2014). Consistent binary classification with generalized performance metrics. Advances in neural information processing systems, 3, 2744\u20132752."},{"issue":"3","key":"176_CR15","doi-asserted-by":"publisher","first-page":"e0118432","DOI":"10.1371\/journal.pone.0118432","volume":"10","author":"T Saito","year":"2015","unstructured":"Saito, T., & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PloS One, 10(3), e0118432.","journal-title":"PloS One"},{"key":"176_CR16","doi-asserted-by":"crossref","unstructured":"Davis, J., and Goadrich, M. (2006). The relationship between Precision-Recall and ROC curves. In Proceedings of the 23rd international conference on machine learning, pp. 233\u2013240.","DOI":"10.1145\/1143844.1143874"},{"key":"176_CR17","volume-title":"An introduction to generalized linear models","author":"AJ Dobson","year":"2018","unstructured":"Dobson, A. J., & Barnett, A. G. (2018). An introduction to generalized linear models. CRC Press."},{"issue":"3","key":"176_CR18","first-page":"297","volume":"1","author":"T Hastie","year":"1986","unstructured":"Hastie, T., & Tibshirani, R. (1986). Generalized additive models. Statistical Science, 1(3), 297\u2013310.","journal-title":"Statistical Science"},{"issue":"1","key":"176_CR19","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/A:1010933404324","volume":"45","author":"L Breiman","year":"2001","unstructured":"Breiman, L. (2001). Random forests. Machine learning, 45(1), 5\u201332.","journal-title":"Machine learning"},{"issue":"10","key":"176_CR20","doi-asserted-by":"publisher","first-page":"1390","DOI":"10.1177\/0010414016666858","volume":"50","author":"C Butcher","year":"2017","unstructured":"Butcher, C., & Goldsmith, B. E. (2017). Elections, ethnicity, and political instability. Comparative Political Studies, 50(10), 1390\u20131419.","journal-title":"Comparative Political Studies"},{"key":"176_CR21","unstructured":"https:\/\/www.systemicpeace.org\/inscrdata.html. Accessed 12 Aug 2022"},{"key":"176_CR22","volume-title":"Codebook for the dyadic militarized interstate dispute data, version 3.10","author":"F Ghosn","year":"2003","unstructured":"Ghosn, F., & Bennett, S. (2003). Codebook for the dyadic militarized interstate dispute data, version 3.10. The Pennsylvania State University."},{"issue":"1","key":"176_CR23","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1017\/S0043887109990219","volume":"62","author":"L-E Cederman","year":"2010","unstructured":"Cederman, L.-E., Wimmer, A., & Min, B. (2010). Why do ethnic groups rebel? New data and analysis. World Politics, 62(1), 87\u2013119.","journal-title":"World Politics"},{"issue":"2","key":"176_CR24","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1177\/0022343316680858","volume":"54","author":"MD Ward","year":"2017","unstructured":"Ward, M. D., & Beger, A. (2017). Lessons from near real-time forecasting of irregular leadership changes. Journal of Peace Research, 54(2), 141\u2013156.","journal-title":"Journal of Peace Research"},{"issue":"6","key":"176_CR25","doi-asserted-by":"publisher","first-page":"651","DOI":"10.1080\/13645579.2020.1729974","volume":"23","author":"S Gorard","year":"2020","unstructured":"Gorard, S. (2020). Handling missing data in numeric analyses. International Journal of Social Research Methodology, 23(6), 651\u2013660.","journal-title":"International Journal of Social Research Methodology"},{"key":"176_CR26","doi-asserted-by":"publisher","DOI":"10.1201\/9780429492259","volume-title":"Flexible imputation of missing data","author":"S Van Buuren","year":"2018","unstructured":"Van Buuren, S. (2018). Flexible imputation of missing data. CRC Press."},{"key":"176_CR27","volume-title":"Classification and regression trees","author":"L Breiman","year":"1984","unstructured":"Breiman, L., Friedman, J., Stone, C. J., & Olshen, R. A. (1984). Classification and regression trees. CRC Press."},{"key":"176_CR28","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v045.i03","author":"S Van Buuren","year":"2010","unstructured":"Van Buuren, S., & Groothuis-Oudshoorn, K. (2010). MICE: Multivariate imputation by chained equations in R. Journal of Statistical Software. https:\/\/doi.org\/10.18637\/jss.v045.i03","journal-title":"Journal of Statistical Software"},{"key":"176_CR29","unstructured":"Alzola, C., and Harrell, F. (2006). An introduction to S and the Hmisc and design libraries. http:\/\/biostat.mc.vanderbilt.edu\/wiki\/pub\/Main\/RS\/sintro.pdf."},{"key":"176_CR30","doi-asserted-by":"publisher","first-page":"321","DOI":"10.1613\/jair.953","volume":"16","author":"NV Chawla","year":"2002","unstructured":"Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: Synthetic minority over-sampling technique. Journal of artificial intelligence research, 16, 321\u2013357.","journal-title":"Journal of artificial intelligence research"},{"issue":"1","key":"176_CR31","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1007\/s10618-012-0295-5","volume":"28","author":"G Menardi","year":"2014","unstructured":"Menardi, G., & Torelli, N. (2014). Training and assessing classification rules with imbalanced data. Data Mining and Knowledge Discovery, 28(1), 92\u2013122.","journal-title":"Data Mining and Knowledge Discovery"},{"issue":"5","key":"176_CR32","doi-asserted-by":"publisher","first-page":"649","DOI":"10.1177\/0022343315585611","volume":"52","author":"R Kennedy","year":"2015","unstructured":"Kennedy, R. (2015). Making useful conflict predictions: Methods for addressing skewed classes and implementing cost-sensitive learning in the study of state failure. Journal of Peace Research, 52(5), 649\u2013664.","journal-title":"Journal of Peace Research"}],"container-title":["Journal of Computational Social Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42001-022-00176-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s42001-022-00176-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42001-022-00176-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,26]],"date-time":"2022-11-26T18:47:03Z","timestamp":1669488423000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s42001-022-00176-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,17]]},"references-count":32,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,11]]}},"alternative-id":["176"],"URL":"https:\/\/doi.org\/10.1007\/s42001-022-00176-6","relation":{},"ISSN":["2432-2717","2432-2725"],"issn-type":[{"type":"print","value":"2432-2717"},{"type":"electronic","value":"2432-2725"}],"subject":[],"published":{"date-parts":[[2022,8,17]]},"assertion":[{"value":"28 November 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 July 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 August 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The author(s) declared no potential conflicts of interest with respect to the research, authorship, and\/or publication of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"All datasets used in this paper are publically available. The sources of data are listed or referenced in Table 2 of the paper.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Data availability"}}]}}