{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,18]],"date-time":"2026-04-18T07:42:31Z","timestamp":1776498151472,"version":"3.51.2"},"reference-count":52,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,11,27]],"date-time":"2025-11-27T00:00:00Z","timestamp":1764201600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,11,27]],"date-time":"2025-11-27T00:00:00Z","timestamp":1764201600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100009482","name":"Santa Clara University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100009482","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["New Gener. Comput."],"published-print":{"date-parts":[[2026,2]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    The Computational Creativity literature stresses the importance of evaluation in computational creativity systems in addition to generative capabilities. But, what about machines that\n                    <jats:italic>only<\/jats:italic>\n                    evaluate? When it comes to co-creative systems, humans often take on the primary evaluative role, while machines assist with the generation of creative artifacts. In this paper, we propose flipping the paradigm, envisioning machines that (only) evaluate humans in the creative space. We call such machines Creative Evaluators whose end goal is to evaluate human-made artifacts and provides feedback to aid humans in the creative process, while the machine itself refrains from directly generating any content. We present a co-creative framework for designing Creative Evaluators, which includes the tension between originality and quality of creative artifacts, and the need for explainability and fairness in machines that take on an evaluative role for human-made artifacts. To illustrate this framework, we present\n                    <jats:italic>Titlevator<\/jats:italic>\n                    , a machine that evaluates titles for AI papers, while also providing feedback to the user on the originality and the quality of the paper being evaluated.\n                  <\/jats:p>","DOI":"10.1007\/s00354-025-00302-8","type":"journal-article","created":{"date-parts":[[2025,11,27]],"date-time":"2025-11-27T07:43:22Z","timestamp":1764229402000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Creative Evaluators: When Machines Evaluate Us"],"prefix":"10.1007","volume":"44","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9013-6902","authenticated-orcid":false,"given":"Kuakou","family":"Bossou","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Margareta","family":"Ackerman","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2025,11,27]]},"reference":[{"key":"302_CR1","unstructured":"Ventura, D.: Mere generation: essential barometer or dated concept. In: Proceedings of the seventh international conference on computational creativity, pp. 17\u201324. Sony CSL, Paris (2016)"},{"key":"302_CR2","unstructured":"Karimi, P., Grace, K., Maher, M.L., Davis, N.: Evaluating creativity in computational co-creative systems. arXiv preprint arXiv:1807.09886 (2018)"},{"key":"302_CR3","unstructured":"Kantosalo, A., Toivonen, H.: Modes for creative human-computer collaboration: Alternating and task-divided co-creativity. In: Proceedings of the seventh international conference on computational creativity, pp. 77\u201384 (2016)"},{"issue":"1","key":"302_CR4","doi-asserted-by":"publisher","first-page":"358","DOI":"10.1016\/j.ejor.2020.07.058","volume":"290","author":"S Chen","year":"2021","unstructured":"Chen, S., Guo, Z., Zhao, X.: Predicting mortgage early delinquency with machine learning methods. Eur. J. Oper. Res. 290(1), 358\u2013372 (2021)","journal-title":"Eur. J. Oper. Res."},{"key":"302_CR5","unstructured":"Santos, V.D., Verspoor, M., Nerbonne, J.: Identifying important factors in essay grading using machine learning. Selected papers in memory of Dr Pavlos Pavlou\u2014Language testing and assessment round the globe: Achievements and experiences. Frankfurt: Peter Lang. Available at http:\/\/urd.let.rug.nl\/nerbonne\/papers\/Santos_et_al-2012-grading.pdf (2012)"},{"key":"302_CR6","doi-asserted-by":"crossref","unstructured":"Ramalingam, V., Pandian, A., Chetry, P., Nigam, H.: Automated essay grading using machine learning algorithm. In: Journal of physics: conference series, vol. 1000, p. 012030. IOP Publishing, Bristol (2018)","DOI":"10.1088\/1742-6596\/1000\/1\/012030"},{"key":"302_CR7","doi-asserted-by":"crossref","unstructured":"Hu, A., Ma, S.: Human interactions and financial investment: a video-based approach. Available at SSRN (2020)","DOI":"10.2139\/ssrn.3583898"},{"key":"302_CR8","unstructured":"Wu, V., Gnanasambandam, C.: A machine-learning approach to venture capital. The McKinsey Quarterly (2017)"},{"key":"302_CR9","unstructured":"Bento, F.R.D.S.R.: Predicting start-up success with machine learning. PhD thesis (2018)"},{"key":"302_CR10","doi-asserted-by":"crossref","unstructured":"Veale, T., P\u00e9rez\u00a0y P\u00e9rez, R.: Leaps and bounds: an introduction to the field of computational creativity. New Gen. Comput. 38(4), 551\u2013563 (2020)","DOI":"10.1007\/s00354-020-00116-w"},{"issue":"6","key":"302_CR11","doi-asserted-by":"publisher","first-page":"265","DOI":"10.1002\/widm.1170","volume":"5","author":"H Toivonen","year":"2015","unstructured":"Toivonen, H., Gross, O.: Data mining and machine learning in computational creativity. Wiley Interdisc. Rev. Data Min. Knowl. Disc. 5(6), 265\u2013275 (2015)","journal-title":"Wiley Interdisc. Rev. Data Min. Knowl. Disc."},{"issue":"1","key":"302_CR12","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1007\/s11023-007-9066-2","volume":"17","author":"G Ritchie","year":"2007","unstructured":"Ritchie, G.: Some empirical criteria for attributing creativity to a computer program. Mind. Mach. 17(1), 67\u201399 (2007)","journal-title":"Mind. Mach."},{"key":"302_CR13","unstructured":"Colton, S., Charnley, J.W., Pease, A.: Computational creativity theory: The FACE and IDEA descriptive models. In: ICCC, pp. 90\u201395 (2011)"},{"issue":"3","key":"302_CR14","doi-asserted-by":"publisher","first-page":"246","DOI":"10.1007\/s12559-012-9156-1","volume":"4","author":"A Jordanous","year":"2012","unstructured":"Jordanous, A.: A standardised procedure for evaluating creative systems: computational creativity evaluation based on what it is to be creative. Cogn. Comput. 4(3), 246\u2013279 (2012)","journal-title":"Cogn. Comput."},{"issue":"3\u20134","key":"302_CR15","first-page":"125","volume":"3","author":"K Grace","year":"2015","unstructured":"Grace, K., Maher, M.L., Fisher, D., Brady, K.: Data-intensive evaluation of design creativity using novelty, value, and surprise. Int. J. Des. Creat. Innov. 3(3\u20134), 125\u2013147 (2015)","journal-title":"Int. J. Des. Creat. Innov."},{"issue":"10","key":"302_CR16","doi-asserted-by":"publisher","first-page":"0162959","DOI":"10.1371\/journal.pone.0162959","volume":"11","author":"A Jordanous","year":"2016","unstructured":"Jordanous, A., Keller, B.: Modelling creativity: identifying key components through a corpus-based approach. PLoS ONE 11(10), 0162959 (2016)","journal-title":"PLoS ONE"},{"key":"302_CR17","doi-asserted-by":"crossref","unstructured":"Boden, M.A.: The turing test and artistic creativity. Kybernetes (2010)","DOI":"10.1108\/03684921011036132"},{"key":"302_CR18","volume-title":"Aaron\u2019s Code: Meta-art, Artificial Intelligence, and the Work of Harold Cohen","author":"P McCorduck","year":"1991","unstructured":"McCorduck, P.: Aaron\u2019s Code: Meta-art, Artificial Intelligence, and the Work of Harold Cohen. Macmillan, New York (1991)"},{"issue":"2","key":"302_CR19","doi-asserted-by":"publisher","first-page":"69","DOI":"10.2307\/3680717","volume":"16","author":"D Cope","year":"1992","unstructured":"Cope, D.: Computer modeling of musical intelligence in emi. Comput. Music. J. 16(2), 69\u201383 (1992)","journal-title":"Comput. Music. J."},{"key":"302_CR20","unstructured":"Pease, A., Colton, S.: On impact and evaluation in computational creativity: A discussion of the turing test and an alternative proposal. In: Proceedings of the AISB symposium on AI and Philosophy, vol. 39 (2011)"},{"key":"302_CR21","unstructured":"P\u00e9rez\u00a0y P\u00e9rez, R.: The three layers evaluation model for computer-generated plots. In: ICCC, pp. 220\u2013229 (2014)"},{"key":"302_CR22","doi-asserted-by":"crossref","unstructured":"Ackerman, M., Loker, D.: Algorithmic songwriting with ALYSIA. In: International conference on evolutionary and biologically inspired music and art, pp. 1\u201316. Springer, New York (2017)","DOI":"10.1007\/978-3-319-55750-2_1"},{"key":"302_CR23","doi-asserted-by":"crossref","unstructured":"P\u00c9rez, R.P.\u00dd., Sharples, M.: Mexica: A computer model of a cognitive account of creative writing. J. Exp. Theor. Artif. Intell. 13(2), 119\u2013139 (2001)","DOI":"10.1080\/09528130010029820"},{"key":"302_CR24","doi-asserted-by":"crossref","unstructured":"Colton, S.: The painting fool: Stories from building an automated painter. In: Computers and Creativity, pp. 3\u201338. Springer, New York (2012)","DOI":"10.1007\/978-3-642-31727-9_1"},{"key":"302_CR25","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611974560","volume-title":"Credit Scoring and Its Applications","author":"L Thomas","year":"2017","unstructured":"Thomas, L., Crook, J., Edelman, D.: Credit Scoring and Its Applications. SIAM, Philadelphia (2017)"},{"key":"302_CR26","doi-asserted-by":"crossref","unstructured":"Leng, Y., Yu, L., Xiong, J.: Deepreviewer: Collaborative grammar and innovation neural network for automatic paper review. In: 2019 International conference on multimodal interaction, pp. 395\u2013403 (2019)","DOI":"10.1145\/3340555.3353766"},{"key":"302_CR27","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2020.116572","volume":"210","author":"W Zhang","year":"2020","unstructured":"Zhang, W., Sjoerds, Z., Hommel, B.: Metacontrol of human creativity: the neurocognitive mechanisms of convergent and divergent thinking. Neuroimage 210, 116572 (2020)","journal-title":"Neuroimage"},{"key":"302_CR28","doi-asserted-by":"crossref","unstructured":"Runco, M.A.: Divergent thinking, creativity, and ideation. (2010)","DOI":"10.1017\/CBO9780511763205.026"},{"key":"302_CR29","unstructured":"Colton, S., Llano, T., Hepworth, R., Charnley, J., Gale, C., Baron, A., Pachet, F., Roy, P., Gerv\u00e1s, P., Collins, N., et al.: The beyond the fence musical and computer says show documentary (2016)"},{"key":"302_CR30","unstructured":"Ventura, D.: How to build a CC system. In: ICCC, pp. 253\u2013260 (2017)"},{"key":"302_CR31","doi-asserted-by":"publisher","first-page":"2318","DOI":"10.1016\/j.procs.2020.03.284","volume":"167","author":"PK Roy","year":"2020","unstructured":"Roy, P.K., Chowdhary, S.S., Bhatia, R.: A machine learning approach for automation of resume recommendation system. Proc. Comput. Sci. 167, 2318\u20132327 (2020)","journal-title":"Proc. Comput. Sci."},{"key":"302_CR32","doi-asserted-by":"publisher","DOI":"10.4324\/9780203508527","volume-title":"The Creative Mind: Myths and Mechanisms","author":"MA Boden","year":"2004","unstructured":"Boden, M.A.: The Creative Mind: Myths and Mechanisms. Routledge, London (2004)"},{"issue":"2","key":"302_CR33","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3167476","volume":"51","author":"C Lamb","year":"2018","unstructured":"Lamb, C., Brown, D.G., Clarke, C.L.: Evaluating computational creativity: an interdisciplinary tutorial. ACM Comput. Surv. (CSUR) 51(2), 1\u201334 (2018)","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"302_CR34","unstructured":"Forbes: First-Movers Seldom Win, While First-Dominators Often Succeed And Achieve Staying-Power. https:\/\/www.forbes.com\/sites\/dileeprao\/2017\/01\/24\/first-movers-seldom-win-first-dominators-mainly-do\/?sh=f207e4319a12"},{"key":"302_CR35","unstructured":"Turnitin: Empower students to do their best, original work. https:\/\/www.turnitin.com\/"},{"issue":"3","key":"302_CR36","doi-asserted-by":"publisher","DOI":"10.1063\/1.3593373","volume":"21","author":"TP Hill","year":"2011","unstructured":"Hill, T.P., Miller, J.: How to combine independent data sets for the same quantity. Chaos Interdisc. J. Nonlinear Sci. 21(3), 033102 (2011)","journal-title":"Chaos Interdisc. J. Nonlinear Sci."},{"key":"302_CR37","unstructured":"Llano, M.T., d\u2019Inverno, M., Yee-King, M., McCormack, J., Ilsar, A., Pease, A., Colton, S.: Explainable computational creativity. In: Proceedings of ICCC (2020)"},{"key":"302_CR38","unstructured":"High-Level Expert Group on AI: Ethics Guidelines for Trustworthy AI. European Commission. https:\/\/ec.europa.eu\/futurium\/en\/ai-alliance-consultation.1.html"},{"key":"302_CR39","doi-asserted-by":"crossref","unstructured":"Gunning, D., Stefik, M., Choi, J., Miller, T., Stumpf, S., Yang, G.-Z.: Xai\u2014explainable artificial intelligence. Sci. Robot. 4(37) (2019)","DOI":"10.1126\/scirobotics.aay7120"},{"key":"302_CR40","doi-asserted-by":"crossref","unstructured":"Gunning, D., Vorm, E., Wang, J.Y., Turek, M.: DARPA\u2019s explainable AI. (XAI) Program, A Retrospective. Wiley Online Library (2021)","DOI":"10.22541\/au.163699841.19031727\/v1"},{"key":"302_CR41","unstructured":"Matt Turek: Explainable Artificial Intelligence (XAI). DARPA. https:\/\/www.darpa.mil\/program\/explainable-artificial-intelligence"},{"key":"302_CR42","unstructured":"Journal of Computational Creativity: About this Journal. Association for Computational Creativity. https:\/\/jcc.computationalcreativity.net\/"},{"issue":"3","key":"302_CR43","first-page":"10","volume":"74","author":"SE Fiarman","year":"2016","unstructured":"Fiarman, S.E.: Unconscious bias: when good intentions aren\u2019t enough. Educ. Leadersh. 74(3), 10\u201315 (2016)","journal-title":"Educ. Leadersh."},{"issue":"6","key":"302_CR44","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3457607","volume":"54","author":"N Mehrabi","year":"2021","unstructured":"Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Comput. Surv. (CSUR) 54(6), 1\u201335 (2021)","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"302_CR45","first-page":"4349","volume":"29","author":"T Bolukbasi","year":"2016","unstructured":"Bolukbasi, T., Chang, K.-W., Zou, J.Y., Saligrama, V., Kalai, A.T.: Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. Adv. Neural. Inf. Process. Syst. 29, 4349\u20134357 (2016)","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"302_CR46","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1007\/s11023-020-09529-4","volume":"31","author":"MSA Lee","year":"2021","unstructured":"Lee, M.S.A., Floridi, L.: Algorithmic fairness in mortgage lending: from absolute conditions to relational trade-offs. Mind. Mach. 31, 165\u2013191 (2021)","journal-title":"Mind. Mach."},{"key":"302_CR47","unstructured":"Rathore, A., Dev, S., Phillips, J.M., Srikumar, V., Zheng, Y., Yeh, C.-C.M., Wang, J., Zhang, W., Wang, B.: Verb: Visualizing and interpreting bias mitigation techniques for word representations. arXiv preprint arXiv:2104.02797 (2021)"},{"issue":"1","key":"302_CR48","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1177\/0165551513511393","volume":"40","author":"S Subotic","year":"2014","unstructured":"Subotic, S., Mukherjee, B.: Short and amusing: the relationship between title characteristics, downloads, and citations in psychology articles. J. Inf. Sci. 40(1), 115\u2013124 (2014)","journal-title":"J. Inf. Sci."},{"issue":"3","key":"302_CR49","doi-asserted-by":"publisher","first-page":"1331","DOI":"10.1007\/s11192-016-1937-y","volume":"107","author":"LB Nair","year":"2016","unstructured":"Nair, L.B., Gibbert, M.: What makes a \u2018good\u2019 title and (how) does it matter for citations? A review and general model of article title attributes in management science. Scientometrics 107(3), 1331\u20131359 (2016)","journal-title":"Scientometrics"},{"issue":"3","key":"302_CR50","doi-asserted-by":"publisher","first-page":"1573","DOI":"10.1007\/s11192-016-2216-7","volume":"110","author":"M Gnewuch","year":"2017","unstructured":"Gnewuch, M., Wohlrabe, K.: Characteristics and citations in economics. Scientometrics 110(3), 1573\u20131578 (2017)","journal-title":"Scientometrics"},{"key":"302_CR51","doi-asserted-by":"crossref","unstructured":"Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019)","DOI":"10.18653\/v1\/D19-1410"},{"key":"302_CR52","unstructured":"Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)"}],"container-title":["New Generation Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00354-025-00302-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00354-025-00302-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00354-025-00302-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,18]],"date-time":"2026-04-18T07:03:02Z","timestamp":1776495782000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00354-025-00302-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,27]]},"references-count":52,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,2]]}},"alternative-id":["302"],"URL":"https:\/\/doi.org\/10.1007\/s00354-025-00302-8","relation":{},"ISSN":["0288-3635","1882-7055"],"issn-type":[{"value":"0288-3635","type":"print"},{"value":"1882-7055","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,27]]},"assertion":[{"value":"21 January 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 September 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 November 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The code that supports the findings of this study is available in the GitHub repository,\n                      \n                      .","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Code availability"}}],"article-number":"3"}}