{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T07:11:42Z","timestamp":1777360302952,"version":"3.51.4"},"reference-count":38,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2022,9,1]],"date-time":"2022-09-01T00:00:00Z","timestamp":1661990400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGEVOlution"],"published-print":{"date-parts":[[2022,9]]},"abstract":"<jats:p>The top image shows a set of scales, which are intended to bring to mind the ideas of balance and fair experimentation which are the focus of our article on genetic programming benchmarks in this issue. Image by Elena Mozhvilo and made available under the Unsplash license on https:\/\/unsplash.com\/photos\/j06gLuKK0GM.<\/jats:p>","DOI":"10.1145\/3578482.3578483","type":"journal-article","created":{"date-parts":[[2022,12,23]],"date-time":"2022-12-23T18:26:08Z","timestamp":1671819968000},"page":"1-19","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Genetic programming benchmarks"],"prefix":"10.1145","volume":"15","author":[{"given":"James","family":"McDermott","sequence":"first","affiliation":[{"name":"University of Galway, Ireland"}]},{"given":"Gabriel","family":"Kronberger","sequence":"additional","affiliation":[{"name":"University of Applied Sciences Upper Austria"}]},{"given":"Patryk","family":"Orzechowski","sequence":"additional","affiliation":[{"name":"University of Pennsylvania and AGH University of Science and Technology, Krakow, Poland"}]},{"given":"Leonardo","family":"Vanneschi","sequence":"additional","affiliation":[{"name":"Universidade Nova de Lisboa, Lisboa; Portugal"}]},{"given":"Luca","family":"Manzoni","sequence":"additional","affiliation":[{"name":"University of Trieste, Trieste, Italy"}]},{"given":"Roman","family":"Kalkreuth","sequence":"additional","affiliation":[{"name":"Leiden University, Netherlands"}]},{"given":"Mauro","family":"Castelli","sequence":"additional","affiliation":[{"name":"Universidade Nova de Lisboa, Lisboa; Portugal"}]}],"member":"320","published-online":{"date-parts":[[2022,12,23]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 2623--2631)","author":"Akiba T.","unstructured":"Akiba , T. , Sano , S. , Yanase , T. , Ohta , T. and Koyama , M ., 2019, July. Optuna: A next-generation hyperparameter optimization framework . In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 2623--2631) . Akiba, T., Sano, S., Yanase, T., Ohta, T. and Koyama, M., 2019, July. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 2623--2631)."},{"key":"e_1_2_1_2_1","volume-title":"International Conference on Learning Representations. ICLR","author":"Balog M.","year":"2017","unstructured":"Balog , M. , Gaunt , A. L. , Brockschmidt , M. , Nowozin , S. , & Tarlow , D. ( 2017 ). Deepcoder: Learning to write programs . In International Conference on Learning Representations. ICLR Balog, M., Gaunt, A. L., Brockschmidt, M., Nowozin, S., & Tarlow, D. (2017). Deepcoder: Learning to write programs. In International Conference on Learning Representations. ICLR"},{"key":"e_1_2_1_3_1","volume-title":"Late Breaking Papers","author":"Burke E. K.","year":"2009","unstructured":"Burke , E. K. , Towards the decathlon challenge of search heuristics.\" Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference : Late Breaking Papers . 2009 . Burke, E. K., et al. \"Towards the decathlon challenge of search heuristics.\" Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers. 2009."},{"key":"e_1_2_1_4_1","volume-title":"Model selection and multimodel inference. A practical information-theoretic approach","author":"Burnham K. P.","year":"1998","unstructured":"Burnham , K. P. Model selection and multimodel inference. A practical information-theoretic approach ( 1998 ). Burnham, K. P. Model selection and multimodel inference. A practical information-theoretic approach (1998)."},{"key":"e_1_2_1_5_1","volume-title":"International Conference on Learning Representations. ICLR","author":"Chen X.","year":"2018","unstructured":"Chen , X. , Liu , C. , & Song , D. ( 2018 , ). Execution-guided neural program synthesis . In International Conference on Learning Representations. ICLR Chen, X., Liu, C., & Song, D. (2018,). Execution-guided neural program synthesis. In International Conference on Learning Representations. ICLR"},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference Companion (pp. 209--210)","author":"Christie L.A.","unstructured":"Christie , L.A. , Brownlee , A.E. and Woodward , J.R ., 2018, July. Investigating benchmark correlations when comparing algorithms with parameter tuning . In Proceedings of the Genetic and Evolutionary Computation Conference Companion (pp. 209--210) . Christie, L.A., Brownlee, A.E. and Woodward, J.R., 2018, July. Investigating benchmark correlations when comparing algorithms with parameter tuning. In Proceedings of the Genetic and Evolutionary Computation Conference Companion (pp. 209--210)."},{"key":"e_1_2_1_7_1","volume-title":"September","author":"Cranmer M.","year":"2020","unstructured":"Cranmer , M. , Py SR : Fast & parallelized symbolic regression in Python\/Julia , September 2020 . 10.5281\/zenodo.4041459 Cranmer, M., PySR: Fast & parallelized symbolic regression in Python\/Julia, September 2020. 10.5281\/zenodo.4041459"},{"key":"e_1_2_1_8_1","volume-title":"International conference on machine learning (pp. 990--998)","author":"Devlin J.","year":"2017","unstructured":"Devlin , J. , Uesato , J. , Bhupatiraju , S. , Singh , R. , Mohamed , A. R. , & Kohli , P. ( 2017 ). Robustfill: Neural program learning under noisy i\/o . In International conference on machine learning (pp. 990--998) . ICML. Devlin, J., Uesato, J., Bhupatiraju, S., Singh, R., Mohamed, A. R., & Kohli, P. (2017). Robustfill: Neural program learning under noisy i\/o. In International conference on machine learning (pp. 990--998). ICML."},{"key":"e_1_2_1_9_1","doi-asserted-by":"crossref","unstructured":"Forsyth R. 1981. BEAGLE - A Darwinian approach to pattern recognition. Kybernetes.  Forsyth R. 1981. BEAGLE - A Darwinian approach to pattern recognition. Kybernetes.","DOI":"10.1108\/eb005587"},{"key":"e_1_2_1_10_1","unstructured":"GitHub Team 2022. GitHub Copilot. https:\/\/github.com\/features\/copilot  GitHub Team 2022. GitHub Copilot. https:\/\/github.com\/features\/copilot"},{"issue":"3","key":"e_1_2_1_11_1","doi-asserted-by":"crossref","first-page":"520","DOI":"10.1109\/TBC.2015.2459851","article-title":"Quality assessment considering viewing distance and image resolution","volume":"61","author":"Gu K.","year":"2015","unstructured":"Gu , K. , Liu , M. , Zhai , G. , Yang , X. and Zhang , W. , 2015 . Quality assessment considering viewing distance and image resolution . IEEE Transactions on Broadcasting , 61 ( 3 ), pp. 520 -- 531 . Gu, K., Liu, M., Zhai, G., Yang, X. and Zhang, W., 2015. Quality assessment considering viewing distance and image resolution. IEEE Transactions on Broadcasting, 61(3), pp.520--531.","journal-title":"IEEE Transactions on Broadcasting"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference (pp. 785--794)","author":"Helmuth T.","unstructured":"Helmuth , T. and Kelly , P ., 2021, June. PSB2: the second program synthesis benchmark suite . In Proceedings of the Genetic and Evolutionary Computation Conference (pp. 785--794) . Helmuth, T. and Kelly, P., 2021, June. PSB2: the second program synthesis benchmark suite. In Proceedings of the Genetic and Evolutionary Computation Conference (pp. 785--794)."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation (pp. 1039--1046)","author":"Helmuth T.","unstructured":"Helmuth , T. and Spector , L ., 2015, July. General program synthesis benchmark suite . In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation (pp. 1039--1046) . Helmuth, T. and Spector, L., 2015, July. General program synthesis benchmark suite. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation (pp. 1039--1046)."},{"key":"e_1_2_1_14_1","volume-title":"European conference on genetic programming (pp. 35--51)","author":"Izzo D.","unstructured":"Izzo , D. , Biscani , F. and Mereta , A ., 2017, April. Differentiable genetic programming . In European conference on genetic programming (pp. 35--51) . Springer, Cham. Izzo, D., Biscani, F. and Mereta, A., 2017, April. Differentiable genetic programming. In European conference on genetic programming (pp. 35--51). Springer, Cham."},{"key":"e_1_2_1_15_1","volume-title":"Genetic Programming: On the Programming of Computers by Means of Natural Selection","author":"Koza J. R.","year":"1992","unstructured":"Koza , J. R. , 1992 . Genetic Programming: On the Programming of Computers by Means of Natural Selection . MIT Press . Koza, J. R., 1992. Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press."},{"key":"e_1_2_1_16_1","unstructured":"Lehman J. etal \"The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities.\" Artificial life 26.2 (2020): 274--306.  Lehman J. et al. \"The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities.\" Artificial life 26.2 (2020): 274--306."},{"key":"e_1_2_1_17_1","volume-title":"Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).","author":"Liao T.","unstructured":"Liao , T. , Taori , R. , Raji , I.D. and Schmidt , L ., 2021, August. Are we learning yet? a meta review of evaluation failures across machine learning . In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). Liao, T., Taori, R., Raji, I.D. and Schmidt, L., 2021, August. Are we learning yet? a meta review of evaluation failures across machine learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)."},{"key":"e_1_2_1_18_1","volume-title":"1--18","author":"McDermott J.","year":"2020","unstructured":"McDermott , J. \" When and why metaheuristics researchers can ignore \"No Free Lunch\" theorems.\" SN Computer Science 1.1 ( 2020 ): 1--18 . McDermott, J. \"When and why metaheuristics researchers can ignore \"No Free Lunch\" theorems.\" SN Computer Science 1.1 (2020): 1--18."},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1016\/j.patrec.2016.01.004","article-title":"Diagnosing a disorder in a classification benchmark","volume":"73","author":"McDermott J.","year":"2016","unstructured":"McDermott , J. and Forsyth , R.S. , 2016 . Diagnosing a disorder in a classification benchmark . Pattern Recognition Letters , 73 , pp. 41 -- 43 . McDermott, J. and Forsyth, R.S., 2016. Diagnosing a disorder in a classification benchmark. Pattern Recognition Letters, 73, pp.41--43.","journal-title":"Pattern Recognition Letters"},{"issue":"1","key":"e_1_2_1_20_1","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1007\/s10710-019-09360-6","article-title":"Cartesian genetic programming: its status and future","volume":"21","author":"Miller J.F.","year":"2020","unstructured":"Miller , J.F. , 2020 . Cartesian genetic programming: its status and future . Genetic Programming and Evolvable Machines , 21 ( 1 ), pp. 129 -- 168 . Miller, J.F., 2020. Cartesian genetic programming: its status and future. Genetic Programming and Evolvable Machines, 21(1), pp.129--168.","journal-title":"Genetic Programming and Evolvable Machines"},{"key":"e_1_2_1_21_1","volume-title":"Mind Children","author":"Moravec H","year":"1988","unstructured":"Moravec , H ( 1988 ), Mind Children , Harvard University Press Moravec, H (1988), Mind Children, Harvard University Press"},{"key":"e_1_2_1_22_1","volume-title":"2015 IEEE Congress on Evolutionary Computation (CEC) (pp. 1152--1159)","author":"Nicolau M.","year":"2015","unstructured":"Nicolau , M. , Agapitos , A. , O'Neill , M. , & Brabazon , A. ( 2015 , May). Guidelines for defining benchmark problems in genetic programming . In 2015 IEEE Congress on Evolutionary Computation (CEC) (pp. 1152--1159) . IEEE. Nicolau, M., Agapitos, A., O'Neill, M., & Brabazon, A. (2015, May). Guidelines for defining benchmark problems in genetic programming. In 2015 IEEE Congress on Evolutionary Computation (CEC) (pp. 1152--1159). IEEE."},{"issue":"4","key":"e_1_2_1_23_1","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1007\/s10710-017-9309-9","article-title":"Understanding grammatical evolution: initialisation","volume":"18","author":"Nicolau M.","year":"2017","unstructured":"Nicolau , M. , 2017 . Understanding grammatical evolution: initialisation . Genetic Programming and Evolvable Machines , 18 ( 4 ), pp. 467 -- 507 . Nicolau, M., 2017. Understanding grammatical evolution: initialisation. Genetic Programming and Evolvable Machines, 18(4), pp.467--507.","journal-title":"Genetic Programming and Evolvable Machines"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference Companion (pp. 1342--1349)","author":"Oliveira L.O.V.","unstructured":"Oliveira , L.O.V. , Martins , J.F.B. , Miranda , L.F. and Pappa , G.L ., 2018, July. Analysing symbolic regression benchmarks under a meta-learning approach . In Proceedings of the Genetic and Evolutionary Computation Conference Companion (pp. 1342--1349) . Oliveira, L.O.V., Martins, J.F.B., Miranda, L.F. and Pappa, G.L., 2018, July. Analysing symbolic regression benchmarks under a meta-learning approach. In Proceedings of the Genetic and Evolutionary Computation Conference Companion (pp. 1342--1349)."},{"key":"e_1_2_1_25_1","volume-title":"Patryk Orzechowski, Ryan J. Urbanowicz, and Jason H. Moore.","author":"Olson R. S.","year":"2017","unstructured":"Olson , R. S. , William La Cava , Patryk Orzechowski, Ryan J. Urbanowicz, and Jason H. Moore. 2017 . PMLB: a large benchmark suite for machine learning evaluation and comparison. BioData Mining 10, 36 (11 Dec 2017), 1--13. 10.1186\/s13040-017-0154-4 Olson, R. S., William La Cava, Patryk Orzechowski, Ryan J. Urbanowicz, and Jason H. Moore. 2017. PMLB: a large benchmark suite for machine learning evaluation and comparison. BioData Mining 10, 36 (11 Dec 2017), 1--13. 10.1186\/s13040-017-0154-4"},{"key":"e_1_2_1_26_1","volume-title":"In International Conference on Learning Representations. ICLR.","author":"Parisotto E.","year":"2017","unstructured":"Parisotto , E. , Mohamed , A. R. , Singh , R. , Li , L. , Zhou , D. , & Kohli , P. ( 2017 ). In International Conference on Learning Representations. ICLR. Parisotto, E., Mohamed, A. R., Singh, R., Li, L., Zhou, D., & Kohli, P. (2017). In International Conference on Learning Representations. ICLR."},{"key":"e_1_2_1_27_1","first-page":"2825","volume-title":"Scikit-learn: Machine learning in Python. the Journal of machine Learning research, 12","author":"Pedregosa F.","year":"2011","unstructured":"Pedregosa , F. , Varoquaux , G. , Gramfort , A. , Michel , V. , Thirion , B. , Grisel , O. , Blondel , M. , Prettenhofer , P. , Weiss , R. , Dubourg , V. and Vanderplas , J. , 2011 . Scikit-learn: Machine learning in Python. the Journal of machine Learning research, 12 , pp. 2825 -- 2830 . Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V. and Vanderplas, J., 2011. Scikit-learn: Machine learning in Python. the Journal of machine Learning research, 12, pp.2825--2830."},{"key":"e_1_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Ponomarenko N. Jin L. Ieremeiev O. Lukin V. Egiazarian K. Astola J. Vozel B. Chehdi K. Carli M. Battisti F. and Kuo C.C.J. 2015. Image database TID2013: Peculiarities results and perspectives. Signal processing: Image communication 30 pp.57--77.  Ponomarenko N. Jin L. Ieremeiev O. Lukin V. Egiazarian K. Astola J. Vozel B. Chehdi K. Carli M. Battisti F. and Kuo C.C.J. 2015. Image database TID2013: Peculiarities results and perspectives. Signal processing: Image communication 30 pp.57--77.","DOI":"10.1016\/j.image.2014.10.009"},{"key":"e_1_2_1_29_1","unstructured":"Roelofs R. Shankar V. Recht B. Fridovich-Keil S. Hardt M. Miller J. and Schmidt L. 2019. A meta-analysis of overfitting in machine learning. Advances in Neural Information Processing Systems 32.  Roelofs R. Shankar V. Recht B. Fridovich-Keil S. Hardt M. Miller J. and Schmidt L. 2019. A meta-analysis of overfitting in machine learning. Advances in Neural Information Processing Systems 32."},{"key":"e_1_2_1_30_1","volume-title":"European conference on genetic programming (pp. 83--96)","author":"Ryan C.","year":"1998","unstructured":"Ryan , C. , Collins , J.J. and O'Neill , M. , 1998 , April. Grammatical evolution: Evolving programs for an arbitrary language . In European conference on genetic programming (pp. 83--96) . Springer, Berlin, Heidelberg. Ryan, C., Collins, J.J. and O'Neill, M., 1998, April. Grammatical evolution: Evolving programs for an arbitrary language. In European conference on genetic programming (pp. 83--96). Springer, Berlin, Heidelberg."},{"key":"e_1_2_1_31_1","doi-asserted-by":"crossref","unstructured":"Salustowicz R. and Schmidhuber J. 1997. Probabilistic incremental program evolution. Evolutionary computation 5(2) pp.123--141.  Salustowicz R. and Schmidhuber J. 1997. Probabilistic incremental program evolution. Evolutionary computation 5(2) pp.123--141.","DOI":"10.1162\/evco.1997.5.2.123"},{"key":"e_1_2_1_32_1","doi-asserted-by":"crossref","unstructured":"Schmidt M. and Lipson H. 2009. Distilling free-form natural laws from experimental data. science 324(5923) pp.81--85.  Schmidt M. and Lipson H. 2009. Distilling free-form natural laws from experimental data. science 324 (5923) pp.81--85.","DOI":"10.1126\/science.1165893"},{"key":"e_1_2_1_33_1","volume-title":"Choose your programming copilot: a comparison of the program synthesis performance of github copilot and genetic programming.\" Proceedings of the Genetic and Evolutionary Computation Conference","author":"Sobania D., M.","year":"2022","unstructured":"Sobania , D., M. Briesch , and F. Rothlauf . \" Choose your programming copilot: a comparison of the program synthesis performance of github copilot and genetic programming.\" Proceedings of the Genetic and Evolutionary Computation Conference . 2022 . Sobania, D., M. Briesch, and F. Rothlauf. \"Choose your programming copilot: a comparison of the program synthesis performance of github copilot and genetic programming.\" Proceedings of the Genetic and Evolutionary Computation Conference. 2022."},{"key":"e_1_2_1_34_1","unstructured":"Stephens T. 2016. Genetic Programming in Python With a Scikit-Learn Inspired API: gplearn. https:\/\/gplearn.readthedocs.io.  Stephens T. 2016. Genetic Programming in Python With a Scikit-Learn Inspired API: gplearn. https:\/\/gplearn.readthedocs.io."},{"key":"e_1_2_1_35_1","doi-asserted-by":"crossref","unstructured":"Udrescu S.-M. and M. Tegmark. AI Feynman: A physics-inspired method for symbolic regression. Science Advances 6.16 (2020): eaay2631.  Udrescu S.-M. and M. Tegmark. AI Feynman: A physics-inspired method for symbolic regression. Science Advances 6.16 (2020): eaay2631.","DOI":"10.1126\/sciadv.aay2631"},{"key":"e_1_2_1_36_1","doi-asserted-by":"crossref","unstructured":"Urbanowicz R. J. J. Kiralis N. A. Sinnott-Armstrong T. Heberling J. M. Fisher and J. H. Moore. \"GAMETES: a fast direct algorithm for generating pure strict epistatic models with random architectures.\" BioData mining 5 no. 1 (2012): 1--14.  Urbanowicz R. J. J. Kiralis N. A. Sinnott-Armstrong T. Heberling J. M. Fisher and J. H. Moore. \"GAMETES: a fast direct algorithm for generating pure strict epistatic models with random architectures.\" BioData mining 5 no. 1 (2012): 1--14.","DOI":"10.1186\/1756-0381-5-16"},{"issue":"2","key":"e_1_2_1_37_1","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1145\/2641190.2641198","article-title":"OpenML: networked science in machine learning","volume":"15","author":"Vanschoren J.","year":"2014","unstructured":"Vanschoren , J. , Van Rijn , J.N. , Bischl , B. and Torgo , L. , 2014 . OpenML: networked science in machine learning . ACM SIGKDD Explorations Newsletter , 15 ( 2 ), pp. 49 -- 60 . Vanschoren, J., Van Rijn, J.N., Bischl, B. and Torgo, L., 2014. OpenML: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2), pp.49--60.","journal-title":"ACM SIGKDD Explorations Newsletter"},{"issue":"2","key":"e_1_2_1_38_1","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1109\/TEVC.2008.926486","article-title":"Order of nonlinearity as a complexity measure for models generated by symbolic regression via pareto genetic programming","volume":"13","author":"Vladislavleva E.J.","year":"2008","unstructured":"Vladislavleva , E.J. , Smits , G.F. and Den Hertog , D. , 2008 . Order of nonlinearity as a complexity measure for models generated by symbolic regression via pareto genetic programming . IEEE Transactions on Evolutionary Computation , 13 ( 2 ), pp. 333 -- 349 . Vladislavleva, E.J., Smits, G.F. and Den Hertog, D., 2008. Order of nonlinearity as a complexity measure for models generated by symbolic regression via pareto genetic programming. IEEE Transactions on Evolutionary Computation, 13(2), pp.333--349.","journal-title":"IEEE Transactions on Evolutionary Computation"}],"container-title":["ACM SIGEVOlution"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3578482.3578483","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3578482.3578483","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:46:53Z","timestamp":1750178813000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3578482.3578483"}},"subtitle":["looking back and looking forward"],"short-title":[],"issued":{"date-parts":[[2022,9]]},"references-count":38,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,9]]}},"alternative-id":["10.1145\/3578482.3578483"],"URL":"https:\/\/doi.org\/10.1145\/3578482.3578483","relation":{},"ISSN":["1931-8499"],"issn-type":[{"value":"1931-8499","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9]]},"assertion":[{"value":"2022-12-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}