{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,8]],"date-time":"2026-02-08T13:11:31Z","timestamp":1770556291250,"version":"3.49.0"},"reference-count":88,"publisher":"Springer Science and Business Media LLC","issue":"1-2","license":[{"start":{"date-parts":[[2024,6,1]],"date-time":"2024-06-01T00:00:00Z","timestamp":1717200000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,10,5]],"date-time":"2024-10-05T00:00:00Z","timestamp":1728086400000},"content-version":"vor","delay-in-days":126,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100019180","name":"HORIZON EUROPE European Research Council","doi-asserted-by":"publisher","award":["01002802,"],"award-info":[{"award-number":["01002802,"]}],"id":[{"id":"10.13039\/100019180","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003130","name":"Fonds Wetenschappelijk Onderzoek","doi-asserted-by":"publisher","award":["G070521N"],"award-info":[{"award-number":["G070521N"]}],"id":[{"id":"10.13039\/501100003130","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Onderzoeksprogramma Artifici\u00eble Intellgentie (AI) Vlaanderen","award":["G0G3220N"],"award-info":[{"award-number":["G0G3220N"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Constraints"],"published-print":{"date-parts":[[2024,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We consider the problem of <jats:italic>perception-based constraint solving<\/jats:italic>, where part of the problem specification is provided <jats:italic>indirectly<\/jats:italic> through an image provided by a user. As a pedagogical example, we use the complete image of a Sudoku grid. While the rules of the puzzle are assumed to be known, the image must be interpreted by a neural network to extract the values in the grid. In this paper, we investigate (1) <jats:italic>a hybrid modeling approach<\/jats:italic> combining machine learning and constraint solving for <jats:italic>joint inference<\/jats:italic>, knowing that blank cells need to be both predicted as being blank and filled-in to obtain a full solution; (2) the effect of <jats:italic>classifier calibration<\/jats:italic> on joint inference; and (3) how to deal with cases where the constraints of the reasoning system are not satisfied. More specifically, in the case of handwritten <jats:italic>user errors<\/jats:italic> in the image, a naive approach fails to obtain a feasible solution even if the interpretation is correct. Our framework <jats:italic>identifies<\/jats:italic> human mistakes by using a constraint solver and helps the user to <jats:italic>correct<\/jats:italic> these mistakes. We evaluate the performance of the proposed techniques on images taken through the Sudoku Assistant Android app, among other datasets. Our experiments show that (1) joint inference can correct classifier mistakes, (2) overall calibration improves the solution quality on all datasets, and (3) estimating and discriminating between user-written and original visual input while reasoning makes for a more robust system, even in the presence of user errors.<\/jats:p>","DOI":"10.1007\/s10601-024-09372-9","type":"journal-article","created":{"date-parts":[[2024,10,5]],"date-time":"2024-10-05T04:01:32Z","timestamp":1728100892000},"page":"112-151","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Perception-based constraint solving for sudoku images"],"prefix":"10.1007","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9122-926X","authenticated-orcid":false,"given":"Maxime","family":"Mulamba","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8675-8178","authenticated-orcid":false,"given":"Jayanta","family":"Mandi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8770-8567","authenticated-orcid":false,"given":"Ali \u0130rfan","family":"Mahmuto\u011fullar\u0131","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2156-2155","authenticated-orcid":false,"given":"Tias","family":"Guns","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,10,5]]},"reference":[{"issue":"1","key":"9372_CR1","doi-asserted-by":"publisher","first-page":"255","DOI":"10.1146\/annurev.cs.01.060186.001351","volume":"1","author":"HJ Levesque","year":"1986","unstructured":"Levesque, H. J. (1986). Knowledge representation and reasoning. Annual Review of Computer Science, 1(1), 255\u2013287.","journal-title":"Annual Review of Computer Science"},{"issue":"1","key":"9372_CR2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1017\/S0140525X00052432","volume":"11","author":"P Smolensky","year":"1988","unstructured":"Smolensky, P. (1988). On the proper treatment of connectionism. Behavioral and Brain Sciences, 11(1), 1\u201323.","journal-title":"Behavioral and Brain Sciences"},{"issue":"01","key":"9372_CR3","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1109\/MC.1987.1663364","volume":"20","author":"SE Fahlman","year":"1987","unstructured":"Fahlman, S. E., & Hinton, G. E. (1987). Connectionist architectures for artificial intelligence. Computer, 20(01), 100\u2013109.","journal-title":"Computer"},{"key":"9372_CR4","doi-asserted-by":"crossref","unstructured":"Zhao, Z. Q., Zheng, P., Xu, S. t., & Wu, X. (2019). Object detection with deep learning: A review. IEEE Transactions on Neural Networks and Learning Systems,30(11), 3212\u20133232.","DOI":"10.1109\/TNNLS.2018.2876865"},{"key":"9372_CR5","doi-asserted-by":"crossref","unstructured":"Graves, A.,\u00a0Jaitly, N., & Mohamed, A.r. (2013). Hybrid speech recognition with deep bidirectional LSTM. In 2013 IEEE workshop on automatic speech recognition and understanding (IEEE) (pp. 273\u2013278).","DOI":"10.1109\/ASRU.2013.6707742"},{"key":"9372_CR6","doi-asserted-by":"publisher","unstructured":"He, K.,\u00a0Zhang, X.,\u00a0Ren, S., &\u00a0Sun, J. (2016). Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 (IEEE Computer Society) (pp. 770\u2013778). https:\/\/doi.org\/10.1109\/CVPR.2016.90","DOI":"10.1109\/CVPR.2016.90"},{"key":"9372_CR7","doi-asserted-by":"crossref","unstructured":"Deng, L.,\u00a0Yu, D., et\u00a0al. (2014). Deep learning: Methods and applications. Foundations and trends\u00ae in signal processing, 7(3\u20134), 197\u2013387.","DOI":"10.1561\/2000000039"},{"key":"9372_CR8","unstructured":"Marcus, G. (2020). The next decade in ai: Four steps towards robust artificial intelligence. arXiv:2002.06177"},{"key":"9372_CR9","doi-asserted-by":"crossref","unstructured":"Zellers, R.,\u00a0Yatskar, M.,\u00a0Thomson, S., &\u00a0Choi, Y. (2018). Neural motifs: scene graph parsing with global context. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 (Computer Vision Foundation \/ IEEE Computer Society) (pp. 5831\u20135840).","DOI":"10.1109\/CVPR.2018.00611"},{"issue":"11","key":"9372_CR10","doi-asserted-by":"publisher","first-page":"12387","DOI":"10.1007\/s10462-023-10448-w","volume":"56","author":"ALC d\u2019Avila Garcez","year":"2023","unstructured":"d\u2019Avila Garcez, A. L. C. (2023). Lamb, neurosymbolic AI: The 3rd wave. Artificial Intelligence Review, 56(11), 12387\u201312406.","journal-title":"Artificial Intelligence Review"},{"key":"9372_CR11","doi-asserted-by":"publisher","first-page":"42200","DOI":"10.1109\/ACCESS.2020.2976199","volume":"8","author":"R Roscher","year":"2020","unstructured":"Roscher, R., Bohn, B., Duarte, M. F., & Garcke, J. (2020). Explainable machine learning for scientific insights and discoveries. Ieee Access, 8, 42200\u201342216.","journal-title":"Ieee Access"},{"key":"9372_CR12","unstructured":"Palm, R.,\u00a0Paquet, U., &\u00a0Winther, O. (2018). Recurrent relational networks. Advances in neural information processing systems, 31."},{"key":"9372_CR13","unstructured":"Wang, P., Donti, P. L.,\u00a0Wilder, B., & Kolter, J. Z. (2019). SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver. In ICML, Proceedings of machine learning research, vol\u00a097 (PMLR) (pp. 6545\u20136554)."},{"key":"9372_CR14","doi-asserted-by":"publisher","unstructured":"Brouard, C.,\u00a0de\u00a0Givry, S., &\u00a0Schiex, T. (2020). Pushing data into CP models using graphical model learning and solving. In H.\u00a0Simonis (Ed), Principles and practice of constraint programming - 26th international conference, CP 2020, Louvain-la-Neuve, Belgium, September 7-11, 2020, Proceedings, Lecture Notes in Computer Science, vol. 12333 (Springer) (pp. 811\u2013827). https:\/\/doi.org\/10.1007\/978-3-030-58475-7_47","DOI":"10.1007\/978-3-030-58475-7_47"},{"key":"9372_CR15","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2021.103522","volume":"299","author":"J Suchan","year":"2021","unstructured":"Suchan, J., Bhatt, M., & Varadarajan, S. (2021). Commonsense visual sensemaking for autonomous driving-on generalised neurosymbolic online abduction integrating vision and semantics. Artificial Intelligence, 299, 103522.","journal-title":"Artificial Intelligence"},{"key":"9372_CR16","unstructured":"Akkaya, I.,\u00a0Andrychowicz, M.,\u00a0Chociej, M.,\u00a0Litwin, M.,\u00a0McGrew, B.,\u00a0Petron, A.,\u00a0Paino, A.,\u00a0Plappert, M.,\u00a0Powell, G.,\u00a0Ribas, R., et\u00a0al. (2019). Solving rubik\u2019s cube with a robot hand. arXiv:1910.07113"},{"key":"9372_CR17","doi-asserted-by":"publisher","unstructured":"Mulamba, M.,\u00a0Mandi, J.,\u00a0Canoy, R., &\u00a0Guns, T. (2020). Hybrid classification and reasoning for image-based constraint solving. In E.\u00a0Hebrard, & N.\u00a0Musliu (Eds.), Integration of constraint programming, artificial intelligence, and operations research - 17th international conference, CPAIOR 2020, Vienna, Austria, September 21-24, 2020, Proceedings, Lecture Notes in Computer Science, vol. 12296 (Springer), pp. 364\u2013380.https:\/\/doi.org\/10.1007\/978-3-030-58942-4_24","DOI":"10.1007\/978-3-030-58942-4_24"},{"issue":"6","key":"9372_CR18","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1109\/MSP.2012.2211477","volume":"29","author":"L Deng","year":"2012","unstructured":"Deng, L. (2012). The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6), 141\u2013142.","journal-title":"IEEE Signal Processing Magazine"},{"key":"9372_CR19","unstructured":"Guns, T.,\u00a0Gamba, E.,\u00a0Mulamba, M.,\u00a0Bleukx, I.,\u00a0Berden, S., &\u00a0Pesa, M. (2022). Sudoku assistant. https:\/\/play.google.com\/store\/apps\/details?id=be.kuleuven.cs.visualsudoku"},{"key":"9372_CR20","doi-asserted-by":"crossref","unstructured":"Marra, G.,\u00a0Duman\u010di\u0107, S.,\u00a0Manhaeve, R., &\u00a0De\u00a0Raedt, L. (2024). From statistical relational to neurosymbolic artificial intelligence: A survey. Artificial Intelligence, 104062.","DOI":"10.1016\/j.artint.2023.104062"},{"key":"9372_CR21","doi-asserted-by":"crossref","unstructured":"Cappart, Q.,\u00a0Ch\u00e9telat, D., Khalil, E. B.,\u00a0Lodi, A.,\u00a0Morris, C., &\u00a0Velickovic, P. (2021). Combinatorial optimization and reasoning with graph neural networks. In IJCAI (ijcai.org) (pp. 4348\u20134355).","DOI":"10.24963\/ijcai.2021\/595"},{"key":"9372_CR22","doi-asserted-by":"crossref","unstructured":"Kotary, J.,\u00a0Fioretto, F., Hentenryck, P. V., &\u00a0Wilder, B. (2021). End-to-end constrained optimization learning: A survey. In IJCAI (ijcai.org) (pp. 4475\u20134482).","DOI":"10.24963\/ijcai.2021\/610"},{"key":"9372_CR23","doi-asserted-by":"publisher","first-page":"3569","DOI":"10.1109\/TIP.2019.2963378","volume":"29","author":"M Paumard","year":"2020","unstructured":"Paumard, M., Picard, D., & Tabia, H. (2020). Deepzzle: Solving visual jigsaw puzzles with deep learning and shortest path optimization. IEEE Transactions on Image Processing, 29, 3569\u20133581.","journal-title":"IEEE Transactions on Image Processing"},{"key":"9372_CR24","doi-asserted-by":"crossref","unstructured":"Noroozi, M., &\u00a0Favaro, P. (2016). Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV (6), Lecture notes in computer science, vol. 9910 (Springer) (pp. 69\u201384).","DOI":"10.1007\/978-3-319-46466-4_5"},{"key":"9372_CR25","unstructured":"Asai, M.,\u00a0Kajino, H.,\u00a0Fukunaga, A., &\u00a0Muise, C. (2021). In Neuro-symbolic artificial intelligence, frontiers in artificial intelligence and applications, vol. 342 (IOS Press) (pp. 52\u201377)."},{"key":"9372_CR26","unstructured":"Mao, J.,\u00a0Gan, C.,\u00a0Kohli, P., Tenenbaum, J. B., &\u00a0Wu, J. (2019). The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In International conference on learning representations. https:\/\/openreview.net\/forum?id=rJgMlhRctm"},{"key":"9372_CR27","doi-asserted-by":"crossref","unstructured":"Stammer, W.,\u00a0Schramowski, P., &\u00a0Kersting, K. (2021). Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations. In CVPR (Computer Vision Foundation \/ IEEE) (pp. 3619\u20133629).","DOI":"10.1109\/CVPR46437.2021.00362"},{"key":"9372_CR28","unstructured":"Shindo, H., Dhami, D. S., &\u00a0Kersting, K. (2021). Neuro-symbolic forward reasoning. CoRR arxiv:2110.09383"},{"key":"9372_CR29","doi-asserted-by":"crossref","unstructured":"Manigrasso, F., Miro, F. D.,\u00a0Morra, L., &\u00a0Lamberti, F. (2021). Faster-LTN: A neuro-symbolic, end-to-end object detection architecture. In ICANN (2), Lecture notes in computer science, vol. 12892 (Springer) (pp. 40\u201352).","DOI":"10.1007\/978-3-030-86340-1_4"},{"key":"9372_CR30","unstructured":"Manhaeve, R.,\u00a0Dumancic, S.,\u00a0Kimmig, A.,\u00a0Demeester, T., &\u00a0De\u00a0Raedt, L. (2018). Deepproblog: Neural probabilistic logic programming. In Advances in neural information processing systems, (pp. 3749\u20133759)."},{"key":"9372_CR31","unstructured":"Dragone, P.,\u00a0Teso, S., &\u00a0Passerini, A. (2021). Neuro-symbolic constraint programming for structured prediction. In NeSy, CEUR Workshop proceedings, vol. 2986 (CEUR-WS.org) (pp. 6\u201314)."},{"key":"9372_CR32","doi-asserted-by":"crossref","unstructured":"Cai, L.,\u00a0Dai, W.,\u00a0Huang, Y.,\u00a0Li, Y., Muggleton, S.H., &\u00a0Jiang, Y. (2021). Abductive learning with ground knowledge base. In IJCAI (ijcai.org) (pp. 1815\u20131821).","DOI":"10.24963\/ijcai.2021\/250"},{"key":"9372_CR33","doi-asserted-by":"publisher","first-page":"103521","DOI":"10.1016\/j.artint.2021.103521","volume":"299","author":"R Evans","year":"2021","unstructured":"Evans, R., Bosnjak, M., Buesing, L., Ellis, K., Reichert, D. P., Kohli, P., & Sergot, M. J. (2021). Making sense of raw input. Artificial Intelligence, 299, 103521.","journal-title":"Artificial Intelligence"},{"key":"9372_CR34","doi-asserted-by":"publisher","unstructured":"Yang, Z.,\u00a0Ishay, A., &\u00a0Lee, J. (2020). NeurASP: Embracing neural networks into answer set programming. In C.\u00a0Bessiere (Ed.), Proceedings of the twenty-ninth international joint conference on artificial intelligence, IJCAI 2020, (ijcai.org) (pp. 1755\u20131762). https:\/\/doi.org\/10.24963\/ijcai.2020\/243","DOI":"10.24963\/ijcai.2020\/243"},{"key":"9372_CR35","first-page":"7972","volume":"35","author":"Y Nandwani","year":"2022","unstructured":"Nandwani, Y., Ranjan, R., Singla, P., et al. (2022). A solver-free framework for scalable learning in neural ilp architectures. Advances in Neural Information Processing Systems, 35, 7972\u20137986.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"9372_CR36","doi-asserted-by":"publisher","unstructured":"Bai, Y.,\u00a0Chen, D., & Gomes, C. P. (2021) CLR-DRNets: Curriculum learning with restarts to solve visual combinatorial games. In L.D. Michel (Ed.), 27th International conference on principles and practice of constraint programming, CP 2021, Montpellier, France (Virtual conference), October 25-29, 2021, LIPIcs, vol. 210, (Schloss Dagstuhl - Leibniz-Zentrum f\u00fcr Informatik) (pp. 17:1\u201317:14). https:\/\/doi.org\/10.4230\/LIPIcs.CP.2021.17","DOI":"10.4230\/LIPIcs.CP.2021.17"},{"issue":"11","key":"9372_CR37","doi-asserted-by":"publisher","first-page":"2278","DOI":"10.1109\/5.726791","volume":"86","author":"Y LeCun","year":"1998","unstructured":"LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278\u20132324.","journal-title":"Proceedings of the IEEE"},{"key":"9372_CR38","unstructured":"An, S., Lee, M. J.,\u00a0Park, S., Yang, H. S., &\u00a0So, J. (2020). An ensemble of simple convolutional neural network models for mnist digit recognition. arxiv:2008.10400"},{"key":"9372_CR39","unstructured":"Augustine, E.,\u00a0Pryor, C.,\u00a0Dickens, C.,\u00a0Pujara, J., Wang, W. Y., &\u00a0Getoor, L. (2022) Visual Sudoku puzzle classification: A suite of collective neuro-symbolic tasks. In NeSy, CEUR workshop proceedings, vol. 3212 (CEUR-WS.org) (pp. 15\u201329)."},{"key":"9372_CR40","doi-asserted-by":"publisher","unstructured":"Wicht, B., &\u00a0Hennebert, J. (2015) Mixed handwritten and printed digit recognition in Sudoku with convolutional deep belief network. In 13th International conference on document analysis and recognition, ICDAR 2015, Nancy, France, August 23-26, 2015 (IEEE Computer Society) (pp. 861\u2013865). https:\/\/doi.org\/10.1109\/ICDAR.2015.7333884","DOI":"10.1109\/ICDAR.2015.7333884"},{"issue":"6","key":"9372_CR41","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1109\/MSP.2012.2211477","volume":"29","author":"L Deng","year":"2012","unstructured":"Deng, L. (2012). The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6), 141\u2013142.","journal-title":"IEEE Signal Processing Magazine"},{"key":"9372_CR42","unstructured":"Cornelio, C.,\u00a0Stuehmer, J., Hu, S. X., & Hospedales, T. M. (2023). Learning where and when to reason in neuro-symbolic inference. In ICLR (OpenReview.net)."},{"key":"9372_CR43","unstructured":"Rossi, F.,\u00a0Van\u00a0Beek, P., &\u00a0Walsh, T. (2006). Handbook of constraint programming (Elsevier)."},{"key":"9372_CR44","unstructured":"Goodfellow, I.,\u00a0Bengio, Y., &\u00a0Courville, A. (2016). Deep learning (MIT Press)."},{"key":"9372_CR45","doi-asserted-by":"crossref","unstructured":"Mattick, A.,\u00a0Mayr, M.,\u00a0Seuret, M.,\u00a0Maier, A., &\u00a0Christlein, V. (2021). SmartPatch: Improving handwritten word imitation with patch discriminators. In ICDAR (1), Lecture notes in computer science, vol. 12821 (Springer) (pp. 268\u2013283).","DOI":"10.1007\/978-3-030-86549-8_18"},{"key":"9372_CR46","unstructured":"Netzer, Y.,\u00a0Wang, T.,\u00a0Coates, A.,\u00a0Bissacco, A.,\u00a0Wu, B., & Ng, A. Y. (2011). Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on deep learning and unsupervised feature learning 2011. http:\/\/ufldl.stanford.edu\/housenumbers\/nips2011_housenumbers.pdf"},{"key":"9372_CR47","doi-asserted-by":"crossref","unstructured":"Cubuk, E. D.,\u00a0Zoph, B.,\u00a0Man\u00e9, D.,\u00a0Vasudevan, V., & Le, Q. V. (2019). AutoAugment: Learning augmentation strategies from data. In CVPR (Computer vision foundation \/ IEEE) (pp. 113\u2013123).","DOI":"10.1109\/CVPR.2019.00020"},{"key":"9372_CR48","unstructured":"Guo, C.,\u00a0Pleiss, G.,\u00a0Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In ICML, Proceedings of machine learning research, vol.\u00a070 (PMLR) (pp. 1321\u20131330)"},{"key":"9372_CR49","unstructured":"Kull, M.,\u00a0Perell\u00f3-Nieto, M.,\u00a0K\u00e4ngsepp, M.,\u00a0de\u00a0Menezes\u00a0e Silva\u00a0Filho, T.,\u00a0Song, H., & Flach, P. A. (2019). Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration. In H.M. Wallach, H.\u00a0Larochelle, A.\u00a0Beygelzimer, F.\u00a0d\u2019Alch\u00e9-Buc, E.B. Fox, & R.\u00a0Garnett (Eds.), Advances in neural information processing systems 32: Annual conference on neural information processing systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada (pp. 12295\u201312305). https:\/\/proceedings.neurips.cc\/paper\/2019\/hash\/8ca01ea920679a0fe3728441494041b9-Abstract.html"},{"key":"9372_CR50","doi-asserted-by":"crossref","unstructured":"Platt, J. C. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in large margin classifiers (MIT Press) (pp. 61\u201374)","DOI":"10.7551\/mitpress\/1113.003.0008"},{"key":"9372_CR51","unstructured":"Naeini, M. P., Cooper, G. F., &\u00a0Hauskrecht, M. (2015). Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the twenty-ninth AAAI conference on artificial intelligence (AAAI Press), AAAI\u201915 (pp. 2901\u20132907)."},{"key":"9372_CR52","unstructured":"Roth, D., Yih, & W. t. (2004). A linear programming formulation for global inference in natural language tasks. In Proceedings of the eighth conference on computational natural language learning (CoNLL-2004) at HLT-NAACL 2004 (Association for Computational Linguistics, Boston, Massachusetts, USA) (pp. 1\u20138). https:\/\/aclanthology.org\/W04-2401"},{"key":"9372_CR53","doi-asserted-by":"crossref","unstructured":"Punyakanok, V.,\u00a0Roth, D., Yih, W. t., &\u00a0Zimak, D. (2004). Semantic role labeling via integer linear programming inference. In COLING 2004: Proceedings of the 20th international conference on computational linguistics (COLING, Geneva, Switzerland) (pp. 1346\u20131352). https:\/\/aclanthology.org\/C04-1197","DOI":"10.3115\/1220355.1220552"},{"key":"9372_CR54","unstructured":"Li, Q.,\u00a0Ji, H., &\u00a0Huang, L. (2013). Joint event extraction via structured prediction with global features. In Proceedings of the 51st annual meeting of the association for computational linguistics (vol. 1: Long papers) (pp. 73\u201382)"},{"key":"9372_CR55","unstructured":"Guns, T., Stuckey, P. J., &\u00a0Tack, G. (2018). Solution dominance over constraint satisfaction problems. arXiv:1812.09207"},{"issue":"4","key":"9372_CR56","doi-asserted-by":"publisher","first-page":"438","DOI":"10.1007\/s10115-004-0175-4","volume":"7","author":"P Tzvetkov","year":"2005","unstructured":"Tzvetkov, P., Yan, X., & Han, J. (2005). TSP: Mining top-k closed sequential patterns. Knowledge and Information Systems, 7(4), 438\u2013457.","journal-title":"Knowledge and Information Systems"},{"key":"9372_CR57","doi-asserted-by":"crossref","unstructured":"Herbei, R., & Wegkamp, M. H. (2006). Classification with reject option. The Canadian Journal of Statistics \/ La Revue Canadienne de Statistique, 34(4), 709\u2013721. http:\/\/www.jstor.org\/stable\/20445230","DOI":"10.1002\/cjs.5550340410"},{"key":"9372_CR58","unstructured":"Hendrickx, K.,\u00a0Perini, L., der Plas, D. V.,\u00a0Meert, W., &\u00a0Davis, J. (2021). Machine learning with a reject option: A survey. CoRR arxiv:2107.11277"},{"issue":"1","key":"9372_CR59","doi-asserted-by":"publisher","first-page":"355","DOI":"10.3233\/JIFS-212233","volume":"43","author":"MS Gondere","year":"2022","unstructured":"Gondere, M. S., Schmidt-Thieme, L., Sharma, D. P., & Scholz, R. (2022). Multi-script handwritten digit recognition using multi-task learning. Journal of Intelligent Fuzzy Systems, 43(1), 355\u2013364.","journal-title":"Journal of Intelligent Fuzzy Systems"},{"key":"9372_CR60","doi-asserted-by":"publisher","unstructured":"Borovec, J.,\u00a0Falcon, W.,\u00a0Nitta, A., Jha, A. H., otaj,\u00a0Brundyn, A.,\u00a0Byrne, D.,\u00a0Raw, N.,\u00a0Matsumoto, S.,\u00a0Koker, T.,\u00a0Ko, B.,\u00a0Oke, A.,\u00a0Sundrani, S., Baruch,\u00a0Clement, C.,\u00a0Poiret, C.,\u00a0Gupta, R.,\u00a0Aekula, H.,\u00a0W\u00e4lchli, A.,\u00a0Phatak, A.,\u00a0Kessler, I.,\u00a0Wang, J.,\u00a0Lee, J.,\u00a0Mehta, S.,\u00a0Yang, Z., &\u00a0O\u2019Donnell, G. (2022) zlapp. Lightning-ai\/lightning-bolts: Minor patch release. https:\/\/doi.org\/10.5281\/zenodo.7447212","DOI":"10.5281\/zenodo.7447212"},{"key":"9372_CR61","unstructured":"Paszke, A.,\u00a0Gross, S.,\u00a0Chintala, S.,\u00a0Chanan, G.,\u00a0Yang, E.,\u00a0DeVito, Z.,\u00a0Lin, Z.,\u00a0Desmaison, A.,\u00a0Antiga, L., &\u00a0Lerer, A. (2017). Automatic differentiation in PyTorch. In NeurIPS Autodiff Workshop."},{"key":"9372_CR62","unstructured":"Guns, T. (2019). Increasing modeling language convenience with a universal n-dimensional array, CPpy as python-embedded example. In Proceedings of the 18th workshop on constraint modelling and Reformulation at CP (Modref 2019), vol.\u00a019"},{"key":"9372_CR63","unstructured":"Perron, L., &\u00a0Furnon, V. (2022). Or-tools. https:\/\/developers.google.com\/optimization\/"},{"key":"9372_CR64","doi-asserted-by":"publisher","unstructured":"Mulamba Ke\u00a0Tchomba, M. A.,\u00a0Mandi, J.,\u00a0Mahmutogullari, I., &\u00a0Guns, T. (2024). Replication data for: Perception-based constraint solving for Sudoku images. https:\/\/doi.org\/10.48804\/3SUHHR","DOI":"10.48804\/3SUHHR"},{"key":"9372_CR65","doi-asserted-by":"crossref","unstructured":"Smock, B.,\u00a0Pesala, R., &\u00a0Abraham, R. (2022). PubTables-1M: Towards comprehensive table extraction from unstructured documents. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 4634\u20134642).","DOI":"10.1109\/CVPR52688.2022.00459"},{"issue":"1\u20132","key":"9372_CR66","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1017\/S1471068411000494","volume":"12","author":"J Wielemaker","year":"2012","unstructured":"Wielemaker, J., Schrijvers, T., Triska, M., & Lager, T. (2012). SWI-Prolog. Theory and Practice of Logic Programming, 12(1\u20132), 67\u201396.","journal-title":"Theory and Practice of Logic Programming"},{"key":"9372_CR67","unstructured":"Loshchilov, I., &\u00a0Hutter, F. (2019). Decoupled weight decay regularization. In ICLR (Poster) (OpenReview.net)."},{"issue":"6","key":"9372_CR68","doi-asserted-by":"publisher","first-page":"94","DOI":"10.3390\/jimaging7060094","volume":"7","author":"G W\u00f6lflein","year":"2021","unstructured":"W\u00f6lflein, G., & Arandjelovi\u0107, O. (2021). Determining chess game state from an image. Journal of Imaging, 7(6), 94.","journal-title":"Journal of Imaging"},{"issue":"1","key":"9372_CR69","doi-asserted-by":"publisher","first-page":"84","DOI":"10.1109\/TCDS.2019.2915763","volume":"12","author":"A Persson","year":"2019","unstructured":"Persson, A., Dos Martires, P. Z., De Raedt, L., & Loutfi, A. (2019). Semantic relational object tracking. IEEE Transactions on Cognitive and Developmental Systems, 12(1), 84\u201397.","journal-title":"IEEE Transactions on Cognitive and Developmental Systems"},{"issue":"5","key":"9372_CR70","doi-asserted-by":"publisher","first-page":"863","DOI":"10.3233\/SW-190362","volume":"10","author":"M Alirezaie","year":"2019","unstructured":"Alirezaie, M., L\u00e4ngkvist, M., Sioutis, M., & Loutfi, A. (2019). Semantic referee: A neural-symbolic framework for enhancing geospatial semantic segmentation. Semantic Web, 10(5), 863\u2013880.","journal-title":"Semantic Web"},{"key":"9372_CR71","doi-asserted-by":"crossref","unstructured":"Du, C.,\u00a0Wang, Y.,\u00a0Yang, Z.,\u00a0Zhou, H.,\u00a0Han, M., & Lai, J. H. (2023) PARCS: A deployment-oriented AI system for robust parcel-level cropland segmentation of satellite images. In Proceedings of the AAAI conference on artificial intelligence.","DOI":"10.1609\/aaai.v37i13.26873"},{"issue":"1","key":"9372_CR72","doi-asserted-by":"publisher","first-page":"1036","DOI":"10.1109\/TPAMI.2022.3150906","volume":"45","author":"G Singh","year":"2022","unstructured":"Singh, G., Akrigg, S., Di Maio, M., Fontana, V., Alitappeh, R. J., Khan, S., Saha, S., Jeddisaravi, K., Yousefi, F., Culley, J., et al. (2022). Road: The road event awareness dataset for autonomous driving. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1), 1036\u20131054.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"issue":"14","key":"9372_CR73","doi-asserted-by":"publisher","first-page":"2525","DOI":"10.3390\/math10142525","volume":"10","author":"L Zhang","year":"2022","unstructured":"Zhang, L., Yin, H., Hui, B., Liu, S., & Zhang, W. (2022). Knowledge-based scene graph generation with visual contextual dependency. Mathematics, 10(14), 2525.","journal-title":"Mathematics"},{"key":"9372_CR74","unstructured":"Chang, M.,\u00a0Ratinov, L., &\u00a0Roth, D. (2007) Guiding semi-supervision with constraint-driven learning. In ACL (The Association for Computational Linguistics)"},{"key":"9372_CR75","first-page":"3379","volume":"2022","author":"T Fang","year":"2022","unstructured":"Fang, T., Do, Q. V., Zhang, H., Song, Y., Wong, G. Y., & See, S. (2022). PseudoReasoner: Leveraging pseudo labels for commonsense knowledge base population. Findings of the Association for Computational Linguistics: EMNLP, 2022, 3379\u20133394.","journal-title":"Findings of the Association for Computational Linguistics: EMNLP"},{"key":"9372_CR76","doi-asserted-by":"publisher","first-page":"1623","DOI":"10.1613\/jair.1.15320","volume":"80","author":"J Mandi","year":"2024","unstructured":"Mandi, J., Kotary, J., Berden, S., Mulamba, M., Bucarey, V., Guns, T., & Fioretto, F. (2024). Decision-focused learning: Foundations, state of the art, benchmark and future opportunities. Journal of Artificial Intelligence Research, 80, 1623\u20131701.","journal-title":"Journal of Artificial Intelligence Research"},{"key":"9372_CR77","first-page":"29944","volume":"35","author":"K Ahmed","year":"2022","unstructured":"Ahmed, K., Teso, S., Chang, K. W., Van den Broeck, G., & Vergari, A. (2022). Semantic probabilistic layers for neuro-symbolic learning. Advances in Neural Information Processing Systems, 35, 29944\u201329959.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"9372_CR78","first-page":"24824","volume":"35","author":"J Wei","year":"2022","unstructured":"Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35, 24824\u201324837.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"9372_CR79","unstructured":"Brown, T. B.,\u00a0Mann, B.,\u00a0Ryder, N.,\u00a0Subbiah, M.,\u00a0Kaplan, J.,\u00a0Dhariwal, P.,\u00a0Neelakantan, A.,\u00a0Shyam, P.,\u00a0Sastry, G.,\u00a0Askell, A.,\u00a0Agarwal, S.,\u00a0Herbert-Voss, A.,\u00a0Krueger, G.,\u00a0Henighan, T.,\u00a0Child, R.,\u00a0Ramesh, A., Ziegler, D. M.,\u00a0Wu, J.,\u00a0Winter, C.,\u00a0Hesse, C.,\u00a0Chen, M.,\u00a0Sigler, E.,\u00a0Litwin, M.,\u00a0Gray, S.,\u00a0Chess, B.,\u00a0Clark, J.,\u00a0Berner, C.,\u00a0McCandlish, S.,\u00a0Radford, A.,\u00a0Sutskever, I., &\u00a0Amodei, D. (2020). Language models are few-shot learners, in NeurIPS."},{"key":"9372_CR80","unstructured":"Yin, S.,\u00a0Fu, C.,\u00a0Zhao, S.,\u00a0Li, K.,\u00a0Sun, X.,\u00a0Xu, T., &\u00a0Chen, E. (2023). A survey on multimodal large language models. arXiv:2306.13549"},{"key":"9372_CR81","unstructured":"Schick, T.,\u00a0Dwivedi-Yu, J.,\u00a0Dess\u00ec, R.,\u00a0Raileanu, R.,\u00a0Lomeli, M.,\u00a0Hambro, E.,\u00a0Zettlemoyer, L.,\u00a0Cancedda, N., &\u00a0Scialom, T. (2024). Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36."},{"key":"9372_CR82","unstructured":"Michailidis, K.,\u00a0Tsouros, D., &\u00a0Guns, T. (2024). Constraint modelling with LLMs using in-context learning. In 30th International conference on principles and practice of constraint programming."},{"key":"9372_CR83","unstructured":"Lin, M.,\u00a0Chen, Q., &\u00a0Yan, S. (2014). Network in network. In Y.\u00a0Bengio, & Y.\u00a0LeCun (Eds.) 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, conference track proceedings. arxiv:1312.4400"},{"key":"9372_CR84","unstructured":"Gonz\u00e1lez, R. C., & Woods, R. E. (1992). Digital image processing (Addison-Wesley)."},{"key":"9372_CR85","unstructured":"Simonyan, K., &\u00a0Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. In Y.\u00a0Bengio, & Y.\u00a0LeCun (Eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, conference track proceedings. arxiv:1409.1556"},{"key":"9372_CR86","doi-asserted-by":"crossref","unstructured":"Deng, J.,\u00a0Dong, W.,\u00a0Socher, R., Li, L. J.,\u00a0Li, K., &\u00a0Fei-Fei, L. (2009) Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (Ieee) (pp. 248\u2013255)","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"9372_CR87","doi-asserted-by":"publisher","unstructured":"LeCun, Y.,\u00a0Bottou, L., Orr, G. B., &\u00a0M\u00fcller, K. (2012). In G.\u00a0Montavon, G.B. Orr, & K.\u00a0M\u00fcller (Eds.) Neural networks: Tricks of the trade - second edition, lecture notes in computer science, vol. 7700 (Springer) (pp. 9\u201348). https:\/\/doi.org\/10.1007\/978-3-642-35289-8_3","DOI":"10.1007\/978-3-642-35289-8_3"},{"key":"9372_CR88","unstructured":"Lindauer, M.,\u00a0Eggensperger, K.,\u00a0Feurer, M.,\u00a0Biedenkapp, A.,\u00a0Deng, D.,\u00a0Benjamins, C.,\u00a0Ruhkopf, T.,\u00a0Sass, R., &\u00a0Hutter, F. (2022). Smac3: A versatile bayesian optimization package for hyperparameter optimization. Journal of Machine Learning Research, 23(54), 1\u20139. http:\/\/jmlr.org\/papers\/v23\/21-0888.html"}],"container-title":["Constraints"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10601-024-09372-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10601-024-09372-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10601-024-09372-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,4]],"date-time":"2024-11-04T11:10:55Z","timestamp":1730718655000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10601-024-09372-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6]]},"references-count":88,"journal-issue":{"issue":"1-2","published-print":{"date-parts":[[2024,6]]}},"alternative-id":["9372"],"URL":"https:\/\/doi.org\/10.1007\/s10601-024-09372-9","relation":{},"ISSN":["1383-7133","1572-9354"],"issn-type":[{"value":"1383-7133","type":"print"},{"value":"1572-9354","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6]]},"assertion":[{"value":"5 September 2024","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 October 2024","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no relevant financial or non-financial interests to disclose.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Financial and non-financial interests"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"Not Applicable.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors have no relevant conflicting interests to disclose.","order":6,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}