{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T01:42:30Z","timestamp":1760060550367,"version":"build-2065373602"},"reference-count":24,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2025,9,2]],"date-time":"2025-09-02T00:00:00Z","timestamp":1756771200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Union","award":["CZ.10.03.01\/00\/22_003\/0000048","CZ.02.01.01\/00\/22_008\/0004590","LL1902","25-17929X","857306"],"award-info":[{"award-number":["CZ.10.03.01\/00\/22_003\/0000048","CZ.02.01.01\/00\/22_008\/0004590","LL1902","25-17929X","857306"]}]},{"name":"Ministry of Education, Youth and Sports within the dedicated program ERC CZ","award":["CZ.10.03.01\/00\/22_003\/0000048","CZ.02.01.01\/00\/22_008\/0004590","LL1902","25-17929X","857306"],"award-info":[{"award-number":["CZ.10.03.01\/00\/22_003\/0000048","CZ.02.01.01\/00\/22_008\/0004590","LL1902","25-17929X","857306"]}]},{"name":"Czech Science Foundation","award":["CZ.10.03.01\/00\/22_003\/0000048","CZ.02.01.01\/00\/22_008\/0004590","LL1902","25-17929X","857306"],"award-info":[{"award-number":["CZ.10.03.01\/00\/22_003\/0000048","CZ.02.01.01\/00\/22_008\/0004590","LL1902","25-17929X","857306"]}]},{"name":"European Union\u2019s Horizon 2020 research and innovation programme","award":["CZ.10.03.01\/00\/22_003\/0000048","CZ.02.01.01\/00\/22_008\/0004590","LL1902","25-17929X","857306"],"award-info":[{"award-number":["CZ.10.03.01\/00\/22_003\/0000048","CZ.02.01.01\/00\/22_008\/0004590","LL1902","25-17929X","857306"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>While neural networks can solve complex geometric problems, as demonstrated by systems like AlphaGeometry, we have limited understanding of how they internally represent and reason about spatial relationships. In this work, we investigate how neural networks develop internal spatial understanding by training Graph Neural Networks and Transformers to predict point positions on a discrete 2D grid from geometric constraints that describe hidden figures. We show that both models develop interpretable internal representations that mirror the geometric structure of the problems they solve. Specifically, we observe that point embeddings self-organize into 2D grid structures during training, and during inference, the models iteratively construct the hidden geometric figures within their embedding spaces. Our analysis reveals how reasoning complexity correlates with prediction accuracy, and shows that models solve constraints through an iterative refinement process, which might resemble continuous optimization. We also find that Graph Neural Networks prove more suitable than Transformers for this type of structured constraint reasoning and scale more effectively to larger problems. These findings provide initial insights into how neural networks can develop structured understanding and contribute to their interpretability.<\/jats:p>","DOI":"10.3390\/make7030093","type":"journal-article","created":{"date-parts":[[2025,9,2]],"date-time":"2025-09-02T16:05:22Z","timestamp":1756829122000},"page":"93","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Geometric Reasoning in the Embedding Space"],"prefix":"10.3390","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3867-644X","authenticated-orcid":false,"given":"David","family":"Moj\u017e\u00ed\u0161ek","sequence":"first","affiliation":[{"name":"Institute for Research and Applications of Fuzzy Modeling, University of Ostrava, 702 00 Ostrava, Czech Republic"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7639-864X","authenticated-orcid":false,"given":"Jan","family":"H\u016fla","sequence":"additional","affiliation":[{"name":"Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague, 160 00 Prague, Czech Republic"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4397-9004","authenticated-orcid":false,"given":"Ji\u0159\u00ed","family":"Jane\u010dek","sequence":"additional","affiliation":[{"name":"Institute for Research and Applications of Fuzzy Modeling, University of Ostrava, 702 00 Ostrava, Czech Republic"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-1861-3778","authenticated-orcid":false,"given":"David","family":"Herel","sequence":"additional","affiliation":[{"name":"Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague, 160 00 Prague, Czech Republic"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3487-784X","authenticated-orcid":false,"given":"Mikol\u00e1\u0161","family":"Janota","sequence":"additional","affiliation":[{"name":"Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague, 160 00 Prague, Czech Republic"}]}],"member":"1968","published-online":{"date-parts":[[2025,9,2]]},"reference":[{"key":"ref_1","first-page":"69736","article-title":"Evaluating cognitive maps and planning in large language models with cogeval","volume":"36","author":"Momennejad","year":"2023","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_2","unstructured":"Wu, W., Mao, S., Zhang, Y., Xia, Y., Dong, L., Cui, L., and Wei, F. (2024, January 10\u201315). Mind\u2019s Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models. Proceedings of the Thirty-Eighth Annual Conference on Neural Information Processing Systems, Vancouver, BC, Canada."},{"key":"ref_3","unstructured":"Yamada, Y., Bao, Y., Lampinen, A.K., Kasai, J., and Yildirim, I. (2023). Evaluating Spatial Understanding of Large Language Models. arXiv."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Li, S., Deng, Y., and Lam, W. (2023, January 6\u201310). DepWiGNN: A Depth-wise Graph Neural Network for Multi-hop Spatial Reasoning in Text. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore.","DOI":"10.18653\/v1\/2023.findings-emnlp.428"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Teodorescu, L., Hofmann, K., and Oudeyer, P.Y. (2022). SpatialSim: Recognizing Spatial Configurations of Objects with Graph Neural Networks. Front. Artif. Intell., 4.","DOI":"10.3389\/frai.2021.782081"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"476","DOI":"10.1038\/s41586-023-06747-5","article-title":"Solving olympiad geometry without human demonstrations","volume":"625","author":"Trinh","year":"2024","journal-title":"Nature"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1007\/BF02328447","article-title":"Basic principles of mechanical theorem proving in elementary geometries","volume":"2","year":"1986","journal-title":"J. Autom. Reason."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Wu, W.T. (2008). On the decision problem and the mechanization of theorem-proving in elementary geometry. Selected Works of Wen-Tsun Wu, World Scientific.","DOI":"10.1142\/9789812791085"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Wong, M.F., Qi, X., and Tan, C.W. (2022). EuclidNet: Deep visual reasoning for constructible problems in geometry. arXiv.","DOI":"10.54364\/AAIML.2023.1152"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"H\u016fla, J., Moj\u017e\u00ed\u0161ek, D., and Janota, M. (2024, January 21\u201325). Understanding GNNs for Boolean Satisfiability through Approximation Algorithms. Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, Boise, ID, USA.","DOI":"10.1145\/3627673.3679813"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Krueger, R., Han, J.M., and Selsam, D. (2021, January 12\u201315). Automatically Building Diagrams for Olympiad Geometry Problems. Proceedings of the CADE, Virtual Event.","DOI":"10.1007\/978-3-030-79876-5_33"},{"key":"ref_12","unstructured":"Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozi\u00e8re, B., Goyal, N., Hambro, E., and Azhar, F. (2023). Llama: Open and efficient foundation language models. arXiv."},{"key":"ref_13","unstructured":"Ivanitskiy, M.I., Spies, A.F., Rauker, T., Corlouer, G., Mathwin, C., Quirke, L., Rager, C., Shah, R., Valentine, D., and Behn, C.G.D. (2023). Structured World Representations in Maze-Solving Transformers. arXiv."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1162\/tacl_a_00004","article-title":"Representation learning for grounded spatial reasoning","volume":"6","author":"Janner","year":"2018","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_15","first-page":"337","article-title":"Z3: An Efficient SMT Solver","volume":"Volume 4963","author":"Ramakrishnan","year":"2008","journal-title":"Proceedings of the 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS), Held as Part of the Joint European Conferences on Theory and Practice of Software (ETAPS)"},{"key":"ref_16","unstructured":"Selsam, D., Lamm, M., B\u00fcnz, B., Liang, P., de Moura, L., and Dill, D.L. (2018). Learning a SAT solver from single-bit supervision. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_18","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"127063","DOI":"10.1016\/j.neucom.2023.127063","article-title":"Roformer: Enhanced transformer with rotary position embedding","volume":"568","author":"Su","year":"2024","journal-title":"Neurocomputing"},{"key":"ref_20","unstructured":"Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, \u0141. (2018). Universal transformers. arXiv."},{"key":"ref_21","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_22","unstructured":"Moj\u017e\u00ed\u0161ek, D., H\u016fla, J., Li, Z., Zhou, Z., and Janota, M. (2025). Neural Approaches to SAT Solving: Design Choices and Interpretability. arXiv."},{"key":"ref_23","unstructured":"Abbe, E., Bengio, S., Lotfi, A., and Rizk, K. (2023, January 23\u201329). Generalization on the unseen, logic reasoning and degree curriculum. Proceedings of the International Conference on Machine Learning, Honolulu, HI, USA."},{"key":"ref_24","unstructured":"Wong, L.S., Grand, G., Lew, A.K., Goodman, N.D., Mansinghka, V.K., Andreas, J., and Tenenbaum, J.B. (2023). From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought. arXiv."}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/7\/3\/93\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:38:00Z","timestamp":1760035080000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/7\/3\/93"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,2]]},"references-count":24,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,9]]}},"alternative-id":["make7030093"],"URL":"https:\/\/doi.org\/10.3390\/make7030093","relation":{},"ISSN":["2504-4990"],"issn-type":[{"type":"electronic","value":"2504-4990"}],"subject":[],"published":{"date-parts":[[2025,9,2]]}}}