{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,25]],"date-time":"2026-01-25T00:26:00Z","timestamp":1769300760565,"version":"3.49.0"},"reference-count":36,"publisher":"Springer Science and Business Media LLC","issue":"9","license":[{"start":{"date-parts":[[2023,5,1]],"date-time":"2023-05-01T00:00:00Z","timestamp":1682899200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,5,1]],"date-time":"2023-05-01T00:00:00Z","timestamp":1682899200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/N509711\/1"],"award-info":[{"award-number":["EP\/N509711\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000769","name":"University of Oxford","doi-asserted-by":"crossref","award":["DeepMind Graduate Scholarship"],"award-info":[{"award-number":["DeepMind Graduate Scholarship"]}],"id":[{"id":"10.13039\/501100000769","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100000769","name":"University of Oxford","doi-asserted-by":"crossref","award":["EP\/T517811\/1"],"award-info":[{"award-number":["EP\/T517811\/1"]}],"id":[{"id":"10.13039\/501100000769","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100000769","name":"University of Oxford","doi-asserted-by":"crossref","award":["EP\/N510129\/1"],"award-info":[{"award-number":["EP\/N510129\/1"]}],"id":[{"id":"10.13039\/501100000769","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001961","name":"AXA Research Fund","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001961","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/R013667\/1"],"award-info":[{"award-number":["EP\/R013667\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003816","name":"Huawei Technologies","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100003816","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100018707","name":"HORIZON EUROPE Reforming and enhancing the European Research and Innovation system","doi-asserted-by":"publisher","award":["964505 (E-pi)"],"award-info":[{"award-number":["964505 (E-pi)"]}],"id":[{"id":"10.13039\/100018707","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2023,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Neural networks have proven to be very powerful at computer vision tasks. However, they often exhibit unexpected behaviors, acting against background knowledge about the problem at hand. This calls for models (i) able to learn from requirements expressing such background knowledge, and (ii) guaranteed to be compliant with the requirements themselves. Unfortunately, the development of such models is hampered by the lack of real-world datasets equipped with formally specified requirements. In this paper, we introduce the ROad event Awareness Dataset with logical Requirements (ROAD-R), the first publicly available dataset for autonomous driving with requirements expressed as logical constraints. Given ROAD-R, we show that current state-of-the-art models often violate its logical constraints, and that it is possible to exploit them to create models that (i) have a better performance, and (ii) are guaranteed to be compliant with the requirements themselves.<\/jats:p>","DOI":"10.1007\/s10994-023-06322-z","type":"journal-article","created":{"date-parts":[[2023,5,1]],"date-time":"2023-05-01T21:27:41Z","timestamp":1682976461000},"page":"3261-3291","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["ROAD-R: the autonomous driving dataset with logical requirements"],"prefix":"10.1007","volume":"112","author":[{"given":"Eleonora","family":"Giunchiglia","sequence":"first","affiliation":[]},{"given":"Mihaela C\u0103t\u0103lina","family":"Stoian","sequence":"additional","affiliation":[]},{"given":"Salman","family":"Khan","sequence":"additional","affiliation":[]},{"given":"Fabio","family":"Cuzzolin","sequence":"additional","affiliation":[]},{"given":"Thomas","family":"Lukasiewicz","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,5,1]]},"reference":[{"key":"6322_CR1","unstructured":"Amodei, D., Olah, C., Steinhardt, J., Christiano, P. F., Schulman, J., & Man\u00e9, D. (2016). Concrete problems in AI safety. arXiv preprint arXiv:1606.06565"},{"issue":"4","key":"6322_CR2","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1007\/s10817-006-9063-9","volume":"37","author":"O Bailleux","year":"2006","unstructured":"Bailleux, O., & Marquis, P. (2006). Some computational aspects of distance-sat. Journal of Automated Reasoning, 37(4), 231\u2013260.","journal-title":"Journal of Automated Reasoning"},{"key":"6322_CR3","doi-asserted-by":"crossref","unstructured":"Carreira, J., & Zisserman, A. (2017). Quo vadis, action recognition? A new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition..","DOI":"10.1109\/CVPR.2017.502"},{"key":"6322_CR4","doi-asserted-by":"crossref","unstructured":"Curtis, K., Awad, G., Rajput, S., & Soboroff, I. (2020). HLVU: A new challenge to test deep understanding of movies the way humans do. In Proceedings of the 2020 International Conference on Multimedia Retrieval.","DOI":"10.1145\/3372278.3390742"},{"key":"6322_CR5","first-page":"1","volume":"7","author":"J Demsar","year":"2006","unstructured":"Demsar, J. (2006). Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 7, 1\u201330.","journal-title":"The Journal of Machine Learning Research"},{"key":"6322_CR6","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1016\/j.artint.2015.08.011","volume":"244","author":"M Diligenti","year":"2017","unstructured":"Diligenti, M., Gori, M., & Sacca, C. (2017). Semantic-based regularization for learning and inference. Artificial Intelligence, 244, 143\u2013165.","journal-title":"Artificial Intelligence"},{"key":"6322_CR7","doi-asserted-by":"crossref","unstructured":"Diligenti, M., Roychowdhury, S., & Gori, M. (2017). Integrating prior knowledge into deep learning. In 2017 16th IEEE international conference on machine learning and applications (ICMLA).","DOI":"10.1109\/ICMLA.2017.00-37"},{"key":"6322_CR8","doi-asserted-by":"crossref","unstructured":"Donadello, I., Serafini, L., & d\u2019Avila Garcez, A. (2017). Logic tensor networks for semantic image interpretation. In Proceedings of IJCAI.","DOI":"10.24963\/ijcai.2017\/221"},{"key":"6322_CR9","unstructured":"Dragone, P., Teso, S., & Passerini, A. (2021). Neuro-symbolic constraint programming for structured prediction. In IJCLR-NeSy."},{"key":"6322_CR50","doi-asserted-by":"publisher","unstructured":"E\u00e9n, N., & S\u00f6rensson, N. (2004). An Extensible SAT-solver. In E. Giunchiglia and A. Tacchella (Eds.), Theory and applications of satisfiability testing. SAT 2003. Lecture Notes in Computer Science, vol. 2919. Berlin, Heidelberg: Springer. https:\/\/doi.org\/10.1007\/978-3-540-24605-3_37","DOI":"10.1007\/978-3-540-24605-3_37"},{"key":"6322_CR10","doi-asserted-by":"crossref","unstructured":"Feichtenhofer, C., Fan, H., Malik, J., & He, K. (2019). SlowFast networks for video recognition. In Proceedings of the IEEE\/CVF international conference on computer vision.","DOI":"10.1109\/ICCV.2019.00630"},{"issue":"2020","key":"6322_CR11","first-page":"9662","volume":"33","author":"E Giunchiglia","year":"2020","unstructured":"Giunchiglia, E., & Lukasiewicz, T. (2020). Coherent hierarchical multi-label classification networks. Advances in Neural Information Processing Systems, 33(2020), 9662\u20139673.","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"2021","key":"6322_CR12","doi-asserted-by":"publisher","first-page":"759","DOI":"10.1613\/jair.1.12850","volume":"72","author":"E Giunchiglia","year":"2021","unstructured":"Giunchiglia, E., & Lukasiewicz, T. (2021). Multi-label classification neural networks with hard logical constraints. Journal of Artificial Intelligence Research, 72(2021), 759\u2013818.","journal-title":"Journal of Artificial Intelligence Research"},{"key":"6322_CR13","doi-asserted-by":"crossref","unstructured":"Giunchiglia, E., Stoian, M. C., & Lukasiewicz, T. (2022). Deep learning with logical constraints. In Proceedings of IJCAI.","DOI":"10.24963\/ijcai.2022\/767"},{"key":"6322_CR14","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition","DOI":"10.1109\/CVPR.2016.90"},{"key":"6322_CR15","doi-asserted-by":"crossref","unstructured":"Hickey, R., & Bacchus, F. (2019). Speeding up assumption-based SAT. In Proceedings of SAT.","DOI":"10.1007\/978-3-030-24258-9_11"},{"key":"6322_CR16","doi-asserted-by":"crossref","unstructured":"Hoernle, N., Karampatsis, R., Belle, V., & Gal, K. (2022). MultiplexNet: Towards fully satisfied logical constraints in neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence.","DOI":"10.1609\/aaai.v36i5.20512"},{"key":"6322_CR17","doi-asserted-by":"crossref","unstructured":"Hu, Z., Ma, X., Liu, Z., Hovy, E., & Xing, E. (2016). Harnessing deep neural networks with logic rules. arXiv preprint arXiv:1603.06318 (2016)","DOI":"10.18653\/v1\/P16-1228"},{"key":"6322_CR18","doi-asserted-by":"crossref","unstructured":"Hu, Z., Yang, Z., Salakhutdinov, R., & Xing, E. (2016). Deep neural networks with massive learned knowledge. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.","DOI":"10.18653\/v1\/D16-1173"},{"key":"6322_CR19","doi-asserted-by":"crossref","unstructured":"Hua, Y., Zhao, Z., Liu, Z., Chen, X., Li, R., & Zhang, H. (2018). Traffic prediction based on random connectivity in deep learning with long short-term memory. In 2018 IEEE 88th Vehicular Technology Conference (VTC-Fall). IEEE.","DOI":"10.1109\/VTCFall.2018.8690851"},{"key":"6322_CR20","doi-asserted-by":"crossref","unstructured":"H\u00e1jek, P. (1998). Metamathematics of Fuzzy Logic. Dordrecht, the Netherlands: Kluwer Academic Publishers.","DOI":"10.1007\/978-94-011-5300-3"},{"key":"6322_CR21","doi-asserted-by":"crossref","unstructured":"Kalogeiton, V., Weinzaepfel, P., Ferrari, V., & Schmid, C. (2017). Action tubelet detector for spatio-temporal action localization. In Proceedings of the IEEE International Conference on Computer Vision.","DOI":"10.1109\/ICCV.2017.472"},{"key":"6322_CR22","doi-asserted-by":"crossref","unstructured":"LeCun, Y., Bottou, L., Orr, G., & M\u00fcller, K. (2012). Efficient backprop. In Neural Networks: Tricks of the Trade. Springer, Heidelberg.","DOI":"10.1007\/978-3-642-35289-8_3"},{"key":"6322_CR23","unstructured":"Li, C. M., & Many\u00e0, F. (2009). Maxsat, hard and soft constraints. In  Handbook of Satisfiability. Frontiers in Artif. Intell. and Appl.,  185(2009), 613-631."},{"key":"6322_CR24","doi-asserted-by":"crossref","unstructured":"Li, T., & Srikumar, V. (2019). Augmenting neural networks with first-order logic. arXiv preprint arXiv:1906.06298 (2019)","DOI":"10.18653\/v1\/P19-1028"},{"key":"6322_CR25","doi-asserted-by":"crossref","unstructured":"Li, D., Qiu, Z., Dai, Q., Yao, T., & Mei, T. (2018). Recurrent tubelet proposal and recognition networks for action detection. In Proceedings of the European conference on computer vision.","DOI":"10.1007\/978-3-030-01231-1_19"},{"issue":"1","key":"6322_CR26","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1177\/0278364916679498","volume":"36","author":"W Maddern","year":"2017","unstructured":"Maddern, W., Pascoe, G., Linegar, C., & Newman, P. (2017). 1 year, 1000 km: The Oxford RobotCar dataset. The International Journal of Robotics Research, 36(1), 3\u201315.","journal-title":"The International Journal of Robotics Research"},{"key":"6322_CR27","doi-asserted-by":"crossref","unstructured":"Mahon, L., Giunchiglia, E., Li, B., & Lukasiewicz, T. (2020). Knowledge graph extraction from videos. In 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA).","DOI":"10.1109\/ICMLA51294.2020.00014"},{"key":"6322_CR28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1471-2105-11-2","volume":"11","author":"L Schietgat","year":"2010","unstructured":"Schietgat, L., Vens, C., Struyf, J., Blockeel, H., Kocev, D., & Dzeroski, S. (2010). Predicting gene function using hierarchical multi-label decision tree ensembles. BMC Bioinformatics, 11, 1\u201314.","journal-title":"BMC Bioinformatics"},{"issue":"1","key":"6322_CR29","doi-asserted-by":"publisher","first-page":"1036","DOI":"10.1109\/TPAMI.2022.3150906","volume":"45","author":"G Singh","year":"2022","unstructured":"Singh, G., Akrigg, S., Maio, M. D., Fontana, V., Alitappeh, R. J., Saha, S., Saravi, K. J., Yousefi, F., Culley, J., Nicholson, T., Omokeowa, J., Khan, S., Grazioso, S., Bradley, A., Gironimo, G. D., & Cuzzolin, F. (2022). ROAD: The road event awareness dataset for autonomous driving. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1), 1036\u20131054.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"6322_CR30","doi-asserted-by":"crossref","unstructured":"Singh, G., & Cuzzolin, F. (2019). Recurrent convolutions for causal 3D CNNs. In Proceedings of the IEEE\/CVF International Conference on Computer Vision Workshops.","DOI":"10.1109\/ICCVW.2019.00183"},{"key":"6322_CR31","volume-title":"Software Engineering","author":"I Sommerville","year":"2011","unstructured":"Sommerville, I. (2011). Software Engineering. London: Pearson."},{"issue":"2008","key":"6322_CR32","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1007\/s10994-008-5077-3","volume":"73","author":"C Vens","year":"2008","unstructured":"Vens, C., Struyf, J., Schietgat, L., Dzeroski, S., & Blockeel, H. (2008). Decision trees for hierarchical multi-label classification. Machine Learning, 73(2008), 185\u2013214.","journal-title":"Machine Learning"},{"key":"6322_CR33","doi-asserted-by":"crossref","unstructured":"Wang, X., Girshick, R., Gupta, A., & He, K. (2018). Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition.","DOI":"10.1109\/CVPR.2018.00813"},{"key":"6322_CR34","doi-asserted-by":"crossref","unstructured":"Wehrmann, J., Cerri, R., & Barros, R. C. (2018). Hierarchical multi-label classification networks. In International Conference on Machine Learning Proceedings, ICML.","DOI":"10.1145\/3019612.3019664"},{"key":"6322_CR35","unstructured":"Xu, J., Zhang, Z., Friedman, T., Liang, Y., & Van den Broeck, G. (2018). A semantic loss function for deep learning with symbolic knowledge. In Proceedings of International Conference on Machine Learning."}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-023-06322-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-023-06322-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-023-06322-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,10]],"date-time":"2023-08-10T17:07:26Z","timestamp":1691687246000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-023-06322-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,1]]},"references-count":36,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2023,9]]}},"alternative-id":["6322"],"URL":"https:\/\/doi.org\/10.1007\/s10994-023-06322-z","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,1]]},"assertion":[{"value":"7 June 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"31 January 2023","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 February 2023","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 May 2023","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"ROAD-R consists of the set of logical constraints in Appendix\u00a0D, on top of the existing dataset ROAD, which is publicly available, and which is linked. Thus, ethical issues related to person identification do not apply to ROAD-R.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}}]}}