{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T02:22:19Z","timestamp":1783477339534,"version":"3.55.0"},"reference-count":125,"publisher":"MIT Press - Journals","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational Linguistics"],"published-print":{"date-parts":[[2020,1]]},"abstract":"<jats:p> To ensure readability, text is often written and presented with due formatting. These text formatting devices help the writer to effectively convey the narrative. At the same time, these help the readers pick up the structure of the discourse and comprehend the conveyed information. There have been a number of linguistic theories on discourse structure of text. However, these theories only consider unformatted text. Multimedia text contains rich formatting features that can be leveraged for various NLP tasks. In this article, we study some of these discourse features in multimedia text and what communicative function they fulfill in the context. As a case study, we use these features to harvest structured subject knowledge of geometry from textbooks. We conclude that the discourse and text layout features provide information that is complementary to lexical semantic information. Finally, we show that the harvested structured knowledge can be used to improve an existing solver for geometry problems, making it more accurate as well as more explainable. <\/jats:p>","DOI":"10.1162\/coli_a_00360","type":"journal-article","created":{"date-parts":[[2019,10,8]],"date-time":"2019-10-08T14:59:06Z","timestamp":1570546746000},"page":"627-665","source":"Crossref","is-referenced-by-count":19,"title":["Discourse in Multimedia: A Case Study in Extracting Geometry Knowledge from Textbooks"],"prefix":"10.1162","volume":"45","author":[{"given":"Mrinmaya","family":"Sachan","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, School of Computer Science, Machine Learning Department."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Avinava","family":"Dubey","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, School of Computer Science, Machine Learning Department"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eduard H.","family":"Hovy","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, School of Computer Science, Language Technologies Institute"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tom M.","family":"Mitchell","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, School of Computer Science, Machine Learning Department"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dan","family":"Roth","sequence":"additional","affiliation":[{"name":"University of Pennsylvania, Department of Computer and Information Science"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eric P.","family":"Xing","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, School of Computer Science, Machine Learning Department"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","reference":[{"key":"bib1","first-page":"245","volume-title":"Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence","author":"Alvin Chris","year":"2014"},{"key":"bib2","first-page":"305","volume-title":"Handbook of Natural Language Processing","author":"Andr\u00e9 Elisabeth","year":"2000"},{"key":"bib3","unstructured":"Andr\u00e9, Elisabeth, Wolfgang Finkler, Winfried Graf, Thomas Rist, Anne Schauder, and Wolfgang Wahlster. 1991. WIP: The automatic synthesis of multimodal presentations. Technical report, University of Saarland."},{"key":"bib4","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-55399-1_19"},{"key":"bib5","volume-title":"Proceedings of the 12th Annual Conference of the Cognitive Science Society","volume":"487","author":"Arens Yigal","year":"1990"},{"key":"bib6","first-page":"1253","volume-title":"IJCAI","author":"Arens Yigal","year":"1993"},{"key":"bib7","doi-asserted-by":"crossref","unstructured":"Arens, Yigal, Eduard H. Hovy, and Mira Vossers. 1992. On the knowledge underlying multimedia presentations. Technical report, University of Southern California Marina Del Rey Information Sciences Inst.","DOI":"10.21236\/ADA278690"},{"key":"bib8","first-page":"808","volume-title":"AAAI","author":"Arens Yigal","year":"1988"},{"key":"bib9","unstructured":"Balasubramanian, Niranjan, Stephen Soderland, Oren Etzioni Mausam, and Robert Bart. 2002. Out of the box information extraction: A case study using bio-medical texts. Technical report, University of Washington."},{"key":"bib10","first-page":"2670","volume-title":"IJCAI 2007, Proceedings of the 20th International Joint Conference on Artificial Intelligence","author":"Banko Michele","year":"2007"},{"key":"bib11","doi-asserted-by":"publisher","DOI":"10.1162\/089120101317066131"},{"key":"bib12","doi-asserted-by":"publisher","DOI":"10.1162\/089120101317066131"},{"key":"bib13","first-page":"1533","volume-title":"Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013","author":"Berant Jonathan","year":"2013"},{"key":"bib14","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.2000.926687"},{"key":"bib15","doi-asserted-by":"publisher","DOI":"10.3115\/1220575.1220646"},{"key":"bib16","first-page":"827","volume-title":"EMNLP-CoNLL","author":"Canisius Sander","year":"2007"},{"key":"bib17","first-page":"23","volume-title":"Proceedings of the Workshop on Pragmatics of Question Answering at HLT-NAACL 2004","author":"Chai Joyce Y.","year":"2004"},{"key":"bib18","first-page":"55","volume-title":"2008 Annual Conference of the Association for Computational Linguistics (ACL)","author":"Chan Yee Seng","year":"2008"},{"key":"bib19","doi-asserted-by":"publisher","DOI":"10.1016\/S0167-9236(02)00100-8"},{"key":"bib20","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2006.152"},{"key":"bib21","first-page":"859","volume-title":"Proceedings of the 25th AAAI Conference on Artificial Intelligence (AAAI-2011)","author":"Chen David L.","year":"2011"},{"key":"bib22","doi-asserted-by":"publisher","DOI":"10.1142\/2196"},{"key":"bib23","first-page":"74","volume-title":"Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction","author":"Clark Peter","year":"2012"},{"issue":"1","key":"bib24","first-page":"11","volume":"13","author":"Cohen Robin","year":"1987","journal-title":"Computational Linguistics"},{"key":"bib25","first-page":"2184","volume-title":"Association for Computational Linguistics (ACL)","author":"Conrath Juliette","year":"2014"},{"key":"bib26","first-page":"110","volume-title":"Proceedings of a Workshop on Text Representation and Domain Modelling: Ideas from Linguistics and AI","author":"Dale Robert","year":"1991"},{"key":"bib27","first-page":"13","volume-title":"Working Notes for the AAAI Fall Symposium on Discourse Structure in Natural Language Understanding and Generation","author":"Dale Robert","year":"1991"},{"key":"bib28","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-1303"},{"key":"bib29","unstructured":"Davis, Tom. 2006. Geometry with computers. Technical report."},{"key":"bib30","first-page":"250","volume-title":"Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics","author":"Denkowski Michael","year":"2010"},{"key":"bib31","first-page":"49","author":"Dijk Teun A. Van","year":"1979","journal-title":"Text Processing"},{"key":"bib32","doi-asserted-by":"publisher","DOI":"10.3115\/1690219.1690239"},{"key":"bib33","author":"Dwyer F. M.","year":"1978","journal-title":"Learning Services"},{"key":"bib34","doi-asserted-by":"publisher","DOI":"10.1145\/1409360.1409378"},{"key":"bib35","first-page":"391","volume-title":"Proceedings of the Nineteenth National Conference on Artificial Intelligence, Sixteenth Conference on Innovative Applications of Artificial Intelligence","author":"Etzioni Oren","year":"2004"},{"key":"bib36","volume-title":"Computers and Thought","author":"Feigenbaum Edward A.","year":"1963"},{"key":"bib37","doi-asserted-by":"publisher","DOI":"10.1145\/49103.1046411"},{"key":"bib38","doi-asserted-by":"publisher","DOI":"10.1109\/2.97249"},{"key":"bib39","first-page":"60","volume-title":"Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1","author":"Feng Vanessa Wei","year":"2012"},{"key":"bib40","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-1048"},{"key":"bib41","doi-asserted-by":"publisher","DOI":"10.1017\/S1471068414000076"},{"key":"bib42","volume-title":"Instructional Message Design: Principles from the Behavioral Sciences","author":"Fleming M. L.","year":"1978"},{"key":"bib43","first-page":"44","volume-title":"International Workshop on Automated Deduction in Geometry","author":"Gao Xiao-Shan","year":"2002"},{"key":"bib44","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1168"},{"key":"bib45","first-page":"150","volume-title":"Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue","author":"Ghosh Sucheta","year":"2012"},{"issue":"3","key":"bib46","first-page":"175","volume":"12","author":"Grosz Barbara J.","year":"1986","journal-title":"Computational Linguistics"},{"key":"bib47","doi-asserted-by":"publisher","DOI":"10.1145\/1993498.1993505"},{"key":"bib48","volume-title":"Designing Instructional Text","author":"Hartley J.","year":"1985"},{"key":"bib49","first-page":"262","volume-title":"Readings in Intelligent User Interfaces","author":"Hovy Eduard H","year":"1998"},{"key":"bib50","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-45221-5_31"},{"key":"bib51","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-1092"},{"key":"bib52","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-1002"},{"key":"bib53","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-017-1616-1"},{"key":"bib54","doi-asserted-by":"publisher","DOI":"10.1016\/S0747-7171(86)80007-4"},{"key":"bib55","first-page":"1062","volume-title":"Proceedings of AAAI-05","author":"Kate Rohit J.","year":"2005"},{"key":"bib56","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46493-0_15"},{"key":"bib57","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.571"},{"key":"bib58","doi-asserted-by":"publisher","DOI":"10.1613\/jair.53"},{"key":"bib59","first-page":"282","volume-title":"Proceedings of the Eighteenth International Conference on Machine Learning, ICML","volume":"1","author":"Lafferty John","year":"2001"},{"key":"bib60","doi-asserted-by":"publisher","DOI":"10.1111\/j.1551-6708.1987.tb00863.x"},{"key":"bib61","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4020-5958-2_5"},{"key":"bib62","first-page":"1294","volume-title":"Association for Computational Linguistics (ACL)","author":"Lei Tao","year":"2013"},{"key":"bib63","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-1003"},{"key":"bib64","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1193"},{"key":"bib65","first-page":"590","volume-title":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1","author":"Liang Percy","year":"2011"},{"key":"bib66","first-page":"74","volume-title":"Text Summarization Branches Out: Proceedings of the ACL-04 Workshop","volume":"8","author":"Lin Chin-Yew","year":"2004"},{"key":"bib67","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324912000307"},{"key":"bib68","author":"Ling Wang","year":"2016","journal-title":"arXiv preprint arXiv:1603.06744"},{"key":"bib69","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1015"},{"key":"bib70","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348407"},{"key":"bib71","first-page":"4574","volume-title":"Advances in Neural Information Processing Systems 29","author":"Liu Chang","year":"2016"},{"key":"bib72","doi-asserted-by":"publisher","DOI":"10.1613\/jair.5002"},{"key":"bib73","volume-title":"Some Aspects of Text Grammars","author":"Longacre Robert E.","year":"1983"},{"key":"bib74","first-page":"147","volume-title":"Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue","author":"Louis Annie","year":"2010"},{"key":"bib75","first-page":"35","volume-title":"Understanding or Retrieval of Documents, AAAI Fall Symposium.","author":"Luc Christophe","year":"1999"},{"key":"bib76","doi-asserted-by":"publisher","DOI":"10.1145\/22949.22950"},{"issue":"8","key":"bib77","first-page":"234","volume":"3","author":"Mann William C.","year":"1988","journal-title":"Text"},{"key":"bib78","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-5010"},{"key":"bib79","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/6754.001.0001","volume-title":"The Theory and Practice of Discourse Parsing and Summarization","author":"Marcu Daniel","year":"2000"},{"key":"bib80","first-page":"251","volume-title":"2010 5th ACM\/IEEE International Conference on Human-Robot Interaction (HRI)","author":"Matuszek Cynthia","year":"2010"},{"key":"bib81","volume-title":"Planning Multimedia Explanations Using Communicative Acts","author":"Maybury M","year":"1998"},{"key":"bib82","doi-asserted-by":"publisher","DOI":"10.1037\/0022-0663.81.2.240"},{"key":"bib83","first-page":"2302","volume-title":"Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI-15)","author":"Mitchell T.","year":"2015"},{"issue":"3","key":"bib84","first-page":"409","volume":"22","author":"Moser Megan","year":"1996","journal-title":"Computational Linguistics"},{"key":"bib85","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P15-1121"},{"key":"bib86","doi-asserted-by":"crossref","unstructured":"Neal, J. G., S. C. Shapiro, C. Y. Thielman, J. R. Gucwa, and J. M. Lammens. 1990. Intelligent multi-media integrated interface project. Technical report RADC-TR-90-128, Calspan UB Research Center, Buffalo, NY.","DOI":"10.21236\/ADA225973"},{"key":"bib87","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-60800-1_31"},{"key":"bib88","first-page":"41","volume-title":"Proceedings of the Association for Computational Linguistics Workshop on Punctuation","author":"Pascual Elsa","year":"1996"},{"key":"bib89","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2005.09.002"},{"key":"bib90","volume-title":"Proceedings of ECCE-5","author":"Petre M.","year":"1990"},{"key":"bib91","doi-asserted-by":"publisher","DOI":"10.1016\/0378-2166(88)90050-1"},{"key":"bib92","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P15-1085"},{"key":"bib93","unstructured":"Reed, Chris and Derek Long. 1997. Generating punctuation in written arguments. Technical report 2694743, Department of Computer Science, University College, London."},{"key":"bib94","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P15-1024"},{"key":"bib95","first-page":"177","volume-title":"Proceedings of the World Congress on Engineering and Computer Science","volume":"1","author":"Saleem Ozair","year":"2012"},{"key":"bib96","volume-title":"Geometry Turned On: Dynamic Software in Learning, Teaching, and Research","author":"Schattschneider Doris","year":"1997"},{"key":"bib97","first-page":"2831","volume-title":"Proceedings of AAAI","author":"Seo Min Joon","year":"2014"},{"key":"bib98","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1171"},{"key":"bib99","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2105-4-20"},{"key":"bib100","first-page":"1488","volume-title":"IJCAI 2009, Proceedings of the 21st International Joint Conference on Artificial Intelligence","author":"Shimizu Nobuyuki","year":"2009"},{"key":"bib101","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46478-7_41"},{"key":"bib102","doi-asserted-by":"publisher","DOI":"10.3115\/1073445.1073475"},{"key":"bib103","first-page":"197","volume-title":"AAAI Workshop on Intelligent Multimedia Interfaces","author":"Stock Oliviero","year":"1993"},{"key":"bib104","first-page":"583","volume":"3","author":"Strehl Alexander","year":"2002","journal-title":"Journal of Machine Learning Research"},{"key":"bib105","volume-title":"The Elements of Style","author":"Strunk William","year":"2007"},{"key":"bib106","doi-asserted-by":"publisher","DOI":"10.3115\/1620754.1620837"},{"key":"bib107","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2007.04.005"},{"key":"bib108","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-12-223260-2.50016-5"},{"key":"bib109","volume-title":"Some Aspects of Text Grammars","author":"Van Dijk Teun A.","year":"1972"},{"key":"bib110","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-58146-5_7"},{"key":"bib111","doi-asserted-by":"publisher","DOI":"10.1007\/11765448_18"},{"key":"bib112","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/K15-2002"},{"key":"bib113","doi-asserted-by":"publisher","DOI":"10.1145\/2682571.2797062"},{"key":"bib114","first-page":"317","volume-title":"Proceedings of the 25th ACM International Conference on Information and Knowledge Management","author":"Wang Shuting","year":"2016"},{"key":"bib115","doi-asserted-by":"publisher","DOI":"10.1007\/BF02328447"},{"key":"bib116","author":"White Michael","year":"1995","journal-title":"CoRR"},{"key":"bib117","volume-title":"Workshop on User Interfaces for Theorem Proving (UITP)","author":"Wilson Sean","year":"2005"},{"key":"bib118","doi-asserted-by":"publisher","DOI":"10.1145\/2815833.2815834"},{"key":"bib119","author":"Yaghmazadeh Navid","year":"2017","journal-title":"CoRR"},{"key":"bib120","doi-asserted-by":"publisher","DOI":"10.1145\/2684822.2685292"},{"key":"bib121","first-page":"702","volume-title":"Proceeding of ACL, Volume 2","author":"Yao Xuchen","year":"2013"},{"key":"bib122","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1041"},{"key":"bib123","first-page":"817","volume-title":"Proceedings of the 11th National Conference on Artificial Intelligence","author":"Zelle John M.","year":"1993"},{"key":"bib124","first-page":"1050","volume-title":"Proceedings of the Thirteenth National Conference on Artificial Intelligence","author":"Zelle John M.","year":"1996"},{"key":"bib125","author":"Zettlemoyer Luke S.","year":"2012","journal-title":"arXiv preprint arXiv:1207.1420"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/coli_a_00360","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:28:27Z","timestamp":1615584507000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/45\/4\/627-665\/93358"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1]]},"references-count":125,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,1]]}},"alternative-id":["10.1162\/coli_a_00360"],"URL":"https:\/\/doi.org\/10.1162\/coli_a_00360","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1]]}}}