{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,3]],"date-time":"2025-10-03T17:48:00Z","timestamp":1759513680052,"version":"3.38.0"},"reference-count":58,"publisher":"China Science Publishing & Media Ltd.","issue":"4","license":[{"start":{"date-parts":[[2021,8,5]],"date-time":"2021-08-05T00:00:00Z","timestamp":1628121600000},"content-version":"vor","delay-in-days":216,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,10,25]]},"abstract":"<jats:p>Recent success of knowledge graphs has spurred interest in applying them in open science, such as on intelligent survey systems for scientists. However, efforts to understand the quality of candidate survey questions provided by these methods have been limited. Indeed, existing methods do not consider the type of on-the-fly content planning that is possible for face-to-face surveys and hence do not guarantee that selection of subsequent questions is based on response to previous questions in a survey. To address this limitation, we propose a dynamic and informative solution for an intelligent survey system that is based on knowledge graphs. To illustrate our proposal, we look into social science surveys, focusing on ordering the questions of a questionnaire component by their level of acceptance, along with conditional triggers that further customise participants' experience. Our main findings are: (i) evaluation of the proposed approach shows that the dynamic component can be beneficial in terms of lowering the number of questions asked per variable, thus allowing more informative data to be collected in a survey of equivalent length; and (ii) a primary advantage of the proposed approach is that it enables grouping of participants according to their responses, so that participants are not only served appropriate follow-up questions, but their responses to these questions may be analysed in the context of some initial categorisation. We believe that the proposed approach can easily be applied to other social science surveys based on grouping definitions in their contexts. The knowledge-graph-based intelligent survey approach proposed in our work allows online questionnaires to approach face-to-face interaction in their level of informativity and responsiveness, as well as duplicating certain advantages of interview-based data collection.<\/jats:p>","DOI":"10.1162\/dint_a_00107","type":"journal-article","created":{"date-parts":[[2021,8,5]],"date-time":"2021-08-05T16:31:06Z","timestamp":1628181066000},"page":"477-506","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":7,"title":["A Knowledge Graph Based Approach to Social Science\n                    Surveys"],"prefix":"10.3724","volume":"3","author":[{"given":"Jeff Z.","family":"Pan","sequence":"first","affiliation":[{"name":"School of Informatics, University of Edinburgh, Edinburgh EH8 9YL, UK"},{"name":"Department of Computing Science, University of Aberdeen, Aberdeen AB24 3FX, UK"}]},{"given":"Elspeth","family":"Edelstein","sequence":"additional","affiliation":[{"name":"School of Language, Literature, Music and Visual Culture, University of Aberdeen, Aberdeen AB24 3FX, UK"}]},{"given":"Patrik","family":"Bansky","sequence":"additional","affiliation":[{"name":"Department of Computing Science, University of Aberdeen, Aberdeen AB24 3FX, UK"}]},{"given":"Adam","family":"Wyner","sequence":"additional","affiliation":[{"name":"School of Law and Department of Computer Science, Swansea University, Swansea, West Glamorgan SA2 8PP, UK"}]}],"member":"2026","published-online":{"date-parts":[[2021,10,25]]},"reference":[{"key":"2021102517000343100_ref1","doi-asserted-by":"crossref","first-page":"435","DOI":"10.1007\/s11135-005-8081-8","article-title":"Conducting online surveys","volume":"40","author":"Van\n                                Selm","year":"2006","journal-title":"Quality\n                        & Quantity"},{"key":"2021102517000343100_ref2","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1007\/978-1-4614-3876-2_2","article-title":"Classification of surveys","volume-title":"Handbook of Survey Methodology for the Social Sciences","author":"Stoop","year":"2012"},{"key":"2021102517000343100_ref3","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1007\/978-1-4614-3876-2_15","article-title":"Repeated cross-sectional surveys using FTF","volume-title":"Handbook of Survey Methodology for the Social Sciences","author":"Stoop","year":"2012"},{"volume-title":"The Sage Encyclopedia of Social Science Research Methods\n                        (2004)","author":"Lewis-Beck","key":"2021102517000343100_ref4"},{"volume-title":"Research methods knowledge base (2007)","author":"Trochim","key":"2021102517000343100_ref5"},{"key":"2021102517000343100_ref6","first-page":"3","article-title":"Knowledge driven intelligent survey systems for\n                        linguists","volume-title":"Joint International Semantic\n                        Technology Conference","author":"Soares","year":"2018"},{"volume-title":"Exploiting linked data and knowledge graphs in large\n                        organisations","year":"2016","author":"Pan","key":"2021102517000343100_ref7"},{"key":"2021102517000343100_ref8","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-49493-7","volume-title":"Reasoning Web: Logical foundation of knowledge graph construction\n                        and querying answering","author":"Pan","year":"2017"},{"key":"2021102517000343100_ref9","first-page":"1","article-title":"Reducing ambiguity in tagging systems with folksonomy search\n                        expansion","volume-title":"The Proceedings of the 6th European\n                        Semantic Web Conference (ESWC2009)","author":"Pan","year":"2009"},{"key":"2021102517000343100_ref10","doi-asserted-by":"crossref","first-page":"1387","DOI":"10.1145\/3209978.3210187","article-title":"Utilizing knowledge graphs for text-centric information\n                        retrieval","volume-title":"Proceedings of the 41st International\n                        ACM SIGIR Conference on Research and Development in Information Retrieval\n                        (SIGIR 2018)","author":"Dietz","year":"2018"},{"key":"2021102517000343100_ref11","first-page":"114","article-title":"Relevance search over schema-rich knowledge\n                        graphs","volume-title":"Proceedings of the 12th ACM\n                        International WSDM Conference (WSDM2019)","author":"Gu","year":"2019"},{"key":"2021102517000343100_ref12","first-page":"950","volume-title":"KGAT: Knowledge graph attention network for recommendation","author":"Wang","year":"2019"},{"key":"2021102517000343100_ref13","first-page":"1327","article-title":"Knowledge driven phenotyping","volume-title":"Proceedings of Medical Informatics Europe (MIE 2020)","author":"Wu","year":"2020"},{"key":"2021102517000343100_ref14","doi-asserted-by":"crossref","first-page":"104877","DOI":"10.1016\/j.tiv.2020.104877","article-title":"Applying knowledge-driven mechanistic inference to\n                        toxicogenomics","volume":"66","author":"Tripodi","year":"2020","journal-title":"Toxicology in Vitro"},{"key":"2021102517000343100_ref15","doi-asserted-by":"crossref","first-page":"678","DOI":"10.1145\/3308560.3317701","article-title":"Knowledge-driven stock trend prediction and explanation via\n                        temporal convolutional network","volume-title":"Proceedings of\n                        the World Wide Web Conference (WWW 2019)","author":"Deng","year":"2019"},{"issue":"1","key":"2021102517000343100_ref16","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1108\/JKM-11-2014-0452","article-title":"Sko types: An entity-based scientific knowledge objects\n                        metadata schema","volume":"19","author":"Xu","year":"2015","journal-title":"Journal of Knowledge\n                        Management"},{"key":"2021102517000343100_ref17","doi-asserted-by":"crossref","first-page":"1327","DOI":"10.1145\/3227609.3227689","article-title":"Towards a knowledge graph for science","volume-title":"Proceedings of the 8th International Conference on Web Intelligence,\n                        Mining and Semantics (WIMS 2018)","author":"Auer","year":"2018"},{"key":"2021102517000343100_ref18","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1007\/s00354-020-00087-y","article-title":"Knowledge-driven intelligent survey systems towards open\n                        science","volume":"38","author":"Edelstein","year":"2020","journal-title":"New Generation Computing"},{"key":"2021102517000343100_ref19","first-page":"669","article-title":"Content based fake news detection using knowledge\n                        graphs","volume-title":"Proceedings of the International\n                        Semantic Web Conference (ISWC2018)","author":"Pan","year":"2018"},{"key":"2021102517000343100_ref20","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-31226-7","volume-title":"Ontology-driven software development","author":"Pan","year":"2013"},{"volume-title":"Semantic Web enabled software engineering","year":"2014","author":"Pan","key":"2021102517000343100_ref21"},{"key":"2021102517000343100_ref22","first-page":"1","article-title":"Towards ontology-driven requirements\n                        engineering","volume-title":"Proceedings of the International\n                        Workshop on Semantic Web Enabled Software Engineering (SWESE2011)","author":"Siegemund","year":"2011"},{"key":"2021102517000343100_ref23","first-page":"429","article-title":"Reasoning driven configuration of linked data content\n                        management systems","volume-title":"Proceedings of the 3rd Joint\n                        International Conference on Semantic Technologies (JIST 2013)","author":"Taylor","year":"2013"},{"key":"2021102517000343100_ref24","first-page":"465","volume-title":"A knowledge graph for industry 4.0","author":"Bader","year":"2020"},{"volume-title":"The description logic handbook: Theory, implementation, and\n                        applications","year":"2003","author":"Baader","key":"2021102517000343100_ref25"},{"key":"2021102517000343100_ref26","first-page":"1434","article-title":"Approximating OWL-DL ontologies","volume-title":"The Proceedings of the 22nd National Conference on Artificial\n                        Intelligence (AAAI-07)","author":"Pan","year":"2007"},{"key":"2021102517000343100_ref27","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1016\/j.artint.2015.10.004","article-title":"Tractable approximate deduction for OWL","volume":"235","author":"Pan","year":"2016","journal-title":"Artificial Intelligence"},{"issue":"1\u20132","key":"2021102517000343100_ref28","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1007\/s10472-010-9187-9","article-title":"Forgetting for knowledge bases in DL-Lite","volume":"58","author":"Wang","year":"2010","journal-title":"Journal of Annals of Mathematics and Artificial\n                        Intelligence"},{"key":"2021102517000343100_ref29","first-page":"117","article-title":"Foundations for uniform interpolation and forgetting in\n                        expressive description logics","volume-title":"Proceedings of\n                        the 22nd International Joint Conference on Artificial Intelligence (IJCAI\n                        2011)","author":"Lutz","year":"2011"},{"issue":"2","key":"2021102517000343100_ref30","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1111\/j.1467-8640.2012.00442.x","article-title":"Eliminating concepts and roles from ontologies in expressive\n                        descriptive logics","volume":"30","author":"Wang","year":"2014","journal-title":"Computing Intelligence"},{"key":"2021102517000343100_ref31","first-page":"28","article-title":"f-SWRL: A fuzzy extension of SWRL","volume":"6","author":"Pan","year":"2006","journal-title":"Journal of Data Semantic"},{"key":"2021102517000343100_ref32","first-page":"119","article-title":"Handling imprecise knowledge with fuzzy description\n                        logic","volume-title":"The Proceedings of the 2006 International\n                        Workshop on Description Logics (DL2006)","author":"Stoilos","year":"2006"},{"key":"2021102517000343100_ref33","first-page":"1","article-title":"A possibilistic extension of description\n                        logics","volume-title":"Proceedings of 2007 International\n                        Workshop on Description Logics (DL2007)","author":"Qi","year":"2007"},{"issue":"4","key":"2021102517000343100_ref34","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1002\/int.20470","article-title":"Extending description logics with uncertainty reasoning in\n                        possibilistic logic","volume":"26","author":"Qi","year":"2011","journal-title":"International Journal of\n                        Intelligent Systems"},{"key":"2021102517000343100_ref35","doi-asserted-by":"crossref","DOI":"10.26530\/OAPEN_603356","volume-title":"The empirical base of linguistics: Grammaticality judgments and\n                        linguistic methodology","author":"Sch\u00fctze","year":"2016"},{"key":"2021102517000343100_ref36","doi-asserted-by":"crossref","first-page":"364","DOI":"10.3389\/fpsyg.2020.00364","article-title":"Acceptable ungrammatical sentences, unacceptable grammatical\n                        sentences, and the role of the cognitive parser","volume":"11","author":"Leivada","year":"2020","journal-title":"Frontiers in Psychology"},{"key":"2021102517000343100_ref37","doi-asserted-by":"crossref","first-page":"242","DOI":"10.1093\/acprof:oso\/9780199367221.003.0008","article-title":"This syntax needs studied","volume-title":"Micro-syntactic variation in North American English","author":"Edelstein","year":"2014"},{"issue":"1","key":"2021102517000343100_ref38","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1215\/00031283-77-1-32","article-title":"At the intersection of regional and social dialects: The case\n                        of like + past participle in American English","volume":"77","author":"Murray","year":"2002","journal-title":"American Speech"},{"volume-title":"The British-Irish dialect quiz","year":"2019","author":"Katz","key":"2021102517000343100_ref39"},{"key":"2021102517000343100_ref40","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1007\/978-3-8349-3722-3_11","article-title":"Dise: Dynamic intelligent survey engine","volume-title":"Quantitative Marketing and Marketing Management","author":"Schlereth","year":"2012"},{"issue":"2","key":"2021102517000343100_ref41","first-page":"175","article-title":"Minijudge: Software for small-scale experimental\n                        syntax","volume":"12","author":"Myers","year":"2007","journal-title":"International Journal of Computational\n                        Linguistics & Chinese Language Processing"},{"issue":"1","key":"2021102517000343100_ref42","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3758\/BRM.41.1.12","article-title":"Timing accuracy of web experiments: A case study using the\n                        webexp software package","volume":"41","author":"Keller","year":"2009","journal-title":"Behavior Research\n                        Methods"},{"issue":"1","key":"2021102517000343100_ref43","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1177\/0098628316677643","article-title":"Psytoolkit: A novel web-based method for running online\n                        questionnaires and reaction-time experiments","volume":"44","author":"Stoet","year":"2017","journal-title":"Teaching of Psychology"},{"volume-title":"ibex 0.3.7 manual (2013)","author":"Drummond","key":"2021102517000343100_ref44"},{"issue":"1","key":"2021102517000343100_ref45","first-page":"4","article-title":"Worldlikeness: A web-based tool for typological\n                        psycholinguistic research","volume":"23","author":"Chen","year":"2017","journal-title":"University of\n                        Pennsylvania Working Papers in Linguistics"},{"issue":"4","key":"2021102517000343100_ref46","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1177\/0098628312456615","article-title":"Participants at your fingertips: Using Amazon's\n                        Mechanical Turk to increase student-faculty collaborative\n                        research","volume":"39","author":"Johnson","year":"2012","journal-title":"Teaching of Psychology"},{"issue":"8","key":"2021102517000343100_ref47","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1111\/j.1749-818X.2011.00295.x","article-title":"Using Mechanical Turk to obtain and analyze English\n                        acceptability judgments","volume":"5","author":"Gibson","year":"2011","journal-title":"Language and Linguistics\n                        Compass"},{"issue":"2","key":"2021102517000343100_ref48","doi-asserted-by":"crossref","first-page":"481","DOI":"10.1007\/s11049-015-9305-9","article-title":"A streamlined approach to online linguistic\n                        surveys","volume":"34","author":"Erlewine","year":"2016","journal-title":"Natural Language & Linguistic\n                        Theory"},{"volume-title":"Survey software buyers'\n                    guide","year":"2019","author":"Capterra","key":"2021102517000343100_ref49"},{"volume-title":"Software advice, buyer's guide","key":"2021102517000343100_ref50"},{"volume-title":"An optimization framework for adaptive questionnaire design\n                        (2004)","author":"Abernethy","key":"2021102517000343100_ref51"},{"issue":"4","key":"2021102517000343100_ref52","doi-asserted-by":"crossref","first-page":"999","DOI":"10.1016\/j.compedu.2009.10.003","article-title":"Ah-questionnaire: An adaptive hierarchical questionnaire for\n                        learning styles","volume":"54","author":"Ortigosa","year":"2010","journal-title":"Computers &\n                        Education"},{"key":"2021102517000343100_ref53","first-page":"1412","article-title":"Implementation of an adaptive questionnaire","volume-title":"Proceedings of the ED-MEDIA Conference","author":"Nokelainen","year":"2001"},{"key":"2021102517000343100_ref54","first-page":"399","article-title":"An adaptive questionnaire for automatic identification of\n                        learning styles","volume-title":"International Conference on\n                        Industrial, Engineering and Other Applications of Applied Intelligent\n                        Systems","author":"Mwamikazi","year":"2014"},{"key":"2021102517000343100_ref55","first-page":"224","article-title":"A dynamic questionnaire to further reduce questions in\n                        learning style assessment","volume-title":"IFIP International\n                        Conference on Artificial Intelligence Applications and Innovations","author":"Mwamikazi","year":"2014"},{"issue":"4","key":"2021102517000343100_ref56","first-page":"311","article-title":"UK scholarly e-book usage: A landmark survey","volume":"60","author":"Nicholas","year":"2008","journal-title":"Aslib Journal of Information Management"},{"key":"2021102517000343100_ref57","first-page":"456","article-title":"An eye tracking study of how pictures influence online\n                        reading","volume-title":"Proceedings of IFIP Conference on\n                        Human-Computer Interaction","author":"Beymer","year":"2007"},{"key":"2021102517000343100_ref58","doi-asserted-by":"crossref","DOI":"10.4135\/9781412986045","volume-title":"Survey questions: Handcrafting the standardized\n                        questionnaire","author":"Converse","year":"1986"}],"container-title":["Data Intelligence"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/direct.mit.edu\/dint\/article-pdf\/3\/4\/477\/1968576\/dint_a_00107.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/direct.mit.edu\/dint\/article-pdf\/3\/4\/477\/1968576\/dint_a_00107.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,14]],"date-time":"2025-03-14T07:43:12Z","timestamp":1741938192000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.sciengine.com\/doi\/10.1162\/dint_a_00107"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":58,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,10,25]]}},"URL":"https:\/\/doi.org\/10.1162\/dint_a_00107","relation":{},"ISSN":["2641-435X"],"issn-type":[{"type":"electronic","value":"2641-435X"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}