{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,24]],"date-time":"2026-01-24T19:29:48Z","timestamp":1769282988874,"version":"3.49.0"},"reference-count":72,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2023,7,14]],"date-time":"2023-07-14T00:00:00Z","timestamp":1689292800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["SMA 1416730"],"award-info":[{"award-number":["SMA 1416730"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Agent-based model (ABM) development needs information on system components and interactions. Qualitative narratives contain contextually rich system information beneficial for ABM conceptualization. Traditional qualitative data extraction is manual, complex, and time- and resource-consuming. Moreover, manual data extraction is often biased and may produce questionable and unreliable models. A possible alternative is to employ automated approaches borrowed from Artificial Intelligence. This study presents a largely unsupervised qualitative data extraction framework for ABM development. Using semantic and syntactic Natural Language Processing tools, our methodology extracts information on system agents, their attributes, and actions and interactions. In addition to expediting information extraction for ABM, the largely unsupervised approach also minimizes biases arising from modelers\u2019 preconceptions about target systems. We also introduce automatic and manual noise-reduction stages to make the framework usable on large semi-structured datasets. We demonstrate the approach by developing a conceptual ABM of household food security in rural Mali. The data for the model contain a large set of semi-structured qualitative field interviews. The data extraction is swift, predominantly automatic, and devoid of human manipulation. We contextualize the model manually using the extracted information. We also put the conceptual model to stakeholder evaluation for added credibility and validity.<\/jats:p>","DOI":"10.3390\/a16070338","type":"journal-article","created":{"date-parts":[[2023,7,14]],"date-time":"2023-07-14T08:40:06Z","timestamp":1689324006000},"page":"338","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["A Largely Unsupervised Domain-Independent Qualitative Data Extraction Approach for Empirical Agent-Based Model Development"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3203-4534","authenticated-orcid":false,"given":"Rajiv","family":"Paudel","sequence":"first","affiliation":[{"name":"Operation Research and Analysis, Idaho National Laboratory, 1955 Fremont Ave., Idaho Falls, ID 83415, USA"}]},{"given":"Arika","family":"Ligmann-Zielinska","sequence":"additional","affiliation":[{"name":"Department of Geography, Environment, and Spatial Sciences, Michigan State University, Geography Building, 673 Auditorium Rd, Room 121, East Lansing, MI 48824, USA"}]}],"member":"1968","published-online":{"date-parts":[[2023,7,14]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"590","DOI":"10.2307\/2392365","article-title":"Qualitative data as an attractive nuisance: The problem of analysis","volume":"24","author":"Miles","year":"1979","journal-title":"Adm. Sci. Q."},{"key":"ref_2","unstructured":"Mortelmans, D. (2019). The Palgrave Handbook of Methods for Media Policy Research, Palgrave Macmillan."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"371","DOI":"10.1016\/S1054-139X(99)00068-3","article-title":"The reason and rhyme of qualitative research: Why, when, and how to use qualitative methods in the study of adolescent health","volume":"25","author":"Rich","year":"1999","journal-title":"J. Adolesc. Health"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1177\/1524839912437370","article-title":"Qualitative research: The importance of conducting research that doesn\u2019t \u201ccount\u201d","volume":"13","author":"Watkins","year":"2012","journal-title":"Health Promot. Pract."},{"key":"ref_5","unstructured":"Kemp-Benedict, E. (2004, January 1). From Narrative to Number: A Role for Quantitative Models in Scenario analysis. Proceedings of the International Congress on Environmental Modelling and Software, Osnabr\u00fcck, Germany."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1287\/inte.27.2.48","article-title":"Modeling for litigation: Mixing qualitative and quantitative approaches","volume":"27","author":"Ackermann","year":"1997","journal-title":"Interfaces"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1002\/1099-1727(200023)16:3<225::AID-SDR195>3.0.CO;2-D","article-title":"Qualitative and quantitative modelling in system dynamics: Some research questions","volume":"16","author":"Coyle","year":"2000","journal-title":"Syst. Dyn. Rev. J. Syst. Dyn. Soc."},{"key":"ref_8","unstructured":"Forbus, K.D., and Falkenhainer, B. (August, January 29). Self-Explanatory Simulations: An Integration of Qualitative and Quantitative Knowledge. Proceedings of the AAAI, Boston, MA, USA."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"103259","DOI":"10.1016\/j.scs.2021.103259","article-title":"Compatibility of quantitative and qualitative data-collection protocols for urban soundscape evaluation","volume":"74","author":"Jo","year":"2021","journal-title":"Sustain. Cities Soc."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"422","DOI":"10.1057\/palgrave.jors.2600700","article-title":"Qualitative vs quantitative modelling: The evolving balance","volume":"50","author":"Wolstenholme","year":"1999","journal-title":"J. Oper. Res. Soc."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Djenontin, I.N.S., Zulu, L.C., and Ligmann-Zielinska, A. (2020). Improving representation of decision rules in LUCC-ABM: An example with an elicitation of farmers\u2019 decision making for landscape restoration in central Malawi. Sustainability, 12.","DOI":"10.3390\/su12135380"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"10","DOI":"10.18564\/jasss.1563","article-title":"Using qualitative evidence to enhance an agent-based modelling system for studying land use change","volume":"13","author":"Polhill","year":"2010","journal-title":"J. Artif. Soc. Soc. Simul."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1037\/qup0000030","article-title":"Mending fences: Defining the domains and approaches of quantitative and qualitative research","volume":"2","author":"Landrum","year":"2015","journal-title":"Qual. Psychol."},{"key":"ref_14","unstructured":"Runck, B. (2018). GeoComputational Approaches to Evaluate the Impacts of Communication on Decision-Making in Agriculture. [Ph.D. Thesis, University of Minnesota]."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Du, J., and Ligmann-Zielinska, A. (2015). The Volatility of Data Space: Topology Oriented Sensitivity Analysis. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0137591"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1016\/j.ecolmodel.2014.01.018","article-title":"Towards better modelling and decision support: Documenting model development, testing, and analysis using TRACE","volume":"280","author":"Grimm","year":"2014","journal-title":"Ecol. Model."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Ligmann-Zielinska, A., Siebers, P.-O., Magliocca, N., Parker, D.C., Grimm, V., Du, J., Cenek, M., Radchuk, V., Arbab, N.N., and Li, S. (2020). \u2018One size does not fit all\u2019: A roadmap of purpose-driven mixed-method pathways for sensitivity analysis of agent-based models. J. Artif. Soc. Soc. Simul., 23.","DOI":"10.18564\/jasss.4201"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1111\/j.1467-8306.2005.00450.x","article-title":"Exploring Complexity in a Human\u2013Environment System: An Agent-Based Spatial Model for Multidisciplinary and Multiscale Integration","volume":"95","author":"An","year":"2005","journal-title":"Ann. Assoc. Am. Geogr."},{"key":"ref_19","unstructured":"Railsback, S.F., and Grimm, V. (2019). Agent-Based and Individual-Based Modeling: A Practical Introduction, Princeton University Press."},{"key":"ref_20","unstructured":"Wilensky, U., and Rand, W. (2015). An Introduction to Agent-Based Modeling: Modeling Natural, Social, and Engineered Complex Systems with NetLogo, Mit Press."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Janssen, M., and Ostrom, E. (2006). Empirically based, agent-based models. Ecol. Soc., 11.","DOI":"10.5751\/ES-01861-110237"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1080\/1747423X.2015.1030463","article-title":"Strategic directions for agent-based modeling: Avoiding the YAAWN syndrome","volume":"11","author":"Evans","year":"2016","journal-title":"J. Land. Use Sci."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1080\/17474230701201349","article-title":"Comparison of empirical methods for building agent-based models in land use science","volume":"2","author":"Robinson","year":"2007","journal-title":"J. Land. Use Sci."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Smajgl, A., and Barreteau, O. (2014). Empirical Agent-Based Modelling-Challenges and Solutions, Springer.","DOI":"10.1007\/978-1-4614-6134-0"},{"key":"ref_25","unstructured":"Seidl, R. (2014, January 1\u20135). Social scientists, qualitative data, and agent-based modeling. Proceedings of the Social Simulation Conference, Barcelona, Spain."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"2760","DOI":"10.1016\/j.ecolmodel.2010.08.019","article-title":"The ODD protocol: A review and first update","volume":"221","author":"Grimm","year":"2010","journal-title":"Ecol. Model."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1016\/j.envsoft.2014.01.029","article-title":"Standardised and transparent model descriptions for agent-based models: Current status and prospects","volume":"55","author":"Balbi","year":"2014","journal-title":"Environ. Model. Softw."},{"key":"ref_28","unstructured":"Ford, A., and Ford, F.A. (1999). Modeling the Environment: An Introduction to System Dynamics Models of Environmental Systems, Island press."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1504\/IJSPM.2012.049817","article-title":"Validation in the agent-based modelling paradigm: Problems and a solution","volume":"7","author":"Heath","year":"2012","journal-title":"Int. J. Simul. Process Model."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"105713","DOI":"10.1016\/j.envsoft.2023.105713","article-title":"Modeling agent decision and behavior in the light of data science and artificial intelligence","volume":"166","author":"An","year":"2023","journal-title":"Environ. Model. Softw."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"13","DOI":"10.18564\/jasss.2687","article-title":"How Do Agents Make Decisions? A Survey","volume":"17","author":"Balke","year":"2014","journal-title":"J. Artif. Soc. Soc. Simul."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Doscher, C., Moore, K., Smallman, C., Wilson, J., and Simmons, D. (2014). Empirical Agent-Based Modelling-Challenges and Solutions: Volume 1, The Characterisation and Parameterisation of Empirical Agent-Based Models, Springer.","DOI":"10.1007\/978-1-4614-6134-0_1"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"783","DOI":"10.1017\/ASC2006112","article-title":"Modelling farmer decision-making: Concepts, progress and challenges","volume":"82","year":"2006","journal-title":"Anim. Sci."},{"key":"ref_34","first-page":"21","article-title":"An integrated approach to simulating behavioural processes: A case study of the lock-in of consumption patterns","volume":"2","author":"Janssen","year":"1999","journal-title":"J. Artif. Soc. Soc. Simul."},{"key":"ref_35","unstructured":"Becu, N., Barreteau, O., Perez, P., Saising, J., and Sungted, S. (2005). Companion Modeling and Multi-Agent Systems for Integrated Natural Resource Management in Asia, International Rice Research Institute."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1268","DOI":"10.1016\/j.envsoft.2010.03.007","article-title":"Modelling with stakeholders","volume":"25","author":"Voinov","year":"2010","journal-title":"Environ. Model. Softw."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1177\/0894439305282346","article-title":"Understanding complex behavior and decision making using ethnographic knowledge elicitation tools (KnETs)","volume":"24","author":"Bharwani","year":"2006","journal-title":"Soc. Sci. Comput. Rev."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"17","DOI":"10.18564\/jasss.2715","article-title":"A context-and scope-sensitive analysis of narrative data to aid the specification of agent behaviour","volume":"18","author":"Edmonds","year":"2015","journal-title":"J. Artif. Soc. Soc. Simul."},{"key":"ref_39","unstructured":"Ghorbani, A., Schrauwen, N., and Dijkema, G.P.J. (2013, January 16\u201320). Using Ethnographic Information to Conceptualize Agent-based Models. Proceedings of the European Social Simulation Association Conference, Warsaw, Poland."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1007\/BF02512229","article-title":"How to build and use agent-based models in social science","volume":"1","author":"Gilbert","year":"2000","journal-title":"Mind Soc."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1016\/j.jenvman.2004.01.010","article-title":"First principles of the MameLuke multi-actor modelling framework for land use change, illustrated with a Philippine case study","volume":"72","author":"Huigen","year":"2004","journal-title":"J. Environ. Manag."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"552","DOI":"10.1016\/j.ipm.2011.07.002","article-title":"Automatically structuring domain knowledge from text: An overview of current research","volume":"48","author":"Clark","year":"2012","journal-title":"Inf. Process. Manag."},{"key":"ref_43","first-page":"47","article-title":"Natural Language Processing for Conceptual Modeling","volume":"3","year":"2009","journal-title":"JDCTA"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1459352.1459355","article-title":"Word sense disambiguation: A survey","volume":"41","author":"Navigli","year":"2009","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Husain, M.S., and Khanum, M.A. (2016, January 4\u20135). Word Sense Disambiguation in Software Requirement Specifications Using WordNet and Association Mining Rule. Proceedings of the Second International Conference on Information and Communication Technology for Competitive Strategies, Udaipur, India.","DOI":"10.1145\/2905055.2905179"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Orkphol, K., and Yang, W. (2019). Word sense disambiguation using cosine similarity collaborates with Word2vec and WordNet. Future Internet, 11.","DOI":"10.3390\/fi11050114"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Fraga, A., Moreno, V., Parra, E., and Garcia, J. (2017, January 5\u20137). Extraction of Patterns Using NLP: Genetic Deafness. Proceedings of the SEKE, Pittsburgh, PA, USA.","DOI":"10.18293\/SEKE2017-204"},{"key":"ref_48","unstructured":"Liddy, E.D. (2023, June 28). Natural Language Processing. Available online: https:\/\/surface.syr.edu\/cgi\/viewcontent.cgi?article=1043&context=istpub."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Loper, E., and Bird, S. (2002, January 7). NLTK: The natural language toolkit. Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, Philadelphia, PA, USA.","DOI":"10.3115\/1118108.1118117"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., and McClosky, D. (2014, January 22\u201327). The Stanford CoreNLP natural language processing toolkit. Proceedings of the 52nd annual meeting of the association for computational linguistics: System demonstrations, Baltimore, MD, USA.","DOI":"10.3115\/v1\/P14-5010"},{"key":"ref_51","unstructured":"Salloum, S.A., Al-Emran, M., Monem, A.A., and Shaalan, K. (2018). Intelligent Natural Language Processing: Trends and Applications, Springer."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1016\/j.inffus.2016.10.004","article-title":"A review of natural language processing techniques for opinion mining systems","volume":"36","author":"Sun","year":"2017","journal-title":"Inf. Fusion."},{"key":"ref_53","unstructured":"Bird, S., Klein, E., and Loper, E. (2009). Natural language processing with Python: Analyzing text with the natural language toolkit, O\u2019Reilly Media Inc."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Nasukawa, T., and Yi, J. (2003, January 23\u201325). Sentiment analysis: Capturing favorability using natural language processing. Proceedings of the 2nd International Conference on Knowledge Capture, Sanibel Island, FL, USA.","DOI":"10.1145\/945645.945658"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Harris, L.R. (1978, January 4\u20136). The ROBOT System: Natural language processing applied to data base query. Proceedings of the 1978 Annual Conference, Washington, DC, USA.","DOI":"10.1145\/800127.804087"},{"key":"ref_56","unstructured":"Lees, B. (1970). WIT Transactions on Information and Communication Technologies, Available online: https:\/\/www.witpress.com\/elibrary\/wit-transactions-on-information-and-communication-technologies\/12\/10537."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1007\/s10707-019-00345-2","article-title":"Using word embeddings to generate data-driven human agent decision-making from natural language","volume":"23","author":"Runck","year":"2019","journal-title":"GeoInformatica"},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Padilla, J.J., Shuttleworth, D., and O\u2019Brien, K. (2019, January 8\u201311). Agent-Based Model Characterization Using Natural Language Processing. Proceedings of the 2019 Winter Simulation Conference (WSC), National Harbor, MD, USA.","DOI":"10.1109\/WSC40007.2019.9004895"},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1111\/gean.12267","article-title":"Future Developments in Geographical Agent-Based Models: Challenges and Opportunities","volume":"53","author":"Heppenstall","year":"2021","journal-title":"Geogr. Anal."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"108982","DOI":"10.1016\/j.knosys.2022.108982","article-title":"Mapping the knowledge frontiers and evolution of decision making based on agent-based modeling","volume":"250","author":"Liang","year":"2022","journal-title":"Knowl.-Based Syst."},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Harmain, H.M., and Gaizauskas, R. (2000, January 11\u201315). CM-Builder: An automated NL-based CASE tool. Proceedings of the ASE 2000 Fifteenth IEEE International Conference on Automated Software Engineering, Grenoble, France.","DOI":"10.1109\/ASE.2000.873649"},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"9","DOI":"10.18564\/jasss.1897","article-title":"UML for ABM","volume":"15","author":"Bersini","year":"2012","journal-title":"J. Artif. Soc. Soc. Simul."},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"12","DOI":"10.18564\/jasss.2838","article-title":"A Call to Arms: Standards for Agent-Based Modeling and Simulation","volume":"18","author":"Collins","year":"2015","journal-title":"J. Artif. Soc. Soc. Simul."},{"key":"ref_64","doi-asserted-by":"crossref","unstructured":"Bakam, I., Kordon, F., Le Page, C., and Bousquet, F. (2000, January 5\u20137). Formalization of a spatialized multiagent model using coloured petri nets for the study of an hunting management system. Proceedings of the International Workshop on Formal Approaches to Agent-Based Systems, Greenbelt, MD, USA.","DOI":"10.1007\/3-540-45484-5_10"},{"key":"ref_65","first-page":"1","article-title":"Agent-based social simulation: Dealing with complexity","volume":"9","author":"Gilbert","year":"2004","journal-title":"Complex. Syst. Netw. Excell."},{"key":"ref_66","doi-asserted-by":"crossref","unstructured":"Miller, J.H., and Page, S.E. (2009). Complex Adaptive Systems: An Introduction to Computational Models of Social Life, Princeton University Press.","DOI":"10.1515\/9781400835522"},{"key":"ref_67","first-page":"100","article-title":"Introduction to information retrieval","volume":"16","author":"Manning","year":"2010","journal-title":"Nat. Lang. Eng."},{"key":"ref_68","first-page":"137","article-title":"Data reduction techniques for large qualitative data sets","volume":"2","author":"Namey","year":"2008","journal-title":"Handb. Team-Based Qual. Res."},{"key":"ref_69","unstructured":"Ramos, J. (2003, January 3\u20138). Using tf-idf to determine word relevance in document queries. Proceedings of the First Instructional Conference on Machine Learning, Piscataway, NJ, USA."},{"key":"ref_70","first-page":"33","article-title":"Mental models of food security in rural Mali","volume":"38","author":"Sanga","year":"2017","journal-title":"Environ. Syst. Decis."},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"1829","DOI":"10.1080\/13658816.2010.490533","article-title":"Applying time-dependent variance-based global sensitivity analysis to represent the dynamics of an agent-based model of land use change","volume":"24","author":"Sun","year":"2010","journal-title":"Int. J. Geogr. Inf. Sci."},{"key":"ref_72","unstructured":"Xiang, X., Kennedy, R., Madey, G., and Cabaniss, S. (2005, January 3). Verification and validation of agent-based scientific simulation models. Proceedings of the Agent-Directed Simulation Conference, San Diego, CA, USA."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/16\/7\/338\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:11:55Z","timestamp":1760127115000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/16\/7\/338"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,14]]},"references-count":72,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2023,7]]}},"alternative-id":["a16070338"],"URL":"https:\/\/doi.org\/10.3390\/a16070338","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,14]]}}}