{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T20:51:55Z","timestamp":1777495915418,"version":"3.51.4"},"reference-count":37,"publisher":"Wiley","issue":"1","license":[{"start":{"date-parts":[[2016,12,27]],"date-time":"2016-12-27T00:00:00Z","timestamp":1482796800000},"content-version":"vor","delay-in-days":361,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"content-domain":{"domain":["asistdl.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Proc. Assoc. Info. Sci. Tech."],"published-print":{"date-parts":[[2016,1]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:p>\n                    This paper reports on the user centered, formative evaluation of tools and the validation of models for the analysis of historical textbooks in the context of the digital history project\n                    <jats:italic>Children and their World<\/jats:italic>\n                    . The goal of the project is to create methods for computer\u2010supported, interactive analysis that can be applied to a large corpus of historical textbooks on history and geography (\u223c5000 volumes). A first version of a tool for text analysis has been created based on a user centered design process, including a contextual study on current work practices of historians, participative design workshops, and the prioritization of requirements with the project stakeholders. In addition, several generations of text models used in these tools have been iteratively evaluated by the historians in the project. In the context of a cooperative validation study, researchers have used these tools to examine existing hypotheses from the field. The method enables the validation of text models regarding established knowledge, provides additional insights into the requirements for tools and visualizations, and helps to strengthen the expert users' trust in the tools. Based on the findings, this paper proposes four principles for the analysis of a corpus of historical texts. Moreover, the methods presented are discussed regarding the application of user centered design in the context of digital humanities projects.\n                  <\/jats:p>","DOI":"10.1002\/pra2.2016.14505301078","type":"journal-article","created":{"date-parts":[[2016,12,27]],"date-time":"2016-12-27T06:08:13Z","timestamp":1482818893000},"page":"1-10","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Methods for user\u2010centered design and evaluation of text analysis tools in a digital history project"],"prefix":"10.1002","volume":"53","author":[{"given":"Ben","family":"Heuwing","sequence":"first","affiliation":[{"name":"Department of Information Science and Natural Language Processing University of Hildesheim Universit\u00e4tsplatz 1 31141 Hildesheim Germany"}]},{"given":"Thomas","family":"Mandl","sequence":"additional","affiliation":[{"name":"Department of Information Science and Natural Language Processing University of Hildesheim Universit\u00e4tsplatz 1 31141 Hildesheim Germany"}]},{"given":"Christa","family":"Womser\u2010Hacker","sequence":"additional","affiliation":[{"name":"Department of Information Science and Natural Language Processing University of Hildesheim Universit\u00e4tsplatz 1 31141 Hildesheim Germany"}]}],"member":"311","published-online":{"date-parts":[[2016,12,27]]},"reference":[{"key":"e_1_2_10_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-12823-8_22"},{"key":"e_1_2_10_3_1","doi-asserted-by":"publisher","DOI":"10.1177\/0957926508088962"},{"key":"e_1_2_10_4_1","doi-asserted-by":"publisher","DOI":"10.1162\/jmlr.2003.3.4-5.993"},{"key":"e_1_2_10_5_1","doi-asserted-by":"publisher","DOI":"10.1093\/jahist\/jau184"},{"key":"e_1_2_10_6_1","volume-title":"Proceedings of the EuroVis Workshop on Visual Analytics","author":"Boukhelifa N.","year":"2015"},{"key":"e_1_2_10_7_1","first-page":"288","article-title":"Reading Tea Leaves: How Humans Interpret Topic Models","volume":"22","author":"Chang J.","year":"2009","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_10_8_1","unstructured":"Chuang J. Gupta S. Manning C. &Heer J.(2013).Topic model diagnostics: Assessing domain relevance via topical alignment. InProceedings of the 30th International Conference on Machine Learning (ICML\u201013)(pp.612\u2013620)."},{"key":"e_1_2_10_9_1","doi-asserted-by":"crossref","unstructured":"Chuang J. Manning C. D. &Heer J.(2012).Termite: Visualization Techniques for Assessing Textual Topic Models. InProceedings of the International Working Conference on Advanced Visual Interfaces(pp.74\u201377). New York NY USA: ACM.http:\/\/doi.org\/10.1145\/2254556.2254572","DOI":"10.1145\/2254556.2254572"},{"key":"e_1_2_10_10_1","doi-asserted-by":"crossref","unstructured":"Chuang J. Ramage D. Manning C. &Heer J.(2012).Interpretation and Trust: Designing Model\u2010driven Visualizations for Text Analysis. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(pp.443\u2013452). New York NY USA: ACM.http:\/\/doi.org\/10.1145\/2207676.2207738","DOI":"10.1145\/2207676.2207738"},{"key":"e_1_2_10_11_1","doi-asserted-by":"publisher","DOI":"10.18352\/lq.8017"},{"key":"e_1_2_10_12_1","unstructured":"Cybulska A. &Vossen P.(2011).Historical event extraction from text. InProceedings of the 5th ACL\u2010HLT Workshop on Language Technology for Cultural Heritage Social Sciences and Humanities(pp.39\u201343). Association for Computational Linguistics."},{"key":"e_1_2_10_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.poetic.2013.08.004"},{"key":"e_1_2_10_14_1","unstructured":"Eisenstein J. Sun I. &Klein L.(2014).Exploratory Thematic Analysis for Historical Newspaper Archives. Presented at the digital humanities 2014. Retrieved fromhttp:\/\/dharchive.org\/paper\/DH2014\/Paper-921.xml"},{"key":"e_1_2_10_15_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0087908"},{"key":"e_1_2_10_16_1","doi-asserted-by":"publisher","DOI":"10.2979\/victorianstudies.54.1.69"},{"issue":"2","key":"e_1_2_10_17_1","article-title":"Building Better Digital Humanities Tools: Toward Broader Audiences and User\u2010Centered Designs","volume":"6","author":"Gibbs F. W.","year":"2012","journal-title":"Digital Humanities Quarterly"},{"key":"e_1_2_10_18_1","doi-asserted-by":"publisher","DOI":"10.1057\/9780230371934_16"},{"key":"e_1_2_10_19_1","unstructured":"Mimno D. Wallach H. M. Talley E. Leenders M. &McCallum A.(2011).Optimizing semantic coherence in topic models. InProceedings of the Conference on Empirical Methods in Natural Language Processing(pp.262\u2013272). Association for Computational Linguistics."},{"key":"e_1_2_10_20_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.20342"},{"key":"e_1_2_10_21_1","unstructured":"Newman D. Lau J. H. Grieser K. &Baldwin T.(2010).Automatic evaluation of topic coherence. InHuman Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics(pp.100\u2013108). Association for Computational Linguistics."},{"key":"e_1_2_10_22_1","unstructured":"Nuessli M.\u2010A. &Kaplan F.(2014).Encoding Metaknowledge for Historical Databases. Indigital humanities 2014. Lausanne. Retrieved fromhttp:\/\/dharchive.org\/paper\/DH2014\/Paper-366.xml"},{"key":"e_1_2_10_23_1","doi-asserted-by":"publisher","DOI":"10.1002\/aris.2008.1440420112"},{"key":"e_1_2_10_24_1","volume-title":"User story mapping","author":"Patton J.","year":"2014"},{"key":"e_1_2_10_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-02146-6"},{"key":"e_1_2_10_26_1","unstructured":"Pirolli P. &Card S.(2005).The Sensemaking Process and Leverage Points for Analyst Technology as Identified through Cognitive Task Analysis. InProceedings of International Conference on Intelligence Analysis(Vol. 5 pp.2\u20134). Mitre McLean VA."},{"key":"e_1_2_10_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2011.04.032"},{"issue":"4","key":"e_1_2_10_28_1","article-title":"Modelling Historians' Information\u2010Seeking Behaviour with an Interdisciplinary and Comparative Approach","volume":"17","author":"Rhee H. L.","year":"2012","journal-title":"Information Research"},{"key":"e_1_2_10_29_1","doi-asserted-by":"crossref","unstructured":"Rieh S. Y. Yang J. Y. Yakel E. &Markey K.(2010).Conceptualizing Institutional Repositories: Using Co\u2010discovery to Uncover Mental Models. InProceedings of the Third Symposium on Information Interaction in Context(pp.165\u2013174). New York NY USA: ACM.http:\/\/doi.org\/10.1145\/1840784.1840809","DOI":"10.1145\/1840784.1840809"},{"key":"e_1_2_10_30_1","unstructured":"Schmidt B. M.(2013).Words Alone: Dismantling Topic Models in the Humanities.Journal of Digital Humanities. Retrieved fromhttp:\/\/journalofdigitalhumanities.org\/2-1\/words-alone-by-benjamin-m-schmidt\/"},{"key":"e_1_2_10_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2809936.2809939"},{"key":"e_1_2_10_32_1","doi-asserted-by":"crossref","unstructured":"Sinclair S. Ruecker S. &Radzikowska M.(2013).Information Visualization for Humanities Scholars.Literary Studies in the Digital Age \u2010 An Evolving Anthology.http:\/\/doi.org\/10.1632\/lsda.2013.6","DOI":"10.1632\/lsda.2013.6"},{"issue":"1","key":"e_1_2_10_33_1","article-title":"New information infrastructures for textbook research at the Georg Eckert Institute","volume":"9","author":"Str\u00f6tgen R.","year":"2014","journal-title":"History of Education & Children's Literature"},{"key":"e_1_2_10_34_1","unstructured":"Thaller M.(2012).Controversies around the Digital Humanities: An Agenda.Historical Social Research\/Historische Sozialforschung 7\u201323."},{"key":"e_1_2_10_35_1","doi-asserted-by":"publisher","DOI":"10.1108\/00220410810844178"},{"key":"e_1_2_10_36_1","doi-asserted-by":"publisher","DOI":"10.29085\/9781856049054.002"},{"key":"e_1_2_10_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-02260-9_5"},{"key":"e_1_2_10_38_1","unstructured":"Yang T.\u2010I. Torget A. J. &Mihalcea R.(2011).Topic modeling on historical newspapers. InProceedings of the 5th ACL\u2010HLT Workshop on Language Technology for Cultural Heritage Social Sciences and Humanities(pp.96\u2013104). Association for Computational Linguistics."}],"container-title":["Proceedings of the Association for Information Science and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fpra2.2016.14505301078","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fpra2.2016.14505301078","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/asistdl.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/pra2.2016.14505301078","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,21]],"date-time":"2025-10-21T17:11:11Z","timestamp":1761066671000},"score":1,"resource":{"primary":{"URL":"https:\/\/asistdl.onlinelibrary.wiley.com\/doi\/10.1002\/pra2.2016.14505301078"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,1]]},"references-count":37,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2016,1]]}},"alternative-id":["10.1002\/pra2.2016.14505301078"],"URL":"https:\/\/doi.org\/10.1002\/pra2.2016.14505301078","archive":["Portico"],"relation":{},"ISSN":["2373-9231","2373-9231"],"issn-type":[{"value":"2373-9231","type":"print"},{"value":"2373-9231","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,1]]},"assertion":[{"value":"2016-12-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}