{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T01:55:38Z","timestamp":1787018138018,"version":"build-2736575974"},"reference-count":76,"publisher":"MIT Press","issue":"3","license":[{"start":{"date-parts":[[2024,5,2]],"date-time":"2024-05-02T00:00:00Z","timestamp":1714608000000},"content-version":"vor","delay-in-days":122,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Automated coherence metrics constitute an efficient and popular way to evaluate topic models. Previous work presents a mixed picture of their presumed correlation with human judgment. This work proposes a novel sampling approach to mining topic representations at a large scale while seeking to mitigate bias from sampling, enabling the investigation of widely used automated coherence metrics via large corpora. Additionally, this article proposes a novel user study design, an amalgamation of different proxy tasks, to derive a finer insight into the human decision-making processes. This design subsumes the purpose of simple rating and outlier-detection user studies. Similar to the sampling approach, the user study conducted is extensive, comprising 40 study participants split into eight different study groups tasked with evaluating their respective set of 100 topic representations. Usually, when substantiating the use of these metrics, human responses are treated as the gold standard. This article further investigates the reliability of human judgment by flipping the comparison and conducting a novel extended analysis of human response at the group and individual level against a generic corpus. The investigation results show a moderate to good correlation between these metrics and human judgment, especially for generic corpora, and derive further insights into the human perception of coherence. Analyzing inter-metric correlations across corpora shows moderate to good correlation among these metrics. As these metrics depend on corpus statistics, this article further investigates the topical differences between corpora, revealing nuances in applications of these metrics.<\/jats:p>","DOI":"10.1162\/coli_a_00518","type":"journal-article","created":{"date-parts":[[2024,5,2]],"date-time":"2024-05-02T14:36:01Z","timestamp":1714660561000},"page":"893-952","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":9,"title":["Aligning Human and Computational Coherence Evaluations"],"prefix":"10.1162","volume":"50","author":[{"given":"Jia Peng","family":"Lim","sequence":"first","affiliation":[{"name":"Singapore Management University, School of Computer and Information Systems, PreferredAI Research Group. jiapeng.lim.2021@smu.edu.sg"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hady W.","family":"Lauw","sequence":"additional","affiliation":[{"name":"Singapore Management University, School of Computer and Information Systems, PreferredAI Research Group. hadywlauw@smu.edu.sg"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2024,9,1]]},"reference":[{"key":"2024092014251242800_bib1","first-page":"13","article-title":"Evaluating topic coherence using distributional semantics","volume-title":"Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) \u2013 Long Papers","author":"Aletras","year":"2013"},{"key":"2024092014251242800_bib2","doi-asserted-by":"publisher","first-page":"13448","DOI":"10.18653\/v1\/2023.acl-long.752","article-title":"Topic-guided sampling for data-efficient multi-domain stance detection","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Arakelyan","year":"2023"},{"key":"2024092014251242800_bib3","article-title":"Wikiextractor","author":"Attardi","year":"2015"},{"issue":"5","key":"2024092014251242800_bib4","doi-asserted-by":"publisher","first-page":"1069","DOI":"10.1145\/502102.502107","article-title":"A unified approach to approximating resource allocation and scheduling","volume":"48","author":"Bar-Noy","year":"2001","journal-title":"Journal of the ACM"},{"issue":"1","key":"2024092014251242800_bib5","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1177\/0265532209340194","article-title":"A Rasch-based validation of the vocabulary size test","volume":"27","author":"Beglar","year":"2010","journal-title":"Language Testing"},{"issue":"8","key":"2024092014251242800_bib6","doi-asserted-by":"publisher","first-page":"1288","DOI":"10.1073\/pnas.45.8.1288","article-title":"A mathematical theory of adaptive control processes","volume":"45","author":"Bellman","year":"1959","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"2","key":"2024092014251242800_bib7","doi-asserted-by":"publisher","first-page":"837","DOI":"10.1177\/0049124118799372","article-title":"Interrater reliability in systematic review methodology: Exploring variation in coder decision-making","volume":"50","author":"Belur","year":"2021","journal-title":"Sociological Methods & Research"},{"key":"2024092014251242800_bib8","doi-asserted-by":"publisher","first-page":"1676","DOI":"10.18653\/v1\/2021.eacl-main.143","article-title":"Cross-lingual contextualized topic models with zero-shot learning","volume-title":"Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume","author":"Bianchi","year":"2021"},{"key":"2024092014251242800_bib9","first-page":"993","article-title":"Latent dirichlet allocation","volume":"3","author":"Blei","year":"2003","journal-title":"Journal of Machine Learning Research"},{"issue":"1","key":"2024092014251242800_bib10","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1177\/02655322211064629","article-title":"Revisiting English language proficiency and its impact on the academic performance of domestic university students in Singapore","volume":"40","author":"Bo","year":"2023","journal-title":"Language Testing"},{"key":"2024092014251242800_bib11","first-page":"2787","article-title":"Translating embeddings for modeling multi-relational data","volume-title":"Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2","author":"Bordes","year":"2013"},{"key":"2024092014251242800_bib12","first-page":"31","article-title":"Normalized (pointwise) mutual information in collocation extraction","author":"Bouma","year":"2009","journal-title":"Proceedings of the Biennial GSCL Conference 2009"},{"key":"2024092014251242800_bib13","article-title":"Towards monosemanticity: Decomposing language models with dictionary learning","author":"Bricken","year":"2023","journal-title":"Transformer Circuits Thread"},{"issue":"131","key":"2024092014251242800_bib14","first-page":"1","article-title":"Decoupling sparsity and smoothness in the dirichlet variational autoencoder topic model","volume":"20","author":"Burkhardt","year":"2019","journal-title":"Journal of Machine Learning Research"},{"key":"2024092014251242800_bib15","first-page":"288","article-title":"Reading tea leaves: How humans interpret topic models","volume-title":"Proceedings of the 22nd International Conference on Neural Information Processing Systems","author":"Chang","year":"2009"},{"key":"2024092014251242800_bib16","doi-asserted-by":"publisher","first-page":"210","DOI":"10.1137\/0214017","article-title":"Arboricity and subgraph listing algorithms","volume":"14","author":"Chiba","year":"1985","journal-title":"SIAM Journal on Computing"},{"issue":"1","key":"2024092014251242800_bib17","first-page":"22","article-title":"Word association norms, mutual information, and lexicography","volume":"16","author":"Church","year":"1990","journal-title":"Computational Linguistics"},{"issue":"6","key":"2024092014251242800_bib18","doi-asserted-by":"publisher","first-page":"791","DOI":"10.1287\/opre.6.6.791","article-title":"A method for solving traveling-salesman problems","volume":"6","author":"Croes","year":"1958","journal-title":"Operations Research"},{"key":"2024092014251242800_bib19","article-title":"Sparse autoencoders find highly interpretable features in language models","author":"Cunningham","year":"2023"},{"key":"2024092014251242800_bib20","doi-asserted-by":"publisher","first-page":"589","DOI":"10.1145\/3178876.3186125","article-title":"Listing k-cliques in sparse real-world graphs*","volume-title":"Proceedings of the 2018 World Wide Web Conference","author":"Danisch","year":"2018"},{"key":"2024092014251242800_bib21","doi-asserted-by":"publisher","first-page":"439","DOI":"10.1162\/tacl_a_00325","article-title":"Topic modeling in embedding spaces","volume":"8","author":"Dieng","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2024092014251242800_bib22","doi-asserted-by":"publisher","first-page":"3824","DOI":"10.18653\/v1\/2021.naacl-main.300","article-title":"Topic model or topic twaddle? Re-evaluating semantic interpretability measures","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Doogan","year":"2021"},{"key":"2024092014251242800_bib23","article-title":"A mathematical framework for transformer circuits","author":"Elhage","year":"2021","journal-title":"Transformer Circuits Thread"},{"issue":"3","key":"2024092014251242800_bib24","doi-asserted-by":"publisher","first-page":"194","DOI":"10.1093\/analys\/63.3.194","article-title":"A probabilistic theory of coherence","volume":"63","author":"Fitelson","year":"2003","journal-title":"Analysis"},{"issue":"2","key":"2024092014251242800_bib25","doi-asserted-by":"publisher","first-page":"373","DOI":"10.1162\/coli_a_00475","article-title":"Reflection of demographic background on word usage","volume":"49","author":"Garimella","year":"2023","journal-title":"Computational Linguistics"},{"key":"2024092014251242800_bib26","doi-asserted-by":"publisher","first-page":"30","DOI":"10.18653\/v1\/2022.emnlp-main.3","article-title":"Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing","author":"Geva","year":"2022"},{"key":"2024092014251242800_bib27","doi-asserted-by":"publisher","first-page":"5228","DOI":"10.1073\/pnas.0307752101","article-title":"Finding scientific topics","volume":"101 Suppl 1","author":"Griffiths","year":"2004","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2024092014251242800_bib28","first-page":"856","article-title":"Online learning for latent dirichlet allocation","volume-title":"Proceedings of the 23rd International Conference on Neural Information Processing Systems - Volume 1","author":"Hoffman","year":"2010"},{"key":"2024092014251242800_bib29","first-page":"Art. 155","article-title":"Is automated topic model evaluation broken? The incoherence of coherence","volume-title":"Neural Information Processing Systems","author":"Hoyle","year":"2021"},{"key":"2024092014251242800_bib30","doi-asserted-by":"publisher","first-page":"5321","DOI":"10.18653\/v1\/2022.findings-emnlp.390","article-title":"Are neural topic models broken?","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2022","author":"Hoyle","year":"2022"},{"key":"2024092014251242800_bib31","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1111\/j.1469-8137.1912.tb05611.x","article-title":"The distribution of the flora in the alpine zone 1","volume":"11","author":"Jaccard","year":"1912","journal-title":"New Phytologist"},{"issue":"2","key":"2024092014251242800_bib32","doi-asserted-by":"publisher","first-page":"494","DOI":"10.1109\/TNNLS.2021.3070843","article-title":"A survey on knowledge graphs: Representation, acquisition, and applications","volume":"33","author":"Ji","year":"2022","journal-title":"IEEE Transactions on Neural Networks and Learning Systems"},{"key":"2024092014251242800_bib33","article-title":"Auto-encoding variational Bayes","volume-title":"2nd International Conference on Learning Representations, ICLR 2014","author":"Kingma","year":"2014"},{"key":"2024092014251242800_bib34","article-title":"Computing krippendorff\u2019s alpha-reliability","author":"Krippendorff","year":"2011"},{"key":"2024092014251242800_bib35","doi-asserted-by":"publisher","first-page":"355","DOI":"10.18653\/v1\/P17-1033","article-title":"Topically driven neural language model","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Lau","year":"2017"},{"key":"2024092014251242800_bib36","doi-asserted-by":"publisher","first-page":"530","DOI":"10.3115\/v1\/E14-1056","article-title":"Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality","volume-title":"Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics","author":"Lau","year":"2014"},{"key":"2024092014251242800_bib37","doi-asserted-by":"publisher","first-page":"8646","DOI":"10.18653\/v1\/2023.emnlp-main.534","article-title":"Disentangling transformer language models as superposed topic models","volume-title":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing","author":"Lim","year":"2023"},{"key":"2024092014251242800_bib38","doi-asserted-by":"publisher","first-page":"13874","DOI":"10.18653\/v1\/2023.acl-long.776","article-title":"Large-scale correlation analysis of automated metrics for topic models","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Lim","year":"2023"},{"issue":"1","key":"2024092014251242800_bib39","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1214\/aoms\/1177730491","article-title":"On a test of whether one of two random variables is stochastically larger than the other","volume":"18","author":"Mann","year":"1947","journal-title":"Annals of Mathematical Statistics"},{"key":"2024092014251242800_bib40","doi-asserted-by":"publisher","first-page":"3143","DOI":"10.1145\/3485447.3512034","article-title":"Topic discovery via latent space clustering of pretrained language model representations","volume-title":"Proceedings of the ACM Web Conference 2022","author":"Meng","year":"2022"},{"key":"2024092014251242800_bib41","doi-asserted-by":"publisher","first-page":"1908","DOI":"10.1145\/3394486.3403242","article-title":"Hierarchical topic mining via joint spherical tree and text embedding","volume-title":"Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","author":"Meng","year":"2020"},{"key":"2024092014251242800_bib42","first-page":"1727","article-title":"Neural variational inference for text processing","volume-title":"Proceedings of The 33rd International Conference on Machine Learning","author":"Miao","year":"2016"},{"issue":"11","key":"2024092014251242800_bib43","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1145\/219717.219748","article-title":"Wordnet: A lexical database for English","volume":"38","author":"Miller","year":"1995","journal-title":"Communications of ACM"},{"key":"2024092014251242800_bib44","first-page":"262","article-title":"Optimizing semantic coherence in topic models","volume-title":"Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing","author":"Mimno","year":"2011"},{"key":"2024092014251242800_bib45","first-page":"262","article-title":"Optimizing semantic coherence in topic models","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","author":"Mimno","year":"2011"},{"key":"2024092014251242800_bib46","doi-asserted-by":"publisher","first-page":"31","DOI":"10.18653\/v1\/S16-1003","article-title":"SemEval-2016 Task 6: Detecting stance in tweets","volume-title":"Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)","author":"Mohammad","year":"2016"},{"issue":"1","key":"2024092014251242800_bib47","doi-asserted-by":"publisher","first-page":"59","DOI":"10.3138\/cmlr.63.1.59","article-title":"How large a vocabulary is needed for reading and listening?","volume":"63","author":"Nation","year":"2006","journal-title":"Canadian Modern Language Review"},{"key":"2024092014251242800_bib48","doi-asserted-by":"publisher","DOI":"10.23915\/distill.00024.001","article-title":"Zoom in: An introduction to circuits","author":"Olah","year":"2020","journal-title":"Distill"},{"key":"2024092014251242800_bib49","article-title":"Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation","volume-title":"Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC\u201906)","author":"Passonneau","year":"2006"},{"key":"2024092014251242800_bib50","doi-asserted-by":"publisher","first-page":"701","DOI":"10.1145\/2623330.2623732","article-title":"DeepWalk: Online learning of social representations","volume-title":"Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Perozzi","year":"2014"},{"key":"2024092014251242800_bib51","doi-asserted-by":"publisher","first-page":"399","DOI":"10.1145\/2684822.2685324","article-title":"Exploring the space of topic coherence measures","volume-title":"WSDM","author":"R\u00f6der","year":"2015"},{"key":"2024092014251242800_bib52","article-title":"Evaluating topic coherence measures","author":"Rosner","year":"2014","journal-title":"ArXiv preprint arXiv 1403.6397"},{"key":"2024092014251242800_bib53","doi-asserted-by":"publisher","first-page":"287","DOI":"10.1162\/tacl_a_00099","article-title":"Comparing apples to apple: The effects of stemmers on topic models","volume":"4","author":"Schofield","year":"2016","journal-title":"Transactions of the Association for Computational Linguistics"},{"issue":"3\u20134","key":"2024092014251242800_bib54","doi-asserted-by":"publisher","first-page":"591","DOI":"10.1093\/biomet\/52.3-4.591","article-title":"An analysis of variance test for normality (complete samples)","volume":"52","author":"Shapiro","year":"1965","journal-title":"Biometrika"},{"key":"2024092014251242800_bib55","first-page":"14681","article-title":"Topic modeling revisited: A document graph-based neural network perspective","volume-title":"Advances in Neural Information Processing Systems 34 \u2013 35th Conference on Neural Information Processing Systems, NeurIPS 2021","author":"Shen","year":"2021"},{"key":"2024092014251242800_bib56","article-title":"Autoencoding variational inference for topic models","volume-title":"ICLR (Poster)","author":"Srivastava","year":"2017"},{"key":"2024092014251242800_bib57","doi-asserted-by":"publisher","first-page":"9348","DOI":"10.18653\/v1\/2023.emnlp-main.581","article-title":"Revisiting automated topic model evaluation with large language models","volume-title":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing","author":"Stammbach","year":"2023"},{"key":"2024092014251242800_bib58","first-page":"39","article-title":"Self-perception of English ability: Is it related to proficiency and\/or class performance?","volume-title":"\u65b0\u6f5f\u5927\u5b66\u8a00\u8a9e\u6587\u5316\u7814\u7a76","author":"Takahashi","year":"2009"},{"key":"2024092014251242800_bib59","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1162\/coli_a_00506","article-title":"Topics in the haystack: Enhancing topic quality through corpus expansion","author":"Thielmann","year":"2024","journal-title":"Computational Linguistics"},{"issue":"12","key":"2024092014251242800_bib60","doi-asserted-by":"publisher","first-page":"2724","DOI":"10.1109\/TKDE.2017.2754499","article-title":"Knowledge graph embedding: A survey of approaches and applications","volume":"29","author":"Wang","year":"2017","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"2024092014251242800_bib61","doi-asserted-by":"publisher","first-page":"166","DOI":"10.18653\/v1\/N19-1015","article-title":"Topic-guided variational auto-encoder for text generation","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Wang","year":"2019"},{"key":"2024092014251242800_bib62","doi-asserted-by":"publisher","first-page":"485","DOI":"10.18653\/v1\/2020.emnlp-main.35","article-title":"Friendly topic assistant for transformer based abstractive summarization","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Wang","year":"2020"},{"issue":"4","key":"2024092014251242800_bib63","doi-asserted-by":"publisher","first-page":"817","DOI":"10.2307\/1912934","article-title":"A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity","volume":"48","author":"White","year":"1980","journal-title":"Econometrica"},{"issue":"6","key":"2024092014251242800_bib64","doi-asserted-by":"publisher","first-page":"80","DOI":"10.2307\/3001968","article-title":"Individual comparisons by ranking methods","volume":"1","author":"Wilcoxon","year":"1945","journal-title":"Biometrics Bulletin"},{"key":"2024092014251242800_bib65","doi-asserted-by":"publisher","first-page":"14734","DOI":"10.18653\/v1\/2023.acl-long.823","article-title":"Information screening whilst exploiting! Multimodal relation extraction with feature denoising and multimodal topic modeling","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Wu","year":"2023"},{"key":"2024092014251242800_bib66","doi-asserted-by":"publisher","first-page":"6005","DOI":"10.1609\/aaai.v32i1.12033","article-title":"Diagnosing and improving topic models by analyzing posterior variability","volume-title":"Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence","author":"Xing","year":"2018"},{"key":"2024092014251242800_bib67","doi-asserted-by":"publisher","first-page":"11387","DOI":"10.18653\/v1\/2023.findings-acl.722","article-title":"Topic-guided self-introduction generation for social media users","volume-title":"Findings of the Association for Computational Linguistics: ACL 2023","author":"Xu","year":"2023"},{"issue":"5","key":"2024092014251242800_bib68","doi-asserted-by":"publisher","first-page":"557","DOI":"10.1007\/BF00992628","article-title":"The impact of English-language proficiency on international graduate students\u2019 perceived academic difficulty","volume":"32","author":"Xu","year":"1991","journal-title":"Research in Higher Education"},{"key":"2024092014251242800_bib69","doi-asserted-by":"publisher","first-page":"144","DOI":"10.1145\/3366423.3380102","article-title":"Graph attention topic modeling network","volume-title":"Proceedings of the Web Conference 2020","author":"Yang","year":"2020"},{"key":"2024092014251242800_bib70","doi-asserted-by":"publisher","first-page":"1955","DOI":"10.1109\/ICDE53745.2022.00192","article-title":"Efficient k-clique listing with set intersection speedup","volume-title":"2022 IEEE 38th International Conference on Data Engineering (ICDE)","author":"Yuan","year":"2022"},{"key":"2024092014251242800_bib71","doi-asserted-by":"publisher","first-page":"6737","DOI":"10.1609\/aaai.v34i04.6152","article-title":"Topic modeling on document networks with adjacent-encoder","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Zhang","year":"2020"},{"key":"2024092014251242800_bib72","doi-asserted-by":"publisher","first-page":"2429","DOI":"10.1145\/3534678.3539310","article-title":"Variational graph author topic modeling","volume-title":"Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","author":"Zhang","year":"2022"},{"key":"2024092014251242800_bib73","doi-asserted-by":"publisher","first-page":"279","DOI":"10.18653\/v1\/2022.naacl-main.21","article-title":"Seed-guided topic discovery with out-of-vocabulary seeds","volume-title":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Zhang","year":"2022"},{"key":"2024092014251242800_bib74","doi-asserted-by":"publisher","first-page":"635","DOI":"10.1109\/ICDM.2017.73","article-title":"Metalda: A topic model that efficiently incorporates meta information","volume-title":"2017 IEEE International Conference on Data Mining (ICDM)","author":"Zhao","year":"2017"},{"key":"2024092014251242800_bib75","first-page":"37","article-title":"Online nonnegative matrix factorization with general divergences","volume-title":"Proceedings of the 20th International Conference on Artificial Intelligence and Statistics","author":"Zhao","year":"2017"},{"key":"2024092014251242800_bib76","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/ICME51207.2021.9428274","article-title":"MNRE: A challenge multimodal dataset for neural relation extraction with visual evidence in social media posts","volume-title":"2021 IEEE International Conference on Multimedia and Expo (ICME)","author":"Zheng","year":"2021"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/50\/3\/893\/2471052\/coli_a_00518.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/50\/3\/893\/2471052\/coli_a_00518.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,20]],"date-time":"2024-09-20T10:25:38Z","timestamp":1726827938000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/50\/3\/893\/120894\/Aligning-Human-and-Computational-Coherence"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":76,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,9,1]]},"published-print":{"date-parts":[[2024,9,1]]}},"URL":"https:\/\/doi.org\/10.1162\/coli_a_00518","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}