{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:17:30Z","timestamp":1750220250116,"version":"3.41.0"},"reference-count":35,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2022,11,25]],"date-time":"2022-11-25T00:00:00Z","timestamp":1669334400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Asian Low-Resour. Lang. Inf. Process."],"published-print":{"date-parts":[[2023,1,31]]},"abstract":"<jats:p>We propose an unsupervised word segmentation model, in which for each unlabelled sentence sample, the learning objective is to maximize the generation probability of the sentence given its all possible segmentations. Such a generation probability can be factorized into the likelihood of each possible segment given the context in a recursive way. To capture both the long- and short-term dependencies, we propose to use a bi-directional neural language model to better extract the features of the segment\u2019s context. Two decoding algorithms were also developed to combine the context features from both directions to generate the final segmentation at the inference time, which helps to reconcile word-boundary ambiguities. Experimental results show that our context-sensitive unsupervised segmentation model achieved state-of-the-art at different evaluation settings on various datasets for Chinese, and the comparable result for Thai.<\/jats:p>","DOI":"10.1145\/3529387","type":"journal-article","created":{"date-parts":[[2022,4,29]],"date-time":"2022-04-29T11:41:22Z","timestamp":1651232482000},"page":"1-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Unsupervised Word Segmentation with Bi-directional Neural Language Model"],"prefix":"10.1145","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4960-015X","authenticated-orcid":false,"given":"Lihao","family":"Wang","sequence":"first","affiliation":[{"name":"Fudan University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4430-5036","authenticated-orcid":false,"given":"Xiaoqing","family":"Zheng","sequence":"additional","affiliation":[{"name":"Fudan University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,11,25]]},"reference":[{"key":"e_1_3_1_2_2","first-page":"1137","article-title":"A neural probabilistic language model","volume":"3","author":"Bengio Yoshua","year":"2003","unstructured":"Yoshua Bengio, R\u00e9jean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. J. Mach. Learn. Res. 3, Feb. (2003), 1137\u20131155.","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1092"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1110"},{"key":"e_1_3_1_5_2","article-title":"Learning phrase representations using RNN encoder-decoder for statistical machine translation","author":"Cho Kyunghyun","year":"2014","unstructured":"Kyunghyun Cho, Bart Van Merri\u00ebnboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014).","journal-title":"arXiv preprint arXiv:1406.1078"},{"key":"e_1_3_1_6_2","volume-title":"Proceedings of the 4th SIGHAN Workshop on Chinese Language Processing","author":"Emerson Thomas","year":"2005","unstructured":"Thomas Emerson. 2005. The Second International Chinese Word Segmentation Bakeoff. In Proceedings of the 4th SIGHAN Workshop on Chinese Language Processing."},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1162\/089120104773633394"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.3115\/1220175.1220260"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cognition.2009.03.008"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33016457"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-13-1927-3_14"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"issue":"3","key":"e_1_3_1_13_2","first-page":"8","article-title":"Chinese word segmentation: A decade review","volume":"21","author":"Huang Changning","year":"2007","unstructured":"Changning Huang and Hai Zhao. 2007. Chinese word segmentation: A decade review. J. Chinese Inf. Process. 21, 3 (2007), 8\u201320.","journal-title":"J. Chinese Inf. Process."},{"key":"e_1_3_1_14_2","volume-title":"Proceedings of the 6th SIGHAN Workshop on Chinese Language Processing","author":"Jin Guangjin","year":"2008","unstructured":"Guangjin Jin and Xiao Chen. 2008. The Fourth International Chinese Language Processing Bakeoff: Chinese word segmentation, named entity recognition and Chinese PoS tagging. In Proceedings of the 6th SIGHAN Workshop on Chinese Language Processing."},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.5555\/1273073.1273129"},{"key":"e_1_3_1_16_2","article-title":"Exploring the limits of language modeling","author":"Jozefowicz Rafal","year":"2016","unstructured":"Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410 (2016).","journal-title":"arXiv preprint arXiv:1602.02410"},{"key":"e_1_3_1_17_2","article-title":"Unsupervised learning of word boundary with description length gain","author":"Kitt Chunyu","year":"1999","unstructured":"Chunyu Kitt and Yorick Wilks. 1999. Unsupervised learning of word boundary with description length gain. EACL 1999: CoNLL-99 Computational Natural Language Learning (1999). Retrieved from: https:\/\/aclanthology.org\/W99-0701.pdf.","journal-title":"EACL 1999: CoNLL-99 Computational Natural Language Learning"},{"key":"e_1_3_1_18_2","article-title":"Segmental recurrent neural networks","author":"Kong Lingpeng","year":"2015","unstructured":"Lingpeng Kong, Chris Dyer, and Noah A. Smith. 2015. Segmental recurrent neural networks. arXiv preprint arXiv:1511.06018 (2015).","journal-title":"arXiv preprint arXiv:1511.06018"},{"key":"e_1_3_1_19_2","volume-title":"Proceedings of 8th International Symposium on Natural Language Processing (SNLP\u201909)","author":"Kosawat Krit","year":"2009","unstructured":"Krit Kosawat. 2009. InterBEST 2009: Thai word segmentation workshop. In Proceedings of 8th International Symposium on Natural Language Processing (SNLP\u201909)."},{"issue":"2","key":"e_1_3_1_20_2","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1080\/10618600.1998.10474772","article-title":"Estimating mixture of Dirichlet process models","volume":"7","author":"MacEachern Steven N.","year":"1998","unstructured":"Steven N. MacEachern and Peter M\u00fcller. 1998. Estimating mixture of Dirichlet process models. J. Computat. Graphic. Statist. 7, 2 (1998), 223\u2013238.","journal-title":"J. Computat. Graphic. Statist."},{"key":"e_1_3_1_21_2","first-page":"383","volume-title":"Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics","author":"Magistry Pierre","year":"2012","unstructured":"Pierre Magistry and Beno\u00eet Sagot. 2012. Unsupervized word segmentation: The case for Mandarin Chinese. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 383\u2013387."},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2010-343"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.5555\/1687878.1687894"},{"key":"e_1_3_1_24_2","first-page":"855","article-title":"The two-parameter Poisson-Dirichlet distribution derived from a stable subordinator","author":"Pitman Jim","year":"1997","unstructured":"Jim Pitman and Marc Yor. 1997. The two-parameter Poisson-Dirichlet distribution derived from a stable subordinator. Ann. Probabil. Apr 1 (1997), 855\u2013900.","journal-title":"Ann. Probabil."},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/5.18626"},{"issue":"4","key":"e_1_3_1_26_2","first-page":"336","article-title":"A statistical method for finding word boundaries in Chinese text","volume":"4","author":"Sproat Richard","year":"1990","unstructured":"Richard Sproat and Chilin Shih. 1990. A statistical method for finding word boundaries in Chinese text. Comput. Process. Chinese Orient. Lang. 4, 4 (1990), 336\u2013351.","journal-title":"Comput. Process. Chinese Orient. Lang."},{"key":"e_1_3_1_27_2","article-title":"Unsupervised neural word segmentation for Chinese via segmental language modeling","author":"Sun Zhiqing","year":"2018","unstructured":"Zhiqing Sun and Zhi-Hong Deng. 2018. Unsupervised neural word segmentation for Chinese via segmental language modeling. arXiv preprint arXiv:1810.03167 (2018).","journal-title":"arXiv preprint arXiv:1810.03167"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.3115\/1220175.1220299"},{"key":"e_1_3_1_29_2","first-page":"1385","volume-title":"Proceedings of the Conference on Advances in Neural Information Processing Systems","author":"Teh Yee W.","year":"2005","unstructured":"Yee W. Teh, Michael I. Jordan, Matthew J. Beal, and David M. Blei. 2005. Sharing clusters among related groups: Hierarchical Dirichlet processes. In Proceedings of the Conference on Advances in Neural Information Processing Systems. 1385\u20131392."},{"key":"e_1_3_1_30_2","first-page":"1","volume-title":"Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data","author":"Teng Zhiyang","year":"2014","unstructured":"Zhiyang Teng, Hao Xiong, and Qun Liu. 2014. Unsupervised joint monolingual character alignment and word segmentation. In Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data. Springer, 1\u201312."},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P15-1171"},{"key":"e_1_3_1_32_2","article-title":"Sequence modeling via segmentations","author":"Wang Chong","year":"2017","unstructured":"Chong Wang, Yining Wang, Po-Sen Huang, Abdelrahman Mohamed, Dengyong Zhou, and Li Deng. 2017. Sequence modeling via segmentations. arXiv preprint arXiv:1702.07463 (2017).","journal-title":"arXiv preprint arXiv:1702.07463"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1162\/COLI_a_00058"},{"key":"e_1_3_1_34_2","volume-title":"Computational Studies of the Most Frequent Chinese Words and Sounds","author":"Yee Suen Ching","year":"1986","unstructured":"Suen Ching Yee. 1986. Computational Studies of the Most Frequent Chinese Words and Sounds, Vol. 3. World Scientific."},{"key":"e_1_3_1_35_2","volume-title":"Proceedings of the 3rd International Joint Conference on Natural Language Processing","author":"Zhao Hai","year":"2008","unstructured":"Hai Zhao and Chunyu Kit. 2008. An empirical comparison of goodness measures for unsupervised Chinese word segmentation with a unified framework. In Proceedings of the 3rd International Joint Conference on Natural Language Processing."},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2018\/640"}],"container-title":["ACM Transactions on Asian and Low-Resource Language Information Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3529387","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3529387","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:30:39Z","timestamp":1750188639000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3529387"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,25]]},"references-count":35,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,1,31]]}},"alternative-id":["10.1145\/3529387"],"URL":"https:\/\/doi.org\/10.1145\/3529387","relation":{},"ISSN":["2375-4699","2375-4702"],"issn-type":[{"type":"print","value":"2375-4699"},{"type":"electronic","value":"2375-4702"}],"subject":[],"published":{"date-parts":[[2022,11,25]]},"assertion":[{"value":"2021-06-17","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-03-28","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-11-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}