{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T04:50:08Z","timestamp":1784263808894,"version":"3.55.0"},"reference-count":62,"publisher":"Association for Computing Machinery (ACM)","issue":"9","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Asian Low-Resour. Lang. Inf. Process."],"published-print":{"date-parts":[[2025,9,30]]},"abstract":"<jats:p>Current computational approaches for analysing or generating code-mixed sentences do not explicitly model \u201cnaturalness\u201d or \u201cacceptability\u201d of code-mixed sentences, but rely on training corpora to reflect distribution of acceptable code-mixed sentences. Modelling human judgement for the acceptability of code-mixed text can help in distinguishing natural code-mixed text and enable quality-controlled generation of code-mixed text. To this end, we construct Cline\u2014a dataset containing human acceptability judgements for English-Hindi\u00a0(en-hi) code-mixed text. Cline is the largest of its kind with 16,642 sentences, consisting of samples sourced from two sources: synthetically generated code-mixed text and samples collected from online social media. Our analysis establishes that popular code-mixing metrics such as CMI, Number of Switch Points, Burstines, which are used to filter\/curate\/compare code-mixed corpora have low correlation with human acceptability judgements, underlining the necessity of our dataset. Experiments using Cline demonstrate that simple Multilayer Perceptron (MLP) models when trained solely using code-mixing metrics as features are outperformed by fine-tuned pre-trained Multilingual Large Language Models (MLLMs). Specifically, among Encoder models XLM-Roberta and Bernice outperform IndicBERT across different configurations. Among Encoder-Decoder models, mBART performs better than mT5, however, Encoder-Decoder models are not able to outperform Encoder-only models. Decoder-only models perform the best when compared with all other MLLMS, with Llama 3.2 - 3B models outperforming similarly sized Qwen, Phi models. Comparison with zero and fewshot capabilitites of ChatGPT show that MLLMs fine-tuned on larger data outperform ChatGPT, providing scope for improvement in code-mixed tasks. Zero-shot transfer from English\u2013Hindi to English-Telugu acceptability judgments using our model checkpoints proves superior to random baselines, enabling application to other code-mixed language pairs and providing further avenues of research. We publicly release our human-annotated dataset, trained checkpoints, code-mix corpus, and code for data generation and model training.<\/jats:p>","DOI":"10.1145\/3748312","type":"journal-article","created":{"date-parts":[[2025,7,14]],"date-time":"2025-07-14T11:43:31Z","timestamp":1752493411000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["From Human Judgements to Predictive Models: Unravelling Acceptability in Code-Mixed Sentences"],"prefix":"10.1145","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7542-6802","authenticated-orcid":false,"given":"Prashant","family":"Kodali","sequence":"first","affiliation":[{"name":"IIIT Hyderabad","place":["Gachibowli, India"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-4807-2402","authenticated-orcid":false,"given":"Anmol","family":"Goel","sequence":"additional","affiliation":[{"name":"IIIT Hyderabad","place":["Gachibowli, India"]},{"name":"TU Darmstadt","place":["Gachibowli, India"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-3366-0231","authenticated-orcid":false,"given":"Likhith","family":"Asapu","sequence":"additional","affiliation":[{"name":"IIIT Hyderabad","place":["Gachibowli, India"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5537-1664","authenticated-orcid":false,"given":"Vamshi Krishna","family":"Bonagiri","sequence":"additional","affiliation":[{"name":"IIIT Hyderabad","place":["Gachibowli, India"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-3479-6258","authenticated-orcid":false,"given":"Anirudh","family":"Govil","sequence":"additional","affiliation":[{"name":"IIIT Hyderabad","place":["Gachibowli, India"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7473-7839","authenticated-orcid":false,"given":"Monojit","family":"Choudhury","sequence":"additional","affiliation":[{"name":"MBZUAI","place":["Masdar City, United Arab Emirates"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5082-2078","authenticated-orcid":false,"given":"Ponnurangam","family":"Kumaraguru","sequence":"additional","affiliation":[{"name":"IIIT Hyderabad","place":["Gachibowli, India"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8705-6637","authenticated-orcid":false,"given":"Manish","family":"Shrivastava","sequence":"additional","affiliation":[{"name":"IIIT Hyderabad","place":["Gachibowli, India"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,9,10]]},"reference":[{"key":"e_1_3_4_2_2","volume-title":"The Cambr idge Handbook of Linguistic Code-Switching","year":"2009","unstructured":"2009. The Cambr idge Handbook of Linguistic Code-Switching. Cambridge University Press."},{"key":"e_1_3_4_3_2","unstructured":"Marah Abdin and Jyoti Aneja et al.2024. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. (2024). arxiv:cs.CL\/2404.14219https:\/\/arxiv.org\/abs\/2404.14219"},{"key":"e_1_3_4_4_2","first-page":"1803","volume-title":"Proceedings of the 12th Language Resources and Evaluation Conference","author":"Aguilar Gustavo","year":"2020","unstructured":"Gustavo Aguilar, Sudipta Kar, and Thamar Solorio. 2020. LinCE: A centralized benchmark for linguistic code-switching evaluation. In Proceedings of the 12th Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France, 1803\u20131813. Retrieved fromhttps:\/\/aclanthology.org\/2020.lrec-1.223"},{"key":"e_1_3_4_5_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.614"},{"key":"e_1_3_4_6_2","doi-asserted-by":"crossref","unstructured":"Peter Auer. 1984. Bilingual conversation. (1984).","DOI":"10.1075\/pb.v.8"},{"key":"e_1_3_4_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3392846"},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1090"},{"key":"e_1_3_4_9_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-5801"},{"key":"e_1_3_4_10_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.747"},{"key":"e_1_3_4_11_2","doi-asserted-by":"crossref","first-page":"6191","DOI":"10.18653\/v1\/2022.emnlp-main.415","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing","author":"DeLucia Alexandra","year":"2022","unstructured":"Alexandra DeLucia, Shijie Wu, Aaron Mueller, Carlos Aguirre, Philip Resnik, and Mark Dredze. 2022. Bernice: A multilingual pre-trained encoder for twitter. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. Abu Dhabi, United Arab Emirates, 6191\u20136205. Retrieved from https:\/\/aclanthology.org\/2022.emnlp-main.415"},{"key":"e_1_3_4_12_2","first-page":"359","volume-title":"Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Eisenstein Jacob","year":"2013","unstructured":"Jacob Eisenstein. 2013. What to do about bad language on the internet. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Lucy Vanderwende, Hal Daum\u00e9 III, and Katrin Kirchhoff (Eds.). Association for Computational Linguistics, Atlanta, Georgia, 359\u2013369. Retrieved from https:\/\/aclanthology.org\/N13-1037"},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4612-4380-9_6"},{"key":"e_1_3_4_14_2","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Fornaciari Tommaso","year":"2021","unstructured":"Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, Massimo Poesio, et\u00a0al. 2021. Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics."},{"key":"e_1_3_4_15_2","volume-title":"Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC\u201916)","author":"Gamb\u00e4ck Bj\u00f6rn","year":"2016","unstructured":"Bj\u00f6rn Gamb\u00e4ck and Amitava Das. 2016. Comparing the level of code-switching in corpora. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC\u201916). Portoro\u017e, Slovenia."},{"issue":"2","key":"e_1_3_4_16_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1177\/003368827700800201","article-title":"The sociolinguistic significance of conversational code-switching","volume":"8","author":"Gumperz John J.","year":"1977","unstructured":"John J. Gumperz. 1977. The sociolinguistic significance of conversational code-switching. RELC Journal 8, 2 (1977), 1\u201334.","journal-title":"RELC Journal"},{"key":"e_1_3_4_17_2","volume-title":"North American Chapter of the Association for Computational Linguistics","author":"Gupta Abhirut","year":"2021","unstructured":"Abhirut Gupta, Aditya Vavre, and Sunita Sarawagi. 2021. Training data augmentation for code-mixed translation. In North American Chapter of the Association for Computational Linguistics."},{"key":"e_1_3_4_18_2","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2017-1429"},{"key":"e_1_3_4_19_2","volume-title":"Coling 1982: Proceedings of the Ninth International Conference on Computational Linguistics","author":"Joshi Aravind K.","year":"1982","unstructured":"Aravind K. Joshi. 1982. Processing of sentences with intra-sentential code-switching. In Coling 1982: Proceedings of the Ninth International Conference on Computational Linguistics. Retrieved from https:\/\/aclanthology.org\/C82-1023"},{"key":"e_1_3_4_20_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.445"},{"key":"e_1_3_4_21_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.329"},{"key":"e_1_3_4_22_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.findings-acl.40"},{"key":"e_1_3_4_23_2","article-title":"A guideline of selecting and reporting intraclass correlation coefficients for reliability research.","volume":"15","author":"Koo Terry K.","year":"2016","unstructured":"Terry K. Koo and Mae Y. Li. 2016. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of chiropractic medicine 15 2 (2016), 155\u201363.","journal-title":"Journal of chiropractic medicine"},{"key":"e_1_3_4_24_2","article-title":"Measuring gradience in speakers\u2019 grammaticality judgements","volume":"36","author":"Lau Jey Han","year":"2014","unstructured":"Jey Han Lau, Alexander Clark, and Shalom Lappin. 2014. Measuring gradience in speakers\u2019 grammaticality judgements. Cognitive Science 36 (2014). Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:11620784","journal-title":"Cognitive Science"},{"key":"e_1_3_4_25_2","volume-title":"Proceedings of the Annual Meeting of the Cognitive Science Society","volume":"36","author":"Lau Jey Han","year":"2014","unstructured":"Jey Han Lau, Alexander Clark, and Shalom Lappin. 2014. Measuring gradience in speakers\u2019 grammaticality judgements. In Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 36."},{"key":"e_1_3_4_26_2","doi-asserted-by":"crossref","first-page":"1202","DOI":"10.1111\/cogs.12414","article-title":"Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge","volume":"41","author":"Lau Jey Han","year":"2017","unstructured":"Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science 41, 5 (2017), 1202\u20131241.","journal-title":"Cognitive Science"},{"key":"e_1_3_4_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF01067625"},{"key":"e_1_3_4_28_2","volume-title":"Interspeech","author":"Lee Grandee","year":"2019","unstructured":"Grandee Lee, Xianghu Yue, and Haizhou Li. 2019. Linguistically motivated parallel data augmentation for code-switch language modeling. In Interspeech."},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00343"},{"key":"e_1_3_4_30_2","article-title":"NLP Augmentation","author":"Ma Edward","year":"2019","unstructured":"Edward Ma. 2019. NLP Augmentation. Retrieved fromhttps:\/\/github.com\/makcedward\/nlpaug. (2019).","journal-title":"https:\/\/github.com\/makcedward\/nlpaug"},{"key":"e_1_3_4_31_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.findings-emnlp.4"},{"key":"e_1_3_4_32_2","volume-title":"Conference on Empirical Methods in Natural Language Processing","author":"Mikhailov Vladislav","year":"2022","unstructured":"Vladislav Mikhailov, Tatiana Shamardina, Max Ryabinin, Alena Pestova, Ivan Smurov, and E. Artemova. 2022. RuCoLA: Russian corpus of linguistic acceptability. In Conference on Empirical Methods in Natural Language Processing. https:\/\/api.semanticscholar.org\/CorpusID:253097947"},{"key":"e_1_3_4_33_2","volume-title":"Sociolinguistics: Method and interpretation","author":"Milroy Lesley","year":"2008","unstructured":"Lesley Milroy and Matthew Gordon. 2008. Sociolinguistics: Method and interpretation. John Wiley & Sons."},{"key":"e_1_3_4_34_2","first-page":"3417","volume-title":"Proceedings of the Thirteenth Language Resources and Evaluation Conference","author":"Mujadia Vandan","year":"2022","unstructured":"Vandan Mujadia and Dipti Sharma. 2022. The LTRC hindi-telugu parallel corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France, 3417\u20133424. Retrieved from https:\/\/aclanthology.org\/2022.lrec-1.365"},{"key":"e_1_3_4_35_2","unstructured":"Pieter Muysken. 2000. Bilingual speech: A typology of code-mixing."},{"key":"e_1_3_4_36_2","volume-title":"Duelling Languages: Grammatical Structure in Codeswitching","author":"Myers-Scotton C.","year":"1997","unstructured":"C. Myers-Scotton. 1997. Duelling Languages: Grammatical Structure in Codeswitching. Clarendon Press. Retrieved from 92039655https:\/\/books.google.co.in\/books?id=NuYdnTyKkdQC"},{"key":"e_1_3_4_37_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1493"},{"key":"e_1_3_4_38_2","doi-asserted-by":"publisher","DOI":"10.1515\/ling.1980.18.7-8.581"},{"key":"e_1_3_4_39_2","volume-title":"Annual Meeting of the Association for Computational Linguistics","author":"Pratapa Adithya","year":"2018","unstructured":"Adithya Pratapa, Gayatri Bhat, Monojit Choudhury, Sunayana Sitaram, Sandipan Dandapat, and Kalika Bali. 2018. Language modeling for code-mixing: The role of linguistic theory based synthetic data. In Annual Meeting of the Association for Computational Linguistics."},{"key":"e_1_3_4_40_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.wnut-1.18"},{"key":"e_1_3_4_41_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00452"},{"key":"e_1_3_4_42_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.eacl-demos.24"},{"key":"e_1_3_4_43_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.243"},{"key":"e_1_3_4_44_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.240"},{"key":"e_1_3_4_45_2","doi-asserted-by":"publisher","DOI":"10.17169\/langsci.b89.100"},{"key":"e_1_3_4_46_2","doi-asserted-by":"publisher","DOI":"10.1017\/S0022226700010537"},{"key":"e_1_3_4_47_2","article-title":"Intraclass correlations: Uses in assessing rater reliability.","volume":"86","author":"Shrout Patrick E.","year":"1979","unstructured":"Patrick E. Shrout and Joseph L. Fleiss. 1979. Intraclass correlations: Uses in assessing rater reliability. Psychological bulletin 86 2 (1979), 420\u20138.","journal-title":"Psychological bulletin"},{"key":"e_1_3_4_48_2","unstructured":"Sunayana Sitaram Khyathi Raghavi Chandu Sai Krishna Rallabandi and Alan W. Black. 2020. A survey of code-switched speech and language processing. (2020). arxiv:cs.CL\/1904.00784"},{"key":"e_1_3_4_49_2","doi-asserted-by":"crossref","DOI":"10.5964\/bioling.8597","article-title":"Continuous acceptability, categorical grammaticality, and experimental syntax","author":"Sprouse Jon","year":"2007","unstructured":"Jon Sprouse. 2007. Continuous acceptability, categorical grammaticality, and experimental syntax. Biolinguistics (2007).","journal-title":"Biolinguistics"},{"key":"e_1_3_4_50_2","doi-asserted-by":"crossref","first-page":"314","DOI":"10.18653\/v1\/2021.inlg-1.34","volume-title":"Proceedings of the 14th International Conference on Natural Language Generation","author":"Srivastava Vivek","year":"2021","unstructured":"Vivek Srivastava and Mayank Singh. 2021. Quality evaluation of the low-resource synthetically generated code-mixed hinglish text. In Proceedings of the 14th International Conference on Natural Language Generation. Association for Computational Linguistics, Aberdeen, Scotland, UK, 314\u2013319. Retrieved from https:\/\/aclanthology.org\/2021.inlg-1.34"},{"key":"e_1_3_4_51_2","unstructured":"Qwen Team. 2024. Qwen2.5: A Party of Foundation Models. (September2024). Retrieved from https:\/\/qwenlm.github.io\/blog\/qwen2.5\/"},{"key":"e_1_3_4_52_2","unstructured":"Tristan Thrush Christopher Potts and Tatsunori Hashimoto. 2024. Improving Pretraining Data Using Perplexity Correlations. (2024). arxiv:cs.CL\/2409.05816https:\/\/arxiv.org\/abs\/2409.05816"},{"key":"e_1_3_4_53_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-emnlp.250"},{"key":"e_1_3_4_54_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.12752"},{"key":"e_1_3_4_55_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00321"},{"key":"e_1_3_4_56_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00290"},{"key":"e_1_3_4_57_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00290"},{"key":"e_1_3_4_58_2","volume-title":"Conference on Computational Natural Language Learning","author":"Winata Genta Indra","year":"2019","unstructured":"Genta Indra Winata, Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2019. Code-switched language models using neural based synthetic data from parallel sentences. In Conference on Computational Natural Language Learning."},{"key":"e_1_3_4_59_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"e_1_3_4_60_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-short.42"},{"key":"e_1_3_4_61_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.eacl-main.242"},{"key":"e_1_3_4_62_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.41"},{"key":"e_1_3_4_63_2","article-title":"Qwen2 technical report","author":"Yang An","year":"2024","unstructured":"An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et\u00a0al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671 (2024).","journal-title":"arXiv preprint arXiv:2407.10671"}],"container-title":["ACM Transactions on Asian and Low-Resource Language Information Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3748312","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,10]],"date-time":"2025-09-10T13:29:27Z","timestamp":1757510967000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3748312"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,10]]},"references-count":62,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2025,9,30]]}},"alternative-id":["10.1145\/3748312"],"URL":"https:\/\/doi.org\/10.1145\/3748312","relation":{},"ISSN":["2375-4699","2375-4702"],"issn-type":[{"value":"2375-4699","type":"print"},{"value":"2375-4702","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,10]]},"assertion":[{"value":"2023-12-02","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-04","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-09-10","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}