{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,19]],"date-time":"2026-02-19T02:25:33Z","timestamp":1771467933992,"version":"3.50.1"},"reference-count":39,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2025,8,13]],"date-time":"2025-08-13T00:00:00Z","timestamp":1755043200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Deputyship for Research &amp; Innovation, Ministry of Education, in Saudi Arabia","award":["MoE-IF-UJ-22-Innov-L"],"award-info":[{"award-number":["MoE-IF-UJ-22-Innov-L"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Imaging"],"abstract":"<jats:p>Food product labels serve as a critical source of information, providing details about nutritional content, ingredients, and health implications. These labels enable Food and Drug Authorities (FDA) to ensure compliance and take necessary health-related and logistics actions. Additionally, product labels are essential for online grocery stores to offer reliable nutrition facts and empower customers to make informed dietary decisions. Unfortunately, product labels are typically available in image formats, requiring organizations and online stores to manually transcribe them\u2014a process that is not only time-consuming but also highly prone to human error, especially with multilingual labels that add complexity to the task. Our study investigates the challenges and effectiveness of leveraging large language models (LLMs) to extract nutritional elements and values from multilingual food product labels, with a specific focus on Arabic and English. A comprehensive empirical analysis was conducted using a manually curated dataset of 294 food product labels, comprising 588 transcribed nutritional elements and values in both languages, which served as the ground truth for evaluation. The findings reveal that while LLMs performed better in extracting English elements and values compared to Arabic, our post-processing techniques significantly enhanced their accuracy, with GPT-4o outperforming GPT-4V and Gemini.<\/jats:p>","DOI":"10.3390\/jimaging11080271","type":"journal-article","created":{"date-parts":[[2025,8,13]],"date-time":"2025-08-13T13:30:38Z","timestamp":1755091838000},"page":"271","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Extract Nutritional Information from Bilingual Food Labels Using Large Language Models"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4959-5888","authenticated-orcid":false,"given":"Fatmah Y.","family":"Assiri","sequence":"first","affiliation":[{"name":"Software Engineering Department, College of Computer Science and Engineering, University of Jeddah, Jeddah 21493, Saudi Arabia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3399-2996","authenticated-orcid":false,"given":"Mohammad D.","family":"Alahmadi","sequence":"additional","affiliation":[{"name":"Software Engineering Department, College of Computer Science and Engineering, University of Jeddah, Jeddah 21493, Saudi Arabia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mohammed A.","family":"Almuashi","sequence":"additional","affiliation":[{"name":"Computer Science and Artificial Intelligence Department, College of Computer Science and Engineering, University of Jeddah, Jeddah 21493, Saudi Arabia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-1794-2749","authenticated-orcid":false,"given":"Ayidh M.","family":"Almansour","sequence":"additional","affiliation":[{"name":"Saudi Food and Drug Authority (SFDA), Riyadh 11671, Saudi Arabia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2025,8,13]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1016\/j.bj.2018.03.003","article-title":"Food safety in the 21st century","volume":"41","author":"Fung","year":"2018","journal-title":"Biomed. J."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Gundimeda, V., Murali, R.S., Joseph, R., and Naresh Babu, N. (2019). An automated computer vision system for extraction of retail food product metadata. First International Conference on Artificial Intelligence and Cognitive Computing, Proceedings of the AI Computing Conference 2018, Beijing, China, 12 September 2018, Springer.","DOI":"10.1007\/978-981-13-1580-0_20"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Fuentes, J.M., Paredes, R., Fulladosa, E., del Mar Gir\u00f3, M., and Claret, A. (2022, January 6\u20138). Automatic Transcription System for Nutritional Information Charts of Spanish Food Products. Proceedings of the VISIGRAPP (4: VISAPP), Virtual Event.","DOI":"10.5220\/0010827600003124"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"449","DOI":"10.1007\/s11760-020-01764-7","article-title":"A novel unified deep neural networks methodology for use by date recognition in retail food package image","volume":"15","author":"Gong","year":"2021","journal-title":"Signal Image Video Process."},{"key":"ref_5","first-page":"5722","article-title":"Integration of LLM in Expiration Date Scanning for Visually Impaired People","volume":"25","author":"Grumeza","year":"2024","journal-title":"Scalable Comput. Pract. Exp."},{"key":"ref_6","first-page":"91","article-title":"A framework for halal products checking interactive application with OCR and AR technologies","volume":"9","author":"Lam","year":"2017","journal-title":"J. Telecommun. Electron. Comput. Eng. (JTEC)"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Akram, T., Rizvi, H.H., Ali, S.A., Hamza, S.M., and Ifthikhar, A. (2020, January 8\u20139). OCR and Barcode based Halal and Health Analyzer. Proceedings of the 2020 International Conference on Information Science and Communication Technology (ICISCT), Karachi, Pakistan.","DOI":"10.1109\/ICISCT49550.2020.9080036"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Pravst, I., Hribar, M., \u017dmitek, K., Bla\u017eica, B., Korou\u0161i\u0107 Seljak, B., and Ku\u0161ar, A. (2022). Branded foods databases as a tool to support nutrition research and monitoring of the food supply: Insights from the slovenian composition and labeling information system. Front. Nutr., 8.","DOI":"10.3389\/fnut.2021.798576"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"129010","DOI":"10.1016\/j.foodchem.2021.129010","article-title":"Development of the Hellenic Food Thesaurus (HelTH), a branded food composition database: Aims, design and preliminary findings","volume":"347","author":"Katidi","year":"2021","journal-title":"Food Chem."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Kapsokefalou, M., Roe, M., Turrini, A., Costa, H.S., Martinez-Victoria, E., Marletta, L., Berry, R., and Finglas, P. (2019). Food composition at present: New challenges. Nutrients, 11.","DOI":"10.3390\/nu11081714"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Marconi, S., Durazzo, A., Camilli, E., Lisciani, S., Gabrielli, P., Aguzzi, A., Gambelli, L., Lucarini, M., and Marletta, L. (2018). Food composition databases: Considerations about complex food matrices. Foods, 7.","DOI":"10.3390\/foods7010002"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Kuang, J., Hua, W., Liang, D., Yang, M., Jiang, D., Ren, B., and Bai, X. (2023, January 21). Visual information extraction in the wild: Practical dataset and end-to-end solution. Proceedings of the International Conference on Document Analysis and Recognition, San Jose, CA, USA.","DOI":"10.1007\/978-3-031-41731-3_3"},{"key":"ref_13","unstructured":"Khasgiwala, A. (2018). Word Recognition in Nutrition Labels with Convolutional Neural Network. [Master\u2019s Thesis, Utah State University]."},{"key":"ref_14","unstructured":"Reibring, J. (2017). Photo OCR for Nutrition Labels. Combining Machine Learning and General Image Processing for Text Detection of American Nutrition Labels. [Master\u2019s Thesis, Department of Signals and Systems Chalmers University of Technology]."},{"key":"ref_15","unstructured":"Matsunaga, N., and Sullivan, R. (2015). Image Processing for the Extraction of Nutritional Information from Food Labels. [Bachelor\u2019s Thesis, Santa Clara University]."},{"key":"ref_16","unstructured":"Marinha, L.R.A. (2015). Automatic Nutritional Information Extraction from Photographic Images of Labels. [Master\u2019s Thesis, Universidade do Porto (Portugal)]."},{"key":"ref_17","unstructured":"Kulyukin, V., Zaman, T., and Andhavarapu, S.K. (2014, January 21\u201324). Effective nutrition label use on smartphones. Proceedings of the 2014 International Conference on Internet Computing (ICOMP), Las Vegas, NV, USA."},{"key":"ref_18","unstructured":"Grubert, O., and Gao, L. (2014). Recognition of Nutrition Facts Labels from Mobile Images, Stanford University. Technical Report;."},{"key":"ref_19","unstructured":"(2024, December 03). INFOODS. Middle East Food Composition Tables, 2017. Available online: https:\/\/www.fao.org\/infoods\/infoods\/tables-and-databases\/en\/."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"104047","DOI":"10.1016\/j.jfca.2021.104047","article-title":"Development of an Arabic food composition database for use in an Arabic online dietary assessment tool (myfood24)","volume":"102","author":"Bawajeeh","year":"2021","journal-title":"J. Food Compos. Anal."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"105299","DOI":"10.1016\/j.jfca.2023.105299","article-title":"The Saudi branded food database: First-phase development (Branded Beverage Database)","volume":"120","author":"Aldhirgham","year":"2023","journal-title":"J. Food Compos. Anal."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"329","DOI":"10.4103\/ijnpnd.ijnpnd_61_24","article-title":"Development of Omani-branded food composition database for an electronic dietary assessment tool","volume":"14","author":"Buthaina","year":"2024","journal-title":"Int. J. Nutr. Pharmacol. Neurol. Dis."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Carter, M.C., Hancock, N., Albar, S.A., Brown, H., Greenwood, D.C., Hardie, L.J., Frost, G.S., Wark, P.A., and Cade, J.E. (2016). Development of a new branded UK food composition database for an online dietary assessment tool. Nutrients, 8.","DOI":"10.3390\/nu8080480"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1016\/j.jfca.2017.07.019","article-title":"A partnership for public health: USDA branded food products database","volume":"64","author":"Kretser","year":"2017","journal-title":"J. Food Compos. Anal."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"744","DOI":"10.1016\/j.jfca.2010.09.001","article-title":"OQALI: A French database on processed foods","volume":"24","author":"Menard","year":"2011","journal-title":"J. Food Compos. Anal."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1016\/j.jfca.2017.07.008","article-title":"Belgian branded food products database: Inform consumers on a healthy lifestyle in a public-private partnership","volume":"64","author":"Seeuws","year":"2017","journal-title":"J. Food Compos. Anal."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"526","DOI":"10.1080\/03670244.2011.620880","article-title":"Nutritional value of 10 traditional dishes of the United Arab Emirates","volume":"50","author":"Habib","year":"2011","journal-title":"Ecol. Food Nutr."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1025","DOI":"10.9728\/dcs.2020.21.6.1025","article-title":"OCR-Based Safety Check System of Packaged Food for Food Inconvenience Patients","volume":"21","author":"Kamis","year":"2020","journal-title":"J. Digit. Contents Soc."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Parkavi, A., Shetty, T.B., Raj, V.A., Upadhyaya, S.B., and Thairani, R. (2023, January 6\u20138). Android application for food label recognition to ensure safe food consumption based on user allergen information leveraging OCR. Proceedings of the 2023 14th International Conference on Computing Communication and Networking Technologies (ICCCNT), Delhi, India.","DOI":"10.1109\/ICCCNT56998.2023.10307054"},{"key":"ref_30","unstructured":"Sheela, A., Sankar, S., Arun Mozhi, S., and Vignesh, C.K. (2024, January 8\u20139). NutriGaze: AI-Driven Dietary Planning with OCR and NLP. Proceedings of the 2024 International Conference on Power, Energy, Control and Transmission Systems (ICPECTS), Chennai, India."},{"key":"ref_31","first-page":"1263","article-title":"Food labelling recognition using ocr for mobile application","volume":"3","author":"Nasir","year":"2018","journal-title":"IUTM Comput. Proc. Innov. Comput. Technol. Appl."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"D\u2019Ambrosio, R., Gallinal, J., Souto, S., and Mayr, C. (2024, January 23\u201326). EasyEat: Personalized Healthy Eating Coach. Proceedings of the 2024 IEEE Global Humanitarian Technology Conference (GHTC), Radnor, PA, USA.","DOI":"10.1109\/GHTC62424.2024.10771570"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Papastratis, I., Konstantinidis, D., Daras, P., and Dimitropoulos, K. (2024). AI nutrition recommendation using a deep generative model and ChatGPT. Sci. Rep., 14.","DOI":"10.1038\/s41598-024-65438-x"},{"key":"ref_34","unstructured":"Hua, A., Preet Dhaliwal, M., Burke, R., and Qin, Y. (2024). NutriBench: A Dataset for Evaluating Large Language Models in Carbohydrate Estimation from Meal Descriptions. arXiv."},{"key":"ref_35","unstructured":"Noever, D., and Noever, S.E.M. (2023). The multimodal and modular ai chef: Complex recipe generation from imagery. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"7577","DOI":"10.1109\/JBHI.2024.3417280","article-title":"Dietary assessment with multimodal ChatGPT: A systematic analysis","volume":"28","author":"Lo","year":"2024","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_37","unstructured":"Jiao, P., Wu, X., Zhu, B., Chen, J., Ngo, C.W., and Jiang, Y. (2024). Rode: Linear rectified mixture of diverse experts for food large multi-modal models. arXiv."},{"key":"ref_38","unstructured":"Song, F., Zhu, B., Hao, Y., and Wang, S. (October, January 29). Enhancing recipe retrieval with foundation models: A data augmentation perspective. Proceedings of the European Conference on Computer Vision, Milan, Italy."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Mohbat, F., and Zaki, M.J. (2024, January 21\u201325). LLaVA-Chef: A Multi-modal Generative Model for Food Recipes. Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, Boise, ID, USA.","DOI":"10.1145\/3627673.3679562"}],"container-title":["Journal of Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-433X\/11\/8\/271\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:26:32Z","timestamp":1760034392000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-433X\/11\/8\/271"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,13]]},"references-count":39,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2025,8]]}},"alternative-id":["jimaging11080271"],"URL":"https:\/\/doi.org\/10.3390\/jimaging11080271","relation":{},"ISSN":["2313-433X"],"issn-type":[{"value":"2313-433X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,13]]}}}