{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T04:04:37Z","timestamp":1776830677072,"version":"3.51.2"},"reference-count":60,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2024,11,26]],"date-time":"2024-11-26T00:00:00Z","timestamp":1732579200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"JSPS KAKENHI","award":["24K20795"],"award-info":[{"award-number":["24K20795"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Imaging"],"abstract":"<jats:p>Food semantic segmentation is of great significance in the field of computer vision and artificial intelligence, especially in the application of food image analysis. Due to the complexity and variety of food, it is difficult to effectively handle this task using supervised methods. Thus, we introduce IngredSAM, a novel approach for open-world food ingredient semantic segmentation, extending the capabilities of the Segment Anything Model (SAM). Utilizing visual foundation models (VFMs) and prompt engineering, IngredSAM leverages discriminative and matchable semantic features between a single clean image prompt of specific ingredients and open-world images to guide the generation of accurate segmentation masks in real-world scenarios. This method addresses the challenges of traditional supervised models in dealing with the diverse appearances and class imbalances of food ingredients. Our framework demonstrates significant advancements in the segmentation of food ingredients without any training process, achieving 2.85% and 6.01% better performance than previous state-of-the-art methods on both FoodSeg103 and UECFoodPix datasets. IngredSAM exemplifies a successful application of one-shot, open-world segmentation, paving the way for downstream applications such as enhancements in nutritional analysis and consumer dietary trend monitoring.<\/jats:p>","DOI":"10.3390\/jimaging10120305","type":"journal-article","created":{"date-parts":[[2024,11,26]],"date-time":"2024-11-26T07:40:32Z","timestamp":1732606832000},"page":"305","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["IngredSAM: Open-World Food Ingredient Segmentation via a Single Image Prompt"],"prefix":"10.3390","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-1605-0213","authenticated-orcid":false,"given":"Leyi","family":"Chen","sequence":"first","affiliation":[{"name":"College of Food Science and Nutritional Engineering, China Agricultural University, Beijing 100083, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bowen","family":"Wang","sequence":"additional","affiliation":[{"name":"D3 Center, Osaka University, 2-1, Yamadaoka, Osaka 5650871, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6330-6723","authenticated-orcid":false,"given":"Jiaxin","family":"Zhang","sequence":"additional","affiliation":[{"name":"Architecture and Design College, Nanchang University, No. 999, Xuefu Avenue, Honggutan New District, Nanchang 330031, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,11,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Wu, X., Fu, X., Liu, Y., Lim, E.P., Hoi, S.C., and Sun, Q. (2021, January 20\u201324). A large-scale benchmark for food image segmentation. Proceedings of the 29th ACM International Conference on Multimedia, Virtual Event.","DOI":"10.1145\/3474085.3475201"},{"key":"ref_2","unstructured":"Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., and Liu, W. (November, January 27). Ccnet: Criss-cross attention for semantic segmentation. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Republic of Korea."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Wang, Q., Dong, X., Wang, R., and Sun, H. (2022, January 10\u201312). Swin transformer based pyramid pooling network for food segmentation. Proceedings of the 2022 IEEE 2nd International Conference on Software Engineering and Artificial Intelligence (SEAI), Xiamen, China.","DOI":"10.1109\/SEAI55746.2022.9832133"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Honbu, Y., and Yanai, K. (2022, January 27\u201330). Unseen food segmentation. Proceedings of the 2022 International Conference on Multimedia Retrieval, Newark, NJ, USA.","DOI":"10.1145\/3512527.3531426"},{"key":"ref_5","unstructured":"Sinha, G., Parmar, K., Azimi, H., Tai, A., Chen, Y., Wong, A., and Xi, P. (2023). Transferring knowledge for food image segmentation using transformers and convolutions. arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Zhu, Z., and Dai, Y. (2023). A New CNN-Based Single-Ingredient Classification Model and Its Application in Food Image Segmentation. J. Imaging, 9.","DOI":"10.20944\/preprints202308.1030.v1"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"He, Y., Xu, C., Khanna, N., Boushey, C.J., and Delp, E.J. (2013, January 15\u201319). Food image analysis: Segmentation, identification and weight estimation. Proceedings of the 2013 IEEE International Conference on Multimedia and Expo (ICME), San Jose, CA, USA.","DOI":"10.1109\/ICME.2013.6607548"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"691","DOI":"10.1108\/09590550710773246","article-title":"Using behavioural and motivational thinking in food segmentation","volume":"35","author":"Hollywood","year":"2007","journal-title":"Int. J. Retail. Distrib. Manag."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Kirillov, A., He, K., Girshick, R., Rother, C., and Doll\u00e1r, P. (2019, January 15\u201320). Panoptic segmentation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00963"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., and Torr, P.H. (2021, January 19\u201325). Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00681"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Okamoto, K., and Yanai, K. (2021, January 10\u201315). UEC-FoodPIX Complete: A large-scale food image segmentation dataset. Proceedings of the Pattern Recognition, ICPR International Workshops and Challenges, Virtual Event.","DOI":"10.1007\/978-3-030-68821-9_51"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Wang, B., Li, L., Verma, M., Nakashima, Y., Kawasaki, R., and Nagahara, H. (2021, January 19\u201325). MTUNet: Few-shot Image Classification with Visual Explanations. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshop, Nashville, TN, USA.","DOI":"10.1109\/CVPRW53098.2021.00259"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Min, W., Liu, L., Wang, Z., Luo, Z., Wei, X., Wei, X., and Jiang, S. (2020, January 12\u201316). Isia food-500: A dataset for large-scale food recognition via stacked global-local attention network. Proceedings of the 28th ACM International Conference on Multimedia, Virtual Event.","DOI":"10.1145\/3394171.3414031"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., and Lo, W.Y. (2023, January 4\u20136). Segment anything. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Paris, France.","DOI":"10.1109\/ICCV51070.2023.00371"},{"key":"ref_15","unstructured":"Brown, T.B. (2020). Language models are few-shot learners. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"208","DOI":"10.1016\/j.aiopen.2023.08.012","article-title":"GPT understands, too","volume":"5","author":"Liu","year":"2024","journal-title":"AI Open"},{"key":"ref_17","unstructured":"Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18\u201324). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, PMLR, Virtual Event."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"10956","DOI":"10.1007\/s10489-022-04072-4","article-title":"Match them up: Visually explainable few-shot image classification","volume":"53","author":"Wang","year":"2022","journal-title":"Appl. Intell."},{"key":"ref_19","unstructured":"Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., and El-Nouby, A. (2023). Dinov2: Learning robust visual features without supervision. arXiv."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"He, K., Chen, X., Xie, S., Li, Y., Doll\u00e1r, P., and Girshick, R. (2022, January 18\u201324). Masked autoencoders are scalable vision learners. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01553"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., and Ballas, N. (2023, January 18\u201322). Self-supervised learning from images with a joint-embedding predictive architecture. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.01499"},{"key":"ref_22","unstructured":"Devlin, J. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Liu, J., Li, L., Xiang, T., Wang, B., and Qian, Y. (2023). Tcra-llm: Token compression retrieval augmented large language model for inference cost reduction. arXiv.","DOI":"10.18653\/v1\/2023.findings-emnlp.655"},{"key":"ref_24","unstructured":"Cheng, Y., Li, L., Xu, Y., Li, X., Yang, Z., Wang, W., and Yang, Y. (2023). Segment and track anything. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Andriiashen, V., van Liere, R., van Leeuwen, T., and Batenburg, K.J. (2021). Unsupervised foreign object detection based on dual-energy absorptiometry in the food industry. J. Imaging, 7.","DOI":"10.3390\/jimaging7070104"},{"key":"ref_26","first-page":"8743","article-title":"Open world entity segmentation","volume":"45","author":"Qi","year":"2022","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Cen, J., Yun, P., Cai, J., Wang, M.Y., and Liu, M. (2021, January 11\u201317). Deep metric learning for open world semantic segmentation. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, BC, Canada.","DOI":"10.1109\/ICCV48922.2021.01505"},{"key":"ref_28","unstructured":"Yang, J., Gao, M., Li, Z., Gao, S., Wang, F., and Zheng, F. (2023). Track anything: Segment anything meets videos. arXiv."},{"key":"ref_29","unstructured":"Zhu, J., Chen, Z., Hao, Z., Chang, S., Zhang, L., Wang, D., Lu, H., Luo, B., He, J.Y., and Lan, J.P. (2023). Tracking anything in high quality. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Zhang, K., and Liu, D. (2023). Customized segment anything model for medical image segmentation. arXiv.","DOI":"10.2139\/ssrn.4495221"},{"key":"ref_31","unstructured":"Chen, T., Mai, Z., Li, R., and Chao, W.l. (2023). Segment anything model (sam) enhanced pseudo labels for weakly supervised semantic segmentation. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Wang, B., Li, L., Nakashima, Y., and Nagahara, H. (2023, January 18\u201322). Learning Bottleneck Concepts in Image Classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.01055"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"102463","DOI":"10.1016\/j.aei.2024.102463","article-title":"Improving facade parsing with vision transformers and line integration","volume":"60","author":"Wang","year":"2024","journal-title":"Adv. Eng. Inform."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Myers, J., Najafian, K., Maleki, F., and Ovens, K. (2024). Efficient Wheat Head Segmentation with Minimal Annotation: A Generative Approach. J. Imaging, 10.","DOI":"10.3390\/jimaging10070152"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Boston, T., Van Dijk, A., and Thackway, R. (2024). U-Net Convolutional Neural Network for Mapping Natural Vegetation and Forest Types from Landsat Imagery in Southeastern Australia. J. Imaging, 10.","DOI":"10.2139\/ssrn.4727252"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Kwenda, C., Gwetu, M., and Fonou-Dombeu, J.V. (2024). Hybridizing Deep Neural Networks and Machine Learning Models for Aerial Satellite Forest Image Segmentation. J. Imaging, 10.","DOI":"10.3390\/jimaging10060132"},{"key":"ref_37","unstructured":"Shen, Q., Yang, X., and Wang, X. (2023). Anything-3d: Towards single-view anything reconstruction in the wild. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"100484","DOI":"10.1016\/j.patter.2022.100484","article-title":"Applications of knowledge graphs for food science and industry","volume":"3","author":"Min","year":"2022","journal-title":"Patterns"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"9932","DOI":"10.1109\/TPAMI.2023.3237871","article-title":"Large scale visual food recognition","volume":"45","author":"Min","year":"2023","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Klotz, J., Rengarajan, V., and Sankaranarayanan, A.C. (2021, January 11\u201317). Fine-grain prediction of strawberry freshness using subsurface scattering. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, BC, Canada.","DOI":"10.1109\/ICCVW54120.2021.00264"},{"key":"ref_41","unstructured":"Dosovitskiy, A. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv."},{"key":"ref_42","first-page":"7537","article-title":"Fourier features let networks learn high frequency functions in low dimensional domains","volume":"33","author":"Tancik","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_43","first-page":"5998","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Zhou, H., Qiao, B., Yang, L., Lai, J., and Xie, X. (2023, January 18\u201322). Texture-guided saliency distilling for unsupervised salient object detection. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.00701"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Tang, L., Jiang, P.T., Xiao, H., and Li, B. (2024). Towards Training-Free Open-World Segmentation via Image Prompt Foundation Models. Int. J. Comput. Vis., 1\u201315.","DOI":"10.1007\/s11263-024-02185-6"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1145\/1015706.1015720","article-title":"\u201cGrabCut\u201d interactive foreground extraction using iterated graph cuts","volume":"23","author":"Rother","year":"2004","journal-title":"ACM Trans. Graph. (TOG)"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Ege, T., Shimoda, W., and Yanai, K. (2019, January 21). A new large-scale food image segmentation dataset and its application to food calorie estimation based on grains of rice. Proceedings of the 5th International Workshop on Multimedia Assisted Dietary Management, Nice, France.","DOI":"10.1145\/3347448.3357162"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"108380","DOI":"10.1016\/j.compeleceng.2022.108380","article-title":"Bayesian deep learning for semantic segmentation of food images","volume":"103","author":"Aguilar","year":"2022","journal-title":"Comput. Electr. Eng."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Sharma, U., Artacho, B., and Savakis, A. (2021). Gourmetnet: Food segmentation using multi-scale waterfall features with spatial and channel attention. Sensors, 21.","DOI":"10.3390\/s21227504"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Wang, X., Zhang, X., Cao, Y., Wang, W., Shen, C., and Huang, T. (2023, January 2\u20136). Seggpt: Towards segmenting everything in context. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Paris, France.","DOI":"10.1109\/ICCV51070.2023.00110"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Dong, X., Wang, W., Li, H., and Cai, Q. (2021, January 7\u20138). Windows attention based pyramid network for food segmentation. Proceedings of the 2021 IEEE 7th International Conference on Cloud Computing and Intelligent Systems (CCIS), Xi\u2019an, China.","DOI":"10.1109\/CCIS53392.2021.9754670"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Xiao, T., Liu, Y., Zhou, B., Jiang, Y., and Sun, J. (2018, January 8\u201314). Unified perceptual parsing for scene understanding. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01228-1_26"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Lan, X., Lyu, J., Jiang, H., Dong, K., Niu, Z., Zhang, Y., and Xue, J. (2023). Foodsam: Any food segmentation. IEEE Trans. Multimed., 1\u201314.","DOI":"10.1109\/TMM.2023.3330047"},{"key":"ref_54","unstructured":"Zhang, R., Jiang, Z., Guo, Z., Yan, S., Pan, J., Ma, X., Dong, H., Gao, P., and Li, H. (2023). Personalize segment anything model with one shot. arXiv."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Wang, X., Wang, W., Cao, Y., Shen, C., and Huang, T. (2023, January 18\u201322). Images speak in images: A generalist painter for in-context visual learning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.00660"},{"key":"ref_56","unstructured":"Guo, J., Hao, Z., Wang, C., Tang, Y., Wu, H., Hu, H., Han, K., and Xu, C. (2024). Data-efficient large vision models through sequential autoregression. arXiv."},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018, January 8\u201314). Encoder-decoder with atrous separable convolution for semantic image segmentation. Proceedings of the European conference on computer vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01234-2_49"},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"2049","DOI":"10.1007\/s00530-023-01088-9","article-title":"The segmented UEC Food-100 dataset with benchmark experiment on food detection","volume":"29","author":"Arslan","year":"2023","journal-title":"Multimed. Syst."},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Yang, C., An, Z., Huang, L., Bi, J., Yu, X., Yang, H., Diao, B., and Xu, Y. (2024, January 17\u201318). CLIP-KD: An Empirical Study of CLIP Model Distillation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR52733.2024.01510"},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Yang, C., Zhou, H., An, Z., Jiang, X., Xu, Y., and Zhang, Q. (2022, January 18\u201324). Cross-image relational knowledge distillation for semantic segmentation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01200"}],"container-title":["Journal of Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-433X\/10\/12\/305\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:39:43Z","timestamp":1760114383000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-433X\/10\/12\/305"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,26]]},"references-count":60,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["jimaging10120305"],"URL":"https:\/\/doi.org\/10.3390\/jimaging10120305","relation":{},"ISSN":["2313-433X"],"issn-type":[{"value":"2313-433X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,26]]}}}