{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T19:04:10Z","timestamp":1774638250979,"version":"3.50.1"},"reference-count":53,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2023,1,4]],"date-time":"2023-01-04T00:00:00Z","timestamp":1672790400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Institute of Diabetes and Digestive","award":["R01DK100796"],"award-info":[{"award-number":["R01DK100796"]}]},{"name":"National Institute of Diabetes and Digestive","award":["R01ADK122473"],"award-info":[{"award-number":["R01ADK122473"]}]},{"name":"Bill &amp; Melinda Gates Foundation","award":["R01DK100796"],"award-info":[{"award-number":["R01DK100796"]}]},{"name":"Bill &amp; Melinda Gates Foundation","award":["R01ADK122473"],"award-info":[{"award-number":["R01ADK122473"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Sensor-based food intake monitoring has become one of the fastest-growing fields in dietary assessment. Researchers are exploring imaging-sensor-based food detection, food recognition, and food portion size estimation. A major problem that is still being tackled in this field is the segmentation of regions of food when multiple food items are present, mainly when similar-looking foods (similar in color and\/or texture) are present. Food image segmentation is a relatively under-explored area compared with other fields. This paper proposes a novel approach to food imaging consisting of two imaging sensors: color (Red\u2013Green\u2013Blue) and thermal. Furthermore, we propose a multi-modal four-Dimensional (RGB-T) image segmentation using a k-means clustering algorithm to segment regions of similar-looking food items in multiple combinations of hot, cold, and warm (at room temperature) foods. Six food combinations of two food items each were used to capture RGB and thermal image data. RGB and thermal data were superimposed to form a combined RGB-T image and three sets of data (RGB, thermal, and RGB-T) were tested. A bootstrapped optimization of within-cluster sum of squares (WSS) was employed to determine the optimal number of clusters for each case. The combined RGB-T data achieved better results compared with RGB and thermal data, used individually. The mean \u00b1 standard deviation (std. dev.) of the F1 score for RGB-T data was 0.87 \u00b1 0.1 compared with 0.66 \u00b1 0.13 and 0.64 \u00b1 0.39, for RGB and Thermal data, respectively.<\/jats:p>","DOI":"10.3390\/s23020560","type":"journal-article","created":{"date-parts":[[2023,1,4]],"date-time":"2023-01-04T03:27:44Z","timestamp":1672802864000},"page":"560","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Food Image Segmentation Using Multi-Modal Imaging Sensors with Color and Thermal Data"],"prefix":"10.3390","volume":"23","author":[{"given":"Viprav B.","family":"Raju","sequence":"first","affiliation":[{"name":"Department Electrical and Computer Engineering, The University of Alabama, Tuscaloosa, AL 35487, USA"}]},{"given":"Masudul H.","family":"Imtiaz","sequence":"additional","affiliation":[{"name":"Department Electrical and Computer Engineering, Clarkson University, Potsdam, NY 13699, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7792-4234","authenticated-orcid":false,"given":"Edward","family":"Sazonov","sequence":"additional","affiliation":[{"name":"Department Electrical and Computer Engineering, The University of Alabama, Tuscaloosa, AL 35487, USA"}]}],"member":"1968","published-online":{"date-parts":[[2023,1,4]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Zhu, F., Bosch, M., Boushey, C.J., and Delp, E.J. (2010, January 26\u201329). An Image Analysis System for Dietary Assessment and Evaluation. Proceedings of the 2010 IEEE International Conference on Image Processing, Hong Kong, China.","DOI":"10.1109\/ICIP.2010.5650848"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Jia, W., Li, Y., Qu, R., Baranowski, T., Burke, L.E., Zhang, H., Bai, Y., Mancino, J.M., Xu, G., and Mao, Z.-H. (2018). Automatic Food Detection in Egocentric Images Using Artificial Intelligence Technology. Public Health Nutr., 1\u201312.","DOI":"10.1017\/S1368980018000538"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Fang, S., Zhu, F., Jiang, C., Zhang, S., Boushey, C.J., and Delp, E.J. (2016, January 25\u201328). Delp A Comparison of Food Portion Size Estimation Using Geometric Models and Depth Images. Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA.","DOI":"10.1109\/ICIP.2016.7532312"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Rahman, M.d.H., Li, Q., Pickering, M., Frater, M., Kerr, D., Bouchey, C., and Delp, E. (2012, January 25\u201329). Food Volume Estimation in a Mobile Phone Based Dietary Assessment System. Proceedings of the 8th International Conference on Signal Image Technology and Internet Based Systems, Sorrento, Italy. SITIS 2012r.","DOI":"10.1109\/SITIS.2012.146"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1016\/j.jfoodeng.2004.04.001","article-title":"Segmentation of Colour Food Images Using a Robust Algorithm","volume":"66","author":"Mery","year":"2005","journal-title":"J. Food Eng."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"He, Y., Khanna, N., Boushey, C.J., and Delp, E.J. (2012, January 17\u201319). Snakes Assisted Food Image Segmentation. Proceedings of the 2012 IEEE 14th International Workshop on Multimedia Signal Processing (MMSP), Banff, AB, Canada.","DOI":"10.1109\/MMSP.2012.6343437"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"025702","DOI":"10.1088\/0957-0233\/26\/2\/025702","article-title":"Saliency-Aware Food Image Segmentation for Personal Dietary Assessment Using a Wearable Computer","volume":"26","author":"Chen","year":"2015","journal-title":"Meas. Sci. Technol."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"He, Y., Xu, C., Khanna, N., Boushey, C.J., and Delp, E.J. (2013, January 15\u201319). Food Image Analysis: Segmentation, Identification and Weight Estimation. Proceedings of the 2013 IEEE International Conference on Multimedia and Expo (ICME), San Jose, CA, USA.","DOI":"10.1109\/ICME.2013.6607548"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Kawano, Y., and Yanai, K. (2013, January 23\u201328). Real-Time Mobile Food Recognition System. Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Portland, OR, USA.","DOI":"10.1109\/CVPRW.2013.5"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Dehais, J., Anthimopoulos, M., and Mougiakakou, S. (2016, January 16). Food Image Segmentation for Dietary Assessment. Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management\u2014MADiMa \u201916, New York, NY, USA.","DOI":"10.1145\/2986035.2986047"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Wang, Y., Liu, C., Zhu, F., Boushey, C.J., and Delp, E.J. (2016, January 25\u201328). Efficient Superpixel Based Segmentation for Food Image Analysis. Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA.","DOI":"10.1109\/ICIP.2016.7532818"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Wang, Y., Zhu, F., Boushey, C.J., and Delp, E.J. (2017, January 17\u201320). Weakly Supervised Food Image Segmentation Using Class Activation Maps. Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China.","DOI":"10.1109\/ICIP.2017.8296487"},{"key":"ref_13","unstructured":"Pfisterer, K.J., Amelard, R., Chung, A.G., Syrnyk, B., MacLean, A., and Wong, A. (2019). Fully-Automatic Semantic Segmentation for Food Intake Tracking in Long-Term Care Homes. arXiv."},{"key":"ref_14","first-page":"462","article-title":"Distinguishing Nigerian Food Items and Calorie Content with Hyperspectral Imaging","volume":"Volume 10590","author":"Battiato","year":"2017","journal-title":"New Trends in Image Analysis and Processing\u2014ICIAP 2017"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"65","DOI":"10.9781\/ijimai.2013.229","article-title":"Infected Fruit Part Detection Using K-Means Clustering Segmentation Technique","volume":"2","author":"Dubey","year":"2013","journal-title":"IJIMAI"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1007\/s40595-014-0028-3","article-title":"An Image Segmentation Approach for Fruit Defect Detection Using K-Means Clustering and Graph-Based Algorithm","volume":"2","author":"Pham","year":"2015","journal-title":"Vietnam J. Comput. Sci."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1111\/jfpe.12054","article-title":"The Potential of Double K-Means Clustering for Banana Image Segmentation: Image Segmentation on Banana","volume":"37","author":"Hu","year":"2014","journal-title":"J. Food Process. Eng."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1186\/s13640-018-0309-3","article-title":"Image Segmentation Based on Adaptive K-Means Algorithm","volume":"2018","author":"Zheng","year":"2018","journal-title":"J. Image Video Proc."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Siswantoro, J., Prabuwono, A.S., Abdullah, A., and Idrus, B. (2015, January 27\u201328). Automatic Image Segmentation Using Sobel Operator and K-Means Clustering: A Case Study in Volume Measurement System for Food Products. Proceedings of the 2015 International Conference on Science in Information Technology (ICSITech), Yogyakarta, Indonesia.","DOI":"10.1109\/ICSITech.2015.7407769"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Feng, Z., Song, L., Duan, J., He, L., Zhang, Y., Wei, Y., and Feng, W. (2022). Monitoring Wheat Powdery Mildew Based on Hyperspectral, Thermal Infrared, and RGB Image Data Fusion. Sensors, 22.","DOI":"10.3390\/s22010031"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Zhang, Y., M\u00fcller, S., Stephan, B., Gross, H.-M., and Notni, G. (2021). Point Cloud Hand\u2013Object Segmentation Using Multimodal Imaging with Thermal and Color Data for Safe Robotic Object Handover. Sensors, 21.","DOI":"10.3390\/s21165676"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Cohen, B., Edan, Y., Levi, A., and Alchanatis, V. (2022). Early Detection of Grapevine (Vitis Vinifera) Downy Mildew (Peronospora) and Diurnal Variations Using Thermal Imaging. Sensors, 22.","DOI":"10.3390\/s22093585"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Bougrine, A., Harba, R., Canals, R., Ledee, R., Jabloun, M., and Villeneuve, A. (2022). Segmentation of Plantar Foot Thermal Images Using Prior Information. Sensors, 22.","DOI":"10.3390\/s22103835"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Bhadoriya, A.S., Vegamoor, V., and Rathinam, S. (2022). Vehicle Detection and Tracking Using Thermal Cameras in Adverse Visibility Conditions. Sensors, 22.","DOI":"10.3390\/s22124567"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Schischmanow, A., Dahlke, D., Baumbach, D., Ernst, I., and Linkiewicz, M. (2022). Seamless Navigation, 3D Reconstruction, Thermographic and Semantic Mapping for Building Inspection. Sensors, 22.","DOI":"10.3390\/s22134745"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/0031-3203(81)90028-5","article-title":"A Survey on Image Segmentation","volume":"13","author":"Fu","year":"1981","journal-title":"Pattern Recognit."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Raju, V., and Sazonov, E. (2019, January 11\u201314). Processing of Egocentric Camera Images from a Wearable Food Intake Sensor. Proceedings of the 2019 SoutheastCon, Huntsville, AL, USA.","DOI":"10.1109\/SoutheastCon42311.2019.9020284"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Raju, V.B., and Sazonov, E. (2022). FOODCAM: A Novel Structured Light-Stereo Imaging System for Food Portion Size Estimation. Sensors, 22.","DOI":"10.3390\/s22093300"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Kim, N., Choi, Y., Hwang, S., Park, K., Yoon, J.S., and Kweon, I.S. (2015, January 28\u201330). Geometrical Calibration of Multispectral Calibration. Proceedings of the 2015 12th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI), Goyang, Republic of Korea.","DOI":"10.1109\/URAI.2015.7358880"},{"key":"ref_30","unstructured":"(Henry\u2019s Blog, 2018). Henry Zhang Methods of Thermal Camera Calibration, Henry\u2019s Blog."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1016\/j.infrared.2011.01.002","article-title":"Calibration and Verification of Thermographic Cameras for Geometric Measurements","volume":"54","author":"Armesto","year":"2011","journal-title":"Infrared Phys. Technol."},{"key":"ref_32","unstructured":"Nikolaev, D.P., Radeva, P., Verikas, A., and Zhou, J. (2019). Robust Low Resolution Thermal Stereo Camera Calibration. Eleventh International Conference on Machine Vision (ICMV 2018), SPIE."},{"key":"ref_33","unstructured":"Brooks, R.R., and Iyengar, S.S. (1998). Multi-Sensor Fusion: Fundamentals and Applications with Software, Prentice-Hall."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"R1","DOI":"10.1088\/0031-9155\/46\/3\/201","article-title":"Medical Image Registration","volume":"46","author":"Hill","year":"2001","journal-title":"Phys. Med. Biol."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1016\/j.media.2016.06.030","article-title":"A Survey of Medical Image Registration\u2014Under Review","volume":"33","author":"Viergever","year":"2016","journal-title":"Med. Image Anal."},{"key":"ref_36","unstructured":"Fonseca, L.M.G., and Kenney, C.S. (1999, January 17\u201320). Control Point Assessment for Image Registration. Proceedings of the XII Brazilian Symposium on Computer Graphics and Image Processing (Cat. No.PR00481), Campinas, Brazil."},{"key":"ref_37","first-page":"891","article-title":"Fully Automatic and Robust Approach for Remote Sensing Image Registration","volume":"Volume 4756","author":"Rueda","year":"2008","journal-title":"Progress in Pattern Recognition, Image Analysis and Applications"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"269","DOI":"10.1016\/S0167-8655(97)00010-X","article-title":"Image Registration by Control Points Pairing Using the Invariant Properties of Line Segments","volume":"18","author":"Wang","year":"1997","journal-title":"Pattern Recognit. Lett."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Robb, R.A. (1994, January 9). Grey Value Correlation Techniques Used for Automatic Matching of CT and MR Brain and Spine Images. Proceedings of the Volume 2359, Visualization in Biomedical Computing 1994, Rochester, MN, USA.","DOI":"10.1117\/12.185182"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1109\/83.650848","article-title":"A Pyramid Approach to Subpixel Registration Based on Intensity","volume":"7","author":"Thevenaz","year":"1998","journal-title":"IEEE Trans. on Image Process."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"100","DOI":"10.2307\/2346830","article-title":"Algorithm AS 136: A K-Means Clustering Algorithm","volume":"28","author":"Hartigan","year":"1979","journal-title":"Appl. Stat."},{"key":"ref_42","unstructured":"Arthur, D., and Vassilvitskii, S. (2007). K-Means++: The Advantages of Careful Seeding. Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, Society for Industrial and Applied Mathematics."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"2499","DOI":"10.1016\/j.dam.2013.04.015","article-title":"Optimising Sum-of-Squares Measures for Clustering Multisets Defined over a Metric Space","volume":"161","author":"Kettleborough","year":"2013","journal-title":"Discret. Appl. Math."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"23","DOI":"10.2307\/2531893","article-title":"A Criterion for Determining the Number of Groups in a Data Set Using Sum-of-Squares Clustering","volume":"44","author":"Krzanowski","year":"1988","journal-title":"Biometrics"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"919","DOI":"10.1214\/aop\/1176993713","article-title":"A Central Limit Theorem for k-Means Clustering","volume":"10","author":"Pollard","year":"1982","journal-title":"Ann. Probab."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"012017","DOI":"10.1088\/1757-899X\/336\/1\/012017","article-title":"Integration K-Means Clustering Method and Elbow Method For Identification of The Best Customer Profile Cluster","volume":"336","author":"Syakur","year":"2018","journal-title":"IOP Conf. Ser. Mater. Sci. Eng."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"5510","DOI":"10.1109\/TIP.2019.2920514","article-title":"Adaptive Morphological Reconstruction for Seeded Image Segmentation","volume":"28","author":"Lei","year":"2019","journal-title":"IEEE Trans. Image Process."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1109\/TFUZZ.2018.2883033","article-title":"Deviation-Sparse Fuzzy C-Means With Neighbor Information Constraint","volume":"27","author":"Zhang","year":"2019","journal-title":"IEEE Trans. Fuzzy Syst."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"1753","DOI":"10.1109\/TFUZZ.2018.2889018","article-title":"Superpixel-Based Fast Fuzzy C-Means Clustering for Color Image Segmentation","volume":"27","author":"Lei","year":"2019","journal-title":"IEEE Trans. Fuzzy Syst."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"2078","DOI":"10.1109\/TFUZZ.2019.2930030","article-title":"Automatic Fuzzy Clustering Framework for Image Segmentation","volume":"28","author":"Lei","year":"2020","journal-title":"IEEE Trans. Fuzzy Syst."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"146182","DOI":"10.1109\/ACCESS.2020.3015270","article-title":"Robust Self-Sparse Fuzzy Clustering for Image Segmentation","volume":"8","author":"Jia","year":"2020","journal-title":"IEEE Access"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1023\/B:VISI.0000022288.19776.77","article-title":"Efficient Graph-Based Image Segmentation","volume":"59","author":"Felzenszwalb","year":"2004","journal-title":"Int. J. Comput. Vis."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Ramesh, A., Raju, V.B., Rao, M., and Sazonov, E. (2021, January 1\u20135). Food Detection and Segmentation from Egocentric Camera Images. Proceedings of the 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Virtual.","DOI":"10.1109\/EMBC46164.2021.9630823"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/2\/560\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T17:58:14Z","timestamp":1760119094000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/2\/560"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,4]]},"references-count":53,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2023,1]]}},"alternative-id":["s23020560"],"URL":"https:\/\/doi.org\/10.3390\/s23020560","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,4]]}}}