{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,24]],"date-time":"2026-10-24T02:25:42Z","timestamp":1792808742194,"version":"4.5.0"},"reference-count":50,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2022,1,9]],"date-time":"2022-01-09T00:00:00Z","timestamp":1641686400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100007917","name":"Agricultural Research Service","doi-asserted-by":"publisher","award":["3050-11210-009-00D"],"award-info":[{"award-number":["3050-11210-009-00D"]}],"id":[{"id":"10.13039\/100007917","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Near-surface cameras, such as those in the PhenoCam network, are a common source of ground truth data in modelling and remote sensing studies. Despite having locations across numerous agricultural sites, few studies have used near-surface cameras to track the unique phenology of croplands. Due to management activities, crops do not have a natural vegetation cycle which many phenological extraction methods are based on. For example, a field may experience abrupt changes due to harvesting and tillage throughout the year. A single camera can also record several different plants due to crop rotations, fallow fields, and cover crops. Current methods to estimate phenology metrics from image time series compress all image information into a relative greenness metric, which discards a large amount of contextual information. This can include the type of crop present, whether snow or water is present on the field, the crop phenology, or whether a field lacking green plants consists of bare soil, fully senesced plants, or plant residue. Here, we developed a modelling workflow to create a daily time series of crop type and phenology, while also accounting for other factors such as obstructed images and snow covered fields. We used a mainstream deep learning image classification model, VGG16. Deep learning classification models do not have a temporal component, so to account for temporal correlation among images, our workflow incorporates a hidden Markov model in the post-processing. The initial image classification model had out of sample F1 scores of 0.83\u20130.85, which improved to 0.86\u20130.91 after all post-processing steps. The resulting time series show the progression of crops from emergence to harvest, and can serve as a daily, local-scale dataset of field states and phenological stages for agricultural research.<\/jats:p>","DOI":"10.3390\/rs14020286","type":"journal-article","created":{"date-parts":[[2022,1,9]],"date-time":"2022-01-09T23:08:26Z","timestamp":1641769706000},"page":"286","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":32,"title":["Classification of Daily Crop Phenology in PhenoCams Using Deep Learning and Hidden Markov Models"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6178-6903","authenticated-orcid":false,"given":"Shawn D.","family":"Taylor","sequence":"first","affiliation":[{"name":"US Department of Agriculture, Agricultural Research Service, Jornada Experimental Range, New Mexico State University, Las Cruces, NM 88003, USA"},{"name":"Oak Ridge Institute for Science and Education (ORISE), Oak Ridge, TN 37830, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1252-6013","authenticated-orcid":false,"given":"Dawn M.","family":"Browning","sequence":"additional","affiliation":[{"name":"US Department of Agriculture, Agricultural Research Service, Jornada Experimental Range, New Mexico State University, Las Cruces, NM 88003, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,1,9]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"111402","DOI":"10.1016\/j.rse.2019.111402","article-title":"Remote sensing for agricultural applications: A meta-review","volume":"236","author":"Weiss","year":"2020","journal-title":"Remote Sens. Environ."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"B\u00e9gu\u00e9, A., Arvor, D., Bellon, B., Betbeder, J., de Abelleyra, D., Ferraz, R.P., Lebourgeois, V., Lelong, C., Sim\u00f5es, M., and Ver\u00f3n, S.R. (2018). Remote sensing and cropping practices: A review. Remote Sens., 10.","DOI":"10.3390\/rs10010099"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1016\/j.rse.2016.11.004","article-title":"Toward mapping crop progress at field scales through fusion of Landsat and MODIS imagery","volume":"188","author":"Gao","year":"2017","journal-title":"Remote Sens. Environ."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1","DOI":"10.34133\/2021\/8379391","article-title":"Mapping Crop Phenology in Near Real-Time Using Satellite Remote Sensing: Challenges and Opportunities","volume":"2021","author":"Gao","year":"2021","journal-title":"J. Remote Sens."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1016\/j.agrformet.2018.11.002","article-title":"Monitoring crop phenology using a smartphone based near-surface remote sensing approach","volume":"265","author":"Hufkens","year":"2019","journal-title":"Agric. For. Meteorol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"103306","DOI":"10.1016\/j.agsy.2021.103306","article-title":"Using PhenoCams to track crop phenology and explain the effects of different cropping systems on yield","volume":"195","author":"Liu","year":"2022","journal-title":"Agric. Syst."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"180028","DOI":"10.1038\/sdata.2018.28","article-title":"Tracking vegetation phenology across diverse North American biomes using PhenoCam imagery","volume":"5","author":"Richardson","year":"2018","journal-title":"Sci. Data"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"222","DOI":"10.1038\/s41597-019-0229-9","article-title":"Tracking vegetation phenology across diverse biomes using Version 2.0 of the PhenoCam Dataset","volume":"6","author":"Seyednasrollah","year":"2019","journal-title":"Sci. Data"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"108147","DOI":"10.1016\/j.ecolind.2021.108147","article-title":"Monitoring agroecosystem productivity and phenology at a national scale: A metric assessment framework","volume":"131","author":"Browning","year":"2021","journal-title":"Ecol. Indic."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Borowiec, M.L., Frandsen, P., Dikow, R., McKeeken, A., Valentini, G., and White, A.E. (2021). Deep learning as a tool for ecology and evolution. EcoEvoRxiv.","DOI":"10.32942\/OSF.IO\/NT3AS"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"533","DOI":"10.1111\/1365-2656.12780","article-title":"A computer vision for animal ecology","volume":"87","author":"Weinstein","year":"2018","journal-title":"J. Anim. Ecol."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"E5716","DOI":"10.1073\/pnas.1719367115","article-title":"Automatically identifying, counting, and describing wild animals in camera-trap images with deep learning","volume":"115","author":"Norouzzadeh","year":"2018","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Conway, A.M., Durbach, I.N., McInnes, A., and Harris, R.N. (2021). Frame-by-frame annotation of video recordings using deep neural networks. Ecosphere, 12.","DOI":"10.1002\/ecs2.3384"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"13151","DOI":"10.1109\/ACCESS.2020.2965462","article-title":"Leveraging Artificial Intelligence for Large-Scale Plant Phenology Studies from Noisy Time-Lapse Images","volume":"8","author":"Correia","year":"2020","journal-title":"IEEE Access"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"109419","DOI":"10.1016\/j.ecolmodel.2020.109419","article-title":"Utilizing machine learning for detecting flowering in mid-range digital repeat photography","volume":"440","author":"Kim","year":"2021","journal-title":"Ecol. Model."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"plaa052","DOI":"10.1093\/aobpla\/plaa052","article-title":"What plant is that? Tests of automated image recognition apps for plant identification on plants from the British flora","volume":"12","author":"Jones","year":"2020","journal-title":"AoB Plants"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"4613","DOI":"10.1073\/pnas.1716999115","article-title":"An explainable deep machine vision framework for plant stress phenotyping","volume":"115","author":"Ghosal","year":"2018","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Kosmala, M., Crall, A., Cheng, R., Hufkens, K., Henderson, S., and Richardson, A. (2016). Season Spotter: Using Citizen Science to Validate and Scale Plant Phenology from Near-Surface Remote Sensing. Remote Sens., 8.","DOI":"10.3390\/rs8090726"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1016\/j.isprsjprs.2021.10.023","article-title":"Monitoring leaf phenology in moist tropical forests by applying a superpixel-based deep learning method to time-series images of tree canopies","volume":"183","author":"Song","year":"2022","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Cao, M., Sun, Y., Jiang, X., Li, Z., and Xin, Q. (2021). Identifying Leaf Phenology of Deciduous Broadleaf Forests from PhenoCam Images Using a Convolutional Neural Network Regression Method. Remote Sens., 13.","DOI":"10.3390\/rs13122331"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Benos, L., Tagarakis, A.C., Dolias, G., Berruto, R., Kateris, D., and Bochtis, D. (2021). Machine Learning in Agriculture: A Comprehensive Updated Review. Sensors, 21.","DOI":"10.3390\/s21113758"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Yalcin, H. (2017, January 7\u201310). Plant phenology recognition using deep learning: Deep-Pheno. Proceedings of the 2017 6th International Conference on Agro-Geoinformatics, Agro-Geoinformatics, Fairfax, VA, USA.","DOI":"10.1109\/Agro-Geoinformatics.2017.8046996"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1007\/s11119-020-09734-2","article-title":"Real-time detection of rice phenology through convolutional neural network using handheld camera images","volume":"22","author":"Han","year":"2021","journal-title":"Precis. Agric."},{"key":"ref_24","first-page":"754","article-title":"A Deep Convolutional Neural Network Based Image Processing Framework for Monitoring the Growth of Soybean Crops","volume":"Volume 2","author":"Alkady","year":"2021","journal-title":"Proceedings of the 2021 ASABE Annual International Virtual Meeting"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1742","DOI":"10.1111\/nph.15591","article-title":"Tracking seasonal rhythms of plants in diverse ecosystems with digital camera imagery","volume":"222","author":"Richardson","year":"2019","journal-title":"New Phytol."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Aasen, H., Kirchgessner, N., Walter, A., and Liebisch, F. (2020). PhenoCams for Field Phenotyping: Using Very High Temporal Resolution Digital Repeated Photography to Investigate Interactions of Growth, Phenology, and Harvest Traits. Front. Plant Sci., 11.","DOI":"10.3389\/fpls.2020.00593"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"754275","DOI":"10.1002\/aps3.11315","article-title":"Methods for broad-scale plant phenology assessments using citizen scientists\u2019 photographs","volume":"8","author":"Barve","year":"2020","journal-title":"Appl. Plant Sci."},{"key":"ref_28","unstructured":"Meier, U. (1997). Growth Stages of Mono-and Dicotyledonous Plants, Blackwell Wissenschafts-Verlag."},{"key":"ref_29","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv."},{"key":"ref_30","unstructured":"Chollet, F. (2018). Keras: The python deep learning library. Astrophys. Source Code Libr."},{"key":"ref_31","unstructured":"Milliman, T., Seyednasrollah, B., Young, A.M., Hufkens, K., Friedl, M.A., Frolking, S., Richardson, A.D., Abraha, M., Allen, D.W., and Apple, M. (2019). PhenoCam Dataset v2.0: Digital Camera Imagery from the PhenoCam Network, 2000\u20132018, ORNL DAAC."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Esmael, B., Arnaout, A., Fruhwirth, R.K., and Thonhauser, G. (2012, January 4\u20137). Improving time series classification using Hidden Markov Models. Proceedings of the 2012 12th International Conference on Hybrid Intelligent Systems (HIS), Pune, India.","DOI":"10.1109\/HIS.2012.6421385"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1016\/j.isprsjprs.2015.04.009","article-title":"A spatial\u2013temporal contextual Markovian kernel method for multi-temporal land cover mapping","volume":"107","author":"Wehmann","year":"2015","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"703","DOI":"10.1109\/TGRS.2015.2463689","article-title":"Improving the Consistency of Multitemporal Land Cover Maps Using a Hidden Markov Model","volume":"54","author":"Abercrombie","year":"2016","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_35","unstructured":"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (2016). TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"McKinney, W. (July, January 28). Data Structures for Statistical Computing in Python. Proceedings of the 9th Python in Science Conference, Austin, TX, USA. SciPy: 2010.","DOI":"10.25080\/Majora-92bf1922-00a"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1038\/s41586-020-2649-2","article-title":"Array programming with NumPy","volume":"585","author":"Harris","year":"2020","journal-title":"Nature"},{"key":"ref_38","first-page":"1","article-title":"Pomegranate: Fast and flexible probabilistic modeling in python","volume":"18","author":"Schreiber","year":"2018","journal-title":"J. Mach. Learn. Res."},{"key":"ref_39","unstructured":"(2021, January 01). Python Software Foundation. 2003. Python Language Reference Manual, version 3.6. Available online: http:\/\/www.python.org."},{"key":"ref_40","unstructured":"R Core Team (2017). R: A Language and Environment for Statistical Computing, R Core Team."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Zeileis, A., and Grothendieck, G. (2005). zoo: S3 Infrastructure for Regular and Irregular Time Series. J. Stat. Softw., 14.","DOI":"10.18637\/jss.v014.i06"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1686","DOI":"10.21105\/joss.01686","article-title":"Welcome to the Tidyverse","volume":"4","author":"Wickham","year":"2019","journal-title":"J. Open Source Softw."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Wickham, H. (2016). ggplot2: Elegant Graphics for Data Analysis, Springer-Verlag.","DOI":"10.1007\/978-3-319-24277-4_9"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Taylor, S.D. (2022). Analysis Code for: Classification of Daily Crop Phenology in PhenoCams Using Deep Learning and Hidden Markov Modelss, CERN.","DOI":"10.1101\/2021.10.20.465168"},{"key":"ref_45","unstructured":"USDA-NASS (2012). The Yield Forecasting Program of NASS. Report SMB 12-01, Technical Report."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"e2019JG005529","DOI":"10.1029\/2019JG005529","article-title":"Simulating agriculture in the Community Land Model version 5","volume":"125","author":"Lombardozzi","year":"2020","journal-title":"J. Geophys. Res. Biogeosciences"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"5679","DOI":"10.1038\/s41598-018-23804-6","article-title":"Intercomparison of phenological transition dates derived from the PhenoCam Dataset V1.0 and MODIS satellite remote sensing","volume":"8","author":"Richardson","year":"2018","journal-title":"Sci. Rep."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"2146","DOI":"10.1016\/j.rse.2010.04.019","article-title":"A Two-Step Filtering approach for detecting maize and soybean phenology with time-series MODIS data","volume":"114","author":"Sakamoto","year":"2010","journal-title":"Remote Sens. Environ."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1023\/A:1007469218079","article-title":"The hierarchical hidden Markov model: Analysis and applications","volume":"32","author":"Fine","year":"1998","journal-title":"Mach. Learn."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1007\/978-3-319-46487-9_3","article-title":"Segmental Spatiotemporal CNNs for Fine-Grained Action Segmentation","volume":"Volume 9907","author":"Leibe","year":"2016","journal-title":"Computer Vision\u2014ECCV 2016"}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/2\/286\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T14:01:46Z","timestamp":1760364106000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/2\/286"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,9]]},"references-count":50,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2022,1]]}},"alternative-id":["rs14020286"],"URL":"https:\/\/doi.org\/10.3390\/rs14020286","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.10.20.465168","asserted-by":"object"}]},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,9]]}}}