{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T03:49:14Z","timestamp":1760240954762,"version":"build-2065373602"},"reference-count":63,"publisher":"MDPI AG","issue":"21","license":[{"start":{"date-parts":[[2019,10,27]],"date-time":"2019-10-27T00:00:00Z","timestamp":1572134400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001665","name":"Agence Nationale de la Recherche","doi-asserted-by":"publisher","award":["ANR-15-CE39-0005"],"award-info":[{"award-number":["ANR-15-CE39-0005"]}],"id":[{"id":"10.13039\/501100001665","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100008982","name":"Qatar National Research Fund","doi-asserted-by":"publisher","award":["NPRP-09-768-1-114"],"award-info":[{"award-number":["NPRP-09-768-1-114"]}],"id":[{"id":"10.13039\/100008982","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The main objective of this work is to study the applicability of ensemble methods in the context of deep learning with limited amounts of labeled data. We exploit an ensemble of neural networks derived using Monte Carlo dropout, along with an ensemble of SVM classifiers which owes its effectiveness to the hand-crafted features used as inputs and to an active learning procedure. In order to leverage each classifier\u2019s respective strengths, we combine them in an evidential framework, which models specifically their imprecision and uncertainty. The application we consider in order to illustrate the interest of our Multiple Classifier System is pedestrian detection in high-density crowds, which is ideally suited for its difficulty, cost of labeling and intrinsic imprecision of annotation data. We show that the fusion resulting from the effective modeling of uncertainty allows for performance improvement, and at the same time, for a deeper interpretation of the result in terms of commitment of the decision.<\/jats:p>","DOI":"10.3390\/s19214664","type":"journal-article","created":{"date-parts":[[2019,10,28]],"date-time":"2019-10-28T04:44:31Z","timestamp":1572237871000},"page":"4664","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Augmenting Deep Learning Performance in an Evidential Multiple Classifier System"],"prefix":"10.3390","volume":"19","author":[{"given":"Jennifer","family":"Vandoni","sequence":"first","affiliation":[{"name":"SATIE - CNRS UMR 8029, Paris-Sud University, Paris-Saclay University, 91405 Orsay CEDEX, France"},{"name":"SAFRAN SA, Safran Tech, Pole Technologie du Signal et de l\u2019Information, 78772 Magny-les-Hameaux, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8494-2289","authenticated-orcid":false,"given":"Sylvie Le","family":"H\u00e9garat-Mascle","sequence":"additional","affiliation":[{"name":"SATIE - CNRS UMR 8029, Paris-Sud University, Paris-Saclay University, 91405 Orsay CEDEX, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7065-4809","authenticated-orcid":false,"given":"Emanuel","family":"Aldea","sequence":"additional","affiliation":[{"name":"SATIE - CNRS UMR 8029, Paris-Sud University, Paris-Saclay University, 91405 Orsay CEDEX, France"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2019,10,27]]},"reference":[{"key":"ref_1","unstructured":"Gal, Y., and Ghahramani, Z. (2016, January 19\u201324). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. Proceedings of the International Conference on Machine Learning (ICML), New York, NY, USA."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"6884","DOI":"10.1109\/TGRS.2018.2845450","article-title":"Tensor-based classification models for hyperspectral data analysis","volume":"56","author":"Makantasis","year":"2018","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Shafer, G. (1976). A Mathematical Theory of Evidence, Princeton University Press.","DOI":"10.1515\/9780691214696"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1016\/0004-3702(94)90026-4","article-title":"The transferable belief model","volume":"66","author":"Smets","year":"1994","journal-title":"Artif. Intell."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.ijar.2016.07.010","article-title":"40 years of Dempster-Shafer theory","volume":"79","author":"Denoeux","year":"2016","journal-title":"Int. J. Approx. Reason."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1000","DOI":"10.1016\/j.ijar.2009.03.006","article-title":"Combination of partially non-distinct beliefs: The cautious-adaptive rule","volume":"50","author":"Kallel","year":"2009","journal-title":"Int. J. Approx. Reason."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1016\/j.ijar.2011.07.006","article-title":"Distances in evidence theory: Comprehensive survey and generalizations","volume":"53","author":"Jousselme","year":"2012","journal-title":"Int. J. Approx. Reason."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1016\/j.patcog.2015.10.014","article-title":"Online active learning of decision trees with evidential data","volume":"52","author":"Ma","year":"2016","journal-title":"Pattern Recognit."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1016\/j.engappai.2018.04.019","article-title":"Evidential framework for Error Correcting Output Code classification","volume":"73","author":"Lachaize","year":"2018","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_10","unstructured":"Pellican\u00f2, N., Aldea, E., and Le H\u00e9garat-Mascle, S. (2017, January 4\u20137). Geometry-Based Multiple Camera Head Detection in Dense Crowds. Proceedings of the 28th British Machine Vision Conference (BMVC)\u20145th Activity Monitoring by Multiple Distributed Sensing Workshop, London, UK."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"166","DOI":"10.1016\/j.ijar.2018.11.007","article-title":"Evidential query-by-committee active learning for pedestrian detection in high-density crowds","volume":"104","author":"Vandoni","year":"2019","journal-title":"Int. J. Approx. Reason."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"743","DOI":"10.1109\/TPAMI.2011.155","article-title":"Pedestrian detection: An evaluation of the state of the art","volume":"34","author":"Dollar","year":"2011","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_13","unstructured":"Dalal, N., and Triggs, B. (2005, January 20\u201325). Histograms of oriented gradients for human detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Diego, CA, USA."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1627","DOI":"10.1109\/TPAMI.2009.167","article-title":"Object detection with discriminatively trained part-based models","volume":"32","author":"Felzenszwalb","year":"2010","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Braik, M., Al-Zoubi, H., and Al-Hiary, H. (2019). Pedestrian detection using multiple feature channels and contour cues with census transform histogram and random forest classifier. Pattern Anal. Appl.","DOI":"10.1007\/s10044-019-00835-x"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Hosang, J., Omran, M., Benenson, R., and Schiele, B. (2015, January 7\u201312). Taking a deeper look at pedestrians. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7299034"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Tian, Y., Luo, P., Wang, X., and Tang, X. (2015, January 7\u201312). Pedestrian detection aided by deep learning semantic tasks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7299143"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Zhang, S., Benenson, R., Omran, M., Hosang, J., and Schiele, B. (2016, January 27\u201330). How Far are We from Solving Pedestrian Detection?. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.141"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zhang, L., Lin, L., Liang, X., and He, K. (2016, January 8\u201316). Is faster R-CNN doing well for pedestrian detection?. Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46475-6_28"},{"key":"ref_20","first-page":"985","article-title":"Scale-aware fast R-CNN for pedestrian detection","volume":"20","author":"Li","year":"2017","journal-title":"IEEE Trans. Multimed."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Sermanet, P., Kavukcuoglu, K., Chintala, S., and LeCun, Y. (2013, January 23\u201328). Pedestrian detection with unsupervised multi-stage feature learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Portland, OR, USA.","DOI":"10.1109\/CVPR.2013.465"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1874","DOI":"10.1109\/TPAMI.2017.2738645","article-title":"Jointly learning deep features, deformable parts, occlusion and classification for pedestrian detection","volume":"40","author":"Ouyang","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Ujjwal, U., Dziri, A., Leroy, B., and Bremond, F. (2018, January 27\u201330). Late Fusion of Multiple Convolutional Layers for Pedestrian Detection. Proceedings of the 15th IEEE International Conference on Advanced Video and Signal-based Surveillance (AVSS), Auckland, New Zealand.","DOI":"10.1109\/AVSS.2018.8639083"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"448","DOI":"10.1162\/neco.1992.4.3.448","article-title":"A practical Bayesian framework for backpropagation networks","volume":"4","author":"MacKay","year":"1992","journal-title":"Neural Comput."},{"key":"ref_25","unstructured":"Neal, R.M. (2012). Bayesian Learning for Neural Networks, Springer Science & Business Media."},{"key":"ref_26","unstructured":"Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. (2015, January 6\u201311). Weight Uncertainty in Neural Network. Proceedings of the International Conference on Machine Learning (ICML), Lille, France."},{"key":"ref_27","unstructured":"Graves, A. (2011, January 12\u201315). Practical variational inference for neural networks. Proceedings of the Advances in Neural Information Processing Systems (NIPS), Granada, Spain."},{"key":"ref_28","first-page":"10","article-title":"Auto-Encoding Variational Bayes","volume":"1050","author":"Kingma","year":"2014","journal-title":"Statistics"},{"key":"ref_29","unstructured":"Damianou, A., and Lawrence, N. (May, January 29). Deep gaussian processes. Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics, Scottsdale, AZ, USA."},{"key":"ref_30","unstructured":"Wan, L., Zeiler, M., Zhang, S., Le Cun, Y., and Fergus, R. (2013, January 16\u201321). Regularization of neural networks using dropconnect. Proceedings of the International Conference on Machine Learning (ICML), Atlanta, GA, USA."},{"key":"ref_31","first-page":"1929","article-title":"Dropout: A simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"ref_32","unstructured":"Kendall, A., Badrinarayanan, V., and Cipolla, R. (2015). Bayesian SegNet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding. arXiv."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Kendall, A., and Cipolla, R. (2016, January 16\u201321). Modelling uncertainty in deep learning for camera relocalization. Proceedings of the 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden.","DOI":"10.1109\/ICRA.2016.7487679"},{"key":"ref_34","unstructured":"Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017, January 4\u20139). Simple and scalable predictive uncertainty estimation using deep ensembles. Proceedings of the Advances in Neural Information Processing Systems (NIPS), Long Beach, CA, USA."},{"key":"ref_35","unstructured":"Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (2013). Intriguing properties of neural networks. arXiv."},{"key":"ref_36","unstructured":"Goodfellow, I.J., Shlens, J., and Szegedy, C. (2015). Explaining and Harnessing Adversarial Examples. arXiv."},{"key":"ref_37","unstructured":"Gal, Y. (2016). Uncertainty in Deep Learning. [Ph.D. Thesis, University of Cambridge]."},{"key":"ref_38","unstructured":"Lempitsky, V., and Zisserman, A. (2010, January 6\u20139). Learning to count objects in images. Proceedings of the Advances in Neural Information Processing Systems (NIPS), Vancouver, BC, Canada."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.patrec.2017.07.007","article-title":"A survey of recent advances in cnn-based single image crowd counting and density estimation","volume":"107","author":"Sindagi","year":"2018","journal-title":"Pattern Recognit. Lett."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Vandoni, J., Aldea, E., and Le H\u00e9garat-Mascle, S. (2019, January 22\u201325). Evaluating Crowd Density Estimators Via Their Uncertainty Bounds. Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.","DOI":"10.1109\/ICIP.2019.8803522"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Hamaguchi, R., Fujita, A., Nemoto, K., Imaizumi, T., and Hikosaka, S. (2018, January 12\u201315). Effective use of dilated convolutions for segmenting small object instances in remote sensing imagery. Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA.","DOI":"10.1109\/WACV.2018.00162"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"918","DOI":"10.1016\/j.nicl.2017.12.022","article-title":"White matter hyperintensity and stroke lesion segmentation and differentiation using convolutional neural networks","volume":"17","author":"Guerrero","year":"2018","journal-title":"NeuroImage Clin."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5\u20139). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Hu, P., and Ramanan, D. (2017, January 21\u201326). Finding tiny faces. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.166"},{"key":"ref_45","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv."},{"key":"ref_46","unstructured":"Ioffe, S., and Szegedy, C. (2015, January 6\u201311). Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the International Conference on Machine Learning (ICML), Lille, France."},{"key":"ref_47","unstructured":"Iglovikov, V., and Shvets, A. (2018). TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation. arXiv."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"2481","DOI":"10.1109\/TPAMI.2016.2644615","article-title":"SegNet: A deep convolutional encoder-decoder architecture for image segmentation","volume":"39","author":"Badrinarayanan","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_49","unstructured":"Denker, J.S., and Lecun, Y. (1991, January 26\u201329). Transforming neural-net output levels to probability distributions. Proceedings of the Advances in Neural Information Processing Systems (NIPS), Denver, Colorado."},{"key":"ref_50","unstructured":"Gal, Y., and Ghahramani, Z. (2015). Bayesian convolutional neural networks with Bernoulli approximate variational inference. arXiv."},{"key":"ref_51","unstructured":"Hoaglin, D.C., Mosteller, F., and Tukey, J.W. (2000). Understanding Robust and Exploratory Data Analysis, Wiley-Interscience. Number Sirsi i9780471384915."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"234","DOI":"10.1016\/j.artint.2007.05.008","article-title":"Conjunctive and disjunctive combination of belief functions induced by nondistinct bodies of evidence","volume":"172","year":"2008","journal-title":"Artif. Intell."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1016\/j.ijar.2018.10.008","article-title":"Evidential split-and-merge: Application to object-based image analysis","volume":"103","author":"Lachaize","year":"2018","journal-title":"Int. J. Approx. Reason."},{"key":"ref_54","first-page":"61","article-title":"Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods","volume":"10","author":"Platt","year":"1999","journal-title":"Adv. Large Margin Classif."},{"key":"ref_55","unstructured":"Vandoni, J., Aldea, E., and Le H\u00e9garat-Mascle, S. (September, January 29). An evidential framework for pedestrian detection in high-density crowds. Proceedings of the 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Lecce, Italy."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Vandoni, J., Le H\u00e9garat-Mascle, S., and Aldea, E. (2018, January 10\u201313). Belief Function Definition for Ensemble Methods-Application to Pedestrian Detection in Dense Crowds. Proceedings of the 21st International Conference on Information Fusion (FUSION), Cambridge, UK.","DOI":"10.23919\/ICIF.2018.8455313"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/0031-3203(95)00067-4","article-title":"A comparative study of texture measures with classification based on featured distributions","volume":"29","author":"Ojala","year":"1996","journal-title":"Pattern Recognit."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"815","DOI":"10.1109\/TPAMI.2009.77","article-title":"Daisy: An efficient dense descriptor applied to wide-baseline stereo","volume":"32","author":"Tola","year":"2010","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Li, M., Bao, S., Dong, W., Wang, Y., and Su, Z. (2013, January 15\u201318). Head-shoulder based gender recognition. Proceedings of the International Conference on Image Processing (ICIP), Melbourne, VIC, Australia.","DOI":"10.1109\/ICIP.2013.6738567"},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1016\/0020-0255(87)90007-7","article-title":"On the Dempster-Shafer framework and new combination rules","volume":"41","author":"Yager","year":"1987","journal-title":"Inf. Sci."},{"key":"ref_61","unstructured":"Aldea, E., and Kiyani, K.H. (2014, January 1\u20135). Hybrid focal stereo networks for pattern analysis in homogeneous scenes. Proceedings of the Asian Conference on Computer Vision (ACCV), Singapore."},{"key":"ref_62","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2015, January 13\u201316). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.123"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/21\/4664\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:29:45Z","timestamp":1760189385000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/21\/4664"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,27]]},"references-count":63,"journal-issue":{"issue":"21","published-online":{"date-parts":[[2019,11]]}},"alternative-id":["s19214664"],"URL":"https:\/\/doi.org\/10.3390\/s19214664","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2019,10,27]]}}}