{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T16:00:15Z","timestamp":1781366415751,"version":"3.54.1"},"publisher-location":"New York, NY, USA","reference-count":47,"publisher":"ACM","license":[{"start":{"date-parts":[[2018,7,12]],"date-time":"2018-07-12T00:00:00Z","timestamp":1531353600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2018,7,12]]},"DOI":"10.1145\/3213846.3213858","type":"proceedings-article","created":{"date-parts":[[2018,7,12]],"date-time":"2018-07-12T17:46:44Z","timestamp":1531417604000},"page":"118-128","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":138,"title":["Identifying implementation bugs in machine learning based image classifiers using metamorphic testing"],"prefix":"10.1145","author":[{"given":"Anurag","family":"Dwarakanath","sequence":"first","affiliation":[{"name":"Accenture Labs, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Manish","family":"Ahuja","sequence":"additional","affiliation":[{"name":"Accenture Labs, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samarth","family":"Sikand","sequence":"additional","affiliation":[{"name":"Accenture Labs, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Raghotham M.","family":"Rao","sequence":"additional","affiliation":[{"name":"Accenture Labs, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"R. P. Jagadeesh Chandra","family":"Bose","sequence":"additional","affiliation":[{"name":"Accenture Labs, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Neville","family":"Dubash","sequence":"additional","affiliation":[{"name":"Accenture Labs, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sanjay","family":"Podder","sequence":"additional","affiliation":[{"name":"Accenture Labs, India"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2018,7,12]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"antares1987. 2017. About Deterministic Behaviour of GP U implementation of tensorflow. https:\/\/github.com\/tensorflow\/tensorflow\/issues\/12871. {Online; accessed 15-Jan-2018}.  antares1987. 2017. About Deterministic Behaviour of GP U implementation of tensorflow. https:\/\/github.com\/tensorflow\/tensorflow\/issues\/12871. {Online; accessed 15-Jan-2018}."},{"key":"e_1_3_2_2_2_1","unstructured":"Earl T Barr Mark Harman Phil McMinn Muzammil Shahbaz and Shin Yoo. 2015.  Earl T Barr Mark Harman Phil McMinn Muzammil Shahbaz and Shin Yoo. 2015."},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"crossref","unstructured":"The oracle problem in software testing: A survey. IEEE transactions on software engineering 41 5 (2015) 507\u2013525.  The oracle problem in software testing: A survey. IEEE transactions on software engineering 41 5 (2015) 507\u2013525.","DOI":"10.1109\/TSE.2014.2372785"},{"key":"e_1_3_2_2_4_1","unstructured":"Olga Belitskaya. 2018.  Olga Belitskaya. 2018."},{"key":"e_1_3_2_2_5_1","unstructured":"Handwritten Letters 2. https:\/\/www.kaggle.com\/ olgabelitskaya\/handwrittenletters- 2. {Online; accessed 12-Jan-2018}.  Handwritten Letters 2. https:\/\/www.kaggle.com\/ olgabelitskaya\/handwrittenletters- 2. {Online; accessed 12-Jan-2018}."},{"key":"e_1_3_2_2_6_1","unstructured":"Chih-Chung Chang and Chih-Jen Lin. 2011.  Chih-Chung Chang and Chih-Jen Lin. 2011."},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1961189.1961199"},{"key":"e_1_3_2_2_8_1","unstructured":"Tsong Y Chen Shing C Cheung and Shiu Ming Yiu. 1998.  Tsong Y Chen Shing C Cheung and Shiu Ming Yiu. 1998."},{"key":"e_1_3_2_2_9_1","unstructured":"Anna Choromanska Mikael Henaff Michael Mathieu G\u00e9rard Ben Arous and Yann LeCun. 2015. The loss surfaces of multilayer networks. In Artificial Intelligence and Statistics. 192\u2013204.  Anna Choromanska Mikael Henaff Michael Mathieu G\u00e9rard Ben Arous and Yann LeCun. 2015. The loss surfaces of multilayer networks. In Artificial Intelligence and Statistics. 192\u2013204."},{"key":"e_1_3_2_2_10_1","unstructured":"Yann N Dauphin Razvan Pascanu Caglar Gulcehre Kyunghyun Cho Surya Ganguli and Yoshua Bengio. 2014. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In Advances in neural information processing systems. 2933\u20132941.   Yann N Dauphin Razvan Pascanu Caglar Gulcehre Kyunghyun Cho Surya Ganguli and Yoshua Bengio. 2014. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In Advances in neural information processing systems. 2933\u20132941."},{"key":"e_1_3_2_2_11_1","unstructured":"Junhua Ding Xiaojun Kang and Xin-Hua Hu. 2017.  Junhua Ding Xiaojun Kang and Xin-Hua Hu. 2017."},{"key":"e_1_3_2_2_12_1","volume-title":"Proceedings of the 2nd International Workshop on Metamorphic Testing","author":"Validating"},{"key":"e_1_3_2_2_13_1","unstructured":"Anurag Dwarakanath Manish Ahuja Samarth Sikand Raghotham M Rao Jagadeesh Chandra Bose R. P Neville Dubash and Sanjay Podder. 2018.  Anurag Dwarakanath Manish Ahuja Samarth Sikand Raghotham M Rao Jagadeesh Chandra Bose R. P Neville Dubash and Sanjay Podder. 2018."},{"key":"e_1_3_2_2_14_1","unstructured":"Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing: Appendix. https:\/\/github.com\/verml\/VerifyML. {Online; accessed 2-May-2018}.  Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing: Appendix. https:\/\/github.com\/verml\/VerifyML. {Online; accessed 2-May-2018}."},{"key":"e_1_3_2_2_15_1","unstructured":"Python Software Foundation. 2013. MutPy 0.4.0. https:\/\/pypi.python.org\/pypi\/ MutPy\/0.4.0. {Online; accessed 15-Jan-2018}.  Python Software Foundation. 2013. MutPy 0.4.0. https:\/\/pypi.python.org\/pypi\/ MutPy\/0.4.0. {Online; accessed 15-Jan-2018}."},{"key":"e_1_3_2_2_16_1","unstructured":"Ian J. Goodfellow Oriol Vinyals and Andrew M. Saxe. 2014.  Ian J. Goodfellow Oriol Vinyals and Andrew M. Saxe. 2014."},{"key":"e_1_3_2_2_17_1","unstructured":"Qualitatively characterizing neural network optimization problems. (2014). arXiv: 1412.6544 http:\/\/arxiv.org\/abs\/1412.6544  Qualitatively characterizing neural network optimization problems. (2014). arXiv: 1412.6544 http:\/\/arxiv.org\/abs\/1412.6544"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_2_2_19_1","unstructured":"IEEE. {n. d.}. IEEE Std 1012-2016 (Revision of IEEE Std 1012-2012\/ Incorporates IEEE Std 1012-2016\/Cor1-2017) - IEEE Standard for System Software and Hardware Verification and Validation. https:\/\/standards.ieee.org\/findstds\/standard\/ 1012- 2016.html. {Online; accessed 7-Jan-2018}.  IEEE. {n. d.}. IEEE Std 1012-2016 (Revision of IEEE Std 1012-2012\/ Incorporates IEEE Std 1012-2016\/Cor1-2017) - IEEE Standard for System Software and Hardware Verification and Validation. https:\/\/standards.ieee.org\/findstds\/standard\/ 1012- 2016.html. {Online; accessed 7-Jan-2018}."},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.62"},{"key":"e_1_3_2_2_21_1","unstructured":"jkschin. 2017. Non-determinism Docs. https:\/\/github.com\/tensorflow\/tensorflow\/ pull\/10636. {Online; accessed 15-Jan-2018}.  jkschin. 2017. Non-determinism Docs. https:\/\/github.com\/tensorflow\/tensorflow\/ pull\/10636. {Online; accessed 15-Jan-2018}."},{"key":"e_1_3_2_2_22_1","unstructured":"Alex Karpathy. 2016. CS231n: Convolutional Neural Networks for Visual Recognition. http:\/\/cs231n.github.io\/convolutionalnetworks\/. {Online; accessed 26-Jan-2018}.  Alex Karpathy. 2016. CS231n: Convolutional Neural Networks for Visual Recognition. http:\/\/cs231n.github.io\/convolutionalnetworks\/. {Online; accessed 26-Jan-2018}."},{"key":"e_1_3_2_2_23_1","unstructured":"Alex Krizhevsky Vinod Nair and Geoffrey Hinton. {n. d.}. The CIFAR-10 dataset. https:\/\/www.cs.toronto.edu\/~kriz\/cifar.html. {Online; accessed 8-Sept-2017}.  Alex Krizhevsky Vinod Nair and Geoffrey Hinton. {n. d.}. The CIFAR-10 dataset. https:\/\/www.cs.toronto.edu\/~kriz\/cifar.html. {Online; accessed 8-Sept-2017}."},{"key":"e_1_3_2_2_24_1","unstructured":"Alex Krizhevsky Ilya Sutskever and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097\u20131105.   Alex Krizhevsky Ilya Sutskever and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097\u20131105."},{"key":"e_1_3_2_2_25_1","unstructured":"Min Lin Qiang Chen and Shuicheng Yan. 2013.  Min Lin Qiang Chen and Shuicheng Yan. 2013."},{"key":"e_1_3_2_2_26_1","unstructured":"Network in network. arXiv preprint arXiv:1312.4400 (2013).  Network in network. arXiv preprint arXiv:1312.4400 (2013)."},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.46"},{"key":"e_1_3_2_2_30_1","unstructured":"Christian Murphy and Gail E Kaiser. 2008.  Christian Murphy and Gail E Kaiser. 2008."},{"key":"e_1_3_2_2_31_1","unstructured":"Improving the dependability of machine learning applications. (2008).  Improving the dependability of machine learning applications. (2008)."},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2009.19"},{"key":"e_1_3_2_2_33_1","volume-title":"NIPS workshop on deep learning and unsupervised feature learning","volume":"2011","author":"Netzer Yuval","year":"2011"},{"key":"e_1_3_2_2_34_1","unstructured":"Mihai Oltean. 2018. Fruits 360 Dataset. https:\/\/www.kaggle.com\/moltean\/fruits\/ data. {Online; accessed 12-Jan-2018}.  Mihai Oltean. 2018. Fruits 360 Dataset. https:\/\/www.kaggle.com\/moltean\/fruits\/ data. {Online; accessed 12-Jan-2018}."},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3052973.3053009"},{"key":"e_1_3_2_2_36_1","unstructured":"Kexin Pei Yinzhi Cao Junfeng Yang and Suman Jana. 2017.  Kexin Pei Yinzhi Cao Junfeng Yang and Suman Jana. 2017."},{"key":"e_1_3_2_2_37_1","unstructured":"DeepXplore: Automated Whitebox Testing of Deep Learning Systems. arXiv preprint arXiv:1705.06640 (2017).  DeepXplore: Automated Whitebox Testing of Deep Learning Systems. arXiv preprint arXiv:1705.06640 (2017)."},{"key":"e_1_3_2_2_38_1","unstructured":"pixabay. {n. d.}. https:\/\/pixabay.com\/en\/photos\/bird\/. {Online; accessed 5-Sept- 2017}.  pixabay. {n. d.}. https:\/\/pixabay.com\/en\/photos\/bird\/. {Online; accessed 5-Sept- 2017}."},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-015-0816-y"},{"key":"e_1_3_2_2_40_1","unstructured":"Scikit-Learn. {n. d.}. Scikit-learn example: Recognizing hand-written digits. http:\/\/scikitlearn.org\/stable\/auto_examples\/classification\/plot_digits_ classification.html. {Online; accessed 6-Sept-2017}.  Scikit-Learn. {n. d.}. Scikit-learn example: Recognizing hand-written digits. http:\/\/scikitlearn.org\/stable\/auto_examples\/classification\/plot_digits_ classification.html. {Online; accessed 6-Sept-2017}."},{"key":"e_1_3_2_2_41_1","volume-title":"Developing Bug-Free Machine Learning Systems With Formal Mathematics. In International Conference on Machine Learning. 3047\u20133056","author":"Selsam Daniel","year":"2017"},{"key":"e_1_3_2_2_42_1","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).  Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)."},{"key":"e_1_3_2_2_43_1","unstructured":"Siwakorn Srisakaokul Zhengkai Wu Angello Astorga Oreoluwa Alebiosu and Tao Xie. 2018. Multiple-Implementation Testing of Supervised Learning Software. (2018).  Siwakorn Srisakaokul Zhengkai Wu Angello Astorga Oreoluwa Alebiosu and Tao Xie. 2018. Multiple-Implementation Testing of Supervised Learning Software. (2018)."},{"key":"e_1_3_2_2_44_1","unstructured":"TensorFlow. {n. d.}. ResNet in TensorFlow. https:\/\/github.com\/tensorflow\/models\/ tree\/master\/official\/resnet. {Online; accessed 28-Dec-2017}.  TensorFlow. {n. d.}. ResNet in TensorFlow. https:\/\/github.com\/tensorflow\/models\/ tree\/master\/official\/resnet. {Online; accessed 28-Dec-2017}."},{"key":"e_1_3_2_2_45_1","doi-asserted-by":"crossref","unstructured":"Yuchi Tian Kexin Pei Suman Jana and Baishakhi Ray. 2017. DeepTest: Automated testing of deep-neural-network-driven autonomous cars. arXiv preprint arXiv:1708.08559 (2017).  Yuchi Tian Kexin Pei Suman Jana and Baishakhi Ray. 2017. DeepTest: Automated testing of deep-neural-network-driven autonomous cars. arXiv preprint arXiv:1708.08559 (2017).","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/25.4.465"},{"key":"e_1_3_2_2_47_1","unstructured":"Xiaoyuan Xie Joshua Ho Christian Murphy Gail Kaiser Baowen Xu and Tsong Yueh Chen. 2009.  Xiaoyuan Xie Joshua Ho Christian Murphy Gail Kaiser Baowen Xu and Tsong Yueh Chen. 2009."},{"key":"e_1_3_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2009.26"}],"event":{"name":"ISSTA '18: International Symposium on Software Testing and Analysis","location":"Amsterdam Netherlands","acronym":"ISSTA '18","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering","SIGPLAN ACM Special Interest Group on Programming Languages"]},"container-title":["Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3213846.3213858","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3213846.3213858","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:07:41Z","timestamp":1750212461000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3213846.3213858"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,7,12]]},"references-count":47,"alternative-id":["10.1145\/3213846.3213858","10.1145\/3213846"],"URL":"https:\/\/doi.org\/10.1145\/3213846.3213858","relation":{},"subject":[],"published":{"date-parts":[[2018,7,12]]},"assertion":[{"value":"2018-07-12","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}