{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T13:40:31Z","timestamp":1783345231100,"version":"3.54.6"},"reference-count":116,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2020,10,4]],"date-time":"2020-10-04T00:00:00Z","timestamp":1601769600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Intelligent Manufacturing Special Fund of Tianjin","award":["20191012"],"award-info":[{"award-number":["20191012"]}]},{"name":"Innovation Research Project of Tianjin University","award":["2020XZC-0042"],"award-info":[{"award-number":["2020XZC-0042"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61872263"],"award-info":[{"award-number":["61872263"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2020,10,31]]},"abstract":"<jats:p>\n            Deep neural network (DNN) has become increasingly popular and DNN testing is very critical to guarantee the correctness of DNN, i.e., the accuracy of DNN in this work. However, DNN testing suffers from a serious efficiency problem, i.e., it is costly to label each test input to know the DNN accuracy for the testing set, since labeling each test input involves multiple persons (even with domain-specific knowledge) in a manual way and the testing set is large-scale. To relieve this problem, we propose a novel and practical approach, called PACE (which is short for\n            <jats:bold>P<\/jats:bold>\n            ractical\n            <jats:bold>AC<\/jats:bold>\n            curacy\n            <jats:bold>E<\/jats:bold>\n            stimation), which selects a small set of test inputs that can precisely estimate the accuracy of the whole testing set. In this way, the labeling costs can be largely reduced by just labeling this small set of selected test inputs. Besides achieving a precise accuracy estimation, to make PACE more practical it is also required that it is interpretable, deterministic, and as efficient as possible. Therefore, PACE first incorporates clustering to interpretably divide test inputs with different testing capabilities (i.e., testing different functionalities of a DNN model) into different groups. Then, PACE utilizes the MMD-critic algorithm, a state-of-the-art example-based explanation algorithm, to select prototypes (i.e., the most representative test inputs) from each group, according to the group sizes, which can reduce the impact of noise due to clustering. Meanwhile, PACE also borrows the idea of adaptive random testing to select test inputs from the minority space (i.e., the test inputs that are not clustered into any group) to achieve great diversity under the required number of test inputs. The two parallel selection processes (i.e., selection from both groups and the minority space) compose the final small set of selected test inputs. We conducted an extensive study to evaluate the performance of PACE based on a comprehensive benchmark (i.e., 24 pairs of DNN models and testing sets) by considering different types of models (i.e., classification and regression models, high-accuracy and low-accuracy models, and CNN and RNN models) and different types of test inputs (i.e., original, mutated, and automatically generated test inputs). The results demonstrate that PACE is able to precisely estimate the accuracy of the whole testing set with only 1.181%\u223c2.302% deviations, on average, significantly outperforming the state-of-the-art approaches.\n          <\/jats:p>","DOI":"10.1145\/3394112","type":"journal-article","created":{"date-parts":[[2020,7,7]],"date-time":"2020-07-07T12:39:27Z","timestamp":1594125567000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":87,"title":["Practical Accuracy Estimation for Efficient Deep Neural Network Testing"],"prefix":"10.1145","volume":"29","author":[{"given":"Junjie","family":"Chen","sequence":"first","affiliation":[{"name":"College of Intelligence and Computing, Tianjin University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhuo","family":"Wu","sequence":"additional","affiliation":[{"name":"Tianjin International Engineering Institute, Tianjin University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zan","family":"Wang","sequence":"additional","affiliation":[{"name":"College of Intelligence and Computing, Tianjin University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hanmo","family":"You","sequence":"additional","affiliation":[{"name":"College of Intelligence and Computing, Tianjin University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lingming","family":"Zhang","sequence":"additional","affiliation":[{"name":"Department of Computer Science, The University of Texas at Dallas, Richardson, TX, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ming","family":"Yan","sequence":"additional","affiliation":[{"name":"College of Intelligence and Computing, Tianjin University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,10,4]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"FastICA. 2019. Retrieved from https:\/\/scikit-learn.org\/stable\/modules\/classes.html#module-sklearn.decomposition.  FastICA. 2019. Retrieved from https:\/\/scikit-learn.org\/stable\/modules\/classes.html#module-sklearn.decomposition."},{"key":"e_1_2_1_2_1","unstructured":"HDBScan. 2019. Retrieved from https:\/\/pypi.org\/project\/hdbscan\/.  HDBScan. 2019. Retrieved from https:\/\/pypi.org\/project\/hdbscan\/."},{"key":"e_1_2_1_3_1","unstructured":"Keras. 2019. Retrieved from https:\/\/keras.io\/.  Keras. 2019. Retrieved from https:\/\/keras.io\/."},{"key":"e_1_2_1_4_1","unstructured":"Github. 2019. MMD-critic algorithms. Retrieved from https:\/\/github.com\/BeenKim\/MMD-critic.  Github. 2019. MMD-critic algorithms. Retrieved from https:\/\/github.com\/BeenKim\/MMD-critic."},{"key":"e_1_2_1_5_1","unstructured":"scikit. 2019. scikit-learn. Retrieved from https:\/\/scikit-learn.org\/stable\/.  scikit. 2019. scikit-learn. Retrieved from https:\/\/scikit-learn.org\/stable\/."},{"key":"e_1_2_1_6_1","unstructured":"Tensorflow. 2019. Retrieved from https:\/\/www.tensorflow.org\/.  Tensorflow. 2019. Retrieved from https:\/\/www.tensorflow.org\/."},{"key":"e_1_2_1_7_1","volume-title":"Introduction to Software Testing","author":"Ammann Paul"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.3390\/computers5010001"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the 6th International Conference on Learning Representations.","author":"Brendel Wieland","year":"2018"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2017.49"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.312"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183440.3183456"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.70"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2016.19"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2017.45"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970366"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00020"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00042"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 32nd European Conference on Object-Oriented Programming. 6:1--6:27","author":"Chen Junjie","year":"2018"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236053"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2889771"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2009.02.022"},{"key":"e_1_2_1_23_1","first-page":"5","article-title":"A new heuristic for test suite reduction. Info","volume":"40","author":"Chen Tsong Yueh","year":"1998","journal-title":"Softw. Technol."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00055"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2012.07.029"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338954"},{"key":"e_1_2_1_27_1","volume-title":"Deepcruiser: Automated guided testing for stateful deep learning systems. arXiv preprint arXiv:1812.05339","author":"Du Xiaoning","year":"2018"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.988497"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.265"},{"key":"e_1_2_1_30_1","volume-title":"Alper Sen, and Alper Cakan.","author":"Gerasimou Simos","year":"2020"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2771783.2771784"},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the 3rd International Conference on Learning Representations.","author":"Goodfellow Ian J.","year":"2015"},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the 3rd International Conference on Learning Representations.","author":"Goodfellow Ian J.","year":"2015"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2610384.2610416"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264835"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/152388.152391"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.2307\/2346830"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the 6th International Conference on Learning Representations.","author":"He Warren","year":"2018"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884791"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00126"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338955"},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the 24th IEEE\/ACM International Conference on Automated Software Engineering. 233--244","author":"Jiang Bo"},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the IEEE\/AIAA 35th Digital Avionics Systems Conference. IEEE, 1--10","author":"Julian Kyle D."},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the 30th International Conference on Neural Information Processing Systems. 2288--2296","author":"Kim Been","year":"2016"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_2_1_46_1","volume-title":"Proceedings of the 5th International Conference on Learning Representations Workshop Track.","author":"Kurakin Alexey","year":"2017"},{"key":"e_1_2_1_47_1","volume-title":"29th AAAI Conference on Artificial Intelligence.","author":"Lai Siwei","year":"2015"},{"key":"e_1_2_1_48_1","volume-title":"Advances in Neural Information Processing Systems","author":"Lee Daniel D."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2950290.2950361"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.5555\/3155562.3155684"},{"key":"e_1_2_1_51_1","volume-title":"Gale","author":"Lewis David D.","year":"1994"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330574"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2007.38"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-NIER.2019.00031"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338930"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2018\/585"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2016.12.038"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1016\/bs.adcom.2018.10.001"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2019.8668044"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2018.00021"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236082"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2019.23415"},{"key":"e_1_2_1_64_1","volume-title":"Test selection for deep learning systems. CoRR abs\/1904.13195","author":"Ma Wei","year":"2019"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2931054"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.21105\/joss.00205"},{"key":"e_1_2_1_67_1","volume-title":"Proceedings of the 21st International Conference on Machine Learning. ACM, 74","author":"Melville Prem"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1090\/dimacs\/015\/09"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMp1606181"},{"key":"e_1_2_1_70_1","volume-title":"Proceedings of the 36th International Conference on Machine Learning. 4901--4911","author":"Odena Augustus"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNN.2006.880980"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1109\/MILCOM.2016.7795300"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_2_1_74_1","volume-title":"Proceedings of the 36th International Conference on Machine Learning. 5231--5240","author":"Qin Yao","year":"2019"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00116251"},{"key":"e_1_2_1_76_1","volume-title":"Proceedings of the 19th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications. 432--448","author":"Ren Xiaoxia"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.536955"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1145\/248233.248262"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.256"},{"key":"e_1_2_1_80_1","volume-title":"Proceedings IEEE International Conference on Software Maintenance. IEEE, 179--188","author":"Rothermel Gregg"},{"key":"e_1_2_1_81_1","volume-title":"Proceedings of the 37th IEEE\/ACM International Conference on Software Engineering. 268--279","author":"Saha Ripon K."},{"key":"e_1_2_1_82_1","volume-title":"Towards crafting text adversarial samples. arXiv preprint arXiv:1707.02812","author":"Samanta Suranjana","year":"2017"},{"key":"e_1_2_1_83_1","volume-title":"Atkeson","author":"Schaal Stefan","year":"1998"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2019.23288"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1145\/3068335"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2016.01.018"},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-NIER.2019.00030"},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.5555\/1613715.1613855"},{"key":"e_1_2_1_89_1","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635921"},{"key":"e_1_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786878"},{"key":"e_1_2_1_91_1","unstructured":"Qingkai Shi Jun Wan Yang Feng Chunrong Fang and Zhenyu Chen. 2019. DeepGini: Prioritizing massive tests to reduce labeling cost. arXiv preprint arXiv:1903.00661.  Qingkai Shi Jun Wan Yang Feng Chunrong Fang and Zhenyu Chen. 2019. DeepGini: Prioritizing massive tests to reduce labeling cost. arXiv preprint arXiv:1903.00661."},{"key":"e_1_2_1_92_1","volume-title":"Proceedings of the 3rd International Conference on Learning Representations.","author":"Simonyan Karen","year":"2015"},{"key":"e_1_2_1_93_1","volume-title":"Advances in Neural Information Processing Systems","author":"Sun Yi","year":"1988"},{"key":"e_1_2_1_94_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238172"},{"key":"e_1_2_1_95_1","volume-title":"Proceedings of the 2nd International Conference on Learning Representations.","author":"Szegedy Christian","year":"2014"},{"key":"e_1_2_1_96_1","first-page":"101","article-title":"A critique and improvement of the CL common language effect size statistics of McGraw and Wong","volume":"25","author":"Vargha Andr\u00e1s","year":"2000","journal-title":"J. Edu. Behav. Stat."},{"key":"e_1_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00126"},{"key":"e_1_2_1_98_1","unstructured":"Zan Wang Ming Yan Shuang Liu Junjie Chen Dongdi Zhang Zhuo Wu and Xiang Chen. 2020. A survey on testing of deep neural networks. J. Softw. (2020). to appear.  Zan Wang Ming Yan Shuang Liu Junjie Chen Dongdi Zhang Zhuo Wu and Xiang Chen. 2020. A survey on testing of deep neural networks. J. Softw. (2020). to appear."},{"key":"e_1_2_1_99_1","unstructured":"Pete Warden. 2017. Speech commands: A public dataset for single-word speech recognition. Retrieved from http:\/\/download.tensorflow. org\/data\/speech_commands_v0.  Pete Warden. 2017. Speech commands: A public dataset for single-word speech recognition. Retrieved from http:\/\/download.tensorflow. org\/data\/speech_commands_v0."},{"key":"e_1_2_1_100_1","volume-title":"Breakthroughs in Statistics","author":"Wilcoxon Frank"},{"key":"e_1_2_1_101_1","doi-asserted-by":"publisher","DOI":"10.1016\/0169-7439(87)80084-9"},{"key":"e_1_2_1_102_1","volume-title":"Proceedings of the 6th International Conference on Learning Representations.","author":"Xiao Chaowei","year":"2018"},{"key":"e_1_2_1_103_1","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330579"},{"key":"e_1_2_1_104_1","doi-asserted-by":"publisher","DOI":"10.1002\/stv.430"},{"key":"e_1_2_1_105_1","unstructured":"Zhe Yu Jeffrey C. Carver Gregg Rothermel and Tim Menzies. 2019. Searching for better test case prioritization schemes: A case study of AI-assisted systematic literature review. arXiv preprint arXiv:1909.07249.  Zhe Yu Jeffrey C. Carver Gregg Rothermel and Tim Menzies. 2019. Searching for better test case prioritization schemes: A case study of AI-assisted systematic literature review. arXiv preprint arXiv:1909.07249."},{"key":"e_1_2_1_106_1","volume-title":"Proceedings of the 27th USENIX Security Symposium. 49--64","author":"Yuan Xuejing"},{"key":"e_1_2_1_107_1","volume-title":"Machine learning testing: Survey, landscapes and horizons","author":"Zhang Jie M.","year":"2020"},{"key":"e_1_2_1_108_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180198"},{"key":"e_1_2_1_109_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2011.6080769"},{"key":"e_1_2_1_110_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2011.26"},{"key":"e_1_2_1_111_1","unstructured":"Long Zhang Xuechao Sun Yong Li and Zhenyu Zhang. 2019. A noise-sensitivity-analysis-based test prioritization technique for deep neural networks. arXiv preprint arXiv:1901.00054.  Long Zhang Xuechao Sun Yong Li and Zhenyu Zhang. 2019. A noise-sensitivity-analysis-based test prioritization technique for deep neural networks. arXiv preprint arXiv:1901.00054."},{"key":"e_1_2_1_112_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238187"},{"key":"e_1_2_1_113_1","doi-asserted-by":"publisher","DOI":"10.1109\/AITest.2019.000-5"},{"key":"e_1_2_1_114_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338931"},{"key":"e_1_2_1_115_1","doi-asserted-by":"publisher","DOI":"10.1109\/QRS.2019.00056"},{"key":"e_1_2_1_116_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380422"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3394112","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3394112","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:47:12Z","timestamp":1750193232000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3394112"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,4]]},"references-count":116,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,10,31]]}},"alternative-id":["10.1145\/3394112"],"URL":"https:\/\/doi.org\/10.1145\/3394112","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,10,4]]},"assertion":[{"value":"2019-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-04-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-10-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}