{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,5]],"date-time":"2026-07-05T21:51:50Z","timestamp":1783288310262,"version":"3.54.6"},"reference-count":61,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2021,9,28]],"date-time":"2021-09-28T00:00:00Z","timestamp":1632787200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100020951","name":"Fonds de Recherche du Qu\u00e9bec","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100020951","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100019217","name":"Institut de VAlorisation des DOnn\u00e9es","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100019217","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2022,1,31]]},"abstract":"<jats:p>\n            Nowadays, we are witnessing an increasing demand in both corporates and academia for exploiting\n            <jats:bold>Deep Learning<\/jats:bold>\n            (\n            <jats:bold>DL<\/jats:bold>\n            ) to solve complex real-world problems. A DL program encodes the network structure of a desirable DL model and the process by which the model learns from the training dataset. Like any software, a DL program can be faulty, which implies substantial challenges of software quality assurance, especially in safety-critical domains. It is therefore crucial to equip DL development teams with efficient fault detection techniques and tools. In this article, we propose\n            <jats:italic>NeuraLint<\/jats:italic>\n            , a model-based fault detection approach for DL programs, using meta-modeling and graph transformations. First, we design a meta-model for DL programs that includes their base skeleton and fundamental properties. Then, we construct a graph-based verification process that covers 23 rules defined on top of the meta-model and implemented as graph transformations to detect faults and design inefficiencies in the generated models (i.e., instances of the meta-model). First, the proposed approach is evaluated by finding faults and design inefficiencies in 28 synthesized examples built from common problems reported in the literature. Then\n            <jats:italic>NeuraLint<\/jats:italic>\n            successfully finds 64 faults and design inefficiencies in 34 real-world DL programs extracted from Stack Overflow posts and GitHub repositories. The results show that\n            <jats:italic>NeuraLint<\/jats:italic>\n            effectively detects faults and design issues in both synthesized and real-world examples with a recall of 70.5% and a precision of 100%. Although the proposed meta-model is designed for feedforward neural networks, it can be extended to support other neural network architectures such as recurrent neural networks. Researchers can also expand our set of verification rules to cover more types of issues in DL programs.\n          <\/jats:p>","DOI":"10.1145\/3470006","type":"journal-article","created":{"date-parts":[[2021,9,28]],"date-time":"2021-09-28T20:49:24Z","timestamp":1632862164000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":28,"title":["Automatic Fault Detection for Deep Learning Programs Using Graph Transformations"],"prefix":"10.1145","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0440-6839","authenticated-orcid":false,"given":"Amin","family":"Nikanjam","sequence":"first","affiliation":[{"name":"K. N. Toosi University of Technology, Iran and SWAT Lab., Polytechnique Montreal, Montr\u00e9al (Qu\u00e9bec) Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Houssem Ben","family":"Braiek","sequence":"additional","affiliation":[{"name":"SWAT Lab., Polytechnique Montreal, Montr\u00e9al (Qu\u00e9bec) Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohammad Mehdi","family":"Morovati","sequence":"additional","affiliation":[{"name":"SWAT Lab., Polytechnique Montreal, Montr\u00e9al (Qu\u00e9bec) Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Foutse","family":"Khomh","sequence":"additional","affiliation":[{"name":"SWAT Lab., Polytechnique Montreal, Montr\u00e9al (Qu\u00e9bec) Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,9,28]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2016. Retrieved February 2 2021 from https:\/\/github.com\/katyprogrammer\/regularization-experiment\/commit\/b93dd636.  2016. Retrieved February 2 2021 from https:\/\/github.com\/katyprogrammer\/regularization-experiment\/commit\/b93dd636."},{"key":"e_1_2_1_2_1","unstructured":"2017. Retrieved February 5 2021 from https:\/\/github.com\/dishen12\/keras_frcnn\/commit\/38413c6.  2017. Retrieved February 5 2021 from https:\/\/github.com\/dishen12\/keras_frcnn\/commit\/38413c6."},{"key":"e_1_2_1_3_1","unstructured":"2017. Retrieved February 5 2021 from https:\/\/github.com\/yumatsuoka\/comp_DNNfw\/commit\/30e0973.  2017. Retrieved February 5 2021 from https:\/\/github.com\/yumatsuoka\/comp_DNNfw\/commit\/30e0973."},{"key":"e_1_2_1_4_1","unstructured":"2018. Retrieved February 5 2021 from https:\/\/github.com\/taashi-s\/UNet_Keras\/commit\/b1b6d93.  2018. Retrieved February 5 2021 from https:\/\/github.com\/taashi-s\/UNet_Keras\/commit\/b1b6d93."},{"key":"e_1_2_1_5_1","unstructured":"2018. Retrieved February 5 2021 from https:\/\/github.com\/keras-team\/keras-applications\/commit\/05ff470.  2018. Retrieved February 5 2021 from https:\/\/github.com\/keras-team\/keras-applications\/commit\/05ff470."},{"key":"e_1_2_1_6_1","unstructured":"2018. Retrieved February 5 2021 from https:\/\/github.com\/mateusz93\/Car-recognition\/commit\/94b36ea.  2018. Retrieved February 5 2021 from https:\/\/github.com\/mateusz93\/Car-recognition\/commit\/94b36ea."},{"key":"e_1_2_1_7_1","unstructured":"2018. Retrieved February 5 2021 from https:\/\/github.com\/tf-encrypted\/tf-encrypted\/issues\/248 (network B).  2018. Retrieved February 5 2021 from https:\/\/github.com\/tf-encrypted\/tf-encrypted\/issues\/248 (network B)."},{"key":"e_1_2_1_8_1","unstructured":"2018. Retrieved February 5 2021 from https:\/\/github.com\/tf-encrypted\/tf-encrypted\/issues\/248 (network C).  2018. Retrieved February 5 2021 from https:\/\/github.com\/tf-encrypted\/tf-encrypted\/issues\/248 (network C)."},{"key":"e_1_2_1_9_1","unstructured":"2020. Keras-based VGG16. Retrieved June 29 2020 https:\/\/github.com\/keras-team\/keras-applications\/blob\/master\/keras_applications\/vgg16.py.  2020. Keras-based VGG16. Retrieved June 29 2020 https:\/\/github.com\/keras-team\/keras-applications\/blob\/master\/keras_applications\/vgg16.py."},{"key":"e_1_2_1_10_1","unstructured":"2020. Tensorflow-based LeNet. Retrieved June 29 2020 https:\/\/github.com\/tensorflow\/models\/blob\/master\/research\/slim\/nets\/lenet.py.  2020. Tensorflow-based LeNet. Retrieved June 29 2020 https:\/\/github.com\/tensorflow\/models\/blob\/master\/research\/slim\/nets\/lenet.py."},{"key":"e_1_2_1_11_1","volume-title":"Retrieved","year":"2021","unstructured":"2021. Replication Package and Source Code of NeuraLint . Retrieved February 15, 2021 https:\/\/github.com\/neuralint\/neuralint. 2021. Replication Package and Source Code of NeuraLint. Retrieved February 15, 2021 https:\/\/github.com\/neuralint\/neuralint."},{"key":"e_1_2_1_12_1","volume-title":"International Conference on Learning Representations. Retrieved","author":"Allamanis Miltiadis","year":"2018","unstructured":"Miltiadis Allamanis , Marc Brockschmidt , and Mahmoud Khademi . 2018 . Learning to represent programs with graphs . In International Conference on Learning Representations. Retrieved June 29, 2020 https:\/\/openreview.net\/forum?id=BJOFETxR-. Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2018. Learning to represent programs with graphs. In International Conference on Learning Representations. Retrieved June 29, 2020 https:\/\/openreview.net\/forum?id=BJOFETxR-."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2019.00078"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/QRS.2019.00059"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2020.110542"},{"key":"e_1_2_1_16_1","unstructured":"Alfredo Canziani Adam Paszke and Eugenio Culurciello. 2016. An analysis of deep neural network models for practical applications. arXiv:1605.07678. https:\/\/arxiv.org\/abs\/1605.07678.  Alfredo Canziani Adam Paszke and Eugenio Culurciello. 2016. An analysis of deep neural network models for practical applications. arXiv:1605.07678. https:\/\/arxiv.org\/abs\/1605.07678."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1774088.1774561"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_2_1_19_1","volume-title":"International Conference on Learning Representations. Retrieved","author":"Dinella Elizabeth","year":"2020","unstructured":"Elizabeth Dinella , Hanjun Dai , Ziyang Li , Mayur Naik , Le Song , and Ke Wang . 2020 . Hoppity: Learning graph transformations to detect and fix bugs in programs . In International Conference on Learning Representations. Retrieved June 29, 2020 https:\/\/openreview.net\/forum. Elizabeth Dinella, Hanjun Dai, Ziyang Li, Mayur Naik, Le Song, and Ke Wang. 2020. Hoppity: Learning graph transformations to detect and fix bugs in programs. In International Conference on Learning Representations. Retrieved June 29, 2020 https:\/\/openreview.net\/forum."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213858"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10009-011-0186-x"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 13th International Conference on Artificial Intelligence and Statistics. 249\u2013256","author":"Glorot Xavier","year":"2010","unstructured":"Xavier Glorot and Yoshua Bengio . 2010 . Understanding the difficulty of training deep feedforward neural networks . In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics. 249\u2013256 . Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics. 249\u2013256."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.5555\/3086952"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.668"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/MODELS.2019.00014"},{"key":"e_1_2_1_26_1","unstructured":"Seyyed Hossein Hasanpour Mohammad Rouhani Mohsen Fayyaz Mohammad Sabokrou and Ehsan Adeli. 2018. Towards principled design of deep convolutional networks: Introducing SimpNet. arXiv:1802.06205. https:\/\/arxiv.org\/abs\/1802.06205.  Seyyed Hossein Hasanpour Mohammad Rouhani Mohsen Fayyaz Mohammad Sabokrou and Ehsan Adeli. 2018. Towards principled design of deep convolutional networks: Introducing SimpNet. arXiv:1802.06205. https:\/\/arxiv.org\/abs\/1802.06205."},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of Machine Learning Research, 2672\u20132680","author":"Hayou Soufiane","year":"2019","unstructured":"Soufiane Hayou , Arnaud Doucet , and Judith Rousseau . 2019 . On the impact of the activation function on deep neural networks training. In International Conference on Machine Learning . Proceedings of Machine Learning Research, 2672\u20132680 . Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. 2019. On the impact of the activation function on deep neural networks training. In International Conference on Machine Learning. Proceedings of Machine Learning Research, 2672\u20132680."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7299173"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.entcs.2005.12.018"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380395"},{"key":"e_1_2_1_32_1","unstructured":"Forrest Iandola Matt Moskewicz Sergey Karayev Ross Girshick Trevor Darrell and Kurt Keutzer. 2014. DenseNet: Implementing efficient ConvNet descriptor pyramids. arXiv:1404.1869. https:\/\/arxiv.org\/abs\/1404.1869.  Forrest Iandola Matt Moskewicz Sergey Karayev Ross Girshick Trevor Darrell and Kurt Keutzer. 2014. DenseNet: Implementing efficient ConvNet descriptor pyramids. arXiv:1404.1869. https:\/\/arxiv.org\/abs\/1404.1869."},{"key":"e_1_2_1_33_1","unstructured":"Forrest N. Iandola Song Han Matthew W. Moskewicz Khalid Ashraf William J. Dally and Kurt Keutzer. 2016. SqueezeNet: AlexNet-level accuracy with 50 fewer parameters and <0.5 MB model size. arXiv:1602.07360. https:\/\/arxiv.org\/abs\/1602.07360.  Forrest N. Iandola Song Han Matthew W. Moskewicz Khalid Ashraf William J. Dally and Kurt Keutzer. 2016. SqueezeNet: AlexNet-level accuracy with 50 fewer parameters and <0.5 MB model size. arXiv:1602.07360. https:\/\/arxiv.org\/abs\/1602.07360."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.5555\/3045118.3045167"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338955"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380378"},{"key":"e_1_2_1_37_1","unstructured":"Roshni G. Iyer Yizhou Sun Wei Wang and Justin Gottschlich. 2020. Software language comprehension using a program-derived semantic graph. arXiv:2004.00768. https:\/\/arxiv.org\/abs\/2004.00768.  Roshni G. Iyer Yizhou Sun Wei Wang and Justin Gottschlich. 2020. Software language comprehension using a program-derived semantic graph. arXiv:2004.00768. https:\/\/arxiv.org\/abs\/2004.00768."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.5555\/2999134.2999257"},{"key":"e_1_2_1_39_1","volume-title":"Retrieved","author":"LeCun Yann","year":"1998","unstructured":"Yann LeCun . 1998 . The MNIST database of handwritten digits . Retrieved June 29, 2020 http:\/\/yann.lecun.com\/exdb\/mnist\/. Yann LeCun. 1998. The MNIST database of handwritten digits. Retrieved June 29, 2020 http:\/\/yann.lecun.com\/exdb\/mnist\/."},{"key":"e_1_2_1_40_1","volume-title":"Retrieved","author":"\u00a0al Yann","year":"2015","unstructured":"Yann LeCun et \u00a0al . 2015 . LeNet-5, convolutional neural networks . Retrieved June 29, 2020 http:\/\/yann.lecun.com\/exdb\/lenet. Yann LeCun et\u00a0al. 2015. LeNet-5, convolutional neural networks. Retrieved June 29, 2020 http:\/\/yann.lecun.com\/exdb\/lenet."},{"key":"e_1_2_1_41_1","volume-title":"Deep learning. Nature 521, 7553","author":"LeCun Yann","year":"2015","unstructured":"Yann LeCun , Yoshua Bengio , and Geoffrey Hinton . 2015. Deep learning. Nature 521, 7553 ( 2015 ), 436\u2013444. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521, 7553 (2015), 436\u2013444."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00279"},{"key":"e_1_2_1_44_1","volume-title":"European Conference on Computer Vision. Springer, 740\u2013755","author":"Lin Tsung-Yi","unstructured":"Tsung-Yi Lin , Michael Maire , Serge Belongie , James Hays , Pietro Perona , Deva Ramanan , Piotr Doll\u00e1r , and C. Lawrence Zitnick . 2014. Microsoft COCO: Common objects in context . In European Conference on Computer Vision. Springer, 740\u2013755 . Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll\u00e1r, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In European Conference on Computer Vision. Springer, 740\u2013755."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2017.05.007"},{"key":"e_1_2_1_46_1","unstructured":"Chigozie Nwankpa Winifred Ijomah Anthony Gachagan and Stephen Marshall. 2018. Activation functions: Comparison of trends in practice and research for deep learning. arXiv:1811.03378. https:\/\/arxiv.org\/abs\/1811.03378.  Chigozie Nwankpa Winifred Ijomah Anthony Gachagan and Stephen Marshall. 2018. Activation functions: Comparison of trends in practice and research for deep learning. arXiv:1811.03378. https:\/\/arxiv.org\/abs\/1811.03378."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_2_1_48_1","volume-title":"Retrieved","author":"Perez Carlos E.","year":"2016","unstructured":"Carlos E. Perez . 2016 . The Meta Model and Meta Meta-Model of Deep Learning . Retrieved June 29, 2020 https:\/\/medium.com\/intuitionmachine\/the-meta-model-and-meta-meta-model-of-deep-learning-10062f0bf74c. Carlos E. Perez. 2016. The Meta Model and Meta Meta-Model of Deep Learning. Retrieved June 29, 2020 https:\/\/medium.com\/intuitionmachine\/the-meta-model-and-meta-meta-model-of-deep-learning-10062f0bf74c."},{"key":"e_1_2_1_49_1","volume-title":"International Workshop on Applications of Graph Transformations with Industrial Relevance. Springer, 479\u2013485","author":"Rensink Arend","year":"2003","unstructured":"Arend Rensink . 2003 . The GROOVE simulator: A tool for state space generation . In International Workshop on Applications of Graph Transformations with Industrial Relevance. Springer, 479\u2013485 . Arend Rensink. 2003. The GROOVE simulator: A tool for state space generation. In International Workshop on Applications of Graph Transformations with Industrial Relevance. Springer, 479\u2013485."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.5555\/1886436.1886447"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.5555\/3305890.3305996"},{"key":"e_1_2_1_52_1","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556. https:\/\/arxiv.org\/abs\/1409.1556.  Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556. https:\/\/arxiv.org\/abs\/1409.1556."},{"key":"e_1_2_1_53_1","volume-title":"Smith and Nicholay Topin","author":"Leslie","year":"2016","unstructured":"Leslie N. Smith and Nicholay Topin . 2016 . Deep convolutional neural network design patterns. arXiv:1611.00847. https:\/\/arxiv.org\/abs\/1611.00847. Leslie N. Smith and Nicholay Topin. 2016. Deep convolutional neural network design patterns. arXiv:1611.00847. https:\/\/arxiv.org\/abs\/1611.00847."},{"key":"e_1_2_1_54_1","volume-title":"Retrieved","author":"Soley Richard Mark","year":"2013","unstructured":"Richard Mark Soley . 2013 . How to Deliver Resilient, Secure, Efficient, and Easily Changed IT Systems in Line with CISQ Recommendations . Retrieved June 29, 2020 https:\/\/www.omg.org\/news\/whitepapers\/CISQ_compliant_IT_Systemsv.4-3.pdf. Richard Mark Soley. 2013. How to Deliver Resilient, Secure, Efficient, and Easily Changed IT Systems in Line with CISQ Recommendations. Retrieved June 29, 2020 https:\/\/www.omg.org\/news\/whitepapers\/CISQ_compliant_IT_Systemsv.4-3.pdf."},{"key":"e_1_2_1_55_1","unstructured":"Jost Tobias Springenberg Alexey Dosovitskiy Thomas Brox and Martin Riedmiller. 2014. Striving for simplicity: The all convolutional net. arXiv:1412.6806. https:\/\/arxiv.org\/abs\/1412.6806.  Jost Tobias Springenberg Alexey Dosovitskiy Thomas Brox and Martin Riedmiller. 2014. Striving for simplicity: The all convolutional net. arXiv:1412.6806. https:\/\/arxiv.org\/abs\/1412.6806."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.5555\/2627435.2670313"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.308"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_2_1_59_1","volume-title":"Retrieved","author":"Zhang Chengwei","year":"2018","unstructured":"Chengwei Zhang . 2018 . How to Choose Last-Layer Activation and Loss Function . Retrieved June 29, 2020 https:\/\/www.dlology.com\/blog\/how-to-choose-last-layer-activation-and-loss-function\/. Chengwei Zhang. 2018. How to Choose Last-Layer Activation and Loss Function. Retrieved June 29, 2020 https:\/\/www.dlology.com\/blog\/how-to-choose-last-layer-activation-and-loss-function\/."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2019.00020"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213866"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3470006","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3470006","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:30:26Z","timestamp":1750188626000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3470006"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,28]]},"references-count":61,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,1,31]]}},"alternative-id":["10.1145\/3470006"],"URL":"https:\/\/doi.org\/10.1145\/3470006","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,28]]},"assertion":[{"value":"2020-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-09-28","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}