{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,14]],"date-time":"2026-02-14T21:44:18Z","timestamp":1771105458486,"version":"3.50.1"},"reference-count":69,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2023,9,28]],"date-time":"2023-09-28T00:00:00Z","timestamp":1695859200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"SPATIAL"},{"name":"European Union\u2019s Horizon 2020 research and innovation programme","award":["101021808"],"award-info":[{"award-number":["101021808"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2023,11,30]]},"abstract":"<jats:p>Billions of distributed, heterogeneous, and resource constrained IoT devices deploy on-device machine learning (ML) for private, fast, and offline inference on personal data. On-device ML is highly context dependent and sensitive to user, usage, hardware, and environment attributes. This sensitivity and the propensity toward bias in ML makes it important to study bias in on-device settings. Our study is one of the first investigations of bias in this emerging domain and lays important foundations for building fairer on-device ML. We apply a software engineering lens, investigating the propagation of bias through design choices in on-device ML workflows. We first identify<jats:italic>reliability bias<\/jats:italic>as a source of unfairness and propose a measure to quantify it. We then conduct empirical experiments for a keyword spotting task to show how complex and interacting technical design choices amplify and propagate<jats:italic>reliability bias<\/jats:italic>. Our results validate that design choices made during model training, like the sample rate and input feature type, and choices made to optimize models, like light-weight architectures, the pruning learning rate, and pruning sparsity, can result in disparate predictive performance across male and female groups. Based on our findings, we suggest low effort strategies for engineers to mitigate bias in on-device ML.<\/jats:p>","DOI":"10.1145\/3591867","type":"journal-article","created":{"date-parts":[[2023,4,11]],"date-time":"2023-04-11T13:30:31Z","timestamp":1681219831000},"page":"1-37","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["Tiny, Always-on, and Fragile: Bias Propagation through Design Choices in On-device Machine Learning Workflows"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9657-9509","authenticated-orcid":false,"given":"Wiebke (Toussaint)","family":"Hutiri","sequence":"first","affiliation":[{"name":"Delft University of Technology, The Netherlands"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4173-031X","authenticated-orcid":false,"given":"Aaron Yi","family":"Ding","sequence":"additional","affiliation":[{"name":"Delft University of Technology, The Netherlands"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5057-9557","authenticated-orcid":false,"given":"Fahim","family":"Kawsar","sequence":"additional","affiliation":[{"name":"Nokia Bell Labs, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1475-3017","authenticated-orcid":false,"given":"Akhil","family":"Mathur","sequence":"additional","affiliation":[{"name":"Nokia Bell Labs, UK"}]}],"member":"320","published-online":{"date-parts":[[2023,9,28]]},"reference":[{"key":"e_1_3_4_2_2","first-page":"6336","volume-title":"Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP\u201919)","author":"Alvarez Raziel","year":"2019","unstructured":"Raziel Alvarez and Hyun Jin Park. 2019. End-to-end streaming keyword spotting. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP\u201919). 6336\u20136340."},{"key":"e_1_3_4_3_2","unstructured":"Colby Banbury Vijay Janapa Reddi Peter Torelli Jeremy Holleman Nat Jeffries Csaba Kiraly Pietro Montino David Kanter Sebastian Ahmed Danilo Pau Urmish Thakker Antonio Torrini Peter Warden Jay Cordaro Giuseppe Di Guglielmo Javier Duarte Stephen Gibellini Videet Parekh Honson Tran Nhan Tran Niu Wenxu and Xu Xuesong. 2021. MLPerf Tiny Benchmark. Retrieved from http:\/\/arxiv.org\/abs\/2106.07597."},{"key":"e_1_3_4_4_2","unstructured":"Colby R. Banbury Vijay Janapa Reddi Max Lam William Fu Amin Fazel Jeremy Holleman Xinyuan Huang Robert Hurtado David Kanter Anton Lokhmotov David Patterson Danilo Pau Jae-sun Seo Jeff Sieracki Urmish Thakker Marian Verhelst and Poonam Yadav. 2020. Benchmarking TinyML Systems: Challenges and Direction. Retrieved from http:\/\/arxiv.org\/abs\/2003.04821."},{"key":"e_1_3_4_5_2","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-022-03050-7"},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.5555\/3157382.3157584"},{"key":"e_1_3_4_7_2","first-page":"1889","volume-title":"Proceedings of Machine Learning Research: Conference on Fairness, Accountability, and Transparency","volume":"81","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of Machine Learning Research: Conference on Fairness, Accountability, and Transparency, Vol. 81. 1889\u20131896."},{"key":"e_1_3_4_8_2","article-title":"TinyTL: Reduce memory, not parameters for efficient on-device learning","author":"Cai Han","year":"2020","unstructured":"Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. 2020. TinyTL: Reduce memory, not parameters for efficient on-device learning. Advances in Neural Information Processing Systems (Dec. 2020).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_4_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468537"},{"key":"e_1_3_4_10_2","volume-title":"Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP\u201914)","author":"Chen Guoguo","year":"2014","unstructured":"Guoguo Chen, Carolina Parada, and Georg Heigold. 2014. Small-footprint keyword spotting using deep neural networks. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP\u201914). IEEE."},{"key":"e_1_3_4_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2019.2921977"},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.6028\/NIST.SP.1500-201"},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.1089\/big.2016.0048"},{"key":"e_1_3_4_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3362743.3362964"},{"issue":"3","key":"e_1_3_4_15_2","article-title":"On-device machine learning: An algorithms and learning theory perspective","volume":"2","author":"Dhar Sauptik","year":"2021","unstructured":"Sauptik Dhar, Junyao Guo, Jiayi Liu, Samarth Tripathi, Unmesh Kurup, and Mohak Shah. 2021. On-device machine learning: An algorithms and learning theory perspective. ACM Trans. Internet Things 2, 3 (2021). Retrieved from http:\/\/arxiv.org\/abs\/1911.00623.","journal-title":"ACM Trans. Internet Things"},{"key":"e_1_3_4_16_2","first-page":"227","volume-title":"Proceedings of the International Symposium on Low Power Electronics and Design","author":"Dieter William R.","year":"2005","unstructured":"William R. Dieter, Srabosti Datta, and Wong Key Kai. 2005. Power reduction by varying sampling rate. In Proceedings of the International Symposium on Low Power Electronics and Design. 227\u2013232."},{"key":"e_1_3_4_17_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2021.103555"},{"key":"e_1_3_4_18_2","first-page":"1","article-title":"Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding","author":"Han Song","year":"2016","unstructured":"Song Han, Huizi Mao, and William J. Dally. 2016. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. In Proceedings of the 4th International Conference on Learning Representations (ICLR\u201916). 1\u201314.","journal-title":"Proceedings of the 4th International Conference on Learning Representations (ICLR\u201916)"},{"key":"e_1_3_4_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2015.2462851"},{"key":"e_1_3_4_20_2","first-page":"3323","article-title":"Equality of opportunity in supervised learning","author":"Hardt Moritz","year":"2016","unstructured":"Moritz Hardt, Eric Price, and Nathan Srebro. 2016. Equality of opportunity in supervised learning. Advances in Neural Information Processing Systems, 3323\u20133331.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_4_21_2","unstructured":"Andrew J. Hawkins. 2019. Serious Safety Lapses Led to Uber\u2019s Fatal Self-driving Crash New Documents Suggest. Retrieved from https:\/\/www.theverge.com\/2019\/11\/6\/20951385\/uber-self-driving-crash-death-reason-ntsb-dcouments."},{"key":"e_1_3_4_22_2","volume-title":"Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU\u201917)","author":"He Yanzhang","year":"2017","unstructured":"Yanzhang He, Rohit Prabhavalkar, Kanishka Rao, Wei Li, Anton Bakhtin, and Ian McGraw. 2017. Streaming small-footprint keyword spotting using sequence-to-sequence models. In Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU\u201917)."},{"key":"e_1_3_4_23_2","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2020-2763"},{"key":"e_1_3_4_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300830"},{"key":"e_1_3_4_25_2","unstructured":"Sara Hooker Nyalleng Moorosi Gregory Clark Samy Bengio and Emily Denton. 2020. Characterising Bias in Compressed Models. Retrieved from https:\/\/arxiv.org\/abs\/2010.03058."},{"key":"e_1_3_4_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/RE.2019.00050"},{"key":"e_1_3_4_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524491.3527308"},{"key":"e_1_3_4_28_2","first-page":"1","article-title":"Multi-scale dense networks for resource efficient image classification","author":"Huang Gao","year":"2018","unstructured":"Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens Van Der Maaten, and Kilian Weinberger. 2018. Multi-scale dense networks for resource efficient image classification. In Proceedings of the 6th International Conference on Learning Representations (ICLR\u201918). 1\u201314.","journal-title":"Proceedings of the 6th International Conference on Learning Representations (ICLR\u201918)"},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445901"},{"key":"e_1_3_4_30_2","doi-asserted-by":"publisher","DOI":"10.1162\/99608f92.762d171a"},{"key":"e_1_3_4_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2016.2545928"},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1915768117\/-\/DCSupplemental.y"},{"key":"e_1_3_4_33_2","unstructured":"Timothy B. Lee. 2019. How Terrible Software Design Decisions Led to Uber\u2019s Deadly 2018 Crash. Retrieved from https:\/\/arstechnica.com\/cars\/2019\/11\/how-terrible-software-design-decisions-led-to-ubers-deadly-2018-crash\/."},{"key":"e_1_3_4_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3506695"},{"key":"e_1_3_4_35_2","first-page":"93","article-title":"Lost in pruning: The effects of pruning neural networks beyond test accuracy","volume":"3","author":"Liebenwein Lucas","year":"2021","unstructured":"Lucas Liebenwein, Cenk Baykal, Brandon Carter, David Gifford, and Daniela Rus. 2021. Lost in pruning: The effects of pruning neural networks beyond test accuracy. Proc. Mach. Learn. Syst. 3 (2021), 93\u2013138.","journal-title":"Proc. Mach. Learn. Syst."},{"key":"e_1_3_4_36_2","unstructured":"Jiayi Liu Samarth Tripathi Unmesh Kurup and Mohak Shah. 2020. Pruning algorithms to accelerate convolutional neural networks for edge applications: A survey. Retrieved from http:\/\/arxiv.org\/abs\/2005.04275."},{"key":"e_1_3_4_37_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10287-022-00425-z"},{"key":"e_1_3_4_38_2","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2020-1765"},{"key":"e_1_3_4_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/IPSN.2018.00048"},{"key":"e_1_3_4_40_2","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS\u201921)","author":"Mazumder Mark","year":"2021","unstructured":"Mark Mazumder, Sharad Chitlangia, Colby Banbury, Yiping Kang, Juan Ciro, Keith Achorn, Daniel Galvez, Mark Sabini, Peter Mattson, David Kanter, Greg Diamos, Pete Warden, Josh Meyer, and Vijay Janapa Reddi. 2021. Multilingual spoken words corpus. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS\u201921)."},{"key":"e_1_3_4_41_2","unstructured":"Ninareh Mehrabi Fred Morstatter Nripsuta Saxena Kristina Lerman and Aram Galstyan. 2019. A survey on bias and fairness in machine learning. Retrieved from https:\/\/arxiv.org\/abs\/1908.09635."},{"key":"e_1_3_4_42_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-statistics-042720-125902"},{"key":"e_1_3_4_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3384419.3430782"},{"key":"e_1_3_4_44_2","article-title":"Privacy expectations and preferences in an IoT world this paper is included in the proceedings of the privacy expectations and preferences in an IoT world","author":"Naeini Pardis Emami","year":"2017","unstructured":"Pardis Emami Naeini, Sruti Bhagavatula, Hana Habib, Martin Degeling, Lujo Bauer, Lorrie Cranor, Norman Sadeh, Santa Clara, Pardis Emami-naeini, Sruti Bhagavatula, Hana Habib, Martin Degeling, Lujo Bauer, Lorrie Faith Cranor, and Norman Sadeh. 2017. Privacy expectations and preferences in an IoT world this paper is included in the proceedings of the privacy expectations and preferences in an IoT world. In Proceedings of the 13th Symposium on Usable Privacy and Security (SOUPS\u201917). Retrieved from https:\/\/www.usenix.org\/conference\/soups2017\/technical-sessions\/presentation\/naeini.","journal-title":"Proceedings of the 13th Symposium on Usable Privacy and Security (SOUPS\u201917)"},{"key":"e_1_3_4_45_2","doi-asserted-by":"publisher","DOI":"10.1002\/9781119121077.ch10"},{"key":"e_1_3_4_46_2","first-page":"600","article-title":"Addressing fairness in classification with a model-agnostic multi-objective algorithm","author":"Padh Kirtan","year":"2021","unstructured":"Kirtan Padh, Diego Antognini, Emma Lejal-Glaude, Boi Faltings, and Claudiu Musat. 2021. Addressing fairness in classification with a model-agnostic multi-objective algorithm. In Proceedings of the 37th Conference on Uncertainty in Artificial Intelligence (UAI\u201921). 600\u2013609.","journal-title":"Proceedings of the 37th Conference on Uncertainty in Artificial Intelligence (UAI\u201921)"},{"key":"e_1_3_4_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3494672"},{"key":"e_1_3_4_48_2","unstructured":"ProKNX. 2022. Smart Home Technology Keeps People Happy at Home for Longer. Retrieved from https:\/\/www.proknx.com\/en\/news\/2020\/smart-home-technology-keeps-people-happy-at-home-for-longer\/."},{"key":"e_1_3_4_49_2","volume-title":"Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS\u201921)","author":"Qian Shangshu","year":"2021","unstructured":"Shangshu Qian, Hung Viet Pham, Thibaud Lutellier, Zeou Hu, Jungwon Kim, Lin Tan, Yaoliang Yu, Jiahao Chen, J. P. Morgan, and Sameena Shah. 2021. Are my deep learning systems fair? An empirical study of fixed-seed training. In Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS\u201921). Retrieved from https:\/\/github.com\/lin-tan\/fairness-variance\/."},{"key":"e_1_3_4_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3375627.3375820"},{"key":"e_1_3_4_51_2","unstructured":"Apple Inc. (Press release). 2021. Apple Advances Its Privacy Leadership with iOS 15 iPadOS 15 macOS Monterey and watchOS 8. Retrieved from https:\/\/www.apple.com\/newsroom\/2021\/06\/apple-advances-its-privacy-leadership-with-ios-15-ipados-15-macos-monterey-and-watchos-8\/."},{"key":"e_1_3_4_52_2","doi-asserted-by":"publisher","DOI":"10.21437\/interspeech.2015-352"},{"key":"e_1_3_4_53_2","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2021-464"},{"key":"e_1_3_4_54_2","article-title":"Hidden technical debt in machine learning systems","author":"Sculley D.","year":"2015","unstructured":"D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Fran\u00e7ois Crespo, and Dan Dennison. 2015. Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_4_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11219-021-09557-y"},{"key":"e_1_3_4_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445865"},{"key":"e_1_3_4_57_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-13-8403-5"},{"issue":"4","key":"e_1_3_4_58_2","doi-asserted-by":"crossref","first-page":"298","DOI":"10.1123\/jmpb.2020-0061","article-title":"Impact of reduced sampling rate on accelerometer-based physical activity monitoring and machine learning activity classification","volume":"4","author":"Small Scott","year":"2021","unstructured":"Scott Small, Sara Khalid, Paula Dhiman, Shing Chan, Dan Jackson, Aiden Doherty, and Andrew Price. 2021. Impact of reduced sampling rate on accelerometer-based physical activity monitoring and machine learning activity classification. J. Measure. Phys. Behav. 4, 4 (2021), 298\u2013310.","journal-title":"J. Measure. Phys. Behav."},{"key":"e_1_3_4_59_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/w17-1606"},{"key":"e_1_3_4_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3485730.3493448"},{"key":"e_1_3_4_61_2","volume-title":"Proceedings of the International Conference on Machine Learning (ICML\u201920)","author":"Toussaint Wiebke","year":"2020","unstructured":"Wiebke Toussaint, Dave Van Veen, Courtney Irwin, Yoni Nachmany, Manuel Barreiro-Perez, Elena D\u00edaz-Pel\u00e1ez, Sara Guerreiro de Sousa, Liliana Mill\u00e1n, Pedro L. S\u00e1nchez, Antonio S\u00e1nchez-Puente, Jes\u00fas Sampedro-G\u00f3mez, P. Ignacio Dorado-D\u00edaz, and V\u00edctor Vicente-Palacios. 2020. Design considerations for high impact, automated echocardiogram analysis. In Proceedings of the International Conference on Machine Learning (ICML\u201920). Retrieved from http:\/\/arxiv.org\/abs\/2006.06292."},{"key":"e_1_3_4_62_2","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2016-1393"},{"key":"e_1_3_4_63_2","unstructured":"Jennifer Pattison Tuohy. 2021. Amazon Alexa\u2019s New Elder Care Service Launches Today. Retrieved from https:\/\/www.theverge.com\/2021\/12\/7\/22822026\/amazon-alexa-together-elder-care-price-features-release-date."},{"key":"e_1_3_4_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3194770.3194776"},{"key":"e_1_3_4_65_2","doi-asserted-by":"crossref","unstructured":"Hugo Villamizar Marcos Kalinowski and Helio Lopes. 2022. A catalogue of concerns for specifying machine learning-enabled systems. Retrieved from http:\/\/arxiv.org\/abs\/2204.07662.","DOI":"10.29327\/1298262.25-12"},{"key":"e_1_3_4_66_2","first-page":"1","article-title":"Bias preservation in machine learning: The legality of fairness metrics under EU non- discrimination law","author":"Wachter Sandra","unstructured":"Sandra Wachter, Brent Mittelstadt, and Chris Russell. [n.d.]. Bias preservation in machine learning: The legality of fairness metrics under EU non- discrimination law. West Virginia Law Review, 1\u201351. Retrieved from https:\/\/ssrn.com\/abstract=3792772.","journal-title":"West Virginia Law Review"},{"key":"e_1_3_4_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2017.7953242"},{"key":"e_1_3_4_68_2","unstructured":"Pete Warden. 2018. Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition. Retrieved from https:\/\/arxiv.org\/abs\/1804.03209."},{"key":"e_1_3_4_69_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01249-6_18"},{"key":"e_1_3_4_70_2","unstructured":"Yundong Zhang Naveen Suda Liangzhen Lai and Vikas Chandra. 2017. Hello edge: Keyword spotting on microcontrollers. Retrieved from https:\/\/arxiv.org\/abs\/1711.07128."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3591867","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3591867","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:37:45Z","timestamp":1750178265000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3591867"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,28]]},"references-count":69,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,11,30]]}},"alternative-id":["10.1145\/3591867"],"URL":"https:\/\/doi.org\/10.1145\/3591867","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,28]]},"assertion":[{"value":"2022-05-20","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-03-06","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-09-28","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}