{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T22:27:15Z","timestamp":1776119235105,"version":"3.50.1"},"reference-count":70,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2024,8,22]],"date-time":"2024-08-22T00:00:00Z","timestamp":1724284800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2024,8,22]]},"abstract":"<jats:p>Emotion-aware video applications (e.g., gaming, online meetings, online tutoring) strive to moderate the content presentations for a more engaging and improved user experience. These services typically deploy a machine-learning model that continuously infers the user's emotion (based on different physiological signals, facial expressions, etc.) to adapt the content delivery. Therefore, to train such models, the emotion ground truth labels also need to be collected continuously. Typically, those are collated as emotion self-reports from users in a continuous manner (using an auxiliary device such as a joystick) when they watch some videos. This process of continuous emotion annotation not only increases the cognitive load and survey fatigue but also significantly deteriorates the viewing experience. To address this problem, we propose a framework, PResUP that probes a user for emotion self-reports opportunistically based on the physiological response variations of the user. Specifically, the framework implements a sequence of phases - (a) user profile construction based on physiological responses, (b) similar user clustering based on the user profile, and (c) training a parameterized activation-guided LSTM (Long short-term memory) model by sharing data among similar users to detect the opportune self-report collection moments. All these steps together help to reduce the continuous emotion annotation overhead by probing at the opportune moments without compromising the annotation quality. We evaluated PResUP by conducting a user study (N=36) during which the participants watched eight videos, and their physiological responses and continuous emotion self-reports were recorded. The key results from this evaluation reveal that PResUP reduces the annotation overhead by reducing the probing rate by an average of 34.80% and detects the opportune probing moments with an average TPR of 80.07% without compromising the annotation quality. Motivated by these findings, we deployed PResUP by performing a follow-up user study (N=18). In this deployment scenario, we also obtained similar performance in terms of the probing rate reduction (average reduction of 38.05%), and opportune moment detection performance (average TPR of 82.26%). These findings underscore the utility of PResUP in reducing the continuous emotion annotation effort during video consumption.<\/jats:p>","DOI":"10.1145\/3678569","type":"journal-article","created":{"date-parts":[[2024,9,9]],"date-time":"2024-09-09T14:36:21Z","timestamp":1725892581000},"page":"1-32","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Towards Reducing Continuous Emotion Annotation Effort During Video Consumption: A Physiological Response Profiling Approach"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-7239-6066","authenticated-orcid":false,"given":"Swarnali","family":"Banik","sequence":"first","affiliation":[{"name":"Department of Computer Science &amp; Information Systems, BITS Pilani Goa, India"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2466-0025","authenticated-orcid":false,"given":"Sougata","family":"Sen","sequence":"additional","affiliation":[{"name":"APPCAIR, Department of Computer Science &amp; Information Systems, BITS Pilani Goa, India"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8458-604X","authenticated-orcid":false,"given":"Snehanshu","family":"Saha","sequence":"additional","affiliation":[{"name":"APPCAIR, Department of Computer Science &amp; Information Systems, BITS Pilani Goa, and HappyMonk AI Labs, India"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0226-0733","authenticated-orcid":false,"given":"Surjya","family":"Ghosh","sequence":"additional","affiliation":[{"name":"APPCAIR, Department of Computer Science &amp; Information Systems, BITS Pilani Goa, India"}]}],"member":"320","published-online":{"date-parts":[[2024,9,9]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/PerComWorkshops53856.2022.9767359"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/T-AFFC.2011.28"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.3390\/bioengineering9110688"},{"key":"e_1_2_1_4_1","volume-title":"Fast activation function approach for deep learning based online anomaly intrusion detection. In 2018 IEEE 4th International Conference on Big Data Security on Cloud (BigDataSecurity)","author":"Alrawashdeh Khaled","unstructured":"Khaled Alrawashdeh and Carla Purdy. 2018. Fast activation function approach for deep learning based online anomaly intrusion detection. In 2018 IEEE 4th International Conference on Big Data Security on Cloud (BigDataSecurity), IEEE International Conference on High Performance and Smart Computing, (HPSC) and IEEE International Conference on Intelligent Data and Security (IDS). IEEE, 5--13."},{"key":"e_1_2_1_5_1","volume-title":"A survey of methods for time series change point detection. Knowledge and information systems 51, 2","author":"Aminikhanghahi Samaneh","year":"2017","unstructured":"Samaneh Aminikhanghahi and Diane J Cook. 2017. A survey of methods for time series change point detection. Knowledge and information systems 51, 2 (2017), 339--367."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2017.06.020"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2024.3359706"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/0005-7916(94)90063-9"},{"key":"e_1_2_1_9_1","volume-title":"Random forests. Machine learning 45","author":"Breiman Leo","year":"2001","unstructured":"Leo Breiman. 2001. Random forests. Machine learning 45 (2001), 5--32."},{"key":"e_1_2_1_10_1","volume-title":"IEMOCAP: Interactive emotional dyadic motion capture database. Language resources and evaluation 42","author":"Busso Carlos","year":"2008","unstructured":"Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. IEMOCAP: Interactive emotional dyadic motion capture database. Language resources and evaluation 42 (2008), 335--359."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.3390\/s22010151"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939785"},{"key":"e_1_2_1_13_1","volume-title":"Support-vector networks. Machine learning 20","author":"Cortes Corinna","year":"1995","unstructured":"Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine learning 20 (1995), 273--297."},{"key":"e_1_2_1_14_1","unstructured":"Roddy Cowie Ellen Douglas-Cowie Susie Savvidou Edelle McMahon Martin Sawey and Marc Schr\u00f6der. 2000. 'FEELTRACE': An instrument for recording perceived emotion in real time. In ITRW Speech-Emotion."},{"key":"e_1_2_1_15_1","volume-title":"Gtrace: General trace program compatible with emotionml. In 2013 humaine association conference on affective computing and intelligent interaction","author":"Cowie Roddy","year":"2013","unstructured":"Roddy Cowie, Martin Sawey, Cian Doherty, Javier Jaimovich, Cavan Fyans, and Paul Stapleton. 2013. Gtrace: General trace program compatible with emotionml. In 2013 humaine association conference on affective computing and intelligent interaction. IEEE, 709--710."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.is.2014.11.009"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3264913"},{"key":"e_1_2_1_18_1","volume-title":"A review and meta-analysis of multimodal affect detection systems. ACM computing surveys (CSUR) 47, 3","author":"D'mello Sidney K","year":"2015","unstructured":"Sidney K D'mello and Jacqueline Kory. 2015. A review and meta-analysis of multimodal affect detection systems. ACM computing surveys (CSUR) 47, 3 (2015), 1--36."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.entcs.2019.04.009"},{"key":"e_1_2_1_20_1","volume-title":"A better activation function for artificial neural networks","author":"Elliott David L","unstructured":"David L Elliott. 1993. A better activation function for artificial neural networks. University of Maryland. Systems Research Center."},{"key":"e_1_2_1_21_1","volume-title":"Dimensions of personality","author":"Eysenck Hans","unstructured":"Hans Eysenck. 2018. Dimensions of personality. Routledge."},{"key":"e_1_2_1_22_1","volume-title":"Analysing connectivity with Granger causality and dynamic causal modelling. Current opinion in neurobiology 23, 2","author":"Friston Karl","year":"2013","unstructured":"Karl Friston, Rosalyn Moran, and Anil K Seth. 2013. Analysing connectivity with Granger causality and dynamic causal modelling. Current opinion in neurobiology 23, 2 (2013), 172--178."},{"key":"e_1_2_1_23_1","volume-title":"IEEE ISR","author":"Gao Qian","year":"2013","unstructured":"Qian Gao, Su Mei Xi, and Young Im Cho. 2013. A multi-agent personalized ontology profile based user preference profile construction method. In IEEE ISR 2013. IEEE, 1--4."},{"key":"e_1_2_1_24_1","first-page":"1","article-title":"Sonicface: Tracking facial expressions using a commodity microphone array","volume":"5","author":"Gao Yang","year":"2021","unstructured":"Yang Gao, Yincheng Jin, Seokmin Choi, Jiyang Li, Junjie Pan, Lin Shu, Chi Zhou, and Zhanpeng Jin. 2021. Sonicface: Tracking facial expressions using a commodity microphone array. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 4 (2021), 1--33.","journal-title":"Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/EMBC.2016.7590834"},{"key":"e_1_2_1_26_1","doi-asserted-by":"crossref","unstructured":"Jeffrey M Girard and Aidan GC Wright. [n. d.]. DARMA: Software for dual axis rating and media annotation. Behavior research methods 50 3 ([n. d.]).","DOI":"10.3758\/s13428-017-0915-5"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICMIC.2015.7409485"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835484"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3536221.3556603"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.2307\/2346830"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3036990"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2020.103801"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2017.2781732"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219845"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514197.3549640"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2018.09.001"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cobeha.2017.09.011"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP40776.2020.9053558"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/TBME.2010.2048568"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2013.01.012"},{"key":"e_1_2_1_41_1","volume-title":"On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics","author":"Mann Henry B","year":"1947","unstructured":"Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics (1947), 50--60."},{"key":"e_1_2_1_42_1","volume-title":"Continuous detection of physiological stress with commodity hardware. ACM transactions on computing for healthcare 1, 2","author":"Mishra Varun","year":"2020","unstructured":"Varun Mishra, Gunnar Pope, Sarah Lord, Stephanie Lewia, Byron Lowens, Kelly Caine, Sougata Sen, Ryan Halter, and David Kotz. 2020. Continuous detection of physiological stress with commodity hardware. ACM transactions on computing for healthcare 1, 2 (2020), 1--30."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445235"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.bbe.2020.04.005"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/NER.2011.5910636"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.5555\/1497470.1497492"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41597-020-00630-y"},{"key":"e_1_2_1_48_1","volume-title":"Searching for activation functions. arXiv preprint arXiv:1710.05941","author":"Ramachandran Prajit","year":"2017","unstructured":"Prajit Ramachandran, Barret Zoph, and Quoc V Le. 2017. Searching for activation functions. arXiv preprint arXiv:1710.05941 (2017)."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1016\/0377-0427(87)90125-7"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0077714"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2883213"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2020.3014842"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3606274.3606278"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/52.3-4.591"},{"key":"e_1_2_1_55_1","volume-title":"d.]. Continuous, real-time emotion annotation: A novel joystick-based analysis framework","author":"Sharma Karan","unstructured":"Karan Sharma, Claudio Castellini, Freek Stulp, and Egon L Van den Broek. [n. d.]. Continuous, real-time emotion annotation: A novel joystick-based analysis framework. IEEE Transactions on Affective Computing 11, 1 ([n. d.])."},{"key":"e_1_2_1_56_1","volume-title":"Alin Albu Schaeffer, and Friedhelm Schwenker","author":"Sharma Karan","year":"2019","unstructured":"Karan Sharma, Claudio Castellini, Egon L van den Broek, Alin Albu Schaeffer, and Friedhelm Schwenker. 2019. A dataset of continuous affect annotations and physiological signals for emotion analysis. Scientific data 6, 1 (2019)."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.3390\/s18072074"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2019.2901673"},{"key":"e_1_2_1_59_1","volume-title":"P-LSTM: A Novel LSTM Architecture for Glucose Level Prediction Problem. In International Conference on Neural Information Processing. Springer, 369--380","author":"Swain Abhijeet","year":"2022","unstructured":"Abhijeet Swain, Vaibhav Ganatra, Snehanshu Saha, Archana Mathur, and Rekha Phadke. 2022. P-LSTM: A Novel LSTM Architecture for Glucose Level Prediction Problem. In International Conference on Neural Information Processing. Springer, 369--380."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3495002"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-70093-9_86"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0014610"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11031-014-9445-y"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818346.2820736"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13042-019-01056-8"},{"key":"e_1_2_1_66_1","volume-title":"Large-scale nonlinear Granger causality for inferring directed dependence from short multivariate time-series data. Scientific reports 11, 1","author":"Wism\u00fcller Axel","year":"2021","unstructured":"Axel Wism\u00fcller, Adora M Dsouza, M Ali Vosoughi, and Anas Abidin. 2021. Large-scale nonlinear Granger causality for inferring directed dependence from short multivariate time-series data. Scientific reports 11, 1 (2021), 7817."},{"key":"e_1_2_1_67_1","volume-title":"Principal component analysis. Chemometrics and intelligent laboratory systems 2, 1--3","author":"Wold Svante","year":"1987","unstructured":"Svante Wold, Kim Esbensen, and Paul Geladi. 1987. Principal component analysis. Chemometrics and intelligent laboratory systems 2, 1--3 (1987), 37--52."},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/T-AFFC.2012.14"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.4304\/jsw.6.8.1529-1536"},{"key":"e_1_2_1_70_1","volume-title":"Chen Wang, Alan Hanjalic, and Pablo Cesar.","author":"Zhang Tianyi","year":"2020","unstructured":"Tianyi Zhang, Abdallah El Ali, Chen Wang, Alan Hanjalic, and Pablo Cesar. 2020. RCEA: Real-time, Continuous Emotion Annotation for Collecting Precise Mobile Video Ground Truth Labels. In ACM CHI. 1--15."}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678569","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3678569","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T14:43:38Z","timestamp":1755787418000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678569"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,22]]},"references-count":70,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,8,22]]}},"alternative-id":["10.1145\/3678569"],"URL":"https:\/\/doi.org\/10.1145\/3678569","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,22]]},"assertion":[{"value":"2024-09-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}