{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T19:00:04Z","timestamp":1776106804507,"version":"3.50.1"},"reference-count":57,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2020,1,20]],"date-time":"2020-01-20T00:00:00Z","timestamp":1579478400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100010221","name":"Higher Education Commision, Pakistan","doi-asserted-by":"publisher","award":["611-19\/P.D\/2017"],"award-info":[{"award-number":["611-19\/P.D\/2017"]}],"id":[{"id":"10.13039\/501100010221","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Coping with stress is crucial for a healthy lifestyle. In the past, a great deal of research has been conducted to use socially assistive robots as a therapy to alleviate stress and anxiety related problems. However, building a fully autonomous social robot which can deliver psycho-therapeutic solutions is a very challenging endeavor due to limitations in artificial intelligence (AI). To overcome AI\u2019s limitations, researchers have previously introduced crowdsourcing-based teleoperation methods, which summon the crowd\u2019s input to control a robot\u2019s functions. However, in the context of robotics, such methods have only been used to support the object manipulation, navigational, and training tasks. It is not yet known how to leverage real-time crowdsourcing (RTC) to process complex therapeutic conversational tasks for social robotics. To fill this gap, we developed Crowd of Oz (CoZ), an open-source system that allows Softbank\u2019s Pepper robot to support such conversational tasks. To demonstrate the potential implications of this crowd-powered approach, we investigated how effectively, crowd workers recruited in real-time can teleoperate the robot\u2019s speech, in situations when the robot needs to act as a life coach. We systematically varied the number of workers who simultaneously handle the speech of the robot (N = 1, 2, 4, 8) and investigated the concomitant effects for enabling RTC for social robotics. Additionally, we present Pavilion, a novel and open-source algorithm for managing the workers\u2019 queue so that a required number of workers are engaged or waiting. Based on our findings, we discuss salient parameters that such crowd-powered systems must adhere to, so as to enhance their performance in response latency and dialogue quality.<\/jats:p>","DOI":"10.3390\/s20020569","type":"journal-article","created":{"date-parts":[[2020,1,21]],"date-time":"2020-01-21T03:04:43Z","timestamp":1579575883000},"page":"569","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Crowd of Oz: A Crowd-Powered Social Robotics System for Stress Management"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0558-6106","authenticated-orcid":false,"given":"Tahir","family":"Abbas","sequence":"first","affiliation":[{"name":"Department of Industrial Design, Eindhoven University of Technology, 5600 MB Eindhoven, The Netherlands"},{"name":"Department of Software Engineering, Mirpur University of Science &amp; Technology (MUST), Mirpur AJK 10250, Pakistan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7333-981X","authenticated-orcid":false,"given":"Vassilis-Javed","family":"Khan","sequence":"additional","affiliation":[{"name":"Department of Industrial Design, Eindhoven University of Technology, 5600 MB Eindhoven, The Netherlands"}]},{"given":"Ujwal","family":"Gadiraju","sequence":"additional","affiliation":[{"name":"L3S Research Center, Leibniz Universit\u00e4t, 30167 Hannover, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5688-4878","authenticated-orcid":false,"given":"Emilia","family":"Barakova","sequence":"additional","affiliation":[{"name":"Department of Industrial Design, Eindhoven University of Technology, 5600 MB Eindhoven, The Netherlands"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2001-7251","authenticated-orcid":false,"given":"Panos","family":"Markopoulos","sequence":"additional","affiliation":[{"name":"Department of Industrial Design, Eindhoven University of Technology, 5600 MB Eindhoven, The Netherlands"}]}],"member":"1968","published-online":{"date-parts":[[2020,1,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Taelman, J., Vandeput, S., Spaepen, A., and Van Huffel, S. (2008, January 23\u201327). Influence of mental stress on heart rate and heart rate variability. Proceedings of the 4th European Conference of the International Federation for Medical and Biological Engineering, Antwerp, Belgium.","DOI":"10.1007\/978-3-540-89208-3_324"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1016\/j.tics.2006.02.002","article-title":"Learning under stress: How does it work?","volume":"10","author":"Pu","year":"2006","journal-title":"Trends Cogn. Sci."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1080\/13811118.2015.1054055","article-title":"Suicidal behavior and psychological distress in university students: A 12-nation study","volume":"20","author":"Eskin","year":"2016","journal-title":"Arch. Suicide Res."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1080\/15332985.2017.1380742","article-title":"Predictors of suicidal ideation among college students: A prospective cohort study","volume":"16","author":"Nam","year":"2018","journal-title":"Soc. Work. Ment. Health"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"505","DOI":"10.1007\/s10902-016-9736-y","article-title":"Stress and subjective well-being among first year UK undergraduate students","volume":"18","author":"Denovan","year":"2017","journal-title":"J. Happiness Stud."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"646","DOI":"10.1016\/j.sbspro.2011.11.288","article-title":"Stress and academic achievement among undergraduate students in Universiti Putra Malaysia","volume":"29","author":"Elias","year":"2011","journal-title":"Procedia-Soc. Behav. Sci."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"55","DOI":"10.4103\/0019-557X.200253","article-title":"Perceived psychological stress among undergraduate medical students: Role of academic factors","volume":"61","author":"Chowdhury","year":"2017","journal-title":"Indian J. Public Health"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Shah, M., Hasan, S., Malik, S., and Sreeramareddy, C.T. (2010). Perceived stress, sources and severity of stress among medical undergraduates in a Pakistani medical school. BMC Med. Educ., 10.","DOI":"10.1186\/1472-6920-10-2"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"468","DOI":"10.1002\/smi.2804","article-title":"Petting away pre-exam stress: The effect of therapy dog sessions on student well-being","volume":"34","author":"Klaiber","year":"2018","journal-title":"Stress Health"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1037\/pro0000177","article-title":"The influence of a socially assistive robot on mood, anxiety, and arousal in children","volume":"49","author":"Crossman","year":"2018","journal-title":"Prof. Psychol. Res. Pract."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Pu, L., Moyle, W., Jones, C., and Todorovic, M. (2018). The Effectiveness of Social Robots for Older Adults: A Systematic Review and Meta-Analysis of Randomized Controlled Studies. Gerontologist.","DOI":"10.1093\/geront\/gny046"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Rose, E.J. (2017, January 11\u201313). Designing for engagement: Using participatory design to develop a social robot to measure teen stress. Proceedings of the 35th ACM International Conference on the Design of Communication, Halifax, NS, Canada.","DOI":"10.1145\/3121113.3121212"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Kim, J., Kim, Y., Kim, B., Yun, S., Kim, M., and Lee, J. (2018, January 21\u201326). Can a Machine Tend to Teenagers\u2019 Emotional Needs?. Proceedings of the Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems-CHI \u201918, Montreal, QC, Canada.","DOI":"10.1145\/3170427.3188548"},{"key":"ref_14","unstructured":"Kang, H., Hebert, M., and Kanade, T. (2011, January 6\u201313). Discovering object instances from scenes of Daily Living. Proceedings of the 2011 IEEE International Conference on Computer Vision, Barcelona, Spain."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1080\/17439760.2014.913671","article-title":"Crowd-powered positive psychological interventions","volume":"9","author":"Morris","year":"2014","journal-title":"J. Posit. Psychol."},{"key":"ref_16","unstructured":"Morris, R. (2011, January 7\u201312). Crowdsourcing workshop: The emergence of affective crowdsourcing. Proceedings of the 2011 Annual Conference Extended Abstracts on Human Factors in Computing Systems, Vancouver, BC, Canada."},{"key":"ref_17","unstructured":"(2020, January 19). Emotional Baggage Check. Available online: https:\/\/emotionalbaggagecheck.com\/."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Bernstein, M.S., Brandt, J., Miller, R.C., and Karger, D.R. (2011, January 16\u201319). Crowds in two seconds. Proceedings of the 24th annual ACM Symposium on User Interface Software and Technology, Proc. UIST\u201911, Santa Barbara, CA, USA.","DOI":"10.1145\/2047196.2047201"},{"key":"ref_19","first-page":"6","article-title":"Crowdsourcing versus the laboratory: Towards human-centered experiments using the crowd","volume":"Volume 10264 LNCS","author":"Gadiraju","year":"2017","journal-title":"Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Lasecki, W.S., Murray, K.I., White, S., Miller, R.C., and Bigham, J.P. (2011, January 16\u201319). Real-time crowd control of existing interfaces. Proceedings of the 24th annual ACM Symposium on User Interface Software and Technology, Santa Barbara, CA, USA.","DOI":"10.1145\/2047196.2047200"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Gouravajhala, S.R., Yim, J., Desingh, K., Huang, Y., Jenkins, O.C., and Lasecki, W.S. (2018, January 5\u20138). EURECA: Enhanced Understanding of Real Environments via Crowd Assistance. Proceedings of the Sixth AAAI Conference on Human Computation and Crowdsourcing, Z\u00fcrich, Switzerland.","DOI":"10.1609\/hcomp.v6i1.13339"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"304","DOI":"10.1016\/j.ijhcs.2005.03.006","article-title":"Evaluating a realistic agent in an advice-giving task","volume":"63","author":"Berry","year":"2005","journal-title":"Int. J. -Hum.-Comput. Stud."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Powers, A., Kiesler, S., Fussell, S., Fussell, S., and Torrey, C. (2007, January 10\u201312). Comparing a computer agent with a humanoid robot. Proceedings of the ACM\/IEEE International Conference on Human-Robot Interaction, Arlington, VA, USA.","DOI":"10.1145\/1228716.1228736"},{"key":"ref_24","unstructured":"(2020, January 19). Pepper | Softbank. Available online: https:\/\/www.softbankrobotics.com\/emea\/en\/pepper."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1231","DOI":"10.1109\/TSMCC.2007.905819","article-title":"Human performance issues and user interface design for teleoperated robots","volume":"37","author":"Chen","year":"2007","journal-title":"IEEE Trans. Syst. Man Cybern. Part C Appl. Rev."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Butler, D.J., Huang, J., Roesner, F., and Cakmak, M. (2015, January 2\u20135). The Privacy-Utility Tradeoff for Remotely Teleoperated Robots. Proceedings of the Tenth Annual ACM\/IEEE International Conference on Human-Robot Interaction, Portland, OR, USA.","DOI":"10.1145\/2696454.2696484"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1177\/1557234X13502463","article-title":"Teleoperation and Beyond for Assistive Humanoid Robots","volume":"9","author":"Goodrich","year":"2013","journal-title":"Rev. Hum. Factors Ergon."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"259","DOI":"10.3102\/0034654318821286","article-title":"Social Robots for Language Learning: A Review","volume":"89","author":"Verhagen","year":"2019","journal-title":"Rev. Educ. Res."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Torta, E., Oberzaucher, J., Werner, F., Cuijpers, R.H., and Juola, J.F. (2013). Attitudes towards Socially Assistive Robots in Intelligent Homes: Results From Laboratory Studies and Field Trials. J. -Hum.-Robot. Interact., 1.","DOI":"10.5898\/JHRI.1.2.Torta"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"530","DOI":"10.1109\/TSMCA.2011.2164243","article-title":"Teleoperation of multiple social robots","volume":"42","author":"Glas","year":"2012","journal-title":"IEEE Trans. Syst. Man Cybern. Part A Syst. Humans"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"345","DOI":"10.3109\/17518423.2012.739212","article-title":"Promoting question-asking in school-aged children with autism spectrum disorders: Effectiveness of a robot intervention compared to a human-trainer intervention","volume":"16","author":"Huskens","year":"2013","journal-title":"Dev. Neurorehabilit."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1007\/s00779-017-1060-y","article-title":"Effects of robots\u2019 intonation and bodily appearance on robot-mediated communicative treatment outcomes for children with autism spectrum disorder","volume":"22","author":"Smeekens","year":"2018","journal-title":"Pers. Ubiquitous Comput."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Sono, T., Satake, S., Kanda, T., and Imai, M. (2019). Walking partner robot chatting about scenery. Adv. Robot.","DOI":"10.1080\/01691864.2019.1610062"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/j.jss.2016.09.015","article-title":"A Survey of the Use of Crowdsourcing in Software Engineering","volume":"126","author":"Mao","year":"2017","journal-title":"J. Syst. Softw."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Von Ahn, L., and Dabbish, L. (2004, January 24\u201329). Labeling images with a computer game. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vienna, Austria.","DOI":"10.1145\/985692.985733"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Bernstein, M.S., Little, G., Miller, R.C., Hartmann, B., Ackerman, M.S., Karger, D.R., Crowell, D., and Panovich, K. (2010, January 3\u20136). Soylent: A Word Processor with a Crowd Inside. Proceedings of the ACM Symposium on User Interface Software and Technology-UIST\u201910, New York, NY, USA.","DOI":"10.1145\/1866029.1866078"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"756","DOI":"10.1038\/nature09304","article-title":"Predicting protein structures with a multiplayer online game","volume":"466","author":"Cooper","year":"2010","journal-title":"Nature"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Yu, L., and Nickerson, J.V. (2011, January 7\u201312). Cooks or cobblers?. Proceedings of the 2011 Annual Conference on Human Factors In Computing Systems-CHI \u201911, Vancouver, BC, Canada.","DOI":"10.1145\/1978942.1979147"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"67","DOI":"10.15346\/hc.v1i1.5","article-title":"Architecting Real-Time Crowd-Powered Systems","volume":"1","author":"Lasecki","year":"2014","journal-title":"Hum. Comput."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Bigham, J.P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R.C., Miller, R., Tatarowicz, A., White, B., and White, S. (2010, January 3\u20136). VizWiz: Nearly Real-Time Answers to Visual Questions. Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, New York, NY, USA.","DOI":"10.1145\/1866029.1866080"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Lasecki, W.S., Wesley, R., Nichols, J., Kulkarni, A., Allen, J.F., and Bigham, J. (2013, January 8\u201311). Chorus: A Crowd-powered Conversational Assistant. Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, St. Andrews Scotland, UK.","DOI":"10.1145\/2501988.2502057"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Huang, T.H.K., Chang, J.C., and Bigham, J.P. (2018, January 21\u201326). Evorus: A Crowd-powered Conversational Assistant Built to Automate Itself over Time. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada.","DOI":"10.1145\/3173574.3173869"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Lasecki, W.S., Thiha, P., Zhong, Y., Brady, E., and Bigham, J.P. (2013, January 21\u201323). Answering visual questions with conversational crowd assistants. Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility, Bellevue, WA, USA.","DOI":"10.1145\/2513383.2517033"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Huang, T.H., Azaria, A., Romero, O.J., and Bigham, J.P. (2019). InstructableCrowd: Creating IF-THEN Rules for Smartphones via Conversations with the Crowd. arXiv.","DOI":"10.15346\/hc.v6i1.104"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Andolina, S., Schneider, H., Chan, J., Klouche, K., Jacucci, G., and Dow, S. (2017, January 27\u201330). Crowdboard: Augmenting in-person idea generation with real-time crowds. Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition, Singapore.","DOI":"10.1145\/3059454.3059477"},{"key":"ref_46","unstructured":"Salisbury, E., Stein, S., and Ramchurn, S.D. (2015, January 4\u20138). Real-time opinion aggregation methods for crowd robotics. Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, Istanbul, Turkey."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"25","DOI":"10.5898\/JHRI\/3.2.Toris","article-title":"The Robot Management System: A Framework for Conducting Human-Robot Interaction Studies Through Crowdsourcing","volume":"3","author":"Toris","year":"2014","journal-title":"J. Hum. Robot. Interact."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Crick, C., Osentoski, S., Jenkins, O.C., and Jay, G. (2011, January 8\u201311). Human and robot perception in large-scale learning from demonstration. Proceedings of the HRI 2011, 6th ACM\/IEEE International Conference on Human-Robot Interaction, Lausanne, Switzerland.","DOI":"10.1145\/1957656.1957788"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Bj\u00f6rling, E.A., Rose, E., and Ren, R. (2018, January 5\u20138). Teen-Robot Interaction: A Pilot Study of Engagement with a Low-fidelity Prototype. Proceedings of the ACM\/IEEE International Conference on Human-Robot Interaction, Chicago, IL, USA.","DOI":"10.1145\/3173386.3177068"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Huang, T.H.K., and Bigham, J.P. (2017, January 23\u201326). A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing. Proceedings of the Fifth AAAI Conference on Human Computation and Crowdsourcing, Qu\u00e9bec City, QC, Canada.","DOI":"10.1609\/hcomp.v5i1.13318"},{"key":"ref_51","first-page":"84","article-title":"Evaluating Human-Machine Interaction for Appropriateness","volume":"2","author":"Webb","year":"2010","journal-title":"LREC Conf. Proc."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1177\/0261927X09351676","article-title":"The psychological meaning of words: LIWC and computerized text analysis methods","volume":"29","author":"Tausczik","year":"2010","journal-title":"J. Lang. Soc. Psychol."},{"key":"ref_53","unstructured":"Miller, W.R., and Rollnick, S. (2013). Motivational Interviewing: Helping People Change, Guilford Press."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Shiwa, T., Kanda, T., Imai, M., Ishiguro, H., and Hagita, N. (2008, January 12\u201315). How quickly should communication robots respond?. Proceedings of the Third International Conference on Human Robot Interaction-HRI \u201908, Amsterdam, The Netherlands.","DOI":"10.1145\/1349822.1349843"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1207\/s15327973rlsi2602_4","article-title":"Acknowledgment Tokens and Speakership Incipiency Revisited","volume":"26","author":"Zimmerman","year":"1993","journal-title":"Res. Lang. Soc. Interact."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Srinivasan, V., and Takayama, L. (2016, January 7\u201312). Help Me Please. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, San Jose, CA, USA.","DOI":"10.1145\/2858036.2858217"},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Caine, K., Sabanovic, S., and Carter, M. (2012, January 5). The effect of monitoring by cameras and robots on the privacy enhancing behaviors of older adults. Proceedings of the Seventh Annual ACM\/IEEE International Conference on Human-Robot Interaction, Boston, MA, USA.","DOI":"10.1145\/2157689.2157807"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/2\/569\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T13:20:18Z","timestamp":1760361618000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/2\/569"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,20]]},"references-count":57,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2020,1]]}},"alternative-id":["s20020569"],"URL":"https:\/\/doi.org\/10.3390\/s20020569","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1,20]]}}}