{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,25]],"date-time":"2025-09-25T18:06:44Z","timestamp":1758823604625,"version":"3.41.0"},"reference-count":64,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2018,9,18]],"date-time":"2018-09-18T00:00:00Z","timestamp":1537228800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2018,9,18]]},"abstract":"<jats:p>Smart appliances with built-in cameras, such as the Nest Cam and Amazon Echo Look, are becoming pervasive. They hold the promise of bringing high fidelity, contextually rich sensing into our homes, workplaces and other environments. Despite recent and impressive advances, computer vision systems are still limited in the types of sensing questions they can answer, and more importantly, do not easily generalize across diverse human environments. In response, researchers have investigated hybrid crowd- and AI-powered methods that collect human labels to bootstrap automatic processes. However, deployments have been small and mostly confined to institutional settings, leaving open questions about the scalability and generality of the approach. In this work, we describe our iterative development of Zensors++, a full-stack crowd-AI camera-based sensing system that moves significantly beyond prior work in terms of scale, question diversity, accuracy, latency, and economic feasibility. We deployed Zensors++ in the wild, with real users, over many months and environments, generating 1.6 million answers for nearly 200 questions created by our participants, costing roughly 6\/10ths of a cent per answer delivered. We share lessons learned, insights gleaned, and implications for future crowd-AI vision systems.<\/jats:p>","DOI":"10.1145\/3264921","type":"journal-article","created":{"date-parts":[[2018,9,19]],"date-time":"2018-09-19T11:58:41Z","timestamp":1537358321000},"page":"1-20","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["Crowd-AI Camera Sensing in the Real World"],"prefix":"10.1145","volume":"2","author":[{"given":"Anhong","family":"Guo","sequence":"first","affiliation":[{"name":"Human-Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"given":"Anuraag","family":"Jain","sequence":"additional","affiliation":[{"name":"Human-Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"given":"Shomiron","family":"Ghose","sequence":"additional","affiliation":[{"name":"Human-Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"given":"Gierad","family":"Laput","sequence":"additional","affiliation":[{"name":"Human-Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"given":"Chris","family":"Harrison","sequence":"additional","affiliation":[{"name":"Human-Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"given":"Jeffrey P.","family":"Bigham","sequence":"additional","affiliation":[{"name":"Human-Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, PA, USA"}]}],"member":"320","published-online":{"date-parts":[[2018,9,18]]},"reference":[{"key":"e_1_2_2_1_1","unstructured":"Amazon. 2018. Echo Look | Hands-Free Camera and Style Assistant with Alexa--includes Style Check to get a second opinion on your outfit. (2018). Retrieved August 8 2018 from https:\/\/www.amazon.com\/Amazon-Echo-Look-Camera-Style-Assistant\/dp\/B0186JAEWK  Amazon. 2018. Echo Look | Hands-Free Camera and Style Assistant with Alexa--includes Style Check to get a second opinion on your outfit. (2018). Retrieved August 8 2018 from https:\/\/www.amazon.com\/Amazon-Echo-Look-Camera-Style-Assistant\/dp\/B0186JAEWK"},{"key":"e_1_2_2_2_1","unstructured":"Amazon Mechanical Turk. 2018. Human intelligence through an API. (2018). Retrieved August 8 2018 from https:\/\/www.mturk.com  Amazon Mechanical Turk. 2018. Human intelligence through an API. (2018). Retrieved August 8 2018 from https:\/\/www.mturk.com"},{"key":"e_1_2_2_3_1","unstructured":"Amazon Web Services. 2018. Amazon ElastiCache. (2018). Retrieved August 8 2018 from https:\/\/aws.amazon.com\/elasticache\/  Amazon Web Services. 2018. Amazon ElastiCache. (2018). Retrieved August 8 2018 from https:\/\/aws.amazon.com\/elasticache\/"},{"key":"e_1_2_2_4_1","unstructured":"Amazon Web Services. 2018. Amazon S3. (2018). Retrieved August 8 2018 from https:\/\/aws.amazon.com\/s3\/  Amazon Web Services. 2018. Amazon S3. (2018). Retrieved August 8 2018 from https:\/\/aws.amazon.com\/s3\/"},{"key":"e_1_2_2_5_1","unstructured":"Amazon Web Services. 2018. Amazon Simple Email Service. (2018). Retrieved August 8 2018 from https:\/\/aws.amazon.com\/ses\/  Amazon Web Services. 2018. Amazon Simple Email Service. (2018). Retrieved August 8 2018 from https:\/\/aws.amazon.com\/ses\/"},{"key":"e_1_2_2_6_1","unstructured":"Amazon Web Services. 2018. Amazon Simple Notification Service (SNS). (2018). Retrieved August 8 2018 from https:\/\/aws.amazon.com\/sns\/  Amazon Web Services. 2018. Amazon Simple Notification Service (SNS). (2018). Retrieved August 8 2018 from https:\/\/aws.amazon.com\/sns\/"},{"key":"e_1_2_2_7_1","unstructured":"Amazon Web Services. 2018. Amazon Simple Queue Service. (2018). Retrieved August 8 2018 from https:\/\/aws.amazon.com\/sqs\/  Amazon Web Services. 2018. Amazon Simple Queue Service. (2018). Retrieved August 8 2018 from https:\/\/aws.amazon.com\/sqs\/"},{"key":"e_1_2_2_8_1","unstructured":"Amazon Web Services. 2018. Amazon Web Services (AWS) -- Cloud Computing Services. (2018). Retrieved August 8 2018 from https:\/\/aws.amazon.com  Amazon Web Services. 2018. Amazon Web Services (AWS) -- Cloud Computing Services. (2018). Retrieved August 8 2018 from https:\/\/aws.amazon.com"},{"key":"e_1_2_2_9_1","unstructured":"Amazon Web Services. 2018. Elastic Load Balancing. (2018). Retrieved August 8 2018 from https:\/\/aws.amazon.com\/elasticloadbalancing\/  Amazon Web Services. 2018. Elastic Load Balancing. (2018). Retrieved August 8 2018 from https:\/\/aws.amazon.com\/elasticloadbalancing\/"},{"key":"e_1_2_2_10_1","unstructured":"Amazon Web Services. 2018. Message Queues. (2018). Retrieved August 8 2018 from https:\/\/aws.amazon.com\/message-queue\/  Amazon Web Services. 2018. Message Queues. (2018). Retrieved August 8 2018 from https:\/\/aws.amazon.com\/message-queue\/"},{"key":"e_1_2_2_11_1","unstructured":"Amazon Web Services. 2018. Redis. (2018). Retrieved August 8 2018 from https:\/\/aws.amazon.com\/redis\/  Amazon Web Services. 2018. Redis. (2018). Retrieved August 8 2018 from https:\/\/aws.amazon.com\/redis\/"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1054972.1055059"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.2165\/00007256-200838120-00006"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047201"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207710"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866080"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/358916.358935"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2481291"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675214"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0968-090X(98)00019-9"},{"key":"e_1_2_2_21_1","doi-asserted-by":"crossref","unstructured":"Paolo Comelli Paolo Ferragina Mario Notturno Granieri and Flavio Stabile. 1995. Optical recognition of motor vehicle license plates. IEEE transactions on Vehicular Technology 44 4 (1995) 790--799.  Paolo Comelli Paolo Ferragina Mario Notturno Granieri and Flavio Stabile. 1995. Optical recognition of motor vehicle license plates. IEEE transactions on Vehicular Technology 44 4 (1995) 790--799.","DOI":"10.1109\/25.467963"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1967.1053964"},{"key":"e_1_2_2_23_1","unstructured":"D-Link. 2018. DCS 932L Wireless Day\/Night Camera. (2018). Retrieved August 8 2018 from https:\/\/eu.dlink.com\/uk\/en\/products\/dcs-932l-day-night-cloud-camera  D-Link. 2018. DCS 932L Wireless Day\/Night Camera. (2018). Retrieved August 8 2018 from https:\/\/eu.dlink.com\/uk\/en\/products\/dcs-932l-day-night-cloud-camera"},{"key":"e_1_2_2_24_1","unstructured":"D-Link. 2018. Sound 8 Motion Detection. (2018). Retrieved August 8 2018 from http:\/\/us.dlink.com\/features\/motion-detection-alerting\/  D-Link. 2018. Sound 8 Motion Detection. (2018). Retrieved August 8 2018 from http:\/\/us.dlink.com\/features\/motion-detection-alerting\/"},{"key":"e_1_2_2_25_1","unstructured":"Django. 2018. The web framework for perfectionists with deadlines. (2018). Retrieved August 8 2018 from https:\/\/www.djangoproject.com  Django. 2018. The web framework for perfectionists with deadlines. (2018). Retrieved August 8 2018 from https:\/\/www.djangoproject.com"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145355"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1378600.1378605"},{"key":"e_1_2_2_28_1","unstructured":"Facebook. 2018. React -- A JavaScript library for building user interfaces. (2018). Retrieved August 8 2018 from https:\/\/reactjs.org  Facebook. 2018. React -- A JavaScript library for building user interfaces. (2018). Retrieved August 8 2018 from https:\/\/reactjs.org"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1620545.1620581"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557186"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2984511.2984518"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025845"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1864349.1864375"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025781"},{"key":"e_1_2_2_35_1","doi-asserted-by":"crossref","unstructured":"Danna Gurari Qing Li Abigale J Stangl Anhong Guo Chi Lin Kristen Grauman Jiebo Luo and Jeffrey P Bigham. 2018. VizWiz Grand Challenge: Answering Visual Questions from Blind People. arXiv preprint arXiv:1802.08218 (2018).  Danna Gurari Qing Li Abigale J Stangl Anhong Guo Chi Lin Kristen Grauman Jiebo Luo and Jeffrey P Bigham. 2018. VizWiz Grand Challenge: Answering Visual Questions from Blind People. arXiv preprint arXiv:1802.08218 (2018).","DOI":"10.1109\/CVPR.2018.00380"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647403"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2851581.2892502"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173869"},{"key":"e_1_2_2_40_1","unstructured":"Ting-Hao Kenneth Huang Yun-Nung Chen and Jeffrey P. Bigham. 2017. Real-time On-Demand Crowd-powered Entity Extraction. ArXiv e-prints (April 2017). arXiv:cs.HC\/1704.03627  Ting-Hao Kenneth Huang Yun-Nung Chen and Jeffrey P. Bigham. 2017. Real-time On-Demand Crowd-powered Entity Extraction. ArXiv e-prints (April 2017). arXiv:cs.HC\/1704.03627"},{"volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP '16)","year":"2016","author":"Kenneth Huang Ting-Hao","key":"e_1_2_2_41_1"},{"volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP '17)","year":"2017","author":"Kaur Harmanpreet","key":"e_1_2_2_42_1"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2602044.2602049"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753604"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702416"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025773"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441912"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2513383.2517033"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2501988.2502057"},{"key":"e_1_2_2_50_1","unstructured":"David Lowe. 2015. The Computer Vision Industry. (2015). Retrieved August 8 2018 from https:\/\/www.cs.ubc.ca\/~lowe\/vision.html  David Lowe. 2015. The Computer Vision Industry. (2015). Retrieved August 8 2018 from https:\/\/www.cs.ubc.ca\/~lowe\/vision.html"},{"volume-title":"Omoby: Visual Search for the iPhone.","year":"2010","key":"e_1_2_2_51_1"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1006\/cviu.2000.0897"},{"key":"e_1_2_2_53_1","unstructured":"Nest. 2018. Nest Aware. (2018). Retrieved August 8 2018 from https:\/\/nest.com\/cameras\/nest-aware\/  Nest. 2018. Nest Aware. (2018). Retrieved August 8 2018 from https:\/\/nest.com\/cameras\/nest-aware\/"},{"key":"e_1_2_2_54_1","unstructured":"Dat Tien Nguyen Firoj Alam Ferda Ofli and Muhammad Imran. 2017. Automatic Image Filtering on Social Networks Using Deep Learning and Perceptual Hashing During Crises. arXiv preprint arXiv:1704.02602 (2017).  Dat Tien Nguyen Firoj Alam Ferda Ofli and Muhammad Imran. 2017. Automatic Image Filtering on Social Networks Using Deep Learning and Perceptual Hashing During Crises. arXiv preprint arXiv:1704.02602 (2017)."},{"key":"e_1_2_2_55_1","doi-asserted-by":"crossref","unstructured":"Shwetak N. Patel Matthew S. Reynolds and Gregory D. Abowd. 2008. Detecting Human Movement by Differential Air Pressure Sensing in HVAC System Ductwork: An Exploration in Infrastructure Mediated Sensing. Springer Berlin Heidelberg Berlin Heidelberg 1--18.  Shwetak N. Patel Matthew S. Reynolds and Gregory D. Abowd. 2008. Detecting Human Movement by Differential Air Pressure Sensing in HVAC System Ductwork: An Exploration in Infrastructure Mediated Sensing. Springer Berlin Heidelberg Berlin Heidelberg 1--18.","DOI":"10.1007\/978-3-540-79576-6_1"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298682"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/2030112.2030151"},{"key":"e_1_2_2_58_1","doi-asserted-by":"crossref","unstructured":"Emmanuel Munguia Tapia Stephen S. Intille and Kent Larson. 2004. Activity Recognition in the Home Using Simple and Ubiquitous Sensors. Springer Berlin Heidelberg Berlin Heidelberg 158--175.  Emmanuel Munguia Tapia Stephen S. Intille and Kent Larson. 2004. Activity Recognition in the Home Using Simple and Ubiquitous Sensors. Springer Berlin Heidelberg Berlin Heidelberg 158--175.","DOI":"10.1007\/978-3-540-24646-6_10"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.1991.139758"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-012-0564-1"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1007\/11428572_5"},{"key":"e_1_2_2_63_1","unstructured":"Christoph Zauner. 2010. Implementation and benchmarking of perceptual image hash functions. (2010).  Christoph Zauner. 2010. Implementation and benchmarking of perceptual image hash functions. (2010)."},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCE.2011.5735484"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/LSP.2016.2603342"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3264921","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3264921","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:07:59Z","timestamp":1750212479000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3264921"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,9,18]]},"references-count":64,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2018,9,18]]}},"alternative-id":["10.1145\/3264921"],"URL":"https:\/\/doi.org\/10.1145\/3264921","relation":{},"ISSN":["2474-9567"],"issn-type":[{"type":"electronic","value":"2474-9567"}],"subject":[],"published":{"date-parts":[[2018,9,18]]},"assertion":[{"value":"2018-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-09-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}