{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,12]],"date-time":"2026-05-12T21:09:36Z","timestamp":1778620176882,"version":"3.51.4"},"reference-count":72,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW1","license":[{"start":{"date-parts":[[2020,5,28]],"date-time":"2020-05-28T00:00:00Z","timestamp":1590624000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2020,5,28]]},"abstract":"<jats:p>Converting widely-available 2D images and videos, captured using an RGB camera, to 3D can help accelerate the training of machine learning systems in spatial reasoning domains ranging from in-home assistive robots to augmented reality to autonomous vehicles. However, automating this task is challenging because it requires not only accurately estimating object location and orientation, but also requires knowing currently unknown camera properties (e.g., focal length). A scalable way to combat this problem is to leverage people's spatial understanding of scenes by crowdsourcing visual annotations of 3D object properties. Unfortunately, getting people to directly estimate 3D properties reliably is difficult due to the limitations of image resolution, human motor accuracy, and people's 3D perception (i.e., humans do not \"see\" depth like a laser range finder). In this paper, we propose a crowd-machine hybrid approach that jointly uses crowds' approximate measurements of multiple in-scene objects to estimate the 3D state of a single target object. Our approach can generate accurate estimates of the target object by combining heterogeneous knowledge from multiple contributors regarding various different objects that share a spatial relationship with the target object. We evaluate our joint object estimation approach with 363 crowd workers and show that our method can reduce errors in the target object's 3D location estimation by over 40%, while requiring only $35$% as much human time. Our work introduces a novel way to enable groups of people with different perspectives and knowledge to achieve more accurate collective performance on challenging visual annotation tasks.<\/jats:p>","DOI":"10.1145\/3392858","type":"journal-article","created":{"date-parts":[[2020,5,29]],"date-time":"2020-05-29T16:01:06Z","timestamp":1590768066000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["C-Reference: Improving 2D to 3D Object Pose Estimation Accuracy via Crowdsourced Joint Object Estimation"],"prefix":"10.1145","volume":"4","author":[{"given":"Jean Y.","family":"Song","sequence":"first","affiliation":[{"name":"University of Michigan - Ann Arbor, Ann Arbor, MI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"John Joon Young","family":"Chung","sequence":"additional","affiliation":[{"name":"University of Michigan - Ann Arbor, Ann Arbor, MI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David F.","family":"Fouhey","sequence":"additional","affiliation":[{"name":"University of Michigan - Ann Arbor, Ann Arbor, MI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Walter S.","family":"Lasecki","sequence":"additional","affiliation":[{"name":"University of Michigan - Ann Arbor, Ann Arbor, MI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,5,29]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461912.2462002"},{"key":"e_1_2_1_2_1","volume-title":"Converting 2D video to 3D: An efficient path to a 3Dexperience","author":"Cao Xun","year":"2011","unstructured":"Xun Cao , Alan C Bovik , Yao Wang , and Qionghai Dai . 2011. Converting 2D video to 3D: An efficient path to a 3Dexperience . IEEE MultiMedia 18, 4 ( 2011 ), 12--17. Xun Cao, Alan C Bovik, Yao Wang, and Qionghai Dai. 2011. Converting 2D video to 3D: An efficient path to a 3Dexperience.IEEE MultiMedia 18, 4 (2011), 12--17."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.409"},{"key":"e_1_2_1_4_1","volume-title":"Advances in Neural Information Processing Systems 29. Curran Associates","author":"Chen Weifeng","unstructured":"Weifeng Chen , Zhao Fu , Dawei Yang , and Jia Deng . 2016. Single-Image Depth Perception in the Wild . In Advances in Neural Information Processing Systems 29. Curran Associates , Inc ., 730--738. Weifeng Chen, Zhao Fu, Dawei Yang, and Jia Deng. 2016. Single-Image Depth Perception in the Wild. In Advances in Neural Information Processing Systems 29. Curran Associates, Inc., 730--738."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00575"},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the SIGCHI conference on human factors in computing systems.","author":"Chen Yan","year":"2020","unstructured":"Yan Chen , Mauli Pandey , Jean Y. Song , Walter S. Lasecki , and Steve Oney . 2020 . Improving Crowd-Supported GUI Testing with Structural Guidance . In Proceedings of the SIGCHI conference on human factors in computing systems. Yan Chen, Mauli Pandey, Jean Y. Song, Walter S. Lasecki, and Steve Oney. 2020. Improving Crowd-Supported GUI Testing with Structural Guidance. In Proceedings of the SIGCHI conference on human factors in computing systems."},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the ACM conference on Computer-Supported Collaborative Work (CSCW '19)","author":"Chung John J.Y.","unstructured":"John J.Y. Chung , Jean Y. Song , Sindhu Kutty , Sungsoo Ray Hong , Juho Kim , and Walter S. Lasecki . 2019. Efficient Elicitation Approaches to Estimate Collective Crowd Answers . In Proceedings of the ACM conference on Computer-Supported Collaborative Work (CSCW '19) . ACM, New York, NY, USA. John J.Y. Chung, Jean Y. Song, Sindhu Kutty, Sungsoo Ray Hong, Juho Kim, and Walter S. Lasecki. 2019. Efficient Elicitation Approaches to Estimate Collective Crowd Answers. In Proceedings of the ACM conference on Computer-Supported Collaborative Work (CSCW '19). ACM, New York, NY, USA."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1026598000963"},{"key":"e_1_2_1_9_1","doi-asserted-by":"crossref","unstructured":"J. E. Cutting and P. M. Vishton. 1995. Perceiving layout and knowing distances: The interaction relative potency and contextual use of different information about depth. In Perception of space and motion. 69--117.  J. E. Cutting and P. M. Vishton. 1995. Perceiving layout and knowing distances: The interaction relative potency and contextual use of different information about depth. In Perception of space and motion. 69--117.","DOI":"10.1016\/B978-012240530-3\/50005-5"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.261"},{"key":"e_1_2_1_11_1","doi-asserted-by":"crossref","unstructured":"Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm.Applied statistics(1979) 20--28.  Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm.Applied statistics(1979) 20--28.","DOI":"10.2307\/2346806"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-88688-4_15"},{"key":"e_1_2_1_13_1","volume-title":"Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture. CoRRabs\/1411.4734","author":"Eigen David","year":"2014","unstructured":"David Eigen and Rob Fergus . 2014. Predicting Depth , Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture. CoRRabs\/1411.4734 ( 2014 ). David Eigen and Rob Fergus. 2014. Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture. CoRRabs\/1411.4734 (2014)."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.264"},{"key":"e_1_2_1_15_1","doi-asserted-by":"crossref","unstructured":"Yun Fei Guodong Rong Bin Wang and Wenping Wang. 2014. Parallel L-BFGS-B algorithm on gpu. Computers &graphics40 1--9.  Yun Fei Guodong Rong Bin Wang and Wenping Wang. 2014. Parallel L-BFGS-B algorithm on gpu. Computers &graphics40 1--9.","DOI":"10.1016\/j.cag.2014.01.002"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/358669.358692"},{"key":"e_1_2_1_17_1","volume-title":"The KITTI Vision Benchmark Suite. In Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Geiger Andreas","year":"2012","unstructured":"Andreas Geiger , Philip Lenz , and Raquel Urtasun . 2012 . Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In Conference on Computer Vision and Pattern Recognition (CVPR). Andreas Geiger, Philip Lenz, and Raquel Urtasun. 2012. Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_2_1_18_1","unstructured":"Andreas Geiger Christian Wojek and Raquel Urtasun. 2011. Joint 3d estimation of objects and scene layout. In Advances in Neural Information Processing Systems. 1467--1475.  Andreas Geiger Christian Wojek and Raquel Urtasun. 2011. Joint 3d estimation of objects and scene layout. In Advances in Neural Information Processing Systems. 1467--1475."},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","unstructured":"R. I. Hartley and A. Zisserman. 2004.Multiple View Geometry in Computer Vision(second ed.). Cambridge University Press ISBN: 0521540518.  R. I. Hartley and A. Zisserman. 2004.Multiple View Geometry in Computer Vision(second ed.). Cambridge University Press ISBN: 0521540518.","DOI":"10.1017\/CBO9780511811685"},{"key":"e_1_2_1_20_1","volume-title":"Models of the effects of prior knowledge on category learning.Journal of Experimental Psychology:Learning, Memory, and Cognition 20, 6","author":"Heit Evan","year":"1994","unstructured":"Evan Heit . 1994. Models of the effects of prior knowledge on category learning.Journal of Experimental Psychology:Learning, Memory, and Cognition 20, 6 ( 1994 ), 1264. Evan Heit. 1994. Models of the effects of prior knowledge on category learning.Journal of Experimental Psychology:Learning, Memory, and Cognition 20, 6 (1994), 1264."},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the European Conference on Computer Vision (ECCV). 0--0.","author":"Hodan Tomas","unstructured":"Tomas Hodan , Rigas Kouskouridas , Tae-Kyun Kim , Federico Tombari , Kostas Bekris , Bertram Drost , Thibault Groueix , Krzysztof Walas , Vincent Lepetit , Ales Leonardis , A Summary of the 4th International Workshop on Recovering 6D Object Pose . In Proceedings of the European Conference on Computer Vision (ECCV). 0--0. Tomas Hodan, Rigas Kouskouridas, Tae-Kyun Kim, Federico Tombari, Kostas Bekris, Bertram Drost, Thibault Groueix, Krzysztof Walas, Vincent Lepetit, Ales Leonardis, et al.2018. A Summary of the 4th International Workshop on Recovering 6D Object Pose. In Proceedings of the European Conference on Computer Vision (ECCV). 0--0."},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the European Conference on Computer Vision (ECCV). 19--34","author":"Hodan Tomas","unstructured":"Tomas Hodan , Frank Michel , Eric Brachmann , Wadim Kehl , Anders GlentBuch , Dirk Kraft , Bertram Drost , Joel Vidal , Stephan Ihrke , Xenophon Zabulis , : Benchmark for 6d object pose estimation . In Proceedings of the European Conference on Computer Vision (ECCV). 19--34 . Tomas Hodan, Frank Michel, Eric Brachmann, Wadim Kehl, Anders GlentBuch, Dirk Kraft, Bertram Drost, Joel Vidal,Stephan Ihrke, Xenophon Zabulis, et al.2018. Bop: Benchmark for 6d object pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV). 19--34."},{"key":"e_1_2_1_23_1","doi-asserted-by":"crossref","unstructured":"D. Hoiem A.A. Efros and M. Hebert. 2005. Geometric Context from a Single Image. InICCV.  D. Hoiem A.A. Efros and M. Hebert. 2005. Geometric Context from a Single Image. InICCV.","DOI":"10.1109\/ICCV.2005.107"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"e_1_2_1_25_1","unstructured":"Stephen James and Edward Johns. 2016. 3d simulation for robot arm control with deep q-learning.arXiv preprintarXiv:1609.03759(2016).  Stephen James and Edward Johns. 2016. 3d simulation for robot arm control with deep q-learning.arXiv preprintarXiv:1609.03759(2016)."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-2099"},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics","author":"Jiang Youxuan","unstructured":"Youxuan Jiang , Jonathan K. Kummerfeld , and Walter S. Lasecki . 2017. Understanding Task Design Trade-offs in Crowdsourced Paraphrase Collection . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics , Vancouver, Canada, 103--109. Youxuan Jiang, Jonathan K. Kummerfeld, and Walter S. Lasecki. 2017. Understanding Task Design Trade-offs in Crowdsourced Paraphrase Collection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Vancouver, Canada, 103--109."},{"key":"e_1_2_1_28_1","volume-title":"accessed","author":"Jones Oliphant T.","year":"2020","unstructured":"Oliphant T. Peterson P. Jones , E. (2001 , accessed 2 January 2020 ). SciPy: open source scientific tools for Python .http:\/\/www.scipy.org Oliphant T. Peterson P. Jones, E. (2001, accessed 2 January 2020). SciPy: open source scientific tools for Python.http:\/\/www.scipy.org"},{"key":"e_1_2_1_29_1","unstructured":"Sanjay Kairam and Jeffrey Heer. [n.d.]. Parting crowds: Characterizing divergent interpretations in crowdsourced annotation tasks(CSCW '16).  Sanjay Kairam and Jeffrey Heer. [n.d.]. Parting crowds: Characterizing divergent interpretations in crowdsourced annotation tasks(CSCW '16)."},{"key":"e_1_2_1_30_1","doi-asserted-by":"crossref","unstructured":"CT Kelley. 1999.Iterative Methods for Optimization. SIAM Publications Philadelphia.  CT Kelley. 1999.Iterative Methods for Optimization. SIAM Publications Philadelphia.","DOI":"10.1137\/1.9781611970920"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441923"},{"key":"e_1_2_1_32_1","volume-title":"2d-to-3d image conversion by learning depth from examples.In2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops","author":"Konrad Janusz","unstructured":"Janusz Konrad , Meng Wang , and Prakash Ishwar . 2012. 2d-to-3d image conversion by learning depth from examples.In2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops . IEEE , 16--22. Janusz Konrad, Meng Wang, and Prakash Ishwar. 2012. 2d-to-3d image conversion by learning depth from examples.In2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. IEEE, 16--22."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3332165.3347927"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 551--562","author":"Lasecki Walter S.","unstructured":"Walter S. Lasecki , Mitchell Gordon , Danai Koutra , Malte F. Jung , Steven P. Dow , and Jeffrey P. Bigham . 2014. Glance:Rapidly coding behavioral video with the crowd . In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 551--562 . Walter S. Lasecki, Mitchell Gordon, Danai Koutra, Malte F. Jung, Steven P. Dow, and Jeffrey P. Bigham. 2014. Glance:Rapidly coding behavioral video with the crowd. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 551--562."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2501988.2502057"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/IST.2009.5071663"},{"key":"e_1_2_1_37_1","volume-title":"Epnp: An accurate o (n) solution to the pnp problem. International journal of computer vision81, 2","author":"Lepetit Vincent","year":"2009","unstructured":"Vincent Lepetit , Francesc Moreno-Noguer , and Pascal Fua . 2009 . Epnp: An accurate o (n) solution to the pnp problem. International journal of computer vision81, 2 (2009), 155. Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua. 2009. Epnp: An accurate o (n) solution to the pnp problem. International journal of computer vision81, 2 (2009), 155."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2568033"},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence (AAAI'12)","author":"Lin Christopher H.","unstructured":"Christopher H. Lin , Mausam Mausam , and Daniel S. Weld . 2012. Dynamically Switching Between Synergistic Workflows for Crowdsourcing . In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence (AAAI'12) . AAAI Press,87--93. Christopher H. Lin, Mausam Mausam, and Daniel S. Weld. 2012. Dynamically Switching Between Synergistic Workflows for Crowdsourcing. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence (AAAI'12). AAAI Press,87--93."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/34.134043"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/34.862199"},{"key":"e_1_2_1_42_1","doi-asserted-by":"crossref","unstructured":"An T Nguyen Matthew Lease and Byron C Wallace. 2019. Explainable modeling of annotations in crowdsourcing.. In IUI. 575--579.  An T Nguyen Matthew Lease and Byron C Wallace. 2019. Explainable modeling of annotations in crowdsourcing.. In IUI. 575--579.","DOI":"10.1145\/3301275.3302276"},{"key":"e_1_2_1_43_1","unstructured":"Shubham Tulsiani Abhinav Gupta Nilesh Kulkarni Ishan Misra. 2019. 3D-RelNet: Joint Object and Relational Network for 3D Prediction. In ICCV.  Shubham Tulsiani Abhinav Gupta Nilesh Kulkarni Ishan Misra. 2019. 3D-RelNet: Joint Object and Relational Network for 3D Prediction. In ICCV."},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 741--754","author":"Orts-Escolano Sergio","unstructured":"Sergio Orts-Escolano , Christoph Rhemann , Sean Fanello , Wayne Chang , Adarsh Kowdle , Yury Degtyarev , David Kim , Philip L Davidson , Sameh Khamis , Mingsong Dou , : Virtual 3d teleportation in real-time . In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 741--754 . Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L Davidson, Sameh Khamis, Mingsong Dou, et al.2016. Holoportation: Virtual 3d teleportation in real-time. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 741--754."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1068\/p260599"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.5555\/2540128.2540496"},{"key":"e_1_2_1_47_1","volume-title":"First AAAI Conference on Human Computation and Crowdsourcing.","author":"Oyama Satoshi","year":"2013","unstructured":"Satoshi Oyama , Yukino Baba , Yuko Sakurai , and Hisashi Kashima . 2013 . EM-based inference of true labels using confidence judgments . In First AAAI Conference on Human Computation and Crowdsourcing. Satoshi Oyama, Yukino Baba, Yuko Sakurai, and Hisashi Kashima. 2013. EM-based inference of true labels using confidence judgments. In First AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_2_1_48_1","unstructured":"Xinlei Pan Yurong You Ziyan Wang and Cewu Lu. 2017. Virtual to real reinforcement learning for autonomous driving. arXiv preprint arXiv:1704.03952(2017).  Xinlei Pan Yurong You Ziyan Wang and Cewu Lu. 2017. Virtual to real reinforcement learning for autonomous driving. arXiv preprint arXiv:1704.03952(2017)."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.528"},{"key":"e_1_2_1_50_1","doi-asserted-by":"crossref","unstructured":"Ramya Ramakrishnan Ece Kamar Besmira Nushi Debadeepta Dey Julie Shah and Eric Horvitz. 2019. Overcoming Blind Spots in the Real World: Leveraging Complementary Abilities for Joint Execution. (2019).  Ramya Ramakrishnan Ece Kamar Besmira Nushi Debadeepta Dey Julie Shah and Eric Horvitz. 2019. Overcoming Blind Spots in the Real World: Leveraging Complementary Abilities for Joint Execution. (2019).","DOI":"10.1609\/aaai.v33i01.33016137"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3126594.3126629"},{"key":"e_1_2_1_52_1","unstructured":"Ashutosh Saxena Jamie Schulte and Andrew Ng. 2007. Depth Estimation using Monocular and Stereo Cues. In IJCAI.  Ashutosh Saxena Jamie Schulte and Andrew Ng. 2007. Depth Estimation using Monocular and Stereo Cues. In IJCAI."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.51"},{"key":"e_1_2_1_54_1","unstructured":"Alice Smith Alice E Smith David W Coit Thomas Baeck David Fogel and Zbigniew Michalewicz. 1997. Penalty functions. (1997).  Alice Smith Alice E Smith David W Coit Thomas Baeck David Fogel and Zbigniew Michalewicz. 1997. Penalty functions. (1997)."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.5555\/1613715.1613751"},{"key":"e_1_2_1_56_1","volume-title":"Lasecki","author":"Song Jean Y.","year":"2019","unstructured":"Jean Y. Song , Raymond Fok , Juho Kim , and Walter S . Lasecki . 2019 . FourEyes: Leveraging Tool Diversity as a Means toImprove Aggregate Accuracy in Crowdsourcing. ACM Transactions on Interactive Intelligent Systems (TiiS) 10, 1 (2019),3. Jean Y. Song, Raymond Fok, Juho Kim, and Walter S. Lasecki. 2019. FourEyes: Leveraging Tool Diversity as a Means toImprove Aggregate Accuracy in Crowdsourcing. ACM Transactions on Interactive Intelligent Systems (TiiS)10, 1 (2019),3."},{"key":"e_1_2_1_57_1","volume-title":"In23rd International Conference on Intelligent User Interfaces (IUI '18)","author":"Song Jean Y.","unstructured":"Jean Y. Song , Raymond Fok , Alan Lundgard , Fan Yang , Juho Kim , and Walter S. Lasecki . 2018. Two Tools Are Better Than One: Tool Diversity As a Means of Improving Aggregate Crowd Performance . In23rd International Conference on Intelligent User Interfaces (IUI '18) . ACM, New York, NY, USA, 559--570. Jean Y. Song, Raymond Fok, Alan Lundgard, Fan Yang, Juho Kim, and Walter S. Lasecki. 2018. Two Tools Are Better Than One: Tool Diversity As a Means of Improving Aggregate Crowd Performance. In23rd International Conference on Intelligent User Interfaces (IUI '18). ACM, New York, NY, USA, 559--570."},{"key":"e_1_2_1_58_1","volume-title":"Proceedings of the 24th International Conference on Intelligent User Interfaces. ACM, 558--569","author":"Song Jean Y.","unstructured":"Jean Y. Song , Stephan J. Lemmer , Michael Xieyang Liu , Shiyan Yan , Juho Kim , Jason J. Corso , and Walter S. Lasecki . 2019. Popup: reconstructing 3D video using particle filtering to aggregate crowd responses . In Proceedings of the 24th International Conference on Intelligent User Interfaces. ACM, 558--569 . Jean Y. Song, Stephan J. Lemmer, Michael Xieyang Liu, Shiyan Yan, Juho Kim, Jason J. Corso, and Walter S. Lasecki. 2019. Popup: reconstructing 3D video using particle filtering to aggregate crowd responses. In Proceedings of the 24th International Conference on Intelligent User Interfaces. ACM, 558--569."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298655"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2010.5650464"},{"key":"e_1_2_1_61_1","unstructured":"Robert J Sternberg and Karin Sternberg. 2016.Cognitive psychology. Nelson Education.  Robert J Sternberg and Karin Sternberg. 2016.Cognitive psychology. Nelson Education."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.177"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2009.5459328"},{"key":"e_1_2_1_64_1","doi-asserted-by":"crossref","unstructured":"Shubham Tulsiani Saurabh Gupta David Fouhey Alexei A. Efros and Jitendra Malik. 2017. Factoring Shape Pose and Layout from the 2D Image of a 3D Scene. arXiv(2017).  Shubham Tulsiani Saurabh Gupta David Fouhey Alexei A. Efros and Jitendra Malik. 2017. Factoring Shape Pose and Layout from the 2D Image of a 3D Scene. arXiv(2017).","DOI":"10.1109\/CVPR.2018.00039"},{"key":"e_1_2_1_65_1","volume-title":"Global optimization by basin-hopping and the lowest energy structures of Lennard-Jones clusters containing up to 110 atoms.The Journal of Physical Chemistry A101, 28","author":"Wales David J","year":"1997","unstructured":"David J Wales and Jonathan PK Doye . 1997. Global optimization by basin-hopping and the lowest energy structures of Lennard-Jones clusters containing up to 110 atoms.The Journal of Physical Chemistry A101, 28 ( 1997 ), 5111--5116. David J Wales and Jonathan PK Doye. 1997. Global optimization by basin-hopping and the lowest energy structures of Lennard-Jones clusters containing up to 110 atoms.The Journal of Physical Chemistry A101, 28 (1997), 5111--5116."},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00346"},{"key":"e_1_2_1_67_1","doi-asserted-by":"crossref","unstructured":"Yaming Wang Xiao Tan Yi Yang Ziyu Li Xiao Liu Feng Zhou and Larry S Davis. 2018. Improving Annotation for 3D Pose Dataset of Fine-Grained Object Categories.arXiv preprint arXiv:1810.09263(2018).  Yaming Wang Xiao Tan Yi Yang Ziyu Li Xiao Liu Feng Zhou and Larry S Davis. 2018. Improving Annotation for 3D Pose Dataset of Fine-Grained Object Categories.arXiv preprint arXiv:1810.09263(2018).","DOI":"10.1109\/ICCVW.2019.00341"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298800"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46484-8_10"},{"key":"e_1_2_1_70_1","volume-title":"IEEE Winter Conference on Applications of Computer Vision. 75--82","author":"Xiang Y.","unstructured":"Y. Xiang , R. Mottaghi , and S. Savarese . 2014. Beyond PASCAL: A benchmark for 3D object detection in the wild . In IEEE Winter Conference on Applications of Computer Vision. 75--82 . Y. Xiang, R. Mottaghi, and S. Savarese. 2014. Beyond PASCAL: A benchmark for 3D object detection in the wild. In IEEE Winter Conference on Applications of Computer Vision. 75--82."},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.470"},{"key":"e_1_2_1_72_1","volume-title":"Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. ACM Transactions on Mathematical Software (TOMS)23, 4","author":"Zhu Ciyou","year":"1997","unstructured":"Ciyou Zhu , Richard H Byrd , Peihuang Lu , and Jorge Nocedal . 1997. Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. ACM Transactions on Mathematical Software (TOMS)23, 4 ( 1997 ),550--560. Ciyou Zhu, Richard H Byrd, Peihuang Lu, and Jorge Nocedal. 1997. Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. ACM Transactions on Mathematical Software (TOMS)23, 4 (1997),550--560."}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3392858","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3392858","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:41:16Z","timestamp":1750200076000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3392858"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,5,28]]},"references-count":72,"journal-issue":{"issue":"CSCW1","published-print":{"date-parts":[[2020,5,28]]}},"alternative-id":["10.1145\/3392858"],"URL":"https:\/\/doi.org\/10.1145\/3392858","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,5,28]]},"assertion":[{"value":"2020-05-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}