{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,27]],"date-time":"2025-08-27T15:36:43Z","timestamp":1756309003283},"reference-count":44,"publisher":"IGI Global","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2012,7,1]]},"abstract":"<p>User studies with mobile devices have typically been cumbersome, since researchers have had to recruit participants, hand out or configure devices, and offer incentives and rewards. The increasing popularity of application stores has allowed researchers to use such mechanisms to recruit participants and conduct large-scale studies in authentic settings with relatively little effort. Most researchers who use application stores do not consider the side-effects or biases that such an approach may introduce. The authors summarize prior work that has reported experiences from using application stores as a recruiting, distribution and study mechanism, and also present a case study of a 4-week long study using the Android Market to deploy an application to over 4000 users that collected data on their mobile phone charging habits. The authors synthesize their own experiences with prior reported findings to discuss the challenges, advantages, limitations and considerations of using application stores as a recruitment and distribution approach for conducting large-scale studies.<\/p>","DOI":"10.4018\/jmhci.2012070102","type":"journal-article","created":{"date-parts":[[2013,1,9]],"date-time":"2013-01-09T20:56:29Z","timestamp":1357764989000},"page":"28-43","source":"Crossref","is-referenced-by-count":23,"title":["Lessons Learned from Large-Scale User Studies"],"prefix":"10.4018","volume":"4","author":[{"given":"Denzil","family":"Ferreira","sequence":"first","affiliation":[{"name":"University of Oulu, Finland"}]},{"given":"Vassilis","family":"Kostakos","sequence":"additional","affiliation":[{"name":"University of Oulu, Finland"}]},{"given":"Anind K.","family":"Dey","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, USA"}]}],"member":"2432","reference":[{"key":"jmhci.2012070102-0","unstructured":"AdMob Mobile Metrics. (2010). January 2010 Mobile Metrics Report. Retrieved February 25, 2010, from http:\/\/metrics.admob.com\/2010\/02\/january-2010-mobile-metrics-report\/"},{"key":"jmhci.2012070102-1","unstructured":"Android Developer. (2009). Status bar guidelines. Retrieved December 2, 2011, from http:\/\/developer.android.com\/guide\/practices\/ui_guidelines\/icon_design_status_bar.html"},{"key":"jmhci.2012070102-2","unstructured":"Android Developer. (2011). Support package. Retrieved December 2, 2011, from http:\/\/developer.android.com\/sdk\/compatibility-library.html"},{"key":"jmhci.2012070102-3","unstructured":"Android Developer Dashboard. (2010). Platform versions. Retrieved September 1, 2010, from http:\/\/developer.android.com\/resources\/dashboard\/platform-versions.html"},{"key":"jmhci.2012070102-4","unstructured":"Bray, T. (2010). Android Developers Blog: Download count problems. Retrieved December 2, 2011, from http:\/\/android-developers.blogspot.com\/2010\/06\/download-count-problems.html"},{"key":"jmhci.2012070102-5","doi-asserted-by":"crossref","unstructured":"Buennemeyer, T. K., Nelson, T. M., Clagett, L. M., Dunning, J. P., Marchany, R. C., & Tront, J. G. (2008). Mobile device profiling and intrusion detection using smart batteries. In Proceedings of the 41st Hawaii International Conference on System Sciences.","DOI":"10.1109\/HICSS.2008.319"},{"key":"jmhci.2012070102-6","unstructured":"Byrne, J. A. (2010). The proper charging of stationary lead-acid batteries (Your battery is only as good as how you charge it). Paper presented at the Stationary Battery Conference and Trade Show, Hollywood, FL."},{"key":"jmhci.2012070102-7","unstructured":"Church, K., & Cherubini, M. (2010). Evaluating mobile user experience in-the-wild: Prototypes, playgrounds and contextual experience sampling. In Proceedings of the Workshop on Research in the Large: Using App Stores, Markets and other Wide Distribution Channels in Ubiquitous Computing Research, Copenhagen, Denmark (pp. 29-32)."},{"key":"jmhci.2012070102-8","unstructured":"Corey, G. P. (2010). Nine ways to murder your battery (These are only some of the ways). Paper presented at the Stationary Battery Conference and Trade Show, Hollywood, FL."},{"key":"jmhci.2012070102-9","doi-asserted-by":"crossref","unstructured":"Cuervo, E., Balasubramanian, A., Cho, D., Wolman, A., Saroiu, S., Chandra, R., & Bahl, P. (2010). MAUI: Making smartphones last longer with code offload. In Proceedings of the 8th International Conference on Mobile Systems, Applications, and Services, San Francisco, CA (pp. 49-62).","DOI":"10.1145\/1814433.1814441"},{"key":"jmhci.2012070102-10","unstructured":"Enck, W., Ongtang, M., & McDaniel, P. (2008). Mitigating android software misuse before it happens (Tech. Rep. No. NAS-TR-0094-2008). State College, PA: Networking and Security Research Center, Pennsylvania State University."},{"key":"jmhci.2012070102-11","doi-asserted-by":"crossref","unstructured":"Ferreira, D., Dey, A. K., & Kostakos, V. (2011). Understanding human-smartphone concerns: A study of battery life. In Proceedings of the 9th International Conference on Pervasive Computing, San Francisco, CA (pp. 19-33).","DOI":"10.1007\/978-3-642-21726-5_2"},{"key":"jmhci.2012070102-12","unstructured":"Gartner Research. (2010). Gartner says worldwide mobile device sales grew 13.8 percent in second quarter of 2010, but competition drove prices down. Retrieved August 12, 2010, from http:\/\/www.gartner.com\/it\/page.jsp?id=1421013"},{"key":"jmhci.2012070102-13","unstructured":"Gartner Research. (2011). Gartner says worldwide mobile device sales to end users reached 1.6 billion units in 2010, smartphone sales grew 72 percent in 2010. Retrieved February 9, 2011, from http:\/\/www.gartner.com\/it\/page.jsp?id=1543014"},{"key":"jmhci.2012070102-14","unstructured":"Girardello, A., & Michahelles, F. (2010). Boostrapping your mobile application on a social market. In Proceedings of the International Conference on Ubiquitous Computing, Copenhagen, Denmark (pp. 0-1)."},{"key":"jmhci.2012070102-15","doi-asserted-by":"crossref","unstructured":"Heer, J., & Bostock, M. (2010). Crowdsourcing graphical perception: Using mechanical turk to assess visualization design. In Proceedings of the International Conference on Human Factors in Computing Systems.","DOI":"10.1145\/1753326.1753357"},{"key":"jmhci.2012070102-16","doi-asserted-by":"crossref","unstructured":"Horton, J., Rand, D., & Zeckhauser, R. (2010). The online laboratory: Conducting experiments in a real labor market (NBER Working Paper w15691). Cambridge, MA: NBER.","DOI":"10.3386\/w15961"},{"key":"jmhci.2012070102-17","author":"P.Ipeirotis","year":"2010","journal-title":"Demographics of Mechanical Turk"},{"key":"jmhci.2012070102-18","doi-asserted-by":"crossref","unstructured":"Kittur, A., Chi, E., & Suh, B. (2008). Crowdsourcing user studies with Mechanical Turk. In Proceedings of the ACM Conference on Human Factors in Computing Systems (pp. 453-456).","DOI":"10.1145\/1357054.1357127"},{"key":"jmhci.2012070102-19","unstructured":"Korn, M. (2010). Understanding use situated in real-world mobile contexts. In Proceedings of the Workshop on Research in the Large: Using App Stores, Markets and other Wide Distribution Channels in Ubiquitous Computing Research, Copenhagen, Denmark."},{"key":"jmhci.2012070102-20","unstructured":"Lawson, S. (2009). Android market needs more filters. Retrieved December 2, 2011, from http:\/\/www.pcworld.com\/article\/161410\/android_market_needs_more_filters_tmobile_says.html"},{"key":"jmhci.2012070102-21","unstructured":"McMillian, D. (2010). iPhone software distribution for mass participation. In Proceedings of the Workshop on Research in the Large: Using App Stores, Markets and other Wide Distribution Channels in Ubiquitous Computing Research, Copenhagen, Denmark (pp. 3-6)."},{"key":"jmhci.2012070102-22","unstructured":"Mechanical Turk. (2005). Amazon Mechanical Turk. Retrieved December 2, 2011, from https:\/\/www.mturk.com\/mturk\/welcome"},{"key":"jmhci.2012070102-23","unstructured":"Michahelles, F. (2010). Getting closer to reality by evaluating released apps? In Proceedings of the Workshop on Research in the Large: Using App Stores, Markets and other Wide Distribution Channels in Ubiquitous Computing Research, Copenhagen, Denmark (pp. 4-5)."},{"key":"jmhci.2012070102-24","unstructured":"Microsoft Research. (2007). Microsoft Research SenseCam. Retrieved February 14, 2010, from http:\/\/research.microsoft.com\/en- us\/um\/cambridge\/projects\/sensecam\/"},{"key":"jmhci.2012070102-25","unstructured":"Miluzzo, E., Lane, D. N., Lu, H., & Campbell, A. (2010). Research in the app store era: Experiences from the CenceMe App Deployment on the iPhone. In Proceedings of the Workshop on Research in the Large: Using App Stores, Markets and other Wide Distribution Channels in Ubiquitous Computing Research, Copenhagen, Denmark."},{"key":"jmhci.2012070102-26","unstructured":"Morrison, A., & Chalmers, M. (2010). SGVis: Analysis of mass participation trial data. In Proceedings of the Workshop on Research in the Large: Using App Stores, Markets and other Wide Distribution Channels in Ubiquitous Computing Research, Copenhagen, Denmark."},{"key":"jmhci.2012070102-27","unstructured":"Morrison, A., Reeves, S., McMillan, D., & Chalmers, M. (2010). Experiences of mass participation in ubicomp research. In Proceedings of the Workshop on Research in the Large: Using App Stores, Markets and other Wide Distribution Channels in Ubiquitous Computing Research, Copenhagen, Denmark."},{"key":"jmhci.2012070102-28","unstructured":"Nokia. (2007). Nokia LifeBlog. Retrieved October 12, 2010, from http:\/\/www.nokia.com\/lifeblog\/"},{"key":"jmhci.2012070102-29","doi-asserted-by":"publisher","DOI":"10.1145\/1508285.1508292"},{"key":"jmhci.2012070102-30","doi-asserted-by":"crossref","unstructured":"Oliver, E. (2010). The challenges in large-scale smartphone user studies. In Proceedings of the 2nd ACM International Workshop on Hot Topics in Planet-scale Measurement, San Francisco, CA (p. 5).","DOI":"10.1145\/1834616.1834623"},{"key":"jmhci.2012070102-31","unstructured":"Oliver, E., & Keshav, S. (2010). Data driven smartphone energy level prediction (Tech. Rep. No. CS-2010-06). Waterloo, ON, Canada: University of Waterloo."},{"key":"jmhci.2012070102-32","author":"P.Ostendorp","year":"2004","journal-title":"Cellular phones, advancements in energy efficiency and opportunities for energy savings"},{"key":"jmhci.2012070102-33","doi-asserted-by":"crossref","unstructured":"Patel, S. N., Kientz, J. A., Hayes, G. R., Bhat, S., & Abowd, G. D. (2006). Farther than you may think: An empirical investigation of the proximity of users to their mobile phones. P. Dourish & A. Friday (Eds.), Proceedings of the 8th International Conference on Ubiquitous Computing (LNCS 4206, pp. 123-140).","DOI":"10.1007\/11853565_8"},{"key":"jmhci.2012070102-34","unstructured":"PhoneGap. (2011). PhoneGap supported features. Retrieved December 2, 2011, from http:\/\/phonegap.com\/about\/features"},{"key":"jmhci.2012070102-35","doi-asserted-by":"crossref","unstructured":"Rahmati, A., Qian, A., & Zhong, L. (2007, September 9-12). Understanding human-battery interaction on mobile phones. In Proceedings of the 9th International Conference on Human Computer Interaction with Mobile Devices and Services, Singapore (pp. 265-272).","DOI":"10.1145\/1377999.1378017"},{"key":"jmhci.2012070102-36","doi-asserted-by":"crossref","unstructured":"Ravi, N., Scott, J., Han, L., & Iftode, L. (2008). Context-aware battery management for mobile phones. In Proceedings of the Sixth Annual IEEE International Conference on Pervasive Computing and Communications (pp. 224-233).","DOI":"10.1109\/PERCOM.2008.108"},{"key":"jmhci.2012070102-37","doi-asserted-by":"publisher","DOI":"10.1145\/1689239.1689243"},{"key":"jmhci.2012070102-38","unstructured":"Research2Guidance. (2011, April). Android market insights. Retrieved December 2, 2011, from http:\/\/www.research2guidance.com\/shop\/index.php\/android-market-insights-april-2011"},{"key":"jmhci.2012070102-39","unstructured":"Rohs, M., Kratz, S., Schleicher, R., Sahami, A., & Schmidt, A. (2010). WorldCupinion: Experiences with an android app for real-time opinion sharing during World Cup Soccer Games. In Proceedings of the Workshop on Research in the Large: Using App Stores, Markets and other Wide Distribution Channels in Ubiquitous Computing Research, Copenhagen, Denmark."},{"key":"jmhci.2012070102-40","doi-asserted-by":"crossref","unstructured":"Schmidt, A. D., Peters, F., Lamour, F., Scheel, C., \u00c7amtepe, S. A., & Albayrak, S. (2009). Monitoring smartphones for anomaly detection. In Proceedings of the 1st International Conference on MOBILe Wireless MiddleWARE, Operating Systems, and Applications (p. 40).","DOI":"10.4108\/ICST.MOBILWARE2008.2492"},{"key":"jmhci.2012070102-41","unstructured":"Schubarth, C. (2011). Android phones, iPhones grow market share. Silicon Valley Journal. Retrieved December 2, 2011, from http:\/\/www.bizjournals.com\/sanjose\/news\/2011\/12\/05\/android-phones-iphones-grow-market.html"},{"key":"jmhci.2012070102-42","doi-asserted-by":"crossref","unstructured":"Zhang, L., Tiwana, B., Dick, R. P., Qian, Z., Mao, Z. M., Wang, Z., & Yang, L. (2010). Accurate online power estimation and automatic battery behavior based power model generation for smartphones. In Proceedings of the Eighth IEEE\/ACM\/IFIP International Conference on Hardware\/Software Codesign and System Synthesis, Scottsdale, AZ (pp. 106-114).","DOI":"10.1145\/1878961.1878982"},{"key":"jmhci.2012070102-43","doi-asserted-by":"crossref","unstructured":"Zheng, P., & Ni, L. M. (2006). Spotlight: The rise of the smart phone. IEEE Distributed Systems Online, 7(3), 0603-o3003.","DOI":"10.1109\/MDSO.2006.22"}],"container-title":["International Journal of Mobile Human Computer Interaction"],"original-title":[],"language":"ng","link":[{"URL":"https:\/\/www.igi-global.com\/viewtitle.aspx?TitleId=68846","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,6,1]],"date-time":"2022-06-01T23:05:32Z","timestamp":1654124732000},"score":1,"resource":{"primary":{"URL":"https:\/\/services.igi-global.com\/resolvedoi\/resolve.aspx?doi=10.4018\/jmhci.2012070102"}},"subtitle":["Using Android Market as a Source of Data"],"short-title":[],"issued":{"date-parts":[[2012,7,1]]},"references-count":44,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2012,7]]}},"URL":"https:\/\/doi.org\/10.4018\/jmhci.2012070102","relation":{},"ISSN":["1942-390X","1942-3918"],"issn-type":[{"value":"1942-390X","type":"print"},{"value":"1942-3918","type":"electronic"}],"subject":[],"published":{"date-parts":[[2012,7,1]]}}}