{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,24]],"date-time":"2026-03-24T03:08:59Z","timestamp":1774321739636,"version":"3.50.1"},"reference-count":104,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2024,7,31]],"date-time":"2024-07-31T00:00:00Z","timestamp":1722384000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGMIS Database"],"published-print":{"date-parts":[[2024,8]]},"abstract":"<jats:p>Determining an appropriate sample size is a critical planning decision in quantitative empirical research. In recent years, there has been a growing concern that researchers have excessively focused on statistical significance in large sample studies to the detriment of effect sizes. This research focuses on a related concern at the other end of the spectrum. We argue that a combination of bias in significant estimates obtained from small samples (compared to their population values) and an editorial preference for the publication of significant results compound to produce marked bias in published small sample studies. We then present a simulation study covering a variety of statistical techniques commonly used to examine structural equation models with latent variables. Our results support our contention that significant results obtained from small samples are likely biased and should be considered with skepticism. We also argue for the need to provide a priori power analyses to understand the behavior of parameter estimates under the small sample conditions we examine.<\/jats:p>","DOI":"10.1145\/3685235.3685238","type":"journal-article","created":{"date-parts":[[2024,7,31]],"date-time":"2024-07-31T13:37:15Z","timestamp":1722433035000},"page":"12-49","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Too Small to Succeed: Small Samples and the p-Value Problem"],"prefix":"10.1145","volume":"55","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1513-3581","authenticated-orcid":false,"given":"Miguel I.","family":"Aguirre-Urreta","sequence":"first","affiliation":[{"name":"Information Systems and Business Analytics, Florida International University, Miami, FL, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7988-7609","authenticated-orcid":false,"given":"Mikko","family":"R\u00f6nkk\u00f6","sequence":"additional","affiliation":[{"name":"School of Business and Economics, University of Jyv\u00e4skyl\u00e4, Jyv\u00e4skyl\u00e4, Finland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-7348-6078","authenticated-orcid":false,"given":"Cameron N.","family":"McIntosh","sequence":"additional","affiliation":[{"name":"Canadian Centre for Justice and Community Safety Statistics, Statistics Canada, Ottawa, ON, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,7,31]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Statistics as principled argument","author":"Abelson R. P.","year":"1995","unstructured":"Abelson, R. P. (1995). Statistics as principled argument. Lawrence Erlbaum Associates."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1111\/ijmr.12102"},{"key":"e_1_2_1_3_1","volume-title":"Sample size determination and statistical power analysis in PLS using R: An annotated tutorial. Communications of the Association for Information Systems, 36(1)","author":"Aguirre-Urreta M. I.","year":"2015","unstructured":"Aguirre-Urreta, M. I., & R\u00f6nkk\u00f6, M. (2015). Sample size determination and statistical power analysis in PLS using R: An annotated tutorial. Communications of the Association for Information Systems, 36(1). http:\/\/aisel.aisnet.org\/cais\/vol36\/iss1\/3"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.25300\/MISQ\/2018\/13587"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.leaqua.2017.01.005"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.leaqua.2017.01.006"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1177\/0149206315619011"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.2307\/248704"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.5465\/amle.9.4.zqr715"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10826-013-9795-1"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.im.2019.05.003"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1002\/smj.2477"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1111\/peps.12111"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1177\/0033294115625265"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.5555\/2017197.2017203"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.291.20.2457"},{"key":"e_1_2_1_17_1","volume-title":"Statistical strategies for small sample research (pp. 307--341)","author":"Chin W.","year":"1999","unstructured":"Chin, W., & Newsted, P. (1999). Structural equation modeling analysis with small samples using partial least squares. In R. Hoyle (Ed.), Statistical strategies for small sample research (pp. 307--341). Sage Publications."},{"key":"e_1_2_1_18_1","volume-title":"Statistical power analysis for the behavioral sciences","author":"Cohen J.","year":"1988","unstructured":"Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.","edition":"2"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.112.1.155"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1177\/096228020101000402"},{"key":"e_1_2_1_21_1","volume-title":"Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis","author":"Cumming G.","year":"2011","unstructured":"Cumming, G. (2011). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. Routledge."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1177\/0956797613504966"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1177\/0013164415607618"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.1990.03440100097014"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1002\/0470870168.ch2"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.25300\/MISQ\/2015\/39.2.02"},{"key":"e_1_2_1_27_1","volume-title":"Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One, 3(8)","author":"Dwan K.","year":"2008","unstructured":"Dwan, K., Altman, D., Arnaiz, J., Bloom, J., Chan, A., Cronin, E., Decullier, E., Easterbrook, P., Von Elm, E., Gamble, C., Ghersi, D., Ioannidis, J., Simes, J., & Williamson, P. (2008). Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One, 3(8)."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.316.7124.61"},{"key":"e_1_2_1_29_1","volume-title":"Recent developments in PLS. Communications of the Association for Information Systems, 44","author":"Evermann J.","year":"2021","unstructured":"Evermann, J., & R\u00f6nkk\u00f6, M. (2021). Recent developments in PLS. Communications of the Association for Information Systems, 44."},{"key":"e_1_2_1_30_1","volume-title":"Do pressures to publish increase scientists' bias? An empirical support from US states data. PLoS One, 5(4)","author":"Fanelli D.","year":"2010","unstructured":"Fanelli, D. (2010). Do pressures to publish increase scientists' bias? An empirical support from US states data. PLoS One, 5(4)."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0024445"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691612459059"},{"issue":"1","key":"e_1_2_1_33_1","first-page":"27","article-title":"Why figures with error bars should replace p values","volume":"217","author":"Fidler F.","year":"2009","unstructured":"Fidler, F., & Loftus, G. (2009). Why figures with error bars should replace p values. Journal of Psychology, 217(1), 27--37.","journal-title":"Journal of Psychology"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.0963-7214.2004.01502008.x"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.1255484"},{"key":"e_1_2_1_36_1","volume-title":"An update and extension to SEM guidelines for administrative and social science research. MIS Quarterly, 35(2), iii--xiv","author":"Gefen D.","year":"2011","unstructured":"Gefen, D., Rigdon, E. E., & Straub, D. W. (2011). An update and extension to SEM guidelines for administrative and social science research. MIS Quarterly, 35(2), iii--xiv."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1093\/oxfordjournals.pan.a004877"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1177\/0049124108318973"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1017\/pan.2017.41"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.5555\/2481655.2481672"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1287\/isre.1070.0123"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0076157"},{"key":"e_1_2_1_43_1","volume-title":"Multivariate data analysis. Cengage Learning","author":"Hair J.","year":"2019","unstructured":"Hair, J., Black, W., Babin, B., & Anderson, R. (2019). Multivariate data analysis. Cengage Learning."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-80519-7"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.1002106"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1108\/IMDS-09-2015-0382"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1198\/000313001300339897"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1177\/0149206316679487"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.2466\/pr0.1997.80.1.337"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1111\/1467-9876.00197"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.0020124"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.25300\/MISQ\/2019\/13980"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.5555\/2017336.2017343"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.5555\/1289800.1289809"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0028086"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1177\/1094428112452760"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1111\/iops.12045"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15327957pspr0203_4"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691611406925"},{"key":"e_1_2_1_61_1","volume-title":"Publication bias in psychology: A diagnosis based on the correlation between effect size and sample size. PLoS One, 9(9)","author":"K\u00fchberger A.","year":"2014","unstructured":"K\u00fchberger, A., Fritz, A., & Schemdl, T. (2014). Publication bias in psychology: A diagnosis based on the correlation between effect size and sample size. PLoS One, 9(9)."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1093\/geronb\/gby065"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1177\/2515245918770963"},{"issue":"1","key":"e_1_2_1_64_1","first-page":"31","article-title":"Let's take the con out of econometrics","volume":"73","author":"Leamer E.","year":"1983","unstructured":"Leamer, E. (1983). Let's take the con out of econometrics. American Economic Review, 73(1), 31--43.","journal-title":"American Economic Review"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1287\/isre.2013.0480"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.5555\/2017274.2017279"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1287\/isre.1070.0132"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.2466\/03.11.PMS.112.2.331-348"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1744-6570.2006.00059.x"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.5334\/e2019a"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15328007SEM0904_8"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1037\/1082-989X.5.2.241"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691612459058"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1177\/0149206314527133"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jom.2012.06.002"},{"key":"e_1_2_1_76_1","volume-title":"simsem: SIMulated Structural Equation Modeling (0.5--15)","author":"Pornprasertmanit S.","year":"2020","unstructured":"Pornprasertmanit, S., Miller, P., & Schoemann, A. (2020). simsem: SIMulated Structural Equation Modeling (0.5--15). http:\/\/CRAN.R-project.org\/package=simsem"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.5555\/2017410.2017423"},{"key":"e_1_2_1_78_1","volume-title":"R: A language and environment for statistical computing (4.1.0)","author":"Core Team","year":"2021","unstructured":"R Core Team. (2021). R: A language and environment for statistical computing (4.1.0). R Foundation for Statistical Computing."},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijresmar.2009.08.001"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.emj.2016.05.006"},{"key":"e_1_2_1_81_1","volume-title":"A critical look at the use of PLS-SEM in MIS Quarterly. MIS Quarterly, 36(1), iii--xiv","author":"Ringle C.","year":"2012","unstructured":"Ringle, C., Sarstedt, M., & Straub, D. (2012). A critical look at the use of PLS-SEM in MIS Quarterly. MIS Quarterly, 36(1), iii--xiv."},{"key":"e_1_2_1_82_1","volume-title":"matrixpls: Matrix-based partial least squares estimation (1.0.12). https:\/\/github.com\/mronkko\/matrixpls","author":"R\u00f6nkk\u00f6 M.","year":"2021","unstructured":"R\u00f6nkk\u00f6, M. (2021). matrixpls: Matrix-based partial least squares estimation (1.0.12). https:\/\/github.com\/mronkko\/matrixpls"},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1177\/1094428112474693"},{"key":"e_1_2_1_84_1","volume-title":"Improvements to PLSc: Remaining problems and simple solutions. [Unpublished working paper.] http:\/\/urn.fi\/URN:NBN:fi:aalto-201603051463","author":"R\u00f6nkk\u00f6 M.","year":"2016","unstructured":"R\u00f6nkk\u00f6, M., McIntosh, C. N., & Aguirre-Urreta, M. I. (2016). Improvements to PLSc: Remaining problems and simple solutions. [Unpublished working paper.] http:\/\/urn.fi\/URN:NBN:fi:aalto-201603051463"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.paid.2015.07.019"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jom.2016.05.002"},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v048.i02"},{"key":"e_1_2_1_88_1","first-page":"1","volume-title":"H. Rothstein, A. Sutton, & M","author":"Rothstein H.","year":"2005","unstructured":"Rothstein, H., Sutton, A., & Borenstein, M. (2005). Publication bias in meta-analyses. In H. Rothstein, A. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 1--7). Wiley."},{"key":"e_1_2_1_89_1","doi-asserted-by":"publisher","DOI":"10.1080\/00031305.1966.10480401"},{"key":"e_1_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.25300\/MISQ\/2016\/40.3.04"},{"key":"e_1_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.5555\/2017345.2017347"},{"key":"e_1_2_1_92_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0033242"},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","DOI":"10.3310\/hta14080"},{"key":"e_1_2_1_94_1","doi-asserted-by":"publisher","DOI":"10.1177\/0001839216629644"},{"key":"e_1_2_1_95_1","doi-asserted-by":"publisher","DOI":"10.1080\/00031305.1995.10476125"},{"key":"e_1_2_1_96_1","doi-asserted-by":"publisher","DOI":"10.2307\/248922"},{"key":"e_1_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0895-4356(99)00161-4"},{"key":"e_1_2_1_98_1","doi-asserted-by":"publisher","DOI":"10.1080\/01973533.2015.1012991"},{"key":"e_1_2_1_99_1","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMsa065779"},{"key":"e_1_2_1_100_1","doi-asserted-by":"publisher","DOI":"10.1080\/01973533.2015.1060240"},{"key":"e_1_2_1_101_1","doi-asserted-by":"publisher","DOI":"10.1080\/00031305.2019.1583913"},{"key":"e_1_2_1_102_1","doi-asserted-by":"publisher","DOI":"10.1191\/0962280205sm415oa"},{"key":"e_1_2_1_103_1","first-page":"1","volume-title":"K. G. J\u00f6reskog & S","author":"Wold H.","year":"1982","unstructured":"Wold, H. (1982). Soft modeling-The basic design and some extensions. In K. G. J\u00f6reskog & S. Wold (Eds.), Systems under indirect observation: Causality, structure, prediction (pp. 1--54). North-Holland."},{"key":"e_1_2_1_104_1","doi-asserted-by":"publisher","DOI":"10.5555\/2017424.2017433"},{"key":"e_1_2_1_105_1","doi-asserted-by":"publisher","DOI":"10.1038\/485298a"}],"container-title":["ACM SIGMIS Database: the DATABASE for Advances in Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3685235.3685238","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3685235.3685238","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:06:17Z","timestamp":1750291577000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3685235.3685238"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,31]]},"references-count":104,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,8]]}},"alternative-id":["10.1145\/3685235.3685238"],"URL":"https:\/\/doi.org\/10.1145\/3685235.3685238","relation":{},"ISSN":["0095-0033","1532-0936"],"issn-type":[{"value":"0095-0033","type":"print"},{"value":"1532-0936","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,31]]},"assertion":[{"value":"2024-07-31","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}