{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T19:56:09Z","timestamp":1780516569040,"version":"3.54.1"},"reference-count":114,"publisher":"MIS Quarterly","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,3,1]]},"abstract":"<jats:p>Content validation is an essential aspect of the scale development process that ensures that measurement instruments capture their intended constructs. However, researchers rarely undertake this core step in behavioral research because it requires costly data collection and specialized expertise. We present RATER (replicable approach to expert ratings), a free web-based system (www.contval.org) that can help the broader research community (scientists, reviewers, students) gain quick and reliable insights into the content validity of measurement instruments. Guided by psychometric measurement theory, RATER evaluates whether a scale\u2019s items correspond to their intended construct, remain distinct from other constructs, and adequately represent all aspects of the construct\u2019s content domain. The system employs two unique artificial intelligence models, RATERC and RATERD, which leverage psychometric scales from 2,443 journal articles spanning eight disciplines and two state-of-the-art large language model architectures (i.e., BERT and GPT). A set of six complementary studies confirms the RATER system\u2019s accuracy, reliability, and usefulness. We find that RATER can augment the scale development and validation process, increasing the validity of findings in behavioral research.<\/jats:p>","DOI":"10.25300\/misq\/2025\/18946","type":"journal-article","created":{"date-parts":[[2025,9,18]],"date-time":"2025-09-18T16:22:11Z","timestamp":1758212531000},"page":"59-86","source":"Crossref","is-referenced-by-count":2,"title":["AI-Augmented Content Validation in Behavioral Research: Development and Evaluation of the RATER System"],"prefix":"10.25300","volume":"50","author":[{"given":"Jean-Charles","family":"Pillet","sequence":"first","affiliation":[{"name":"TBS Business School Toulouse, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kai R.","family":"Larsen","sequence":"additional","affiliation":[{"name":"Leeds School of Business, University of Colorado Boulder Boulder, CO, U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Dobolyi","sequence":"additional","affiliation":[{"name":"Leeds School of Business, University of Colorado Boulder Boulder, CO, U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Magno","family":"Queiroz","sequence":"additional","affiliation":[{"name":"College of Business, Florida Atlantic University Boca Raton, FL, U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Abram","family":"Handler","sequence":"additional","affiliation":[{"name":"Leeds School of Business, University of Colorado Boulder Boulder, CO, U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jan Ketil","family":"Arnulf","sequence":"additional","affiliation":[{"name":"BI Norwegian Business School, Norwegian Defence University College Oslo, Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rajeev","family":"Sharma","sequence":"additional","affiliation":[{"name":"Deakin Business School, Deakin University, Victoria, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"10933","published-online":{"date-parts":[[2026,3,1]]},"reference":[{"issue":"2","key":"2026022713465366100_b1-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"441","DOI":"10.1287\/isre.2024.editorial.v35.n2","article-title":"Pathways for design research on artificial intelligence","volume":"35","author":"Abbasi","year":"2024","journal-title":"Information Systems Research"},{"key":"2026022713465366100_b2-03_ma_10_25300_misq_2025_18946","unstructured":"Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., Lee, J. R., Lee, Y. T., Li, Y., Liu, W., Mendes, C. C. T., Nguyen, A., Price, E., de Rosa, G., Saarikivi, O., \u2026 Zhang, Y. (2024). Phi-4 Technical report (Version 1). arXiv. https:\/\/arxiv.org\/abs\/2412.08905"},{"key":"2026022713465366100_b3-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1007\/978-3-319-58097-5_12","article-title":"Scale development using Twitter data: Applying contemporary natural language processing methods in IS Research","volume-title":"Analytics and data science: Advances in research and pedagogy","author":"Agogo","year":"2018"},{"issue":"1","key":"2026022713465366100_b4-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1111\/joms.12058","article-title":"Methodological wishes for the next decade and how to make wishes come true","volume":"51","author":"Aguinis","year":"2014","journal-title":"Journal of Management Studies"},{"issue":"1","key":"2026022713465366100_b5-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"83","DOI":"10.5465\/annals.2016.0011","article-title":"What you see is what you get? Enhancing methodological transparency in management research","volume":"12","author":"Aguinis","year":"2018","journal-title":"Academy of Management Annals"},{"issue":"1","key":"2026022713465366100_b6-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"569","DOI":"10.1016\/j.jagp.2024.01.067","article-title":"An ounce of prevention is worth a pound of cure: Improving research quality before data collection","volume":"1","author":"Aguinis","year":"2014","journal-title":"Annual Review of Organizational Psychology and Organizational Behavior"},{"key":"2026022713465366100_b7-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","DOI":"10.1177\/01492063241291542","article-title":"Grammatical redundancy in scales: Using the \u201cConGRe\u201d process to create better measures","author":"Alley","year":"2024","journal-title":"Journal of Management, Advance online publication"},{"issue":"2","key":"2026022713465366100_b8-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"214","DOI":"10.1016\/j.sapharm.2018.03.066","article-title":"Evaluation of methods used for estimating content validity","volume":"15","author":"Almanasreh","year":"2019","journal-title":"Research in Social and Administrative Pharmacy"},{"issue":"5","key":"2026022713465366100_b9-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"732","DOI":"10.1037\/0021-9010.76.5.732","article-title":"Predicting the performance of measures in a confirmatory factor analysis with a pretest assessment of their substantive validities","volume":"76","author":"Anderson","year":"1991","journal-title":"Journal of Applied Psychology"},{"key":"2026022713465366100_b10-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1016\/B978-0-12-819200-9.00013-2","article-title":"Semantic and ontological structures of psychological attributes","volume-title":"Measuring and modeling persons and situations","author":"Arnulf","year":"2021"},{"issue":"9","key":"2026022713465366100_b11-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0106361","article-title":"Predicting survey responses: How and why semantics shape survey statistics on organizational behaviour","volume":"9","author":"Arnulf","year":"2014","journal-title":"PloS One"},{"key":"2026022713465366100_b12-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2021.720559","article-title":"Semantic algorithms in the assessment of attitudes and personality","volume":"12","author":"Arnulf","year":"2021","journal-title":"Frontiers in Psychology"},{"key":"2026022713465366100_b13-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2022.903077","article-title":"The interactive reading task: Transformer-based automatic item generation","volume":"5","author":"Attali","year":"2022","journal-title":"Frontiers in Artificial Intelligence"},{"issue":"1","key":"2026022713465366100_b14-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v067.i01","article-title":"Fitting linear mixed-effects models using lme4","volume":"67","author":"Bates","year":"2015","journal-title":"Journal of Statistical Software"},{"issue":"4","key":"2026022713465366100_b15-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"278","DOI":"10.1561\/1700000058","article-title":"Measurement in Marketing","volume":"12","author":"Baumgartner","year":"2019","journal-title":"Foundations and Trends\u00ae in Marketing"},{"key":"2026022713465366100_b16-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","DOI":"10.1016\/j.newideapsych.2024.101121","article-title":"Partnering with AI for instrument development: Possibilities and pitfalls","volume":"76","author":"Beghetto","year":"2025","journal-title":"New Ideas in Psychology"},{"issue":"5","key":"2026022713465366100_b17-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"605","DOI":"10.1037\/apl0000362","article-title":"Job crafting revisited: Implications of an extended framework for active changes at work","volume":"104","author":"Bindl","year":"2019","journal-title":"Journal of Applied Psychology"},{"issue":"1","key":"2026022713465366100_b18-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"1","DOI":"10.2307\/3250956","article-title":"Validation in information systems research: A state-of-the-art assessment","volume":"25","author":"Boudreau","year":"2001","journal-title":"MIS Quarterly"},{"key":"2026022713465366100_b19-03_ma_10_25300_misq_2025_18946","unstructured":"Bucher, M. J. J., & Martini, M. (2024). Fine-tuned \u201csmall\u201d LLMs (still) significantly outperform zero-shot generative AI models in text classification. arXiv. https:\/\/arxiv.org\/abs\/2406.08660"},{"issue":"3","key":"2026022713465366100_b20-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"451","DOI":"10.1287\/isre.2017.0704","article-title":"Thinking about measures and measurement in positivist research: A proposal for refocusing on fundamentals","volume":"28","author":"Burton-Jones","year":"2017","journal-title":"Information Systems Research"},{"issue":"1","key":"2026022713465366100_b21-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1093\/schbul\/sbz105","article-title":"Using machine learning in psychiatry: The need to establish a framework that nurtures trustworthiness","volume":"46","author":"Chandler","year":"2020","journal-title":"Schizophrenia Bulletin"},{"key":"2026022713465366100_b22-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","DOI":"10.3389\/feduc.2023.858273","article-title":"Automatic item generation: Foundations and machine learning-based approaches for assessments","volume":"8","author":"Circi","year":"2023","journal-title":"Frontiers in Education"},{"issue":"4","key":"2026022713465366100_b23-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"1199","DOI":"10.5465\/amj.2014.0506","article-title":"Flourishing via workplace relationships: Moving beyond instrumental support","volume":"59","author":"Colbert","year":"2016","journal-title":"Academy of Management Journal"},{"issue":"4","key":"2026022713465366100_b24-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"599","DOI":"10.1037\/a0036374","article-title":"Scale indicators of social exchange relationships: A comparison of relative content validity","volume":"99","author":"Colquitt","year":"2014","journal-title":"Journal of Applied Psychology"},{"issue":"10","key":"2026022713465366100_b25-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"1243","DOI":"10.1037\/apl0000406","article-title":"Content validation guidelines: Evaluation criteria for definitional correspondence and definitional distinctiveness","volume":"104","author":"Colquitt","year":"2019","journal-title":"Journal of Applied Psychology"},{"issue":"2","key":"2026022713465366100_b26-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"679","DOI":"10.25300\/MISQ\/2022\/15516","article-title":"When Constructs Become Obsolete: A Systematic Approach to Evaluating and Updating Constructs for Information Systems Research","volume":"46","author":"Compeau","year":"2022","journal-title":"MIS Quarterly"},{"issue":"12","key":"2026022713465366100_b27-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"1351","DOI":"10.1037\/apl0000815","article-title":"From alpha to omega and beyond! A look at the past, present, and (possible) future of psychometric soundness in the Journal of Applied Psychology","volume":"105","author":"Cortina","year":"2020","journal-title":"Journal of Applied Psychology"},{"issue":"4","key":"2026022713465366100_b28-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"343","DOI":"10.1521\/pedi.1992.6.4.343","article-title":"The five-factor model of personality and its relevance to personality disorders","volume":"6","author":"Costa","year":"1992","journal-title":"Journal of Personality Disorders"},{"key":"2026022713465366100_b29-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyt.2020.533949","article-title":"Deep learning-based natural language processing for screening psychiatric patients","volume":"11","author":"Dai","year":"2021","journal-title":"Frontiers in Psychiatry"},{"key":"2026022713465366100_b30-03_ma_10_25300_misq_2025_18946","unstructured":"DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., \u2026 Zhang, Z. (2025). DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv. https:\/\/arxiv.org\/abs\/2501.12948"},{"key":"2026022713465366100_b31-03_ma_10_25300_misq_2025_18946","volume-title":"Scale development: Theory and applications","author":"DeVellis","year":"2017","edition":"4th"},{"key":"2026022713465366100_b32-03_ma_10_25300_misq_2025_18946","author":"Devlin","year":"2019","journal-title":"BERT: Pre-training of deep bidirectional transformers for language understanding"},{"issue":"4","key":"2026022713465366100_b33-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1177\/2158244016671770","article-title":"Comparing traditional and crowdsourcing methods for pretesting survey questions","volume":"6","author":"Edgar","year":"2016","journal-title":"SAGE Open"},{"key":"2026022713465366100_b34-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"226","DOI":"10.1016\/j.copsyc.2021.09.019","article-title":"Using semantic similarity to understand the psychological constructs related to prosociality","volume":"44","author":"Evans","year":"2021","journal-title":"Current Opinion in Psychology"},{"key":"2026022713465366100_b35-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/P18-1082","article-title":"Hierarchical neural story generation","author":"Fan","year":"2018"},{"issue":"4","key":"2026022713465366100_b36-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"456","DOI":"10.1177\/2515245920952393","article-title":"Measurement schmeasurement: Questionable measurement practices and how to avoid them","volume":"3","author":"Flake","year":"2020","journal-title":"Advances in Methods and Practices in Psychological Science"},{"issue":"3","key":"2026022713465366100_b37-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1037\/h0057532","article-title":"A new readability yardstick","volume":"32","author":"Flesch","year":"1948","journal-title":"Journal of Applied Psychology"},{"issue":"2","key":"2026022713465366100_b38-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"265","DOI":"10.1177\/10944281231155771","article-title":"Transforming\u201d Personality Scale Development: Illustrating the Potential of State-of-the-Art Natural Language Processing","volume":"27","author":"Fyffe","year":"2024","journal-title":"Organizational Research Methods"},{"issue":"10","key":"2026022713465366100_b39-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"727","DOI":"10.17705\/1jais.00469","article-title":"Controlling for lexical closeness in survey research: A demonstration on the technology acceptance model","volume":"18","author":"Gefen","year":"2017","journal-title":"Journal of the Association for Information Systems"},{"issue":"1","key":"2026022713465366100_b40-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"crossref","first-page":"196","DOI":"10.1093\/jcr\/ucx047","article-title":"Crowdsourcing consumer research","volume":"44","author":"Goodman","year":"2017","journal-title":"Journal of Consumer Research"},{"issue":"3","key":"2026022713465366100_b41-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"crossref","first-page":"494","DOI":"10.1037\/met0000540","article-title":"Let the algorithm speak: How to use neural networks for automatic item generation in psychological scale development","volume":"29","author":"G\u00f6tz","year":"2024","journal-title":"Psychological Methods"},{"key":"2026022713465366100_b42-03_ma_10_25300_misq_2025_18946","unstructured":"Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., \u2026 Ma, Z. (2024). The Llama 3 herd of models (Version 3). arXiv. https:\/\/arxiv.org\/abs\/2407.21783"},{"key":"2026022713465366100_b43-03_ma_10_25300_misq_2025_18946","unstructured":"Han, D., Han, M., & Unsloth Team. (2023). Unsloth. https:\/\/github.com\/unslothai\/unsloth"},{"issue":"2","key":"2026022713465366100_b44-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"138","DOI":"10.1177\/1094428113520185","article-title":"It\u2019s not me, it\u2019s you: Miscomprehension in surveys","volume":"17","author":"Hardy","year":"2014","journal-title":"Organizational Research Methods"},{"issue":"1","key":"2026022713465366100_b45-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"226","DOI":"10.1002\/jcpy.1320","article-title":"Scale use and abuse: Towards best practices in the deployment of scales","volume":"33","author":"Haws","year":"2023","journal-title":"Journal of Consumer Psychology"},{"issue":"3","key":"2026022713465366100_b46-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"238","DOI":"10.1037\/1040-3590.7.3.238","article-title":"Content validity in psychological assessment: A functional approach to concepts and methods","volume":"7","author":"Haynes","year":"1995","journal-title":"Psychological Assessment"},{"key":"2026022713465366100_b47-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2111.09543","author":"He","year":"2021","journal-title":"DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing (Version 4)"},{"issue":"6","key":"2026022713465366100_b48-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"2596","DOI":"10.1177\/0149206319850280","article-title":"Scale adaptation in organizational science research: A review and best-practice recommendations","volume":"45","author":"Heggestad","year":"2019","journal-title":"Journal of Management"},{"issue":"4","key":"2026022713465366100_b49-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"1011","DOI":"10.1111\/peps.12543","article-title":"The AI-IP: Minimizing the guesswork of personality scale item development through artificial intelligence","volume":"76","author":"Hernandez","year":"2022","journal-title":"Personnel Psychology"},{"issue":"2","key":"2026022713465366100_b50-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1177\/109442819922004","article-title":"An analysis of variance approach to content validation","volume":"2","author":"Hinkin","year":"1999","journal-title":"Organizational Research Methods"},{"issue":"2","key":"2026022713465366100_b51-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"435","DOI":"10.25300\/MISQ\/2015\/39.2.08","article-title":"Mobile application usability: Conceptualization and instrument development","volume":"39","author":"Hoehle","year":"2015","journal-title":"MIS Quarterly"},{"issue":"3","key":"2026022713465366100_b52-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"crossref","first-page":"459","DOI":"10.1037\/0022-006X.47.3.459","article-title":"Item subtlety and face validity in personality assessment","volume":"47","author":"Holden","year":"1979","journal-title":"Journal of Consulting and Clinical Psychology"},{"key":"2026022713465366100_b53-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","DOI":"10.1016\/j.paid.2023.112307","article-title":"Expanding the methodological toolbox: Machine-based item desirability ratings as an alternative to human-based ratings","volume":"213","author":"Hommel","year":"2023","journal-title":"Personality and Individual Differences"},{"issue":"2","key":"2026022713465366100_b54-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"749","DOI":"10.1007\/s11336-021-09823-9","article-title":"Transformer-based deep neural language modeling for construct-specific automatic item generation","volume":"87","author":"Hommel","year":"2022","journal-title":"Psychometrika"},{"issue":"1","key":"2026022713465366100_b55-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","DOI":"10.1145\/3700597","article-title":"Human-AI synergy in survey development: Implications from large language models in business and research","volume":"16","author":"Ke","year":"2025","journal-title":"ACM Transactions on Management Information Systems"},{"key":"2026022713465366100_b56-03_ma_10_25300_misq_2025_18946","volume-title":"Foundations of behavioral research","author":"Kerlinger","year":"1973"},{"key":"2026022713465366100_b57-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1471-2288-11-174","article-title":"Conducting online expert panels: A feasibility and experimental replicability study","volume":"11","author":"Khodyakov","year":"2011","journal-title":"BMC Medical Research Methodology"},{"key":"2026022713465366100_b58-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2021.601679","article-title":"Computational language assessments of harmony in life\u2014not satisfaction with life or rating scales\u2014correlate with cooperative behaviors","volume":"12","author":"Kjell","year":"2021","journal-title":"Frontiers in Psychology"},{"key":"2026022713465366100_b59-03_ma_10_25300_misq_2025_18946","first-page":"326","article-title":"Multivariate confidence calibration for object detection","author":"K\u00fcppers","year":"2020"},{"key":"2026022713465366100_b60-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"crossref","unstructured":"Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention (Version 1). arXiv. https:\/\/arxiv.org\/abs\/2309.06180","DOI":"10.1145\/3600006.3613165"},{"issue":"4","key":"2026022713465366100_b61-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"574","DOI":"10.1177\/10944281221115374","article-title":"Construct development and validation in three practical steps: Recommendations for reviewers, editors, and authors","volume":"26","author":"Lambert","year":"2023","journal-title":"Organizational Research Methods"},{"key":"2026022713465366100_b62-03_ma_10_25300_misq_2025_18946","unstructured":"Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019). ALBERT: A lite BERT for self-supervised learning of language representations (Version 6). arXiv. https:\/\/arxiv.org\/abs\/1909.11942"},{"issue":"3","key":"2026022713465366100_b63-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"crossref","first-page":"529","DOI":"10.25300\/MISQ\/2016\/40.3.01","article-title":"A tool for addressing construct identity in literature reviews and meta-analyses","volume":"40","author":"Larsen","year":"2016","journal-title":"MIS Quarterly"},{"issue":"4","key":"2026022713465366100_b64-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"1267","DOI":"10.25300\/MISQ\/2024\/18064","article-title":"Validity in design science","volume":"49","author":"Larsen","year":"2025","journal-title":"MIS Quarterly"},{"key":"2026022713465366100_b65-03_ma_10_25300_misq_2025_18946","first-page":"272","article-title":"Validity in design science research","author":"Larsen","year":"2020"},{"key":"2026022713465366100_b66-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"crossref","DOI":"10.1109\/HICSS.2008.165","article-title":"Exploring the semantic validity of questionnaire scales","author":"Larsen","year":"2008"},{"key":"2026022713465366100_b67-03_ma_10_25300_misq_2025_18946","unstructured":"Larsen, K. R., Sharma, R., Queiroz, M., Arnulf, J. K., & Pillet, J.-C. (2023). Use of natural language processing techniques in the construct and instrument development process. In Proceedings of the European Conference on Information Systems. https:\/\/aisel.aisnet.org\/ecis2023_rp\/383"},{"issue":"1","key":"2026022713465366100_b68-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"crossref","first-page":"563","DOI":"10.1111\/j.1744-6570.1975.tb01393.x","article-title":"A quantitative approach to content validity","volume":"28","author":"Lawshe","year":"1975","journal-title":"Personnel Psychology"},{"key":"2026022713465366100_b69-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1007\/978-1-4613-3087-5_2","article-title":"What should be done with equity theory? New approaches to the study of fairness in social relationships","volume-title":"Social exchange: Advances in theory and research","author":"Leventhal","year":"1980"},{"issue":"4","key":"2026022713465366100_b70-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"1733","DOI":"10.25300\/MISQ\/2020\/15323","article-title":"TheoryOn: A design framework and system for unlocking behavioral knowledge through ontology learning","volume":"44","author":"Li","year":"2020","journal-title":"MIS Quarterly"},{"key":"2026022713465366100_b71-03_ma_10_25300_misq_2025_18946","unstructured":"Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., & Han, S. (2023). AWQ: Activation-aware weight quantization for LLM compression and acceleration (Version 5). arXiv. https:\/\/arxiv.org\/abs\/2306.00978"},{"key":"2026022713465366100_b72-03_ma_10_25300_misq_2025_18946","unstructured":"Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach (Version 1). arXiv. https:\/\/arxiv.org\/abs\/1907.11692"},{"issue":"3","key":"2026022713465366100_b73-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"crossref","first-page":"323","DOI":"10.1177\/0092070303031003011","article-title":"The dangers of poor construct conceptualization","volume":"31","author":"MacKenzie","year":"2003","journal-title":"Journal of the Academy of Marketing Science"},{"issue":"2","key":"2026022713465366100_b74-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"293","DOI":"10.2307\/23044045","article-title":"Construct measurement and validation procedures in MIS and behavioral research: Integrating new and existing techniques","volume":"35","author":"MacKenzie","year":"2011","journal-title":"MIS Quarterly"},{"key":"2026022713465366100_b75-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"276","DOI":"10.11613\/BM.2012.031","article-title":"Interrater reliability: The kappa statistic","author":"McHugh","year":"2012","journal-title":"Biochemia Medica"},{"issue":"9","key":"2026022713465366100_b76-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"crossref","first-page":"741","DOI":"10.1037\/0003-066X.50.9.741","article-title":"Validity of psychological assessment: Validation of inferences from persons\u2019 responses and performances as scientific inquiry into score meaning","volume":"50","author":"Messick","year":"1995","journal-title":"American Psychologist"},{"key":"2026022713465366100_b77-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"crossref","DOI":"10.1002\/9781118838860","volume-title":"Cognitive interviewing methodology","author":"Miller","year":"2014"},{"issue":"3","key":"2026022713465366100_b78-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"192","DOI":"10.1287\/isre.2.3.192","article-title":"Development of an instrument to measure the perceptions of adopting an information technology innovation","volume":"2","author":"Moore","year":"1991","journal-title":"Information Systems Research"},{"key":"2026022713465366100_b79-03_ma_10_25300_misq_2025_18946","volume-title":"Machine learning","author":"Murphy","year":"2012"},{"issue":"134","key":"2026022713465366100_b80-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","DOI":"10.1098\/rsif.2017.0213","article-title":"The coefficient of determination R2 and intra-class correlation coefficient from generalized linear mixed-effects models revisited and expanded","volume":"14","author":"Nakagawa","year":"2017","journal-title":"Journal of The Royal Society Interface"},{"key":"2026022713465366100_b81-03_ma_10_25300_misq_2025_18946","unstructured":"OpenAI\n          . (2024). Fine-tuning now available for GPT-4o. https:\/\/openai.com\/index\/gpt-4o-fine-tuning"},{"issue":"4","key":"2026022713465366100_b82-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"842","DOI":"10.1111\/isj.12428","article-title":"Improving scale adaptation practices in information systems research: Development and validation of a cognitive validity assessment method","volume":"33","author":"Pillet","year":"2023","journal-title":"Information Systems Journal"},{"issue":"1","key":"2026022713465366100_b83-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"539","DOI":"10.1146\/annurev-psych-120710-100452","article-title":"Sources of method bias in social science research and recommendations on how to control it","volume":"63","author":"Podsakoff","year":"2012","journal-title":"Annual Review of Psychology"},{"issue":"2","key":"2026022713465366100_b84-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1177\/1094428115624965","article-title":"Recommendations for creating better concept definitions in the organizational, behavioral, and social sciences","volume":"19","author":"Podsakoff","year":"2016","journal-title":"Organizational Research Methods"},{"issue":"2","key":"2026022713465366100_b85-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","DOI":"10.1145\/3673906","article-title":"Language models for online depression detection: A review and benchmark analysis on remote interviews","volume":"16","author":"Qin","year":"2024","journal-title":"ACM Transactions on Management Information Systems"},{"key":"2026022713465366100_b86-03_ma_10_25300_misq_2025_18946","unstructured":"Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training (Technical report). https:\/\/cdn.openai.com\/research-covers\/language-unsupervised\/language_understanding_paper.pdf"},{"issue":"1","key":"2026022713465366100_b87-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"1","DOI":"10.2307\/25148826","article-title":"Toward improving the relevance of information systems research to practice: The role of applicability checks","volume":"32","author":"Rosemann","year":"2008","journal-title":"MIS Quarterly"},{"issue":"3","key":"2026022713465366100_b88-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"380","DOI":"10.1037\/met0000244","article-title":"The Semantic Scale Network: An online tool to detect semantic overlap of psychological scales and prevent scale redundancies","volume":"25","author":"Rosenbusch","year":"2020","journal-title":"Psychological Methods"},{"issue":"5","key":"2026022713465366100_b89-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"472","DOI":"10.1037\/apl0000422","article-title":"Political affiliation and employment screening decisions: The role of similarity and identification processes","volume":"105","author":"Roth","year":"2019","journal-title":"Journal of Applied Psychology"},{"issue":"3","key":"2026022713465366100_b90-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1037\/apl0000206","article-title":"A century of progress in industrial and organizational psychology: Discoveries and the next century","volume":"102","author":"Salas","year":"2017","journal-title":"Journal of Applied Psychology"},{"key":"2026022713465366100_b91-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"787","DOI":"10.17705\/1CAIS.04736","article-title":"Empirical test guidelines for content validity: Wash, rinse, and repeat until clean","volume":"47","author":"Schmitz","year":"2020","journal-title":"Communications of the Association for Information Systems"},{"issue":"2","key":"2026022713465366100_b92-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"385","DOI":"10.1016\/0149-2063(93)90058-U","article-title":"Improving construct measurement in management research: Comments and a quantitative approach for assessing the theoretical content adequacy of paper-and-pencil survey-type instruments","volume":"19","author":"Schriesheim","year":"1993","journal-title":"Journal of Management"},{"issue":"2","key":"2026022713465366100_b93-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"469","DOI":"10.5465\/amj.2016.1061","article-title":"Too busy to be fair? The effect of workload and rewards on managers\u2019 justice rule adherence","volume":"62","author":"Sherf","year":"2019","journal-title":"Academy of Management Journal"},{"issue":"1","key":"2026022713465366100_b94-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"102","DOI":"10.1037\/1040-3590.12.1.102","article-title":"On the sins of short-form development","volume":"12","author":"Smith","year":"2000","journal-title":"Psychological Assessment"},{"issue":"2","key":"2026022713465366100_b95-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1037\/met0000545","article-title":"Improving content validity evaluation of assessment instruments through formal content validity analysis","volume":"30","author":"Spoto","year":"2025","journal-title":"Psychological Methods"},{"issue":"1","key":"2026022713465366100_b96-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"167","DOI":"10.1111\/j.1744-6570.2002.tb00108.x","article-title":"Issues and strategies for reducing the length of self-report scales","volume":"55","author":"Stanton","year":"2002","journal-title":"Personnel Psychology"},{"key":"2026022713465366100_b97-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"crossref","unstructured":"Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., & Zhou, D. (2020). MobileBERT: A compact task-agnostic BERT for resource-limited devices (Version 2). arXiv. https:\/\/arxiv.org\/abs\/2004.02984","DOI":"10.18653\/v1\/2020.acl-main.195"},{"issue":"8","key":"2026022713465366100_b98-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"842","DOI":"10.1037\/apl0000300","article-title":"Negative beliefs about accepting coworker help: Implications for employee attitudes, job performance, and reputation","volume":"103","author":"Thompson","year":"2018","journal-title":"Journal of Applied Psychology"},{"issue":"1","key":"2026022713465366100_b99-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"1","DOI":"10.5465\/19416520.2013.759433","article-title":"A critical assessment of charismatic\u2019s transformational leadership research: Back to the drawing board?","volume":"7","author":"Van Knippenberg","year":"2013","journal-title":"Academy of Management Annals"},{"key":"2026022713465366100_b100-03_ma_10_25300_misq_2025_18946","article-title":"Attention is all you need","author":"Vaswani","year":"2017"},{"issue":"3","key":"2026022713465366100_b101-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"425","DOI":"10.2307\/30036540","article-title":"User acceptance of information technology: Toward a unified view","volume":"27","author":"Venkatesh","year":"2003","journal-title":"MIS Quarterly"},{"key":"2026022713465366100_b102-03_ma_10_25300_misq_2025_18946","article-title":"Reducing conceptual redundancy of Trust in IT constructs: A content validation of complementary measures","author":"Vitari","year":"2025"},{"issue":"3","key":"2026022713465366100_b103-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1037\/1040-3590.16.3.231","article-title":"Focus groups in psychological assessment: Enhancing content validity by consulting members of the target population","volume":"16","author":"Vogt","year":"2004","journal-title":"Psychological Assessment"},{"issue":"6","key":"2026022713465366100_b104-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"629","DOI":"10.1016\/j.jom.2004.08.002","article-title":"A theory of formal conceptual definitions: Developing theory-building measurement instruments","volume":"22","author":"Wacker","year":"2004","journal-title":"Journal of Operations Management"},{"issue":"1","key":"2026022713465366100_b105-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"214","DOI":"10.1177\/17411432231155730","article-title":"The more the merrier? A network analysis of construct content validity in school leadership literature","volume":"53","author":"Wang","year":"2025","journal-title":"Educational Management Administration & Leadership"},{"issue":"3","key":"2026022713465366100_b106-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"392","DOI":"10.2307\/2346786","article-title":"Symbolic Description of Factorial Models for Analysis of Variance","volume":"22","author":"Wilkinson","year":"1973","journal-title":"Applied Statistics"},{"key":"2026022713465366100_b107-03_ma_10_25300_misq_2025_18946","author":"Wolf","year":"2019","journal-title":"HuggingFace\u2019s transformers: state-of-the-art natural language processing (Version 5)"},{"key":"2026022713465366100_b108-03_ma_10_25300_misq_2025_18946","unstructured":"Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019). XLNet: Generalized autoregressive pretraining for language understanding (Version 2). arXiv. https:\/\/arxiv.org\/abs\/1906.08237"},{"key":"2026022713465366100_b109-03_ma_10_25300_misq_2025_18946","unstructured":"Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., \u2026 Qiu, Z. (2025). Qwen2.5 Technical report (Version 2). arXiv. https:\/\/arxiv.org\/abs\/2412.15115"},{"issue":"7","key":"2026022713465366100_b110-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"crossref","first-page":"888","DOI":"10.1037\/apl0000386","article-title":"Influencing how one is seen by potential talent: Organizational impression management among recruiting firms","volume":"104","author":"Yu","year":"2019","journal-title":"Journal of Applied Psychology"},{"issue":"1","key":"2026022713465366100_b111-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"324","DOI":"10.5465\/amj.2016.0094","article-title":"Introducing team mindfulness and considering its safeguard role against conflict transformation and social undermining","volume":"61","author":"Yu","year":"2018","journal-title":"Academy of Management Journal"},{"issue":"2","key":"2026022713465366100_b112-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"165","DOI":"10.15171\/jcs.2015.017","article-title":"Design and implementation content validity study: Development of an instrument for measuring patient-centered communication","volume":"4","author":"Zamanzadeh","year":"2015","journal-title":"Journal of Caring Sciences"},{"issue":"3","key":"2026022713465366100_b113-03_ma_10_25300_misq_2025_18946","doi-asserted-by":"publisher","first-page":"779","DOI":"10.5465\/amj.2016.1344","article-title":"Not in my pay grade: The relational benefit of pay grade dissimilarity","volume":"63","author":"Zhang","year":"2020","journal-title":"Academy of Management Journal"},{"key":"2026022713465366100_b114-03_ma_10_25300_misq_2025_18946","author":"Zhong","year":"2023","journal-title":"Can ChatGPT understand too? A comparative study on ChatGPT and fine-tuned Bert"}],"container-title":["MIS Quarterly"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/misq.umn.edu\/misq\/article-pdf\/50\/1\/59\/19925\/03_ma_10.25300_misq_2025_18946.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/misq.umn.edu\/misq\/article-pdf\/50\/1\/59\/19925\/03_ma_10.25300_misq_2025_18946.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T18:47:03Z","timestamp":1772218023000},"score":1,"resource":{"primary":{"URL":"https:\/\/misq.umn.edu\/misq\/article\/50\/1\/59\/3270\/AI-Augmented-Content-Validation-in-Behavioral"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,1]]},"references-count":114,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,3,1]]},"published-print":{"date-parts":[[2026,3,1]]}},"URL":"https:\/\/doi.org\/10.25300\/misq\/2025\/18946","relation":{},"ISSN":["0276-7783","2162-9730"],"issn-type":[{"value":"0276-7783","type":"print"},{"value":"2162-9730","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,1]]}}}