{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T18:07:18Z","timestamp":1790100438335,"version":"4.0.1"},"reference-count":29,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T00:00:00Z","timestamp":1790035200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100014735","name":"JKA Foundation","doi-asserted-by":"publisher","award":["2026M-321"],"award-info":[{"award-number":["2026M-321"]}],"id":[{"id":"10.13039\/501100014735","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS One"],"abstract":"<jats:p>Computational capacity and knowledge of humans are more limited than those of large language models (LLMs). However, in buzzer quizzes, human trivia experts can often identify the correct answer even from insufficient information such as only a few words. Investigating how they can make fast and accurate judgments is expected to highlight new characteristics of human intelligence, but little is known about that. In this exploratory and case-based analysis, we predicted that trivia experts and LLMs would differ in which words\/phrases in a question are important for identifying the answer, and compared experts\u2019 performance with LLMs\u2019 performance in Japanese trivia questions through an LLM-as-a-judge approach: We regarded LLMs as evaluators and then used their outputs as comparative tools for experts\u2019 evaluations. First, we constructed a quiz question processing system that tokenized question texts based on morphological analysis and then numerically evaluated the importance of each token using GPT-4o\/GPT-4.1. Then, we conducted a behavioral experiment wherein actual trivia experts were asked to numerically evaluate the importance of each token, just as LLMs had performed. As a result, trivia experts treated a variety of words\/phrases as important, even if each word\/phrase did not appear to be strongly associated with the answer. Their evaluation scores tended to accumulate faster than those of the LLMs. This may indicate that trivia experts can make inductive inferences faster (e.g., finding a common concept even from few items). More advanced question-answering systems may be designed by applying trivia experts\u2019 cognitive processes to LLMs\u2019 information processing, and our findings may provide a scaffolding toward such goals.<\/jats:p>","DOI":"10.1371\/journal.pone.0358332","type":"journal-article","created":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T17:30:27Z","timestamp":1790098227000},"page":"e0358332","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["Attempts to investigate what words trivia experts focus on in quiz questions: Human-AI comparison through \u201cLLM-as-a-judge\u201d approach"],"prefix":"10.1371","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9665-3863","authenticated-orcid":true,"given":"Masaru","family":"Shirasuna","sequence":"first","affiliation":[{"name":"Shizuoka University","place":["Hamamatsu-shi, Shizuoka, Japan"],"department":["Faculty of Informatics"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuto","family":"Yoshida","sequence":"additional","affiliation":[{"name":"Shizuoka University","place":["Hamamatsu-shi, Shizuoka, Japan"],"department":["Faculty of Informatics"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,9,22]]},"reference":[{"key":"pone.0358332.ref001","author":"A Chatterjee","year":"2025"},{"issue":"11","key":"pone.0358332.ref002","doi-asserted-by":"crossref","first-page":"873","DOI":"10.1016\/j.tics.2020.09.001","article-title":"Understanding human intelligence through human limitations","volume":"24","author":"TL Griffiths","year":"2020","journal-title":"Trends Cogn Sci"},{"issue":"1","key":"pone.0358332.ref003","doi-asserted-by":"crossref","first-page":"451","DOI":"10.1146\/annurev-psych-120709-145346","article-title":"Heuristic decision making","volume":"62","author":"G Gigerenzer","year":"2011","journal-title":"Annu Rev Psychol"},{"key":"pone.0358332.ref004","article-title":"Resource-rational analysis: understanding human cognition as the optimal use of limited computational resources","volume":"43","author":"F Lieder","year":"2019","journal-title":"Behav Brain Sci"},{"issue":"5","key":"pone.0358332.ref005","doi-asserted-by":"crossref","first-page":"839","DOI":"10.1177\/17456916231180597","article-title":"Psychological AI: designing algorithms informed by human psychology","volume":"19","author":"G Gigerenzer","year":"2024","journal-title":"Perspect Psychol Sci"},{"key":"pone.0358332.ref006","unstructured":"Izawa T. Deconstruction of quiz thinking. Asahi Shimbun Publications Inc.; 2021."},{"key":"pone.0358332.ref007","unstructured":"Ichikawa T. The Utopia of quiz: the culture of amateur quiz competitions. In: Eureka. Seidosha; 2020. pp. 121\u20139."},{"key":"pone.0358332.ref008","doi-asserted-by":"crossref","unstructured":"Shirasuna M. The evidence accumulation-based framework to explain human cognitive processes for considering quiz questions: a pilot study in Japan; 2024 [cited 2025 Nov 16]. Database: PsyArXiv [Internet]. Available from: https:\/\/osf.io\/preprints\/psyarxiv\/8q9u3_v1","DOI":"10.31234\/osf.io\/8q9u3"},{"issue":"2","key":"pone.0358332.ref009","first-page":"352","article-title":"Judgments under uncertainty in a buzzer quiz: a case study based on behavioral data in a quiz competition.","volume":"31","author":"M Shirasuna","year":"2024","journal-title":"Cogn Stud Bull Jpn Cogn Sci Soc"},{"key":"pone.0358332.ref010","unstructured":"Tamura T. Taming premonition: a phenomenological study of competitive quiz. In: Eureka. Seidosha; 2020. pp. 95\u2013103."},{"issue":"2","key":"pone.0358332.ref011","doi-asserted-by":"crossref","first-page":"292","DOI":"10.3758\/s13421-018-0869-6","article-title":"Association and response accuracy in the wild","volume":"47","author":"S Bhatia","year":"2019","journal-title":"Mem Cognit"},{"key":"pone.0358332.ref012","author":"P Rodriguez","year":"2019"},{"issue":"1","key":"pone.0358332.ref013","doi-asserted-by":"crossref","first-page":"47","DOI":"10.5715\/jnlp.31.47","article-title":"Achievements and challenges in Japanese question answering: insights from quiz competition results","volume":"31","author":"T Ariyama","year":"2024","journal-title":"J Nat Lang Process"},{"issue":"4","key":"pone.0358332.ref014","doi-asserted-by":"crossref","first-page":"1867","DOI":"10.3758\/s13423-024-02469-5","article-title":"Episodic-semantic linkage for $1000: new semantic knowledge is more strongly coupled with episodic memory in trivia experts","volume":"31","author":"MK Thieu","year":"2024","journal-title":"Psychon Bull Rev"},{"key":"pone.0358332.ref015","article-title":"A Survey on LLM-as-a-Judge","author":"J Gu","year":"2025","journal-title":"ArXiv"},{"key":"pone.0358332.ref016","author":"M Gao","year":"2023"},{"key":"pone.0358332.ref017","author":"L Zheng","year":"2023"},{"key":"pone.0358332.ref018","author":"L Zhu","year":"2025"},{"issue":"1","key":"pone.0358332.ref019","doi-asserted-by":"crossref","first-page":"32290","DOI":"10.1038\/s41598-025-17188-7","article-title":"Evaluating the ability of large language models to predict human social decisions","volume":"15","author":"F Xiao","year":"2025","journal-title":"Sci Rep"},{"key":"pone.0358332.ref020","article-title":"Quantifying uncert-AI-nty: testing the accuracy of llms\u2019 confidence judgments","author":"TN Cash","year":"2025","journal-title":"Mem Cognit"},{"issue":"42","key":"pone.0358332.ref021","doi-asserted-by":"crossref","DOI":"10.1073\/pnas.2518443122","article-title":"The simulation of judgment in LLMs","volume":"122","author":"E Loru","year":"2025","journal-title":"Proc Natl Acad Sci U S A"},{"key":"pone.0358332.ref022","author":"T Hagendorff","year":"2024"},{"key":"pone.0358332.ref023","first-page":"230","author":"T Kudo","year":"2004","journal-title":"Applying conditional random fields to japanese morphological analysis. In: Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics"},{"issue":"1","key":"pone.0358332.ref024","doi-asserted-by":"crossref","first-page":"24956","DOI":"10.1038\/s41598-025-09850-x","article-title":"Effect of interacting minds on response criterion in two person perceptual decision making","volume":"15","author":"T Kawachi","year":"2025","journal-title":"Sci Rep"},{"issue":"1","key":"pone.0358332.ref025","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v080.i01","article-title":"brms: an R package for Bayesian multilevel models using Stan","volume":"80","author":"PC B\u00fcrkner","year":"2017","journal-title":"J Stat Softw"},{"issue":"7","key":"pone.0358332.ref026","doi-asserted-by":"crossref","first-page":"2364","DOI":"10.1111\/cogs.12668","article-title":"How do creative experts practice new skills? Exploratory practice in breakdancers","volume":"42","author":"D Shimizu","year":"2018","journal-title":"Cogn Sci"},{"issue":"7","key":"pone.0358332.ref027","doi-asserted-by":"crossref","first-page":"1092","DOI":"10.1177\/1747021820918533","article-title":"A kind of magic: enhanced detection of pantomimed grasps in professional magicians","volume":"73","author":"D Quarona","year":"2020","journal-title":"Q J Exp Psychol (Hove)"},{"key":"pone.0358332.ref028","article-title":"Humans or LLMs as the Judge? A study on judgement biases","author":"GH Chen","year":"2024","journal-title":"Database: ArXiv"},{"key":"pone.0358332.ref029","doi-asserted-by":"crossref","first-page":"4823","DOI":"10.1007\/s00500-025-10854-y","article-title":"A comprehensive review of frugal artificial intelligence: challenges, applications, and the road to sustainable AI","volume":"29","author":"N Girdhar","year":"2025","journal-title":"Soft Comput"}],"container-title":["PLOS One"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0358332","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T17:30:35Z","timestamp":1790098235000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0358332"}},"subtitle":[],"editor":[{"given":"Musa","family":"Aydin","sequence":"first","affiliation":[{"name":"Samsun University: Samsun Universitesi","place":["T\u00dcRKIYE"]}],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,9,22]]},"references-count":29,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2026,9,22]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0358332","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,22]]}}}