{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T09:59:34Z","timestamp":1775815174609,"version":"3.50.1"},"reference-count":41,"publisher":"Association for Computing Machinery (ACM)","issue":"9","license":[{"start":{"date-parts":[[2021,8,24]],"date-time":"2021-08-24T00:00:00Z","timestamp":1629763200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"DARPA, MCS program","award":["N66001-19-2-4031"],"award-info":[{"award-number":["N66001-19-2-4031"]}]},{"name":"NSF","award":["IIS-1524371, IIS-1714566"],"award-info":[{"award-number":["IIS-1524371, IIS-1714566"]}]},{"name":"DARPA, the CwC program","award":["W911NF-15-1-0543"],"award-info":[{"award-number":["W911NF-15-1-0543"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Commun. ACM"],"published-print":{"date-parts":[[2021,9]]},"abstract":"<jats:p>Commonsense reasoning remains a major challenge in AI, and yet, recent progresses on benchmarks may seem to suggest otherwise. In particular, the recent neural language models have reported above 90% accuracy on the Winograd Schema Challenge (WSC), a commonsense benchmark originally designed to be unsolvable for statistical models that rely simply on word associations. This raises an important question---whether these models have truly acquired robust commonsense capabilities or they rely on spurious biases in the dataset that lead to an overestimation of the true capabilities of machine commonsense.<\/jats:p>\n          <jats:p>\n            To investigate this question, we introduce WinoGrande, a large-scale dataset of 44k problems, inspired by the original WSC, but adjusted to improve both the scale and the hardness of the dataset. The key steps of the dataset construction consist of (1) large-scale crowdsourcing, followed by (2) systematic bias reduction using a novel AFLITE algorithm that generalizes human-detectable\n            <jats:italic>word associations<\/jats:italic>\n            to machine-detectable\n            <jats:italic>embedding associations.<\/jats:italic>\n            Our experiments demonstrate that state-of-the-art models achieve considerably lower accuracy (59.4%-79.1%) on WINOGRANDE compared to humans (94%), confirming that the high performance on the original WSC was inflated by spurious biases in the dataset.\n          <\/jats:p>\n          <jats:p>Furthermore, we report new state-of-the-art results on five related benchmarks with emphasis on their dual implications. On the one hand, they demonstrate the effectiveness of WINOGRANDE when used as a resource for transfer learning. On the other hand, the high performance on all these benchmarks suggests the extent to which spurious biases are prevalent in all such datasets, which motivates further research on algorithmic bias reduction.<\/jats:p>","DOI":"10.1145\/3474381","type":"journal-article","created":{"date-parts":[[2021,8,24]],"date-time":"2021-08-24T14:28:46Z","timestamp":1629815326000},"page":"99-106","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":232,"title":["WinoGrande"],"prefix":"10.1145","volume":"64","author":[{"given":"Keisuke","family":"Sakaguchi","sequence":"first","affiliation":[{"name":"Allen Institute for AI, Seattle, WA"}]},{"given":"Ronan Le","family":"Bras","sequence":"additional","affiliation":[{"name":"Allen Institute for AI, Seattle, WA"}]},{"given":"Chandra","family":"Bhagavatula","sequence":"additional","affiliation":[{"name":"Allen Institute for AI, Seattle, WA"}]},{"given":"Yejin","family":"Choi","sequence":"additional","affiliation":[{"name":"University of Washington &amp; Allen Institute for AI, Seattle, WA"}]}],"member":"320","published-online":{"date-parts":[[2021,8,24]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"On adversarial removal of hypothesis-only bias in natural language inference. *SEM","author":"Belinkov Y.","year":"2019","unstructured":"Belinkov , Y. , Poliak , A. , Shieber , S. , Van Durme , B. , Rush , A. On adversarial removal of hypothesis-only bias in natural language inference. *SEM ( 2019 ), 256--262. Belinkov, Y., Poliak, A., Shieber, S., Van Durme, B., Rush, A. On adversarial removal of hypothesis-only bias in natural language inference. *SEM (2019), 256--262."},{"key":"e_1_2_1_2_1","volume-title":"Establishing a human baseline for the winograd schema challenge. MAICS","author":"Bender D.","year":"2015","unstructured":"Bender , D. Establishing a human baseline for the winograd schema challenge. MAICS ( 2015 ), 30--45. Bender, D. Establishing a human baseline for the winograd schema challenge. MAICS (2015), 30--45."},{"key":"e_1_2_1_3_1","volume-title":"Multinomial adversarial networks for multi-domain text classification. NAACL","author":"Chen X.","year":"2018","unstructured":"Chen , X. , Cardie , C. Multinomial adversarial networks for multi-domain text classification. NAACL ( 2018 ), 1226--1240. Chen, X., Cardie, C. Multinomial adversarial networks for multi-domain text classification. NAACL (2018), 1226--1240."},{"key":"e_1_2_1_4_1","volume-title":"Deep reinforcement learning for mention-ranking coreference models. EMNLP","author":"Clark K.","year":"2016","unstructured":"Clark , K. , Manning , C.D. Deep reinforcement learning for mention-ranking coreference models. EMNLP ( 2016 ), 2256--2262. Clark, K., Manning, C.D. Deep reinforcement learning for mention-ranking coreference models. EMNLP (2016), 2256--2262."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2701413"},{"key":"e_1_2_1_6_1","volume-title":"Human tests of materials for the winograd schema challenge Unpublished manuscript","author":"Davis E.","year":"2016","unstructured":"Davis , E. , Morgenstern , L. , Ortiz , C. Human tests of materials for the winograd schema challenge Unpublished manuscript ( 2016 ). https:\/\/cs.nyu.edu\/faculty\/davise\/papers\/WS2016SubjectTests.pdf, 2016. Davis, E., Morgenstern, L., Ortiz, C. Human tests of materials for the winograd schema challenge Unpublished manuscript (2016). https:\/\/cs.nyu.edu\/faculty\/davise\/papers\/WS2016SubjectTests.pdf, 2016."},{"key":"e_1_2_1_7_1","volume-title":"Toutanova","author":"Devlin J.","year":"1810","unstructured":"Devlin , J. , Chang , M.-W. , Lee , K. , Toutanova , K. BERT : Pre-training of de ep bidirectional transformers for language understanding. arXiv: 1810 .04805 (2018). Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 (2018)."},{"key":"e_1_2_1_8_1","volume-title":"Easy victories and uphill battles in coreference resolution. EMNLP","author":"Durrett G.","year":"2013","unstructured":"Durrett , G. , Klein , D. Easy victories and uphill battles in coreference resolution. EMNLP ( 2013 ), 1971--1982. Durrett, G., Klein, D. Easy victories and uphill battles in coreference resolution. EMNLP (2013), 1971--1982."},{"key":"e_1_2_1_9_1","volume-title":"Adversarial removal of demographic attributes from text data. EMNLP","author":"Elazar Y.","year":"2018","unstructured":"Elazar , Y. , Goldberg , Y. Adversarial removal of demographic attributes from text data. EMNLP ( 2018 ), 11--21. Elazar, Y., Goldberg, Y. Adversarial removal of demographic attributes from text data. EMNLP (2018), 11--21."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-4004"},{"key":"e_1_2_1_11_1","volume-title":"Removing gender and number cues for difficult pronominal anaphora resolution. ACL","author":"Emami A.","year":"2019","unstructured":"Emami , A. , Trichelair , P. , Trischler , A. , Suleman , K. , Schulz , H. , Cheung , J.C.K The KnowRef coreference corpus : Removing gender and number cues for difficult pronominal anaphora resolution. ACL ( 2019 ), 3952--3961. Emami, A., Trichelair, P., Trischler, A., Suleman, K., Schulz, H., Cheung, J.C.K The KnowRef coreference corpus: Removing gender and number cues for difficult pronominal anaphora resolution. ACL (2019), 3952--3961."},{"key":"e_1_2_1_12_1","volume-title":"Are we modeling the task or the annotator? An investigation of annotator bias in natural language understanding datasets. arXiv:1908.07898","author":"Geva M.","year":"2019","unstructured":"Geva , M. , Goldberg , Y. , Berant , J. Are we modeling the task or the annotator? An investigation of annotator bias in natural language understanding datasets. arXiv:1908.07898 ( 2019 ). Geva, M., Goldberg, Y., Berant, J. Are we modeling the task or the annotator? An investigation of annotator bias in natural language understanding datasets. arXiv:1908.07898 (2019)."},{"key":"e_1_2_1_13_1","volume-title":"SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. *SEM","author":"Gordon A.","year":"2012","unstructured":"Gordon , A. , Kozareva , Z. , Roemmele , M. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. *SEM ( 2012 ), 394--398. Gordon, A., Kozareva, Z., Roemmele, M. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. *SEM (2012), 394--398."},{"key":"e_1_2_1_14_1","volume-title":"Commonsense causal reasoning using millions of personal stories. AAAI","author":"Gordon A.S.","year":"2011","unstructured":"Gordon , A.S. , Bejan , C.A. , Sagae , K. Commonsense causal reasoning using millions of personal stories. AAAI ( 2011 ), 1180--1185. Gordon, A.S., Bejan, C.A., Sagae, K. Commonsense causal reasoning using millions of personal stories. AAAI (2011), 1180--1185."},{"key":"e_1_2_1_15_1","volume-title":"Reporting bias and knowledge acquisition. AKBC","author":"Gordon J.","year":"2013","unstructured":"Gordon , J. , van Durme , B. Reporting bias and knowledge acquisition. AKBC ( 2013 ), 25--30. Gordon, J., van Durme, B. Reporting bias and knowledge acquisition. AKBC (2013), 25--30."},{"key":"e_1_2_1_16_1","volume-title":"Annotation artifacts in natural language inference data. NAACL","author":"Gururangan S.","year":"2018","unstructured":"Gururangan , S. , Swayamdipta , S. , Levy , O. , Schwartz , R. , Bowman , S. , Smith , N.A. Annotation artifacts in natural language inference data. NAACL ( 2018 ), 107--112. Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., Smith, N.A. Annotation artifacts in natural language inference data. NAACL (2018), 107--112."},{"key":"e_1_2_1_17_1","volume-title":"A hybrid neural network model for commonsense reasoning. arXiv:1907.11983","author":"He P.","year":"2019","unstructured":"He , P. , Liu , X. , Chen , W. , Gao , J. A hybrid neural network model for commonsense reasoning. arXiv:1907.11983 ( 2019 ). He, P., Liu, X., Chen, W., Gao, J. A hybrid neural network model for commonsense reasoning. arXiv:1907.11983 (2019)."},{"key":"e_1_2_1_18_1","doi-asserted-by":"crossref","unstructured":"Khashabi D. Khot T. Sabharwal A. Tafjord O. Clark P. Hajishirzi H. Unifiedqa: Crossing format boundaries with a single qa system. arXiv preprint arXiv:2005.00700 (2020).  Khashabi D. Khot T. Sabharwal A. Tafjord O. Clark P. Hajishirzi H. Unifiedqa: Crossing format boundaries with a single qa system. arXiv preprint arXiv:2005.00700 (2020).","DOI":"10.18653\/v1\/2020.findings-emnlp.171"},{"key":"e_1_2_1_19_1","volume-title":"A surprisingly robust trick for the winograd schema challenge. ACL","author":"Kocijan V.","year":"2019","unstructured":"Kocijan , V. , Cretu , A.-M. , Camburu , O.-M. , Yordanov , Y. , Lukasiewicz , T. A surprisingly robust trick for the winograd schema challenge. ACL ( 2019 ), 4837--4842. Kocijan, V., Cretu, A.-M., Camburu, O.-M., Yordanov, Y., Lukasiewicz, T. A surprisingly robust trick for the winograd schema challenge. ACL (2019), 4837--4842."},{"key":"e_1_2_1_20_1","volume-title":"Adversarial filters of dataset biases. ICML","author":"Le Bras R.","year":"2020","unstructured":"Le Bras , R. , Swayamdipta , S. , Bhagavatula , C. , Zellers , R. , Peters , M. , Sabharwal , A. , Choi , Y. Adversarial filters of dataset biases. ICML ( 2020 ). Le Bras, R., Swayamdipta, S., Bhagavatula, C., Zellers, R., Peters, M., Sabharwal, A., Choi, Y. Adversarial filters of dataset biases. ICML (2020)."},{"key":"e_1_2_1_21_1","volume-title":"Stanford's multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. CoNLL: Shared Task","author":"Lee H.","year":"2011","unstructured":"Lee , H. , Peirsman , Y. , Chang , A. , Chambers , N. , Surdeanu , M. , Jurafsky , D. Stanford's multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. CoNLL: Shared Task ( 2011 ). Lee, H., Peirsman, Y., Chang, A., Chambers, N., Surdeanu, M., Jurafsky, D. Stanford's multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. CoNLL: Shared Task (2011)."},{"key":"e_1_2_1_22_1","volume-title":"AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning","author":"Levesque H.J.","year":"2011","unstructured":"Levesque , H.J. , Davis , E. , Morgenstern , L. The winograd schema challenge . In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning ( 2011 ). Levesque, H.J., Davis, E., Morgenstern, L. The winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning (2011)."},{"key":"e_1_2_1_23_1","volume-title":"Tttttackling winogrande schemas. arXiv preprint arXiv:2003.08380","author":"Lin S.-C.","year":"2020","unstructured":"Lin , S.-C. , Yang , J.-H. , Nogueira , R. , Tsai , M.-F. , Wang , C.-J. , Lin , J. Tttttackling winogrande schemas. arXiv preprint arXiv:2003.08380 ( 2020 ). Lin, S.-C., Yang, J.-H., Nogueira, R., Tsai, M.-F., Wang, C.-J., Lin, J. Tttttackling winogrande schemas. arXiv preprint arXiv:2003.08380 (2020)."},{"key":"e_1_2_1_24_1","volume-title":"Commonsense knowledge enhanced embeddings for solving pronoun disambiguation problems in winograd schema challenge. arXiv:1611.04146","author":"Liu Q.","year":"2016","unstructured":"Liu , Q. , Jiang , H. , Ling , Z.-H. , Zhu , X. , Wei , S. , Hu , Y. Commonsense knowledge enhanced embeddings for solving pronoun disambiguation problems in winograd schema challenge. arXiv:1611.04146 ( 2016 ). Liu, Q., Jiang, H., Ling, Z.-H., Zhu, X., Wei, S., Hu, Y. Commonsense knowledge enhanced embeddings for solving pronoun disambiguation problems in winograd schema challenge. arXiv:1611.04146 (2016)."},{"key":"e_1_2_1_25_1","volume-title":"Roberta: A robustly optimized bert pretraining approach. ArXiv, abs\/1907.11692","author":"Liu Y.","year":"2019","unstructured":"Liu , Y. , Ott , M. , Goyal , N. , Du , J. , Joshi , M.S. , Chen , D. , Levy , O. , Lewis , M. , Zettlemoyer , L.S. , Stoyanov , V. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs\/1907.11692 ( 2019 ). Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M.S., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L.S., Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs\/1907.11692 (2019)."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v37i1.2639"},{"key":"e_1_2_1_27_1","volume-title":"Probing neural network comprehension of natural language arguments. ACL","author":"Niven T.","year":"2019","unstructured":"Niven , T. , Kao , H.-Y. Probing neural network comprehension of natural language arguments. ACL ( 2019 ), 4658--4664. Niven, T., Kao, H.-Y. Probing neural network comprehension of natural language arguments. ACL (2019), 4658--4664."},{"key":"e_1_2_1_28_1","volume-title":"Solving hard coreference problems. NAACL","author":"Peng H.","year":"2015","unstructured":"Peng , H. , Khashabi , D. , Roth , D. Solving hard coreference problems. NAACL ( 2015 ), 809--819. Peng, H., Khashabi, D., Roth, D. Solving hard coreference problems. NAACL (2015), 809--819."},{"key":"e_1_2_1_29_1","volume-title":"Hypothesis only baselines in natural language inference. *SEM","author":"Poliak A.","year":"2018","unstructured":"Poliak , A. , Naradowsky , J. , Haldar , A. , Rudinger , R. , Van Durme , B. Hypothesis only baselines in natural language inference. *SEM ( 2018 ), 180--191. Poliak, A., Naradowsky, J., Haldar, A., Rudinger, R., Van Durme, B. Hypothesis only baselines in natural language inference. *SEM (2018), 180--191."},{"key":"e_1_2_1_30_1","volume-title":"Language models are unsupervised multitask learners. OpenAI Blog","author":"Radford A.","year":"2019","unstructured":"Radford , A. , Wu , J. , Child , R. , Luan , D. , Amodei , D. , Sutskever , I. Language models are unsupervised multitask learners. OpenAI Blog ( 2019 ), 777--789. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I. Language models are unsupervised multitask learners. OpenAI Blog (2019), 777--789."},{"key":"e_1_2_1_31_1","volume-title":"Resolving complex cases of definite pronouns: The winograd schema challenge. EMNLP-CoNLL","author":"Rahman A.","year":"2012","unstructured":"Rahman , A. , Ng , V. Resolving complex cases of definite pronouns: The winograd schema challenge. EMNLP-CoNLL ( 2012 ). Rahman, A., Ng, V. Resolving complex cases of definite pronouns: The winograd schema challenge. EMNLP-CoNLL (2012)."},{"key":"e_1_2_1_32_1","volume-title":"AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning","author":"Roemmele M.","year":"2011","unstructured":"Roemmele , M. , Bejan , C.A. , Gordon , A.S. Choice of plausible alternatives: An evaluation of commonsense causal reasoning . In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning ( 2011 ). Roemmele, M., Bejan, C.A., Gordon, A.S. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning (2011)."},{"key":"e_1_2_1_33_1","volume-title":"Gender bias in coreference resolution. NAACL","author":"Rudinger R.","year":"2018","unstructured":"Rudinger , R. , Naradowsky , J. , Leonard , B. , Van Durme , B. Gender bias in coreference resolution. NAACL ( 2018 ), 15--20. Rudinger, R., Naradowsky, J., Leonard, B., Van Durme, B. Gender bias in coreference resolution. NAACL (2018), 15--20."},{"key":"e_1_2_1_34_1","volume-title":"Handling multiword expressions in causality estimation. IWCS","author":"Sasaki S.","year":"2017","unstructured":"Sasaki , S. , Takase , S. , Inoue , N. , Okazaki , N. , Inui , K. Handling multiword expressions in causality estimation. IWCS ( 2017 ). Sasaki, S., Takase, S., Inoue, N., Okazaki, N., Inui, K. Handling multiword expressions in causality estimation. IWCS (2017)."},{"key":"e_1_2_1_35_1","volume-title":"Creativity from Constraints: The Psychology of Breakthrough","author":"Stokes P.D.","year":"2005","unstructured":"Stokes , P.D. Creativity from Constraints: The Psychology of Breakthrough . Springer Publishing Company , New York, NY , 2005 . Stokes, P.D. Creativity from Constraints: The Psychology of Breakthrough. Springer Publishing Company, New York, NY, 2005."},{"key":"e_1_2_1_36_1","volume-title":"On the evaluation of commonsense reasoning in natural language understanding. arXiv:1811.01778","author":"Trichelair P.","year":"2018","unstructured":"Trichelair , P. , Emami , A. , Cheung , J.C.K. , Trischler , A. , Suleman , K. , Diaz , F. On the evaluation of commonsense reasoning in natural language understanding. arXiv:1811.01778 ( 2018 ). Trichelair, P., Emami, A., Cheung, J.C.K., Trischler, A., Suleman, K., Diaz, F. On the evaluation of commonsense reasoning in natural language understanding. arXiv:1811.01778 (2018)."},{"key":"e_1_2_1_37_1","volume-title":"A simple method for commonsense reasoning. arXiv:1806.02847","author":"Trinh T.H.","year":"2018","unstructured":"Trinh , T.H. , Le , Q.V. A simple method for commonsense reasoning. arXiv:1806.02847 ( 2018 ). Trinh, T.H., Le, Q.V. A simple method for commonsense reasoning. arXiv:1806.02847 (2018)."},{"key":"e_1_2_1_38_1","volume-title":"Performance impact caused by hidden bias of training data for recognizing textual entailment. LREC","author":"Tsuchiya M.","year":"2018","unstructured":"Tsuchiya , M. Performance impact caused by hidden bias of training data for recognizing textual entailment. LREC ( 2018 ), 1506--1511. Tsuchiya, M. Performance impact caused by hidden bias of training data for recognizing textual entailment. LREC (2018), 1506--1511."},{"key":"e_1_2_1_39_1","first-page":"433","article-title":"Computing machinery and intelligence","volume":"59","author":"Turing A.M","unstructured":"Turing , A.M . Computing machinery and intelligence . Mind 59 , 236(1950), 433 -- 460 . Turing, A.M. Computing machinery and intelligence. Mind 59, 236(1950), 433--460.","journal-title":"Mind"},{"key":"e_1_2_1_40_1","volume-title":"Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv:1905.00537","author":"Wang A.","year":"2019","unstructured":"Wang , A. , Pruksachatkun , Y. , Nangia , N. , Singh , A. , Michael , J. , Hill , F. , Levy , O. , Bowman , S.R. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv:1905.00537 ( 2019 ). Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.R. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv:1905.00537 (2019)."},{"key":"e_1_2_1_41_1","volume-title":"Swag: A large-scale adversarial dataset for grounded commonsense inference. EMNLP","author":"Zellers R.","year":"2018","unstructured":"Zellers , R. , Bisk , Y. , Schwartz , R. , Choi , Y. Swag: A large-scale adversarial dataset for grounded commonsense inference. EMNLP ( 2018 ), 93--104. Zellers, R., Bisk, Y., Schwartz, R., Choi, Y. Swag: A large-scale adversarial dataset for grounded commonsense inference. EMNLP (2018), 93--104."}],"container-title":["Communications of the ACM"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3474381","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3474381","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3474381","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:49Z","timestamp":1750191529000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3474381"}},"subtitle":["an adversarial winograd schema challenge at scale"],"short-title":[],"issued":{"date-parts":[[2021,8,24]]},"references-count":41,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2021,9]]}},"alternative-id":["10.1145\/3474381"],"URL":"https:\/\/doi.org\/10.1145\/3474381","relation":{},"ISSN":["0001-0782","1557-7317"],"issn-type":[{"value":"0001-0782","type":"print"},{"value":"1557-7317","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8,24]]},"assertion":[{"value":"2021-08-24","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}