{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,2]],"date-time":"2026-09-02T14:31:05Z","timestamp":1788359465920,"version":"build-2803163510"},"reference-count":38,"publisher":"Public Library of Science (PLoS)","issue":"12","license":[{"start":{"date-parts":[[2024,12,5]],"date-time":"2024-12-05T00:00:00Z","timestamp":1733356800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"The Scientific and Technological Research Council of Turkiye","award":["1059B192202678, 3211054"],"award-info":[{"award-number":["1059B192202678, 3211054"]}]},{"DOI":"10.13039\/100000055","name":"National Institute on Deafness and Other Communication Disorders","doi-asserted-by":"publisher","award":["R01 DC020715"],"award-info":[{"award-number":["R01 DC020715"]}],"id":[{"id":"10.13039\/100000055","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"publisher","award":["R21 CA273665"],"award-info":[{"award-number":["R21 CA273665"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"crossref","award":["R25 LM014214"],"award-info":[{"award-number":["R25 LM014214"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:p>Breast cancer, a widespread and life-threatening disease, necessitates precise diagnostic tools for improved patient outcomes. Tumor-Infiltrating Lymphocytes (TILs), reflective of the immune response against cancer cells, are pivotal in understanding breast cancer behavior. However, inter-observer variability in TILs scoring methods poses challenges to reliable assessments. This study introduces a novel and interpretable proof-of-principle framework comprising two innovative inter-observer agreement measures. The first method, Boundary-Weighted Fleiss\u2019 Kappa (BWFK), addresses tissue segmentation predictions, focusing on mitigating disagreements along tissue boundaries. BWFK enhances the accuracy of stromal segmentation, providing a nuanced assessment of inter-observer agreement. The second proposed method, the Distance Based Cell Agreement Algorithm (DBCAA), eliminates the need for ground truth annotations in cell detection predictions. This innovative approach offers versatility across histopathological analyses, overcoming data availability challenges. Both methods were applied to assess inter-observer agreement using a clinical image dataset consisting of 25 images of invasive ductal breast carcinoma tissue, each annotated by four pathologists, serving as a proof-of-principle. Experimental investigations demonstrated that the BWFK method yielded gains of up to 32% compared to the standard Fleiss\u2019 Kappa model. Furthermore, a procedure for conducting clinical validations of artificial intelligence (AI) based cell detection methods was elucidated. Thoroughly validated on a clinical dataset, the framework contributes to standardized, reliable, and interpretable inter-observer agreement assessments. This study is the first examination of inter-observer agreements in stromal segmentation and lymphocyte detection for the TILs scoring problem. The study emphasizes the potential impact of these measures in advancing histopathological image analysis, fostering consensus in TILs scoring, and ultimately improving breast cancer diagnostics and treatment planning. The source code and implementation guide for this study are accessible on our GitHub page, and the full clinical dataset is available for academic and research purposes on Kaggle.<\/jats:p>","DOI":"10.1371\/journal.pone.0314450","type":"journal-article","created":{"date-parts":[[2024,12,5]],"date-time":"2024-12-05T13:40:16Z","timestamp":1733406016000},"page":"e0314450","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":5,"title":["An interpretable framework for inter-observer agreement measurements in TILs scoring on histopathological breast images: A proof-of-principle study"],"prefix":"10.1371","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7110-5569","authenticated-orcid":true,"given":"Abdulkerim","family":"Capar","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dursun Ali","family":"Ekinci","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-5500-4255","authenticated-orcid":true,"given":"Mucahit","family":"Ertano","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"M. Khalid Khan","family":"Niazi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Erva Bengu","family":"Balaban","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ibrahim","family":"Aloglu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Meryem","family":"Dogan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9600-3078","authenticated-orcid":true,"given":"Ziyu","family":"Su","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fugen Vardar","family":"Aker","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Metin Nafi","family":"Gurcan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2024,12,5]]},"reference":[{"key":"pone.0314450.ref001","doi-asserted-by":"crossref","first-page":"475","DOI":"10.1007\/s00223-021-00931-3","article-title":"Expression Profiling of Selected Immune Genes and Trabecular Microarchitecture in Breast Cancer Skeletal Metastases Model: Effect of \u03b1\u2013Tocopherol Acetate Supplementation","volume":"110","author":"R Badraoui","year":"2022","journal-title":"Calcif Tissue Int"},{"key":"pone.0314450.ref002","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s40425-016-0165-6","article-title":"Clinical significance of tumor-infiltrating lymphocytes in breast cancer","volume":"4","author":"SE Stanton","year":"2016","journal-title":"J Immunother Cancer"},{"key":"pone.0314450.ref003","first-page":"1003","article-title":"Allium subhirsutum L. as a Potential Source of Antioxidant and Anticancer Bioactive Molecules: HR-LCMS Phytochemical Profiling","volume":"9","author":"R Badraoui","year":"2020","journal-title":"In Vitro and In Vivo Pharmacological Study. Antioxidants"},{"key":"pone.0314450.ref004","doi-asserted-by":"crossref","first-page":"223","DOI":"10.3390\/cells10020223","article-title":"Immune infiltrates in breast cancer: recent updates and clinical implications","volume":"10","author":"MV Dieci","year":"2021","journal-title":"Cells"},{"key":"pone.0314450.ref005","author":"Roberto Salgado","journal-title":"TILs in Breast Cancer"},{"key":"pone.0314450.ref006","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1093\/annonc\/mdu450","article-title":"The evaluation of tumor-infiltrating lymphocytes (TILs) in breast cancer: recommendations by an International TILs Working Group 2014","volume":"26","author":"R Salgado","year":"2015","journal-title":"Annals of Oncology"},{"key":"pone.0314450.ref007","doi-asserted-by":"crossref","first-page":"1155","DOI":"10.1038\/modpathol.2016.109","article-title":"Standardized evaluation of tumor-infiltrating lymphocytes in breast cancer: results of the ring studies of the international immuno-oncology biomarker working group","volume":"29","author":"C Denkert","year":"2016","journal-title":"Modern Pathology"},{"key":"pone.0314450.ref008","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1097\/PAP.0000000000000162","article-title":"Assessing Tumor-infiltrating Lymphocytes in Solid Tumors: A Practical Review for Pathologists and Proposal for a Standardized Method From the International Immunooncology Biomarkers Working Group: Part 1: Assessing the Host Immune Response, TILs in Invasive Breast Carcinoma and Ductal Carcinoma In Situ, Metastatic Tumor Deposits and Areas for Further Research","volume":"24","author":"S Hendry","year":"2017","journal-title":"Adv Anat Pathol"},{"key":"pone.0314450.ref009","doi-asserted-by":"crossref","first-page":"150","DOI":"10.1038\/s41523-021-00346-1","article-title":"The tale of TILs in breast cancer: A report from The International Immuno-Oncology Biomarker Working Group","volume":"7","author":"K El Bairi","year":"2021","journal-title":"NPJ Breast Cancer"},{"key":"pone.0314450.ref010","doi-asserted-by":"crossref","first-page":"498","DOI":"10.1002\/path.6155","article-title":"Pitfalls in machine learning\u2010based assessment of tumor\u2010infiltrating lymphocytes in breast cancer: A report of the International Immuno\u2010Oncology Biomarker Working Group on Breast Cancer","volume":"260","author":"J Thagaard","year":"2023","journal-title":"J Pathol"},{"key":"pone.0314450.ref011","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1016\/j.semcancer.2018.07.001","article-title":"Scoring of tumor-infiltrating lymphocytes: From visual estimation to machine learning","volume":"52","author":"F Klauschen","year":"2018","journal-title":"Semin Cancer Biol"},{"key":"pone.0314450.ref012","doi-asserted-by":"crossref","first-page":"2130","DOI":"10.1038\/s41379-021-00865-z","article-title":"Interobserver variability in the assessment of stromal tumor-infiltrating lymphocytes (sTILs) in triple-negative invasive breast carcinoma influences the association with pathological complete response: the IVITA study","volume":"34","author":"MR Van Bockstal","year":"2021","journal-title":"Modern Pathology"},{"key":"pone.0314450.ref013","doi-asserted-by":"crossref","first-page":"2339","DOI":"10.3390\/diagnostics13142339","article-title":"Assessment of the Tumor\u2013Stroma Ratio and Tumor-Infiltrating Lymphocytes in Colorectal Cancer: Inter-Observer Agreement Evaluation.","volume":"13","author":"A Kazemi","year":"2023","journal-title":"Diagnostics"},{"key":"pone.0314450.ref014","doi-asserted-by":"crossref","first-page":"103492","DOI":"10.1016\/j.ebiom.2021.103492","article-title":"A Computational Tumor-Infiltrating Lymphocyte Assessment Method Comparable with Visual Reporting Guidelines for Triple-Negative Breast Cancer","volume":"70","author":"P Sun","year":"2021","journal-title":"EBioMedicine"},{"key":"pone.0314450.ref015","doi-asserted-by":"crossref","first-page":"1199","DOI":"10.3390\/cancers15041199","article-title":"ONEST (Observers Needed to Evaluate Subjective Tests) Analysis of Stromal Tumour-Infiltrating Lymphocytes (sTILs) in Breast Cancer and Its Limitations","volume":"15","author":"B Cserni","year":"2023","journal-title":"Cancers (Basel)"},{"key":"pone.0314450.ref016","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1038\/s41523-023-00577-4","article-title":"Deep learning model improves tumor-infiltrating lymphocyte evaluation and therapeutic response prediction in breast cancer","volume":"9","author":"S Choi","year":"2023","journal-title":"NPJ Breast Cancer"},{"key":"pone.0314450.ref017","doi-asserted-by":"crossref","first-page":"45","DOI":"10.4103\/jpi.jpi_83_20","article-title":"A Pathologist-Annotated Dataset for Validating Artificial Intelligence: A Project Description and Pilot Study","volume":"12","author":"SN Dudgeon","year":"2021","journal-title":"J Pathol Inform"},{"key":"pone.0314450.ref018","doi-asserted-by":"crossref","first-page":"104966","DOI":"10.1016\/j.compbiomed.2021.104966","article-title":"Comparing deep learning-based automatic segmentation of breast masses to expert interobserver variability in ultrasound imaging","volume":"139","author":"JM Webb","year":"2021","journal-title":"Comput Biol Med"},{"key":"pone.0314450.ref019","first-page":"11","article-title":"Assessing Inter-Annotator Agreement for Medical Image Segmentation","author":"F Yang","year":"2023","journal-title":"IEEE Access"},{"key":"pone.0314450.ref020","doi-asserted-by":"crossref","first-page":"915","DOI":"10.1118\/1.3528204","article-title":"The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): A Completed Reference Database of Lung Nodules on CT Scans","volume":"38","author":"SG Armato","year":"2011","journal-title":"Med Phys"},{"key":"pone.0314450.ref021","first-page":"11","article-title":"NuCLS: A scalable crowdsourcing approach and dataset for nucleus classification and segmentation in breast cancer","author":"M Amgad","year":"2022","journal-title":"Gigascience"},{"key":"pone.0314450.ref022","doi-asserted-by":"crossref","first-page":"105084","DOI":"10.1109\/ACCESS.2022.3210542","article-title":"Segmentation of Tissues and Proliferating Cells in Light-Sheet Microscopy Images of Mouse Embryos Using Convolutional Neural Networks","volume":"10","author":"LD Lo Vercio","year":"2022","journal-title":"IEEE Access"},{"key":"pone.0314450.ref023","doi-asserted-by":"crossref","first-page":"4399","DOI":"10.1038\/s41598-022-08355-1","article-title":"Cell segmentation for immunofluorescence multiplexed images using two-stage domain adaptation and weakly labeled data for pre-training","volume":"12","author":"W Han","year":"2022","journal-title":"Sci Rep"},{"key":"pone.0314450.ref024","first-page":"552","author":"C Kang","year":"2023","journal-title":"Variability Matters: Evaluating Inter-Rater Variability in Histopathology for Robust Cell Detection"},{"key":"pone.0314450.ref025","first-page":"86","article-title":"Intraclass correlations: Uses in assessing rater reliability","author":"PE Shrout","year":"1979","journal-title":"Psychol Bull"},{"key":"pone.0314450.ref026","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1016\/S0140-6736(86)90837-8","article-title":"Altman DouglasG. STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT","volume":"327","author":"J Martin Bland","year":"1986","journal-title":"The Lancet"},{"key":"pone.0314450.ref027","first-page":"570","article-title":"Difference versus Mean Plots. Annals of Clinical Biochemistry","volume":"34","author":"JM Bland","year":"1997","journal-title":"International Journal of Laboratory Medicine"},{"key":"pone.0314450.ref028","doi-asserted-by":"crossref","first-page":"1308","DOI":"10.1093\/ije\/dyr109","article-title":"A graphical method for assessing agreement with the mean between multiple observers using continuous measures","volume":"40","author":"M Jones","year":"2011","journal-title":"Int J Epidemiol"},{"key":"pone.0314450.ref029","doi-asserted-by":"crossref","first-page":"e4483","DOI":"10.1136\/bmj.e4483","article-title":"Pearson\u2019s correlation coefficient","volume":"345","author":"P. Sedgwick","year":"2012","journal-title":"BMJ"},{"key":"pone.0314450.ref030","unstructured":"Ozgun Unlu, Zeyd Ozer, Abdulkerim Capar. easypath. Jun 2022 [cited 16 Nov 2023]. Available: https:\/\/easypath.argenit.com"},{"key":"pone.0314450.ref031","author":"F H\u00f6rst","year":"2023","journal-title":"CellViT: Vision Transformers for Precise Cell Segmentation and Classification"},{"key":"pone.0314450.ref032","doi-asserted-by":"crossref","first-page":"101563","DOI":"10.1016\/j.media.2019.101563","article-title":"Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images","volume":"58","author":"S Graham","year":"2019","journal-title":"Med Image Anal"},{"key":"pone.0314450.ref033","first-page":"234","author":"O Ronneberger","year":"2015","journal-title":"U-Net: Convolutional Networks for Biomedical Image Segmentation"},{"key":"pone.0314450.ref034","doi-asserted-by":"crossref","first-page":"3461","DOI":"10.1093\/bioinformatics\/btz083","article-title":"Structured crowdsourcing enables convolutional segmentation of histology images","volume":"35","author":"M Amgad","year":"2019","journal-title":"Bioinformatics"},{"key":"pone.0314450.ref035","unstructured":"Roboflow: Give your software the power to see objects in images and video. [cited 1 Jan 2024]. Available: https:\/\/roboflow.com"},{"key":"pone.0314450.ref036","unstructured":"LabelMe. The Open annotation tool. [cited 1 Jan 2024]. Available: http:\/\/labelme.csail.mit.edu\/Release3.0"},{"key":"pone.0314450.ref037","unstructured":"Labelbox | Data-centric AI Platform for Building & Using AI. [cited 1 Jan 2024]. Available: https:\/\/labelbox.com"},{"key":"pone.0314450.ref038","unstructured":"TILs Interobserver Agreement Measures. [cited 12 May 2024]. Available: https:\/\/github.com\/cialab\/TILs_Interobserver_Agreement_Measures"}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0314450","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,5]],"date-time":"2024-12-05T13:40:38Z","timestamp":1733406038000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0314450"}},"subtitle":[],"editor":[{"given":"Semir","family":"Vranic","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2024,12,5]]},"references-count":38,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2024,12,5]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0314450","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,5]]}}}