{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T17:38:22Z","timestamp":1782927502648,"version":"3.54.5"},"reference-count":55,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2020,11,25]],"date-time":"2020-11-25T00:00:00Z","timestamp":1606262400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004561","name":"Ministry of Education and Science of the Republic of Kazakhstan","doi-asserted-by":"publisher","award":["BR05236839"],"award-info":[{"award-number":["BR05236839"]}],"id":[{"id":"10.13039\/501100004561","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Symmetry"],"abstract":"<jats:p>Mass media not only reflect the activities of state bodies but also shape the informational context, sentiment, depth, and significance level attributed to certain state initiatives and social events. Multilateral and quantitative (to the practicable extent) assessment of media activity is important for understanding their objectivity, role, focus, and, ultimately, the quality of the society\u2019s \u201cfourth power\u201d. The paper proposes a method for evaluating the media in several modalities (topics, evaluation criteria\/properties, classes), combining topic modeling of the text corpora and multiple-criteria decision making. The evaluation is based on an analysis of the corpora as follows: the conditional probability distribution of media by topics, properties, and classes is calculated after the formation of the topic model of the corpora. Several approaches are used to obtain weights that describe how each topic relates to each evaluation criterion\/property and to each class described in the paper, including manual high-level labeling, a multi-corpora approach, and an automatic approach. The proposed multi-corpora approach suggests assessment of corpora topical asymmetry to obtain the weights describing each topic\u2019s relationship to a certain criterion\/property. These weights, combined with the topic model, can be applied to evaluate each document in the corpora according to each of the considered criteria and classes. The proposed method was applied to a corpus of 804,829 news publications from 40 Kazakhstani sources published from 01 January 2018 to 31 December 2019, to classify negative information on socially significant topics. A BigARTM model was derived (200 topics) and the proposed model was applied, including to fill a table of the analytical hierarchical process (AHP) and all of the necessary high-level labeling procedures. Experiments confirm the general possibility of evaluating the media using the topic model of the text corpora, because an area under receiver operating characteristics curve (ROC AUC) score of 0.81 was achieved in the classification task, which is comparable with results obtained for the same task by applying the BERT (Bidirectional Encoder Representations from Transformers) model.<\/jats:p>","DOI":"10.3390\/sym12121945","type":"journal-article","created":{"date-parts":[[2020,11,25]],"date-time":"2020-11-25T21:55:06Z","timestamp":1606341306000},"page":"1945","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Classification of Negative Information on Socially Significant Topics in Mass Media"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3727-043X","authenticated-orcid":false,"given":"Ravil I.","family":"Mukhamediev","sequence":"first","affiliation":[{"name":"Institute of Cybernetics and Information Technology, Satbayev University (KazNRTU), Satpayev str., 22A, Almaty 050013, Kazakhstan"},{"name":"Department of Natural Science and Computer Technologies, ISMA University, Lomonosov str., 1, LV-1011 Riga, Latvia"},{"name":"Institute of Information and Computational Technologies, Pushkin str., 125, Almaty 050010, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kirill","family":"Yakunin","sequence":"additional","affiliation":[{"name":"Institute of Cybernetics and Information Technology, Satbayev University (KazNRTU), Satpayev str., 22A, Almaty 050013, Kazakhstan"},{"name":"Institute of Information and Computational Technologies, Pushkin str., 125, Almaty 050010, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7378-9212","authenticated-orcid":false,"given":"Rustam","family":"Mussabayev","sequence":"additional","affiliation":[{"name":"Institute of Information and Computational Technologies, Pushkin str., 125, Almaty 050010, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Timur","family":"Buldybayev","sequence":"additional","affiliation":[{"name":"Information-Analytical Center, Dostyk str., 18, Nur-Sultan 010000, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5271-9071","authenticated-orcid":false,"given":"Yan","family":"Kuchin","sequence":"additional","affiliation":[{"name":"Institute of Information and Computational Technologies, Pushkin str., 125, Almaty 050010, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sanzhar","family":"Murzakhmetov","sequence":"additional","affiliation":[{"name":"Institute of Information and Computational Technologies, Pushkin str., 125, Almaty 050010, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4203-800X","authenticated-orcid":false,"given":"Marina","family":"Yelis","sequence":"additional","affiliation":[{"name":"Institute of Cybernetics and Information Technology, Satbayev University (KazNRTU), Satpayev str., 22A, Almaty 050013, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,11,25]]},"reference":[{"key":"ref_1","unstructured":"Edelman, R. (2020, April 25). Edelman Trust Barometer. Available online: https:\/\/www.edelman.com\/research\/2019-edelman-trust-barometer."},{"key":"ref_2","unstructured":"Briggs, A., and Cobley, P. (1998). Promotional strategies and media power. The Media: An Introduction, Longman."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Bushman, B., and Whitaker, J. (2012). Media Influence on Behavior. Encyclopedia of Human Behavior, Elsevier.","DOI":"10.1016\/B978-0-12-375000-6.00386-4"},{"key":"ref_4","first-page":"29","article-title":"Media Effects","volume":"Volume 3","author":"Don","year":"2015","journal-title":"International Encyclopedia of the Social & Behavioral Sciences"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1016\/j.cogsys.2018.12.018","article-title":"Human-machine interaction: A case study on fake news detection using a backtracking based on a cognitive system","volume":"55","author":"Ko","year":"2019","journal-title":"Cogn. Syst. Res."},{"key":"ref_6","unstructured":"Bushman, B., and Whitaker, J. (2020, November 24). Media Influence on Behavior. Reference Module in: Neuroscience and Biobehavioral Psychology. Available online: http:\/\/scitechconnect.elsevier.com\/neurorefmod\/."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Mishra, S., Rizoiu, M.A., and Xie, L. (2016). Feature driven and point process approaches for popularity prediction. Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, Association for Computing Machinery.","DOI":"10.1145\/2983323.2983812"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Tatar, A., Antoniadis, P., Amorim, M.D., and Fdida, S. (2012, January 26\u201329). Ranking News Articles Based on Popularity Prediction. Proceedings of the 2012 IEEE\/ACM International Conference on Advances in Social Networks Analysis and Minng, Istanbul, Turkey.","DOI":"10.1109\/ASONAM.2012.28"},{"key":"ref_9","unstructured":"Bandari, R., Asur, S., and Huberman, B.A. (2020, September 20). The Pulse of News in Social Media: Forecasting Popularity. Available online: https:\/\/arxiv.org\/pdf\/1202.0332.pdf."},{"key":"ref_10","unstructured":"Bauer, M.W., and Suerdem, A. (2016). Developing science culture indicators through text mining and online media monitoring. OECD Blue Sky Forum on Science and Innovation Indicators, LSE Research."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Willaert, T., Van Eecke, P., Beuls, K., and Steels, L. (2020). Building Social Media Observatories for Monitoring Online Opinion Dynamics. Soc. Media Soc., 6.","DOI":"10.1177\/2056305119898778"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1177\/0963662514551506","article-title":"Can media monitoring be a proxy for public opinion about technoscientific controversies? The case of the Italian public debate on nuclear power","volume":"25","author":"Neresini","year":"2016","journal-title":"Public Underst. Sci."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Thanasopon, B., Sumret, N., Buranapanitkij, J., and Netisopakul, P. (2017). Extraction and evaluation of popular online trends: A case of Pantip.com. Proceedings of the 2017 9th International Conference on Information Technology and Electrical Engineering (ICITEE), IEEE.","DOI":"10.1109\/ICITEED.2017.8250454"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Macharia, S. (2020). Global Media Monitoring Project (GMMP). Int. Encycl. Gend. Media Commun., 1\u20136.","DOI":"10.1002\/9781119429128.iegmc074"},{"key":"ref_15","unstructured":"Barysevich, A. (2020, May 19). Top of the Best Social Media Monitoring Tools. Available online: https:\/\/www.socialmediatoday.com\/news\/20-of-the-best-social-media-monitoring-tools-to-consider\/545036\/."},{"key":"ref_16","unstructured":"Agilitypr (2020, May 19). Media Monitoring Ultimate Guide. Available online: https:\/\/www.agilitypr.com\/media-monitoring-ultimate-guide\/."},{"key":"ref_17","unstructured":"Newberry, C. (2020, May 19). Social Media Monitoring Tools. Available online: https:\/\/blog.hootsuite.com\/social-media-monitoring-tools."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Barile, F., Ricci, F., Tkalcic, M., Magnini, B., Zanoli, R., Lavelli, A., and Speranza, M. (2019, January 14\u201317). A News Recommender System for Media Monitoring. Proceedings of the IEEE\/WIC\/ACM International Conference on Web Intelligence, Thessaloniki, Greece.","DOI":"10.1145\/3350546.3352510"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1016\/j.tourman.2016.09.009","article-title":"Mining meaning from online ratings and reviews: Tourist satisfaction analysis using latent dirichlet allocation","volume":"59","author":"Guo","year":"2017","journal-title":"Tour. Manag."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"102034","DOI":"10.1016\/j.ipm.2019.04.002","article-title":"An evaluation of document clustering and topic modelling in two online social networks: Twitter and Reddit","volume":"57","author":"Curiskis","year":"2020","journal-title":"Inf. Process. Manag."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Nikulchev, E., Ilin, D., Silaeva, A., Kolyasnikov, P., Belov, V., Runtov, A., Pushkin, P., Laptev, N., Alexeenko, A., and Magomedov, S. (2020). Digital Psychological Platform for Mass Web-Surveys. Data, 5.","DOI":"10.3390\/data5040095"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Basnyat, B., Anam, A., Singh, N., Gangopadhyay, A., and Roy, N. (2017). Analyzing Social Media Texts and Images to Assess the Impact of Flash Floods in Cities. 2017 IEEE International Conference on Smart Computing (SMARTCOMP), IEEE.","DOI":"10.1109\/SMARTCOMP.2017.7946987"},{"key":"ref_23","first-page":"91","article-title":"Methods for calculating the relevance of text fragments based on thematic models in the problem of automatic annotation","volume":"14","author":"Mashechkin","year":"2013","journal-title":"Comput. Methods Program."},{"key":"ref_24","unstructured":"Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2020, September 15). BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. Available online: https:\/\/arxiv.org\/abs\/1810.04805."},{"key":"ref_25","first-page":"161","article-title":"Review and experimental comparison of text clustering methods","volume":"29","author":"Parkhomenko","year":"2017","journal-title":"Proc Inst. Syst. Program. Russ. Acad. Sci."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"693","DOI":"10.20537\/2076-7633-2012-4-4-693-706","article-title":"Regularization, robustness and sparseness of probabilistic thematic models","volume":"4","author":"Vorontsov","year":"2012","journal-title":"Comput. Res. Modeling"},{"key":"ref_27","first-page":"993","article-title":"Latent dirichlet allocation","volume":"3","author":"Blei","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Vorontsov, K., Frei, O., Apishev, M., Romov, P., and Dudarenko, M. (2015). BigARTM: Open Source Library for Regularized Multimodal Topic Modeling of Large Collections. International Conference on Analysis of Images, Soc. Networks and Texts, Springer.","DOI":"10.1007\/978-3-319-26123-2_36"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"226","DOI":"10.1016\/j.ijpe.2014.11.008","article-title":"A decision support system for supplier selection and order allocation in stochastic, multi-stakeholder and multi-criteria environments","volume":"166","author":"Scott","year":"2015","journal-title":"Int. J. Prod. Econ."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"13947","DOI":"10.3390\/su71013947","article-title":"Sustainable and renewable energy: An overview of the application of multiple criteria decision-making techniques and approaches","volume":"7","author":"Mardani","year":"2015","journal-title":"Sustainability"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1016\/j.eiar.2014.09.002","article-title":"Creating a spatial multi-criteria decision support system for energy related integrated environmental impact assessment","volume":"52","author":"Wanderer","year":"2015","journal-title":"Environ. Impact Assess. Rev."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"19","DOI":"10.2478\/v10108-011-0002-5","article-title":"Towards a new approach for disambiguation in NLP by multiple criterian decision-aid","volume":"95","author":"Hoceini","year":"2011","journal-title":"Prague Bull. Math. Linguistics"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"596","DOI":"10.1016\/j.rser.2016.11.191","article-title":"A review of multi criteria decision making (MCDM) towards sustainable renewable energy development","volume":"69","author":"Kumar","year":"2017","journal-title":"Renew. Sustain. Energy Rev."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1109\/21.87068","article-title":"On ordered weighted averaging aggregation operators in multi criteria decision making","volume":"18","author":"Yager","year":"1988","journal-title":"IEEE Trans. Syst. Man Cybern."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1002\/mcda.428","article-title":"A new method for scoring additive multi-attribute value models using pairwise rankings of alternatives","volume":"15","author":"Hansen","year":"2008","journal-title":"J. Multi-Criteria Decis. Anal."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Figueira, J., Vincent, M., and Bernard, R. (2005). ELECTRE methods. Multiple Criteria Decision Analysis: State of the Art Surveys, Springer.","DOI":"10.1007\/b100605"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"486","DOI":"10.1016\/0377-2217(94)90282-8","article-title":"Topsis for MODM","volume":"76","author":"Lai","year":"1994","journal-title":"Eur. J. Oper. Res."},{"key":"ref_38","unstructured":"Detlof, V.W., and Fischer, G.W. (1975). Multi-attribute utility theory: Models and assessment procedures. Utility, Probability, and Human Decision Making, Springer."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"647","DOI":"10.1287\/mnsc.31.6.647","article-title":"A Preference Ranking Organization Method: (The PROMETHEE Method for Multiple Criteria Decision-Making)","volume":"31","author":"Brans","year":"1985","journal-title":"Manag. Sci."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"514","DOI":"10.1016\/j.ejor.2006.01.020","article-title":"Extended VIKOR method in comparison with outranking methods","volume":"178","author":"Opricovic","year":"2007","journal-title":"Eur. J. Oper. Res."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Saaty, T. (1989). Group decision making and the AHP. The Analytic Hierarchy Process, Springer.","DOI":"10.13033\/isahp.y1988.042"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"2554","DOI":"10.1016\/j.renene.2010.10.037","article-title":"PV site suitability analysis using GIS-based spatial fuzzy multi-criteria evaluation","volume":"36","author":"Charabi","year":"2011","journal-title":"Renew. Energy"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1016\/j.renene.2016.10.054","article-title":"Developing a novel risk-based methodology for multi-criteria decision making in marine renewable energy applications","volume":"102","author":"Abaei","year":"2017","journal-title":"Renew. Energy"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"122275","DOI":"10.1109\/ACCESS.2019.2937627","article-title":"Multi-Criteria Spatial Decision Making Support System for Renewable Energy Development in Kazakhstan","volume":"7","author":"Mukhamediev","year":"2019","journal-title":"IEEE Access."},{"key":"ref_45","unstructured":"Saaty, T.L. (1990). Decision Making for Leaders: The Analytic Hierarchy ProcessfFor Decisions in a Complex World, RWS Publications."},{"key":"ref_46","first-page":"5","article-title":"About measuring the intangible. An approach to relative measurements based on the principal eigenvector of a pairwise comparison matrix","volume":"2","author":"Saati","year":"2015","journal-title":"Cloud Sci."},{"key":"ref_47","first-page":"259","article-title":"Informative features of bias and reliability of electronic Mass Media","volume":"2","author":"Ospanova","year":"2019","journal-title":"Sociologia"},{"key":"ref_48","first-page":"277","article-title":"Methodology for determining informative features of news texts and checking their significance","volume":"3","author":"Atanayeva","year":"2019","journal-title":"Sci. Asp."},{"key":"ref_49","first-page":"87","article-title":"Experiments to evaluate mass media based on the thematic model of the text corpus","volume":"7","author":"Mukhamediev","year":"2020","journal-title":"Cloud Sci."},{"key":"ref_50","unstructured":"Yakunin, K. (2020, September 14). This Repo Presents Data Illustrating Results Obtained by Applying Multi Model Mass Media Assessment (M4a) to a Corpora of News Publication from Kazakhstan Media. Available online: https:\/\/github.com\/KindYAK\/M4A-Data."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Peters, M., Ruder, S., and Smith, N. (2019). To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks. arXiv.","DOI":"10.18653\/v1\/W19-4302"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"60","DOI":"10.17323\/1998-0663.2019.4.60.72","article-title":"The design of the structure of the software system for processing text document corpus","volume":"13","author":"Barakhnin","year":"2019","journal-title":"Bus. Inform."},{"key":"ref_53","unstructured":"Yakunin, K. (2020, September 14). Media Monitoring System. Available online: https:\/\/github.com\/KindYAK\/NLPMonitor."},{"key":"ref_54","unstructured":"Yakunin, K. (2020, September 14). Airflow DAGs for NLPMonitor. Available online: https:\/\/github.com\/KindYAK\/NLPMonitor-DAGs."},{"key":"ref_55","unstructured":"Mimno, D., Wallach, H., Talley, E., Leenders, M., and McCallum, A. (2011, January 27\u201331). Optimizing Semantic Coherence in Topic Models. Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, Edinburgh, UK."}],"container-title":["Symmetry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-8994\/12\/12\/1945\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:37:10Z","timestamp":1760179030000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-8994\/12\/12\/1945"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,25]]},"references-count":55,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2020,12]]}},"alternative-id":["sym12121945"],"URL":"https:\/\/doi.org\/10.3390\/sym12121945","relation":{},"ISSN":["2073-8994"],"issn-type":[{"value":"2073-8994","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,11,25]]}}}