{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,8]],"date-time":"2025-11-08T18:00:46Z","timestamp":1762624846693,"version":"3.41.0"},"reference-count":70,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2023,3,31]],"date-time":"2023-03-31T00:00:00Z","timestamp":1680220800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"LOCOSS","award":["PID2020-114615RB-I00"],"award-info":[{"award-number":["PID2020-114615RB-I00"]}]},{"name":"CoSCA","award":["PGC2018-094905-B-I00"],"award-info":[{"award-number":["PGC2018-094905-B-I00"]}]},{"name":"MBTI4A","award":["P20-00067-FR"],"award-info":[{"award-number":["P20-00067-FR"]}]},{"DOI":"10.13039\/501100011688","name":"ECSEL Joint Undertaking","doi-asserted-by":"crossref","award":["101007260"],"award-info":[{"award-number":["101007260"]}],"id":[{"id":"10.13039\/501100011688","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2023,4,30]]},"abstract":"<jats:p>There are numerous domains in which information systems need to deal with uncertain information. These uncertainties may originate from different reasons such as vagueness, imprecision, incompleteness, or inconsistencies, and in many cases, they cannot be neglected. In this article, we are interested in representing and processing uncertain information in domain models, considering the stakeholders\u2019 beliefs (opinions). We show how to associate beliefs to model elements and how to propagate and operate with their associated uncertainty so that domain experts can individually reason about their models enriched with their personal opinions. In addition, we address the challenge of combining the opinions of different domain experts on the same model elements, with the goal to come up with informed collective decisions. We provide different strategies and a methodology to optimally merge individual opinions.<\/jats:p>","DOI":"10.1145\/3542947","type":"journal-article","created":{"date-parts":[[2022,6,8]],"date-time":"2022-06-08T10:00:24Z","timestamp":1654682424000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["Dealing with Belief Uncertainty in Domain Models"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7779-8810","authenticated-orcid":false,"given":"Lola","family":"Burgue\u00f1o","sequence":"first","affiliation":[{"name":"Open University of Catalonia, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2939-5803","authenticated-orcid":false,"given":"Paula","family":"Mu\u00f1oz","sequence":"additional","affiliation":[{"name":"ITIS Software, Universidad de M\u00e1laga, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9639-0186","authenticated-orcid":false,"given":"Robert","family":"Claris\u00f3","sequence":"additional","affiliation":[{"name":"Open University of Catalonia, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2418-2489","authenticated-orcid":false,"given":"Jordi","family":"Cabot","sequence":"additional","affiliation":[{"name":"ICREA, Open University of Catalonia, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0295-0520","authenticated-orcid":false,"given":"S\u00e9bastien","family":"G\u00e9rard","sequence":"additional","affiliation":[{"name":"CEA List, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8139-9986","authenticated-orcid":false,"given":"Antonio","family":"Vallecillo","sequence":"additional","affiliation":[{"name":"ITIS Software, Universidad de M\u00e1laga, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,3,31]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-62522-1_12"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/SAI.2014.6918171"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10270-019-00741-0"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3239372"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/MODELS.2019.00013"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/MiSE.2019.00011"},{"key":"e_1_3_3_8_2","first-page":"91","volume-title":"Proc. of CAiSE","author":"Burgue\u00f1o Loli","year":"2021","unstructured":"Loli Burgue\u00f1o, Robert Claris\u00f3, S\u00e9bastien G\u00e9rard, Shuai Li, and Jordi Cabot. 2021. An NLP-based architecture for the autocompletion of partial domain models. In Proc. of CAiSE, Vol. 12751. Springer, 91\u2013106."},{"key":"e_1_3_3_9_2","article-title":"Belief Fusion Plugin - Git Repository","author":"Burgue\u00f1o Loli","year":"2021","unstructured":"Loli Burgue\u00f1o, Paula Mu\u00f1oz, Robert Clariso, Jordi Cabot, Sebastien Gerard, and Antonio Vallecillo. 2021. Belief Fusion Plugin - Git Repository. https:\/\/github.com\/atenearesearchgroup\/belief-fusion-plugin.","journal-title":"https:\/\/github.com\/atenearesearchgroup\/belief-fusion-plugin"},{"key":"e_1_3_3_10_2","doi-asserted-by":"crossref","unstructured":"Loli Burgue\u00f1o Paula Mu\u00f1oz Robert Claris\u00f3 Jordi Cabot and Antonio Vallecillo. 2021. Dealing with beliefs in domain models \u2014 Companion website. http:\/\/atenea.lcc.uma.es\/projects\/BeliefsInModels.","DOI":"10.1145\/3542947"},{"key":"e_1_3_3_11_2","first-page":"325","volume-title":"11th Annual Conference on Privacy, Security and Trust","author":"Ceolin Davide","year":"2013","unstructured":"Davide Ceolin, Archana Nottamkandath, and Wan Fokkink. 2013. Semi-automated assessment of annotation trustworthiness. In 11th Annual Conference on Privacy, Security and Trust. IEEE, 325\u2013332."},{"key":"e_1_3_3_12_2","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1007\/978-1-4614-6230-9_13","volume-title":"Situation Awareness with Systems of Systems","author":"Ceolin Davide","year":"2013","unstructured":"Davide Ceolin, Willem Robert Van Hage, Guus Schreiber, and Wan Fokkink. 2013. Assessing trust for determining the reliability of information. In Situation Awareness with Systems of Systems. Springer, 209\u2013228."},{"key":"e_1_3_3_13_2","first-page":"592","volume-title":"Proc. of KR\u201918","author":"Console Marco","year":"2018","unstructured":"Marco Console, Paolo Guagliardo, and Leonid Libkin. 2018. Propositional and predicate logics of incomplete information. In Proc. of KR\u201918. AAAI Press, 592\u2013601."},{"key":"e_1_3_3_14_2","unstructured":"Andrew Critch. 2016. Credence\u2014Using subjective probabilities to express belief strengths. (Feb.2016). http:\/\/acritch.com\/credence\/."},{"key":"e_1_3_3_15_2","doi-asserted-by":"crossref","DOI":"10.1002\/9781119286387","volume-title":"Theory of Probability: A Critical Introductory Treatment","author":"Finetti Bruno de","year":"2017","unstructured":"Bruno de Finetti. 2017. Theory of Probability: A Critical Introductory Treatment. John Wiley & Sons."},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35813-5_9"},{"key":"e_1_3_3_17_2","first-page":"43","volume-title":"35th International Conference on Software Engineering (ICSE\u201913)","author":"Esfahani Naeem","year":"2013","unstructured":"Naeem Esfahani, Sam Malek, and Kaveh Razavi. 2013. GuideArch: Guiding the exploration of architectural solution space under uncertainty. In 35th International Conference on Software Engineering (ICSE\u201913). IEEE, 43\u201352."},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10270-017-0594-9"},{"key":"e_1_3_3_19_2","first-page":"573","volume-title":"ICSE\u201912","author":"Famelis Michalis","year":"2012","unstructured":"Michalis Famelis, Rick Salay, and Marsha Chechik. 2012. Partial models: Towards modeling and reasoning with uncertainty. In ICSE\u201912. IEEE Press, 573\u2013583."},{"key":"e_1_3_3_20_2","volume-title":"An Introduction to Probability Theory and Its Applications","author":"Feller W.","year":"2008","unstructured":"W. Feller. 2008. An Introduction to Probability Theory and Its Applications. Wiley."},{"key":"e_1_3_3_21_2","volume-title":"Domain-Specific Languages","author":"Fowler Martin","year":"2011","unstructured":"Martin Fowler. 2011. Domain-Specific Languages. Addison-Wesley."},{"key":"e_1_3_3_22_2","first-page":"125","volume-title":"Proc. of the FoSER Workshop at FSE\/SDP 2010","author":"Garlan David","year":"2010","unstructured":"David Garlan. 2010. Software engineering in an uncertain world. In Proc. of the FoSER Workshop at FSE\/SDP 2010. ACM, 125\u2013128."},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.scico.2007.01.013"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/MODELS.2015.7338262"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/CCDC.2014.6852575"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10270-016-0568-3"},{"key":"e_1_3_3_27_2","unstructured":"Eyke H\u00fcllermeier and Willem Waegeman. 2019. Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods. (2019). arxiv:cs.LG\/1910.09457."},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1142\/S0218488501000831"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-42337-1"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.23919\/ICIF.2017.8009820"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568239"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-018-9627-4"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1016\/0951-8320(95)00073-B"},{"key":"e_1_3_3_34_2","volume-title":"Uncertainty Theory (5th ed.)","author":"Liu Baoding","year":"2018","unstructured":"Baoding Liu. 2018. Uncertainty Theory (5th ed.). Springer."},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/MODELS50736.2021.00015"},{"key":"e_1_3_3_36_2","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1145\/2489850.2489859","volume-title":"International Workshop on Software Engineering for Systems-of-Systems","author":"Lytra Ioanna","year":"2013","unstructured":"Ioanna Lytra and Uwe Zdun. 2013. Supporting architectural decision making for systems-of-systems design under uncertainty. In International Workshop on Software Engineering for Systems-of-Systems. 43\u201346."},{"key":"e_1_3_3_37_2","unstructured":"Miguel Angel Manzano Helena de Felipe-Rodr\u00edguez and Laura Gago-G\u00f3mez. 2018. DICTOMAGRED: Diccionario de Toponimia Magreb\u00ed. https:\/\/dictomagred.usal.es\/."},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3284179.3284318"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.3390\/informatics6020020"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3362789.3362833"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.5381\/jot.2020.19.3.a1"},{"key":"e_1_3_3_42_2","unstructured":"Jakob Nielsen. 2020. How Many Test Users in a Usability Study?https:\/\/www.nngroup.com\/articles\/how-many-test-users\/."},{"key":"e_1_3_3_43_2","first-page":"25","volume-title":"URSW","author":"Nottamkandath Archana","year":"2014","unstructured":"Archana Nottamkandath, Jasper Oosterman, Davide Ceolin, Wan J. Fokkink, et\u00a0al. 2014. Automated evaluation of crowdsourced annotations in the cultural heritage domain. In URSW. 25\u201336."},{"key":"e_1_3_3_44_2","volume-title":"Object Constraint Language (OCL) Specification. Version 2.4","author":"Group Object Management","year":"2014","unstructured":"Object Management Group. 2014. Object Constraint Language (OCL) Specification. Version 2.4. OMG Document formal\/2014-02-03."},{"key":"e_1_3_3_45_2","volume-title":"Unified Modeling Language (UML) Specification. Version 2.5","author":"Group Object Management","year":"2015","unstructured":"Object Management Group. 2015. Unified Modeling Language (UML) Specification. Version 2.5. OMG Document formal\/2015-03-01."},{"key":"e_1_3_3_46_2","volume-title":"Precise Semantics for Uncertainty Modeling (PSUM) RFP","author":"Group Object Management","year":"2017","unstructured":"Object Management Group. 2017. Precise Semantics for Uncertainty Modeling (PSUM) RFP. https:\/\/www.omg.org\/cgi-bin\/doc.cgi?ad\/2017-12-1OMG Document ad\/2017-12-1."},{"key":"e_1_3_3_47_2","first-page":"59","volume-title":"Proc. of OCL@MODELS\u201919 (CEUR Workshop Proceedings)","volume":"2513","author":"Ortiz Victor","year":"2019","unstructured":"Victor Ortiz, Loli Burgue\u00f1o, Antonio Vallecillo, and Martin Gogolla. 2019. Native support for UML and OCL primitive datatypes enriched with uncertainty in USE. In Proc. of OCL@MODELS\u201919 (CEUR Workshop Proceedings), Vol. 2513. CEUR-WS.org, 59\u201366. http:\/\/ceur-ws.org\/Vol-2513\/paper5.pdf."},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/2602576.2602582"},{"key":"e_1_3_3_49_2","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1145\/2568088.2568095","volume-title":"5th ACM\/SPEC International Conference on Performance Engineering","author":"Perez-Palacin Diego","year":"2014","unstructured":"Diego Perez-Palacin and Raffaela Mirandola. 2014. Uncertainties in the modeling of self-adaptive systems: A taxonomy and an example of availability evaluation. In 5th ACM\/SPEC International Conference on Performance Engineering. 3\u201314."},{"key":"e_1_3_3_50_2","article-title":"ACM SIGSOFT empirical standards","volume":"2010","author":"Ralph Paul","year":"2020","unstructured":"Paul Ralph, Sebastian Baltes, Domenico Bianculli, Yvonne Dittrich, Michael Felderer, Robert Feldt, Antonio Filieri, Carlo Alberto Furia, Daniel Graziotin, Pinjia He, Rashina Hoda, Natalia Juristo, Barbara A. Kitchenham, Romain Robbes, Daniel M\u00e9ndez, Jefferson Molleri, Diomidis Spinellis, Miroslaw Staron, Klaas-Jan Stol, Damian A. Tamburri, Marco Torchiano, Christoph Treude, Burak Turhan, and Sira Vegas. 2020. ACM SIGSOFT empirical standards. CoRR abs\/2010.03525 (2020).","journal-title":"CoRR"},{"key":"e_1_3_3_51_2","volume-title":"Handbook of Usability Testing. Second Edition: How to Plan, Design, and Conduct Effective Tests","author":"Rubin Jeff","year":"2008","unstructured":"Jeff Rubin and Dana Chisnell. 2008. Handbook of Usability Testing. Second Edition: How to Plan, Design, and Conduct Effective Tests. Wiley Publishing, Inc."},{"key":"e_1_3_3_52_2","article-title":"Controlling AI","author":"Russell Stuart","year":"2020","unstructured":"Stuart Russell and Frank Chen. 2020. Controlling AI. podcast. https:\/\/a16z.com\/2020\/01\/16\/controlling-ai-human-compatible\/.","journal-title":"podcast"},{"key":"e_1_3_3_53_2","volume-title":"Artificial Intelligence. A Modern Approach (3rd ed.)","author":"Russell Stuart J.","year":"2010","unstructured":"Stuart J. Russell and Peter Norvig. 2010. Artificial Intelligence. A Modern Approach (3rd ed.). Prentice Hall."},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00766-013-0170-y"},{"key":"e_1_3_3_55_2","first-page":"1","volume-title":"Proc. of BigMDE @ STAF\u201915 (CEUR Workshop Proceedings)","volume":"1406","author":"Scheidgen Markus","year":"2015","unstructured":"Markus Scheidgen. 2015. Generation of large random models for benchmarking. In Proc. of BigMDE @ STAF\u201915 (CEUR Workshop Proceedings), Vol. 1406. CEUR-WS.org, 1\u201310."},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2003.1231147"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1515\/9780691214696"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10270-021-00924-8"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-21920-8_52"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3233027.3233029"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442391.3442407"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-25964-2_8"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3239372.3239410"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10270-020-00832-3"},{"key":"e_1_3_3_65_2","first-page":"3084","volume-title":"Determining Usability Test Sample Size (2nd ed.)","author":"Turner Carl W.","year":"2006","unstructured":"Carl W. Turner, James R. Lewis, and Jakob Nielsen. 2006. Determining Usability Test Sample Size (2nd ed.). Vol. 3. CRC Press, 3084\u20133088."},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.456"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.23919\/ICIF.2018.8455615"},{"key":"e_1_3_3_68_2","volume-title":"DSL Engineering - Designing, Implementing and Using Domain-Specific Languages","author":"Voelter Markus","year":"2013","unstructured":"Markus Voelter, Sebastian Benz, Christian Dietrich, Birgit Engelmann, Mats Helander, Lennart C. L. Kats, Eelco Visser, and Guido Wachsmuth. 2013. DSL Engineering - Designing, Implementing and Using Domain-Specific Languages. dslbook.org. http:\/\/www.dslbook.org."},{"key":"e_1_3_3_69_2","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1007\/978-3-540-45143-3_2","volume-title":"Empirical Methods and Studies in Software Engineering, Experiences from ESERNET","author":"Wohlin Claes","year":"2003","unstructured":"Claes Wohlin, Martin H\u00f6st, and Kennet Henningsson. 2003. Empirical research methods in software engineering. In Empirical Methods and Studies in Software Engineering, Experiences from ESERNET, Reidar Conradi and Alf Inge Wang (Eds.). Vol. 2765. 7\u201323."},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-29044-2"},{"key":"e_1_3_3_71_2","doi-asserted-by":"crossref","DOI":"10.1007\/978-94-010-0646-0","volume-title":"Fuzzy Set Theory \u2013 and Its Applications (4th ed.)","author":"Zimmermann Hans-Juergen","year":"2001","unstructured":"Hans-Juergen Zimmermann. 2001. Fuzzy Set Theory \u2013 and Its Applications (4th ed.). Springer Science+Business Media."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3542947","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3542947","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:49:32Z","timestamp":1750182572000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3542947"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,31]]},"references-count":70,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,4,30]]}},"alternative-id":["10.1145\/3542947"],"URL":"https:\/\/doi.org\/10.1145\/3542947","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"type":"print","value":"1049-331X"},{"type":"electronic","value":"1557-7392"}],"subject":[],"published":{"date-parts":[[2023,3,31]]},"assertion":[{"value":"2021-07-02","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-05-08","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-03-31","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}