{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,8]],"date-time":"2026-02-08T06:03:03Z","timestamp":1770530583237,"version":"3.49.0"},"reference-count":38,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2022,10,26]],"date-time":"2022-10-26T00:00:00Z","timestamp":1666742400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001691","name":"JSPS KAKENHI","doi-asserted-by":"publisher","award":["JP20K12511"],"award-info":[{"award-number":["JP20K12511"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Informatics"],"abstract":"<jats:p>In user experience evaluation (UXE), it is generally accepted that the order in which users perform tasks when using a product is often random rather than fixed. UXE based on these so-called randomly ordered tasks is challenging. Although several articles have been published on UXE, none have proposed a technique to evaluate the significance of randomly ordered tasks. In this study, we propose a new approach to predict final user satisfaction based on UX related to randomly ordered tasks. We aimed to study the importance of task order in the UX. In the main experiment, 60 participants completed questionnaires about satisfaction while performing a series of tasks on a travel agency website. Among the machine learning models tested, we found that accounting for the order or sequence of actions actually performed by users in a support vector machine (SVM) algorithm with a polynomial kernel produced the most accurate predictions of final user satisfaction (97%). These findings indicate that some machine learning techniques can comprehend participants\u2019 randomly ordered UX data. Moreover, using random ordering, which accounts for the actual order of actions performed by users, can significantly impact the prediction of final user satisfaction.<\/jats:p>","DOI":"10.3390\/informatics9040085","type":"journal-article","created":{"date-parts":[[2022,10,26]],"date-time":"2022-10-26T05:11:33Z","timestamp":1666761093000},"page":"85","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Using Random Ordering in User Experience Testing to Predict Final User Satisfaction"],"prefix":"10.3390","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9968-2309","authenticated-orcid":false,"given":"Kitti","family":"Koonsanit","sequence":"first","affiliation":[{"name":"Department of Computer Science, Graduate School of Systems Design, Tokyo Metropolitan University, Tokyo 191-0065, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daiki","family":"Hiruma","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Graduate School of Systems Design, Tokyo Metropolitan University, Tokyo 191-0065, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vibol","family":"Yem","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Graduate School of Systems Design, Tokyo Metropolitan University, Tokyo 191-0065, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nobuyuki","family":"Nishiuchi","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Graduate School of Systems Design, Tokyo Metropolitan University, Tokyo 191-0065, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,10,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"3136","DOI":"10.3390\/jtaer16070171","article-title":"Predicting Final User Satisfaction Using Momentary UX Data and Machine Learning Techniques","volume":"16","author":"Koonsanit","year":"2021","journal-title":"J. Theor. Appl. Electron. Commer. Res."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1080\/10447318.2015.1064648","article-title":"Assessing user satisfaction in the era of user experience: Comparison of the SUS, UMUX, and UMUX-LITE as a function of product experience","volume":"31","author":"Borsci","year":"2015","journal-title":"Int. J. Hum.-Comput. Interact."},{"key":"ref_3","first-page":"55","article-title":"Customer Relationship Management (CRM), Customer Satisfaction and Customer Lifetime Value in Retail","volume":"15","author":"Kaul","year":"2017","journal-title":"Rev. Prof. Manag. J. New Delhi Inst. Manag."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1476","DOI":"10.1109\/JPROC.2016.2637878","article-title":"A survey on web tracking: Mechanisms, implications, and defenses","volume":"105","author":"Bujlow","year":"2017","journal-title":"Proc. IEEE"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Hussain, J., Ali Khan, W., Hur, T., Muhammad Bilal, H.S., Bang, J., Ul Hassan, A., Afzal, M., and Lee, S. (2018). A Multimodal Deep Log-Based User Experience (UX) Platform for UX Evaluation. Sensors, 18.","DOI":"10.3390\/s18051622"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Feng, L., and Wei, W. (2019). An empirical study on user experience evaluation and identification of critical UX issues. Sustainability, 11.","DOI":"10.3390\/su11082432"},{"key":"ref_7","first-page":"43","article-title":"Impact of customer relationship management on customer satisfaction and its role towards customer loyalty and retention practices in the hotel sector","volume":"7","author":"Angamuthu","year":"2015","journal-title":"BVIMSR\u2019s J. Manag. Res."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1080\/10548408.2015.1130108","article-title":"Impact of Customer Relationship Management on Customer Satisfaction: The Case of a Budget Hotel Chain","volume":"34","author":"Rahimi","year":"2017","journal-title":"J. Travel Tour. Mark."},{"key":"ref_9","first-page":"1","article-title":"The Impact of Software User Experience on Customer Satisfaction","volume":"21","author":"Badran","year":"2018","journal-title":"J. Manag. Inf. Decis. Sci."},{"key":"ref_10","unstructured":"Roto, V., Law, E., Vermeeren, A., and Hoonhout, J. (2011, January 15\u201317). User experience white paper: Bringing clarity to the concept of user experience. Proceedings of the Dagstuhl Seminar on Demarcating User Experience, Wadern, Germany."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Marti, P., and Iacono, I. (2016, January 11\u201314). Anticipated, momentary, episodic, remembered: The many facets of User eXperience. Proceedings of the 2016 Federated Conference on Computer Science and Information Systems (FedCSIS), Gdansk, Poland.","DOI":"10.15439\/2016F302"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"473","DOI":"10.1016\/j.intcom.2011.06.005","article-title":"UX Curve: A method for evaluating long-term user experience","volume":"23","author":"Kujala","year":"2011","journal-title":"Interact Comput."},{"key":"ref_13","unstructured":"Kujala, S., Roto, V., V\u00e4\u00e4n\u00e4nen, K., Karapanos, E., and Sinnel\u00e4, A. (2022, January 12). Guidelines How to Use the UX Curve Method 2013. Available online: https:\/\/www.researchgate.net\/publication\/242019830_Guidelines_how_to_use_the_UX_Curve_method."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"86","DOI":"10.20943\/01201605.8693","article-title":"UX Graph Tool for Evaluating the User Satisfaction","volume":"13","author":"Hashizume","year":"2016","journal-title":"Int. J. Comput. Sci. Issue"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1007\/978-3-319-39510-4_31","article-title":"UX Graph and ERM as Tools for Measuring Kansei Experience","volume":"Volume 9731","author":"Kurosu","year":"2016","journal-title":"Proceedings of the 18th International Conference on Human-Computer Interaction. Theory, Design, Development and Practice"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Sukamto, R.A., Wibisono, Y., and Agitya, D.G. (2020, January 21\u201322). Enhancing The User Experience of Portal Website using User-Centered Design Method. Proceedings of the 2020 6th International Conference on Science in Information Technology (ICSITech), Palu, Indonesia.","DOI":"10.1109\/ICSITech49800.2020.9392044"},{"key":"ref_17","first-page":"28","article-title":"User Experience Factors that Influence Users\u2019 Satisfaction of Using Digital Library","volume":"9","author":"Pushparaja","year":"2021","journal-title":"Open Int. J. Inform."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"113","DOI":"10.7903\/ijecs.2004","article-title":"Consequences Of User Experience in A Gamified E-Commerce Platform","volume":"13","author":"Mominzada","year":"2021","journal-title":"Int. J. Electron. Commer. Stud."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"752","DOI":"10.18178\/ijmlc.2020.10.6.1001","article-title":"Towards machine learning based analysis of quality of user experience (QoUE)","volume":"10","author":"Nwakanma","year":"2020","journal-title":"Int. J. Mach. Learn. Comput."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"310","DOI":"10.1108\/JOSM-08-2012-0163","article-title":"The cumulative effect of satisfaction with discrete transactions on share of wallet","volume":"3","author":"Keiningham","year":"2014","journal-title":"J. Serv. Manag."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1007\/s11002-020-09522-y","article-title":"Timing of apology after service failure: The moderating role of future interaction expectation on customer satisfaction","volume":"31","author":"Min","year":"2020","journal-title":"Mark. Lett."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"107939","DOI":"10.1016\/j.cie.2022.107939","article-title":"A machine learning-based iterative design approach to automate user satisfaction degree prediction in smart product-service system","volume":"165","author":"Cong","year":"2022","journal-title":"Comput. Ind. Eng."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"256","DOI":"10.1002\/hfm.20951","article-title":"The peak\u2013end rule in evaluating product user experience: The chronological evaluation of past impressive episodes on overall satisfaction","volume":"32","author":"Doi","year":"2022","journal-title":"Hum. Factors Ergon. Manuf. Serv. Ind."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1186\/s40537-019-0224-1","article-title":"A machine learning approach to analyze customer satisfaction from airline tweets","volume":"6","author":"Kumar","year":"2019","journal-title":"J. Big Data"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1007\/s12193-016-0222-y","article-title":"SVM-based feature selection methods for emotion recognition from multimodal data","volume":"11","year":"2017","journal-title":"J. Multimodal User Interfaces"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1207","DOI":"10.1162\/089976600300015565","article-title":"New support vector algorithms","volume":"12","author":"Smola","year":"2000","journal-title":"Neural Comput."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"670","DOI":"10.1080\/0951192X.2019.1571240","article-title":"Extraction of affective responses from customer reviews: An opinion mining and machine learning approach","volume":"33","author":"Li","year":"2020","journal-title":"Int. J. Comput. Integr. Manuf."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1804","DOI":"10.1177\/0962280213502437","article-title":"A comparison of machine learning methods for classification using simulation with multiple real data examples from mental health studies","volume":"25","author":"Khondoker","year":"2016","journal-title":"Stat. Methods Med. Res."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1016\/j.aca.2012.11.007","article-title":"Sample size planning for classification models","volume":"760","author":"Beleites","year":"2013","journal-title":"Anal. Chim. Acta"},{"key":"ref_30","first-page":"202","article-title":"The problem of sampling in qualitative research","volume":"2","author":"Oppong","year":"2013","journal-title":"Asian J. Manag. Sci. Educ."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1613\/jair.953","article-title":"SMOTE: Synthetic minority over-sampling technique","volume":"16","author":"Chawla","year":"2002","journal-title":"J. Artif. Intell. Res."},{"key":"ref_32","first-page":"1","article-title":"Class imbalance problem in data mining review","volume":"2","author":"Longadge","year":"2013","journal-title":"Int. J. Comput. Sci. Netw."},{"key":"ref_33","first-page":"559","article-title":"Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning","volume":"18","author":"Nogueira","year":"2017","journal-title":"J. Mach. Learn. Res."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1504\/IJKESDP.2011.039875","article-title":"Borderline over-sampling for imbalanced data classification","volume":"3","author":"Nguyen","year":"2011","journal-title":"Int. J. Knowl. Eng. Soft Data Paradig."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1145\/1007730.1007735","article-title":"A study of the behavior of several methods for balancing machine learning training data","volume":"6","author":"Batista","year":"2004","journal-title":"ACM SIGKDD Explor. Newsl."},{"key":"ref_36","first-page":"245","article-title":"Preventing \u201coverfitting\u201d of cross-validation data","volume":"Volume 97","author":"Ng","year":"1997","journal-title":"Proceedings of the ICML"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Yadav, S., and Shukla, S. (2016, January 27\u201328). Analysis of k-fold cross-validation over hold-out validation on colossal datasets for quality classification. Proceedings of the 2016 IEEE 6th International Conference on Advanced Computing (IACC), Bhimavaram, India.","DOI":"10.1109\/IACC.2016.25"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Ben-Hur, A., and Weston, J. (2010). A user\u2019s guide to support vector machines. Data Mining Techniques for the Life Sciences, Springer.","DOI":"10.1007\/978-1-60327-241-4_13"}],"container-title":["Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-9709\/9\/4\/85\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:02:50Z","timestamp":1760144570000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-9709\/9\/4\/85"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,26]]},"references-count":38,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["informatics9040085"],"URL":"https:\/\/doi.org\/10.3390\/informatics9040085","relation":{},"ISSN":["2227-9709"],"issn-type":[{"value":"2227-9709","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,26]]}}}