{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T17:37:58Z","timestamp":1776101878182,"version":"3.50.1"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2017,8,24]],"date-time":"2017-08-24T00:00:00Z","timestamp":1503532800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2017,8,24]],"date-time":"2017-08-24T00:00:00Z","timestamp":1503532800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Hum. Cent. Comput. Inf. Sci."],"published-print":{"date-parts":[[2017,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Self-reporting is used as a subjective measure of usability study of technology solutions. In assistive technology research, more than often the \u2018a coordinator\u2019 directly assist the \u2018subject\u2019 in the scoring process. This makes the rating process slower and also introduces bias, such as, \u2018Forer effect\u2019 and\/or \u2018Hawthorne\u2019 effect. To address these issues we propose to use technology mediated interaction between the \u2018subject\u2019 and \u2018the coordinator\u2019 in evaluating assistive technology solutions. The goal is to combine both the qualitative and quantitative scores to create a relatively unbiased rating system. Empirical studies were performed on two different datasets in order to illustrate the utility of the proposed approach. It was observed that, the proposed hybrid rating is relatively unbiased for usability study.<\/jats:p>","DOI":"10.1186\/s13673-017-0104-7","type":"journal-article","created":{"date-parts":[[2017,6,24]],"date-time":"2017-06-24T11:41:20Z","timestamp":1498304480000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Rethinking self-reported measure in subjective evaluation of assistive technology"],"prefix":"10.1186","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8205-4939","authenticated-orcid":false,"given":"Gahangir","family":"Hossain","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2017,8,24]]},"reference":[{"issue":"2","key":"104_CR1","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1145\/2141943.2141945","volume":"4","author":"A Sears","year":"2012","unstructured":"Sears A, Hanson VL (2012) Representing users in accessibility research. ACM Trans Access. Comput 4(2):7. doi:10.1145\/2141943.2141945","journal-title":"ACM Trans Access. Comput"},{"issue":"1","key":"104_CR2","doi-asserted-by":"publisher","first-page":"367","DOI":"10.2466\/pr0.1985.57.2.367","volume":"57","author":"DH Dickson","year":"1985","unstructured":"Dickson DH, Kelly IW (1985) The \u2018Barnum Effect\u2019 in personality assessment: a review of the literature. Psychol Rep 57(1):367\u2013382","journal-title":"Psychol Rep"},{"issue":"1","key":"104_CR3","first-page":"27","volume":"12","author":"HM Parsons","year":"1991","unstructured":"Parsons HM (1991) Hawthorne: an early OBM experiment. J Org Behav Manag 12(1):27\u201343","journal-title":"J Org Behav Manag"},{"issue":"3","key":"104_CR4","doi-asserted-by":"publisher","first-page":"208","DOI":"10.1037\/h0089007","volume":"14","author":"RT Brown","year":"1999","unstructured":"Brown RT, Reynolds CR, Whitaker JS (1999) Bias in mental testing since bias in mental testing. Sch Psychol Q 14(3):208","journal-title":"Sch Psychol Q"},{"key":"104_CR5","doi-asserted-by":"crossref","unstructured":"Hart SG (2006) NASA-task load index (NASA-TLX); 20\u00a0years later. In Proceedings of the human factors and ergonomics society annual meeting. Sage Publications, Thousand Oaks, pp 904\u2013908","DOI":"10.1177\/154193120605000909"},{"key":"104_CR6","doi-asserted-by":"crossref","unstructured":"Hossain G, Shaik AS, Yeasin M (2011) Cognitive load and usability analysis of R-MAP for the people who are blind or visual impaired. In Proceedings of the 29th ACM international conference on design of communication. ACM, New York, pp 137\u2013144","DOI":"10.1145\/2038476.2038503"},{"key":"104_CR7","unstructured":"Hossain G, Yeasin M (2013) Collaboration gaps in disabilities sensemaking: Deaf and blind communication perspective\u201d, Proceedings of the ACM conference on computer supported cooperative work (CSCW) CIS Workshop 2013, Feb 24, 2013, San Ontario"},{"issue":"3","key":"104_CR8","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1016\/S0166-4115(08)62386-9","volume":"1","author":"SG Hart","year":"1988","unstructured":"Hart SG, Staveland LE (1988) Development of NASA-TLX (task load index): results of empirical and theoretical research. Hum Mental Workload 1(3):139\u2013183","journal-title":"Hum Mental Workload"},{"key":"104_CR9","doi-asserted-by":"publisher","first-page":"297","DOI":"10.1007\/BF02310555","volume":"16","author":"LJ Cronbach","year":"1951","unstructured":"Cronbach LJ (1951) Coefficient alpha and the internal structure of tests. Psychometrika 16:297\u2013333","journal-title":"Psychometrika"},{"issue":"2","key":"104_CR10","first-page":"39","volume":"14","author":"N Kano","year":"1984","unstructured":"Kano N, Seraku N, Takahashi F, Tsuji S (1984) Attractive quality and must-be quality. J Jpn Soc Qual Control 14(2):39\u201348","journal-title":"J Jpn Soc Qual Control"},{"issue":"9580","key":"104_CR11","doi-asserted-by":"publisher","first-page":"2171","DOI":"10.1016\/S0140-6736(07)61013-7","volume":"369","author":"HO Dickinson","year":"2007","unstructured":"Dickinson HO, Parkinson KN, Ravens-Sieberer U, Schirripa G, Thyen U, Arnaud C, Beckung E, et al. (2007) Self-reported quality of life of 8\u201312-year-old children with cerebral palsy: a cross-sectional European study. Lancet 369(9580):2171\u20132178","journal-title":"Lancet"},{"issue":"4","key":"104_CR12","doi-asserted-by":"publisher","first-page":"281","DOI":"10.1017\/S0012162205000526","volume":"47","author":"M White-Koning","year":"2005","unstructured":"White-Koning M, Arnaud C, Bourdet-Loub\u00e8re S, Bazex H, Colver A, Grandjean H (2005) Subjective quality of life in children with intellectual impairment\u2014how can it be assessed? Dev Med Child Neurol 47(4):281\u2013285","journal-title":"Dev Med Child Neurol"},{"issue":"1","key":"104_CR13","doi-asserted-by":"publisher","first-page":"e27","DOI":"10.2196\/mhealth.3422","volume":"3","author":"SR Stoyanov","year":"2015","unstructured":"Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M (2015) Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR mhealth uhealth 3(1):e27","journal-title":"JMIR mhealth uhealth"},{"issue":"1","key":"104_CR14","doi-asserted-by":"publisher","first-page":"016009","DOI":"10.1088\/1741-2552\/14\/1\/016009","volume":"14","author":"DJ McFarland","year":"2016","unstructured":"McFarland DJ, Parvaz MA, Sarnacki WA, Goldstein RZ, Wolpaw JR (2016) Prediction of subjective ratings of emotional pictures by EEG features. J Neural Eng 14(1):016009","journal-title":"J Neural Eng"},{"key":"104_CR15","doi-asserted-by":"publisher","unstructured":"Dai NT, Kuang X, Tang G (2016) Differential weighting of objective versus subjective measures in performance evaluation: experimental evidence. Eur Accounting Rev pp 1\u201320. doi:10.1080\/09638180.2016.1234402","DOI":"10.1080\/09638180.2016.1234402"},{"key":"104_CR16","first-page":"1065","volume":"6","author":"L Kilman","year":"2015","unstructured":"Kilman L, Zekveld AA, H\u00e4llgren M, R\u00f6nnberg J (2015) Subjective ratings of masker disturbance during the perception of native and non-native speech. Front Psychol 6:1065","journal-title":"Front Psychol"},{"issue":"3","key":"104_CR17","first-page":"114","volume":"9","author":"HH Friedman","year":"1999","unstructured":"Friedman HH, Amoo T (1999) Rating the rating scales. J Mark Manag 9(3):114\u2013123","journal-title":"J Mark Manag"},{"issue":"6","key":"104_CR18","doi-asserted-by":"publisher","first-page":"611","DOI":"10.1037\/0021-843X.88.6.611","volume":"88","author":"S Krantz","year":"1979","unstructured":"Krantz S, Hammen CL (1979) Assessment of cognitive bias in depression. J Abnorm Psychol 88(6):611","journal-title":"J Abnorm Psychol"},{"key":"104_CR19","volume-title":"The psychology of the psychic","author":"DF Marks","year":"2000","unstructured":"Marks DF (2000) The psychology of the psychic, 2nd edn. Pro-metheus Books, New York","edition":"2"},{"key":"104_CR20","doi-asserted-by":"publisher","first-page":"357","DOI":"10.1007\/BF02293814","volume":"43","author":"D Andrich","year":"1978","unstructured":"Andrich D (1978) A rating formulation for ordered response categories. Psychometrika 43:357\u2013374","journal-title":"Psychometrika"},{"issue":"14","key":"104_CR21","doi-asserted-by":"publisher","first-page":"966","DOI":"10.1080\/00140130210166951","volume":"45","author":"J Annett","year":"2002","unstructured":"Annett J (2002) Subjective rating scales: science or art? Ergonomics 45(14):966\u2013987","journal-title":"Ergonomics"},{"issue":"2","key":"104_CR22","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1080\/17470216508416423","volume":"17","author":"I Pollack","year":"1965","unstructured":"Pollack  I(1965) Iterative techniques for unbiased rating scales. Q J Exp Psychol 17(2):139\u2013148","journal-title":"Q J Exp Psychol"},{"key":"104_CR23","unstructured":"Krauss-Whitbourne, S (2012) When it comes to personality tests, skepticism is a good thing. Psychol Today. https:\/\/www.psychologytoday.com\/blog\/fulfillment-any-age\/201008\/when-itcomes-personality-tests-dose-skepticism-is-good-thing. Accessed 25 Nov 2012"},{"key":"104_CR24","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1109\/2.121503","volume":"25","author":"J Nielsen","year":"1992","unstructured":"Nielsen J (1992) The usability engineering life cycle. Computer 25:12\u201322","journal-title":"Computer"},{"key":"104_CR25","volume-title":"Experimental psy-chology","author":"BH Kantowitz","year":"2009","unstructured":"Kantowitz BH, Roediger HLIII, Elmes DG (2009) Experimental psy-chology. Cengage Learning, Boston"},{"key":"104_CR26","doi-asserted-by":"publisher","first-page":"350","DOI":"10.1037\/1040-3590.8.4.350","volume":"8","author":"N Schmitt","year":"1996","unstructured":"Schmitt N (1996) Uses and abuses of coefficient alpha. Psychol Assess 8:350\u2013353. doi:10.1037\/1040-3590.8.4.350","journal-title":"Psychol Assess"},{"key":"104_CR27","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1177\/0146621605278814","volume":"30","author":"R Zinbarg","year":"2006","unstructured":"Zinbarg R, Yovel I, Revelle W, McDonald R (2006) Estimating generalizability to a universe of indicators that all have an attribute in common: a comparison of estimators for alpha. Appl Psychol Meas 30:121\u2013144. doi:10.1177\/0146621605278814","journal-title":"Appl Psychol Meas"},{"key":"104_CR28","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1007\/s11336-003-0974-7","volume":"70","author":"R Zinbarg","year":"2005","unstructured":"Zinbarg R, Revelle W, Yovel I, Li W (2005) Cronbach\u2019s, Revelle\u2019s, and McDonald\u2019s: their relations with each other and two alternative conceptualiza-tions of reliability. Psychometrika 70:123\u2013133. doi:10.1007\/s11336-003-0974-7","journal-title":"Psychometrika"},{"key":"104_CR29","doi-asserted-by":"publisher","DOI":"10.1111\/bjop.12046","author":"TJ Dunn","year":"2013","unstructured":"Dunn TJ, Baguley T, Brunsden V (2013) From alpha to omega: a prac-tical solution to the pervasive problem of internal consistency estimation. Br J Psychol. doi:10.1111\/bjop.12046","journal-title":"Br J Psychol"},{"issue":"1","key":"104_CR30","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1136\/qshc.6.1.25","volume":"6","author":"L Redpath","year":"1997","unstructured":"Redpath L, Stacey A, Pugh E, Holmes E (1997) Use of the critical incident tech-nique in primary care in the audit of deaths by suicide. Qual Health Care 6(1):25\u201328","journal-title":"Qual Health Care"},{"key":"104_CR31","first-page":"278","volume":"34","author":"LL Thurstone","year":"1927","unstructured":"Thurstone LL (1927) A law of comparative judgement. Psychol Rev 34:278\u2013286","journal-title":"Psychol Rev"},{"key":"104_CR32","unstructured":"Russell I (2012) The delta rule. University of Hartford, West Hartford. Accessed 5 Nov 2012"}],"container-title":["Human-centric Computing and Information Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13673-017-0104-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13673-017-0104-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13673-017-0104-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,7,30]],"date-time":"2021-07-30T08:12:26Z","timestamp":1627632746000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1186\/s13673-017-0104-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,8,24]]},"references-count":32,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2017,12]]}},"alternative-id":["104"],"URL":"https:\/\/doi.org\/10.1186\/s13673-017-0104-7","relation":{},"ISSN":["2192-1962"],"issn-type":[{"value":"2192-1962","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,8,24]]},"assertion":[{"value":"26 November 2016","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 June 2017","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 August 2017","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"23"}}