{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,11]],"date-time":"2026-02-11T19:50:52Z","timestamp":1770839452460,"version":"3.50.1"},"reference-count":68,"publisher":"Emerald","issue":"4","license":[{"start":{"date-parts":[[2014,10,28]],"date-time":"2014-10-28T00:00:00Z","timestamp":1414454400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2014,10,28]]},"abstract":"<jats:sec>\n               <jats:title content-type=\"abstract-heading\">Purpose<\/jats:title>\n               <jats:p> \u2013 An alarming number of spreadsheet faults have been reported in the literature, indicating that effective and easy-to-apply spreadsheet testing techniques are not available for \u201cnon-technical,\u201d end-user programmers. The purpose of this paper is to alleviate the problem by introducing a metamorphic testing (MT) technique for spreadsheets. <\/jats:p>\n            <\/jats:sec>\n            <jats:sec>\n               <jats:title content-type=\"abstract-heading\">Design\/methodology\/approach<\/jats:title>\n               <jats:p> \u2013 The paper discussed four common challenges encountered by end-user programmers when testing a spreadsheet. The MT technique was then discussed and how it could be used to solve the common challenges was explained. An experiment involving several \u201creal-world\u201d spreadsheets was performed to determine the viability and effectiveness of MT. <\/jats:p>\n            <\/jats:sec>\n            <jats:sec>\n               <jats:title content-type=\"abstract-heading\">Findings<\/jats:title>\n               <jats:p> \u2013 The experiment confirmed that MT is highly effective in spreadsheet fault detection, and yet MT is a general technique that can be easily used by end-user programmers to test a large variety of spreadsheet applications. <\/jats:p>\n            <\/jats:sec>\n            <jats:sec>\n               <jats:title content-type=\"abstract-heading\">Originality\/value<\/jats:title>\n               <jats:p> \u2013 The paper provides a detailed discussion of some common challenges of spreadsheet testing encountered by end-user programmers. To the best of the authors knowledge, the paper is the first that includes an empirical study of how effective MT is in spreadsheet fault detection from an end-user programmer's perspective.<\/jats:p>\n            <\/jats:sec>","DOI":"10.1108\/itp-01-2013-0004","type":"journal-article","created":{"date-parts":[[2014,10,23]],"date-time":"2014-10-23T12:29:16Z","timestamp":1414067356000},"page":"440-462","source":"Crossref","is-referenced-by-count":8,"title":["How can non-technical end users effectively test their spreadsheets?"],"prefix":"10.1108","volume":"27","author":[{"given":"Pak-Lok","family":"Poon","sequence":"first","affiliation":[]},{"given":"Fei-Ching","family":"Kuo","sequence":"additional","affiliation":[]},{"given":"Huai","family":"Liu","sequence":"additional","affiliation":[]},{"given":"Tsong","family":"Yueh Chen","sequence":"additional","affiliation":[]}],"member":"140","reference":[{"key":"key2020122801550833800_b1","doi-asserted-by":"crossref","unstructured":"Abraham, R.\n                and \n                  Erwig, M.\n                (2006), \u201cAutoTest: a tool for automatic test case generation in spreadsheets\u201d, Proceedings of the IEEE Symposium on Visual Languages & Human-Centric Computing, Brighton, pp. 43-50.","DOI":"10.1109\/VLHCC.2006.11"},{"key":"key2020122801550833800_b2","doi-asserted-by":"crossref","unstructured":"Acu\u00f1a, S.T.\n                and \n                  Juristo, N.\n                (2004), \u201cAssigning people to roles in software projects\u201d, Software \u2013 Practice & Experience, Vol. 34 No. 7, pp. 675-696.","DOI":"10.1002\/spe.586"},{"key":"key2020122801550833800_b3","unstructured":"Alavi, M.\n                and \n                  Weiss, I.R.\n                (1989), \u201cManaging the risks associated with end-user computing\u201d, in \n                  Nelson, R.R.\n                (Ed.), End-User Computing: Concepts, Issues, and Applications, Wiley, New York, NY, pp. 231-248."},{"key":"key2020122801550833800_b4","unstructured":"Ayalew, Y.\n                (2007), \u201cA user-centered approach for testing spreadsheets\u201d, International Journal of Computing & ICT Research, Vol. 1 No. 1, pp. 76-84."},{"key":"key2020122801550833800_b5","unstructured":"Barr, S.\n               , \n                  Foley, R.\n                and \n                  McMullen, M.\n                (1994), \u201cTowards a quality management system for end-user application development\u201d, Proceedings of the Software Quality Management Conference, Edinburgh, pp. 26-28."},{"key":"key2020122801550833800_b6","doi-asserted-by":"crossref","unstructured":"Benson, D.H.\n                (1983), \u201cA field study of end-user computing: findings and issues\u201d, MIS Quarterly, Vol. 7 No. 4, pp. 35-45.","DOI":"10.2307\/248745"},{"key":"key2020122801550833800_b8","doi-asserted-by":"crossref","unstructured":"Boehm, B.W.\n                and \n                  Basili, V.R.\n                (2001), \u201cSoftware defect reduction top 10 list\u201d, IEEE Computer, Vol. 34 No. 1, pp. 135-137.","DOI":"10.1109\/2.962984"},{"key":"key2020122801550833800_b9","unstructured":"Cale, E.G.\n                (1994), \u201cQuality issues for end-user-developed software\u201d, Journal of Systems Management, Vol. 45 No. 1, pp. 36-39."},{"key":"key2020122801550833800_b10","doi-asserted-by":"crossref","unstructured":"Caulkins, J.P.\n               , \n                  Morrison, E.L.\n                and \n                  Weidemann, T.\n                (2007), \u201cSpreadsheet errors & decision making: evidence from field interviews\u201d, Journal of Organizational & End User Computing, Vol. 19 No. 3, pp. 1-23.","DOI":"10.4018\/joeuc.2007070101"},{"key":"key2020122801550833800_b11","doi-asserted-by":"crossref","unstructured":"Chan, W.K.\n               , \n                  Chen, T.Y.\n               , \n                  Lu, H.\n               , \n                  Tse, T.H.\n                and \n                  Yau, S.S.\n                (2005), \u201cA metamorphic approach to integration testing of context-sensitive middleware-based applications\u201d, Proceedings of the 5th International Conference on Quality Software, Melbourne, pp. 241-249.","DOI":"10.1109\/QSIC.2005.3"},{"key":"key2020122801550833800_b12","unstructured":"Chen, T.Y.\n               , \n                  Cheung, S.C.\n                and \n                  Yiu, S.\n                (1998), \u201cMetamorphic testing: a new approach for generating next test cases\u201d, Technical Report No. HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science & Technology, Hong Kong."},{"key":"key2020122801550833800_b13","doi-asserted-by":"crossref","unstructured":"Chen, T.Y.\n               , \n                  Ho, J.W.K.\n               , \n                  Liu, H.\n                and \n                  Xie, X.\n                (2009), \u201cAn innovative approach for testing bioinformatics programs using metamorphic testing\u201d, BMC Bioinformatics, Vol. 10 No. 24, doi:10.1186\/1471-2105-10-24.","DOI":"10.1186\/1471-2105-10-24"},{"key":"key2020122801550833800_b14","doi-asserted-by":"crossref","unstructured":"Chen, T.Y.\n               , \n                  Tse, T.H.\n                and \n                  Zhou, Z.Q.\n                (2003), \u201cFault-based testing without the need of oracles\u201d, Information & Software Technology, Vol. 45 No. 1, pp. 1-9.","DOI":"10.1016\/S0950-5849(02)00129-5"},{"key":"key2020122801550833800_b15","doi-asserted-by":"crossref","unstructured":"Christy, C.T.\n                (2006), Engineering with the Spreadsheet: Structural Engineering Templates Using Excel, American Society of Civil Engineers, Reston, VA.","DOI":"10.1061\/9780784408278"},{"key":"key2020122801550833800_b16","unstructured":"Clermont, M.\n               , \n                  Hanin, C.\n                and \n                  Mittermeir, R.T.\n                (2002), \u201cA spreadsheet auditing tool evaluated in an industrial context\u201d, Proceedings of the Annual Conference of the European Spreadsheet Risks Interest Group, Cardiff, pp. 35-47."},{"key":"key2020122801550833800_b17","doi-asserted-by":"crossref","unstructured":"Cragg, P.\n                and \n                  King, M.\n                (1993), \u201cSpreadsheet modelling abuse: an opportunity for OR?\u201d, Journal of Operational Research Society, Vol. 44 No. 8, pp. 743-752.","DOI":"10.1057\/jors.1993.133"},{"key":"key2020122801550833800_b18","doi-asserted-by":"crossref","unstructured":"Dewey, B.R.\n                (1998), \u201cProblem-solving tools for engineering students\u201d, Proceedings of the 28th Annual Frontiers in Education Conference (FIE \u201998), Tempe, AZ, pp. 1050-1055.","DOI":"10.1109\/FIE.1998.738557"},{"key":"key2020122801550833800_b19","doi-asserted-by":"crossref","unstructured":"Ding, J.\n               , \n                  Wu, T.\n               , \n                  Lu, J.Q.\n                and \n                  Hu, X.-H.\n                (2010), \u201cSelf-checked metamorphic testing of an image processing program\u201d, Proceedings of the 4th IEEE International Conference on Secure Software Integration & Reliability Improvement, Singapore, pp. 190-197.","DOI":"10.1109\/SSIRI.2010.25"},{"key":"key2020122801550833800_b20","unstructured":"Doll, A.\n                (2000), \u201cPerforming metallurgical calculations in a spreadsheet\u201d, available at: www.agd consulting.ca\/CalcPaper (accessed October 11, 2014)."},{"key":"key2020122801550833800_b21","doi-asserted-by":"crossref","unstructured":"Fisher II, M.\n                and \n                  Rothermel, G.\n                (2005), \u201cThe EUSES spreadsheet corpus: a shared resource for supporting experimentation with spreadsheet dependability mechanisms\u201d, Proceedings of the 1st Workshop on End-User Software Engineering, St Louis, MO, pp. 47-51.","DOI":"10.1145\/1083231.1083242"},{"key":"key2020122801550833800_b22","doi-asserted-by":"crossref","unstructured":"Fisher II, M.\n               , \n                  Rothermel, G.\n               , \n                  Brown, D.\n               , \n                  Cao, M.\n               , \n                  Cook, C.\n                and \n                  Burnett, M.\n                (2006), \u201cIntegrating automated test generation into the WYSIWYT spreadsheet testing methodology\u201d, ACM Transactions on Software Engineering & Methodology, Vol. 15 No. 2, pp. 150-194.","DOI":"10.1145\/1131421.1131423"},{"key":"key2020122801550833800_b23","unstructured":"Franz, L.A.\n                and \n                  Shih, J.C.\n                (1994), \u201cEstimating the value of inspections & early testing for software projects\u201d, Hewlett-Packard Journal, Vol. 45 No. 6, pp. 60-67."},{"key":"key2020122801550833800_b24","doi-asserted-by":"crossref","unstructured":"Gripenberg, P.\n                (2011), \u201cComputer self-efficacy in the information society\u201d, Information Technology & People, Vol. 24 No. 3, pp. 303-331.","DOI":"10.1108\/09593841111158392"},{"key":"key2020122801550833800_b25","doi-asserted-by":"crossref","unstructured":"Grochtmann, M.\n                and \n                  Grimm, K.\n                (1993), \u201cClassification-trees for partition testing\u201d, Software Testing, Verification & Reliability, Vol. 3 No. 2, pp. 63-82.","DOI":"10.1002\/stvr.4370030203"},{"key":"key2020122801550833800_b26","unstructured":"Grossman, T.A.\n                (2003), \u201cAccuracy in spreadsheet modelling systems\u201d, Proceedings of the Annual Conference of the European Spreadsheet Risks Interest Group, Dublin, pp. 91-97."},{"key":"key2020122801550833800_b27","unstructured":"Grossman, T.A.\n                and \n                  \u00d6zl\u00fck\n                (2010), \u201cSpreadsheets grow up: three spreadsheet engineering methodologies for large financial planning models\u201d, Proceedings of the Annual Conference of the European Spreadsheet Risks Interest Group, London, available at: http:\/\/arxiv.org\/pdf\/1008.4174.pdf"},{"key":"key2020122801550833800_b28","doi-asserted-by":"crossref","unstructured":"Harrison, W.\n                (2004), \u201cThe dangers of end-user programming\u201d, IEEE Software, Vol. 21 No. 4, pp. 5-7.","DOI":"10.1109\/MS.2004.13"},{"key":"key2020122801550833800_b29","unstructured":"Hill, M.C.\n                and \n                  Barnes, W.A.\n                (2011), \u201cEnd-user computing applications: implications for internal auditors and managers\u201d, The CPA Journal, Vol. 81 No. 7, pp. 67-71."},{"key":"key2020122801550833800_b30","unstructured":"Institute of Electrical & Electronics Engineers (IEEE)\n                (1990), IEEE Standard 610.12-1990: IEEE Standard Glossary of Software Engineering Terminology, IEEE, New York, NY."},{"key":"key2020122801550833800_b31","unstructured":"Institute of Electrical & Electronics Engineers (IEEE)\n                (2008), IEEE Standard 1028\u2122-2008: IEEE Standard for Software Reviews & Audits, IEEE, New York, NY."},{"key":"key2020122801550833800_b32","doi-asserted-by":"crossref","unstructured":"Jee, E.\n               , \n                  Yoo, J.\n               , \n                  Cha, S.\n                and \n                  Bae, D.\n                (2009), \u201cA data flow-based structural testing technique for FBD programs\u201d, Information & Software Technology, Vol. 51 No. 7, pp. 1131-1139.","DOI":"10.1016\/j.infsof.2009.01.003"},{"key":"key2020122801550833800_b33","unstructured":"Jorgenson, P.C.\n                (2008), Software Testing: A Craftsman's Approach, 3rd ed., Auerbach Publications, Boca Raton, FL."},{"key":"key2020122801550833800_b34","doi-asserted-by":"crossref","unstructured":"Kandt, R.K.\n                (2009), \u201cExperiences in improving flight software development processes\u201d, IEEE Software, Vol. 26 No. 3, pp. 58-64.","DOI":"10.1109\/MS.2009.66"},{"key":"key2020122801550833800_b35","unstructured":"Kaner, C.\n               , \n                  Falk, J.\n                and \n                  Nguyen, H.Q.\n                (1999), Testing Computer Software, 2nd ed., Wiley, New York, NY."},{"key":"key2020122801550833800_b36","doi-asserted-by":"crossref","unstructured":"Kantabutra, V.\n                (1996), \u201cOn hardware for computing exponential & trigonometric functions\u201d, IEEE Transactions on Computers, Vol. 45 No. 3, pp. 328-339.","DOI":"10.1109\/12.485571"},{"key":"key2020122801550833800_b37","unstructured":"Laudon, K.C.\n                and \n                  Laudon, J.P.\n                (2010), Management Information Systems: Managing the Digital Firm, 11th ed., Pearson, Upper Saddle River, NJ."},{"key":"key2020122801550833800_b38","doi-asserted-by":"crossref","unstructured":"Lee, S.M.\n               , \n                  Kim, Y.R.\n                and \n                  Lee, J.\n                (1995), \u201cAn empirical study of the relationships among end user information systems acceptance, training and effectiveness\u201d, Journal of Management Information Systems, Vol. 12 No. 2, pp. 189-202.","DOI":"10.1080\/07421222.1995.11518086"},{"key":"key2020122801550833800_b39","doi-asserted-by":"crossref","unstructured":"Lei, Y.\n               , \n                  Carver, R.H.\n               , \n                  Kacker, R.\n                and \n                  Kung, D.\n                (2007), \u201cA combinatorial testing strategy for concurrent programs\u201d, Software Testing, Verification & Reliability, Vol. 17 No. 2, pp. 207-225.","DOI":"10.1002\/stvr.369"},{"key":"key2020122801550833800_b40","doi-asserted-by":"crossref","unstructured":"Liu, H.\n               , \n                  Kuo, F.-C.\n               , \n                  Towey, D.\n                and \n                  Chen, T.Y.\n                (2014), \u201cHow effectively does metamorphic testing alleviate the oracle problem?\u201d, IEEE Transactions on Software Engineering, Vol. 40 No. 1, pp. 4-22.","DOI":"10.1109\/TSE.2013.46"},{"key":"key2020122801550833800_b42","doi-asserted-by":"crossref","unstructured":"McDaid, K.\n                and \n                  Rust, A.\n                (2009), \u201cTest-driven development for spreadsheet risk management\u201d, IEEE Software, Vol. 26 No. 5, pp. 31-36.","DOI":"10.1109\/MS.2009.143"},{"key":"key2020122801550833800_b43","unstructured":"McDonough, D.\n                (2004), \u201cSpreadsheets and nuclear reactors must never mix!\u201d, available at: http:\/\/it.toolbox. com\/blogs\/spreadsheets\/spreadsheets-nuclear-reactors-must-never-mix-771 (accessed October 11, 2014)."},{"key":"key2020122801550833800_b41","doi-asserted-by":"crossref","unstructured":"Mason, D.\n                and \n                  Willcocks, L.\n                (1991), \u201cManagers, spreadsheets & computing growth: contagion or control?\u201d, Information Systems Journal, Vol. 1 No. 2, pp. 115-128.","DOI":"10.1111\/j.1365-2575.1991.tb00031.x"},{"key":"key2020122801550833800_b44","doi-asserted-by":"crossref","unstructured":"Meyer, B.\n                (2008), \u201cSeven principles of software testing\u201d, IEEE Computer, Vol. 41 No. 8, pp. 99-101.","DOI":"10.1109\/MC.2008.306"},{"key":"key2020122801550833800_b45","doi-asserted-by":"crossref","unstructured":"Mittermeir, R.\n                and \n                  Clermont, M.\n                (2002), \u201cFinding high-level structures in spreadsheet programs\u201d, Proceedings of the 9th Working Conference on Reverse Engineering, Richmond, VA, pp. 221-232.","DOI":"10.1109\/WCRE.2002.1173080"},{"key":"key2020122801550833800_b46","unstructured":"Myers, G.J.\n                (2004), The Art of Software Testing, 2nd ed., Wiley, Hoboken, NJ."},{"key":"key2020122801550833800_b47","doi-asserted-by":"crossref","unstructured":"Panko, R.R.\n                (1998), \u201cWhat we know about spreadsheet errors\u201d, Journal of End User Computing, Vol. 10 No. 2, pp. 15-21.","DOI":"10.4018\/joeuc.1998040102"},{"key":"key2020122801550833800_b48","doi-asserted-by":"crossref","unstructured":"Panko, R.R.\n                (1999), \u201cApplying code inspection to spreadsheet testing\u201d, Journal of Management Information Systems, Vol. 16 No. 2, pp. 159-176.","DOI":"10.1080\/07421222.1999.11518250"},{"key":"key2020122801550833800_b49","unstructured":"Panko, R.R.\n                (2006), \u201cRecommended practices for spreadsheet testing\u201d, Proceedings of the Annual Conference of the European Spreadsheet Risks Interest Group, Cambridge, pp. 73-84."},{"key":"key2020122801550833800_b50","doi-asserted-by":"crossref","unstructured":"Panko, R.R.\n                (2009), \u201cTwo experiments in reducing overconfidence in spreadsheet development\u201d, in \n                  Clarke, S.\n                (Ed.), Evolutionary Concepts in End User Productivity & Performance: Applications for Organizational Progress, Information Science Reference, Hershey, PA, pp. 131-149.","DOI":"10.4018\/978-1-60566-136-0.ch009"},{"key":"key2020122801550833800_b51","doi-asserted-by":"crossref","unstructured":"Panko, R.R.\n                and \n                  Aurigemma, S.\n                (2010), \u201cRevising the Panko-Halverson taxonomy of spreadsheet errors\u201d, Decision Support Systems, Vol. 49 No. 2, pp. 235-244.","DOI":"10.1016\/j.dss.2010.02.009"},{"key":"key2020122801550833800_b52","doi-asserted-by":"crossref","unstructured":"Peng, G.\n               , \n                  Wang, Y.\n                and \n                  Kasuganti, R.\n                (2011), \u201cTechnological embeddedness & household computer adoption\u201d, Information Technology & People, Vol. 24 No. 4, pp. 414-436.","DOI":"10.1108\/09593841111182269"},{"key":"key2020122801550833800_b53","doi-asserted-by":"crossref","unstructured":"Pierson, J.K.\n               , \n                  Forcht, K.A.\n                and \n                  Teer, F.P.\n                (1990), \u201cDetermining documentation requirements for user developed applications\u201d, Information & Management, Vol. 19 No. 1, pp. 21-31.","DOI":"10.1016\/0378-7206(90)90011-6"},{"key":"key2020122801550833800_b54","unstructured":"Pryor, L.\n                (2004), \u201cWhen, why and how to test spreadsheets\u201d, Proceedings of the Annual Conference of the European Spreadsheet Risks Interest Group, Klagenfurt, pp. 145-151."},{"key":"key2020122801550833800_b55","unstructured":"Rahuma, K.M.\n               , \n                  Sondi, F.\n               , \n                  Milad, M.\n                and \n                  AbuGheit, M.\n                (2013), \u201cComparison between spreadsheet and specialized programs in calculating the effect of scale deposition on the well flow performance\u201d, Journal of Petroleum & Gas Engineering, Vol. 4 No. 4, pp. 69-80."},{"key":"key2020122801550833800_b56","doi-asserted-by":"crossref","unstructured":"Rajalingham, K.\n               , \n                  Chadwick, D.\n               , \n                  Knight, B.\n                and \n                  Edwards, D.\n                (2000), \u201cQuality control in spreadsheets: a software engineering-based approach to spreadsheet development\u201d, Proceedings of the 33rd Annual Hawaii International Conference on System Sciences, Maui, HI, pp. 1-9.","DOI":"10.1109\/HICSS.2000.926736"},{"key":"key2020122801550833800_b57","doi-asserted-by":"crossref","unstructured":"Ronen, B.\n               , \n                  Palley, M.A.\n                and \n                  Lucas, H.C. Jr\n                (1989), \u201cSpreadsheet analysis & design\u201d, Communications of the ACM, Vol. 32 No. 1, pp. 84-93.","DOI":"10.1145\/63238.63244"},{"key":"key2020122801550833800_b58","doi-asserted-by":"crossref","unstructured":"Sajaniemi, J.\n                (2000), \u201cModeling spreadsheet audit: a rigorous approach to automatic visualization\u201d, Journal of Visual Languages & Computing, Vol. 11 No. 1, pp. 49-82.","DOI":"10.1006\/jvlc.1999.0142"},{"key":"key2020122801550833800_b59","doi-asserted-by":"crossref","unstructured":"Scaffidi, C.\n               , \n                  Shaw, M.\n                and \n                  Myers, B.\n                (2005), \u201cEstimating the numbers of end users & end user programmers\u201d, Proceedings of the IEEE Symposium on Visual Languages & Human-Centric Computing, Dallas, TX, pp. 207-214.","DOI":"10.1109\/VLHCC.2005.34"},{"key":"key2020122801550833800_b61","doi-asserted-by":"crossref","unstructured":"Segura, S.\n               , \n                  Hierons, R.M.\n               , \n                  Benavides, D.\n                and \n                  Ruiz-Cort\u00e9s, A.\n                (2011), \u201cAutomated metamorphic testing on the analyses of feature models\u201d, Information & Software Technology, Vol. 53 No. 3, pp. 245-258.","DOI":"10.1016\/j.infsof.2010.11.002"},{"key":"key2020122801550833800_b62","unstructured":"Senders, J.W.\n                and \n                  Moray, N.P.\n                (1991), Human Error: Cause, Prediction, & Reduction, Lawrence Erlbaum, Hillsdale, NH."},{"key":"key2020122801550833800_b63","unstructured":"Sommerville, I.\n                (2011), Software Engineering, 9th ed., Pearson, Boston, MA."},{"key":"key2020122801550833800_b64","doi-asserted-by":"crossref","unstructured":"Taylor, M.J.\n               , \n                  Moynihan, E.P.\n                and \n                  Wood-Harper, A.T.\n                (1998), \u201cEnd-user computing & information systems methodologies\u201d, Information Systems Journal, Vol. 8 No. 1, pp. 85-96.","DOI":"10.1046\/j.1365-2575.1998.00005.x"},{"key":"key2020122801550833800_b65","unstructured":"Tukiainen, M.\n                (2000), \u201cUncovering effects of programming paradigms: errors in two spreadsheet systems\u201d, Proceedings of the 12th Workshop of the Psychology of Programming Interest Group, Cosenza, pp. 247-266."},{"key":"key2020122801550833800_b66","doi-asserted-by":"crossref","unstructured":"Vilkomir, S.A.\n               , \n                  Kapoor, K.\n                and \n                  Bowen, J.P.\n                (2003), \u201cTolerance of control-flow testing criteria\u201d, Proceedings of the 27th Annual International Computer Software & Applications Conference, Dallas, TX, pp. 182-187.","DOI":"10.1109\/CMPSAC.2003.1245339"},{"key":"key2020122801550833800_b67","doi-asserted-by":"crossref","unstructured":"Xie, X.\n               , \n                  Ho, J.W.K.\n               , \n                  Murphy, C.\n               , \n                  Kaiser, G.\n               , \n                  Xu, B.\n                and \n                  Chen, T.Y.\n                (2011), \u201cTesting & validating machine learning classifiers by metamorphic testing\u201d, Journal of Systems & Software, Vol. 84 No. 4, pp. 544-558.","DOI":"10.1016\/j.jss.2010.11.920"},{"key":"key2020122801550833800_b68","doi-asserted-by":"crossref","unstructured":"Zhou, Z.Q.\n               , \n                  Zhang, S.J.\n               , \n                  Hagenbuchner, M.\n               , \n                  Tse, T.H.\n               , \n                  Kuo, F.-C.\n                and \n                  Chen, T.Y.\n                (2012), \u201cAutomated functional testing of online search services\u201d, Software Testing, Verification & Reliability, Vol. 22 No. 4, pp. 221-243.","DOI":"10.1002\/stvr.437"},{"key":"key2020122801550833800_frd1","unstructured":"Boehm, B.W.\n                (1981), Software Engineering Economics, Prentice Hall, Upper Saddle River, NJ."},{"key":"key2020122801550833800_frd2","doi-asserted-by":"crossref","unstructured":"Schultheis, R.\n                and \n                  Sumner, M.\n                (1994), \u201cThe relationship of application risks to application controls: a study of microcomputer based spreadsheet application\u201d, Journal of End User Computing, Vol. 6 No. 2, pp. 11-18.","DOI":"10.4018\/joeuc.1994040102"}],"container-title":["Information Technology &amp; People"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/ITP-01-2013-0004","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/ITP-01-2013-0004\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/ITP-01-2013-0004\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T21:54:00Z","timestamp":1753394040000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/itp\/article\/27\/4\/440-462\/431019"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,10,28]]},"references-count":68,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2014,10,28]]}},"alternative-id":["10.1108\/ITP-01-2013-0004"],"URL":"https:\/\/doi.org\/10.1108\/itp-01-2013-0004","relation":{},"ISSN":["0959-3845"],"issn-type":[{"value":"0959-3845","type":"print"}],"subject":[],"published":{"date-parts":[[2014,10,28]]}}}