{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,30]],"date-time":"2026-01-30T03:33:53Z","timestamp":1769744033497,"version":"3.49.0"},"reference-count":54,"publisher":"Association for Computing Machinery (ACM)","issue":"1","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2007,2]]},"abstract":"<jats:p>\n            Test designers widely believe that the overall effectiveness and cost of software testing depends largely on the type and number of test cases executed on the software. This article shows that the\n            <jats:italic>test oracle<\/jats:italic>\n            , a mechanism that determines whether a software is executed correctly for a test case, also significantly impacts the fault detection effectiveness and cost of a test case. Graphical user interfaces (GUIs), which have become ubiquitous for interacting with today's software, have created new challenges for test oracle development. Test designers manually \u201cassert\u201d the expected values of specific properties of certain GUI widgets in each test case; during test execution, these assertions are used as test oracles to determine whether the GUI executed correctly. Since a test case for a GUI is a sequence of events, a test designer must decide: (1) what to assert; and (2) how frequently to check an assertion, for example, after each event in the test case or after the entire test case has completed execution. Variations of these two factors significantly impact the fault-detection ability and cost of a GUI test case. A technique to declaratively specify different types of automated GUI test oracles is described. Six instances of test oracles are developed and compared in an experiment on four software systems. The results show that test oracles do affect the fault detection ability of test cases in different and interesting ways: (1) Test cases significantly lose their fault detection ability when using \u201cweak\u201d test oracles; (2) in many cases, invoking a \u201cthorough\u201d oracle at the end of test case execution yields the best cost-benefit ratio; (3) certain test cases detect faults only if the oracle is invoked during a small \u201cwindow of opportunity\u201d during test execution; and (4) using thorough and frequently-executing test oracles can compensate for not having long test cases.\n          <\/jats:p>","DOI":"10.1145\/1189748.1189752","type":"journal-article","created":{"date-parts":[[2007,4,5]],"date-time":"2007-04-05T19:20:08Z","timestamp":1175800808000},"page":"4","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":120,"title":["Designing and comparing automated test oracles for GUI-based software applications"],"prefix":"10.1145","volume":"16","author":[{"given":"Qing","family":"Xie","sequence":"first","affiliation":[{"name":"University of Maryland, College Park, MD"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Atif M.","family":"Memon","sequence":"additional","affiliation":[{"name":"University of Maryland, College Park, MD"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2007,2]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.825766"},{"key":"e_1_2_1_2_1","volume-title":"Tech. Rep. CIS-TR-01-02","author":"Baresi L.","year":"2001","unstructured":"Baresi , L. and Young , M . 2001 . Test oracles. Tech. Rep. CIS-TR-01-02 , University of Oregon, Department of Computer and Information Science , Eugene, Oregon . August. http:\/\/www.cs.uoregon.edu\/~michal\/pubs\/oracles.html. Baresi, L. and Young, M. 2001. Test oracles. Tech. Rep. CIS-TR-01-02, University of Oregon, Department of Computer and Information Science, Eugene, Oregon. August. http:\/\/www.cs.uoregon.edu\/~michal\/pubs\/oracles.html."},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the 5th International Conference on Verification, Model Checking and Abstract Interpretation (VMCAI) (Venice). Springer Verlag. 44--57","author":"Barringer H.","unstructured":"Barringer , H. , Goldberg , A. , Havelund , K. , and Sen , K . 2004. Rule-Based runtime verification . In Proceedings of the 5th International Conference on Verification, Model Checking and Abstract Interpretation (VMCAI) (Venice). Springer Verlag. 44--57 . Barringer, H., Goldberg, A., Havelund, K., and Sen, K. 2004. Rule-Based runtime verification. In Proceedings of the 5th International Conference on Verification, Model Checking and Abstract Interpretation (VMCAI) (Venice). Springer Verlag. 44--57."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1049\/sej.1991.0040"},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the 15th International Software\/Internet Quality Week Conference. Software Research","author":"Blackburn M.","unstructured":"Blackburn , M. , Busser , R. , and Nauman , A . 2002. Interface-Driven model-based test generation of Java test drivers . In Proceedings of the 15th International Software\/Internet Quality Week Conference. Software Research , San Francisco, CA. Blackburn, M., Busser, R., and Nauman, A. 2002. Interface-Driven model-based test generation of Java test drivers. In Proceedings of the 15th International Software\/Internet Quality Week Conference. Software Research, San Francisco, CA."},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the 8th International Workshop on Formal Methods for Industrial Critical Systems (FMICS). Elsevier Science. 73--89","author":"Burdy L.","unstructured":"Burdy , L. , Cheon , Y. , Cok , D. , Ernst , M. , Kiniry , J. , Leavens , G. T. , Leino , K. R. M. , and Poll , E . 2003. An overview of JML tools and applications . In Proceedings of the 8th International Workshop on Formal Methods for Industrial Critical Systems (FMICS). Elsevier Science. 73--89 . Burdy, L., Cheon, Y., Cok, D., Ernst, M., Kiniry, J., Leavens, G. T., Leino, K. R. M., and Poll, E. 2003. An overview of JML tools and applications. In Proceedings of the 8th International Workshop on Formal Methods for Industrial Critical Systems (FMICS). Elsevier Science. 73--89."},{"key":"e_1_2_1_7_1","volume-title":"ECOOP: Proceedings of the 16th European Conference on Object-Oriented Programming","author":"Cheon Y.","unstructured":"Cheon , Y. and Leavens , G. T . 2002. A simple and practical approach to unit testing: The JML and JUnit way . In ECOOP: Proceedings of the 16th European Conference on Object-Oriented Programming ( London). Springer Verlag. 231--255. Cheon, Y. and Leavens, G. T. 2002. A simple and practical approach to unit testing: The JML and JUnit way. In ECOOP: Proceedings of the 16th European Conference on Object-Oriented Programming (London). Springer Verlag. 231--255."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/239098.239116"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/193173.195401"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/192218.192221"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/302405.302634"},{"key":"e_1_2_1_12_1","volume-title":"ICSE: Proceedings of the 25th International Conference on Software Engineering. IEEE Computer Society","author":"Elbaum S.","unstructured":"Elbaum , S. , Karre , S. , and Rothermel , G . 2003. Improving web application testing with user session data . In ICSE: Proceedings of the 25th International Conference on Software Engineering. IEEE Computer Society , Washington, DC. 49--59. Elbaum, S., Karre, S., and Rothermel, G. 2003. Improving web application testing with user session data. In ICSE: Proceedings of the 25th International Conference on Software Engineering. IEEE Computer Society, Washington, DC. 49--59."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 2nd International Conference on eXtreme Programming and Flexible Processes in Software Engineering (Villasimius","author":"Finsterwalder M.","year":"2001","unstructured":"Finsterwalder , M. 2001 . Automating acceptance tests for GUI applications in an extreme programming environment . In Proceedings of the 2nd International Conference on eXtreme Programming and Flexible Processes in Software Engineering (Villasimius , Sardinia, Italy). Addison-Wesley. 114--117. Finsterwalder, M. 2001. Automating acceptance tests for GUI applications in an extreme programming environment. In Proceedings of the 2nd International Conference on eXtreme Programming and Flexible Processes in Software Engineering (Villasimius, Sardinia, Italy). Addison-Wesley. 114--117."},{"key":"e_1_2_1_14_1","doi-asserted-by":"crossref","unstructured":"Gall P. L. and Arnould A. 1995. Formal specifications and test: Correctness and oracle. In COMPASS\/ADT. Springer Verlag Oslo Norway. 342--358.   Gall P. L. and Arnould A. 1995. Formal specifications and test: Correctness and oracle. In COMPASS\/ADT. Springer Verlag Oslo Norway. 342--358.","DOI":"10.1007\/3-540-61629-2_52"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/357139.357140"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/367008.367020"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/505863.505892"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1023\/B:FORM.0000017721.39909.4b"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting. Special Sessions: Demonstrations","volume":"2","author":"Hicinbothom J. H.","unstructured":"Hicinbothom , J. H. and Zachary , W. W . 1993. A tool for automatically generating transcripts of human-computer interaction . In Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting. Special Sessions: Demonstrations , vol. 2 . Human Factors and Ergonomics Society, Seattle. 1042. Hicinbothom, J. H. and Zachary, W. W. 1993. A tool for automatically generating transcripts of human-computer interaction. In Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting. Special Sessions: Demonstrations, vol. 2. Human Factors and Ergonomics Society, Seattle. 1042."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/253228.253435"},{"key":"e_1_2_1_21_1","volume-title":"Java Unit Testing Framework","author":"Unit","year":"2003","unstructured":"J Unit , Java Unit Testing Framework . 2003 . JUnit , Java unit testing framework. http:\/\/junit.sourceforge.net. JUnit, Java Unit Testing Framework. 2003. JUnit, Java unit testing framework. http:\/\/junit.sourceforge.net."},{"key":"e_1_2_1_22_1","first-page":"2","article-title":"The black art of GUI testing","volume":"19","author":"Kepple L. R.","year":"1994","unstructured":"Kepple , L. R. 1994 . The black art of GUI testing . Dr. Dobb's J. Softw. Tools 19 , 2 (Feb.), 40. Kepple, L. R. 1994. The black art of GUI testing. Dr. Dobb's J. Softw. Tools 19, 2 (Feb.), 40.","journal-title":"Dr. Dobb's J. Softw. Tools"},{"key":"e_1_2_1_23_1","volume-title":"Mac: A framework for run-time correctness assurance of real-time systems. Tech. Rep. MS-CIS-98-37","author":"Kim M.","year":"1998","unstructured":"Kim , M. , Viswanathan , M. , Ben-Abdallah , H. , Kannan , S. , Lee , I. , and Sokolsky , O . 1998 . Mac: A framework for run-time correctness assurance of real-time systems. Tech. Rep. MS-CIS-98-37 , University of Pennsylvania , Department of Computer and Information Science. Kim, M., Viswanathan, M., Ben-Abdallah, H., Kannan, S., Lee, I., and Sokolsky, O. 1998. Mac: A framework for run-time correctness assurance of real-time systems. Tech. Rep. MS-CIS-98-37, University of Pennsylvania, Department of Computer and Information Science."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.1985.230345"},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the 11th International Software\/Internet Quality Week. Software Research","author":"Marick B.","year":"1998","unstructured":"Marick , B. 1998 . When should a test be automated? In Proceedings of the 11th International Software\/Internet Quality Week. Software Research , San Francisco, CA. Marick, B. 1998. When should a test be automated? In Proceedings of the 11th International Software\/Internet Quality Week. Software Research, San Francisco, CA."},{"key":"e_1_2_1_26_1","first-page":"41","article-title":"Bypassing the GUI","volume":"4","author":"Marick B.","year":"2002","unstructured":"Marick , B. 2002 . Bypassing the GUI . Softw. Test. Quality Eng. Mag. 4 , 5, 41 -- 47 . Marick, B. 2002. Bypassing the GUI. Softw. Test. Quality Eng. Mag. 4, 5, 41--47.","journal-title":"Softw. Test. Quality Eng. Mag."},{"key":"e_1_2_1_27_1","volume-title":"WCRE: Proceedings of the 10th Working Conference on Reverse Engineering (Victoria, BC, Canada). IEEE Computer Society. 260--269","author":"Memon A.","unstructured":"Memon , A. , Banerjee , I. , and Nagarajan , A . 2003a. GUI ripping: Reverse engineering of graphical user interfaces for testing . In WCRE: Proceedings of the 10th Working Conference on Reverse Engineering (Victoria, BC, Canada). IEEE Computer Society. 260--269 . Memon, A., Banerjee, I., and Nagarajan, A. 2003a. GUI ripping: Reverse engineering of graphical user interfaces for testing. In WCRE: Proceedings of the 10th Working Conference on Reverse Engineering (Victoria, BC, Canada). IEEE Computer Society. 260--269."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1002\/smr.v17:1"},{"key":"e_1_2_1_29_1","volume-title":"Advances in Computers","author":"Memon A. M.","unstructured":"Memon , A. M. 2003. Advances in GUI testing . In Advances in Computers , Vol. 57 , Marvin V. Zelkowitz, ed. Academic Press , New York. Memon, A. M. 2003. Advances in GUI testing. In Advances in Computers, Vol. 57, Marvin V. Zelkowitz, ed. Academic Press, New York."},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the IEEE International Conference on Automated Software Engineering","author":"Memon A. M.","unstructured":"Memon , A. M. , Banerjee , I. , and Nagarajan , A . 2003b. What test oracle should I use for effective GUI testing? In Proceedings of the IEEE International Conference on Automated Software Engineering ( Montreal, Canada). IEEE Computer Society. 164--173. Memon, A. M., Banerjee, I., and Nagarajan, A. 2003b. What test oracle should I use for effective GUI testing? In Proceedings of the IEEE International Conference on Automated Software Engineering (Montreal, Canada). IEEE Computer Society. 164--173."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/355045.355050"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.908959"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/940071.940088"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/503209.503244"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2005.117"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/200968.200971"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/271771.271793"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/186258.186508"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.667877"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/186258.187158"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/143062.143100"},{"key":"e_1_2_1_42_1","unstructured":"Robinson H. 1999. Using pre-oracled data in model-based testing. http:\/\/www.geocities.com\/harry_robinson_testing\/pre-oracled.htm.  Robinson H. 1999. Using pre-oracled data in model-based testing. http:\/\/www.geocities.com\/harry_robinson_testing\/pre-oracled.htm."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/52.88944"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/143062.143098"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/1027092.1027093"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/186258.187168"},{"key":"e_1_2_1_47_1","unstructured":"Software Research Inc. 2003. Capture-Replay tool 2003. http:\/\/soft.com.  Software Research Inc. 2003. Capture-Replay tool 2003. http:\/\/soft.com."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/52.73746"},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the Workshop on Automated Program Analysis, Testing and Verification","author":"Vieira M. E.","unstructured":"Vieira , M. E. , Dias , M. S. , and Richardson , D. J . 2000. Object-Oriented specification-based testing using UML statechart diagrams . In Proceedings of the Workshop on Automated Program Analysis, Testing and Verification ( Limerick, Ireland). ACM Press, New York. Vieira, M. E., Dias, M. S., and Richardson, D. J. 2000. Object-Oriented specification-based testing using UML statechart diagrams. In Proceedings of the Workshop on Automated Program Analysis, Testing and Verification (Limerick, Ireland). ACM Press, New York."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/154183.154200"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/25.4.465"},{"key":"e_1_2_1_52_1","volume-title":"Proceedings of the 12th International Symposium Software Reliability Engineering","author":"White L.","unstructured":"White , L. , AlMezen , H. , and Alzeidi , N . 2001. User-Based testing of GUI sequences and their interactions . In Proceedings of the 12th International Symposium Software Reliability Engineering ( Hong Kong). IEEE Press. 54--63. White, L., AlMezen, H., and Alzeidi, N. 2001. User-Based testing of GUI sequences and their interactions. In Proceedings of the 12th International Symposium Software Reliability Engineering (Hong Kong). IEEE Press. 54--63."},{"key":"e_1_2_1_53_1","volume-title":"Software Engineering: An Introduction","author":"Wohlin C.","year":"2000","unstructured":"Wohlin , C. , Runeson , P. , Host , M. , Ohlsson , M. C. , Regnell , B. , and Wesslen , A . 2000 . Experimentation in Software Engineering: An Introduction . Kluwer Academic , Norwell, MA . Wohlin, C., Runeson, P., Host, M., Ohlsson, M. C., Regnell, B., and Wesslen, A. 2000. Experimentation in Software Engineering: An Introduction. Kluwer Academic, Norwell, MA."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/267580.267590"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1189748.1189752","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T18:49:10Z","timestamp":1672253350000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1189748.1189752"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2007,2]]},"references-count":54,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2007,2]]}},"alternative-id":["10.1145\/1189748.1189752"],"URL":"https:\/\/doi.org\/10.1145\/1189748.1189752","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2007,2]]},"assertion":[{"value":"2007-02-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}