{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,2]],"date-time":"2025-08-02T18:07:39Z","timestamp":1754158059522,"version":"3.41.2"},"reference-count":36,"publisher":"Emerald","issue":"5","license":[{"start":{"date-parts":[[2011,9,13]],"date-time":"2011-09-13T00:00:00Z","timestamp":1315872000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2011,9,13]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-heading\">Purpose<\/jats:title><jats:p>In large system development efforts organizational and managerial issues are often more challenging than technical ones. One of the key concerns of managers is the control and evaluation of the overall development effort. The purpose of this paper was to analyze the evaluation of system development process in a setting where there are multiple stakeholders in multiple organizations that develop a common information system (IS). The case is a consortium of universities that has developed a common student registrar system over a period of more than ten years.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Design\/methodology\/approach<\/jats:title><jats:p>The authors used a social process model to analyze the episodes and encounters in the system development and evaluation process. The research approach can be characterized as action case.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Findings<\/jats:title><jats:p>The authors found that in the early stages of system development the evaluation was emergent or even non\u2010existent. As the system was developed further and issues, such as delays and missing features, arose, there was a need for better control of the process. Thus, the evaluation process evolved through a series of critical encounters into a more proactive one.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Research limitations\/implications<\/jats:title><jats:p>The authors studied only one case so broad generalizations directly from the case cannot be made. However, the results could hold true for similar settings. The authors believe that the results are interesting to both researchers and practitioners since in outsourcing relationships the consortium kind of arrangement is becoming more common.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Originality\/value<\/jats:title><jats:p>The authors analyze the evolution of IS evaluation processes in a setting with multiple client and vendor organizations and reveal the complexities of system evaluation in such settings. Moreover, they introduce three new concepts: reactive, transitional and proactive evaluation processes. These concepts can help researchers and practitioners to better understand the nature of the IS evaluation processes and to possibly choose an evaluation approach that has a better \u201cfit\u201d with the development situation at hand.<\/jats:p><\/jats:sec>","DOI":"10.1108\/14637151111166150","type":"journal-article","created":{"date-parts":[[2011,9,24]],"date-time":"2011-09-24T07:24:56Z","timestamp":1316849096000},"page":"711-731","source":"Crossref","is-referenced-by-count":3,"title":["Emerging evaluation processes in consortium\u2010based outsourced system development"],"prefix":"10.1108","volume":"17","author":[{"given":"Antti","family":"Nurmi","sequence":"first","affiliation":[]},{"given":"Petri","family":"Hallikainen","sequence":"additional","affiliation":[]},{"given":"Matti","family":"Rossi","sequence":"additional","affiliation":[]}],"member":"140","reference":[{"key":"key2022031719510906600_b1","doi-asserted-by":"crossref","unstructured":"Ackermann, F., Franco, L.A., Gallupe, B. and Parent, M. (2005), \u201cGSS for multi\u2010organizational collaboration: reflections on process and content\u201d, Group Decision and Negotiation, Vol. 14, pp. 307\u201031.","DOI":"10.1007\/s10726-005-0317-4"},{"key":"key2022031719510906600_b2","doi-asserted-by":"crossref","unstructured":"Baskerville, R.L. (1999), \u201cInvestigation information systems with action research\u201d, Communications of the Association for Information Systems Research, Vol. 2, pp. 1\u201032.","DOI":"10.17705\/1CAIS.00219"},{"key":"key2022031719510906600_b3","doi-asserted-by":"crossref","unstructured":"Braa, K. and Vidgen, R. (1999), \u201cInterpretation, intervention and reduction in the organizational laboratory: a framework for in\u2010context information systems research\u201d, Accounting, Management and Information Technologies, Vol. 9, pp. 25\u201047.","DOI":"10.1016\/S0959-8022(98)00018-6"},{"key":"key2022031719510906600_b4","doi-asserted-by":"crossref","unstructured":"Choudhury, V. and Sabherwal, R. (2003), \u201cPortfolios of control in outsourced software development projects\u201d, Information Systems Research, Vol. 14 No. 3, pp. 291\u2010314.","DOI":"10.1287\/isre.14.3.291.16563"},{"key":"key2022031719510906600_b5","doi-asserted-by":"crossref","unstructured":"Eisenhardt, K.M. (1985), \u201cControl: organizational and economic approaches\u201d, Management Science, Vol. 31 No. 2, pp. 134\u201049.","DOI":"10.1287\/mnsc.31.2.134"},{"key":"key2022031719510906600_b6","unstructured":"Eklin, M. (2005), \u201cTietoj\u00e4rjestelm\u00e4kehitys Sosioteknisen\u00e4 Prosessimallina. Tapaustutkimus Oodi\u2010tietoj\u00e4rjestelm\u00e4st\u00e4\u201d, unpublished Master's thesis, Helsinki School of Economics, Helsinki."},{"key":"key2022031719510906600_b7","doi-asserted-by":"crossref","unstructured":"Farbey, B., Land, F. and Target, D. (1992), \u201cEvaluating investments in IT\u201d, Journal of Information Technology, Vol. 7 No. 2, pp. 109\u201022.","DOI":"10.1177\/026839629200700207"},{"key":"key2022031719510906600_b8","doi-asserted-by":"crossref","unstructured":"Farbey, B., Land, F. and Targett, D. (1999), \u201cMoving IS evaluation forward: learning themes and research issues\u201d, Journal of Strategic Information Systems, Vol. 8 No. 2, pp. 189\u2010207.","DOI":"10.1016\/S0963-8687(99)00021-9"},{"key":"key2022031719510906600_b9","unstructured":"Gallivan, M.J. (1999), \u201cAnalyzing IT outsourcing relationships as alliances among multiple clients and vendors\u201d, in Wonseok, O. (Ed.), Proceedings of the Thirty\u2010second Hawaii International Conference on System Sciences, Maui, HI, USA, p. 7066."},{"key":"key2022031719510906600_b10","unstructured":"Giddens, A. (1984), The Constitution of Society, Polity Press, Cambridge."},{"key":"key2022031719510906600_b11","doi-asserted-by":"crossref","unstructured":"Hallikainen, P., Heikkil\u00e4, J., Peffers, K., Saarinen, T. and Wijnhoven, F. (1998), \u201cEvaluating information technology projects in finland: procedures, follow\u2010through, decision\u2010making and perceived evaluation quality\u201d, Journal of Global Information Management, Vol. 6 No. 4, pp. 23\u201033.","DOI":"10.4018\/jgim.1998100103"},{"key":"key2022031719510906600_b12","unstructured":"Heiskanen, A., Newman, M. and Simil\u00e4, J. (1996), \u201cSoftware contracting: a process model approach\u201d, International Conference on Information Systems, Cleveland, OH, pp. 51\u201062."},{"key":"key2022031719510906600_b13","doi-asserted-by":"crossref","unstructured":"Heiskanen, A., Newman, M. and Simil\u00e4, J. (2000), \u201cThe social dynamics of software development\u201d, Accounting, Management and Information Technologies, Vol. 10 No. 1, pp. 1\u201032.","DOI":"10.1016\/S0959-8022(99)00013-2"},{"key":"key2022031719510906600_b14","doi-asserted-by":"crossref","unstructured":"Huxham, C. (1991), \u201cFacilitating collaboration: issues in multi\u2010organizational group decision support in voluntary, informal collaborative settings\u201d, The Journal of the Operational Research Society, Vol. 42 No. 12, pp. 1037\u201045.","DOI":"10.1057\/jors.1991.198"},{"key":"key2022031719510906600_b15","doi-asserted-by":"crossref","unstructured":"Jones, S. and Hughes, J. (2001), \u201cUnderstanding IS evaluation as a complex social process: a case study of a UK local authority\u201d, European Journal of Information Systems, Vol. 10 No. 4, pp. 189\u2010203.","DOI":"10.1057\/palgrave.ejis.3000405"},{"key":"key2022031719510906600_b16","doi-asserted-by":"crossref","unstructured":"Keen, P.G.W. (1981), \u201cInformation systems and organizational change\u201d, Communications of the ACM, Vol. 24 No. 1, pp. 24\u201033.","DOI":"10.1145\/358527.358543"},{"key":"key2022031719510906600_b17","doi-asserted-by":"crossref","unstructured":"Kirsch, L.J. (1996), \u201cThe management of complex tasks in organizations: controlling the systems development process\u201d, Organization Science, Vol. 7 No. 1, pp. 1\u201021.","DOI":"10.1287\/orsc.7.1.1"},{"key":"key2022031719510906600_b18","doi-asserted-by":"crossref","unstructured":"Larsen, M.A. and Myers, M.D. (1999), \u201cWhen success turns into failure: a package\u2010driven business process re\u2010engineering project in the financial services industry\u201d, Journal of Strategic Information Systems, Vol. 8 No. 4, pp. 395\u2010417.","DOI":"10.1016\/S0963-8687(00)00025-1"},{"key":"key2022031719510906600_b20","doi-asserted-by":"crossref","unstructured":"Mcaulay, L., Doherty, N. and Keval, N. (2002), \u201cThe stakeholder dimension in information systems evaluation\u201d, Journal of Information Technology, Vol. 17 No. 4, pp. 241\u201055.","DOI":"10.1080\/0268396022000017734"},{"key":"key2022031719510906600_b19","doi-asserted-by":"crossref","unstructured":"Martinsuo, M. and Lehtonen, P. (2007), \u201cProgram and its initiation in practice: development program initiation in a public consortium\u201d, International Journal of Project Management, Vol. 25, pp. 337\u201045.","DOI":"10.1016\/j.ijproman.2007.01.011"},{"key":"key2022031719510906600_b21","unstructured":"Mohr, L.B. (1987), \u201cInnovation theory: an assessment from the vantage point of the new electronic technology in organizations\u201d, in Pennings, J.M. and Buitendam, A. (Eds), New Technology as Organizational Innovation: The Development and Diffusion of Microelectronics, Ballinger, Cambridge, MA."},{"key":"key2022031719510906600_b22","doi-asserted-by":"crossref","unstructured":"Newman, M. and Noble, F. (1990), \u201cUser involvement as an interaction process: a case study\u201d, Information Systems Research, Vol. 1 No. 1, pp. 89\u2010113.","DOI":"10.1287\/isre.1.1.89"},{"key":"key2022031719510906600_b23","doi-asserted-by":"crossref","unstructured":"Newman, M. and Robey, D. (1992), \u201cA social process model of user\u2010analyst relationships\u201d, MIS Quarterly, Vol. 16 No. 2, pp. 249\u201066.","DOI":"10.2307\/249578"},{"key":"key2022031719510906600_b24","doi-asserted-by":"crossref","unstructured":"Noble, F. and Newman, M. (1993), \u201cIntegrated system, autonomous departments; organizational invalidity and system change in a university\u201d, Journal of Management Studies, Vol. 30 No. 2, pp. 195\u2010219.","DOI":"10.1111\/j.1467-6486.1993.tb00301.x"},{"key":"key2022031719510906600_b25","unstructured":"Nonaka, I. (1991), \u201cThe knowledge\u2010creating company\u201d, Harvard Business Review, Vol. 69 No. 6, pp. 96\u2010104."},{"key":"key2022031719510906600_b26","doi-asserted-by":"crossref","unstructured":"Remenyi, D. and Sherwood\u2010Smith, M. (1999), \u201cMaximise information systems value by continuos participative evaluation\u201d, Logistics Information Management, Vol. 12 No. 1, pp. 14\u201031.","DOI":"10.1108\/09576059910256222"},{"key":"key2022031719510906600_b27","unstructured":"Saarinen, V. (2001), \u201cKohti Valtakunnallista Opintotietoj\u00e4rjestelm\u00e4\u00e4 \u2013 tapaustutkimus Kuuden yliopiston Yhteisty\u00f6st\u00e4\u201d, unpublished Master's thesis, University of Helsinki."},{"key":"key2022031719510906600_b28","doi-asserted-by":"crossref","unstructured":"Serafeimidis, V. and Smithson, S. (2000), \u201cInformation systems evaluation in practice: a case study of organizational change\u201d, Journal of Information Technology, Vol. 15 No. 2, pp. 93\u2010105.","DOI":"10.1177\/026839620001500202"},{"key":"key2022031719510906600_b29","doi-asserted-by":"crossref","unstructured":"Smithson, S. and Hirschheim, R. (1998), \u201cAnalysing information systems evaluation: another look at an old problem\u201d, European Journal of Information Systems, Vol. 7 No. 3, p. 158.","DOI":"10.1057\/palgrave.ejis.3000304"},{"key":"key2022031719510906600_b30","doi-asserted-by":"crossref","unstructured":"Stockdale, R. and Standing, C. (2006), \u201cAn interpretive approach to evaluating information systems: a content, context, process framework\u201d, European Journal of Operational Research, No. 173, pp. 1090\u2010102.","DOI":"10.1016\/j.ejor.2005.07.006"},{"key":"key2022031719510906600_b31","doi-asserted-by":"crossref","unstructured":"Tiwana, A. (2008), \u201cDoes technological modularity substitute for control? A study of alliance performance in software outsourcing\u201d, Strategic Management Journal, Vol. 29 No. 7, pp. 769\u201080.","DOI":"10.1002\/smj.673"},{"key":"key2022031719510906600_b32","doi-asserted-by":"crossref","unstructured":"Walsham, G. (1995), \u201cInterpretive case studies in IS research: nature and method\u201d, European Journal of Information Systems, Vol. 4 No. 2, pp. 74\u201081.","DOI":"10.1057\/ejis.1995.9"},{"key":"key2022031719510906600_b33","doi-asserted-by":"crossref","unstructured":"Ward, J., Taylor, P. and Bond, P. (1996), \u201cEvaluation and realisation of IS\/IT benefits: an empirical study of current practice\u201d, European Journal of Information Systems, Vol. 4 No. 4, p. 214.","DOI":"10.1057\/ejis.1996.3"},{"key":"key2022031719510906600_b34","unstructured":"Wehrs, W.E. (1999), \u201cA road map for IS\/IT evaluation\u201d, in Mahmood, M.A. and Szewczak, E. (Eds), Measuring Information Technology Investment Payoff: Contemporary Approaches, Idea Group Publishing, Hershey, PA."},{"key":"key2022031719510906600_b35","doi-asserted-by":"crossref","unstructured":"Weick, K.E. (1976), \u201cEducational organizations as loosely coupled systems\u201d, Administrative Science Quarterly, Vol. 21 No. 1, pp. 1\u201019.","DOI":"10.2307\/2391875"},{"key":"key2022031719510906600_b36","unstructured":"Willcocks, L. and Lester, S. (1993), \u201cHow do organizations evaluate and control information systems investments? Recent UK survey evidence\u201d, in Avison, D.E., Kendall, J.E. and Degross, J.I. (Eds), IFIP WG8.2 Working Group Information Systems Development: Human, Social, and Organizational Aspects, Noordwijkerhout, pp. 15\u201039."}],"container-title":["Business Process Management Journal"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/14637151111166150","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/14637151111166150\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/14637151111166150\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,25]],"date-time":"2025-07-25T00:36:28Z","timestamp":1753403788000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/bpmj\/article\/17\/5\/711-731\/256548"}},"subtitle":[],"editor":[{"given":"Stefan","family":"Smolnik","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2011,9,13]]},"references-count":36,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2011,9,13]]}},"alternative-id":["10.1108\/14637151111166150"],"URL":"https:\/\/doi.org\/10.1108\/14637151111166150","relation":{},"ISSN":["1463-7154"],"issn-type":[{"type":"print","value":"1463-7154"}],"subject":[],"published":{"date-parts":[[2011,9,13]]}}}