{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,28]],"date-time":"2026-02-28T06:49:21Z","timestamp":1772261361601,"version":"3.50.1"},"reference-count":46,"publisher":"Copernicus GmbH","issue":"3","license":[{"start":{"date-parts":[[2012,5,11]],"date-time":"2012-05-11T00:00:00Z","timestamp":1336694400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/3.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Geosci. Model Dev."],"abstract":"<jats:p>Abstract. We summarise results from a workshop on \"Model Benchmarking and Quality Assurance\" of the EU-Network of Excellence ACCENT, including results from other activities (e.g. COST Action 732) and publications. A formalised evaluation protocol is presented, i.e. a generic formalism describing the procedure of how to perform a model evaluation. This includes eight steps and examples from global model applications which are given for illustration. The first and important step is concerning the purpose of the model application, i.e. the addressed underlying scientific or political question. We give examples to demonstrate that there is no model evaluation per se, i.e. without a focused purpose. Model evaluation is testing, whether a model is fit for its purpose. The following steps are deduced from the purpose and include model requirements, input data, key processes and quantities, benchmark data, quality indicators, sensitivities, as well as benchmarking and grading. We define \"benchmarking\" as the process of comparing the model output against either observational data or high fidelity model data, i.e. benchmark data. Special focus is given to the uncertainties, e.g. in observational data, which have the potential to lead to wrong conclusions in the model evaluation if not considered carefully.<\/jats:p>","DOI":"10.5194\/gmd-5-611-2012","type":"journal-article","created":{"date-parts":[[2012,5,11]],"date-time":"2012-05-11T05:14:11Z","timestamp":1336713251000},"page":"611-618","source":"Crossref","is-referenced-by-count":10,"title":["The ACCENT-protocol: a framework for benchmarking and model evaluation"],"prefix":"10.5194","volume":"5","author":[{"given":"V.","family":"Grewe","sequence":"first","affiliation":[]},{"given":"N.","family":"Moussiopoulos","sequence":"additional","affiliation":[]},{"given":"P.","family":"Builtjes","sequence":"additional","affiliation":[]},{"given":"C.","family":"Borrego","sequence":"additional","affiliation":[]},{"given":"I. S. A.","family":"Isaksen","sequence":"additional","affiliation":[]},{"given":"A.","family":"Volz-Thomas","sequence":"additional","affiliation":[]}],"member":"3145","published-online":{"date-parts":[[2012,5,11]]},"reference":[{"key":"ref1","unstructured":"ACCENT (Atmospheric Composition Change the European Network of Excellence): Answers to the Urbino Questions, ACCENTs first policy-driven synthesis, edited by: Raes F. and Hjorth J., published by: ACCENT Secretariat, University Urbino, Italy, ISBN 92-79-02413-2, http:\/\/www4.nilu.no\/portal\/publications\/accent-series-reports, 2006."},{"key":"ref2","doi-asserted-by":"crossref","unstructured":"ACCENT (Atmospheric Composition Change the European Network of Excellence): Editorial, edited by: Fuzzi, S. and Maione, M., Atmos. Environm., 43, 5136\u20135137, 2009.","DOI":"10.1016\/j.atmosenv.2009.08.006"},{"key":"ref3","doi-asserted-by":"crossref","unstructured":"Aghedo, A. M., Bowman, K. W., Shindell, D. T., and Faluvegi, G.: The impact of orbital sampling, monthly averaging and vertical resolution on climate chemistry model evaluation with satellite observations, Atmos. Chem. Phys., 11, 6493\u20136514, https:\/\/doi.org\/10.5194\/acp-11-6493-2011, 2011.","DOI":"10.5194\/acp-11-6493-2011"},{"key":"ref4","unstructured":"AIAA (American Institute of Aeronautics and Astronautics): Guide for the Verification and Validation of Computational Fluid Dynamics Simulations, American Institute of Aeronautics and Astronautics, AIAA-G-077-1998, Reston, VA, USA, 1998."},{"key":"ref5","doi-asserted-by":"crossref","unstructured":"Braesicke, P. and Pyle, J.: Sensitivity of dynmaics and ozone to different representations of SSTs in the Unified Model, Q. J. Roy. Meteor. Soc., 130, 2033\u20132045, 2004.","DOI":"10.1256\/qj.03.183"},{"key":"ref6","unstructured":"Britter, R. and Schatzmann, M.: Model evaluation guidance and protocol document, COST Action 732, 28 pp., ISBN 3-00-018312-4, Hamburg, Germany, 2007a."},{"key":"ref7","unstructured":"Britter, R. and Schatzmann, M.: Background and justification document to support the model evaluation guidance and protocol, COST Action 732, p. 88, ISBN 3-00-018312-4, Hamburg, Germany, 2007b."},{"key":"ref8","doi-asserted-by":"crossref","unstructured":"Burkhardt, U. and K\u00e4rcher, B.: Global radiative forcing from contrail cirrus, Nature Climate Change, 1, 54\u201358, 2011.","DOI":"10.1038\/nclimate1068"},{"key":"ref9","doi-asserted-by":"crossref","unstructured":"Crutzen, P. J. and Arnold, F.: Nitric acid cloud formation in the cold Antarctic stratosphere: a major cause for the springtime &quot;ozone hole&quot;, Nature, 324, 651\u2013655, 1986.","DOI":"10.1038\/324651a0"},{"key":"ref10","unstructured":""},{"key":"ref11","doi-asserted-by":"crossref","unstructured":"Douglass, A. R., Prather, M. J., Hall, T. M., Strahan, S. E., Rasch, P. J., Sparling, L. C., Coy, L., and Rodriguez, J. M.: Choosing meteorological input for the global modeling initiative assessment of high-speed aircraft, J. Geophys. Res., 104, 27545\u201327564, 1999.","DOI":"10.1029\/1999JD900827"},{"key":"ref12","unstructured":""},{"key":"ref13","doi-asserted-by":"crossref","unstructured":"%Fuglestvedt, Jan S., Terje Berntsen, Gunnar Myhre, Kristin Rypdal and Ragnhild Bieltvedt Skeie, %Climate forcing from the Transport Sectors. Proceedings of the National Academy of Sciences (PNAS), %vol 105 (no. 2): pp. 454-458. %2008.","DOI":"10.1073\/pnas.0702958104"},{"key":"ref14","doi-asserted-by":"crossref","unstructured":"Garny, H., Dameris, M., and Stenke, A.: Impact of prescribed SSTs on climatologies and long-term trends in CCM simulations, Atmos. Chem. Phys., 9, 6017\u20136031, https:\/\/doi.org\/10.5194\/acp-9-6017-2009, 2009.","DOI":"10.5194\/acp-9-6017-2009"},{"key":"ref15","unstructured":""},{"key":"ref16","doi-asserted-by":"crossref","unstructured":"Grewe, V.: Impact of climate variability on tropospheric ozone, Sci. Total Environ., 374, 167\u2013181, 2007.","DOI":"10.1016\/j.scitotenv.2007.01.032"},{"key":"ref17","doi-asserted-by":"crossref","unstructured":"Grewe, V. and Sausen, R.: Comment on &quot;Quantitative performance metrics for stratospheric-resolving chemistry-climate models&quot; by Waugh and Eyring (2008), Atmos. Chem. Phys., 9, 9101\u20139110, https:\/\/doi.org\/10.5194\/acp-9-9101-2009, 2009.","DOI":"10.5194\/acp-9-9101-2009"},{"key":"ref18","doi-asserted-by":"crossref","unstructured":"Grewe, V., Dameris, M., Fichter, C., and Sausen, R.: Impact of aircraft NOx emissions. Part 1: Interactively coupled climate-chemistry simulations and sensitivities to climate-chemistry feedback, lightning and model resolution, Meteorol. Z., 3, 177\u2013186, 2002.","DOI":"10.1127\/0941-2948\/2002\/0011-0177"},{"key":"ref19","doi-asserted-by":"crossref","unstructured":"%Grewe, V., %Technical Note: A diagnostic for ozone contributions of various NO$\\\\rm_x$ %emissions in multi-decadal chemistry-climate model simulations, Atmos. Chem. Physc. 4, 327-342, 2004. %","DOI":"10.5194\/acpd-4-327-2004"},{"key":"ref20","unstructured":"%Grewe, V., %Impact of lightning on air chemistry and climate, IN: Lightning: Principles, Instruments and Applications, %Eds H.D. Betz, U Schumann, P. Laroche, Springer, 2009. %"},{"key":"ref21","unstructured":""},{"key":"ref22","doi-asserted-by":"crossref","unstructured":"%Hoor, P., J. Borken-Kleefeld, D. Caro, O. Dessens, O. Endresen, M. Gauss, V. Grewe, %D. Hauglustaine, I. S. A. Isaksen, P. J\u00f6ckel, J. Lelieveld, E. Meijer, D. Olivie, %M. Prather, C. Schnadt Poberaj, J. Staehelin, Q. Tang, J. van Aardenne, %P. van Velthoven, and R. Sausen, %The impact of traffic emissions on atmospheric ozone and OH: results from QUANTIFY, %Atmos. Chem. Phys., 9, 3113-3136, 2009.","DOI":"10.5194\/acp-9-3113-2009"},{"key":"ref23","doi-asserted-by":"crossref","unstructured":"J\u00f6ckel, P., Kerkweg, A., Pozzer, A., Sander, R., Tost, H., Riede, H., Baumgaertner, A., Gromov, S., and Kern, B.: Development cycle 2 of the Modular Earth Submodel System (MESSy2), Geosci. Model Dev., 3, 717\u2013752, https:\/\/doi.org\/10.5194\/gmd-3-717-2010, 2010.","DOI":"10.5194\/gmd-3-717-2010"},{"key":"ref24","unstructured":"Kawa, S. R., Anderson, J. G., Baughcum, S. L., Brock, C. A., Brune, W. H., Cohen, R. C., Kinnison, D. E., Newman, P. A., Rodriguez, J. M., Stolarski, R. S., Waugh, D., and Wofsy, S. C.: Assessment of the Effects of High-Speed Aircraft in the Stratosphere: 1998, NASA-Report, NASA\/TP-1999-209237, 1999."},{"key":"ref25","doi-asserted-by":"crossref","unstructured":"Lary, D. J. and Aulov, O.: Space-based measurements of HCl: Intercomparison and historical context, J. Geophys. Res., 113, D15S04, https:\/\/doi.org\/10.1029\/2007JD008715, 2008.","DOI":"10.1029\/2007JD008715"},{"key":"ref26","doi-asserted-by":"crossref","unstructured":"Lee, D. S., Pitari, ,G., Grewe, V., Gierens, K., Penner, J. E., Petzold, A., Prather, M. J., Schumann, U., Bais, A., Berntsen, T., Iachetti, D., Lim, L. L., and Sausen, R.: Transport impacts on atmosphere and climate: Aviation, Atmos. Environm. 44, 4678\u20134734, 2010.","DOI":"10.1016\/j.atmosenv.2009.06.005"},{"key":"ref27","unstructured":"Moussiopoulos, N. and Isaksen, I.: Proceedings of the Workshop on Model Benchmarking and Quality Assurance,ACCENT Series Reports 2.2007, 60 p., http:\/\/www4.nilu.no\/farcry_accent\/download.cfm?DownloadFile=3DCD0697-17A4-8DB2-8C63CCDF7478BCB6, 2007."},{"key":"ref28","unstructured":"Park, J. H., Ko, M. K. W., Jackman, C. H., Plumb, R. A., Kaye, J. A., and Sage, K. H.: M&amp;M-2, NASA: Models and Measurements Intercomparison II. TM-1999-209554, 502 pp., 1999."},{"key":"ref29","doi-asserted-by":"crossref","unstructured":"Parrish, D. D., Millet, D. B., and Goldstein, A. H.: Increasing ozone in marine boundary layer inflow at the west coasts of North America and Europe, Atmos. Chem. Phys., 9, 1303\u20131323, https:\/\/doi.org\/10.5194\/acp-9-1303-2009, 2009.","DOI":"10.5194\/acp-9-1303-2009"},{"key":"ref30","unstructured":"Penner, J. E., Lister, D. H., Griggs, D. J., Dokken, D. J., and McFarland, M.: Aviation and the Global Atmosphere. A special report of IPCC Working Groups I and III, 373, Cambridge Univ. Press, Cambridge, UK, 1999."},{"key":"ref31","doi-asserted-by":"crossref","unstructured":"%Pfister, G.G., L. K. Emmons, P. G. Hess, J.-F. Lamarque, J. J. Orlando, S. Walters, %A. Guenther, P. I. Palmer, and P. J. Lawrence, %Contribution of isoprene to chemical budgets: A model tracer study with the NCAR CTM MOZART-4 %J. Geophys. Res., 113, D5, http:\/\/dx.doi.org\/10.1029\/2007JD008948https:\/\/doi.org\/10.1029\/2007JD008948, 2008","DOI":"10.1029\/2007JD008948"},{"key":"ref32","doi-asserted-by":"crossref","unstructured":"Richter, A. and Burrows, J. P.: Tropospheric NO2 from GOME measurements, Adv. Space Res., 29, 1673\u20131683, 2002.","DOI":"10.1016\/S0273-1177(02)00100-X"},{"key":"ref33","doi-asserted-by":"crossref","unstructured":"Russell III, J. M., Gordley, L. L., Park, J. H., Drayson, S. R., Hesketh, D. H., Cicerone, R. J., Tuck, A. F., Frederick, J. E., Harries, J. E., and Crutzen, P.: The Halogen Occultation Experiment, J. Geophys. Res., 98, 10777\u201310797, 1993.","DOI":"10.1029\/93JD00799"},{"key":"ref34","unstructured":"Saltelli, A., Chan, K., and Scott, E. M.: Sensitivity Analysis, John Wiley &amp; Sons, Chichester, New York, 2000."},{"key":"ref35","doi-asserted-by":"crossref","unstructured":"Schlesinger, S., Crosbie, R. E., Gagn\\\\'{n}e, R. E., Innis, G. S., Lalwani, C. S., Loch, J., Sylvester, R. J., Wright, R. D., Kheir, N., and Bartos, D.: Terminology for Model Credibility, Simulation, 32, 103\u2013104, 1979.","DOI":"10.1177\/003754977903200304"},{"key":"ref36","doi-asserted-by":"crossref","unstructured":"Stevenson, D. S., Dentener, F. J., Schultz, M. G., Ellingsen, K., van Noije, T. P. C., Wild, O., Zeng, G., Amann, M., Atherton, C. S., Bell, N., Bergmann, D. J., Bey, I., Butler, T., Cofala, J., Collins, W. J., Derwent, R. G., Doherty, R. M., Drevet, J., Eskes, H. J., Fiore, A. M., Gauss, M., Hauglustaine, D. A., Horowitz, L. W., Isaksen, I. S. A., Krol, M. C., Lamarque, J.-F., Lawrence, M. G., Montanaro, V., M\u00fcller, J.-F., Pitari, G., Prather, M. J., Pyle, J. A., Rast, S., Rodriguez, J. M., Sanderson, M. G., Savage, N. H., Shindell, D. T., Strahan, S. E., Sudo, K., and Szopa, S.: Multimodel ensemble simulations of present-day and near-future tropospheric ozone, J. Geophys. Res., 111, D08301, https:\/\/doi.org\/10.1029\/2005JD006338, 2006.","DOI":"10.1029\/2005JD006338"},{"key":"ref37","unstructured":"SPARC Report on the Evaluation of Chemistry-Climate Models: SPARC Report No. 5, edited by: Eyring, V., Shepherd, T. G., and Waugh, D. W., WCRP-132, WMO\/TD-No. 1526, http:\/\/www.atmosp.physics.utoronto.ca\/SPARC, 2010."},{"key":"ref38","doi-asserted-by":"crossref","unstructured":"Taylor, K. E.: Summarizing multiple aspects of model performance in a single diagram, J. Geophys. Res., 106, 7183\u20137192, 2001.","DOI":"10.1029\/2000JD900719"},{"key":"ref39","unstructured":""},{"key":"ref40","unstructured":"%TRADEOFF - Final report of the EU-project TRADEOFF: Aircraft emissions: Contribution of various climate compounds to %changes in composition and radiative forcing - tradeoff to reduce atmospheric impact, 2003. %"},{"key":"ref41","doi-asserted-by":"crossref","unstructured":"%Wang, Z.S., Chien, C.-J., and Tonnesen, G.S., %Development of a tagged species source apportionment algorithm to characterize three-dimensional transport and transformation of %precursors and secondary pollutants, %J. Geophys. Res., 114, D21, http:\/\/dx.doi.org\/10.1029\/2008JD010846https:\/\/doi.org\/10.1029\/2008JD010846, 2009.","DOI":"10.1029\/2008JD010846"},{"key":"ref42","doi-asserted-by":"crossref","unstructured":"van Noije, T. P. C., Eskes, H. J., Dentener, F. J., Stevenson, D. S., Ellingsen, K., Schultz, M. G., Wild, O., Amann, M., Atherton, C. S., Bergmann, D. J., Bey, I., Boersma, K. F., Butler, T., Cofala, J., Drevet, J., Fiore, A. M., Gauss, M., Hauglustaine, D. A., Horowitz, L. W., Isaksen, I. S. A., Krol, M. C., Lamarque, J.-F., Lawrence, M. G., Martin, R. V., Montanaro, V., M\u00fcller, J.-F., Pitari, G., Prather, M. J., Pyle, J. A., Richter, A., Rodriguez, J. M., Savage, N. H., Strahan, S. E., Sudo, K., Szopa, S., and van Roozendael, M.: Multi-model ensemble simulations of tropospheric NO2 compared with GOME retrievals for the year 2000, Atmos. Chem. Phys., 6, 2943\u20132979, https:\/\/doi.org\/10.5194\/acp-6-2943-2006, 2006.","DOI":"10.5194\/acp-6-2943-2006"},{"key":"ref43","doi-asserted-by":"crossref","unstructured":"%Wild, O., Modelling the global tropospheric ozone budget: exploring the %variability in current models, Atmos. Chem. Phys. 7, 2643-2660, 2007.","DOI":"10.5194\/acp-7-2643-2007"},{"key":"ref44","doi-asserted-by":"crossref","unstructured":"Waugh, D. W. and Eyring, V.: Quantitative performance metrics for stratospheric-resolving chemistry-climate models, Atmos. Chem. Phys., 8, 5699\u20135713, https:\/\/doi.org\/10.5194\/acp-8-5699-2008, 2008.","DOI":"10.5194\/acp-8-5699-2008"},{"key":"ref45","unstructured":"WMO: Scientific assessment of ozone depletion: 2010, World Meteorological Organisation, Geneva, 2011."},{"key":"ref46","unstructured":"%Zimmermann, P., Ozone in the troposphere over Europe: A source segregated analysis from a global point of view, %Proccedings of EUROTRAC Symposium '98, Eds P.M. Borrell and P Borell, WITPress, Southampton, 1999."}],"container-title":["Geoscientific Model Development"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/gmd.copernicus.org\/articles\/5\/611\/2012\/gmd-5-611-2012.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,15]],"date-time":"2025-02-15T08:02:32Z","timestamp":1739606552000},"score":1,"resource":{"primary":{"URL":"https:\/\/gmd.copernicus.org\/articles\/5\/611\/2012\/"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2012,5,11]]},"references-count":46,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2012]]}},"URL":"https:\/\/doi.org\/10.5194\/gmd-5-611-2012","relation":{"has-preprint":[{"id-type":"doi","id":"10.5194\/gmdd-4-3599-2011","asserted-by":"subject"},{"id-type":"doi","id":"10.5194\/gmdd-4-3599-2011","asserted-by":"object"}]},"ISSN":["1991-9603"],"issn-type":[{"value":"1991-9603","type":"electronic"}],"subject":[],"published":{"date-parts":[[2012,5,11]]}}}