{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T22:11:51Z","timestamp":1772835111466,"version":"3.50.1"},"reference-count":27,"publisher":"Oxford University Press (OUP)","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2015,1,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>The World Conference on Stock Assessment Methods (July 2013) included a workshop on testing assessment methods through simulations. The exercise was made up of two steps applied to datasets from 14 representative fish stocks from around the world. Step 1 involved applying stock assessments to datasets with varying degrees of effort dedicated to optimizing fit. Step 2 was applied to a subset of the stocks and involved characteristics of given model fits being used to generate pseudo-data with error. These pseudo-data were then provided to assessment modellers and fits to the pseudo-data provided consistency checks within (self-tests) and among (cross-tests) assessment models. Although trends in biomass were often similar across models, the scaling of absolute biomass was not consistent across models. Similar types of models tended to perform similarly (e.g. age based or production models). Self-testing and cross-testing of models are a useful diagnostic approach, and suggested that estimates in the most recent years of time-series were the least robust. Results from the simulation exercise provide a basis for guidance on future large-scale simulation experiments and demonstrate the need for strategic investments in the evaluation and development of stock assessment methods.<\/jats:p>","DOI":"10.1093\/icesjms\/fst237","type":"journal-article","created":{"date-parts":[[2014,1,18]],"date-time":"2014-01-18T20:25:54Z","timestamp":1390076754000},"page":"19-30","source":"Crossref","is-referenced-by-count":82,"title":["Simulation testing the robustness of stock assessment models to error: some results from the ICES strategic initiative on stock assessment methods"],"prefix":"10.1093","volume":"72","author":[{"given":"J.J.","family":"Deroba","sequence":"first","affiliation":[{"name":"NOAA NMFS, 166 Water Street, Woods Hole, MA, USA"}]},{"given":"D.S.","family":"Butterworth","sequence":"additional","affiliation":[{"name":"Marine Resource Assessment and Management Group (MARAM), Department of Mathematics and Applied Mathematics, University of Cape Town, University Private Bag, Rondebosch 7701, South Africa"}]},{"suffix":"Jr","given":"R.D.","family":"Methot","sequence":"additional","affiliation":[{"name":"NOAA NMFS, 2725 Montlake Blvd. E, Seattle, WA, USA"}]},{"given":"J.A.A.","family":"De Oliveira","sequence":"additional","affiliation":[{"name":"Centre for Environment, Fisheries and Aquaculture Science (Cefas), Lowestoft Laboratory, Pakefield Road, Lowestoft, SuffolkNR33 OHT, UK"}]},{"given":"C.","family":"Fernandez","sequence":"additional","affiliation":[{"name":"ICES, H.C. Andersens Boulevard 44-46, DK Copenhagen V, Denmark"}]},{"given":"A.","family":"Nielsen","sequence":"additional","affiliation":[{"name":"National Institute of Aquatic Resources, Technical University of Denmark, Charlottenlund Castle, 2920 Charlottenlund, Denmark"}]},{"given":"S.X.","family":"Cadrin","sequence":"additional","affiliation":[{"name":"University of Massachusetts, School for Marine Science and Technology, 200 Mill Road, Suite 325, Fairhaven, MA, USA"}]},{"given":"M.","family":"Dickey-Collas","sequence":"additional","affiliation":[{"name":"ICES, H.C. Andersens Boulevard 44-46, DK Copenhagen V, Denmark"},{"name":"Wageningen Institute of Marine Resources and Ecosystem Studies (IMARES), Haringkade 1, 1976 Ijmuiden, Netherlands"}]},{"given":"C.M.","family":"Legault","sequence":"additional","affiliation":[{"name":"NOAA NMFS, 166 Water Street, Woods Hole, MA, USA"}]},{"given":"J.","family":"Ianelli","sequence":"additional","affiliation":[{"name":"NOAA NMFS, 7600 Sand Point Way NE, Seattle, WA, USA"}]},{"given":"J.L.","family":"Valero","sequence":"additional","affiliation":[{"name":"Center for the Advancement of Population Assessment Methodology (CAPAM), 8901 La Jolla Shores Drive, La Jolla, CA, USA"}]},{"given":"C.L.","family":"Needle","sequence":"additional","affiliation":[{"name":"Marine Scotland \u2013 Science, The Marine Laboratory, PO Box 101, 375 Victoria Road, Aberdeen AB11 9DB, UK"}]},{"given":"J.M.","family":"O'Malley","sequence":"additional","affiliation":[{"name":"NOAA NMFS, 1845 Wasp Blvd., Building 176, Honolulu, HI, USA"}]},{"given":"Y-J.","family":"Chang","sequence":"additional","affiliation":[{"name":"Joint Institute for Marine and Atmospheric Research, Pacific Islands Fisheries Science Center, University of Hawaii,Honolulu, HI, USA"}]},{"given":"G.G.","family":"Thompson","sequence":"additional","affiliation":[{"name":"NOAA NMFS, 7600 Sand Point Way NE, Seattle, WA, USA"}]},{"given":"C.","family":"Canales","sequence":"additional","affiliation":[{"name":"Instituto de Fomento Pesquero (IFOP), Avda. Blanco Encalada 839, Valparaiso, Chile"}]},{"given":"D.P.","family":"Swain","sequence":"additional","affiliation":[{"name":"Fisheries and Oceans Canada, Gulf Fisheries Centre, PO Box 5030, Moncton, NB E1C 9B6, Canada"}]},{"given":"D.C.M.","family":"Miller","sequence":"additional","affiliation":[{"name":"Wageningen Institute of Marine Resources and Ecosystem Studies (IMARES), Haringkade 1, 1976 Ijmuiden, Netherlands"}]},{"given":"N.T.","family":"Hintzen","sequence":"additional","affiliation":[{"name":"Wageningen UR, Institute for Marine Resources and Ecosystem Studies (IMARES), PO Box 68, 1970 AB Ijmuiden, Netherlands"}]},{"given":"M.","family":"Bertignac","sequence":"additional","affiliation":[{"name":"Ifremer, Unit\u00e9 Sciences et Technologies Halieutiques, ZI de la pointe du diable, CS 10070, 29280 Plouzan\u00e9, France"}]},{"given":"L.","family":"Ibaibarriaga","sequence":"additional","affiliation":[{"name":"Marine Research Division, AZTI-Tecnalia, Txatxarramendi ugartea z\/g, E-48395 Sukarrieta, Bizkaia, Spain"}]},{"given":"A.","family":"Silva","sequence":"additional","affiliation":[{"name":"IPMA-Instituto Portugu\u00eas do Mar e da Atmosfera, I.P. Av. Brasilia, 1449-006 Lisboa, Portugal"}]},{"given":"A.","family":"Murta","sequence":"additional","affiliation":[{"name":"IPMA-Instituto Portugu\u00eas do Mar e da Atmosfera, I.P. Av. Brasilia, 1449-006 Lisboa, Portugal"}]},{"given":"L.T.","family":"Kell","sequence":"additional","affiliation":[{"name":"ICCAT Secretariat, Coraz\u00f3n de Mar\u00eda 8, 28002 Madrid, Spain"}]},{"given":"C.L.","family":"de Moor","sequence":"additional","affiliation":[{"name":"Marine Resource Assessment and Management Group (MARAM), Department of Mathematics and Applied Mathematics, University of Cape Town, University Private Bag, Rondebosch 7701, South Africa"}]},{"given":"A.M.","family":"Parma","sequence":"additional","affiliation":[{"name":"Centro Nacional Patag\u00f3nico, Blvd. Brown 2915, 9120 Puerto Madryn, Chugut, Argentina"}]},{"given":"C.M.","family":"Dichmont","sequence":"additional","affiliation":[{"name":"CSIRO Wealth from Oceans Flagship, Queensland Biosciences Precinct, 306 Carmody Road, St. Lucia, QLD 4067, Australia"}]},{"given":"V.R.","family":"Restrepo","sequence":"additional","affiliation":[{"name":"International Seafood Sustainability Foundation, 805 15th Street NW, Washington, DC 20005, USA"}]},{"given":"Y.","family":"Ye","sequence":"additional","affiliation":[{"name":"Food and Agriculture Organization of the United Nations, Vialle delle Terme di Caracalla, 00153 Rome, Italy"}]},{"given":"E.","family":"Jardim","sequence":"additional","affiliation":[{"name":"European Commission Joint Research Center, TP 051, Via Enrico Fermi 2749, 21027 Ispra (VA), Italy"}]},{"given":"P.D.","family":"Spencer","sequence":"additional","affiliation":[{"name":"NOAA NMFS, 7600 Sand Point Way NE, Seattle, WA, USA"}]},{"given":"D.H.","family":"Hanselman","sequence":"additional","affiliation":[{"name":"NOAA NMFS, 17109 Pt. Lena Loop Road, Juneau, AK, USA"}]},{"given":"J.","family":"Blaylock","sequence":"additional","affiliation":[{"name":"Integrated Statistics, 172 Shearwater Way, Falmouth, MA, USA"}]},{"given":"M.","family":"Mood","sequence":"additional","affiliation":[{"name":"Integrated Statistics, 172 Shearwater Way, Falmouth, MA, USA"}]},{"given":"P.-J. F.","family":"Hulson","sequence":"additional","affiliation":[{"name":"NOAA NMFS, 17109 Pt. Lena Loop Road, Juneau, AK, USA"}]}],"member":"286","published-online":{"date-parts":[[2014,1,18]]},"reference":[{"key":"2019120606532126200_FST237C1","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-74075-1","volume-title":"Model Based Inference in the Life Sciences","author":"Anderson","year":"2008"},{"key":"2019120606532126200_FST237C2","doi-asserted-by":"crossref","first-page":"544","DOI":"10.1139\/f04-199","article-title":"Model averaging to estimate rebuilding targets for overfished stocks","volume":"62","author":"Brodziak","year":"2005","journal-title":"Canadian Journal of Fisheries and Aquatic Sciences"},{"key":"2019120606532126200_FST237C3","article-title":"Bayesian multimodel inference for stock assessment","author":"Brodziak","journal-title":"ICES Journal of Marine Science"},{"key":"2019120606532126200_FST237C4","doi-asserted-by":"crossref","first-page":"613","DOI":"10.1093\/icesjms\/fsm003","article-title":"Why a management procedure approach? Some positives and negatives","volume":"64","author":"Butterworth","year":"2007","journal-title":"ICES Journal of Marine Science"},{"key":"2019120606532126200_FST237C5","doi-asserted-by":"crossref","first-page":"985","DOI":"10.1006\/jmsc.1999.0532","article-title":"Experiences in the evaluation and implementation of management procedures","volume":"56","author":"Butterworth","year":"1999","journal-title":"ICES Journal of Marine Science"},{"key":"2019120606532126200_FST237C6","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1016\/j.fishres.2013.03.015","article-title":"Performance of stock assessments with misspecified age- and time-varying natural mortality","volume":"146","author":"Deroba","year":"2013","journal-title":"Fisheries Research"},{"key":"2019120606532126200_FST237C7","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1016\/j.fishres.2008.01.008","article-title":"Evaluating alternative estimators of fishery management reference points","volume":"94","author":"Haltuch","year":"2008","journal-title":"Fisheries Research"},{"key":"2019120606532126200_FST237C8","first-page":"232pp","volume-title":"Report of the Working Group on Methods of Fish Stock Assessments","author":"ICES","year":"2004"},{"key":"2019120606532126200_FST237C9","first-page":"250pp","volume-title":"Report of the Working Group on Methods of Fish Stock Assessment (WGMG), 10\u201319 October 2011","author":"ICES","year":"2011"},{"key":"2019120606532126200_FST237C10","first-page":"249pp","volume-title":"Working Group on Methods of Fish Stock Assessments (WGMG), 8\u201312 October 2012","author":"ICES","year":"2012"},{"key":"2019120606532126200_FST237C11","first-page":"15 pp","volume-title":"Report on the Classification of Stock Assessment Methods developed by SISAM","author":"ICES","year":"2012"},{"key":"2019120606532126200_FST237C12","doi-asserted-by":"crossref","first-page":"640","DOI":"10.1093\/icesjms\/fsm012","article-title":"FLR: an open-source framework for the evaluation and development of management strategies","volume":"64","author":"Kell","year":"2007","journal-title":"ICES Journal of Marine Science"},{"key":"2019120606532126200_FST237C13","doi-asserted-by":"crossref","first-page":"834","DOI":"10.1006\/jmsc.1999.0544","article-title":"An evaluation of management procedures for implementing a precautionary approach in the ICES context for North Sea plaice","volume":"56","author":"Kell","year":"1999","journal-title":"ICES Journal of Marine Science"},{"key":"2019120606532126200_FST237C14","first-page":"133p","volume-title":"Stock assessment of Georges Bank yellowtail flounder for 2012","author":"Legault","year":"2012"},{"key":"2019120606532126200_FST237C15","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1016\/j.fishres.2008.06.014","article-title":"Evaluating methods for estimating process and observation error variances in statistical catch-at-age analysis","volume":"94","author":"Linton","year":"2008","journal-title":"Fisheries Research"},{"key":"2019120606532126200_FST237C16","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1111\/j.1467-2979.2012.00473.x","article-title":"Measuring uncertainty in fisheries stock assessment: the delta method, bootstrap, and MCMC","volume":"14","author":"Magnusson","year":"2012","journal-title":"Fish and Fisheries"},{"key":"2019120606532126200_FST237C18","doi-asserted-by":"crossref","first-page":"473","DOI":"10.1006\/jmsc.1999.0481","article-title":"The retrospective problem in sequential population analysis: an investigation using cod fishery and simulated data","volume":"56","author":"Mohn","year":"1999","journal-title":"ICES Journal of Marine Science"},{"key":"2019120606532126200_FST237C19","volume-title":"NOAA Fisheries Toolbox, Version 3.1","author":"National Oceanic and Atmospheric Administration (NOAA)","year":"2013"},{"key":"2019120606532126200_FST237C20","volume-title":"Improving Fish Stock Assessments","author":"National Research Council (NRC)","year":"1998"},{"key":"2019120606532126200_FST237C21","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1080\/19425120.2011.611005","article-title":"A simulation-based method to determine model misspecification: examples using natural mortality and population dynamics models","volume":"3","author":"Piner","year":"2011","journal-title":"Marine and Coastal Fisheries"},{"key":"2019120606532126200_FST237C22","doi-asserted-by":"crossref","first-page":"631","DOI":"10.1071\/MF01008","article-title":"Evaluation of management tools for Australia's South East Fishery. 2. How well can management quantities be estimated?","volume":"53","author":"Punt","year":"2002","journal-title":"Marine and Freshwater Research"},{"key":"2019120606532126200_FST237C23","first-page":"217","article-title":"A meta-analytic approach to quantifying scientific uncertainty in stock assessments","volume":"109","author":"Ralston","year":"2011","journal-title":"Fishery Bulletin"},{"key":"2019120606532126200_FST237C24","volume-title":"Analyses of simulated data sets in support of the NRC study on stock assessment methods","author":"Restrepo","year":"1998"},{"key":"2019120606532126200_FST237C25","first-page":"pp. 78","volume-title":"Characterizing and presenting assessment uncertainty. SEDAR Procedural Workshop IV","author":"Carmichael","year":"2010"},{"key":"2019120606532126200_FST237C26","doi-asserted-by":"crossref","first-page":"927","DOI":"10.1071\/MF11006","article-title":"Performance of a fisheries catch-at-age model (Stock Synthesis) in data-limited situations","volume":"62","author":"Wetzel","year":"2011","journal-title":"Marine and Freshwater Research"},{"key":"2019120606532126200_FST237C27","doi-asserted-by":"crossref","first-page":"2275","DOI":"10.1139\/f06-111","article-title":"Performance of time-varying catchability estimators in statistical catch-at-age analysis","volume":"63","author":"Wilberg","year":"2006","journal-title":"Canadian Journal of Fisheries and Aquatic Sciences"},{"key":"2019120606532126200_FST237C28","first-page":"714","article-title":"Approaches to the management of waterfowl under uncertainty","volume":"25","author":"Williams","year":"1997","journal-title":"Wildlife Society Bulletin"}],"container-title":["ICES Journal of Marine Science"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/icesjms\/article-pdf\/72\/1\/19\/31224159\/fst237.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/icesjms\/article-pdf\/72\/1\/19\/31224159\/fst237.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,12,6]],"date-time":"2019-12-06T06:58:51Z","timestamp":1575615531000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/icesjms\/article\/72\/1\/19\/815567"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,1,18]]},"references-count":27,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2014,1,18]]},"published-print":{"date-parts":[[2015,1,1]]}},"URL":"https:\/\/doi.org\/10.1093\/icesjms\/fst237","relation":{},"ISSN":["1095-9289","1054-3139"],"issn-type":[{"value":"1095-9289","type":"electronic"},{"value":"1054-3139","type":"print"}],"subject":[],"published-other":{"date-parts":[[2015,1]]},"published":{"date-parts":[[2014,1,18]]}}}