Report by the Institute of Marine and Antarctic Studies: Reproducing the mortality model in Neira 2011
Andrew Wadsley has made several recent statements in the media (e.g. Tasmania Times 26/8/2012) that Neira’s (2011) calculation of spawning biomass is wrong and that his results cannot be reproduced.
We have conducted an internal review of Neira’s analysis and requested an independent external review from SARDI. Both supported the analysis by Neira and showed that his method was easily reproducible.
Despite making this review known (Attachment 1), Wadsley’s message persists:- that this work is bad science, is falsified, scientific method hasn’t been followed and that our results have been fabricated to suit industry. He has been active in the media with this message and it’s been repeated by others including the Tasmanian Greens and the Tasmanian Conservation Trust. This message has also been communicated through mainstream media (eg ABC News TV and radio) and in Parliament and more recently presented at a public rally.
We do not wish to debate the merits of Neira’s study in the media, but believe that a more detailed explanation of why we support his work is warranted.
Neira 2011’s mortality model
Neira (2011) produced a spawning stock biomass estimate using the daily egg production method (DEPM), a method that is widely used to produce biomass estimates for pelagic fish stocks.
A central component of the DEPM is the mortality model, which estimates the daily egg production, P0. The data in DEPM studies are typically skewed and noisy. Consequently care needs to be taken with the method used to fit the mortality model to the data. Neira (2011) used two methods: non-linear regression (although citing Lo et al., 1996, the actual method is a standard non-linear regression using non-linear least squares (NLS)) and a generalized linear model with a negative binomial error distribution (Cubillos et al., 2007; Neira and Lyle 2011). These methods are well established in the DEPM peer reviewed literature and provided comparable results.
Neira (2011) describes the methods he used with the following statement:
“Two functions were fitted to the daily egg abundance-at-age data, namely the traditional least squares non-linear regression (NLS) model (Lo et al., 1996), and a generalized linear model (GLM) using a negative binomial error distribution (Cubillos et al., 2007; Neira and Lyle, 2011).”
This description is sufficient to enable fisheries scientists with experience in DEPM studies or a statistical background to repeat the analysis as the methods are standard and the name provides sufficient description (in the same way basic statistical operations are not referenced in scientific reports). For example, non-linear regression is a completely standard routine and NLS refers to non-linear least squares – the statistical method used to fit non-linear regressions. Indeed many statistical tools like R have routines called “NLS”.
However, to someone new to DEPM studies or applied statistics these methods may be unfamiliar. Should such a person want to apply these methods they may choose to examine the citations (although perusal of a statistics textbook or Google would provide a succinct description). In this case the chosen citation for non-linear regression is not the most helpful.
Lo et al. (1996) only mentions non-linear regression in the methods section on temperature dependent mortality:
“All coefficients were estimated by nonlinear regression (Chambers and Hastie 1992) assuming additive errors.”
Reference to the actual method used to fit the mortality model was given as Picquelle and Stauffer (1985), and no mention of NLS.
Fortunately the topic can be found in many statistical textbooks or through internet searches for “NLS statistics” or “NLS regression” or “non linear regression (NLS)”.
Andrew Wadsley’s Analyses
Wadsley published an analysis of Neira’s mortality curve in the Tasmanian Times (TT) article: “Margiris: UTAS VC must investigate” (Attachment 2) (Note that he then subsequently published additional analyses as errors in his approach were progressively identified by respondents to the article).
Wadsley may be unfamiliar with non-linear regression and NLS and has consequently resorted to the citations in Neira to determine what non-linear regression (NLS) is. Unfortunately those citations do not give an explicit account of NLS and Wadsley has interpreted unrelated sections of those texts to be a description of NLS.
Here we examine how Wadsley’s analyses vary from Neira’s and why they are flawed.
Source: This method is described in Wadsley’s original analysis (Attachment 2).
Description: This method fitted an exponential curve in Microsoft Excel. The method was only applied to non-zero abundances.
Flaws in Wadsley’s approach:
1. Zero egg abundances are valid and supply informative data which has been completely disregarded.
2. Microsoft Excel fits exponential curves using linear regression (fitted with ordinary least squares) on log transformed data. This is a completely different method to those applied by Neira, which consequently is expected to provide a different answer.
The closest method in the literature is the log-linear model, however this has a substantial negative downward bias that must be corrected (Ward et al., 2011) and was not performed by Wadsley. Applying this correction to Dr. Wadsley’s approach (where only positive abundances are considered) increases P0 to exceed Neira’s original estimate (ie implies a higher harvest than would be indicated through the Niera analysis).
Origin: Neira (2011) cannot be interpreted to justify this method. It suggests that Wadsley was either i) unaware of the difference between linear regression of log transformed data and non-linear regression or ii) unaware that the two methods were likely to provide substantially different answers.
Source: This method is described as the “Picquelle and Stauffer” method in Wadsley’s subsequent analysis (Attachment 3).
Description: This method claims to have “used the inbuilt non-linear trend function in MS Excel to calculate NLS exponential trends”
Flaw in Wadsley’s approach: MS Excel has no in built non-linear regression methods. Non linear regressions (using non linear least squares,) must be fitted using custom application of the Solver add-in or other third party add-ins. In fact Wadsley has applied the same method as Method 1 (which we confirmed by obtaining Wadsley’s results by applying his Method 1 to the new data set).
Origin: Neira (2011) cannot be interpreted to justify this method. Wadsley’s statement above indicates that he is unaware of the difference between fitting log transformed data using linear regression and using non-linear regression. This distinction is crucial in DEPM studies.
Source: This method is described as the “Lo et al.” method in Wadsley’s subsequent analysis (Attachment 3).
Description: This method bins the data into half day groups and calculates the mean age and abundance for each group of data points. Method 1 is then applied to these means (note that there are no zero abundances in these binned data points).
Flaw in Wadsley’s approach: This is not the method used in Neira. There is no rationale for this approach and it severely reduces the available data for fitting the mortality model. This is evident from the variability in parameters observed by Wadsley (P0 varies by a factor of four between Wadsley’s analyses of 8a and 8b).
Origin: Lo et al. (1996) binned their data in this manner before using an unspecified method to fit the mortality model. Wadsley has mistakenly considered this as a possible description of NLS (It is not labelled or referred to by Lo et al. 1996 as such). In Lo et al. (2005), they state that they have discontinued using this aggregation step.
The methods used by Neira (2011) have been found to be completely reproducible and have been independently verified by both IMAS and SARDI.
In addition to using an inappropriate method to reproduce Neira’s results, Wadsley appears to have misunderstood the procedures used in the literature, following the wrong citations in attempting to justify his claims. Scientists familiar with DEPM studies or common statistical terminology would not have the same problem.
Nancy Lo, widely considered to be the doyen of DEPM, provided an unsolicited critique of Wadley’s original discussion paper, and reached a similar conclusion to ours (Attachment 4)
However, as DEPM studies are likely to come under greater scrutiny (and from individuals with a limited fisheries background), we suggest the following:
- Citations for standard statistical methods should be to statistical papers (rather than other DEPM papers) and names of corresponding R functions to be stated in the text (where applicable).
- The analysis method and data should be publicly and readily available.
- While seminal papers in DEPM (e.g. Lo et al. 1996) are frequently cited, recent papers that are more closely aligned with the methods in the study should be cited.
- Recognising the inherent variability of the input data, multiple statistical methods should be investigated as part of sensitivity analysis when reporting DEPM studies.
Cubillos, L.A., Ruiz, P., Claramunt, G., Gacitua, S., Nunez, S., Castro, L.R., Riquelme, K., Alarcon, C., Oyarzun, C. and Sepulveda, A. (2007). Spawning, daily egg production, and spawning stock biomass estimation for common sardine (Strangomera bentincki) and anchovy (Engraulis ringens) off central southern Chile in 2002. Fisheries Research 86: 228-240.
Lo, N.C.H., Macewicz B.J. and Griffith D.A. (2005). Spawning biomass of Pacific sardine (Sardinops sargax), from 1994-2004 off California. CalCOFI Report 46, 93-112.
Lo, N.C.H., Ruiz, Y.A.G., Cervantes, M.J., Moser, H.G. and Lynn, R.J. (1996). Egg production and spawning biomass of Pacific sardine (Sardinops sagax) in 1994, determined by the daily egg production method. CalCOFI Report 37, 160-174.
Neira, F.J. (2011) Application of daily egg production to estimate biomass of jack mackerel, Trachurus declivis – a key fish species in the pelagic ecosystem of south-eastern Australia. IMAS Report, 42p.
Neira, F.J. and Lyle, J.M. (2011). DEPM-based spawning biomass of Emmelichthys nitidus (Emmelichthyidae) to underpin a developing mid-water trawl fishery in south-eastern Australia. Fisheries Research 110, 236-243.
Picquelle, S. and Stauffer, G. (1985). Parameter estimation for an egg production method of northern anchovy biomass assessment. In R. Lasker (Editor), An egg production method for estimating spawning biomass of pelagic fish: application to the northern anchovy, Engraulis mordax, pp: 7-15. NOAA Technical Report NMFS 36.
Ward,T.M, Burch,P., McLeay, L.J., and Ivey, A.R. (2011): Use of the Daily Egg Production Method for Stock Assessment of Sardine, Sardinops sagax; Lessons Learned over a Decade of Application off Southern Australia, Reviews in Fisheries Science, 19, 1-20.
Wadsley, A. Super Trawler: The UTAS Vice-Chancellor must investigate.
Tasmania Times 2012-08-26. http://tasmaniantimes.com/index.php?/weblog/article/super-trawler-the-utas-vice-chancellor-must-investigate/
- Wadsley’s claim that IMAS science is wrong.
- Wadsley, A. (2012a). The Commonwealth Small Pelagic Fishery. Review of Estimates of Jack Mackerel Biomass.
- Wadsley, A. (2012b). The Commonwealth Small Pelagic Fishery. Review of Estimates of Jack Mackerel Biomass.
- Nancy Lo’s comments on ‘The commonwealth small pelagic fishery: review of estimates of jack Mackerel biomass by Dr. Andrew Wadsley’
- Changes in the Gillnet, Hook and Trap Sector of the Southern and Eastern Scalefish and Shark Fishery to Protect Dolphins
- Draft Shark Plan 2
- Changes in the Gillnet, Hook and Trap Sector of the Southern and Eastern Scalefish and Shark Fishery
- Eastern Tuna and Billfish Fishery Management arrangements booklet 2011
- Freedom of Information