Reflections on the Skegg report pt.2

[Wandering Eyes]

Dr Martin Lally

Director, Capital Financial Consultants Ltd

lallym@xtra.co.nz

In an earlier comment on the Skegg Report, I noted that the authors did not provide any empirical analysis in support of their conclusion that the elimination strategy should continue to be pursued in New Zealand.  However, in para 6, they did refer to a published paper that concluded that an elimination rather than a mitigation strategy has to date yielded the best outcomes for health (lower covid death rate), the economy (lower GDP losses) and civil liberties (lower average government restrictions). https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(21)00978-8/fulltext?

This paper compares the average death rates, GDP losses and restrictions on movement in the five OECD countries that consistently aimed for elimination with the rest that did not.  Everything is claimed to be better in the first group: a lower average death rate, smaller average GDP losses, and lower average restrictions on civil liberties.  Elimination is defined as “Maximum action to control SARS-CoV-2 and stop community transmission as quickly as possible.”  The five OECD countries claimed by them to have done so are Australia, Iceland, Japan, New Zealand, and South Korea.

This seems like the Holy Grail; if true, there would be no need for trading off liberty and GDP losses for lower covid deaths, and therefore no need for a cost-benefit analysis.  However, as usual, if it seems too good to be true, it isn’t.  The first problem is that the five countries that supposedly undertook the “maximum action to control SARS-CoV-2 and stop community transmission as quickly as possible” include Iceland, Japan and South Korea.  At the very least, maximum action to control covid as quickly as possible would involve border closures and lockdowns.  However, none of these three locked down and Iceland additionally did not even close its borders.

I conveyed this concern to the lead author of the paper (Professor Oliu-Barton), and he replied that their classification of countries relied in part on the ratio of the Stringency Index to the covid deaths during the period when the covid deaths were very low, with the five countries in question having high values for this ratio. This seemed rather subjective so I asked for further details to see if I could reproduce their result (which is fundamental to the credibility of scientific research).  I have yet to receive a reply to that.  Nevertheless, the following seems clear.

1. The authors are not measuring the extent to which countries took the “maximum action to control covid” but are instead measuring how quicklygovernments reacted.  So, the conclusion of the paper (that elimination is superior to mitigation on all dimensions) is not supported by their analysis.  Instead, their analysis supports the conclusion (at most) that acting quickly produces the best outcomes on all dimensions.  Even this conclusion may be too strong because:
2. The analysis in their paper considers only one possible variable that could explain deaths, economic losses and loss of liberties: how fast a government acts. They identify the five fastest movers in the OECD and find that these countries had on average much lower death rates than the other OECD countries (and other benefits), and then attribute this to them moving quickly.  However, had they instead conjectured (very reasonably) that being an island mattered, and identified the island nations amongst their OECD set, they would have found that these island nations were exactly the same five countries that they identify as the fastest movers (South Korea is effectively an island too), and then found their average death rate was much less than the other OECD countries, and would then presumably have attributed this to them being islands.  So, their paper would then have been entitled “Being an Island Creates Best Outcomes for Health, the Economy and Civil Liberties.”  Neither of these two approaches would be satisfactory.  Since death rates may be driven by many variables, a multivariate analysis is essential.  In my own analysis I used multiple regression, and found the following variables to be statistically significant in explaining death rates for countries: population density (low is good), population (low is good), whether it is an island (good), and the date of its first covid death (later is better, to provide more time for preparation).  I also found that the last variable was very closely correlated with how quickly a government acted, measured by the time interval between a country reaching 54% on the Oxford Stringency Index for government restrictions and the date of its first death (high values are best). See pp. 4-8 of:

https://www.medrxiv.org/content/10.1101/2021.07.15.21260606v1

3. Conclusions from statistical analysis require tests for statistical significance.  The authors do not perform any such tests.  This is particularly unsatisfactory in respect of their graph of GDP outcomes for their two sets of countries, which are very similar.  Had they conducted statistical tests, they might have found that economic outcomes were statistically indistinguishable between the two groups, and therefore avoided making the claim that a particular type of government policy produced the “best outcomes for the economy”.
4. The analysis in their paper uses classification data on government policy (countries are classified as fast movers or not) rather than numerical data.  The latter is more powerful if it can be done, because it avoids the somewhat arbitrary dividing line between the two groups and uses inter-group variation as well as variation between the groups. Furthermore, it could be done in this case.  For example, in my analysis, I quantified the speed of government actions by the time interval from reaching 54% on the Oxford Stringency Index until the first covid death.  It may be that the authors used classification data because their assessment of government policy was subjective.  If so, then there is the further problem that they may have been subconsciously biased towards judging these five countries to be the fastest movers because they already knew that they had the lowest death rates in the OECD data set. Such subconscious bias risks turning their analysis into advocacy rather than scientific analysis.  In addition, using classification data requires them to choose the dividing line between the two groups of countries, and this too exposes them to subconscious bias in choosing to include only five countries in the elimination group.
5. Any analysis on whether government policy has favourable effects on covid death rates is exposed to the problem of reverse causality, i.e., government decisions may have been driven by observation of the death rate as well as affecting the death rate. This should have been tested for. The authors do not conduct any such tests.  By contrast, I conduct such tests in the Appendix to my paper.
6. Drawing conclusions about which government actions produce the best outcomes on the basis of a cross-country analysis, as the authors do, can at best only offer conclusions that are valid in general, i.e., they might be true for 60% of countries but not for the other 40%.  Since policy is made by individual countries, the conclusions would then be worthless to individual governments.  The better approach is to conduct a cost-benefit analysis for each individual country, as I have done in my paper (and in a parallel analysis for Australia).

In summary, the article is not measuring the thing it claims to be measuring, and the analysis fails to consider more than one explanatory variable, and it presents no tests of statistical significance, and it uses classification data rather than numerical data, and it conducts no tests of reverse causality, and its results have no value for an individual government.

The paper should not then have been relied upon by the Skegg Committee.  In fact, it is hard to believe that the members of that Committee even read the paper; had they done so, they would surely have noticed that three of the five countries claimed to be following an elimination strategy did not even lock down.  Had they noticed that, and then contacted the authors of the paper for an explanation, and received the explanation I did, it would then have been apparent that the article was not in fact assessing the merits of elimination but the merits of moving quickly.  Moving quickly is good, as the article finds (and I do too) but it does not imply that elimination is better than (say) not locking down and taking other mitigation measures to minimise deaths.  If your house is burning, it may not be clear what action you should take to fight the fire or mitigate the damage but it is obvious that any action you take should be taken quickly.  Likewise the sun rises in the east.

The Skegg Report contains one other piece of empirical evidence on outcomes to date.  In para 8, the report notes that the benefits of New Zealand’s approach can be illustrated by comparing it with Scotland, with much the same population but which experienced 10,000 deaths compared to our 26.  Cherry picking one country is advocacy, not scientific analysis.  Furthermore, if one is going to cherry pick Scotland, one would have to ask how Scotland would have fared had it followed exactly the same policy as New Zealand.  Unlike New Zealand, Scotland has a land border with a place (England) that followed a much less stringent approach than us, and England in turn is separated from the European continent by only 20 miles, with a tunnel connecting them.  With these natural disadvantages, it is unlikely that Scotland would have experienced 26 deaths had it followed exactly the same policy as New Zealand.  What then is the point of citing Scotland’s deaths, other than to suggest (wrongly) that our very low death rate was due entirely to policy and not also to geography?

 

View the full article