Pache.co Information Technologies

Covid-19 modeller Shaun Hendy formally demanded a press release retracting Productivity Commission work that he disagreed with last year. And Hendy’s blasts on Twitter appear to have left the commission unwilling to engage with his
academic critique and fearful of an escalating public spat with political implications so close to an election.

Ultimately, the commission rejected Hendy’s demands and quietly stood by a paper that put the cost of an extra five days of level 4 lockdown in April, 2020 (the Anzac weekend extension) at a net $741 million. The commission defended the work as an exploration of how to go about weighing the costs and benefits of specific elements of the response to Covid-19, within the broader context of an elimination policy. Though Hendy and his colleagues contest its method and findings, it remains published on the independent Crown entity’s website.

Hendy, a professor of physics, was then head of Te Pūnaha Matatini (TPM), the Auckland University-based research centre which the Government has relied on for most of its modelling advice through the pandemic, and which has received, or is contracted to receive, some $6 million for the work. That is the vast majority of the Government’s spending for Covid response modelling work and related analysis.

Hendy has been widely fêted for his contribution during the pandemic, but his run-in with the Productivity Commission suggests his engagement with those he disagrees with, especially on Twitter, has sometimes been unproductive.

Hendy defended the episode as broadly necessary in an environment where public policy was malleable, and in which he feared that the commission’s work (incorporating, he says, a critical misunderstanding of disease dynamics) could be used to undermine New Zealand’s Covid-19 elimination policy. In speaking to the Herald last week, he did accept that his tweeting at the time may have been counterproductive in some ways: “if it did put them off, then I would have to concede that maybe that wasn’t the best way to engage … so yes, in terms of our ability to work with them it probably wasn’t the best thing. On the other hand, once things do become public domain, you do need to react quickly to them.”

For its part, the commission recoiled at what it viewed as Hendy’s aggressive, public demands. Internal emails at the time, released under the Official Information Act (OIA), show considerable consternation within the group over what were characterised as Hendy’s “attacks” and highlighting specific questions economists there had about whether his objections were as conclusive as he claimed.

Hendy and his TPM colleagues used an article on the Spinoff website to outline their academic objections. Ultimately, though, it was a claim by Hendy – on Twitter and in a Newshub article – that appears to have set the tone for the two parties’ terse and ultimately unproductive dealings: Hendy said his team had reached out to the commission when it was writing the cost-benefit piece, only to be rebuffed.

‘Too busy to talk to us’

On August 7, both the Herald and Newshub carried stories of the commission’s cost-benefit analysis, which had been supplied to the Treasury and which was released that month by the department under the OIA.

In both the Newshub story and on Twitter, Hendy contended that the commission’s work contained a critical misunderstanding (he was not interviewed for the Herald piece, which was written as an opinion piece by this journalist). He also maintained that his group had approached the commission at the time of the cost-benefit work, only to be turned away.

In a chain of tweets he wrote: “We actually approached @NZprodcom after some blogging they did around the time they were working on this paper, but we were told they were too busy to talk to us.”

We actually approached @NZprocom after some blogging they did around the time they were working on this paper, but we were told they were too busy to talk to us. 6/N

— Shaun Hendy (@hendysh)

That allegation particularly piqued the commission’s chairman Murray Sherwin. “Did they reach out to us only to be turned away?” Sherwin wrote to his staff under the email subject line: “Shaun bites”. Later he wrote: “I’m deeply stung by the line that we refused to engage: it simply isn’t true and runs counter to our culture of positive engagement…”

Emails released by the commission show that while Hendy himself hadn’t been in touch, his TPM colleague, modeller Rachelle Binny, had been.

On May 1, just a few days after New Zealand moved from level 4 lockdown to level 3, Binny made contact through the commission’s general mailbox. Her note was passed to Dave Heatley, then a principal adviser at the commission, who replied that afternoon.

“I’ve been following your work. I’m tied up [this] afternoon completing a cost-benefit analysis of the 5-day extension of level 4. Once that’s out of the way I’m keen to chat. My contact details are below,” Heatley wrote.

Binny replied, noting there was no rush. She wrote a short note comparing her own and Heatley’s work calculating reff values [disease reproduction values] for a range of countries including New Zealand. They appeared to be roughly in agreement.

“Looking forward to touching base whenever suits,” she wrote, and ended: “Hope the cost-benefit analysis goes well.”

It appears that neither party made any further contact.

Heatley left the commission this year and now works as an independent economist. Speaking to the Herald last month, he said: “[In August, 2020] Hendy completely misrepresented a short email exchange I had with Rachelle Binny, one of his co-authors. Fearing further misrepresentation, I decided against further direct communication with that group.”

Hendy’s letter

Hendy’s formal letter to the commission landed on August 10. In it he reiterated his critique, that the commission’s work contained a misunderstanding related to the usefulness of level 4, and therefore of its benefits.

He wrote: “Given the significance of this misunderstanding and its potential consequences in terms of harm to health and the economy, I ask that the commission:

• Publish a statement on its website retracting the paper, supported by a press release outlining its misunderstanding;

• Investigate why peer review from domain experts in disease dynamics was not sought prior to the completion of the report; and

• Set clear expectations that in future appropriate domain expertise will be utilised when producing such advice.”

The commission’s misunderstanding, Hendy claimed, was essentially this:

“When Covid-19 numbers are low, as they were in late April in New Zealand, disease spread is highly stochastic, meaning that there is a probability an outbreak will fade whether or not the effective reproduction number, Re, is less than one. The commission’s report fails to understand this key point and therefore fails to include the benefits of elimination that might flow from a level 4 extension in reducing case numbers …”

Unfortunately the report by the @NZprocom referred to in this story seems to have been based on a misunderstanding of disease dynamics. I am glad their advice wasn’t followed. 1/N https://t.co/mKlGweERYD

— Shaun Hendy (@hendysh)

In the Spinoff article, published the same day, Hendy and his co-authors claimed that “… at the time our model showed that an additional five days at a higher alert level could increase the probability of elimination by up to 30 percentage points.”

Emails of the time show Heatley and others within the commission, including commissioner Andrew Sweet, were unable to pinpoint the exact basis for Hendy’s criticism.

Sweet, who acted as one of the internal peer reviewers for Heatley’s paper, went looking for the “up to 30 percentage points” figure in TPM’s work of the time. One of the parameters of the cost-benefit analysis was that it was based only on the data and literature available at the time of the decision to remain in level 4 over the Anzac Day weekend.

Sweet appeared to come up empty-handed: “Now that I’ve got the draft rebuttal out, I’m pulling out the papers they [TPM] publicly released in April or earlier. There’s nothing so far about that 30 per cent number.”

Hendy confirmed in a recent interview: “There wasn’t anything publicly available on this [30 per cent figure] at the time, so they would have needed to reach out to us to find that out. This is why it would have been useful for them to seek peer review from disease modellers who may have spotted their mistake.”

In another email sent between colleagues at the commission, Heatley wrote: “his [Hendy’s] critique ignores another, also relevant, tail risk. He talks of ‘locking in’ elimination – yet, as current circumstances in Victoria and Auckland … clearly show, such analysis should also account for the possibility that elimination is short-lived – and thus its benefits also need to be probabilistically discounted. To not do so would be an equally grave error. It’s a great pity that these issues can’t be explored collaboratively.”

Speaking last month, Heatley said: “No one has perfect knowledge or perfect models. Knowledge advances through a mixture of inquiry, publication, challenge and respect for others’ viewpoints and contributions. We should expect academics to embody these values. Hendy’s attempt to suppress research whose conclusions he didn’t agree with is, in my view, behavior unbecoming a professor.”

Drafting the commission’s reply

Ultimately, the commission decided against engaging with Hendy on academic particulars, specifically whether the model it used was fatally flawed.

“Looks well worded – and importantly focuses on the purposes of Dave’s work, rather than a tit-for-tat discussion about each assumption underlying it,” commissioner Gail Pacheco wrote in an email to others at the commission relaying her thoughts on the draft letter Sherwin was preparing in reply to Hendy. Pacheco and Sherwin had also peer-reviewed Heatley’s work; in addition, it was peer-reviewed by two external economists.

“Two aspects I wasn’t sure of are: the paragraph about the toy model. My sense by that stage of the letter, is that some of the key points have been made; and this para may antagonize him [Hendy] – and start a new debate about complex versus simple models …

“…that is potentially the sort of thing he could latch on [to] and start a new Twitter thread on – twisting the words out of context to say @prodcom thinks complex models are too difficult, or something like that – even though that is not your inference at all…”

The “toy model” reference (a description of the commission’s model that Hendy had used in the Newshub story) was ultimately dropped.

Heatley appeared to agree with Pacheco. He replied: “as Gail says, Hendy has likely made up his mind. So the more of his points we counter the more he will dig in. So the best we can hope for is some form of detente – where he sees no additional benefit (or adverse consequences) from further attacks …”

Murray Sherwin’s reply

Sherwin began his letter to Hendy: “I have been surprised, and disappointed, by both the content and the tone of your comments.

“We stand by the paper particularly given its purpose and the context of its production. As noted in the paper, it was produced after the decision to extend the level 4 lockdown by 5 days. It was never intended as advice for that decision nor as a critique. We had no intention to publish the paper.

“Rather, the paper was intended as a desktop exploration of the relative costs and benefits of different Covid-19 responses with the express purpose of exploring the data requirements, the nature of judgments required and the sensitivity of the results to various parameters. It serves that purpose for the commission.”

Sherwin also noted the cordial email exchange between Binny and Heatley that he’d reviewed.

“Careful and skilled calibration of interventions as we negotiate between the Covid-19 health risks and the employment/business/economic and other health-related risks will yield long lasting dividends,” he wrote.

“Given the inevitable heightened sensitivity around these issues in the runup to the election and even more so with the current return to levels 3 and 2, it is not our intention to respond publicly to these matters,” Sherwin closed.

At the time, an election was slated for September 19. It was ultimately delayed to October 17 because a resurgence of Covid-19 in Auckland prompted a level 3 lockdown in the city, and level 2 measures elsewhere. Public servants are under particularly stiff standards of political neutrality in the three-month run-up to an election.

Hendy tweets again

Sherwin delivered the letter of reply to Hendy on August 14. The same day, Hendy tweeted an excerpt of it, followed by his objection to it on Twitter.

Sigh. @NZprocom are not going to correct the misunderstanding in their paper, saying that it was just a desktop exercise for their purposes and they didn’t intend to publish it. pic.twitter.com/Ybz3mRsXiV

— Shaun Hendy (@hendysh)

After nearly a decade at its helm, Sherwin retired from the commission at the end of 2020. Sweet spoke to the Herald on the commission’s behalf last month:

“Shaun tweeted selected sentences from the letter Murray Sherwin, our previous chair, sent to him. However, he never formally replied to [the letter]. And as noted, we felt that engaging indirectly through media such as Twitter was unlikely to be constructive … given the potential sensitivity of the issue, we wanted to avoid indirect engagement with other parties through public forums such as Twitter. We felt that such indirect engagement would run the risk of an unconstructive dialogue and result in the substance of our arguments being lost … We were ultimately unsuccessful in engaging with TPM in a direct, constructive dialogue.”

The response to Covid-19 has since moved on. Level 3 was shown to be effective at keeping the reproduction of the virus low enough to achieve elimination until the arrival of the Delta variant, and the Government did not resort to the heavy stringency of level 4 again until Delta arrived in Auckland in August, 2021. Te Pūnaha Matatini produced its own cost-benefit analysis concluding that level 4 was probably cheaper than achieving elimination by more relaxed measures, because such measures are required for a longer period. In turn, it was critiqued by the Wellington-based economist Martin Lally.

Apart from the Heatley analysis and the single piece of TPM work (which has not been formally peer-reviewed), cost-benefit analysis has so far appeared surprisingly absent from the Government’s Covid response decision-making, even now that the more complicated circumstances of a broad elimination policy have been swept away.