Report warns HNZ IT faults could 'snowball' after layoffs

Despite multiple assessments Mrs A's cancer went undiagnosed for months. Photo: Getty Images
The Health NZ register rates 14 out of the 40 risks as "almost certain" to happen, with "severe" consequences. File photo: Getty Images
Hospitals face a greater risk of many more cyber attacks because of the mass layoff of IT staff, according to a Health NZ Te Whatu Ora document.

A leaked risk register shows there is the potential for massive one-off impacts, as well as everyday erosion of safety, in the northern region covering all of Auckland's hospitals, due to the the proposed culling of more than 1100 data and digital jobs.

The risk register, compiled from staff feedback, gives multiple examples about the impact on the northern region's services:

"With fewer D&D (data and digital) staff to troubleshoot, maintain, and restore essential hospital systems... outages would last longer, severely impacting patient care, administrative functions, and operational efficiency."

The register rates 14 out of the 40 risks as "almost certain" to happen, with "severe" consequences. Another seven are almost as bad.

Faults could "snowball", and patches and protections could fail on the region's old IT systems.

The register said doctors and nurses could be forced to leave front-line duties to plug day-to-day gaps, or may struggle to master IT upgrades without enough helpdesk support.

"There is a significant number of these calls are from front-line, patient-facing services when a device is stopping the ability of a clinician to deliver patient care."

It predicted major failures - such as occurred in May and June this year - might drag out for many more days.

It revealed that a network failure in August was directly attributed to an old device that disrupted medical wards and the renal department at Whangārei Hospital for 15 minutes.

This would get worse, the register said.

"Reducing the number of data and digital network and communications personnel in the northern region will significantly impact the network infrastructure that underpins all clinical services. This reduction will worsen existing technical debt and associated incidents."

The PSA union said the register was prepared inside Health NZ Te Whatu Ora to test the risks ahead of its proposal to cut IT staff by 47 percent.

"This is cold, hard evidence that the government is prepared to risk the lives of patients to save money," PSA national health lead Ashok Shankar said.

The register - obtained by 1 News - used red ink to denote "extreme" risks, and was full of red.

Similar risks could be expected across all other 19 health districts, which also have old systems.

The risk revelations come after Health Commissioner Lester Levy and chief executive Margie Apa designed a rolling series of proposed job culls to to try to reverse operating losses of more than $140 million a month.

On cyber warfare, the register asked: "'With the new normal, what will be the chance of a Waikato 2021-esque cyber blackout in the next 12 months?", and answered: "Risk of more frequent and major cyber events, system-wide outages. Breach of privacy, loss of fidelity of digital systems and clinical records."

Waikato patient data was put up for sale on the dark web after the ransomware attack crippled hospital services for weeks.

Health NZ last week told RNZ: "Investment in cybersecurity has continued and is not included in the activities stopped or deferred".

Incidents of concern

The register detailed three recent alarming incidents that might have been worse with fewer data and digital staff:

• In June, after a Transpower pylon fell over, it took 50 HNZ digital staff one day and six hours to find power workarounds for all of Northland's clinical sites. With fewer people, that might have taken almost three days, the register said.

• In May, a code fault rendered almost 600 workstations unusable across the region. It lasted almost seven days, but "fewer resources available would push resolution out by at least another week".

• Also in May, a power surge took out more than 20 clinical apps for more than six hours, involving more than 50 digital staff in multiple teams to fix. The mop-up would take days with fewer of them and cuts could see "prolonged impact to clinical and patient care, increasing impact to health outcomes".

The register listed ways to reduce the risks - or 'mitigations' - often citing 'business continuity plans' that depend on other types of IT.

But many of the mitigations had major limitations. For instance, it said while old hardware could be replaced, but the scale of this was huge, with three-quarters of the region's hardware being too old in some cases. Capital spending on data and digital projects has already been cut by hundreds of millions of dollars.

"The projected decline of network infrastructure offers no mitigation options that would adequately safeguard clinical services," the register said.

"Trust will erode in D&D's ability to support front line clinical delivery and may impact patient experience," it warned.

"The individual impact and fallout from each incident will add up, creating longer backlogs where previous issues are still unresolved.

"There will be limited or no staff that have the required deep understanding of systems, how they work and interact."

As for cyclones and emergencies like Covid-19, there might be "significantly reduced" IT backup for emergency health workers.

"Emergency situations will not have timely access to D&D resources to help with planning and execution of plans to a response, resulting in impacts to clinical staff burnout and patient care.

"Significantly reduced ability to rapidly stand up digital communications, devices and infrastructure to support emergency command centres, mobile response and field capabilities."

'A bald-faced lie'

The PSA said the register showed patients would die if the government pressed ahead.

"The government has told New Zealanders continually and loudly that its cuts will not impact the frontline of hospital services," Shankar said in a statement.

"These documents expose that for what it is - a bald-faced lie."

Health Minister Dr Shane Reti said Health NZ was managing its consultation process with staff.

"The minister has been constantly reassured that the reset is being undertaken in a careful and considered way," he said in a statement.

Health NZ Te Whatu Ora chief executive Margie Apa said the register was a "collection of perceptions and perspectives taken from workshops between September and November with Data and Digital staff and clinical leaders as we sought to understand potential risks if proposals were to proceed".

She said the organisation would prioritise continuity and timely access to critical clinical information systems as part of its decision making.

"Our teams have very good expertise that we value to inform decisions which is why giving feedback through consultation is important.

"The document does not reflect the current reality or what will happen as a result of proposed changes. This is because decisions have not been made.

"The consultation is about getting the feedback from our staff to help inform our decisions."

Everyday problems

Everyday health functions were also at severe risk from the cuts, the register found.

Clinicians would take longer to learn to use upgrades, and if there were fewer helpdesk staff, they would be forced to "work without the IT access to the clinical systems they need to perform their role efficiently. I expect this to impact patient care", said the register.

It listed risks of "suboptimal digital systems, further degradation of outdated/unfit-for-purpose IT tools leading to reduction in care delivery, quality and safety, efficiency and effectiveness".

"Rural and remote hospital services will disproportionately be at greater risk of not having timely resolution to issues compared with the metro sites."

The risks are not all in the future, and are tied to IT projects that have already had the plug pulled.

RNZ reported last week that 136 IT projects were being stopped or deferred.

One of those stopped was a project to improve non-clinical task management at Te Toka Tumai Auckland City, Counties Manukau and Waitematā hospitals.

"Districts to remain on incumbent systems, noting imminent risk of catastrophic failure Te Toke Tumai," said the register.

It foresaw patients "backing up in ED, not getting to theatre, not receiving pharmaceuticals or having access to treatment, required in a timely manner".

Health NZ has huge numbers of old and vulnerable servers and apps, and patching to protect them might become another casualty.

"Reduced patching capabilities will increase the vulnerability portfolio and increase security risk to dangerous levels, exposing HNZ to potential cyber attacks and data loss.

"Worst case scenario all patching activities would be stopped, as the risk to continue without the correct staffing numbers would be too high and risk of business/clinical impact would be increased."