A study which would have rated all babies born in New Zealand this year for their risk of child abuse was damned in a scientific review by a University of Otago public health statistician.
Nevil Pierse, in his review which accompanied the Ministry of Social Development's application to a health and disability ethics committee last November, said the whole predictive risk modelling (PRM) study should be stopped.
He was one of two academics commissioned by the ministry to provide differing views to the committee about the merit and potential public benefit of the work.
Dr Pierse, who is the deputy director of the University of Otago's He Kainga Oranga/Housing and Health Research programme, said it would do nothing to address the underlying causes of child abuse and was largely an ''excuse for inaction''.
No decision had been made to proceed with the study at the time the ethics committee application was lodged, and the ministry wanted the application heard in secret.
It cited part of the Official Information Act dealing with maintaining constitutional conventions to protect the confidentiality of advice by ministers and officials.
The application was withdrawn shortly after it was lodged, on the direction of incoming minister Ms Tolley, although this did not become public until July this year.
It was revealed she had written ''Not on my watch! These are children, not lab rats'' on the proposal's briefing papers.
The prospective study would have assigned risk scores to newborn babies, developed from administrative data held by Ministry of Health (maternity data), Ministry of Social Development (benefit data) and Child Youth and Family.
Data on the children and their parents/caregivers and siblings would have been linked at weekly or fortnightly intervals over two years, the research protocol document shows.
A follow-up at two years would have checked how many considered at high risk had come to the attention of authorities.
The application said findings from the study would establish whether an ''operational'' use of PRM was feasible and would allow identification and technical difficulties to be resolved before any ''go live''.
The study would also provide ''laboratory'' evidence on whether PRM could improve ''triage decision-making'' for children who were the subject of reported concerns.
Dr Pierse said while preventing child abuse was highly important, the ability to ''marginally better predict which children will suffer child abuse is of little value''.
The problem was the rate of child abuse in New Zealand, the causes of which centred around poverty and its symptoms - marginalisation, violence, deprivation, mental health and drug abuse, he said.
''Research to solve child abuse without addressing the broader social context is of very marginal value and potentially distracting to key players, such as the Minister of Social Development.''
Study without prevention
Ms Tolley's concern was that the study would involve observing ''potentially vulnerable infants'' for the two years of the study as they aged and moved through health and family support services, without taking any preventive action.
The ethics committee application document shows researchers planned to identify district health boards with ''well-developed early identification systems'' willing to participate.
They would have compiled a list of infants born in 2015 whose parents or caregivers were identified as potential candidates for an offer of ''intensive preventive services''.
The overlap between the DHB lists and those identified through PRM would be compared at the end of two years to see if PRM could improve early identification.
Dr Pierse said PRM claimed to identify the groups most at risk of maltreatment, but this might not even be useful practice.
''It also potentially reduces the focus on helping those children at risk who are harder to identify.''
Realistically, more resources and more interventions were needed.
As it was likely much abuse and neglect went undetected, PRM would at best reaffirm existing knowledge or biases within the established Children Youth and Family service framework.
It might encourage less observation of those communities ''that have either the resources to oppose CYF [preventing the substantiation], the ability to hide abuse, or both''.
Quality of data questionable
He was also critical of the quality and accuracy of the data which would be used, describing it as sub-optimal.
This would lead to the ''classic'' garbage-in, garbage-out modelling problem.
It was questionable whether the model would do as well in identifying the risk of ''true maltreatment'' as an experienced social worker or paediatrician, given all the relevant information.
The accuracy of prediction for individual situations was low and the 1% of children at highest risk were likely not to suffer abuse in the absence of any intervention.
Even the narrowest interventions available to reduce child abuse would have significant benefits to the children's and parents' education, health and family functioning.
It made little sense to restrict such interventions to those at high risk of abuse.
Dr Pierse said given the research's lack of merit, the large amount of time and effort spent on the excellent science design of the observation study was hugely disappointing.
The research would do nothing to improve scientific knowledge, he said.
'Very compelling'
The other reviewer, professor of evidence-informed practice at the University of Melbourne School of Health Sciences, Department of Social Work, Aron Shlonsky, was supportive of the project and the research protocol, describing it as ''very compelling'' and a ''thoughtful and high quality piece of work''.
The era of ''Big Data'' meant predictive analytics was entering a new phase.
''Rather than pen and paper measures, we will soon be looking at prognostic dashboards that are generated from multiple databases containing millions of records, and this will be done in real time.
"The question is no longer whether we can use information to predict behaviour, but how we use this technology. My contention is that we cannot avoid this revolution.''
The best minds should be brought together to find ways to use data which could offer an ''unprecedented opportunity to improve people's lives without predetermining them''.
Using PRM could reduce individual bias in risk assessment, allowing the service provider to focus on high quality, contextual assessments and delivering effective services.
''In situations where there are scarce resources, such a predictive endeavour makes a great deal of sense.''
However, he said a risk tool was no substitute for high quality practice.
There was also a risk successfully targeting those considered high risk could result in fewer universal services, a ''grave unintended consequence''.
Prediction of risk must be followed by ''effective services to ameliorate risk or the entire endeavour are in vain'' and it was good to see this acknowledged in the research protocol, he said.
He suggested one longer term advantage to using PRM could be that it could assist in predicting what type of intervention would be best with specific individuals or families.
''If we can figure out our best bet of what to offer to whom and when, we can better take advantage of predictive analytics and eventually move from a culture of risk to a culture of opportunity.''
One of his concerns with PRM used to predict maltreatment was that maltreatment was difficult to define as a behaviour.
Each type of maltreatment could theoretically have a different set of causes.
''Should they all be lumped together? Testing each type of maltreatment separately and then combining types that respond similarly within the model is highly advised,'' Prof Shlonksy said.
Complicated setting
Dr Pierse also questioned whether PRM would perform well in the complicated setting of child abuse which involved complex correlated problems including violence, sexual abuse and neglect which might have differing risk factors and ''perhaps interventions''.
The application document acknowledged there had been mixed reactions from academic and practitioner communities about this use of PRM, which is untried.
Work on PRM since 2012 has included a feasibility study, an ethical review, an ethical review on Maori issues, peer reviews and a privacy impact assessment.
There was a risk that findings from the research could be misinterpreted by some audiences and feed existing social prejudices and stereotypes, and the need to mitigate the potential for such harm was recognised.
Other ethical issues noted in the study application were the possibility of doing harm through a breach of confidentiality, the use of data without individuals' consent, as well as the risk identified by Ms Tolley about identifying children considered at high risk without offering a routine response to the risk.
The future of the use of PRM to identify children considered at risk of abuse is not yet clear.
Since the stopping of the observational study, the Ministry of Social Development has been given approval to proceed with a test expected to involve 20 historic anonymised cases.
Participating intake social workers will be divided into two groups - one will have the PRM score and training on how to use it, the other will have case study information without the PRM.
The decisions made by all participants will be compared with a consensus view of what would be the best decision, developed by an expert panel.
The ministry considers this testing, which is expected to cost up to $100,000, should be completed early next year.
It will generate more information on whether, in practice, risks can be mitigated and the benefits outweigh the harms.
This would be crucial to deciding whether to fully test PRM.
Any subsequent testing ''in a real-life setting'' would require new discussion with the privacy commissioner ''and potentially a new privacy impact assessment'', the ministry said.
Last December, the ministry also advised Ms Tolley that if a decision was made to use PRM in the live setting in the CYF contact centre, it would be done without creating a list of ''potentially high need'' children.
Instead, the system would run on an ''ask and answer'' basis where information for any child would only be extracted and combined once the child had been the subject of a report of concern and a request for the information had been made by an intake social worker.
