It was, he explained, a hand-to-mouth upbringing in which the family lived on bread and dripping and one main meal a day.
Mr Hubbard told this reporter, his humiliation still vivid, of the character-defining day when the milkman came to deliver two pints of milk, for which his mother had no money to pay.
He left having given one pint and saying he would put the cost "on the slate".
"I vowed then I would never be poor," Mr Hubbard recalled.
It was a promise he was to keep.
At one stage, he was considered the South Island's richest man, with a fortune estimated by the National Business Review in 2008 to be $650 million.
All that has been wiped out with the receivership of South Canterbury Finance.
Those close to Mr Hubbard and his wife, Margaret, known as Jean, say yesterday's events would have shaken the couple, who built their reputation and fortune on helping others, and keeping to their word.
Uncharacteristically for a man who intentionally flew under the radar, he yesterday delivered a broadside at SCF directors and the Government over the receivership.
"It has been deeply frustrating and hurtful, over the last nine months, to have been sidelined by my fellow SCF directors, and subsequently straitjacketed by the Government regulators, from working to save South Canterbury [Finance]," he told The New Zealand Herald.
He and his wife had poured an estimated $400 million of their own cash and equity into the ailing finance company, and he said yesterday it was a body blow when the Government took control of SCF in June.
"Surely they realised that by freezing me out and taking over control of my affairs that they would be dealing a body blow to South Canterbury Finance."
He said yesterday was a big day for the regulators and a sad day for investors.
"Instead, they bring down the boom, take me out, freeze my access to my personal funds, and now so many families, small businesses, farms and enterprises ... are going to be seriously suffering," he said.
Mr Hubbard said some people might consider his management style old-fashioned, but he had never defrauded a single investor of a single cent.
Mr Hubbard grew up as one of five children in a three-roomed home in Forth St, Dunedin, in the 1930s.
The Depression had forced his plumber father to find work planting pine trees at Waipori, leaving his wife at home to bring up the children.
He recalled a childhood playing on Signal Hill, and carving his name on a monument.
Last May, Mr Hubbard told the New Zealand Listener he thought about becoming a lawyer, but that meant studying Latin and attending Otago Boys' High School, a step his father refused as it was a "snobs' school and we're working class".
He finally relented and allowed his son to attend King Edward Technical College, but it was Sunday school and Bible class that shaped his Christian ideals and further influenced his life.
On leaving school, he worked on a Taieri dairy farm and then as a clerk with Trustees Executors, where he was encouraged to sit school certificate and university entrance before studying accountancy part time at the University of Otago and then the University of Canterbury after transferring north with the company.
In 1953, he and Jean moved to Timaru, where he worked part time as an accountant at Craighead Diocesan School before establishing the Hubbard and Churcher accountancy practice.
It became central to a business empire that included SCF, Helicopters New Zealand, Scales Corporation, which was involved in shipping and horticulture, and extensive farming investments, including a 33% stake in Dairy Holdings.
Mr Hubbard estimated he had given $200 million to charities and causes during his life.
Five years ago, he was diagnosed with kidney failure and, initially, had to travel to Christchurch several days a week for dialysis.
The wife of a plasterer saved from bankruptcy by Mr Hubbard several years earlier offered him one of her kidneys.
When he refused the offer she spent a month learning how to administer dialysis and now oversees his treatment three times a week.
In the 1980s, when farming went through tough times, he rescued many third and fourth-generation farming families with no-interest loans, and he has helped hundreds of young farmers and aspiring businessmen to get a start.
Allan Hubbard was a "tycoon" with a style all of his own.
That evening two years ago in his Timaru offices when he met Mr Key, he was humbly attired in a hand-knitted jersey, corduroy trousers and socks and sandals.
The chief executive of South Canterbury Finance at the time, Lachie McLeod, put his chairman's casual appearance down to "Allan being Allan".
It wasn't a sign of disrespect.
It was in much the same vein as Mr Hubbard driving a VW Beetle, bought secondhand 36 years ago.
