Dr Sam was speaking ahead of the Otago School of Physical Education's symposium, The Future of Sport in Small Nations, where researchers, policy-makers, administrators and sports journalists will examine the challenges countries with small populations face in the world of international sport.
If New Zealand wanted to continue to punch above its weight as it did at the London Olympics - where it came fourth in the overall medal count relative to population - it would need to invest more money to keep talent here, Dr Sam said.
While the most obvious form of the "brawn drain" was the steady stream of rugby players leaving for Europe and Japan, New Zealand also faced a tough task keeping its coaching and high-performance staff.
This would only become more difficult as countries starting to develop their sport systems, such as India and Brazil, or "emerging middle powers" France, Italy and Korea, invested more in sport.
Such countries could, for example, decide to poach New Zealand's top rowing or Sevens coaching staff.
"If the country and the politicians decide that is a priority, then certainly there will have to be ever-increasing amounts of money put into high performance sports," he said.
"There are only so many medals to go around." Asked how New Zealand had continued to stay competitive, Dr Sam said Sport New Zealand would say "narrowed investment" and "cherry-picking" top sports and athletes had kept New Zealand ahead of the pack.
"To some extent that has worked, but it is hard to differentiate between that and the fact that the amount of money into high performance sport has quadrupled in the last 10 years."
Government funding into sport had grown from $17 million in 2002 to $76 million in 2011-12, he said.
The symposium will be held from November 21-23.
