It was some of the strongest language by a senior US official to describe the strained ties between Washington and Islamabad.
"It is difficult to achieve peace in Afghanistan as long as there is safe haven for terrorists in Pakistan," said Panetta.
"It is very important for Pakistan to take steps. It is an increasing concern, the issue of safe haven, and we are reaching the limits of our patience."
He was speaking in the Afghan capital Kabul where he held talks with military leaders amid rising violence in the war against the Taliban.
Secretary of State Hillary Clinton also implicitly defended Washington's use of drone strikes against suspected militants, just days after one of them killed Abu Yahya al-Libi, al Qaeda's second-ranking leader, in northwest Pakistan.
"We will always maintain our right to use force against groups such as al Qaeda that have attacked us and still threaten us with imminent attack," Clinton said in Istanbul at a meeting of the Global Counterterrorism Forum, a US-and Turkish-chaired group.
Pakistan has termed the attacks as illegal and a violation of its sovereignty. The United States has long pushed Islamabad to do more to help in the war against militancy.
Panetta urged Pakistan to go after the Haqqani militant network, one of the United States' most feared enemies in Afghanistan, and said Washington would exert diplomatic pressure and take any other steps needed to protect its forces.
"It is an increasing concern that safe havens exist and those like the Haqqanis make use of that to attack our forces," he said.
"We are reaching the limits of our patience for that reason. It is extremely important for Pakistan to take action to prevent (giving) the Haqqanis safe havens, and for terrorists to use their country as a safety net to conduct attacks on our forces."
Panetta blamed the group for an attack last week on a US base in the east in which several insurgents, including some wearing suicide vests, used rocket-propelled grenades.
The attack was foiled, but it underlined the challenge facing Western and Afghan forces in the east where insurgents take advantage of the steep, forested terrain and the Pakistani border to launch attacks and then slip back, commanders say.
"What happened the other day in Salerno is an indication that they are going to continue to come at us and, let me be clear, anybody who attacks US soldiers is our enemy and we are going to take them on. We have got to be able to defend ourselves," Panetta told US troops earlier at Kabul airport.
Washington appears to be looking to other allies in the region for help in the face of Pakistan's foot-dragging. Panetta arrived in Kabul after a visit to India, Pakistan's old enemy, where he urged New Delhi to take a more active role in Afghanistan.
NATO has signed an agreement with three countries to the north of Afghanistan for land routes as the US-led alliance begins a withdrawal of its forces from the country next year.
NATO chief Anders Fogh Rasmussen said earlier this week the "reverse transit" deal was signed with Uzbekistan, Kazakhstan and Kyrgyzstan.
Pakistan closed the shorter and cheaper routes through its territory last year to protest a cross-border NATO air attack that killed 24 Pakistani soldiers. Discussions to reopen the Pakistan routes have stalled.
Resupplying troops in Afghanistan through the Northern Distribution Network is about two and a half times more expensive than shipping items through Pakistan, a US defence official told Reuters, speaking on condition of anonymity.
The United States wants Pakistan to launch a full-scale offensive in the North Waziristan border region to go after the Haqqani group, which is close to the Afghan Taliban and al Qaeda.
"We have made that clear time and time again, and will continue to make clear, that it is an intolerable situation to have those people attacking our people, our forces and to have the convenience of being able to return to safe havens in Pakistan," Panetta said.
Pakistan has strong traditional links with the Afghan Taliban and other militant groups. Islamabad denies that it uses them as proxies to gain leverage in Afghanistan ahead of any settlement to the war, or in case a civil war breaks out after most foreign combat troops leave in 2014.
General Martin Dempsey, the chairman of the US military's Joint Chiefs of Staff, suggested the withdrawal timeline added urgency to the need to tackle the Haqqani threat.
"We've got to get (eastern Afghanistan) and the Haqqani influence reduced in order to meet our timelines for the transition that we're moving toward, and at the end of '14," Dempsey told reporters at the Pentagon.
Anti-American feelings runs deep in Pakistan over issues like drones, and with a general election expected in early 2013, no politician will want to be seen as soft on the Americans.
US President Barack Obama will have to look tough against militancy during in this presidential election year in the United States.
