Three Lebanese soldiers and an Israeli officer have been killed in the first serious border clash since Israel's 2006 conflict with Lebanon's Hezbollah.
A Lebanese journalist also died in the fighting. Lebanon says troops opened fire after Israeli troops entered its territory. Israel denied the charge.
PM Benjamin Netanyahu said Israel would respond "aggressively" to any attack.
Hezbollah leader Sheikh Hassan Nasrallah said militants would not "stand idle" if Israel attacked again.
In a speech transmitted via video link to thousands of supporters massed in Hezbollah's stronghold in Beirut's southern suburbs, Sheikh Nasrallah said his militants had been ready to fight in the border clash, but that he had ordered them to stand down to avoid an escalation.
However, he warned that future violence would be met with violence.
"From now on, if the army is attacked in any area where the resistance has a presence or a say, the resistance will not stand by idly. We will cut off the Israeli hand that reaches out to attack the Lebanese army," he said.
Sheikh Nasrallah's speech was a scheduled appearance marking the four-year anniversary of the end of the 2006 war between Hezbollah and Israel.
The US and the EU have urged restraint on all sides.
The Lebanese army says Israeli soldiers crossed the border to uproot a tree which was blocking their view near the Lebanese village of Adaysseh.
A Lebanese army spokesman said troops had fired warning shots and Israel had responded with fire from artillery positions and helicopters.
The Lebanese army confirmed to the BBC that three of its soldiers had been killed and four wounded. The al-Akhbar newspaper confirmed that one of its journalists, Assaf Abu Rahhal, had also been killed.
In a statement, Lebanon's Prime Minister Saad Hariri said Lebanese sovereignty had been violated.
However, UN peacekeepers in southern Lebanon said later on Tuesday there was no evidence that Israeli soldiers had crossed the border, the BBC's Wyre Davies reports.
Israeli Prime Minister Benjamin Netanyahu said in a statement: "Israel has responded and shall respond aggressively in the future to any attempt to disrupt the calm along the northern border or to harm residents of the north or the soldiers protecting them."
An Israeli army spokeswoman told the BBC that two officers had been attacked by "snipers from the Lebanese armed forces" in a "well-planned ambush" during maintenance operations on Israel's side of the border, near the town of Kiryat Shemona.
Earlier, the head of Israel's Northern Command, Maj-Gen Gadi Eizenkot, said the officers had been standing between 300 metres and 400 metres from the border, observing Israeli soldiers removing bushes near the border fence.
"Sniper fire was directed at their position and as a result two commanders were injured as a result of the firing," he said.
The dead officer was named as Lt-Col Dov Harari, 45. The Israeli captain wounded along with him was "critically wounded" but was said to be in a stable condition in a Haifa hospital.
Maj-Gen Eisenkot said he believed the incident was "a one-time event". "We received requests and demands from the highest ranks in the Lebanese army to cease fire," he added.
The UN peacekeeping force stationed in southern Lebanon, Unifil, has urged both sides to show "maximum restraint" following the clash.
Announcing it was in touch with both governments, Washington said it was "extremely concerned" at the violence and urged "maximum restraint" from both sides.
"The region has enough tension as it is," US state department spokesman PJ Crowley said.
"The last thing that we want to see is that this incident expand into something more significant."
EU foreign policy chief Baroness Ashton urged both sides to take "immediate steps" to prevent "a further escalation of tension or any form of further violence".
Israel's Haaretz newspaper said the incident could have been caused by one of the sides misidentifying the correct location of the border.
The exchange comes a day after rockets were fired at the Israeli resort of Eilat.
A stray rocket killed one person in the nearby port of Aqaba in Jordan.