Malaysian troops are negotiating with about 100 men from the Philippines holed up in a remote part of Sabah on Borneo island, officials say.
The men, some reportedly armed, landed in Lahad Datu town earlier this week.
Police say they have identified themselves as the "royal army" of the Sulu Sultanate, which has a historic claim to the area.
Malaysian and Philippine officials are in discussions over the incident, many details of which remain unclear.
Some reports said that the men had denied links to militant groups in the southern Philippines, and had raised the Philippine flag in the village.
"We have sent a team to negotiate with them," Malaysian Inspector-General of Police Tan Sri Ismail Omar was quoted by the country's state news agency Bernama as saying.
"Discussion is proceeding well and we have told them to leave Sabah peacefully, as we do not want any situation which can threaten the security of the people."
Malaysian Foreign Minister Anifah Aman has talked to his Philippine counterpart, Albert del Rosario, over the phone about the matter, says a statement from the Philippine Foreign Ministry.
Officials from both countries were also able "to clarify that "the activity of this Filipino group was not sanctioned by the Philippine government", the statement says.
The Philippine ministry statement put the size of the group at 200 but indicated permanent residents of Sabah could be among the group.
Tighter security measures have also been put in place in the Philippine islands near the area, the statement added.
Malaysia's Sabah state shares a sea border with the southern Philippines, which is home to a number of Islamic militant and kidnap-for ransom groups. The journey between the two can take only a few hours.
It formed part of the Sulu Sultanate - which once spread over several southern Philippine islands as well as parts of Borneo - before it was designated a British protectorate in the 1800s.
Sabah became part of Malaysia in 1963, and the country still pays a token rent to the Sulu Sultanate each year.