PM Malcolm Turnbull has called on all Australians to respect the country's values, or feel free to leave.
Speaking one week after the shooting of a police worker in an apparent terror attack, Mr Turnbull also said anyone preaching hatred or intolerance was undermining the country.
Police accountant Curtis Cheng was gunned down in Sydney Parramatta's suburb last Friday.
His killer, 15-year-old Farhad Jabar, was shot dead by police at the scene.
Police believe Jabar had been radicalised. Four people suspected of being linked to the attack were arrested in raids in western Sydney this week, but most have now been released without charge.
On Friday after holding talks with Muslim leaders, Mr Turnbull said violent extremism was "a challenge to the fundamental Australian value" of mutual respect in a multicultural society.
"It is not compulsory to live in Australia," he said. "If you find Australian values are, you know, unpalatable, then there's a big wide world out there and people have got freedom of movement."
"Those who seek to gnaw away at that social fabric are not part of the Australian dream, they are not advancing the interests of our great country."
He encouraged Australians to "call out the language of hatred wherever it is practised".
Earlier on Friday, the chairman of the mosque in Parramatta, Neil El-Kadomi, told worshippers that they should appreciate what Australia had done for them.
"We live in Australia," he said, in comments quoted by Fairfax Media. "Australia gives us lots of benefits, there's opportunity and you are allowed to practise your religion freely."
He told Fairfax Media there was no place for extremism at the mosque, which was the last place visited by Jabar before he carried out his attack and was regularly used by some of the men arrested this week.
Mr Kadomi said he would tell worshippers on Friday: "If you don't like Australia, leave."
Last week's shooting came amid heightened anxiety in Australia about the influence of domestic and overseas radical Islamist groups.
Around 120 Australians are currently fighting with IS in Syria and Iraq, and almost 30 Australians are believed to have been killed, a spokesperson for the Minister for Justice told the BBC last month.
Officials are worried about the security implications from those who return home, and from those who support them.
More than 160 Australian passports have either been refused, suspended or cancelled on national security grounds.