Parliament has a "clear" right not to accept the European Court of Human Rights' ruling prisoners must get the vote, Chris Grayling has said.
But the justice secretary warned there would be "consequences" for the UK's position in Europe if MPs chose to defy the judgement.
The government has insisted it will not support the granting of voting rights to prisoners, and most MPs agree.
Mr Grayling said he was thinking "very carefully" about how to act.
The UK has been on a collision course with the European Court of Human Rights since it ruled in 2005 that it was a breach of human rights to deny prisoners a vote.
The court said it was up to individual countries to decide which inmates should be denied the right to vote from jail, but that a total ban was illegal.
In May this year, it gave the UK six months to outline how it proposed to change the law on prisoner votes.
But Prime Minister David Cameron this week reiterated his promise inmates would not get the vote while his government was in power.
However, Attorney General Dominic Grieve warned defying the Strasbourg court could be seen "as a move away from out strict adherence to human rights laws".
Mr Grayling declined to say whether he could contemplate the UK withdrawing from the European Convention on Human Rights, but said he was preparing a "very clear plan" to feature in the Conservative manifesto for the next general election.
He told BBC One's Andrew Marr Show: "It is very clear that most people in the political world in the UK don't want to give votes to prisoners.
"What Dominic Grieve was arguing was that we have to be very careful about how we approach the issue.
"The reality is that we are signed up to the European Convention on Human Rights and the European Court of Human Rights.
"If we therefore choose to disagree with a ruling from that court, we have to understand that we are taking a significant step outside that international commitment.
"I am thinking very carefully about how we do the right thing for the UK."
He continued: "Parliament has, in clear case law, the right to say to the European Court of Human Rights 'We do not agree with you'."
But he said the Law Lord had passed a ruling that said Parliament must understand there were political consequences if it chose to disagree with the European court.
The Convention drawn up in the 1950s to counter the kind of abuses seen in the Soviet Union under Stalin, but was now being used for purposes which its authors never intended, Mr Grayling said.
"These are difficult international issues. We don't want to take steps which destabilise progress in other parts of Europe towards improved human rights, better judicial systems and fairer criminal justice systems," he said.