Justice Secretary Ken Clarke has said it is "nonsense" to suggest that murderers and rapists are going to be given the right to vote in elections.
He also rejected calls for the UK to withdraw from the European Convention on Human Rights over the issue.
The government has been warned it must allow some prison inmates to vote to comply with a ruling by the European Court of Human Rights.
MPs will vote on the matter on Thursday with some Tories expected to rebel.
Both the cabinet and shadow cabinet have been told to abstain in the vote. Backbench MPs and the rest of Labour's front bench will have a free vote - although their decision will not be binding on the government.
Shadow Justice Secretary Sadiq Khan said: "It isn't - nor has it ever been - Labour's policy to give prisoners the vote.
"It is important that MPs get to debate this issue, but despite numerous requests by me the government has refused to share the legal advice on which their decision to grant certain prisoners the vote is based nor have they answered the other questions I have asked.
"There are therefore questions about whether Thursday's motion is compliant with our obligations under the European Convention of Human Rights. As a result, it is right that the shadow cabinet abstain on Thursday."
David Cameron told MPs at Prime Minister's Questions: "I don't see any reason why prisoners should get the vote.
"This is not a situation I want this country to be in, and I am sure that you will have a lively debate on Thursday when the House of Commons will make its views known."
The prime minister has said he feels "physically ill" at the thought of granting convicted criminals the right to vote but he had to abide by the European Court's ruling.
Some Conservative MPs have reacted angrily to the PM's stance and have called on him to withdraw from the UK from the European Convention on Human Rights over the issue - something that has been ruled out by ministers.
Tory MP Priti Patel asked Mr Cameron at prime minister's questions for an assurance that Parliament will have the final say.
And she added: "In view of the public disdain of these unelected bureaucrats in Strasbourg, will the prime minister defend our country from any further sanctions from Europe?"
Mr Cameron replied that he had "every sympathy" with Ms Patel's views.
Attorney General Dominic Grieve will lead the debate for the government in the Commons, but will set out the legal position surrounding prisoners' votes rather than arguing for the Government's position that the vote should be given to the "absolute minimum" number of inmates.
Ken Clarke told BBC Radio 4's Today programme: "Human rights law tends to be applied to rather unattractive and unpleasant people.
"It's easy to give human rights or freedom of speech to people with big popular support. These issues always come up with people who are not very popular.
"Prisoners are quite rightly being punished and not popular. Some of them are going to get the vote but probably as few as we can give them consistent with our legal obligations."
Mr Clarke said prisoners serving longer sentences would not be given the vote - but the cut-off point would be decided at a later date.
He told Today "the government does not defy the ruling of courts whose jurisdiction it has always accepted," saying the previous Labour government had "messed about" since the ruling was made five years ago and the coalition was "grasping the nettle".
It comes a report by a committee of MPs says that the UK would be breaking international law if it does not grant some prisoners the vote and could face compensation claims.
"The evidence we have received from our witnesses, including a former Lord Chancellor, is that, however morally justifiable it might be, this current situation is illegal under international law founded on the UK's treaty obligations," the report, by the Political and Constitutional Reform Committee, concludes.
Aidan O'Neill QC, a leading human rights lawyer, told the committee Britain was unlikely to be expelled from the Council of Europe for failing to comply.
But he added it would be a "great pity" if the UK, as a founding member of the European Court of Human Rights "were to set an example to other States in the Council of Europe that they do not have to abide by the law".
At present, in the UK, only prisoners on remand are allowed to vote.
In 2005, the European Court of Human Rights ruled that blanket ban unlawful, and in June, the Council of Europe, which enforces the court's decrees, urged the coalition government to rectify the situation.
The government says it has been advised that unless the law is changed it could face compensation claims from prisoners costing well over £100m.
Thursday's debate was secured by senior Conservative backbencher David Davis and former Labour cabinet minister Jack Straw who convinced a new Commons committee to allow a vote on the issue.
The voting rights of prisoners is a UK-wide issue and will affect Scotland and Northern Ireland even though the administration of justice is devolved.