The justice secretary has insisted he will not abolish short prison terms.
Ken Clarke told Parliament there was often no alternative for the "irritating recidivist offender".
But he said the number of short sentences should be reduced and replaced with "more credible community punishment sentences".
Short terms have drawn scrutiny as the cost and population of jails rises. The prison population in England and Wales stands at 84,702.
In the Commons on Tuesday, Philip Davies MP said: "The vast majority of people who serve short sentences only do so because they've been given community sentence after community sentence and they've failed.
"The last thing to do with those people is to give them another community sentence for it to fail once again."
Mr Clarke told him: "I largely agree. You've probably been upset by reports that I am minded to abolish short prison sentences.
"I have always expressed precisely the opposite feeling - this has never been my view, that we should abolish all short prison sentences.
"With the kind of irritating recidivist offender who's causing a lot of damage, if they offend over and over again, there is no alternative quite often but to a short prison sentence."
But, he said: "There are too many of them and there are cases where you can avoid the use of short prison sentences but we've got to have some very effective alternative when you do."
"We have to... provide magistrates with the full range of alternatives - more credible community punishment sentences, with a properly punitive element which might have a better chance of rehabilitating the offender."