Postal voting is open to fraud on an "industrial scale" and is "unviable" in its current form, a top judge has said.
Richard Mawrey QC, who tries cases of electoral fraud, told the BBC that people should not be able to apply for postal votes as a matter of course.
"On demand" postal voting had not boosted turnout or simplified the process for the vulnerable, he said.
But the Electoral Commission said it would not be "proportionate" to end postal voting altogether.
The government also said it had no plans to abolish the current system, saying it had made it easier for many people to vote.
Since 2001, anyone on the electoral roll has been able to apply for a postal ballot.
In January, the Electoral Commission said it was particularly concerned about 16 council areas in England.
In some of those areas, the BBC has heard allegations that political activists are pressuring families into voting for their candidate, or taking ballot papers away to fill them in, which is illegal.
BBC Radio 4's File on 4 spoke to a man in Derby who said his mother had her postal vote cast for her by activists who turned up at her house and pressurised her into letting them fill in her ballot paper.
He said: "Campaigners came to the house and they asked my mum to vote for them and actually my mum, not being able to read English, she didn't know where to put the cross, so one of the people put the cross in the box for her and said, 'There you go now you can just sign it and we will take it off you.'
"Your votes are supposed to be anonymous but when people come to your house and are literally doing it for you that is not you voting is it? That is them voting for you and voting for themselves."
Mr Mawrey, a deputy High Court judge and election commissioner, said in one case last year he had come across 14 different ways that postal ballots could be manipulated.
"Postal voting on demand, however many safeguards you build into it, is wide open to fraud," he told Radio 4's Today.
"And it's open to fraud on a scale that will make election rigging a possibility and indeed in some areas a probability."
Furthermore, he suggested postal voting had not achieved its other stated objectives, such as increasing turnout or helping particularly vulnerable groups who might find voting difficult to do it more easily.
"All it actually does is create a convenience for members of the public who would rather fill in something in their own home."
Turnout has increased in each of the past two general elections, rising from 59.4% in 2001 to 65.1% in 2010.
But returning officer Ray Morgan, who is also the chief executive of Woking Borough Council, said: "I don't think any election that I've personally officiated over since 2006 has been totally fair and honest."
The government is introducing individual electoral registration from June this year, which ministers say will help stamp out some abuses.
It will mean that anyone who wants a postal vote will have to apply individually and prove their identity.
At the moment there are no identity checks and households can be registered for multiple votes.
The UK Independence Party warned the authorities not to "turn their backs" on the problem, saying Mr Mawrey's comments were "extremely alarming".
Jenny Watson, chair of the Electoral Commission, told the Today programme that political parties should have to sign up to a strengthened code of conduct that the commission would regulate.
"We are talking about the behaviour of unscrupulous campaigners who act in an improper way to put pressure on people. It is that behaviour that needs to be tackled," she said.
"You can't punish voters for the behaviour of unscrupulous campaigners and that's what abolishing postal voting on demand would do."
Cabinet Office minister Greg Clark said he recognised the problem of fraud but believed postal voting had been "very important in making sure that people can access the franchise".
He said: "I think it particularly important that a relatively small number of cases of abuse, which need to be addressed and clamped down very firmly, don't prevent other people - the vast majority of people - using it in a law-abiding way."
Labour MP Graham Allen, chair of the Commons Political and Constitutional Reform Committee, said the focus should be on increasing the number of people registered to vote.
"It is a great pity that the BBC is unwittingly being a part of a co-ordinated political campaign to remove the vote potentially from millions of people," he said.
"It should spend a proportionate amount of energy on revealing the enormous scandal of millions not on the electoral register."
The full story will be broadcast on File on 4 on BBC Radio 4 on Tuesday, 11 March, at 20:00 GMT.