There is a "very strong case" for abolishing jail terms of less than six months in England and Wales, Justice Secretary David Gauke has said.
He said shorter terms were not working for many inmates and courts should focus more on community rehabilitation.
He acknowledged there were "closely defined exceptions" such as people convicted of violent or sexual crimes.
Penal reform campaigners welcomed Mr Gauke's comments but fellow Tory MP Philip Davies said they were "idiotic".
In a speech in central London, the justice secretary called for a more "imaginative" approach to sentencing.
"We should be extremely cautious about continuing to increase sentences as a routine response to concerns over crime," he said.
There should be a "national debate about what justice, including punishment, should look like for our modern times", he added.
According to Mr Gauke, more than 250,000 custodial sentences of six months or less and more than 300,000 for 12 months or less were handed out in the past five years.
He added nearly two thirds of those offenders go on to commit a further crime within a year of being released.
Under the proposals, which could require legislation, short prison sentences would be replaced by "robust" community orders.
The announcement on Saturday that GPS tagging technology would be rolled out nationwide to monitor offenders forms part of the plans.
Mr Gauke challenged the view that there is only a choice between "soft" and "hard" justice.
"In my view, we should be talking about smart justice," he said.
In Scotland, a presumption against prison sentences of less than three months is already in place and is due to be extended to 12 months.
Official figures from 2018 show those sentenced to short jail terms in Scotland were reconvicted almost twice as often in 12 months than those given community payback orders, which in most cases include unpaid work in the community.
By Danny Shaw, BBC home affairs correspondent
Penal reform has been on the government's agenda for the past three years - but what David Gauke is proposing goes much further.
In February 2016, then Prime Minister David Cameron said the system was "stuck in the dark ages" but his most far-reaching proposals were for weekend imprisonment and greater governor autonomy.
Two of Mr Gauke's predecessors, said they wanted to see fewer people locked up but neither suggested making changes to sentencing.
The current justice secretary, however, said legislation to restrict the use of short sentences should be "explored" claiming there would be cross-party backing.
It is a bold step for a Conservative minister to propose a measure knowing some in his party will criticise him for being "soft" on criminals.
But after 13 months in the post Mr Gauke clearly feels he has the evidence to back up his case and the authority to carry it through.
Charities working to reform penal policy welcomed the speech.
The chief executive of Revolving Doors Agency, which campaigns for shorter prison sentences, said terms of less than six months were "ineffective and disruptive".
Christina Marriott said short sentences "contribute to prison churn and chaos, making it harder to rehabilitate the people who do need to be there".
The Prison Reform Trust said Mr Gauke was "establishing a reputation as a thoughtful, balanced policy thinker, driven by evidence not preconception".
The cabinet minister is not the only voice in the government to advocate reform. In January, prisons minister Rory Stewart told the Daily Telegraph short sentences were "long enough to damage you and not long enough to heal you".
However, Conservative MP Philip Davies said Mr Gauke's proposals to abolish short terms were "bonkers".
"In virtually every case the offender has been given community sentence after community sentence and they are only sent to prison because they have failed to stop their offending," he said.