The Colombian government says almost 7m hectares (17m acres) of land have been stolen by armed groups involved in the country's civil conflict over the past 25 years.
Agriculture Minister Juan Camilo Restrepo said hundreds of thousands of families were driven from their land.
He said paramilitaries, rebels and drug dealers had used violence and fraud to take the land.
Colombia this week passed a law which aims to return the land to its owners.
In an interview with the Colombian newspaper El Tiempo, Mr Restrepo said getting that land back would be a "gigantic administrative and judicial task".
The minister said four million hectares of land had been abandoned by owners too scared to return for fear of being killed by right-wing paramilitaries, left-wing guerrilla groups or drug trafficking gangs.
Another two million hectares had been extorted from their rightful owners.
And some 500,000 hectares of government land had been misappropriated by corrupt officials, Mr Restrepo said.
He said most of the stolen land was still in the hands of those who had seized it through violent means or their front men.
But some had been sold on and now belonged to people who had bought the land in good faith, rendering the process of establishing the rightful owner more difficult.
On Wednesday, the Colombian Senate approved a law to compensate victims of Colombia's civil conflict, which also aims to return the stolen land.
President Juan Manuel Santos called the passing of the law a historic move, but experts have estimated it could take a decade to restore the land to its lawful owners.
Illegal armed groups have already tried to undermine the process and several campaigners for land rights have been killed in recent months.