Department for Culture, Media and Sport staff face redundancy
All civil servants working in the Department for Culture, Media and Sport (DCMS) have received letters telling them they are at risk of redundancy.
It follows the successful delivery of the Olympic Games and the Queen's Diamond Jubilee celebrations, on which many of the officials worked.
The PCS union said the cuts were the "definition of cruelty".
The government said the DCMS would be smaller and more flexible, "a model for future civil service working".
The department needs to fill a £2.2m funding gap by April next year.
It is expecting to cut its workforce by about 20% and further reduce costs by moving to cheaper office accommodation.
The PCS, which represents thousands of public sector employees, said the work of the civil servants had "touched every person in the UK" and the cuts showed ministers had "little regard for staff".
General secretary Mark Serwotka said: "This legacy from Jeremy Hunt is a new definition of cruelty, threatening staff with the sack while the Olympic spirit is still alive across the country.
"We will do everything we can to fight these and other cuts that are not only not working, they are making our economic situation worse."
The department, which has been tasked with finding a 50% saving in its administration costs, said it was opening a voluntary redundancy scheme but would move to a compulsory scheme, if too few people volunteered.
A spokesman for DCMS said the move was timed to come after the Olympics because the department would be "facing different priorities" in the coming years.
Of the 410 civil servants working in the department currently, approximately 80 will be made redundant by the beginning of the next financial year.
The department will also be moving out if of its plush accommodation close to Trafalgar Square down the road to 100 Parliament Street, which will be shared with Treasury civil servants and is expected to be "considerably cheaper".
A spokesman for DCMS said: "The department is playing a full part in the cross-government work to reduce the deficit, and this includes making 50% cuts to our administrative costs.
"Much progress has already been made particularly in reducing non-pay costs such as accommodation, but it is clear that more needs to be done.
"The post-Olympics DCMS will be a smaller, more flexible organisation, a mixture of permanent and temporary staff - we believe it will become a model for future civil service working.
"To get there, however, a number of posts will be cut and we hope to achieve this through a voluntary redundancy scheme and a fair selection process for staff remaining."