Grant Shapps says both rail operators and Network Rail are to blame for poor performance.
Transport correspondent, BBC News
"Trains must run on time" has been a mantra of Transport Secretary Grant Shapps since he took up his post in the summer.
The opposite has been true on parts of the UK's railways so far this winter.
It's been chaotic on West Midlands Railway and London North Western Railway - both of which are run by West Midlands Trains.
Passengers on that network had a torrid time in the autumn and say things have only slightly improved since then.
Season ticket holders there have already been given a rebate so their fares don't increase this year.
Andy Street, the West Midlands Mayor, has given the company an ultimatum to get its house in order by the end of this month.
South Western Railway is also under pressure to deliver a more punctual service.
And then there is the case of Northern. That company's future hangs in the balance.
One theme is a tendency by train companies to try and deliver overly-ambitious timetables which they are ultimately unable to deliver.
Train companies have to try and live up to commitments they made to government under their franchising agreements.
But the franchising system of today is now widely seen as far too rigid and that's why the government's rail review will soon announce deep reform to the structures that underpin our railways today.
The Transport Secretary Grant Shapps said this morning that he intends to end Northern's franchise to run trains through the region, including Cumbria.
Mr Shapps was speaking as the annual rail fare increase saw some fares go up by almost 3%.
Northen's record has been blighted by problems with insufficient drivers to provide reliable services, reaching their worst when trains stopped entirely on the Lakes Line in Cumbria after a botched introduction of new timetables.
It's completely unacceptable to have a situation where trains almost routinely don't run to a routine, don't run on time.
I will absolutely bring that situation to an end."