Burton stunned Hereford with injury-time strikes from Aaron Webster and Billy Kee.
Rob Purdie gave the hosts the lead from the penalty spot after Calvin Zola fouled Nathan Elder.
Zola made amends by levelling from Jacques Maghoma's cross before Tom Barkhuizen restored the Bulls' lead with a low drive.
That looked to be it before Webster equalised from 20 yards and Kee won it with a fine strike in the 93rd minute.
VIEWS FROM THE DRESSING ROOM
Burton manager Paul Peschisolido told BBC Radio Derby:
"I tell you what - it's absolutely killing me this is. Do you think we could ever go into a game and lead from the front and get two or three goals? It's never going to happen.
"But what a response. In the first half, some of the defending was absolutely atrocious, but the second half was typical Burton.
"We came back from the dead after a few stern words at half-time. They had a never-say-die attitude and I'm delighted to get three points."