Kevin Blackwell's first home match in charge of Bury started in the worst possible way as Swindon's James Collins scored in the first minute.
The striker's first league goal for the Robins came from a well-judged 20-yard lob over keeper Trevor Carson.
Bury fought back and it took a fine Jay McEveley tackle to deny David Healy.
In the second half, Swindon took charge with Carson saving a volley from striker Adam Rooney but the Robins were not able extend their lead.
VIEWS FROM THE DRESSING ROOM
Bury manager Kevin Blackwell told BBC Radio Manchester:
"It's alright people saying we are being unlucky but sometimes you have to take responsibility and certain players didn't do that. I'm not going to accept it and I've said to the board we're going to have to change players.
"I'm well aware what's wrong, I have to change things and I'm going to. Some of the players aren't good enough and I shall shift them out as best I can.
"Some of the signings are garbage, but some of the players are good players and they need some help. They're getting bogged down trying to help other people.
"This will be a test of my coaching and managerial skills, that's for sure."