Sunderland 1-3 Arsenal
Aaron Ramsey scored twice as Arsenal beat bottom club Sunderland, who had boss Paolo Di Canio sent off.
Arsenal debutant Mesut Ozil set up Olivier Giroud for an early goal, with Craig Gardner equalising from the spot after the break.
But after Ramsey volleyed in, Di Canio was infuriated when Jozy Altidore was denied a leveller after referee Martin Atkinson failed to play an advantage.
Ramsey scored a fine second as Arsenal moved to the Premier League summit.
The result was their fifth consecutive win after losing on the opening day to Aston Villa.
Although the Gunners were on top for long periods in the game, Sunderland were understandably aggrieved they did not register a second equaliser via Altidore halfway through the second period.
After wrestling with Bacary Sagna just outside the box, the United States striker squeezed a shot past Wojciech Szczesny which trickled over the line before it was cleared, only for Atkinson to bring play back for a Sunderland free-kick.
The judgement was especially frustrating for the hosts as it would have avoided any potential goal-line controversy - the new Goal Decision System was used to perfection by showing the ball had marginally crossed the line.
Di Canio was later sent to the stands and has yet to see his team win this season. Since winning his first two games in charge after taking over on 31 March, the Italian has only garnered three points from 24.
Still, having been over-run in a first half where Ozil excelled, he will be heartened by the manner of his team's second-half reply, where they showed superior energy and could have gone ahead with two other decent chances.
They will need that spirit as, following their next match against fellow strugglers West Brom, they face Liverpool and Manchester United.
The hosts were fortunate to still be in the game at the break as Ozil made an instant impression following his £42.5m move from Real Madrid by demanding the ball early on and providing a host of chances.
Theo Walcott could easily have had a hat-trick before the break but found Sunderland goalkeeper Keiren Westwood in impressive form, as he twice denied the England winger after he was set clear by Jack Wilshere and Ozil.
Arsenal played without a designated left-winger as Ozil drifted into the middle, but the German drifted left to collect Wilshere's pass for the opening goal, and crossed simply for Giroud to score his first Premier League goal outside London.
The Black Cats, who had five of their 14 summer signings in the side, contributed to a flowing opening with Modibo Diakite heading against the bar and Altidore going close.
But despite their probing, they were presented with a gift for the equaliser when Laurent Koscielny felled Adam Johnson in the penalty area after 48 minutes.
The ball was heading away from goal, but Koscielny dived in and Gardner made no mistake from the spot, three minutes after coming on at half-time.
The goal signified a huge turnaround in the game as Steven Fletcher, making his first start since March, had a goal disallowed with Ki Sung-Yeung also guilty of straying offside as Gardner hit the post.
Just after the hour mark, though, the Gunners regrouped and it led to a fine strike by Ramsey, who volleyed in from 18 yards after Carl Jenkinson's right-wing cross.
Altidore's disallowed goal then cast a defining narrative in the match and once Ramsey tucked in after a sweet pass from Giroud for his fifth of the season, Sunderland could not find a way back.