Edinburgh survived a tense finish to secure their first Pro12 away win of the season and their first victory at the Dragons since 2006.
Willem Nel and Matt Scott crossed for Edinburgh's tries with full-back Jack Cuthbert kicking 13 points.
Replacement Ross Wardle scored a late try for the hosts, who put pressure on the visitors during the closing stages.
But Alan Solomons' side held on to secure their second league win of 2014.
The match at Rodney Parade had been rearranged after the original scheduled fixture in February was called off due to a waterlogged pitch.
Dragons had Wales wing Hallam Amos back in their starting line-up after two months out with an ankle injury.
Amos came close to giving the home side the perfect start when he was within inches of the try line before the ball was knocked out of his hands by Sam Hidalgo-Clyne.
Tovey put the Dragons in command with two penalties during the opening half hour with the home side dominating territory and possession.
Cuthbert broke Edinburgh's duck with a long-range penalty on 33 minutes and the visitors began to grow into the game.
It was the visitors who crossed for the game's opening try when Scott feed Nel to drive over under a pile of bodies.
Cuthbert landed the conversion but an infringement by Edinburgh handed Dragons a penalty which Tovey scored to bring his side within a point at the end of the first half.
Fly-half Tovey added a fourth penalty, following an infringement by Nel, to give the home side a 12-10 early in the second half.
But Scott's weaving run just two minutes later produced a second Edinburgh try.
The Scotland centre slipped through a gap 30 metres from the line to ground for a try, which was confirmed after deliberation by TMO Derek Bevan.
Cuthbert converted the try before he added his third penalty of the game to extend Edinburgh's advantage.
Edinburgh were reduced to 14 men when lock Izak van der Westhuizen was shown a yellow card and the home side capitalised on their numerical advantage.
Wardle drove over from a matter of metres from the resultant attacking line-out and pass from Wayne Evans with Tovey converting.
Dragons pushed for the try which would have secured victory and had three scrums on the Edinburgh line but the seven-man pack held out an move up to seventh in the table.
Newport Gwent Dragons: Dan Evans; Tom Prydie, Pat Leach, Jack Dixon, Hallam Amos; Jason Tovey, Richie Rees; Phil Price, Elliot Dee, Duncan Bell, Andrew Coombs (capt), Matthew Screech, Lewis Evans, Taulupe Faletau, Nic Cudd.
Replacements: Sam Parry, Owen Evans, Bruce Douglas, Cory Hill, Jevon Groves, Wayne Evans, Ross Wardle, Will Harries.
Edinburgh: Jack Cuthbert; Sam Hidalgo-Clyne, Matt Scott, Andries Strauss, Tom Brown; Harry Leonard, Grayson Hart; Alasdair Dickinson, Ross Ford, Willem Nel, Grant Gilchrist, Izak van der Westhuizen, Mike Coman (capt), Cornell Du Preez, David Denton.
Replacements: James Hilterbrand, Wicus Blaauw, Simon Berghan, Ollie Atkins, Roddy Grant, Sean Kennedy, Carl Bezuidenhout, Sam Beard.
Referee: Leo Colgan (IRFU)
Assistant Referees: Simon Rees, Greg Morgan (both WRU)
Citing Commissioner: John Charles (WRU)
TMO: Derek Bevan (WRU)