Moody's Investor Services has cut its outlook on Japan's credit rating to "negative" from "stable" citing concerns about debt levels.
Moody's currently rates Japan's government debt at an Aa2 level.
In January, rival rating agency Standard & Poor's downgraded Japan's credit rating from AA to AA-, also citing debt concerns.
Earlier this month, Japan was overtaken by China as the world's second-largest economy.
Japan has been trying to boost its economic growth and as a result government spending and borrowing has increased.
Moody's said that the government needed to do more to cut borrowing levels.
Japan currently has the highest government debt levels of any industrialised nation.
Moody's said that it cut its outlook on the credit rating on Japan because of "heightened concern that economic and fiscal policies may not prove strong enough to achieve the government's deficit reduction target".
Also they said the government's policies would probably not be able to "contain the inexorable rise in debt, which already is well above levels in other advanced economies".
Analysts said that the move by Moody's was widely expected after S&P's decision to cut its rating.
However, they added that while it may have a limited impact on the bond and currency markets, it could have broader political implications.
"Politicians or the finance ministry could use this as a reason to push for fiscal reform, which could include a sales tax hike," said Satoru Ogasawara, an economist at Credit Suisse.
"It's not as if the main opposition Liberal Democratic Party doesn't want fiscal reform."
At 5%, Japan's current sales tax is the lowest among major economies.
Increasing that figure is a vital part of Japan's efforts to rein in its public debt, analysts say.
But it is a tricky situation for the government, not least because a tax increase may slow consumer spending and hamper an already fragile economic recovery.
According to Nomura Securities, if the sales tax rate was doubled it could cut half a percentage point from Japan's gross domestic product growth rate in the year the increase was implemented.
It would then knock another 0.8 percentage point from GDP the following year.
Mitsubishi Research Institute estimates it could knock as much as 2 percentage points off the real growth rate.
One of the reasons is that a tax increase could trigger a rush of purchases before the tax hike, followed by a subsequent slump in consumer spending, the analysts said.
Some economists have suggested that the government should consider incremental tax hikes to avoid such an impact.