Top Gear presenter Jeremy Clarkson caused offence when he described a car as "speciale needs" - a play on its proper name, Ferrari F430 Speciale.
Media regulator Ofcom investigated after receiving two complaints.
However, it did not censure the BBC as the broadcaster had deleted the comment from repeats and the iPlayer version, and apologised for causing offence.
Clarkson described the car owned by co-presenter James May as looking "like a simpleton".
In response to Ofcom's inquiry, the BBC said his description was a "light hearted reference to the look of the car (the front of which has the appearance of a broad smile) in contrast to a newer model, which was praised by Mr Clarkson."
In its ruling, Ofcom said: "The BBC said that it was the car itself that was the subject of the fun being poked at and its owner... James May."
The episode of the popular motoring show was broadcast at 2130 BST on 1 August - a fact Ofcom took into account, saying that viewers should expect programmes "to contain more challenging content and humour after the watershed".
Ofcom ruled that "the comments made by Jeremy Clarkson in this instance were capable of causing offence."
It went on: "While obviously intended as a joke and not aimed directly at an individual with learning difficulties, the comment could easily be understood as ridiculing people in society with a particular physical disability or learning difficulty."
It said it acknowledged that Top Gear was "well known for its irreverent style and sometimes outspoken humour and banter between the presenters".
In concluding that the investigation was resolved, it said that the BBC had acted immediately and voluntarily in removing the comment from repeat screenings and the iPlayer.
"It had also apologised for any offence caused by the comments, underlining that there was no intent to make fun of those with special needs."
Ofcom frequently receives complaints about Top Gear.
Last year it censured the show for depicting a graphic - albeit spoof - suicide.
The previous year complaints about another joke by Clarkson, about lorry drivers killing prostitutes, were not upheld.