Camelot failed in his bid to become the first Triple Crown winner since 1970 as 25-1 shot Encke was the surprise winner of the St Leger at Doncaster.
Camelot, the 2-5 favourite, raced at the back of the field in the early stages but looked to be making good progress with half a mile to go.
He challenged two furlongs from home, but the Mahmood Al Zarooni-trained Encke went clear.
Camelot finished three-quarters of a length back, ahead of Michelangelo.
The defeat for the 2,000 Guineas and Epsom Derby winner prevented Aidan O'Brien from becoming the first man to train all five British Classic winners in a season.
O'Brien, whose son Joseph was on board Camelot, said: "It wasn't what we thought it was going to be. It was a steadily-run race and Joseph said he was a little bit fresh with him, but that was probably always going to happen in a slowly-run race.
"You have to take your time on him over a mile and six which he (Joseph) did and when he got out he just stayed on rather than quickened.
"In the Guineas he quickened and in the Derby he quickened, but he just stayed on here.
Camelot was bidding to become only the 16th horse to win the three English Classics in one season. Nijinksy, ridden by Lester Piggott, was the last to achieve the feat 42 years ago.
O'Brien added: "He ran a great race but just got beat. It's disappointing for everybody but that's the way it is. That's racing.
"We expected him to win and if I thought they were going to go that steady I would have had a pacemaker in or two pacemakers. But that's my fault.
"I'm not sure if he'll run again this year. If I thought there was a chance he was going to stay in training next season he might not run."
John Gosden's Michelangelo was third under Frankie Dettori, a further three lengths away.
For Encke's jockey Mickael Barzalona, victory in the world's oldest Classic, which was first run in 1776, comes after he landed the 2011 Epsom Derby on Pour Moi.
And the Frenchman was praised by Godolphin racing manager Simon Crisford as Sheikh Mohammed's group won the Leger for a sixth time.
"Mickael rode a beautiful race and when he kicked at the two-pole he put the race to bed really," said Crisford.
"We weren't sure about his stamina going into the race but he's quite stoutly bred, so we thought it wouldn't be a problem.
"What we loved about that race was the turn of foot he showed, that stands very well for next season and he will stay in training.
"I would imagine it's very unlikely (that he will run again this season) but we will see what Sheikh Mohammed wants to do."
Elsewhere, at the Irish St Leger, Royal Diamond held off Massiyn in a pulsating finish at Curragh.
O'Brien's Fame And Glory, ridden by Jamie Spencer, was in front a quarter of a mile from home.
But 16-1 shot Royal Diamond, trained by Tommy Carmody and ridden by Niall McCullagh, moved powerfully and stayed on to get the verdict from Massiyn, with Brown Panther third.