Tony Hall is to step down as director general of the BBC in the summer, after seven years in the role.
Lord Hall said the decision had been hard, adding: "If I followed my heart I would genuinely never want to leave."
But he said he felt it was important the BBC had the same leader for the BBC's mid-term review in 2022 and the renewal of its charter in 2027.
The National Gallery subsequently announced he had been appointed chair of its board of trustees.
The BBC's chairman, Sir David Clementi, said the search for the BBC's next leader would begin "within the next few weeks".
He added that the BBC was "committed to selecting the best-qualified person for the job".
BBC Newsnight understands Lord Hall had wanted to stay until the BBC's centenary in 2022, but after "big discussions with the board" it was decided he would go earlier.
The programme's political editor Nicholas Watt said the timing was being interpreted in government circles as "something of a masterstroke" by Sir David, whose term as chairman ends in 2021.
"Had Tony Hall waited until 2022 it would have been a new chairman of the BBC making the appointment, and that new chairman would have been appointed by Boris Johnson's government," said the correspondent.
James Purnell, the former Labour politician who became the BBC's director of radio and education in 2016, is currently the bookmakers' favourite to take on the role.
Media commentators are also speculating, with The Independent's Adam Sherwin suggesting Channel 4's chief executive Alex Mahon might be in the frame, along with Apple's Jay Hunt and Charlotte Moore, the BBC's director of content.
The Guardian also points to these female candidates but also suggests former Ofcom chief Sharon White and Gail Rebuck, chair of Penguin Random House UK, might be in contention.
Newsnight's Nicholas Watt said he was hearing that the cabinet "are expecting the BBC to sound them out on this appointment", and there was "no appetite" in the government for "a BBC lifer" who is not interested in reform.
Speaking of Lord Hall, Sir David said the corporation had been "lucky to have him".
He described him as "an inspirational creative leader" who had "led the BBC with integrity and a passion for our values".
Culture secretary Nicky Morgan thanked Lord Hall for his service, saying he had made "a huge contribution to public service broadcasting in his career".
"In this ever changing broadcast landscape the next DG will need to build on Lord Hall's success," she wrote on Twitter.
'Change has been tough'
In a letter to staff, Lord Hall said he believed he would be "leaving the BBC in a much stronger place than when I joined".
His appointment in 2012 followed the resignation of George Entwistle in the wake of the Newsnight Lord McAlpine row.
Lord Hall said the BBC felt "a very different organisation" that was "more innovative, more open, more inclusive, more efficient [and] more commercially aware".
What does the director general do?
The director general, or DG, is the chief executive officer of the BBC, its editor-in-chief.
The person in the post is the editorial, operational and creative leader of the corporation, with responsibility for a worldwide workforce running services across television, radio and online.
The DG and the BBC board are responsible for the effective running of the BBC, delivering its public service and commercial services - including BBC Studios - both at home and abroad.
The DG is appointed by the BBC board. As of April 2019, the holder of the post is paid £450,000 a year.
"Change has been tough at times," he wrote. "But I believe our recent record of transformation stands comparison with virtually any other creative organisation in the world."
Lord Hall takes over at the National Gallery in London from interim chair Sir John Kingman, who has been in the post since Hannah Rothschild's departure in September 2019.
He said: "I am proud to take on the role of its chair. The National Gallery isn't just about serving those who already love art, but reaching a wider audience and future generations."
His appointment was welcomed by Baroness Morgan, who said he would bring "a wealth of experience" to the role.
Analysis - BBC media editor Amol Rajan
Director general of the BBC is one of the most privileged, but also one of the most relentless and tough jobs in Britain - and it gets tougher every day, because of the technological context.
His successor will need to combine world-class political, commercial, editorial and managerial talent, while coming under a relentless barrage of criticism from all fronts.
The question of who gets it will depend on where the BBC Board and its chairman, Sir David Clementi, want to place their emphasis. Someone with commercial nous, or someone who can charm Boris Johnson and Dominic Cummings? Someone with a track record at managing talent - or someone who can make a brave, correct call on a Panorama investigation?
The perfect candidate will be able to do all this - and therefore doesn't exist.
Lord Hall's departure from the BBC comes amid another turbulent time for the broadcaster, with issues around equal pay disputes, political bias, diversity and TV licences at the top of its agenda.
Earlier this month presenter Samira Ahmed won an employment tribunal in a dispute over equal pay, while radio presenter Sarah Montague has confirmed she won a settlement and an apology from the BBC after being treated "unequally".
It also found itself in the spotlight back in 2018 when Sir Cliff Richard won a High Court case against the broadcaster over its coverage of a police raid on his home.
Last year TV presenter and campaigner June Sarpong was appointed the BBC's first director of creative diversity in a bid to improve the BBC's "on-air talent portrayal".
Sarpong said it had been a "great pleasure and honour" to work under Lord Hall's "visionary leadership".
"I cannot overstate the support he has shown me in helping to start the process of inclusive change," she wrote on Twitter. "He will be sincerely missed."
Other developments during Lord Hall's tenure include the growth of BBC iPlayer, the increasing success of BBC Studios. and the launches of BBC Sounds and streaming service Britbox.
It was also announced last year that free TV licences for all over 75s would be scrapped and replaced by a scheme based on pension credit benefit.
Last week Lord Hall outlined a plan that will see at least two thirds of the corporation's staff based outside London by 2027.
Lord Hall - a mini biography
Lord Hall, 68, joined the corporation as a trainee in the Belfast newsroom in 1973.
He became editor of the Nine O'Clock News at the age of 34 and was appointed chief executive of BBC News in 1996.
While there, he launched BBC Radio 5 Live, the news channel, the BBC News website and the parliament channel.
He left in 2001 to become the chief executive of the the Royal Opera House until 2013, and was also deputy chairman of Channel 4.
He was also on the board of the London organising committee for the Olympic Games before returning to the BBC in the top job.
Lord Hall, whose official title is The Lord Hall of Birkenhead, was made a cross-bench peer in 2010.
Reaction to news
Bonnie Greer, who served alongside Lord Hall on the Royal Opera House board, noted that the BBC had been going through a difficult period.
"I think it's been a lot of pressure for the organisation and a lot of pressure for him," the playwright and critic said on The Victoria Derbyshire Show.
Talent agent Jonathan Shalit said Lord Hall had been "fantastic" in the director general role and that it was "normal" for someone in his position to move on after seven years.
"He was going to be moving on soon," he told the same BBC programme. "I think the point he makes about making sure one person oversees the midway point of the licence fee is important."
Philippa Childs, head of broadcast trade union Bectu, said Lord Hall had been "a strong advocate for the BBC, its staff [and] the licence fee".
She said the outgoing director general had always "acted with integrity and transparency" during their "extremely positive and productive working relationship".
Veteran presenter David Dimbleby said Lord Hall "revived the place" because of his passion for broadcasting.
On the future of the BBC, Dimbleby added: "It's always worth remembering, I think in the end, people trust the BBC more than the politicians who try to diminish it.
"I'm not saying there aren't major, major problems with the BBC at the moment; I think there are."
Former BBC chairman Michael Grade said he did not believe that Sir David, or the BBC board, would allow the government to interfere with the appointment of the director general.