Sajid Javid has promised to "do whatever it takes" to put right problems faced by the Windrush generation after he succeeded Amber Rudd as home secretary.
Mr Javid said as a second generation migrant he was "angry" at the treatment of those caught up in the saga.
He also disowned the "hostile environment" tag attached to the government's migration policy.
Labour said he would be judged on getting "justice" for people affected.
Facing MPs' questions for the first time as home secretary, just a few hours after getting the job, Mr Javid said the difficulties faced by "longstanding pillars of the community" should never have happened, adding: "I thought that it could be my mum, my brother, my uncle or even me."
He told MPs: "I want to start by making a pledge, a pledge to those from the Windrush generation who have been in this country for decades and yet have struggled to navigate through the immigration system. This never should have been the case and I will do whatever it takes to put it right."
And he said he would not be using the phrase "hostile environment" to describe immigration laws introduced by Theresa May when she was home secretary.
He told MPs: "I think the terminology is incorrect, I think it's a phrase that is unhelpful and does not represent the values as a country."
Mr Javid was appointed after Ms Rudd quit on Sunday evening, saying she "inadvertently" misled MPs over immigration removal targets.
Ms Rudd told MPs last week the Home Office did not have targets for removing illegal immigrants, but on Sunday the Guardian published a letter in which Ms Rudd set out her "ambitious but deliverable" aim to deport 10% more illegal immigrants over the "next few years" to Theresa May.
When asked if she should take some personal responsibility for her home secretary's resignation, Mrs May said: "When I was home secretary, yes, there were targets in terms of removing people from the country who were here illegally.
"If you talk to members of the public they want to ensure we are dealing with people who are here illegally."
She said she was "very sorry" to see Ms Rudd go, adding: "I think she can look back with pride as home secretary."
Mr Javid, a former investment banker and MP for Bromsgrove since 2010, has been communities secretary for about 18 months.
The 48-year old, who previously served as business and culture secretaries, led the government's response to last year's Grenfell Tower fire disaster.
BBC political editor Laura Kuenssberg's view
Politically neat, generally welcomed by colleagues - In the recent canon of Tory events the relatively smooth landing of this appointment is an achievement in itself.
But moving Sajid Javid in, after Amber Rudd took herself out, does not end the prime minister's problems. She and Mr Javid need to move fast to cauterise the political wounds from Windrush.
He can, and did, make a more compelling and personal case in the House of Commons, showing what seems real anger about what has happened to those caught up in Windrush, and unafraid to use his own family story to display it too.
Colleagues have called him a "good operator", and "compassionate and empathetic". Many in Westminster are pointing to his own family history as the ultimate Tory dream - the boy whose dad arrived in Britain with £1 in his pocket, who through hard work ended up in the cabinet, with a portrait of Margaret Thatcher on his wall.
His appointment is also a landmark, he is the first politician from an ethnic minority to take on one of the great offices of state - the biggest jobs in cabinet.
But whatever the presentation and the political messaging, the realities of the Windrush fiasco affect real lives. It's not going to be fixed with a new face or a more sympathetic soundbite.
Windrush was the issue which led to change
Ms Rudd's departure came after she faced mounting criticism over her handling of the Windrush scandal and immigration policy.
The Windrush generation settled legally in post-war Britain and automatically got the right-to-remain in the UK - but the UK government did not keep a record of everyone in that position.
Some people who do not have the paperwork to prove they are in the UK legally have been detained, lost their jobs and denied access to medical care.
This has prompted calls for the government to abandon its "hostile environment" policy on illegal immigration, which Ms Rudd and Mrs May continued to defend, although Mr Javid told MPs he preferred the phrase "compliant environment".
He told MPs Windrush cases were being investigated "as a matter of urgency", with more than 100 processed so far and 500 appointments scheduled.
Responding for Labour, shadow home secretary Diane Abbott said: "The Windrush generation was my parents' generation. I believe - and most British people believe - that they have been treated appallingly.
She added that Mr Javid "will be judged not on the statements he makes this afternoon: he will be judged on what he does to put the situation right and get justice for the Windrush generation".
To anger from Labour benches, the home secretary replied that Ms Abbott did not "have a monopoly" on anger at the situation.
The task ahead for new home secretary
The new home secretary faces a massive task. And none of it - from terrorism through to the Windrush crisis - is easy to fix, writes BBC home affairs correspondent Dominic Casciani.
The Home Office sits at the heart of some of the most complex and challenging decisions in government - and, if his predecessors' experiences are any guide, Sajid Javid should probably give up now on the hope of having much of a life beyond his red ministerial boxes.
First on the list for any incoming home secretary is appreciating the scale of the security threat.
Every day, he will see a dozen dossiers from the police and intelligence agencies asking for permission to intercept the communications of people who are a threat to the UK. This is his first responsibility: keeping the UK safe.
Opposition points finger at May
Opposition parties have targeted Mrs May - under whose Home Office tenure the "hostile environment" policy was introduced - over the Windrush row.
Ms Abbott said "all roads lead back" to the prime minister who, as the "architect of this crisis" - must come before the Commons to explain "whether she knew that Amber Rudd was misleading Parliament and the public last week".
"Many of the elements of this hostile environment originated under Theresa May and, most important of all, it was in 2014 that she passed legislation which removed the protection from deportation which up until then had applied to Commonwealth citizens," Ms Abbott told BBC One's Breakfast.
The SNP's home affairs spokeswoman Joanna Cherry said: "It is the prime minister who has created the fundamental reasons for the Windrush scandal."
Following Mr Javid's promotion, No 10 also announced former Northern Ireland secretary James Brokenshire was returning to the cabinet as housing, communities and local government secretary while International Development Secretary Penny Mordaunt would take Ms Rudd's other role as women's and equalities minister.
How the immigration 'targets' row unfolded
When Ms Rudd was being grilled by a committee of MPs on Wednesday about the Windrush saga she told them there were no removals targets for illegal immigrants - comments subsequently contradicted by a 2015 inspection report.
She later admitted "local" targets for voluntary removals had been set but she told the Commons on Thursday she had not been aware of them.
But the Guardian reported a June 2017 memo from an official, copied to Ms Rudd, that referred to targets. The newspaper also published a letter at the weekend, from January 2017, where Ms Rudd told Theresa May about plans to restructure her department and increase removals "over the next few years".
Sources told the BBC that on Saturday and Sunday Ms Rudd and her officials did a thorough search of all documents and found other references to operational targets which she felt she should have been aware of.
The Windrush generation reacts
Some of the people who have been caught up in the migration saga that led to Ms Rudd's resignation have been giving their views.
Winston Walker, who came to the UK from Jamaica as a toddler in 1966, said the home secretary had been "a bit of a scapegoat for something that her predecessor created".
"We have to wait and see if anything changes," he said.
"I don't know. It's us that are sat in limbo waiting."
Tanya Simms, who was born in the UK to parents who came from Jamaica in the 1960s and had her passport application rejected, said the situation "still needs to be handled".
"Until we've got an outcome and something has actually changed, it's not a victory. It's not a victory until everybody is out of this situation," she said.
"It's not fair. People have been kept prisoner in the country they are born in. Their lives have stopped."
Whitfield Francis, who came to England aged nine and has been unable to work for four years because he does not have the necessary documentation, added: "Unfortunately Rudd's lost her job but she'll probably be able to find a new job quicker than I can."