UK passports should allow people to define themselves as "X" instead of male or female, campaigners say.
Stonewall, a lesbian, gay, bisexual and transgender (LGBT) rights group, said it would benefit people who face difficulties at passport control.
The government said it was reviewing the way gender was marked in official documents, including passports.
Since 2011, Australian passport-holders have been able to choose X if their gender is indeterminate.
Stonewall's proposal comes days after HSBC let its customers choose non-gender specific titles such as Mx, M and Misc for their bank accounts.
It is estimated that about 650,000 people identify as trans in the UK. Stonewall said many were "afraid to travel abroad" because they feared intrusive questions at passport control.
Tara Stone, a member of the Stonewall Trans Advisory Group, told BBC Radio 4's Today programme X would give some trans people, those who identify as non-binary - neither male nor female - "an option that wasn't prescriptive".
"Some trans people find it very validating of their identity to have that gender designator," she said. "And additionally, in terms of getting access to other things in daily life, having a piece of identification that marks out your gender is actually really useful."
Ms Stone said the X option would also be of great benefit to intersex people who are born with a mixture of male and female sex characteristics.
Stonewall set out several other recommendations to help trans people be "accepted without exception", including:
- Removing the terms "gender reassignment" and "transsexual" - which it sees as outdated - from the Equality Act
- Allowing trans people to change their gender on legal documents without a medical letter
- Raising more awareness of "transphobic" hate crime so that people know how to report it
- Getting more trans people represented in the media as positive role models
Stonewall said existing legislation urgently needed reform, describing the Gender Recognition Act as "outdated".
It said people were forced to choose a gender for pension calculations and insurance policies but that current rules "fall short of best practice".
The Gender Recognition Act, which allows trans people to change their legal gender, is also under review by the government.
A government spokesman said: "We have committed to reviewing the [act] to look at ways of streamlining and de-medicalising the process for changing a person's legal gender.
"Alongside this we are investing £3m to tackle homophobic, biphobic and transphobic bullying."
He added: "We are committed to delivering further positive changes for transgender people."
Stonewall said some trans people had told them members of the public were "hostile" towards them.
Alex Drummond, a trans woman from Cardiff, said: "If we don't see trans people, then it's easy to hold hostility. Familiarity will breed acceptance."
She said it was important for people to see that those who do not identify with a gender also "have friends, and jobs and lives, and be happy".
She added: "If you have to live in stealth to be successful, then you're not living."