From secret handshakes and rolled-up trouser legs to accusations of nepotism, bullying and the repression of reform, the Freemasons' reputation has not always been positive.
But with six million members worldwide, including more than 200,000 in the UK, negative publicity has not dampened people's desire to join.
"At first I thought it was a bit odd, but then you start to understand what it's all about," says Anna, a senior Freemason from one of England's two women-only masonic societies. "It can be quite addictive."
Freemasonry in England was, for 200 years, only open to men. That changed in the 20th Century.
The first female lodge opened in 1908 with a male Grand Master. His successors have all been female, however, and men are no longer permitted to join the lodge.
Anna - not her real name - has been a member for 21 years and has completed all three Freemasonry degrees, making her one of England's most senior female members.
"We do the same rituals, we do the same ceremonies but we are completely separate to [the men]," she says.
Anna is a confident, intelligent woman who neither looks nor sounds like someone who would want to join an organisation often thought of as a stuffy, elitist boys' club. So why did she?
"My husband's a mason and he said 'I think you'd like it, why don't you join?' I think partly so I wouldn't moan at him going off to meetings!"
Michael, a man in his late forties who has been a member for six years, says, like Anna, he joined on the recommendation of somebody close to him.
"It was an invite from a friend," he says. "I didn't know anything about it.
"I like to trust people, I'm very loyal - that kind of thing [being part of a community] appealed to me. That's what it is all about."
Unsurprisingly, many Freemasons are vague when discussing the fraternity's purpose, but two things crop up repeatedly when you ask why they enjoy being members.
Michael says the Freemasons are among the biggest charitable donors in the UK, with £33m raised last year by more than 180 masonic groups.
Anna, too, lists the charitable side of membership as something she enjoys.
"[My job] is such a shallow profession so it's nice to do something worthwhile," she says.
Admirable though charity may be, people donate all the time without joining secretive societies.
What else, then, compels people to not only join but remain committed members?
For many, Freemasonry is a social network.
"Over the years you build up relationships with people," says Michael. "We become friends.
"One person invites someone in and they generally know each other anyway.
"It's a bit like Facebook in that way. You will come across people you've met in the past, certainly if you've lived in the area for a number of years.
"We come from all walks of life and professions but it becomes a network.
"I use that expression carefully because the network is not to be used for your own personal benefit - that is something they stress."
But what is a network for if it does not operate for the benefit of those involved?
"The perception is that Freemasons pat each other on the back and look after each other in ways other friendship groups don't, that is a complete myth," explains Michael.
"It is frowned upon. It is not to be used for your own personal benefit. It really is for others."
Anna adds: "I don't know a single other woman who's worked in [my industry].
"It has never helped me [in my career].
"I think in the past there probably have been cases of [nepotism] but I've never known of a single case in the women's Freemasons. I know people think there is but that's true of any society, isn't it?"
Each lodge meets four times a year officially to welcome new members in ceremonies, the contents of which have always been a closely-guarded secret.
"It's all based on King Solomon's Temple," says Anna. "It's an allegory, slightly grounded in religion."
While members are discouraged from discussing politics or religion, belief in a higher power has been, historically, a requisite to join.
"The best way to explain it is that it's like a play, which everyone has a part in," she continues.
"The Worshipful Master is like the lead actor, who has the most to say in it.
"When you go through your three ceremonies there are things you have to learn - you have questions you have to learn answers to.
"When you're the master, when you host the ceremony, which can take an hour or so, you do the majority of the talking."
The scripts themselves are never revealed publicly - but not, Michael says, because of any "skulduggery or anything weird".
"Weird" or not, does a secret society that can trace its history back to the Middle Ages really have a role in modern society. And if so, what is it?
"I couldn't answer for the men but for us it's to show us [women] in a favourable light," says Anna. "It's all about self-development. It's a journey to learn about yourself."
Michael adds: "It gives you a moral code in terms of bringing out the best side of human nature."
- There are about 4,700 female Freemasons in the UK and 200,000 male Freemasons in England and Wales with more under the Grand Lodge of Scotland.
- Freemasons meet in a temple, which they call a "lodge", as that is where ancient stonemasons met when working on a church or cathedral
- Lodges are grouped by region, roughly in line with the old county boundaries
- Freemasons wear aprons, because of the supposed evolution of freemasonry from the stonemasons
- The "third degree" is the final stage before becoming a fully fledged Mason. The ceremony involves close questioning, which is where the expression "giving someone the third degree" originates
- Famous Freemasons have included Sir Winston Churchill, Sir Arthur Conan Doyle, Rudyard Kipling and Oscar Wilde
*The names of interviewees have been changed to protect their identities.