Retail stores are turning to facial recognition technology and customer tracking tools to fight against shoplifters. BBC Future investigates.

A man walks up to the front door of a jeweller in the centre of Rotterdam and buzzes to enter but it doesn’t budge. He waits. While he lingers by the door, a facial recognition camera quickly scans his face and cross references the image with a watch list of known shoplifters from the local police department. It turns out he has a criminal record for shoplifting and the jeweller doesn’t want him on the premises.

That was one example of a pilot called FotoSwitch in 2011, a program run by the Rotterdam Rihnmond police department, the Netherlands’ Ministry of Security and Justice, and the Dutch Federation of Gold and Silver, aided by Spanish biometrics firm Herta Security.

The pilot gave jewellers an opportunity to quickly screen customers before they entered. The door would also stay locked if a person was wearing sunglasses or something obscuring their face.

It’s the latest way retailers have tried to combat theft. But is it enough to tip the battle in their favour?

New tricks

Figures from the UK’s Office of National Statistics show the number of shoplifting offences totalled 326,464 between April 2014 and March 2015 in the UK, compared to 321,078 and 300,623 for the previous two years. Shoplifting appears to be growing, and thieves are using new tricks to try and steal goods, like “flash robs”, where groups of thieves co-ordinate via social media.

Steve Rowen of US-based Retail Systems Research (RSR) says that among the retailers it usually surveys, the challenge of preventing stock being pilfered by shoplifters is a constant. This has created a need for more intensive tools not only for surveillance but for managing a store in general, from staffing to presenting products. “CCTV, let’s be honest, you couldn’t use it for more than a basic general description of a person,” says Rowen.

CCTV is a classic method for getting a glimpse of your suspect but it can hit a dead end if it doesn’t have anyone to compare it to – a list of known or suspected shoplifters, for example.

More companies are now looking at using facial recognition to keep their stock safe, says Joseph Rosenkrantz, the CEO of FaceFirst, which develops facial recognition technology for multiple industries. He says, only a few years ago, the accessibility of facial recognition was out of bounds for most retailers – it simply cost too much.

Stolen tools and electronics, for example, are easier to resell, making them an attractive target

Herta’s technology needs just the “slightest glimpse” to match against a database and can recognise up to 20 or 30 faces in a crowd, says Gary Lee, Herta Security’s international business development manager. The company remains tight-lipped about who its clients are but it is currently testing its facial recognition system with a large electronics chain.

Rosenkrantz from FaceFirst says its technology is mostly used by grocery stores, DIY stores, and big box retailers. Stolen tools and electronics, for example, are easier to resell, making them an attractive target.

Links to database

So how does this facial recognition work? When a match is determined, the store is alerted with details of the shoplifter such as a photo that was taken during their last arrest as well as other details such as their name, the nature of the crime, and where they stole from in the past.

“All that information is assembled and emailed just to the adjacent manager or person in charge of security at the location, which is important,” says Rosenkrantz. “Let’s say there are 500 stores and a person walks into a store in Essex, you don’t want to notify managers in the other stores. The system is location aware, which is a must for this to work.”

Stores using facial recognition have to get their own access to a database of known shoplifters, which is often done in collaboration with local police.

More stores are opting for biometric security, with more than a quarter of respondents in a recent survey admitting they have recently used facial recognition. It’s partly to keep their stock safe, but also to understand traffic flow and keep tabs on who visits the store.

Facial recognition technology and high-definition digital cameras can be a huge investment for stores, says Rowen, and often they are looking for further ways to use the technology, making it more cost-effective. Not only could it be used to track possible shoplifters but the tech can give stores a view into who visits their stores regularly, which demographics they fit into, or what aisles they frequent.

Almost half of the stores said they were in favour of some kind of facial recognition technology

“We talked to a large department a number of years ago that said the biggest thing that they had taken away from the cameras that they brought in to combat shrink [theft] was that they were staffing their stores in entirely the wrong places at the entirely wrong time of day,” says Rowen.

In the survey mentioned, almost half of the stores said they were in favour of some kind of facial recognition technology and only 7% believed the technology was intrusive. But that raises another issue. Should more be done to make customers aware of the watching eyes that may now be tracking them while doing their spot of shopping?

Lee believes that this is just an extension of the CCTV surveillance that we’re used to already amidst our daily lives. “If it’s used for good things and to protect them and protect businesses, I think it’s no problem,” he says. But he adds that it will only be tolerated he says as long it’s “playing within the rules”.

Too intrusive?

Privacy will remain a major concern. In September of this year the UK’s Home Office published a report encouraging greater oversight on the handling of biometric data, which includes the kind of material the facial recognition system will be studying.

With these kinds of concerns in mind, a selection of start-ups have tried to find their own way of tracking customers without gathering personal images like their face.

Netra, based in Massachusetts in the US, examines a store’s surveillance footage and picks out identifying features in both the products and customers that will help in understanding what has happened in the aftermath of an incident.

“We are extracting very abstract appearance characteristics so the privacy of the person is never compromised,” says CTO Shashi Kant. Netra’s software detects things like the colour of a person’s clothes, their hair length, or if they have a backpack or handbag.

Stores still depend on the use of cameras, which are only as good as what they can see

“In most situations we have enough appearance characteristics to differentiate between humans. It’s not just humans; what they’re carrying, for example, like backpack or a shopping cart. All these things factor into the appearance,” he says.

Prism has created a similar tool, which turns “cameras into intelligent data centres” and mines existing CCTV footage for events that occur in the store like the movement of customers or products from shelves. “We took more of a privacy-enabled approach… not necessarily saying I’m tracking Person A,” explains the company’s operational manager Bob Cutting. He says software like Prism works most efficiently when it’s used with other safeguards already in place such as RFID tags and alarms.

“All systems require some kind of visual verification,” he adds. Stores need a means to prove that a person that set off an alarm is the one that actually lifted the product and not a decoy.

Physical limitations

Whether it’s biometric scanners or more abstract tracking tools, stores still depend on the use of cameras, which are only as good as what they can see. “There’s an art form to camera placement,” says Cutting.

FaceFirst, which says it typically trains cameras on entrances rather than aisles, and it usually needs 90% accuracy in a facial scan to set off an alert but this can be adjusted. Scenarios like an airport for example would opt for a wider net, says Rosenkrantz, where the system would trigger for a lower rate of accuracy just to be on the safe side.

Facial recognition software usually requires a largely frontal view of the face as well as the eyes in order to recognise someone. “We do need a minimum amount of information,” says Herta Security’s Lee. Someone turning to the side or obscuring their appearance slightly could throw the software off. “This is a physical limitation,” says Lee.

But the expectation is for the technology to develop further and become even more nuanced in the coming years and this will attract more clients.

“Retailers are reaching out and testing more and more exotic technologies like facial recognition,” says Rosenkrantz, who claims the market for facial recognition in stores will only become more common.

Tech may be able to spot suspicious shoppers but it can let you down when it comes to policing employees that may be stealing

As this becomes more ordinary across the board, more questions will be raised too. For example, in June, pro-privacy groups in the US walked out of discussions with the Department of Commerce over creating new standards for using facial-scanning software.

In November, US retail giant Walmart revealed that it tested out facial recognition in stores, at least for several months. It found that the tech may be able to spot suspicious shoppers but it can let you down when it comes to policing employees that may be stealing.

The retail sector will continue to wrestle with the need for better security, respecting privacy, and investing smartly in technology. Getting the balance right may take some time. The likes of the jewellery shop in Rotterdam, however, hope that the future means would-be jewel thieves are very much left out in the cold.

Follow us on Facebook, TwitterGoogle+LinkedIn and Instagram.