“Media is full of these facts about the future,” suggests Ahlberg. “So, what we asked ourselves is could we build a machine – we used to jokingly call it a DVR for everything spoken and written about the future – and organise that data meticulously into a dataset, and set it up as a cool user experience?”
The company did just that, and today, Recorded Future’s software is being used by private customers and government clients interested in data-mining the future.
One example of how Recorded Future’s technology could be used, according to Ahlberg, might be someone following pharmaceutical stocks. The user clicks on a specific date, and Recorded Future shows all of the data mined from public information about things that are supposed to take place on that date, such as a review by the US Food and Drug Administration, the release of a new drug, or the expiration of a drug patent.
All this information can hold vital clues for those in the know. And it could be a step towards predictive analytics’ ultimate prize: picking winning stocks. But Ahlberg and other researchers know this is a long way off. “Sometimes I find people want magic predictions,” Ahlberg says. “That doesn’t reflect reality.”
Others agree this is still a tricky area. Gloor, who has used his models to try to outsmart the markets, has so far had only limited success. However, the work has thrown up at least one intriguing result: the models work best with alternative energy stocks. “There we have the tree-huggers, and tree-huggers are honest,” he says. “They talk about new developments that correspond to alternative energy.”
Chen also concedes that financial markets are harder to predict than movies. “It’s not easy to predict a stock return: You can predict the movement, volume and volatility,” he says. “The return is still the Holy Grail.”
Even if such analysis cannot always make precise forecasts, its potential for forecasting trends and events has increasingly attracted interest from the US government. Recorded Future, whose technology can also mine social media to forecast political protests, such as Occupy Wall Street, or track criminal activity, such as cyber attacks, has already attracted interest from the national security community. In-Q-Tel, the venture capital firm founded by the CIA, has invested in the company.
In fact, the intelligence community’s interest in predictive modelling, particularly based on social media, has been growing over the past few years, especially in light of the Arab spring protests, which were at least partly fuelled by social media. Last year, the Intelligence Advanced Research Projects Activity (Iarpa), a research and development arm of the US intelligence community, launched a project called Open Source Indicators, designed to mine information from social media and other public data and to come up with predictions.
The Pentagon has also launched a number of forecasting projects in recent years, hoping, for example, to predict insurgent behaviour. Mark Maybury, the US Air Force chief scientist, likens this sort of human-data collection – whether from social media, foreign news, or elsewhere – as something akin to the images collected by drones flying over Afghanistan. Only instead of information about bombs, it is collecting information about how people behave.
“At the strategic national level, you’d like to do things like predict state failure,” said Maybury of the modelling work the Air Force is doing. “More tactically, you’d like to be able to do things like discover illicit shipping routes, human trafficking routes, and narcotics routes.”