How do game companies share massive files?

Battlefield 4 Developing Battlefield 4 proved to be a technical challenge for its publisher Electronics Arts

Electronic Arts' multi-player shooter "Battlefield 4" exploded onto the computer games scene earlier this month, giving players a realistic taste of military combat in the 21st Century.

Technology of Business

But the logistics needed to develop a complex game like Battlefield 4 were certainly not child's play.

In particular, handling the massive amounts of data that makes up the game - huge graphics files, sound clips, and the game code itself - proved to be a severe technical challenge for the California-based games developer and publisher.

That's because some of the complete game files were as large as 50GB, and future games with more advanced graphics for the new Xbox One console and Sony's PlayStation 4 are likely to be even bigger.

During development these files have to be distributed around the world, explains Steve Scivally, EA's technical director.

Start Quote

The game builds got larger and took longer. There was only one choice: transfer less content”

End Quote Steve Scivally EA's technical director

"EA is made up of a large group of development sites and a few dedicated QA (quality assurance) test centres. New game builds need to be transferred to the test centres as quickly as possible to locate defects and improve quality."

The problem was that sending a 50GB game file between sites over EA's own network often took hours - or even days.

In some cases it would have taken less time to move EA's staff to the data than to move the data to the sites where those people worked, and all this had the effect of slowing down project cycles and the overall pace of the game's development.

More selective

Solving this problem involved abandoning the use of its own computer network for large file distribution, relying instead on the internet to send data to its sites around the world from a central data repository in the cloud.

It also involved being more selective in which data was actually moved around the world, Mr Scivally explains: "The game builds got larger and took longer. There was only one choice: transfer less content."

Battlefield 4 Developers will produce different versions of their games for different games consoles

To do this EA installed devices called cloud storage controllers, supplied by a California-based company called Panzura, in each of its sites.

These cloud storage controllers download and store copies of "master" game files which live in the cloud. The clever part is what happens whenever a change is made to a file at any site anywhere in the world.

When this happens the cloud controller spots that a change has been made to its copy of the file, and transmits just this change, or "delta", back to the cloud so that the master copy can be updated.

This delta - which is often less than 5% of the entire file - is then transmitted from the cloud to all the controllers at the other sites, and their local copies are updated.

Multiple game versions
Alasdair Monk Alasdair Monk says using a system like Dropbox would be too expensive for his firm

To minimise the amount of data that needs to be sent around the world two other technologies are also used: compression - which reduced the size of a file - and something called deduplication.

Deduplication involves spotting chunks of data that are identical to other chunks of data stored at the same location, and then storing only a single copy.

It is useful for EA because the company produces versions of its games for different game platforms such as Sony's PlayStation and Microsoft's Xbox, and because each version is almost, but not quite, identical. Deduplication means that one batch of data can be transmitted to remote sites to update multiple versions of the game.

Using this system EA's developers and testers around the world can access the latest game files in a matter of minutes rather than hours, helping to ensure that Battlefield 4 was ready for release on schedule, Mr Scivally says.

Start Quote

The understanding of BitTorrent has been maturing”

End Quote Christian Averill BitTorrent director of communications

EA's story highlights a widespread problem many companies and individuals are encountering. Files often need to be shared, but increasingly the size of these files are becoming too large share using simple methods like email.

It's the reason that cloud-based services such as Dropbox and others like it have become so popular.

Files encrypted and reassembled

But for small companies that need to share large files frequently these types of services can be too expensive, and for that reason some are turning to more innovative solutions.

"I work with 50-200MB design files and I am constantly sharing stuff with our team of 15 developers," says Alasdair Monk, design director at London-based payments start up GoCardless. "Using Dropbox to do this would be far too expensive."

BitTorrent Sync graphic BitTorrent says its Sync tool is ideal for large uncompressed files

Instead he turned to BitTorrent - a system for sharing large files using peer-to-peer technology that's usually associated with pirating films, music and video games.

BitTorrent offers a free product called BitTorrent Sync, which sends information between users' PCs and eliminates the need to upload data to a third party's computer servers. Mr Monk uses it to make large design files stored on his computer available to GoCardless's developers very rapidly, even if they are working on a laptop from home.

It works quickly because files are split up into encrypted chunks and sent out to multiple BitTorrent Sync users within the company.

When someone else needs one of Mr Monk's files, it is reassembled from chunks stored on some of these users' computers - whichever ones can supply the chunks most quickly. "Even when developers or I am are out on the road, the speed that they can download one of my large files is phenomenal," he says.

Reliability of cloud storage?

Christian Averill, BitTorrent's director of communications, says over one million people are using BitTorrent Sync, and that the association that many people have between BitTorrent and film piracy seems not to have been an issue.

"The understanding of BitTorrent has been maturing and most people know that we have no affiliation with such activity," he says.

He adds that a system like BitTorrent Sync means that there is no need to store data in the cloud, which can be a benefit for companies in some industries that are subject to strict regulatory requirements

There's also the fact that cloud storage is only as reliable as the companies that operate it - something that EA learned the hard way.

Its master game files used to be held in cloud storage supplied by a company called Nirvanix, but in September the company suddenly announced that it would be closing down at the end of the month.

That left EA - along with all Nirvanix's other customers - scrambling to retrieve its cloud data and move it to another cloud before the Nirvanix storage was switched off for good.

More on This Story

The BBC is not responsible for the content of external Internet sites

More Business stories

RSS

BBC Business Live

  1.  
    09:07: SPIRIT PUBS TAKEOVER

    More analysis on Greene King and C&C's offers for the Spirit pub chain. "The really interesting development, in our view, would be for all three businesses to combine. The result would be a leading pubco and a brewer, cider maker and distributor with a much stronger portfolio of brands," say analysts at City broker Canaccord Genuity this morning. "C&C also own Tennents, the leading Scottish lager. Belhaven (Greene King's Scottish estate) is a leading customer."

     
  2.  
    08:51: BANK OF DAVE Radio 5 live

    Dave Fishwick, the minibus salesman who founded the "Bank of Dave" is on Radio 5 live. What does the banking market need? "What we desperately need out there is challenger banks," he says. What's also needed is "tighter control of the bigger banks to prevent greed and corruption," he adds.

     
  3.  
    08:36: MARKET REPORT

    The fallout from Tesco's results yesterday continues today. The supermarket is the biggest faller on the FTSE 100 Index so far this morning down 2.3% to 167.05p. European markets are down generally. The FTSE 100 is down 0.45% to 6390.17, Germany's Dax is down 0.40% to 9011.29 and France's Cac-40 has fallen 050% to 4136.95.

     
  4.  
    08:23: PEARSON EARNINGS
    penguin

    Pearson's chief executive John Fallon says the firm's £50m cost-cutting programme is on track and "momentum in digital, services and emerging market education is building, which will drive a leaner, more cash generative, faster growing business from 2015."

     
  5.  
    08:06: SPIRIT PUBS TAKEOVER

    Could this be the beginning of a bidding war? The pub chain has said it has rejected a takeover proposal from Irish cider maker C&C Group today. Spirit is already in talks with brewer and pub owner Greene King about its proposed £723m takeover offer. C&C, the maker of Magners and Bulmers, has until 20 November to announce a firm takeover offer.

     
  6.  
    07:51: PEARSON EARNINGS

    Pearson has reported flat underlying revenue for nine months to the end of September and a 1% fall in in what it calls headline growth for the period. It blames the strength of sterling against key emerging market currencies for the fall. Penguin Random House has performed well in the third quarter, it adds without giving detail. It says the integration of its businesses is "progressing well and is on track to deliver benefits in 2015 and beyond".

     
  7.  
    07:38: HIKMA WARNING

    Hikma Pharmaceuticals says it has received a warning letter US Food and Drug Administration after an inspection at its manufacturing plant in Portugal. "In the letter, the agency raised issues related to investigations and environmental monitoring at the facility," said the firm, which is taking the letter "very seriously."

     
  8.  
    07:26: TSB EARNINGS
    The TSB logo

    Impairments - that is, bad loans - fell to £23m from £32.2m, said TSB. Loans rose 7.7% to £22bn compared to a year ago, but fell from a peak of £23bn six months ago. TSB won 9.7% of all new or switched bank accounts, it said, adding £500m of deposits.

     
  9.  
    07:18: PEARSON FINANCE CHIEF STEPS DOWN

    Robin Freestone, chief financial officer of Financial Times and Penguin Random House owner Pearson has announced he is standing down after 10 years with the firm, including eight in his current role. He will probably leave the firm in 2015 after a successor has been found, said the firm.

     
  10.  
    07:08: TSB EARNINGS

    TSB third-quarter profit before tax fell 14% to £33.1m compared with the same time a year ago, after operating expenses rose. But revenue swelled 18% to £199m

     
  11.  
    06:54: EU PAYMENT Radio 5 live

    Sarah Hewin of Standard Chartered on 5 live says the payment has to be made in the next few months. That could mean more borrowing, she says.

     
  12.  
    06:41: EU PAYMENT Radio 5 live
    British Prime Minister David Cameron

    Sarah Hewin of Standard Chartered is explaining why the UK has to pay an extra £1.7bn to the EU on 5 live. "The UK has been doing better since 1997 than we thought and that's resulted in this extra payment. The Netherlands will pay more, while France and Germany get a rebate."

     
  13.  
    06:29: AMAZON RESULTS Radio 5 live

    Paul Kavanagh of wealth manager Killik is talking about Amazon's loss-making results last night. "It begs the question about what is happening here with this strategy. The shares fell 11% in after hours [in the US]." Investors may be growing tired of ever-more sales expansion with little profit to show for it, he tells 5 live.

     
  14.  
    06:20: CHALLENGER BANKS Radio 5 live

    Paul Kavanagh of wealth manager Killik says it's difficult for banks to persuade customers they offer something new. When a challenger bank succeeds, the larger banks often take the best ideas, he says on 5 live.

     
  15.  
    06:12: CHALLENGER BANKS Radio 5 live

    Steve Davies is still on 5 live. He says challenger banks are forcing their larger competitors to think more about the customer and service - think Metro bank opening on Sundays. Competing on rates is more difficult, he says. TSB results coming up later.

     
  16.  
    06:03: CHALLENGER BANKS Radio 5 live

    Steve Davies of accountants PwC is on 5 live talking about so-called challenger banks. Can they challenge the largest high street banks? "They have to be able to offer something a little bit different," he says. "The challenge is around innovation," he says. Customer is key, he adds.

     
  17.  
    06:00: Howard Mustoe Business reporter

    Good morning. Get in touch via email blizlivepage@bbc.co.uk and twitter @BBCBusiness.

     
  18.  
    06:00: Matthew West Business reporter

    Morning folks. It's Friday, we're nearly at the weekend. But before that, TSB kicks off bank earnings season and Shire has financials out as well. There's also some service sector data but the big bit of data is the first estimate of third quarter GDP. We'll bring you it all as it happens as always.

     

Features

From BBC Capital

Programmes

  • The smartphone that answers backClick Watch

    Smartphones get smarter – the prototypes that talk and say ouch when you drop them

BBC © 2014 The BBC is not responsible for the content of external sites. Read more.

This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.