How do game companies share massive files?

Battlefield 4 Developing Battlefield 4 proved to be a technical challenge for its publisher Electronics Arts

Electronic Arts' multi-player shooter "Battlefield 4" exploded onto the computer games scene earlier this month, giving players a realistic taste of military combat in the 21st Century.

Technology of Business

But the logistics needed to develop a complex game like Battlefield 4 were certainly not child's play.

In particular, handling the massive amounts of data that makes up the game - huge graphics files, sound clips, and the game code itself - proved to be a severe technical challenge for the California-based games developer and publisher.

That's because some of the complete game files were as large as 50GB, and future games with more advanced graphics for the new Xbox One console and Sony's PlayStation 4 are likely to be even bigger.

During development these files have to be distributed around the world, explains Steve Scivally, EA's technical director.

Start Quote

The game builds got larger and took longer. There was only one choice: transfer less content”

End Quote Steve Scivally EA's technical director

"EA is made up of a large group of development sites and a few dedicated QA (quality assurance) test centres. New game builds need to be transferred to the test centres as quickly as possible to locate defects and improve quality."

The problem was that sending a 50GB game file between sites over EA's own network often took hours - or even days.

In some cases it would have taken less time to move EA's staff to the data than to move the data to the sites where those people worked, and all this had the effect of slowing down project cycles and the overall pace of the game's development.

More selective

Solving this problem involved abandoning the use of its own computer network for large file distribution, relying instead on the internet to send data to its sites around the world from a central data repository in the cloud.

It also involved being more selective in which data was actually moved around the world, Mr Scivally explains: "The game builds got larger and took longer. There was only one choice: transfer less content."

Battlefield 4 Developers will produce different versions of their games for different games consoles

To do this EA installed devices called cloud storage controllers, supplied by a California-based company called Panzura, in each of its sites.

These cloud storage controllers download and store copies of "master" game files which live in the cloud. The clever part is what happens whenever a change is made to a file at any site anywhere in the world.

When this happens the cloud controller spots that a change has been made to its copy of the file, and transmits just this change, or "delta", back to the cloud so that the master copy can be updated.

This delta - which is often less than 5% of the entire file - is then transmitted from the cloud to all the controllers at the other sites, and their local copies are updated.

Multiple game versions
Alasdair Monk Alasdair Monk says using a system like Dropbox would be too expensive for his firm

To minimise the amount of data that needs to be sent around the world two other technologies are also used: compression - which reduced the size of a file - and something called deduplication.

Deduplication involves spotting chunks of data that are identical to other chunks of data stored at the same location, and then storing only a single copy.

It is useful for EA because the company produces versions of its games for different game platforms such as Sony's PlayStation and Microsoft's Xbox, and because each version is almost, but not quite, identical. Deduplication means that one batch of data can be transmitted to remote sites to update multiple versions of the game.

Using this system EA's developers and testers around the world can access the latest game files in a matter of minutes rather than hours, helping to ensure that Battlefield 4 was ready for release on schedule, Mr Scivally says.

Start Quote

The understanding of BitTorrent has been maturing”

End Quote Christian Averill BitTorrent director of communications

EA's story highlights a widespread problem many companies and individuals are encountering. Files often need to be shared, but increasingly the size of these files are becoming too large share using simple methods like email.

It's the reason that cloud-based services such as Dropbox and others like it have become so popular.

Files encrypted and reassembled

But for small companies that need to share large files frequently these types of services can be too expensive, and for that reason some are turning to more innovative solutions.

"I work with 50-200MB design files and I am constantly sharing stuff with our team of 15 developers," says Alasdair Monk, design director at London-based payments start up GoCardless. "Using Dropbox to do this would be far too expensive."

BitTorrent Sync graphic BitTorrent says its Sync tool is ideal for large uncompressed files

Instead he turned to BitTorrent - a system for sharing large files using peer-to-peer technology that's usually associated with pirating films, music and video games.

BitTorrent offers a free product called BitTorrent Sync, which sends information between users' PCs and eliminates the need to upload data to a third party's computer servers. Mr Monk uses it to make large design files stored on his computer available to GoCardless's developers very rapidly, even if they are working on a laptop from home.

It works quickly because files are split up into encrypted chunks and sent out to multiple BitTorrent Sync users within the company.

When someone else needs one of Mr Monk's files, it is reassembled from chunks stored on some of these users' computers - whichever ones can supply the chunks most quickly. "Even when developers or I am are out on the road, the speed that they can download one of my large files is phenomenal," he says.

Reliability of cloud storage?

Christian Averill, BitTorrent's director of communications, says over one million people are using BitTorrent Sync, and that the association that many people have between BitTorrent and film piracy seems not to have been an issue.

"The understanding of BitTorrent has been maturing and most people know that we have no affiliation with such activity," he says.

He adds that a system like BitTorrent Sync means that there is no need to store data in the cloud, which can be a benefit for companies in some industries that are subject to strict regulatory requirements

There's also the fact that cloud storage is only as reliable as the companies that operate it - something that EA learned the hard way.

Its master game files used to be held in cloud storage supplied by a company called Nirvanix, but in September the company suddenly announced that it would be closing down at the end of the month.

That left EA - along with all Nirvanix's other customers - scrambling to retrieve its cloud data and move it to another cloud before the Nirvanix storage was switched off for good.

More on This Story

The BBC is not responsible for the content of external Internet sites

More Business stories

RSS

BBC Business Live

  1.  
    CREDIT SUISSE PROBE 06:38:
    Banking logo

    In case you missed this, Credit Suisse is investigating two of its traders in London over allegations of misconduct. According to reports, two employees on its European equities sales team are under scrutiny over their electronic communications. One trader is said to have improperly shared client information with her husband, who works at a rival bank.

     
  2.  
    THE POOND 06:30: Radio 5 live

    Brenda Kelly, chief market strategist at IG is now talking on Wake Up to Money about Scottish independence and that poll showing a gain for "Yes". "The news that the gap between the yes and no votes from YouGov has narrowed had a downward effect on the pound," she says. Her company estimates there's a mere 20% chance of independence.

     
  3.  
    Via Twitter Adam Parsons Business Correspondent

    tweets: "Andrew Tyrie: customers' confidence in banks "has been badly shaken...and will take years to fix. We haven't solved all these problems" #WUTM"

     
  4.  
    SUPERMARKET PERFORMANCE 06:11: Radio 5 live
    Tesco trolleys

    Brenda Kelly, chief market strategist at IG, is talking about supermarkets on Wake Up to Money. "Aldi and Lidl are really starting to take market share."Tesco's profit warning has led all supermarkets' share prices lower, she says as the discounters gain market share.

     
  5.  
    BANK REFORM 06:05: Radio 5 live

    Wake up to Money has Andrew Tyrie, the chairman of the House of Commons Treasury Committee and scourge of the bank bosses. He tells the programme: "The [banks'] spirit is willing often at the top, but sometimes the flesh is weak. These are very large institutions and they are finding it difficult to get the trickle-down" of new management ideas."

     
  6.  
    06:02: Howard Mustoe Business reporter

    Good morning! Get in touch wit us via email bizlivepage@bbc.co.uk or on twitter @BBCBusiness

     
  7.  
    06:01: Rebecca Marston Business reporter, BBC News

    Good morning. Sex and drugs and GDP. An unlikely lyric but it is going to be news. Officially, the 2012 economic figures will include income made on the black market, so we'll have those when they're released later. And more. Oh yes.

     

Features

From BBC Capital

Programmes

  • A tankHARDtalk Watch

    The West looks 'really weak' against a 'power drunk' Russia, says a senior Ukrainian diplomat

BBC © 2014 The BBC is not responsible for the content of external sites. Read more.

This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.