How do game companies share massive files?

Battlefield 4 Developing Battlefield 4 proved to be a technical challenge for its publisher Electronics Arts

Electronic Arts' multi-player shooter "Battlefield 4" exploded onto the computer games scene earlier this month, giving players a realistic taste of military combat in the 21st Century.

Technology of Business

But the logistics needed to develop a complex game like Battlefield 4 were certainly not child's play.

In particular, handling the massive amounts of data that makes up the game - huge graphics files, sound clips, and the game code itself - proved to be a severe technical challenge for the California-based games developer and publisher.

That's because some of the complete game files were as large as 50GB, and future games with more advanced graphics for the new Xbox One console and Sony's PlayStation 4 are likely to be even bigger.

During development these files have to be distributed around the world, explains Steve Scivally, EA's technical director.

Start Quote

The game builds got larger and took longer. There was only one choice: transfer less content”

End Quote Steve Scivally EA's technical director

"EA is made up of a large group of development sites and a few dedicated QA (quality assurance) test centres. New game builds need to be transferred to the test centres as quickly as possible to locate defects and improve quality."

The problem was that sending a 50GB game file between sites over EA's own network often took hours - or even days.

In some cases it would have taken less time to move EA's staff to the data than to move the data to the sites where those people worked, and all this had the effect of slowing down project cycles and the overall pace of the game's development.

More selective

Solving this problem involved abandoning the use of its own computer network for large file distribution, relying instead on the internet to send data to its sites around the world from a central data repository in the cloud.

It also involved being more selective in which data was actually moved around the world, Mr Scivally explains: "The game builds got larger and took longer. There was only one choice: transfer less content."

Battlefield 4 Developers will produce different versions of their games for different games consoles

To do this EA installed devices called cloud storage controllers, supplied by a California-based company called Panzura, in each of its sites.

These cloud storage controllers download and store copies of "master" game files which live in the cloud. The clever part is what happens whenever a change is made to a file at any site anywhere in the world.

When this happens the cloud controller spots that a change has been made to its copy of the file, and transmits just this change, or "delta", back to the cloud so that the master copy can be updated.

This delta - which is often less than 5% of the entire file - is then transmitted from the cloud to all the controllers at the other sites, and their local copies are updated.

Multiple game versions
Alasdair Monk Alasdair Monk says using a system like Dropbox would be too expensive for his firm

To minimise the amount of data that needs to be sent around the world two other technologies are also used: compression - which reduced the size of a file - and something called deduplication.

Deduplication involves spotting chunks of data that are identical to other chunks of data stored at the same location, and then storing only a single copy.

It is useful for EA because the company produces versions of its games for different game platforms such as Sony's PlayStation and Microsoft's Xbox, and because each version is almost, but not quite, identical. Deduplication means that one batch of data can be transmitted to remote sites to update multiple versions of the game.

Using this system EA's developers and testers around the world can access the latest game files in a matter of minutes rather than hours, helping to ensure that Battlefield 4 was ready for release on schedule, Mr Scivally says.

Start Quote

The understanding of BitTorrent has been maturing”

End Quote Christian Averill BitTorrent director of communications

EA's story highlights a widespread problem many companies and individuals are encountering. Files often need to be shared, but increasingly the size of these files are becoming too large share using simple methods like email.

It's the reason that cloud-based services such as Dropbox and others like it have become so popular.

Files encrypted and reassembled

But for small companies that need to share large files frequently these types of services can be too expensive, and for that reason some are turning to more innovative solutions.

"I work with 50-200MB design files and I am constantly sharing stuff with our team of 15 developers," says Alasdair Monk, design director at London-based payments start up GoCardless. "Using Dropbox to do this would be far too expensive."

BitTorrent Sync graphic BitTorrent says its Sync tool is ideal for large uncompressed files

Instead he turned to BitTorrent - a system for sharing large files using peer-to-peer technology that's usually associated with pirating films, music and video games.

BitTorrent offers a free product called BitTorrent Sync, which sends information between users' PCs and eliminates the need to upload data to a third party's computer servers. Mr Monk uses it to make large design files stored on his computer available to GoCardless's developers very rapidly, even if they are working on a laptop from home.

It works quickly because files are split up into encrypted chunks and sent out to multiple BitTorrent Sync users within the company.

When someone else needs one of Mr Monk's files, it is reassembled from chunks stored on some of these users' computers - whichever ones can supply the chunks most quickly. "Even when developers or I am are out on the road, the speed that they can download one of my large files is phenomenal," he says.

Reliability of cloud storage?

Christian Averill, BitTorrent's director of communications, says over one million people are using BitTorrent Sync, and that the association that many people have between BitTorrent and film piracy seems not to have been an issue.

"The understanding of BitTorrent has been maturing and most people know that we have no affiliation with such activity," he says.

He adds that a system like BitTorrent Sync means that there is no need to store data in the cloud, which can be a benefit for companies in some industries that are subject to strict regulatory requirements

There's also the fact that cloud storage is only as reliable as the companies that operate it - something that EA learned the hard way.

Its master game files used to be held in cloud storage supplied by a company called Nirvanix, but in September the company suddenly announced that it would be closing down at the end of the month.

That left EA - along with all Nirvanix's other customers - scrambling to retrieve its cloud data and move it to another cloud before the Nirvanix storage was switched off for good.

More on This Story

The BBC is not responsible for the content of external Internet sites

More Business stories

RSS

BBC Business Live

  1.  
    INTEREST RATES 07:31:

    Bank of England governor Mark Carney is in all the newspapers today following his speech to the Commonwealth business conference yesterday. The FT interprets Mr Carney's speech as revealing deep concerns about household indebtedness and the effect an interest rate rise will have on households' ability to service their debts.

     
  2.  
    UNILEVER RESULTS 07:23:

    Unilever, the food giant, has said first half net profit rose to 3bn euros (£2.37bn) from 2.68bn euros even as sales fell. Its cost savings plan is working well, it said. Excluding businesses sold or otherwise disposed of, sales rose 3.7%.

     
  3.  
     
  4.  
    WINTER STORM COMPENSATION 07:19:
    SSE logo at the SSE Training Centre in Perth

    SSE and UK Power Networks (IKPN) are to pay an additional £3.3m for delays in getting power back up and running to households in southern England following last winter's storms. That's on top of the £4.7m the two energy companies are paying out to customers under guaranteed standards and in goodwill payments. It brings the total paid out by the two companies to £8m.

     
  5.  
    ENERGY PROBE 07:04: Breaking News

    The Competition and Markets Authority (CMA) will carry out an independent investigation to see if there are any features of the energy market which "prevent, restrict or distort competition and, if so, what action might be taken to remedy them," it says. It will report back by Christmas day 2015.

     
  6.  
    INTEREST RATES 06:56: Radio 5 live

    Kathleen Brooks of FOREX.com thinks January will be the time for a rate increase from the Bank of England. Maybe a 25 basis point increase to 0.75%, she says. Bank governor, Mark Carney, said yesterday there may be more labour supply than previously thought. Wages growth is thus being held back.

     
  7.  
    RUSSIAN SANCTIONS 06:44: BBC Radio 4
    Russian President Vladimir Putin

    Kathleen Brooks has now popped up on the Today programme. She says sanctions against Russia "are toothless at this point....Russia has a lot of energy that the West wants." She adds divisions in the West over energy policy have been exposed by the fact "the UK and France have a spat about sanctions and that just plays into Vladimir Putin's hands." Ms Brooks says there have "also been rumours that Russia has been unofficially propping up the rouble. The sanctions are not going to hurt the European Union's energy links to Russia."

     
  8.  
    ENERGY PRICES 06:31: Radio 5 live

    If you have to steel yourself with a nice cup of sweet tea before opening your latest energy bill, spare a thought for Nigel and Linda Brotherton, who were told by Npower that their monthly direct debit was going to increase from £87 to £53,480,062. Npower has apologised and the incorrect bill has been cancelled. "It's a massive shock," he tells 5 live.

     
  9.  
    GLAXOSMITHKLINE 06:24: Radio 5 live
    gsk

    David Buik of Panmure Gordon is helping us digest the GlaxoSmithKline results on 5 live. "Sales to China is only about 3% of its total business," and so the bribery scandal there is an unwelcome distraction.

     
  10.  
    ROUBLE 06:15: Radio 5 live

    Kathleen Brooks of FOREX.com is on 5 live talking about Russia. The Russia rouble has been "resilient" bearing in mind the new sanctions against Russia. "They are able to shrug off geopolitical risk quite quickly," she says. President Putin may be "breathing a sigh of relief," she adds.

     
  11.  
    POUND 06:01: Radio 5 live

    The pound buys about 1.27 euros, its strongest in nearly two years - don't get excited that's not what you'll get from your local foreign exchange. Still, Kathleen Brooks, a research director at FOREX.com, is on 5 live talking about it. The strong pound has helped ease inflation, she says. "Over the period of the financial crisis we had sky high inflation," she adds. The recovery is consumer-driven, which is aided by a stronger pound, which means cheaper imports.

     
  12.  
    06:00: Howard Mustoe Business reporter

    Good morning! Get in touch via email at bizlivepage@bbc.co.uk or on twitter @BBCBusiness

     
  13.  
    06:00: Matthew West Business Reporter

    Good morning folks. So what's new? Facebook has posted some stonking profits for its second quarter, the US has - just - lifted its ban on flights into Tel Aviv and it's Thursday, so General Motors has decided to recall another couple of hundred thousand cars overnight. There's plenty more to come, stay with us.

     

Features

  • Shinji Mikamo's father's watchTime peace

    The story of the watch that survived Hiroshima


  • Hamas rally in the West Bank village of Yatta, 2006Hamas hopes

    Why the Palestinian group won't back down yet


  • Mary WayaQueen of the court

    Woman who beat "fat" jibes to turn Malawi into netball champions


From BBC Capital

Programmes

  • A man holds a sign which reads Bring Back Our GirlsHARDtalk Watch

    Why there is still hope and optimism for the rescue of Nigeria’s kidnapped schoolgirls

BBC © 2014 The BBC is not responsible for the content of external sites. Read more.

This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.