How do game companies share massive files?

Battlefield 4 Developing Battlefield 4 proved to be a technical challenge for its publisher Electronics Arts

Electronic Arts' multi-player shooter "Battlefield 4" exploded onto the computer games scene earlier this month, giving players a realistic taste of military combat in the 21st Century.

Technology of Business

But the logistics needed to develop a complex game like Battlefield 4 were certainly not child's play.

In particular, handling the massive amounts of data that makes up the game - huge graphics files, sound clips, and the game code itself - proved to be a severe technical challenge for the California-based games developer and publisher.

That's because some of the complete game files were as large as 50GB, and future games with more advanced graphics for the new Xbox One console and Sony's PlayStation 4 are likely to be even bigger.

During development these files have to be distributed around the world, explains Steve Scivally, EA's technical director.

Start Quote

The game builds got larger and took longer. There was only one choice: transfer less content”

End Quote Steve Scivally EA's technical director

"EA is made up of a large group of development sites and a few dedicated QA (quality assurance) test centres. New game builds need to be transferred to the test centres as quickly as possible to locate defects and improve quality."

The problem was that sending a 50GB game file between sites over EA's own network often took hours - or even days.

In some cases it would have taken less time to move EA's staff to the data than to move the data to the sites where those people worked, and all this had the effect of slowing down project cycles and the overall pace of the game's development.

More selective

Solving this problem involved abandoning the use of its own computer network for large file distribution, relying instead on the internet to send data to its sites around the world from a central data repository in the cloud.

It also involved being more selective in which data was actually moved around the world, Mr Scivally explains: "The game builds got larger and took longer. There was only one choice: transfer less content."

Battlefield 4 Developers will produce different versions of their games for different games consoles

To do this EA installed devices called cloud storage controllers, supplied by a California-based company called Panzura, in each of its sites.

These cloud storage controllers download and store copies of "master" game files which live in the cloud. The clever part is what happens whenever a change is made to a file at any site anywhere in the world.

When this happens the cloud controller spots that a change has been made to its copy of the file, and transmits just this change, or "delta", back to the cloud so that the master copy can be updated.

This delta - which is often less than 5% of the entire file - is then transmitted from the cloud to all the controllers at the other sites, and their local copies are updated.

Multiple game versions
Alasdair Monk Alasdair Monk says using a system like Dropbox would be too expensive for his firm

To minimise the amount of data that needs to be sent around the world two other technologies are also used: compression - which reduced the size of a file - and something called deduplication.

Deduplication involves spotting chunks of data that are identical to other chunks of data stored at the same location, and then storing only a single copy.

It is useful for EA because the company produces versions of its games for different game platforms such as Sony's PlayStation and Microsoft's Xbox, and because each version is almost, but not quite, identical. Deduplication means that one batch of data can be transmitted to remote sites to update multiple versions of the game.

Using this system EA's developers and testers around the world can access the latest game files in a matter of minutes rather than hours, helping to ensure that Battlefield 4 was ready for release on schedule, Mr Scivally says.

Start Quote

The understanding of BitTorrent has been maturing”

End Quote Christian Averill BitTorrent director of communications

EA's story highlights a widespread problem many companies and individuals are encountering. Files often need to be shared, but increasingly the size of these files are becoming too large share using simple methods like email.

It's the reason that cloud-based services such as Dropbox and others like it have become so popular.

Files encrypted and reassembled

But for small companies that need to share large files frequently these types of services can be too expensive, and for that reason some are turning to more innovative solutions.

"I work with 50-200MB design files and I am constantly sharing stuff with our team of 15 developers," says Alasdair Monk, design director at London-based payments start up GoCardless. "Using Dropbox to do this would be far too expensive."

BitTorrent Sync graphic BitTorrent says its Sync tool is ideal for large uncompressed files

Instead he turned to BitTorrent - a system for sharing large files using peer-to-peer technology that's usually associated with pirating films, music and video games.

BitTorrent offers a free product called BitTorrent Sync, which sends information between users' PCs and eliminates the need to upload data to a third party's computer servers. Mr Monk uses it to make large design files stored on his computer available to GoCardless's developers very rapidly, even if they are working on a laptop from home.

It works quickly because files are split up into encrypted chunks and sent out to multiple BitTorrent Sync users within the company.

When someone else needs one of Mr Monk's files, it is reassembled from chunks stored on some of these users' computers - whichever ones can supply the chunks most quickly. "Even when developers or I am are out on the road, the speed that they can download one of my large files is phenomenal," he says.

Reliability of cloud storage?

Christian Averill, BitTorrent's director of communications, says over one million people are using BitTorrent Sync, and that the association that many people have between BitTorrent and film piracy seems not to have been an issue.

"The understanding of BitTorrent has been maturing and most people know that we have no affiliation with such activity," he says.

He adds that a system like BitTorrent Sync means that there is no need to store data in the cloud, which can be a benefit for companies in some industries that are subject to strict regulatory requirements

There's also the fact that cloud storage is only as reliable as the companies that operate it - something that EA learned the hard way.

Its master game files used to be held in cloud storage supplied by a company called Nirvanix, but in September the company suddenly announced that it would be closing down at the end of the month.

That left EA - along with all Nirvanix's other customers - scrambling to retrieve its cloud data and move it to another cloud before the Nirvanix storage was switched off for good.

More on This Story

The BBC is not responsible for the content of external Internet sites

More Business stories

RSS

BBC Business Live

  1.  
    PETS PROFITS 12:18:
    Chart showing Pets ad Home's share price

    Pets at Home's share price is nearly 6% higher to 180.10p, following its first quarter trading update this morning. But it's worth pointing out that investors who bought on its first full day of trading on the Stock Exchange on 12 March this year are still well out of pocket. The Pets retailer was valued at 245p a share or £1.3bn in March. It is valued at just under £1bn at today's share price.

     
  2.  
    PEUGOEOT RECOVERS 11:56:
    Carlos Tavares, new chief executive of PSA Peugeot Citroen, April 2014

    In case Peugeot's troubles passed you by, in April this year the car manufacturer announced a major recovery plan after recording net losses of 7.2bn euros ($10bn; £5.9bn) over the previous two years. The plan included cutting costs and slashing the numbers of models in its range from 45 to 26.

     
  3.  
    AIRBUS PROFITS TAKE OFF 11:40:
    Model of an Airbus A350 passenger

    European aerospace company Airbus expects its revenue to remain stable this year despite a big cancellation recently for some of its large A350 planes. The company says revenue rose 7% in the second quarter from the previous year, to 14.6bn euros ($19.5bn: £11.5bn). That helped boost its net income (profit) to 696m euros from 531m euros. In June Emirates airline cancelled an order for 70 A350s, opting for competitor Boeing instead.

     
  4.  
    PEUGEOT RECOVERS 11:25:

    French car marker Peugeot Citroen has narrowed its first-half losses and posted the first positive contribution from its core motoring division in three years. It has narrowed its net loss to 114m euros (£90m) from 471m euros a year earlier. The troubled car maker says its recovery plan is already producing results on all fronts, chief financial officer Jean Baptiste de Chatillon said.

     
  5.  
    SHAGGY DOG TALE 11:15:
    People patting dog

    Way too much good stuff in this to summarise: The story of the railway collection dogs who from Victorian times to the 1950s went about the railways charming passengers into giving them money. Have to pick this out though: "Some dogs were less than honest... [it was] discovered in the 1860s that Brighton Bob was using some of his money to buy biscuits at a bakery."

     
  6.  
    SPANISH GDP 10:59:

    Spain's economy grow at 0.6% in the second quarter, according to official figures, up from 0.4% in the first three months of the year. Yesterday the government raised its growth forecast for the year, saying it would be close to 1.5% this year and could reach 2% next year. The reading was the fastest quarterly rise since the last three months of 2007.

     
  7.  
    ITV PROFITS 10:45:
    ITV still

    ITV's results proclaim the company is now the biggest unscripted independent production company in the US after buying 80% of Leftfield Entertainment. The studio produces reality programmes such as Pawn Stars, Counting Cars, American Restoration and Real Housewives of New Jersey.

     
  8.  
    TAX CREDIT EXTENSION 10:30: Via Email Kevin Peachey Personal finance reporter, BBC News

    "HMRC says deadline for hundreds of thousands of people to renew their tax credits has been extended from 31 July to 6 August, owing to PCS strike action"

     
  9.  
    BANKER BONUSES 10:17:

    Swift reaction to the new plans from the Bank on bonuses. John Cridland, CBI director general, says, broadly, it's a good idea to align performance, behaviour and pay, but: "As these new rules are amongst the toughest in world, we need to be careful we don't create uncertainty which might make it increasingly hard to attract talent to London."

     
  10.  
    BANKER BONUSES 10:13:

    The PRA and the FCA (Financial Conduct Authority) are consulting on the bonus proposals. Martin Wheatley, head of the FCA said: "How a firm conducts its business and treats its customers must be at the heart of how it operates. This has to start at the top. Today's consultations mark a fundamental change in the regulators' ability to hold individuals to account, which is what the public expects of us. It will also build on the cultural change we are beginning to see in the boardrooms of firms across the country."

     
  11.  
    BANKER BONUSES 10:10:

    More: "The PRA [Prudential Regulation Authority] has also today published final rules on clawback which introduce a seven-year minimum period for clawback from the date of award. These rules will come into force on 1 January 2015."

     
  12.  
    BANKER BONUSES 10:07:

    More from the Bank of England's plans to improve bankers' accountability. Some are proposals and the Bank is consulting on them. They are not all new, firm, rules.

     
  13.  
    BANKER BONUSES 10:05:
    Bank of England

    The Bank of England's plans for making bankers more accountable is now released. It says "Increasing the alignment between risk and reward over the longer term, by requiring firms to defer payment of variable remuneration (e.g. bonuses) for a minimum of five or seven years depending on seniority, with a phased approach to vesting."

     
  14.  
    TURTLE TROUBLE 09:53:
    Teenage Mutant Ninja Turtles

    The new Teenage Mutant Ninja Turtles movie in Australia has caused more excitement than usual. The poster shows the turtles jumping from an exploding skyscraper, standard for an action movie, nor necessarily that offensive. That is until you look at the release date at the bottom of the poster. Paramount Pictures has apologised: "We are deeply sorry to have used that artwork."

     
  15.  
    RUSSIA SANCTIONS 09:35:

    Credit and debit card company Visa is one that isn't bothered by tightening sanctions on Russia. "The new package of US economic sanctions is not influencing the operations of Visa in Russia and does not force Visa to halt or block operations of financial institutions who have fallen under the sanctions," it said in a statement. "We are continuing to process transactions in a normal way."

     
  16.  
    NINTENDO LOSS 09:20:
    Nintendo characters

    Videogames giant Nintendo reports a £57m net loss for the April-June quarter. Higher costs are to blame. Sales were down by 8.4%.

     
  17.  
    BARCLAYS PROFITS 09:03:

    Barclays Bank is currently top of the risers on the FTSE 100. Its share price is up 3.3% at 226.30p, so it looks like investors are responding positively to its half year results. They've a way to make up though. Shares are about 30% down compared with last year. Earlier, boss Antony Jenkins told Today: "We are ahead of the targets we have set and in the next few quarters the market will reflect this in the share price." It's made a start.

     
  18.  
    PETS PROFITS 08:50:
    People and pets

    Stock market debutant Pets at Home has released its first trading update. It covers the 12 weeks to 17 July. Like-for-like sales are up 4.1%. Total revenue is 10.4% higher at £210.8m, driven, it says, by new store openings and strong food, accessories and services trade.

     
  19.  
    MARKETS UPDATE 08:41:

    The markets aren't up to much this morning - seems to be the Russia sanctions issue that's weighing things down. Currently:

    • The FTSE 100 is up 4 at 6811.36
    • Germany's Dax is up 9 at 9663.02
    • France's Cac 40 is down 6 at 4359.61
    • The Pound is down a touch at $1.693 and at 1 euro 26.3.
     
  20.  
    ANA PROFITS 08:31:
    Japan scene

    Japan's All Nippon Airways (ANA) has reported a return to profits in the three months to June. The better result was thanks to expansion at a Tokyo airport and changes to its pension plan. Net profit came in at 3.5bn yen (£20m) against a loss of 6.6bn yen. Sales were 10% higher.

     
  21.  
    BANKER BONUSES 08:24: Radio 5 live

    Shadow chancellor, Ed Balls, says the previous Labour government should have been tougher on bankers' bonuses. "Most of the criticism of the Labour government from the banking sector and the Conservative party was that we were much too tough on the banks. Now in retrospect, those criticisms were wrong because we should have been tougher," he tells 5 live. He points out that no one at the time was pressing them to crack down further.

     
  22.  
    AMAZON INDIA Via Email Simon Atkinson Editor, India Business Report

    The editor of India Business Report in Mumbai emails: "Online shopping is still in its infancy in India but growing fast. The market's led by local players but Amazon is upping its presence - and today said it's investing $2bn in its India operations. But restrictions on e-commerce here mean it can't hold its own stock like it does in the UK and US. - for now at least it is only a 'platform' for others to sell through."

     
  23.  
    BAT PROFITS 08:07:
    A pile of cigarettes

    Tobacco giant British American Tobacco reports a fall in profits to £2.6bn in the six months to 301 June from £2.9bn a year earlier. It blames the strength of the pound but revenue is also lower, down 10% to £6.8bn in the period. Volume - which measures the number of actual cigarettes sold - fell 0.4%.

     
  24.  
    BARCLAYS PROFITS 07:55: BBC Radio 4

    Back to Barclays profits for a moment as Antony Jenkins also tells Today that staff at his bank are changing their ways: "Staff at Barclays are fully behind what we're trying to do with our culture change programme." He says staff know this is not only "the right thing to do" but also the way to better profits.

     
  25.  
    ELECTRICITY CUT 07:43:
    Pylons

    Regulator Ofgem has announced a cut in distribution charges that will mean a £12 a year average reduction in electricity bills. And, incidentally, offers us the opportunity to publish a nice picture of pylons.

     
  26.  
    BANKER BONUSES 07:32: BBC Radio 4

    Barclays boss Antony Jenkins tells the Today programme his bank can already take action against mis-behaving bankers: "If someone has done something wrong and performed badly we have the right too claw the bonus back today."

     
  27.  
    HEADLINES
  28.  
    BARCLAYS PROFITS 07:27: BBC Radio 4

    Antony Jenkins, chief executive of Barclays is on the Today programme: "These are an encouraging set of results... Our capital position has never been stronger."

     
  29.  
    ITV PROFITS 07:27:

    ITV boss Adam Crozier says the broadcaster's "share of viewing" improved during its second quarter helped by the World Cup. He says he is confident of ITV's Autumn schedule of both new and returning drama and entertainment will help keep audience figures high. Meanwhile, ITV has benefitted from the economic recovery - specifically an improved advertising market.

     
  30.  
    BARCLAYS PROFITS 07:19:

    Investment bank income at Barclays fell 18%, reflecting a fall in customers. This follows allegations about malpractice in "dark pool" trading. Essentially, these are private stock markets and are the latest area of banking to be probed by regulators.

     
  31.  
    ITV PROFITS 07:19:

    ITV says total external revenues rose 7% to £1.2bn in the six months to 30 June, while revenue from its online, pay and interactive TV unit was up 20% to £67m.

     
  32.  
    BARCLAYS PROFITS 07:13:
    Barclays logo

    More on Barclays: Statutory pre-tax profit was £2.5bn (2013: £1.7bn), reflecting the fact bank had to set aside another £900m for PPI redress Read the full release here.

     
  33.  
    ITV PROFITS 07:10:
    ITV logo

    A strong set of numbers from broadcaster ITV this morning. Annual pre-tax profits are up 40% to £250m in the six months to 30 June compared with £179m for the same period last year.

     
  34.  
    BARCLAYS PROFITS 07:03:

    Barclays profit before tax is down 10% at £3.84bn.

     
  35.  
    BRITISH GAS BOSS 07:03:
    Iain Conn

    In all the excitement over bankers' bonuses we nearly forgot this. British Gas owner Centrica has succeeded in its pursuit of Iain Conn, confirming he will become its new chief executive from January 2015, succeeding Sam Laidlaw who is retiring. Mr Conn joins from BP where he has been chief executive, of BP's refining and marketing division,for the past seven years.

     
  36.  
    BARCLAYS PROFITS 06:52: BBC Radio 4
    Pedestrians pass a branch of Barclays Bank in the rain in London

    Michael Hewson, chief market analyst at CMC Markets is talking to the Today programme about Barclays interim results - coming up imminently. He says the investment banking arm of Barclays is "not listening" to new boss Antony Jenkins who has been trying to clean up the bank's reputation and practices.

     
  37.  
    DRIVERLESS CARS 06:44:
    Nissan car

    It's going to be a bank-heavy day today let's face it, but just to provide a break from all that, the government will be announcing changes in the law that will pave the way for driverless cars to take to Britain's roads next year. The government wants the UK to become a leader in developing the technology. In December, the Treasury said it would create a £10m prize to fund a town or city to become a testing ground for the cars.

     
  38.  
    MUSLIM ACCOUNTS 06:36:

    HSBC has told three Muslim organisations it will close their bank accounts. These are the Finsbury Park Mosque in North London, a think-tank on Islamic issues called the Cordoba Foundation based in West London, and a Muslim charity in Bolton called the Ummah Welfare Trust, which works in 20 countries giving aid. HSBC says the decisions were "absolutely not based on race and religion".

     
  39.  
    BANKER BONUSES 06:30: Radio 5 live

    More from Ms Mangwana on Wake Up to Money. She says the proposed seven year rule may be more about changing culture in banking and the way in which bankers view their bonuses. But she also points out bonuses are generally paid in tranches that vest over a number of years, already (commonly anything between three and five years). "That's the current formula and there are [already] mechanisms to reclaim those bonuses," she says.

     
  40.  
    TWITTER SHARES 06:21:
    Twitter

    In case this happened too late for you, Twitter shares rocketed 30% on stronger-than-expected financial results. Revenue more than doubled in the second quarter. Shares rose to $50 in after hours trading. Still down on its high of $74.73, hit in December.

     
  41.  
    BANKER BONUSES 06:10: Radio 5 live

    Samantha Mangwana, employment lawyer at Slater Gordon told Wake Up to Money seven years is a long time to hold a bonus and regulators may well find it difficult to reclaim money. It is highly likely bankers will have gone and spent the money already, she says, and have nothing that the Bank of England can reclaim.

     
  42.  
    BANKER BONUSES 06:07: BBC World News
    Tom Stephenson

    Those new rules on bankers' bonuses are expected to recommend a claw-back period of seven years. Tom Stevenson from Fidelity Worldwide on BBC World News says they could have been tougher: "One of the suggestions was that bankers could be jailed for a significant fall in profits - that's quite something isn't it. Even so, being able to claw back bonuses for seven years is pretty draconian."

     
  43.  
    06:02: Matthew West Business Reporter

    Good morning folks. It's looking like a busy day today. We also have trading updates from ITV and house builder Taylor Wimpey. As always you can get in touch via email at bizlivepage@bbc.co.uk and on twitter @bbcbusiness

     
  44.  
    06:00: Rebecca Marston Business reporter, BBC News

    Welcome again to the Live page. We're going to be banking heavy. There's Barclays results - in about an hour - and later this morning the Bank of England will release new restrictions on bankers' bonuses, said to be the toughest in the world. We'll see.

     

Features

From BBC Capital

Programmes

  • A factory in JapanThe Travel Show Watch

    Factory infatuation – why Japan’s industrial compounds are drawing large crowds at night

BBC © 2014 The BBC is not responsible for the content of external sites. Read more.

This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.