Cloud = x86 and open source

From a high-level perspective, the basic architecture of Facebook is not that different from other high performance web services.

However, Facebook is the poster child of the new generation of Cloud applications. It's hugely popular and very interactive, and as such it requires much more scalability and availability than your average website that mostly serves up information.

The "Cloud Application" generation did not turn to the classic high-end redundant platforms with heavy Relational Database Management Systems. A combination of x86 scale-out clusters, open source websoftware, and "no SQL" is the foundation that Facebook, Twitter, Google and others build upon.

However, facebook has improved several pieces of the Open Source software puzzle to make them more suited for extreme scalability. Facebook chose PHP as its presentation layer as it is simple to learn, write, and read. However, PHP is very CPU and memory intensive.

According to Facebook’s own numbers, PHP is about 39 times slower than C++ code. Thus it was clear that Facebook had to solve this problem first. The traditional approach is to rewrite the most performance critical parts in C++ as PHP Extensions, but Facebook tried a different solution: the engineers developed HipHop, a source code transformer. Hiphop transforms the PHP source code into faster C++ code and compiles it with g++.

The next piece in the Facebook puzzle is Memcached. Memcached is an in-RAM object caching system with some very cool features. Memcached is a distributed caching system, which means a memcached cache can span many servers. The "cache" is thus in fact a collection of smaller caches. It basically recuperates unused RAM that your operating system would probably waste on less efficient file system caching. These “cache nodes” do not sync or broadcast and as a result the memory cache is very scalable.

Facebook quickly became the world's largest user of memcached and improved memcached vastly. They ported it to 64-bit, lowered TCP memory usage, distributed network processing over multiple cores (instead of one), and so on. Facebook mostly uses memcached to alleviate database load.

Facebook Technology Overview The Facebook Open Compute Servers
POST A COMMENT

68 Comments

View All Comments

  • Menetlaus - Thursday, November 03, 2011 - link

    Power savings absolutely justifies the work they did in customizing.

    20W less power consumption x 24/7/365 operation = 175KW.h (per server per year)
    175KW.h x $0.1/kw.h = $17.50 in power savings/year

    Just looking at the final image in the article there are easily 30 racks of 30 servers visable (30 x 30 x 17.50 =) $15 750/year in power saving.

    Since most power going into a computer ends up as wasted heat, if the 900 servers (from above) were consuming the additioanl 20W this would be ~18KW of additional heat being produced which needs to be cooled. This offers additional operational and capital cost savings due to the smaller cooling requirements.

    Water cooling may be a more efficient way of pulling heat out of the server rack, but the additional parts to move the water around the facility and to cool it adds to the total costs. Water is more efficient because it carries more heat/volume than air and with the piping the heat can be taken outside of the server room, while fans heat the air around the servers where another method of removing the heat is then required.

    The custom power supply at 270V and custom motherboard aren't really that difficult to get, as so many makers of each part already do custom designs for major PC makers (Dell/HP/etc). The difference between 208v and 270v from an electrical design standpoint isn't a big change, neither is removing parts from a motherboard.

    In short it's the economy of scale. You or I wouldn't be able to do this for a dozen personal systems as the costs would be huge per system, on the other hand for anyone managing 1'000's of servers the 20W/per adds up quick.
    Reply
  • iwod - Thursday, November 03, 2011 - link

    And i am guessing Facebook has at least 10 times more then what is shown on that image. Reply
  • DanNeely - Thursday, November 03, 2011 - link

    Hundreds or thousands of times more is more likely. FB's grown to the point of building its own data centers instead of leasing space in other peoples. Large data centers consume multiple megawatts of power. At ~100W/box, that's 5-10k servers per MW (depending on cooling costs); so that's tens of thousands of servers/data center and data centers scattered globally to minimize latency and traffic over longhaul trunks. Reply
  • pandemonium - Friday, November 04, 2011 - link

    I'm so glad there are other people out there - other than myself - that sees the big picture of where these 'miniscule savings' goes. :) Reply
  • npp - Thursday, November 03, 2011 - link

    What you're talking about is how efficient the power factor correction circuits of those PSUs are, and not how power efficient the units their self are... The title is a bit misleading. Reply
  • NCM - Thursday, November 03, 2011 - link

    "Only" 10-20% power savings from the custom power distribution????

    When you've got thousands of these things in a building, consuming untold MW, you'd kill your own grandmother for half that savings. And water cooling doesn't save any energy at all—it's simply an expensive and more complicated way of moving heat from one place to another.

    For those unfamiliar with it, 480 VAC three-phase is a widely used commercial/industrial voltage in USA power systems, yielding 277 VAC line-to-ground from each of its phases. I'd bet that even those light fixtures in the data center photo are also off-the-shelf 277V fluorescents of the kind typically used in manufacturing facilities with 480V power. So this isn't a custom power system in the larger sense (although the server level PSUs are custom) but rather some very creative leverage of existing practice.

    Remember also that there's a double saving from reduced power losses: first from the electricity you don't have to buy, and then from the power you don't have to use for cooling those losses.
    Reply
  • npp - Thursday, November 03, 2011 - link

    I don't remember arguing that 10% power savings are minor :) Maybe you should've posted your thoughts as a regular post, and not a reply. Reply
  • JohanAnandtech - Thursday, November 03, 2011 - link

    Good post but probably meant to be a reply to erwinerwinerwin ;-) Reply
  • NCM - Thursday, November 03, 2011 - link

    Johan writes: "Good post but probably meant to be a reply to erwinerwinerwin ;-)"

    Exactly.
    Reply
  • tiro_uspsss - Thursday, November 03, 2011 - link

    Is it just me, or does placing the Xeons *right* next to each other seem like a bad idea in regards to heat dissipation? :-/

    I realise the aim is performance/watt but, ah, is there any advantage, power usage-wise, if you were to place the CPUs further apart?
    Reply

Log in

Don't have an account? Sign up now