Performance Testing

Before I go any further, just a warning that this is dense, and even for me there are an almost overwhelming number of tables. I struggled with how to present data and settled on the format below, which consists of individiaual tables and a few graphs at the end in case you're a visual person. There's a third option however, which is the google docs spreadsheet I used while collecting data. I've made it public for your enjoyment and it contains the exact same data. (Update: Google Apps is acting wonky and not letting me share the spreadsheet quite yet—it will be posted in a little while. Update 2: Here we go, finally got the link to the spreadsheet for your persual)

First up is the received signal strength, which will show just how much difference that extra ~140 mW will buy. In reality, it’s surprisingly hard to see the difference, but again that increase in strength is only a few dBm, so I suppose it not being very dramatic here is to be expected since this is also a logarithmic scale. That said, the Gen 5 does post numbers correspondingly above the Gen 4 on average; remember that closer to 0 is higher power and thus better. The difference is actually more visible on 2.4GHz than 5GHz based on these numbers.

agrCtlRSSI (dBm) Comparison—2.4GHz (Closer to 0 is Better)
  2011 MacBook Pro (BCM4331,3x3:3) 2010 MacBook Pro
(BCM4322,2x2:2)
Apple Airport Extreme 4th Gen 5th Gen 4th Gen 5th Gen
Office (1) -46 -44 -45 -40
Hallway (2) -52 -50 -52 -52
Downstairs (3) -71 -70 -77 -77
Kitchen (4) -84 -81 -85 -84

agrCtlRSSI (dBm) Comparison—5GHz (Closer to 0 is Better)
  2011 MacBook Pro (BCM4331,3x3:3) 2010 MacBook Pro
(BCM4322,2x2:2)
Apple Airport Extreme 4th Gen 5th Gen 4th Gen 5th Gen
Office (1) -47 -48 -44 -47
Hallway (2) -59 -60 -54 -52
Downstairs (3) -81 -81 -82 -81
Kitchen (4) N/A N/A N/A N/A

Next is MCS (Modulation Coding Scheme) which shows how fast the card is connecting to the 802.11n network. Here we can see how much the 5th generation airport extreme improves MCS selection in a number of cases, especially the three spatial stream scenarios on the 2011 MacBook Pro.

MCS Comparison—2.4GHz (Higher is Better)
  2011 MacBook Pro (BCM4331,3x3:3) 2010 MacBook Pro
(BCM4322,2x2:2)
Lenovo X300
(Intel 6300 3x3:3)
Apple Airport Extreme 4th Gen 5th Gen 4th Gen 5th Gen 4th Gen 5th Gen
Office (1) 22 23 14 15 23 23
Hallway (2) 11 22 13 15 22 22
Downstairs (3) 4 12 0 11 21 13
Kitchen (4) 0 8 0 9 5 12

MCS Comparison—5GHz (Higher is Better)
  2011 MacBook Pro (BCM4331,3x3:3) 2010 MacBook Pro
(BCM4322,2x2:2)
Lenovo X300
(Intel 6300 3x3:3)
Apple Airport Extreme 4th Gen 5th Gen 4th Gen 5th Gen 4th Gen 5th Gen
Office (1) 23 23 15 15 23 23
Hallway (2) 21 22 15 14 12 22
Downstairs (3) 1 9 1 2 5 5
Kitchen (4) N/A N/A N/A N/A 3 3

Locations 3 and 4, which are in challenging environments, see massive increases, previously going from the lowest possible (or not even 802.11n) rate up to much faster rates on both MacBooks.

AFS is our next test, where we transfer a 500MB zip file up and down from an AFS server and average the throughput. On the downstream side of things, the improvements aren’t substantial until we’re in a challenging RF scenario downstairs;, here the new generation wireless card in the Airport Extreme makes a huge difference in throughput on 2.4GHz, and similarly on 5GHz, though at the farthest location it’s still impossible to connect to 5GHz.

AFS File Transfer Performance (Downstream)—2.4GHz (Mbps—Higher is Better)
  2011 MacBook Pro (BCM4331,3x3:3) 2010 MacBook Pro
(BCM4322,2x2:2)
Apple Airport Extreme 4th Gen 5th Gen 4th Gen 5th Gen
Office (1) 97.3 96.8 64.2 76.6
Hallway (2) 85.5 90.5 65.3 71.8
Downstairs (3) 2.67 68.7 2.1 39.4
Kitchen (4) 1.34 33.6 N/A 16.5

AFS File Transfer Performance (Downstream)—5GHz (Mbps—Higher is Better)
  2011 MacBook Pro (BCM4331,3x3:3) 2010 MacBook Pro
(BCM4322,2x2:2)
Apple Airport Extreme 4th Gen 5th Gen 4th Gen 5th Gen
Office (1) 140.6 148.4 95.3 122.9
Hallway (2) 86.3 130.9 84.9 99.5
Downstairs (3) 16.1 39.3 6.6 19.5
Kitchen (4) N/A N/A N/A N/A

When it comes to upstream, the results are dramatic both on 2.4GHz and 5GHz. Throughput is almost always over double, thanks probably in part to the better front end and receive sensitivity of the Airport Extreme’s new wireless stack.

AFS File Transfer Performance (Upstream)—2.4GHz (Mbps—Higher is Better)
  2011 MacBook Pro (BCM4331,3x3:3) 2010 MacBook Pro
(BCM4322,2x2:2)
Apple Airport Extreme 4th Gen 5th Gen 4th Gen 5th Gen
Office (1) 54.0 128.6 52.6 81.5
Hallway (2) 45.3 105.1 30.9 78.1
Downstairs (3) 4.0 37.6 2.8 32.7
Kitchen (4) N/A 10.5 N/A 15.3

AFS File Transfer Performance (Upstream)—5GHz (Mbps—Higher is Better)
  2011 MacBook Pro (BCM4331,3x3:3) 2010 MacBook Pro
(BCM4322,2x2:2)
Apple Airport Extreme 4th Gen 5th Gen 4th Gen 5th Gen
Office (1) 186.7 214.5 150 134.2
Hallway (2) 175.2 195.8 145.7 119.5
Downstairs (3) 19.8 34.3 11.6 21.6
Kitchen (4) N/A N/A N/A N/A

Iperf is finally up, which we can run on the X300 in addition to both Macs. Here on downstream the results are improved pretty substantially for the 3x3:3 2011 MacBook Pro, and across the board for the challenging downstairs RF scenarios. The same applies on 5GHz as well, and in the best case, we can push nearly 300 Mbps on the new MacBook Pro. It’s a dramatic improvement in best case throughput if you have the right card, though the Intel card ends up performing similarly on 5GHz with both the new and old Airport Extreme card. More on that in a second.

iperf 2.0.5 (Downstream)—2.4GHz (Mbps—Higher is Better)
  2011 MacBook Pro (BCM4331,3x3:3) 2010 MacBook Pro
(BCM4322,2x2:2)
Lenovo X300
(Intel 6300 3x3:3)
Apple Airport Extreme 4th Gen 5th Gen 4th Gen 5th Gen 4th Gen 5th Gen
Office (1) 112 161 71.9 87.4 88.3 84.8
Hallway (2) 91.7 138 69.1 78.1 75.8 84.7
Downstairs (3) 5.89 117 2.22 46.9 51.9 68.7
Kitchen (4) 1.49 35.2 N/A 23.7 14.4 46.1

iperf 2.0.5 (Downstream)—5GHz (Mbps—Higher is Better)
  2011 MacBook Pro (BCM4331,3x3:3) 2010 MacBook Pro
(BCM4322,2x2:2)
Lenovo X300
(Intel 6300 3x3:3)
Apple Airport Extreme 4th Gen 5th Gen 4th Gen 5th Gen 4th Gen 5th Gen
Office (1) 213 311 92.2 136 101 112
Hallway (2) 119 243 95.4 97.2 98 106
Downstairs (3) 17.5 47.9 6.83 21.3 41.5 57.5
Kitchen (4) N/A N/A N/A N/A 13.8 11.1

On the upstream with Iperf things improve dramatically across the board on 2.4GHz, and marginally improve on 5GHz with the new WLAN card inside the Airport Extreme.

iperf 2.0.5 (Upstream)—2.4GHz (Mbps—Higher is Better)
  2011 MacBook Pro (BCM4331,3x3:3) 2010 MacBook Pro
(BCM4322,2x2:2)
Lenovo X300
(Intel 6300 3x3:3)
Apple Airport Extreme 4th Gen 5th Gen 4th Gen 5th Gen 4th Gen 5th Gen
Office (1) 57.7 159 64.2 97.7 65.1 126
Hallway (2) 33.1 120 24 98.4 42.3 113
Downstairs (3) 4.42 36.7 4.26 33.6 20.6 38.4
Kitchen (4) 1.89 11.6 1.50 8.51 7.39 21.1

iperf 2.0.5 (Upstream)—5GHz (Mbps—Higher is Better)
  2011 MacBook Pro (BCM4331,3x3:3) 2010 MacBook Pro
(BCM4322,2x2:2)
Lenovo X300
(Intel 6300 3x3:3)
Apple Airport Extreme 4th Gen 5th Gen 4th Gen 5th Gen 4th Gen 5th Gen
Office (1) 215 302 168 191 154 148
Hallway (2) 196 252 169 174 150 144
Downstairs (3) 15.4 41 7.97 22.9 26.8 39.4
Kitchen (4) N/A N/A N/A N/A 7.55 8.66

At the end of the day, the new Airport Extreme dramatically improves throughput in the best case and in a few regions where signal was previously unusable. In the worst case (location 4), performance improves from being essentially unusable to totally fine, and in the case of the 2010MBP goes from not being able to connect at all to pushing 23 Mbps.

So the unanswered question is how the 3x3:3 2011MBP manages to be much faster compared to the 3x3:3 Intel 6300 card, and I suspect the answer might be that the combination of BCM4331 on the client and BCM4331 on the AP enables Apple to use Broadcom’s frame bursting high speed modes—aka modern speedbooster. Careful observers will note in addition that while Iperf over 40 MHz 802.11n (with a link rate of 450 Mbps) delivers 311 Mbps of downstream, the same test on AFS is around 150 Mbps down, possibly due to compression. In addition, note how the Intel card lags behind in locations 1 and 2 until signal gets lower and then becomes competitive again. To me, this definitely seems to indicate some Broadcom-to-Broadcom enhancements are at play. Frankly for Apple this makes sense considering their top to bottom ecosystem control; if you have the ability to choose the card in the AP and the client, why not go with a solution that offers benefits?

Airport Extreme vs. Time Capsule

The next question is how the Time Capsule compares. Rather than re-run all 128 data-points (and then multiple tests per scenario for the purposes of averaging and removing outliers), I decided to use a subset and see whether performance and range is the same on the Time Capsule in those cases. As we showed before, the Time Capsule and Airport Extreme use the same exact wireless card, though gain is different on the Time Capsule antennas than the Airport Extreme.

For this testing, I just used the 2011MBP with its 3x3:3 radio, and ran through signal measurements and Iperf. First up is how signal strength looks.

agrCtlRSSI (dBm) Comparison (Closer to 0 is Better)
  2.4GHz 5GHz
  Airport Extreme Time Capsule Airport Extreme Time Capsule
Office (1) -44 -41 -48 -49
Hallway (2) -50 -51 -60 -60
Downstairs (3) -70 -68 -81 -83
Kitchen (4) -81 -81 N/A N/A

You can pretty much immediately tell that things are very similar. On 2.4GHz and 5GHz the two are very comparable except in a few odd cases. Of course the propagation isn’t going to be identical between the two, but subjectively it’s close.

Next is MCS, and here things are again close, with the Time Capsule narrowly edging out the Airport Extreme on 2.4GHz, and things being very equal on 5GHz. It’s looking the same so far.

MCS Comparison (Higher is Better)
  2.4GHz 5GHz
  Airport Extreme Time Capsule Airport Extreme Time Capsule
Office (1) 23 23 23 23
Hallway (2) 22 23 22 23
Downstairs (3) 12 13 9 8
Kitchen (4) 8 9 N/A N/A

So what about performance now with Iperf? On the downstream side of things, the Airport Extreme comes out on top narrowly in all but the second location on 2.4 and 5GHz. But the difference is minuscule.

iperf 2.0.5 (Downstream) (Mbps—Higher is Better)
  2.4GHz 5GHz
  Airport Extreme Time Capsule Airport Extreme Time Capsule
Office (1) 161 157 311 306
Hallway (2) 138 154 243 246
Downstairs (3) 117 103 47.9 44.9
Kitchen (4) 35.2 29.6 N/A N/A

Upstream is a similar story, with the two being very close outside locations 3 and 4, where the Time Capsule narrowly edges the Airport Extreme out on 2.4GHz.

iperf 2.0.5 (Upstream) (Mbps—Higher is Better)
  2.4GHz 5GHz
  Airport Extreme Time Capsule Airport Extreme Time Capsule
Office (1) 159 153 302 304
Hallway (2) 120 129 252 251
Downstairs (3) 36.7 55.5 41 32.1
Kitchen (4) 11.6 17.3 N/A N/A

All said and done however, the two are incredibly close and despite the difference in gain that the FCC docs would lead you to believe, are virtually indistinguishable in some real-world testing. In my mind, if you’re concerned about WiFi performance, the Time Capsule and Airport Extreme both perform equally well.

Graphs

If you're a more visual person and find that wall of tables and text too daunting, Anand also made some awesome graphs which I would be remiss to not reproduce here for your viewing pleasure.

First up is AFS performance compared on the 2011MBP between the 4th and 5th generation Airport Extreme.

 

Again the main improvements with AFS (real-world file transfer) happen out at the extremes where previously signal was unusable on 2.4GHz, and likewise on 5GHz. That really tells the story of the (sometimes dramatic) difference that the higher power WLAN solution in the 5th generation makes over the 4th generation.
 
Next are two charts showing again that for WiFi purposes the Time Capsule and Airport Extreme are virtually identical.

Within the margin of error, we see the Time Capsule and Airport Extreme perform very, very closely.

WiFi Testing Methodology Disk Performance - Airport Extreme vs. Time Capsule
POST A COMMENT

103 Comments

View All Comments

  • deadshort - Sunday, August 07, 2011 - link

    Fantastic: thanks for doing that! Spread the word!

    If the main motherboard and chipset are the same, I guess it stands to reason that a more powerful radio would take a bit more power. ~250 Wh/day, or about 5% of my fairly careless daily consumption, is worth knowing but not worrying about.
    Reply
  • Brian Klug - Sunday, August 07, 2011 - link

    Yeah, I should note that right now idle seems to be around 8-9 watts without much change.

    -Brian
    Reply
  • jalin2 - Monday, August 08, 2011 - link

    I'm in the market to get a new router and was set on getting the E4200. Now I think I'm going to hold off and wait until your router comparison article. Any idea on timeframe when that'll be released? Reply
  • jackwong - Sunday, August 14, 2011 - link

    I would rather go with Apple, the only good Linksys router I have is the WRT54. I also have the e2500 few months ago, I can't believe a 2011 router doesn't has gigabyte ports... and it has connection problem everyday...

    I would rather go with Apple Extreme base station unless the e4200 is way cheaper.
    Reply
  • melgross - Monday, August 08, 2011 - link

    I've had a lot of Linksys routers over the years, stupidly, because every one died. That includes some commercial models. I will NEVER use another Linksys product again.

    I've got one Extreme, and two Expresses, and have never had a problem with them. As the review says, they just work.

    I've got an old house, built in 1925. Most interior walls are brick. All the walls and ceilings have .75" wood over the brick, with gal steel mesh over that, with .75" mortar, and a .25" layer of plaster over that. The damn thing is as close to a Faraday cage as anyone will ever live in. I've removed some of that when re-doing the kitchen and downstairs bathroom as well as the two bathrooms upstairs by having everything stripped to the beams and replaced with double .5" Sheetrock, but still...
    Reply
  • Guspaz - Monday, August 08, 2011 - link

    Perhaps some mention should be made of the fact that Apple advertises the drive in the Time Capsule as a "server-grade hard drive".

    You do mention that it's very much a consumer disk (in fact, the most "consumer" you can get since it's not even a black or RAID edition), but Apple is really billing the thing as server-grade, which is false advertising.
    Reply
  • jwoelich - Monday, August 08, 2011 - link

    "Fortunately, a source inside Western Digital was willing to explain matters; "If you enter in the drive number on NewEgg you're going to see it come up as a desktop drive. That platform is actually built on a lot of other products for a lot of different OEMs. And the specs could change depending on whatever program we're building for. Needless to say Apple has very stringent requirements that are very specific to them and very unique to them, and that drive has actually been developed and is unique for that Apple product. That unique Apple logo, and what we call 'to the right of the dash,' if you will, indicate that this drive is for a very particular partner to us and this drive is dedicated to that particular audience. You could not buy this same drive at NewEgg or Amazon."

    When we asked whether the variation of the WD20EARS drive is rated for a 1-million-hour MTBF, our source confirmed that it was: "We don't spec our desktop drives with MTBF because our customers don't purchase in that manner, but this particular drive for this particular OEM with these unique requirements does meet those specifications."

    Western Digital says otherwise.
    Reply
  • repoman27 - Monday, August 08, 2011 - link

    lowlymarine did mention it in a comment two days ago, to which I pointed out that for price per GB and performance per watt the Caviar Green is tough to beat, and therefore perfect for servers. Just considering how many Google uses, I would be surprised if there is another line of drives more deployed in servers at the moment than Caviar Green.

    Not all "servers" are designed for crazy amounts of IOPS, massive queue depths, or incredibly high availability. For backup or redundancy, which is what the Time Capsule is marketed for, you want big, cheap and low-power.

    No offense, jwoelich, but although the quote may be legitimate, I have a seriously hard time buying that line of utter crap. We're pretty much down to a three horse race in the storage arena, and just like Intel or AMD might bin things differently or cater to large OEMs in various ways, they basically just pump out a small range of identical products by the millions. There are no "magical drives" that are perfect for servers but not sold by Newegg. The way I read it, Apple had a very stringent range of specifications, to which a bog standard Caviar Green drive happens to adhere perfectly (i.e. low price per GB and low power requirements.)
    Reply
  • jwoelich - Tuesday, August 09, 2011 - link

    I don't see where anyone stated anything being 'magical', just that if you were to go onto Newegg, you couldn't just order one of these specific drives. Now, does that mean that there is more to it than Western Digital simply taking any WD Green drive, testing it to conform with the criteria set forth by Apple, then throw on a new sticker? Or did WD design/modify a product specifically to meet that criteria? More likely, WD crafted a Frankendrive using the criteria Apple set forth, and used the bare minimum of higher quality components that would more likely ensure that criteria was met, which was probably nothing more than power consumption, size, speed and a greater MTBF. Reply
  • LoneWolf15 - Monday, August 08, 2011 - link

    "Some more searching revealed the solution—pin 20 (wireless disable) needs to be taped over to signal the card that the wireless disable switch (which doesn’t exist, since this is the WAN port) is in the on position. A quick surgical application of tape, and the card worked perfectly—take that, Lenovo security. As an aside, what a completely pointless and trivial barrier this is—the Mini PCIe standard (and moreover WiFi notebook cards themselves with U.FL connectors) are designed to be completely and absolutely interchangeable. The notion that this provides any added security (when the adjacent slot is completely unguarded) or—even more absurd, convenience—is nothing short of a surrealist notion."

    Lenovo isn't the only one. HP also does the exact same thing; they whitelist cards in the BIOS, so that only their branded version of Intel's (or Broadcom's) wireless cards work.

    I don't think security has anything to do with it. I think two factors are involved --1) that they don't want to support any other wireless cards than their own, and 2) that they want you to buy from them, establishing vendor lock-in.
    Reply

Log in

Don't have an account? Sign up now