More Than Throughput: Next Generation Wi-Fi Testing with Ixia's WaveDevice

Wi-Fi Performance: iPad Pro and Pixel C

Now that we’ve gone over the basics of how Wi-Fi works, we can get to the true focus of today's article: the results. As this article is something of a preview of what we're working on behind the scenes, I’m still figuring out how I want to condense and present these results for future articles, so for now we’ll be looking at raw data generated from WaveDevice software. As you might be able to guess, it turns out that there are a number of differences in Wi-Fi performance that are readily apparent once you start running tests using WaveDevice.

In the interest of setting a baseline for performance, I elected to compare two tablets that I had on hand that could run WaveAgent software. Apple's 12.9" iPad Pro uses a BCM4355 solution with 2x2 802.11ac, while Google's Pixel C uses a 2x2 802.11ac device. Judging by the system firmware of the Pixel C it looks like we’re looking at BCM4350 chip, so at a high level there really shouldn’t be a huge delta between the iPad Pro and Pixel C.

To start off we can look at the range vs rate test, which is designed to see how the device performs in response to fading base station transmit power. In the case of the iPad Pro, at 15 dBm transmit power the device reported -33 dBm RSSI (received signal strength indicator). It's important to note that the IEEE 802.11 standard doesn't really define RSSI beyond a unitless value, but in the cases we're interested in RSSI is really more a reference to received power, where dBm is the unit of power rather than watts due to the huge differences in power from good to poor reception. However, in the interest of focusing on the rate at which throughput decreases the test sweep in transmit power was 0 dBm to -50 dBm with a 5 dBm step. With this sort of data, we can actually see the kind of throughput that the device sustains for a given RSSI level and for a given transmit power. Of course, there are a number of other statistics that can be examined here as previously discussed, but basically the main takeaway is that the iPad Pro is capable of sustaining 600 Mbps and approaches 0 Mbps at -45 dBm transmit power. Given that we’re looking at a ~47 dB path loss from the transmitter to the receiver, this basically means that the iPad Pro is capable of sustaining non-zero throughput all the way out to roughly -90-95 dBm RSSI.

If you think back to the explanation of the physical layer of Wi-Fi, the reason why this is important is because received power is not quite the same thing as signal to noise ratio. While having high received power does improve your signal to noise ratio, if your receiver has a great deal of phase noise to begin with from poor amplifier design or some other issue in the chain, your throughput is going to fall flat even if the device can transmit/receive effectively to/from the access point.

For the Pixel C, things aren't quite as rosy. In this case I recorded a -43 dBm RSSI at 15 dBm, which is already quite concerning to start with. I attempt to maximize RSSI and throughput in my tests, so it's likely that the Pixel C either has a highly directional antenna, insufficient antenna gain, or a significant impedance mismatch somewhere leading to significant signal reflection. These are all unquestionably hardware problems that are unlikely to be solved by any software changes. The Pixel C is also highly unstable as it approaches the edge of reception, so the test above stops at -30 dBm transmit power because attempting to go to -35 or -40 dBm resulted in the device disconnecting from the network. Resolving this required restarting WaveAgent, WaveDevice, and deleting all saved SSIDs from the Pixel C. I also had to change the SSID of the test AP, so to have any usable results at all it was necessary to adjust the test parameters for the Pixel C.

Putting these issues aside, it's obvious that the Pixel C is underperforming here regardless of how we go about it. Maximum throughput is well below what the iPad Pro can achieve even at short ranges, and the same is true even when compensating for the delta in RSSI. -30 dBm transmit power on the Google Pixel C is equal to -40 dBm transmit power for the iPad Pro when equalizing RSSI, so the iPad Pro can sustain roughly 50% higher throughput even at the extremes of reception. Equalizing RSSI means that we're still ignoring the antenna and other portions of the RF front-end, so it's entirely possible that the delta is even worse given that I couldn't achieve anywhere near -30 dBm RSSI on the Google Pixel C regardless of device orientation within the RF isolation chamber.

As mentioned previously, WaveDevice actually allows for a deeper look at what’s going on behind higher-level failures. Out of curiosity, I decided to run a simple upload throughput test at 15 dBm transmit power at the Pixel C’s highest possible throughput rate, and I found that it’s basically unable to use the highest throughput 256 QAM because there’s too much noise between each point on the constellation to tell what the device intended to transmit. Even when it can use MCS 9 (256 QAM and only 1 redundant bit) the Pixel C averages roughly a 3-4% EVM error, while the iPad Pro was closer to 1-2% at 256 QAM. And though 3-4% might sound like a small value, 256 QAM leaves very little room for error. I regularly saw drops down to MCS 7 (64 QAM, 1 redundant bit) even in ideal cases which resulted in noticeable drops in throughput during this simple test. I'm hesitant to go any further in the analysis here since we don't know enough about the design of the Pixel C's Wi-Fi subsystem, but an OEM would be able to use this information to start searching for potential sources of phase noise. It may be that we're looking at something like improper impedance matching somewhere in the system, amplifiers that are either poorly selected or poorly integrated, and/or a phase-locked loop somewhere that isn’t set up or designed properly for this task.

Moving on to the next test of interest, we can take a look at how these two devices perform in our roaming test. While I'm still experimenting with this for use in full reviews, for now I set up a test with a 10 Mbps load and a starting transmit power of 10 dBm, going down to -40 dBm for the noise floor with a 3 dBm step every second. 64 access points are used for this test, and all of them are on the same channel which should make this easier as the device doesn’t need to scan on all channels for the next access point to jump to. This is a fairly aggressive test, as I’ve run this test on a few devices and nothing is 100% perfect here, although some devices are clearly better than others.

In the case of the iPad Pro, we see a median roam delay of 42ms, which is reasonably respectable given the 10 Mbps traffic and fairly aggressive transmit power changes. However, the Google Pixel C seriously falls short here despite using a similar Wi-Fi chipset as I encountered a number of times when the Pixel C dropped from the network entirely and was unable to complete the test. Even when it didn’t fall off the network, the median roam delay was 682ms, which is pretty much guaranteed to result in some kind of disruption to things like VOIP calls, video conferencing, and similar problems. Ultimately the issue here is that roaming is a very common scenario a device will need to handle, as any office, school, or convention center is pretty much guaranteed to have multiple access points with the same SSID and authentication. There’s also the strong possibility that each access point is on a different channel, which would only increase roam latency figures relative to what I was able to test.

Pixel C Roam Latency

Needless to say, WaveDevice is an incredibly powerful system when it comes to providing insight into parts of Wi-Fi that have effects at the user experience level. Observant readers might have noticed that there's no Traffic Mix test here, and it turns out that the two devices we're looking at don't have particular issues with the Traffic Mix test. However, given the data shared with Ixia on devices they've tested, it's likely that the need for Traffic Mix testing will appear sooner rather than later. For full reviews we'll be including this test as well for completeness.