Original Link: http://www.anandtech.com/show/8229/hp-labs-the-science-of-technology



Welcome to HP's Houston Campus

Recently HP invited journalists from a variety of publications down to their Houston location for a tour of the Science Labs – facilities and labs dedicated to designing, testing, debugging, analyzing, and improving HP products. The groups we met with were largely tasked with improving commercial products, so the focus was more on business and enterprise products like workstations (desktop and mobile) and servers, and not so much on the consumer variants of these products. I'll admit that as this is the first time I've had a chance to go on a tour with any of the major OEMs, I didn't quite know what to expect. Needless to say, there were some cool demonstrations, a lot of interesting information, and a few experiences that were quite awesome.


Houston: Mountains Not Included

Just a few brief items of interest before we start with the tour, the HP campus in Houston is home to around 10,000 employees and contractors, and it was originally part of Compaq before the merger (or acquisition, depending on how you want to look at things). Of the 10,000 or so personnel at the Houston site, it's worth noting that only a small percentage (maybe 5%, give or take) are directly involved with the labs we toured on this visit. Other aspects of the Houston location include server assembly, research and development, administrative, marketing, customer experience, etc. – basically all the normal stuff you'd see at any large OEM location.

If you happen to follow me on Twitter, I actually decided to put the service to good use for a change and tweeted a bunch of photos throughout the day, so you were able to follow along. I have plenty of other photos along with a few videos to share, and today I'll discuss what we were shown and the nature of the testing and validation in much more detail than the short 140 character Twitter posts would allow. Plus I'm not wandering around the HP campus trying to type on my smartphone, which will certainly make writing easier.



HP Multimedia Lab

Things started off somewhat slowly in the Multimedia Lab, where we were shown some of the testing that's being done to validate their workstation hardware and software. The new ZBook 15 and 17 were a major topic here, as these are the first two HP mobile workstations to support Intel's Thunderbolt 2.0 technology (and the first to support Thunderbolt as well, I believe). There was no mention or sight of the ZBook 14, if you're wondering – that's basically a different sort of ZBook, focused more on style and portability than on being a true mobile workstation.

HP showed us a demonstration of a single ZBook 17 streaming two 4K videos from an external Thunderbolt 2 array (from SSDs I believe, though it may have been HDDs) to two 4K displays over a single daisy-chained Thunderbolt 2 connection, and while this was going on they also started a copy from another Thunderbolt 2 SSD enclose on the same chain to a second Thunderbolt 2 SSD enclosure. Add all of that up and it's an impressive amount of data being shunted around, and throughout the demonstration the 4K video continued to roll with no apparent issues.

HP's DreamColor premium LCDs were also discussed, and we've talked about that in past reviews. DreamColor displays use 10-bit panels that come factory calibrated, and they're targeted at imaging/multimedia professionals. It's not hard to see how having accurate colors can really help certain workloads – film crews are one easy example, and HP commented on how they have customers that love the DreamColor panels as they no longer need to check footage on a larger display to verify the quality – they can simply look at it on their mobile workstation display and if needed reshoot a scene right there. For "normal" users the benefits aren't quite so dramatic, but then regular users probably aren't going to spend the money for a DreamColor display.

Another demonstration consisted of a single ZBook 17 powering five separate displays, with two displays running off the Intel integrated graphics while three other displays were powered by the NVIDIA Quadro GPU. Besides the integrated display panel (running off Intel HD Graphics), there was a VGA output (also running off the Intel iGPU), two DisplayPort panels in daisy chain off a DP 1.2 port (NVIDIA), and an Apple Thunderbolt display (NVIDIA). Not all of the displays had active content, but two were showing videos, one had Office open (Excel along with a stock ticker), another had IE running, and the laptop was showing the five connected displays. This may not be a common use case for most people, but it was good to see that HP is actually testing the possibilities.

One item HP did note was that the earlier Apple Thunderbolt displays were not fully validated for Windows use, so there can occasionally be problems. While they did have an Apple Thunderbolt display, the general advice was for Windows users to stick with devices that have been specifically certified to work with Windows.

Moving down to another aisle of testing equipment brought us more ZBook workstations running additional tests. One ZBook 17 laptop had four displays all running as a single large surface off of NVIDIA's Quadro GPU through the use of NVIDIA's Mosaic technology – basically, their alternative to Eyefinity. The screens in question were showing DiRT 3 (I think – it might have been some other Codemasters racing title), and while the looping "attract" demo did run, it wasn't too surprising to see that frame rates were likely in the sub-20 FPS range. Four 1080p displays off a single mobile dGPU? Yeah, that might be asking a bit too much.



HP Software Testing Lab

The Software Testing Lab again focused more on HP's ZBook and EliteBook mobile workstations, as well as the ProBook line, plus a few business all-in-one systems for good measure. The one figure that was thrown out was that over 133,000 hours of testing goes into each ProBook, EliteBook, and ZBook that comes out, which is all done to ensure high reliability and compatibility. The demonstrations were a bit more mundane, unfortunately.

First up was a demonstration of… a fingerprint scanner. I could almost see the eyes of the various editors glaze over on this one. I'm pretty sure everyone is familiar with how fingerprint scanners work, and other than some additional behind-the-scenes work that HP apparently does to ensure their systems are secure, there wasn't much to add.

More interesting was the new feature being rolled out in the latest ZBook and ProBook systems. Dubbed HP Stratus, it's not exactly a revolution, but HP's newest mobile workstations will now support the ability to download and update the BIOS, all from within the UEFI environment. That means full Internet access will work, and the software will be able to connect to HP's servers, check for a new version of the BIOS – and it will know exactly what model of hardware you're using, so no guesswork on the part of the user is required – and update the BIOS. I've seen the ability to flash the BIOS in quite a few laptops and motherboards now, but I'm not sure I've encountered any that allow you to download BIOS updates directly.

A secondary technology related to this is HP Sure Start, a form of BIOS Recovery. First, there's now a second backup copy of the BIOS and firmware on the new systems. Various motherboards have had dual BIOS chips for a while, but not many laptops have included this feature to my knowledge. More importantly, each time the system is powered on, the two BIOS revisions are compared, and if there is any corruption in the primary BIOS – e.g. from a failed BIOS update, or a virus, or something else – the system will automatically recover from the backup master BIOS and then continue booting. And in the event of a BIOS update, the primary BIOS gets updated, and when the system reboots and everything validates properly, the master BIOS is then updated to match the primary BIOS.

It's not clear precisely how long the validations takes – HP said around five seconds I believe – and if a recovery is necessary it will add to the boot time, but that's better than a bricked laptop certainly. An indicator LED on the front of the chassis will light up if BIOS recovery is needed, and the result is that at least on these new mobile workstations (ZBook 15 and 17), updating the BIOS and potential BIOS corruption will largely be a thing of the past.

On our way to the next testing lab, we also passed through an assembly area for servers, which was basically a large warehouse full of parts and work areas. You can see images of this area in the above gallery as well.



HP Environmental Lab

Here's where things started to get fun, as the Environmental Lab has a variety of "torture chambers" that are used to test hardware, packaging, components, etc. in extreme conditions. What sort of extreme conditions and testing are we talking about? Let's first start by noting that the Environmental Lab tends to use destructive testing – so things are actually tested until they break. Destroying thousands of dollars of hardware a day can't help but be entertaining, right? I think one of the HP engineers stated that the labs destroy over $5000 of hardware a day in their testing. It's not just destroyed for laughs, of course – when a product does eventually fail, they then look at how and why it failed and determine if they can improve/fix the product to avoid a particular type of failure, or if the point at which the product failed is actually in the "safe" zone.

Testing included shock tests, where products are subjected to repeatable drops using heavy equipment; vibration testing basically had a gigantic subwoofer that would shake the products (usually for hours at a time). Another test area had a large metal compress that could be used for testing the durability of product packaging – so HP knows how many boxes/pallets they can stack on top of each other. Temperature, altitude, and humidity chambers could torture devices with extremes of heat/cold, altitude (pressure), or humidity. Most of these chambers were somewhat smaller in nature, designed to test individual products; we'll see other larger chambers in some of the other labs. There were also a variety of drop tests – ranging from a device to test how laptop hard drives would respond to shorter drops (with different drive cage material) up to larger tests for dropping products or packaging from several feet above the ground.

Of course not all of the environmental testing is destructive. One area of the labs was a moderate sized semi-anechoic chamber used for noise testing. (It's only "semi-anechoic" as the floor is still a hard surface, but the walls and ceiling all have noise-absorbing materials.) I've seen photos of anechoic chambers before, but this was my first time actually stepping inside of one. It's a very cool experience, and when everyone stopped moving/talking it was amazingly quiet – even with the noisy vibration testing machinery just a short walk down the hallway from the chamber! The level of silence was almost oppressive, and even talking in the room things sounded weird, as we're normally used to hearing reflections of our own voices bouncing off the walls and ceiling.

The chamber itself is of course used to conduct noise testing, and it's large enough to handle up to full server racks if needed. HP uses anywhere from one microphone (e.g. for an operator seated in front of a laptop) to as many as a dozen or so microphones for their noise testing. There are markings around the room for where microphones need to be located for specific types of testing, and the equipment is capable of measuring noise levels well below 20 dB, which is less than what most people would normally notice. There was a second smaller semi-anechoic chamber for testing noise emissions from smaller devices as well (e.g. smartphones and tablets), but we didn't spend much time in there as it could only hold a few people and we had already experienced the larger room.



HP Materials Lab

The Materials Lab was more of a forensics sort of lab, where they can look at products using microscopes ranging in power from 10-20X up to 1000X for optical, and they also had a Scanning Electron Microscope for analysis of molecular structures. Other devices included an FTIR (Fourier Transform Infrared Spectrometer) that can be used to identify various organic substances – did your laptop battery really leak on your nightstand, or is that actually cat urine? (Yes, they had someone who thought their – now non-functional – laptop was leaking where the cat turned out to be the culprit.) XRF (X-Ray Fluorescence) meanwhile can be used to determine exactly how much of various metals is present, and whether or not they're the appropriate thickness.

When deeper analysis is required – and when you don't mind destroying the product – the labs can also cut devices and components in half (or pieces) and create polished blocks showing a cross-section of the hardware. These blocks can then be analyzed under a microscope, and again the thickness of various materials is studies. So for example if the solder balls where a chip is mounted onto a PCB are too small, or have too much voiding, they can see this under a microscope.

There was a real-time X-Ray device as well, which allowed them to look at an X-Ray image of an object and move it around, zoom in on areas, etc. Another piece of equipment measured the amount of force required to push/pull on an object, so they could see how much effort is required to remove a PSU from a server, or a card from a PCIe slot, or even a USB device from a USB socket.

At this point we also started to run into areas where the testing overlapped other labs a bit, but each lab focuses on different types of testing. The Materials Lab for instance also had a bunch of thermal cycling chambers, but they were larger than those in the Environmental Lab and could be used for extended testing. One example was a sort of server motherboard mockup that had 48 DIMM slots, and they would place several of these in temperature cycling chambers where they would be subjected to cycles from 0C to 100C and back repeatedly…for around four months! When/if there were failures, they could see these from the measurements and then investigate what happened.

Wrapping up the Materials Lab, we have two final items. First was an area focused on the use of strain gauges. This wasn't something I had encountered before, but in essence strain gauges use a small sensing surface with wires coming from it that can measure the amount of flex/strain placed on a specific part of a product. There was a laptop motherboard where HP was able to determine that one part of the board was flexing too much during installation into the chassis, resulting in failures during the assembly process. Also shown was a laptop where the display was wired up with six points of strain gauges, and they could then measure how much flex in the display and cover occurred when the product was opened/closed, moved around, etc. All of these could be monitored in real-time from a system connected to the other end of the strain gauges.

The final area of the Materials Lab was for HALT (Highly Accelerated Life Test), but we were running short on time so we didn't spend too much time there. In short, HALT uses temperature, vibration, and other stresses to find weaknesses in a product. The testing typically goes beyond specifications to really exercise a product and find out where potential issues may lie rather than simply testing within limits. Again, this is a destructive test so it's not so much "pass/fail" as it is "find out why it failed". HALT is frequently done earlier in the design process to find and correct problems when it's relatively easy to do so.



Reliability Lab

Much of what we saw at the Reliability Lab was just a variation on the Environmental and Materials Labs. Here, complete products – notebooks, desktops, workstations, servers, tablets, etc. – are subjected to vibration, bumps, thermals, and even electromagnetic discharge. One demonstration had a "gun" of sorts that would send out a controlled static shock. There was an older HP EliteBook laptop (which had likely been abused for many such demonstrations) where the shock caused it to reset. Another thermal chamber was set to 85F and had a Windows tablet running a loop of 3DMark 06 while connected to an external display.

Besides the temperature/vibration/electromagnetic testing, the Reliability Lab also had long-term wear testing that was taking place. One station had a machine that would repeatedly open and close a couple of laptops several times a minute (and one display had a section that had failed during this test). Another test was for a barcode scanner and magnetic card reader, which had a robotic arm pressing the scan button and periodically swiping a card. Elsewhere there was a "keyboard exerciser" where keys on a keyboard would be pressed a couple times per second, and there was a separate machine in the room that could be used to measure the amount of force required to actuate the keys on a keyboard – or the buttons on a mouse – so that they could see how the keyboards would wear over time. In a related set of tests, keyboards were tested for wear and tear on their lettering with a robotic arm constantly rubbing the keys, and a similar device would press on the edges of a laptop display repeatedly.

HP noted that testing in many of these apparatuses might go on for weeks or longer to see how they would hold up under continued use (and abuse). While I can't comment specifically on how well the testing mimics real-world usage (e.g. the nub rubbing on a keyboard isn't quite the same as an actual finger that will leave oily residue), it's definitely good to see this level of attention to detail.

Electromagnetic Lab

The final lab was actually at a separate HP location, located a few miles away from the main HP Houston campus. The Houston Regulatory Lab has been around for 30 years and is used for testing compliance with various government regulations. When it was first created, it had several open air testing areas – it was miles away from any other major buildings so this was sufficient. Over time, the proliferation of new technologies eventually required the creation of shielded environments.

The most impressive was a 10-meter long room (probably 4 meters wide and 4 meters tall as well), which at first looks like another Semi-Anechoic chamber. In fact this is actually a room for testing radio emissions from a device, with two large antenna structures located on different ends of the room. The outside of the room is shielded with thousands of ferrite squares, and inside of those are the usual pyramid-shaped foam structures you'd see in a semi-anechoic chamber, only these are of different sizes and they are also carbon laced to help them absorb radio waves. The larger "spikes" are on the most reflective surfaces, while smaller spikes are used near the corners of the room where the reflected waves are apparently easier to absorb (or something like that). Needless to say, the instant they closed the door to the room all of our cell phones immediately lost connection, though all it took was for the door to be opened and even twenty feet away I was able to reconnect to the HP wireless network.

Other testing done at this lab included testing of WiFi transmit strength in the Satimo chamber. Here they had a "Stargate" ring of antennas around a small pedestal. The pedestal could then rotate while the ring of antennas would take readings, and the result is a chart of WiFi strength for a device. Ideally you want a device to have equally good reception in every direction, so that it doesn't matter where the users is facing or how they're holding the device. The result of the testing would generate color blobs that show signal strength.

Finally, there were additional rooms testing how devices would react to various forms of interference; this was called "immunity" testing. Some of the interference would be induced over the power cables, USB cables, serial/printer cables, display cables, etc. Tests were also done for brown-outs, voltage interrupts, power surges, and so forth. One test had equipment that would simulate a lightning strike on the power grid as one extreme example, and nearby was another electrostatic testing device where they could repeatedly shock laptops and other devices to see how they would respond.



HP Tour Wrap-Up

And there you have it: all of the testing that goes into HP's commercial line of products. In other words, this testing is done on their enterprise class hardware and software, and considering the cost of all the testing facilities you can understand why such hardware tends to cost quite a bit more than consumer hardware. Companies always talk about how much more testing and validation goes into their business class products, but this is the first time I've really had a chance to tour a major OEM and see exactly what sort of testing they do. It was an informative experience for sure, though at the end of the day it only makes me wish consumer products could receive a bit more TLC.

One thing I can't really comment on unfortunately is how HP's testing compares to that of other companies like Dell, Lenovo, etc. Do they do more than the competition, or "about the same"? My understanding is that HP does most of their testing in-house while Dell as an example apparently outsources a lot of testing, but I'm not sure if that's true – or if it's really even "bad" to outsource testing to a third-party lab. There are potential advantages to both types of testing (and I'm sure HP has third-party labs that help with testing at times as well), and ultimately it's the end user experience that matters.

To that end, I can say that my experience over the years with all of the mobile workstations has been quite good. Such notebooks aren't nearly as portable as an Ultrabook, and in some cases the design language can be a bit stale, but if someone tells me they want a laptop that will last at least five years before it needs to be replaced, my advice is always going to be: "go check out the business laptops from the major OEMs; they cost more but if you want high-end materials and build quality that's the best way to get it." By comparison, consumer laptops tend to be built with a disposable mentality – they need to last at least a year without too many failures, but after that all bets are off. I've seen budget consumer laptops that keep on kicking for five years, sure, but I've also seen many that go belly up within two or three years. Whether it's worth paying the premium for a more robust build quality is debatable, but there's no question that enterprise grade hardware has to pass far more stringent standards.

There were a few additional demonstrations of the robustness of HP's hardware, though I unfortunately didn't get videos/pictures as I wasn't in a good location. In our pre-tour briefing, one of the engineers stood dropped and ElitePad prototype on the ground and then stood on top of it, and of course it continued to work without trouble. Someone also mentioned a demonstration of where one of the project managers for the Z-line of workstations stood on a ZBook 15 that was lying on the floor with one foot (so all of his weight was on that foot), but I didn't personally witness that feat – though I have seen videos of similar demonstrations on YouTube. (Note to self: don't try this at home!)

Wrapping things up, the historical aspects of the tour were also entertaining. I saw quite a lot of old hardware still kicking throughout the labs – including many of the test areas that were still using old Compaq 486 machines (unless the internals had been upgraded). I also saw quite a few instances of Windows XP and even Windows 2000 still running; why replace something that still works, right? There were old posters from HP and Compaq scattered around the facilities as well – anyone looking for a hot new Compaq 386/33? (Yes, I actually lusted after such systems at one point, as I first started using PCs in the 8088/8086 era; the first PC I purchased with my own money was a 286/12 if you're wondering – and it only cost $2000!)

As for the HP employees, I was surprised how many people hailed back to the Compaq days, as the merger took place around 2001. The Houston campus was originally part of Compaq, so maybe it's not that surprising, but there were a lot of section heads that have been with the company for more than 20 years. There was also plenty of "new blood" kicking around as well, of course. Will any of them be giving me a tour in another 20 years? That's probably a question best not to ponder, but if I'm still around I'll be sure to grab my walker! And if you want a few additional images, the last gallery includes all the pictures I snapped on my smartphone (most of which were already posted via Twitter).

Log in

Don't have an account? Sign up now