Power Consumption and Thermal Performance

The pfSense installation in the Supermicro SuperServer E302-9D was configured with the default ruleset first, as described in an earlier section. The only free 1000Mbps LAN port was then connected to a spare LAN port in the sink. Two instances of the iPerf3 server were initialized on each of the sink's interfaces connected to the DUT. The firewall rules for the newly connected interface were modified to allow the benchmark traffic. Two iPerf3 clients were started in each connected interface of both the source and the conductor - one in normal, and the other in reverse mode. This benchmark was allowed to run for one hour in an attempt to saturate the duplex link of all of the DUT's interfaces (other than the ones connected to the management network and IPMI). After one hour, the source and sink were turned off, followed after some time by the DUT itself. The power consumption at the wall was recorded during the whole process using an Ubiquiti mFi mPower unit.

The E302-9D pfSense installation idles at around 70W. At full load with all network interfaces active, the power consumption reaches 90W. Keeping just the IPMI active (allowing for the BMC to remotely power up the server) costs slightly more than 15W. Keeping in mind the target market for the system, it would be good on Supermicro's part to see if the 15W number can be reduced further. pfSense / FreeBSD is not a particularly power-efficient OS. Having observed the idle power consumption of both Windows Server 2019 and Ubuntu 20.04 LTS on the same system to be in the high 40s, the inefficiency of pfSense was slightly disappointing. In common firewall applications / deployments in datacenters and server racks, this is not much of a concern (as the system is likely to be never idle), but, embedded applications may not always be in high-traffic mode. Some optimizations from the OS side / Intel drivers may help here.

Towards the end of the stress test, we also captured the temperature sensors' outputs as conveyed by Supermicro's IPMIView tool. The CPU temperature of 73C was well within the 90C limit. However, the SSD was a bit too hot at 82C throughout, as was the MB_10G, VRM, and DIMMs between 80 and 92C. The SSD was partly our fault (the power-hungry Mushkin Sandforce-based SATA SSD is definitely not something to be recommended for a tightly enclosed passively cooled system like the E302-9D, particularly when the SSD makes no contact with the metal casing).

The FLIR One Pro thermal camera was used to take thermal photographs of the chassis at the end of the stress test. Temperatures of around 70C were noticed at various points.

Under normal conditions with light traffic (i.e, power consumption remaining around 70W), temperatures were around 60C - 65C. Additional thermal photographs are available in the above gallery. Given the temperature profile, the unit is best placed away from where one's hands might accidentally touch it.

On the thermal solution itself, Supermicro has done an excellent job in cooling down the SoC (as evident from the bright spot directly above the SoC in the thermal photographs and the teardown gallery earlier in the piece). The heat-sink is well thought-out and blends well with the chassis in terms of contact area. It is not clear whether anything can be done about the VRMs and DIMMs, but users should definitely consider a low-power SSD or ensure that the installed SSD has a chance for its heat to be conductively taken away.

Packet Processing Benchmarks with pkt-gen Miscellaneous Aspects and Concluding Remarks
Comments Locked

34 Comments

View All Comments

  • GreenReaper - Tuesday, July 28, 2020 - link

    The D-1541 only gets ~160% of the performance, that is - under ideal conditions. In practice we tend to average one to two core usage; and scaling for DB operations falls off after four, so the D-1521 may have been the faster CPU for us. (It also meant it was cheaper, yet came with NVMe SSD.)
  • herozeros - Saturday, August 1, 2020 - link

    Had no idea on the price jump on SoC with quickassist, question answered thoroughly, cheers!
  • TrevorH - Tuesday, July 28, 2020 - link

    I notice that it does have an HTML5 remote console so it's not locked to java for that.
  • GreenReaper - Tuesday, July 28, 2020 - link

    I'd love one of these under my desk to go with my HP MicroServer Gen8. Can't justify it, of course, but maybe in a few years they'll end up available at clearance prices or on the second-hand market.
  • Foeketijn - Wednesday, July 29, 2020 - link

    I am hoping for a ryzen gen 11. So far I've skipped the gen 10.
    Microserver without IPMI/iLo. Thats just silly.
  • Spunjji - Wednesday, July 29, 2020 - link

    +1 on that. Don't even care if it's Zen 1 or Zen+ for cost reasons - seems like the perfect fit.

    Raven Ridge would also be a solid option.
  • hrana - Tuesday, July 28, 2020 - link

    Great review but I need some context with your testing methodology. How do the 8C, 12C, and 16C variants perform? If I want a 10G router for everything except IPsec, what do I need today in terms of hardware today for pfsense? Some say pf has its own limitations such that throwing hardware at it is not successful. It would be good if your team could help us better understand using the above methodology.
  • Bp_968 - Tuesday, July 28, 2020 - link

    I wasn't terribly impressed with PFsense. It was blocking my own website (hosted on godaddy at the time and running WordPress) and was blocking it without any explanation or reasonable way to stop blocking it. I dropped by the forums and tried to get some help and instead got 3 pages of tinfoil hat paranoia about how I was probably a russian hacker trying to take over their machines through the forum. This is the offical pfsense forum btw... one guy finally decided I wasn't smart enough to be a russian hacker and then more or less threw his hands up saying sometimes it doesnt like certain types of traffic/websites/etc but hopefully it will get fixed in the future.

    It finally was fixed, by a Ubiquiti edgerouter.
  • ruthan - Wednesday, July 29, 2020 - link

    Can someone explain me, why to paid $1500 for overprice network switch with just 2 x 10 Gb/s ports? What is wrong with classic networking hardware - standalone boxes?
  • PeachNCream - Wednesday, July 29, 2020 - link

    There's flexibility to do more with this system than merely act as a network switch since its running general purpose hardware. Is that worth $1500 if all you need is a switch? Of course not - go buy a switch and save some money.

Log in

Don't have an account? Sign up now