Supermicro SuperServer E302-9D Review: A Fanless 10G pfSense Powerhouse
by Ganesh T S on July 28, 2020 3:00 PM EST- Posted in
- Networking
- Intel
- Supermicro
- 10GBase-T
- Xeon-D
- SFP+
- 10GbE
- ASpeed
- Skylake-D
pfSense Configuration for Benchmarking
A perusal of the FreeBSD firewall performance evaluation guidelines and the accompanying infrastructure helped us narrow down the scope of testing. As elaborated in the section covering the testing methodology, the DUT was configured in various states and the iPerf3 regular TCP benchmark and the pkt-gen sweep for different packet sizes were run for traffic passing through the firewall. A test of the L3 forwarding capabilities of the DUT was also performed using the ipgen benchmark while keeping in mind its stimulus-generating machine limited nature.
Supermicro E302-9D as pfSense Firewall - Benchmarked Modes | |
Mode | DUT Commands / Rules |
Router | sysctl net.inet.ip.forwarding=1 pfctl -d |
PF (No Filters) | sysctl net.inet.ip.forwarding=1 pfctl -e pfctl -F all |
PF (Default Ruleset) | sysctl net.inet.ip.forwarding=1 pfctl -e (Additional firewall rules specified at end of sub-section) |
PF (NAT Mode) | sysctl net.inet.ip.forwarding=1 pfctl -e pfctl -F all -f /home/username/nat.pf |
PF (IPSec) | sysctl net.inet.ip.forwarding=1 pfctl -e (Additional firewall rules specified at end of sub-section) |
The table above summarizes the different states of evaluation and the shell commands used to place the DUT in that mode.
The additional firewall rules for the PF (Default Ruleset) case (added using easyrule / firewall log view) are as below:
pass in quick on ixl2 inet from 172.16.0.0/24 to 172.16.1.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on ixl2 inet from 172.16.0.0/24 to 172.16.10.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on ixl3 inet from 172.16.1.0/24 to 172.16.0.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on ixl3 inet from 172.16.1.0/24 to 172.16.11.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on ixl0 inet from 172.16.10.0/24 to 172.16.0.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on ixl0 inet from 172.16.10.0/24 to 172.16.11.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on ixl1 inet from 172.16.11.0/24 to 172.16.1.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on ixl1 inet from 172.16.11.0/24 to 172.16.10.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on igb3 inet from 172.16.20.0/24 to 172.16.21.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on igb2 inet from 172.16.21.0/24 to 172.16.20.0/24 flags S/SA keep state label "USER_RULE"
The contents of the /home/username/nat.pf file referenced in the PF (NATMode) row of the table are as below:
set limit states 100000000
nat on ixl0 from 172.16.0.0/16 to any -> ixl0
nat on ixl1 from 172.16.0.0/16 to any -> ixl1
nat on igb2 from 172.16.0.0/16 to any -> igb2
pass in quick all keep state
pass out quick all keep state
The IPsec evaluation doesn't follow the steps outlined for the other modes. Instead of using both the source and the sink, along with iPerf3 and pkt-gen programs running on either side, only the source and the DUT are used. A baseline iPerf3 run between the source and the DUT (with no IPsec communication) is used for comparison. The communication between the two sets of ports is configured for IPsec using the script template below (invoked from the shell as an argument to the setkey -f command). The previous security policies and associations are flushed prior to the invocation.
flush;
spdflush;
# Host to host ESP
# Security Associations
add 172.16.0.2 172.16.0.1 esp 0x10001 -E
add 172.16.0.1 172.16.0.2 esp 0x10002 -E
add 172.16.1.2 172.16.1.1 esp 0x10003 -E
add 172.16.1.1 172.16.1.2 esp 0x10004 -E
# Security Policies
spdadd 172.16.0.2 172.16.0.1 any -P in IPsec esp/tunnel/172.16.0.2-172.16.0.1/require;
spdadd 172.16.0.1 172.16.0.2 any -P out IPsec esp/tunnel/172.16.0.1-172.16.0.2/require;
spdadd 172.16.1.2 172.16.1.1 any -P in IPsec esp/tunnel/172.16.1.2-172.16.1.1/require;
spdadd 172.16.1.1 172.16.1.2 any -P out IPsec esp/tunnel/172.16.1.1-172.16.1.2/require;
The template above is for the DUT side, with the one on the source side being similar (the in and out are reversed in the security policies section).
The next section provides additional benchmark processing details along with the results for both iPerf3 and ipgen tests. That is followed by a discussion of pkt-gen benchmark results.
34 Comments
View All Comments
GreenReaper - Tuesday, July 28, 2020 - link
The D-1541 only gets ~160% of the performance, that is - under ideal conditions. In practice we tend to average one to two core usage; and scaling for DB operations falls off after four, so the D-1521 may have been the faster CPU for us. (It also meant it was cheaper, yet came with NVMe SSD.)herozeros - Saturday, August 1, 2020 - link
Had no idea on the price jump on SoC with quickassist, question answered thoroughly, cheers!TrevorH - Tuesday, July 28, 2020 - link
I notice that it does have an HTML5 remote console so it's not locked to java for that.GreenReaper - Tuesday, July 28, 2020 - link
I'd love one of these under my desk to go with my HP MicroServer Gen8. Can't justify it, of course, but maybe in a few years they'll end up available at clearance prices or on the second-hand market.Foeketijn - Wednesday, July 29, 2020 - link
I am hoping for a ryzen gen 11. So far I've skipped the gen 10.Microserver without IPMI/iLo. Thats just silly.
Spunjji - Wednesday, July 29, 2020 - link
+1 on that. Don't even care if it's Zen 1 or Zen+ for cost reasons - seems like the perfect fit.Raven Ridge would also be a solid option.
hrana - Tuesday, July 28, 2020 - link
Great review but I need some context with your testing methodology. How do the 8C, 12C, and 16C variants perform? If I want a 10G router for everything except IPsec, what do I need today in terms of hardware today for pfsense? Some say pf has its own limitations such that throwing hardware at it is not successful. It would be good if your team could help us better understand using the above methodology.Bp_968 - Tuesday, July 28, 2020 - link
I wasn't terribly impressed with PFsense. It was blocking my own website (hosted on godaddy at the time and running WordPress) and was blocking it without any explanation or reasonable way to stop blocking it. I dropped by the forums and tried to get some help and instead got 3 pages of tinfoil hat paranoia about how I was probably a russian hacker trying to take over their machines through the forum. This is the offical pfsense forum btw... one guy finally decided I wasn't smart enough to be a russian hacker and then more or less threw his hands up saying sometimes it doesnt like certain types of traffic/websites/etc but hopefully it will get fixed in the future.It finally was fixed, by a Ubiquiti edgerouter.
ruthan - Wednesday, July 29, 2020 - link
Can someone explain me, why to paid $1500 for overprice network switch with just 2 x 10 Gb/s ports? What is wrong with classic networking hardware - standalone boxes?PeachNCream - Wednesday, July 29, 2020 - link
There's flexibility to do more with this system than merely act as a network switch since its running general purpose hardware. Is that worth $1500 if all you need is a switch? Of course not - go buy a switch and save some money.