Scale-Out Big Data Benchmark: ElasticSearch

ElasticSearch is an open source, full text search engine that can be run on a cluster relatively easy. It's basically like an open source version of Google Search that can be deployed in an enterprise. It should be one of the poster-children of scale-out software and is one of the representatives of the so called "Big Data" technologies. Thanks to Kirth Lammens, one of the talented researchers at my lab, we have developed a benchmark that searches through all the Wikipedia content (+/- 40GB). Elasticsearch is – like many Big Data technologies – built on Java.

We are not sure why, but installing IBM's JDK caused a lot of headaches. For some reason the JVM stopped working in the middle of our tests. We got the same behavior running Apache Spark. This could be a result of our lack of experience with the IBM JDK, or the fact that the Linux LE ecosytem is still young. To cut a long story short, we ended up useing OpenJDK 8, which is part of the Ubuntu 15.04 distribution. OpenJDK is very similar to and based upon the same code as Oracle's HotSpot JDK.

We limited the systems to one socket to avoid the issues associated with garbage collection pauses and other scaling issues. There is reason why many Java benchmarks on these massive machines are using multiple JVMs.

Elastic Search

Although the POWER8 can probably perform a bit better with the IBM JDK, performance is in the same league as the best Xeons. Meanwhile as a further point of comparison we also included the score of the Xeon D from our previous article.

Database Performance: MySQL Energy and Pricing
Comments Locked

146 Comments

View All Comments

  • FunBunny2 - Friday, November 6, 2015 - link

    "The z10 processor was co-developed with and shares many design traits with the POWER6 processor, such as fabrication technology, logic design, execution unit, floating-point units, bus technology (GX bus) and pipeline design style, i.e., a high frequency, low latency, deep (14 stages in the z10), in-order pipeline." from the Wiki.

    Yes, the z continues the CISC ISA from the 360 (well, sort of) rather than hardware RISC, but as Intel (amongst others) has demonstrated, CISC ISA doesn't have to be in hardware. In fact, the 360/30 (lowest tier) was wholly emulated, as was admitted then. Today, we'd say "micro-instructions". All those billions of transistors could have been used to implement X86 in hardware, but Intel went with emulation, sorry micro-ops.

    What matters is the underlying fab tech. That's not going anywhere.
  • FunBunny2 - Friday, November 6, 2015 - link

    ^^ should have gone to KevinG!!
  • Kevin G - Saturday, November 7, 2015 - link

    The GX bus in the mainframes was indeed shared by POWER chips as that enabled system level component sharing (think chipsets).

    However, attributes like the execution unit and the pipeline depth are different between the POWER6 and z10. At a bird's eye view, they do look similar but the implementation is genuinely different.

    Other features like SMT were introduced with the POWER5 but only the most recent z13 chip has 2 way SMT. Features like out-of-order execution, SMT, SIMD were once considered too exotic to validate in the mainframe market that needed absolute certainty in its hardware states. However, recent zArch chips have implemented these features, sometimes decades after being introduced in POWER.

    The other thing is that IBM has been attempting to get get more and more of the zArch instruction set to be executed by hardware and no microcode. Roughly 75% to 80% of instructions are handled by microcode (there is a bit of a range here as some are conditional to use microcode).
  • JohanAnandtech - Saturday, November 7, 2015 - link

    I believe that benchmark uses about 8 threads and not very well either? Secondly, it is probably very well optimized for SSE/AVX. So you can imagine that the POWER8 will not be very good at it, unless we manually optimize it for Altivec/VSX. And that is beyond my skills :-)
  • UrQuan3 - Monday, December 21, 2015 - link

    I'm sure no one is still reading this as I'm posting over a month later, but...

    I tested handbrake/x264 on a bunch of cross-platform builds including Raspberry Pi 2. I found it would take 24 RPi2s to match a single i5-4670K. That was a gcc compiled handbrake on Raspbian vs the heavily optimized DL copy for Windows. Not too bad really. Also, x264 seems to scale fairly well with the number of cores. Still, POWER8 unoptimized would be interesting, though not a fair test.

    BTW, I'd encourage you to use a more standard Linux version than 6-month experimental little-endian version of Ubuntu. The slides you show advertise support for Ubuntu 14.04 LTS, not 15.04. For something this new, you may need the latest, but that is often not the case.
  • stun - Friday, November 6, 2015 - link

    @Johan You might want to fix "the platform" hyperlink at the bottom of page 4. It is invalid.
  • JohanAnandtech - Friday, November 6, 2015 - link

    Thanks and fixed.
  • Ahkorishaan - Friday, November 6, 2015 - link

    Couldn't read past the graphic on page 1. It's 2015 IBM, time to use a font that doesn't look like a toddler's handwriting.
  • xype - Sunday, November 8, 2015 - link

    To be fair, it seems that the slide is meant for management types… :P
  • Jtaylor1986 - Friday, November 6, 2015 - link

    Using decimals instead of commas to denote thousands is jarring to your North American readers.

Log in

Don't have an account? Sign up now