Apache Spark 2.x Benchmarking

Last, but not least, we have Apache Spark. Apache Spark is the poster child of Big Data processing. Speeding up Big Data applications is the top priority project at the university lab I work for (Sizing Servers Lab of the University College of West-Flanders), so we produced a benchmark that uses many of the Spark features and is based upon real world usage.

The test is described in the graph above. We first start with 300 GB of compressed data gathered from the CommonCrawl. These compressed files are a large amount of web archives. We decompress the data on the fly to avoid a long wait that is mostly storage related. We then extract the meaningful text data out of the archives by using the Java library "BoilerPipe". Using the Stanford CoreNLP Natural Language Processing Toolkit, we extract entities ("words that mean something") out of the text, and then count which URLs have the highest occurrence of these entities. The Alternating Least Square algorithm is then used to recommend which URLs are the most interesting for a certain subject.

To get better scaling, we run with 4 executors. Researcher Esli Heyvaert reconfigured the Spark benchmark so it could run on Apache Spark 2.1.1.

Here are the results:

Apache Spark 2.1.1

(*) EPYC and Xeon E5 V4 are older results, run on Kernel 4.8 and a slightly older Java 1.8.0_131 instead of 1.8.0_161. Though we expect that the results would be very similar on kernel 4.13 and Java 1.8.0_161, as we did not see much difference on the Skylake Xeon between those two setups.

Data processing is very parallel and extremely CPU intensive, but the shuffle phases require a lot of memory interactions. The time spent on storage I/O is negligible. The ALS phase does not scale well over many threads, but is less than 4% of the total testing time.

The ThunderX2 delivers 87% of the performance of the twice as expensive EPYC 7601. Since this benchmark scales well with the number of cores, we can estimate that the Xeon 6148 will score around 4.8. So while the ThunderX2 can not really threaten the Xeon Platinum 8176, it gives the Gold 6148 and its ilk a run for their money.

Java Performance: Huge Pages Investigated What We Can Conclude: So Far
Comments Locked

97 Comments

View All Comments

  • imaheadcase - Wednesday, May 23, 2018 - link

    I really think Anandtech needs to branch into different websites. Its very strange and unappealing to certain users to have business/consumer/random reviews/phone info all bunched together.

    Ever since anand actually left it really did venture into more a business/insider based website with random stuff thrown in. It is in no way a bad thing, its just like this review for instance would not appeal to %95 of readers normally. Everyone likes technology naturally that comes to this website, but its a fine line between talking about high end server components that are out of reach to people who just read the article on the mini-itx gaming motherboard. lol
  • Andrei Frumusanu - Wednesday, May 23, 2018 - link

    You're always free to skip articles, nobody's forcing you to read it.
  • boeush - Wednesday, May 23, 2018 - link

    I guess he'd prefer the site content to be grouped in some manner roughly mirroring market segmentation. For instance: consumer, professional, enterprise, exotic/HPC. As opposed to jumbling everything together. Personally, I don't mind - but then, I'm not known for obsessive-compulsive organizing, either :)
  • BurntMyBacon - Thursday, May 24, 2018 - link

    Given the large differences in tech, focus, needs, and trends, I wouldn't mind breaking out Phones and perhaps servers into their own sections. I think there is more than enough overlap to keep consumer and professional desktop/laptop/workstation together, but that is entirely up to how deeply you want to divide things up. On the other hand, you'll want all of it to show up on the front page in some form, or it'll look like the site doesn't have much activity. Perhaps separate pipelines for each category could work. That all said, I don't really mind just skipping over articles that don't interest me. :)
  • imaheadcase - Thursday, May 24, 2018 - link

    Please, that is just lazy excuse. Even news websites have catagory based on the news you interested in. Anandtech literally had a review of a gaming motherboard then a high end server thing, and newz feed gets filled with phone and other news.
  • name99 - Thursday, May 24, 2018 - link

    God, you must REALLY hate Twitter then...

    I argue with Andrei a lot, but every so often he writes a sentence like "You're always free to skip articles, nobody's forcing you to read it" that makes me want to clap him on the back and say "yes, YOU get it" :-)
  • Threska - Sunday, May 27, 2018 - link

    Taken to it's logical extreme the front page could be a dumping ground cesspool and the retort would be "you don't have to wade through any of it" which sounds witty but doesn't solve anything, but over time would lead to the predictable outcome of people leaving.
  • imaheadcase - Sunday, May 27, 2018 - link

    I do hate twitter, but because it has no valid purpose other than to get customer service done faster with companies because it reflects more on them because public venue. Its mostly just a rant inducing place, or a place that is basically just texting anyways since everyone just wants you to send a DM.

    The whole idea of saying "you are free to skip it" is kinda silly thing to say on the internet now. Especially since more and more you can filter things according to what you want. Not only that, but with the tight competition with views from tech websites its in best interest to have more options.

    Even the layout of website never changed. I mean have you ever been to website without a adblocker on? They don't even advertise tech related stuff on it. Its just stupid clickbait stuff.

    Keep in mind, this is not a complaint about articles itself, its just how they are posted. I love this site, been coming to it ever since i built first pc when i was a kid. But its focus is all over the place now vs years ago out what its posting. I'm half thinking one day i will see a review of electronic toothbrush then next day new CPU.
  • GreenReaper - Monday, June 4, 2018 - link

    I'd be fine with that, as long as it was the best darn toothbrush in town!
  • Threska - Sunday, May 27, 2018 - link

    Accessing through RSS might be a better solution especially with a good reader. Just needs accurate tags to match.

Log in

Don't have an account? Sign up now