As a top real-time, mobile-first sports website, Bleacher Report has to be able to handle vast traffic spikes. The site gets 1.5 billion page views per month and 250,000 users at its peak, and sends out more than three billion push notifications each month.
The site was originally written on the Ruby on Rails framework but Bleacher Report reached the point where they could no longer scale it, according to Dave Marks, BR's senior engineering director.
BR had some "pretty unique" challenges around concurrency, as users see vast streams of content merged together when they log into the Team Stream app, he explains. "This was very expensive from a resource perspective and not easily cacheable as it's unique to each user," Marks adds.
The 50-strong development team considered various options, including Node.js and Go, but settled upon Elixir, a programming language built on top of Erlang, which is 31 years old and was originally built for the telecoms industry. About 15 people work solely on Elixir within BR, he explained.
"It allows you to do lots of things very quickly and is based on lots of processes all happening at once, which is very good for what we do," says lead engineer Ben Marx. "We chose Elixir for those performance advantages, and also similarities to Ruby syntax."
Marx, who joined in May 2015, partly came to BR because he was interested in learning more about Elixir and it was one of the biggest adoptions yet of the language, which is about five years old.
"We were one of the first. José Valim, the creator of Elixir, was in town so came by and gave a talk. Chris McCord, creator of Phoenix, is always on Slack and so on. A consulting company published a lot on how to use it. So it was mainly trial and error, reading online, figuring out what to do," he says.
BR was helped by Erlang Solutions, which came to do a quality assurance check on its apps when they were released to production for the first time in Elixir in August 2015.
"I talked to some CTOs I know who used Erlang and they all recommended Erlang Solutions. They were able to come in with expertise, help us with best practice and give us confidence going forward that the systems would be efficient and reliable," Marks said.
Benefits of Elixir
One of the main benefits the team has found is that Elixir runs on a fraction of the infrastructure previously required and without external vendor dependencies, saving BR vast sums of money, according to Marks.
"Elixir has proven so efficient that testing the limits of our services became a challenge unto itself, requiring investment in new benchmarking tools and strategies. In a recent test, for example, our most heavily trafficked service was able to handle 8x our average traffic load, without autoscaling, before the database proved to be a bottleneck," he said.
The new language has led to cleaner code base and much less technical debt, according to Marx. It has also increased the speed of development and some impressive performance improvements.
"On our monolith we needed roughly 150 servers to power the more intensive portions of BR. Following our move to Elixir we're now able to power those same functions on five servers and we're probably overprovisioned. We could probably get away with it on two," Marks says.
For dev teams considering adopting Elixir, the basic advice is to start with something small, according to Erlang Solutions' head of Elixir Claudio Ortolina.
"Most get good results in the first three to six months of development, then you can dig a big deeper," he says.
"Elixir is code that can do many things at once. It has a solid, bullet proof approach but it requires rewiring your brain a bit as a developer, especially if you've never come across it before," Ortolina adds.