Online gambling firm Bet365 has swapped Java for a lesser known programming language, Erlang, enabling it to reduce the complexity of software underpinning its core betting platforms and scale systems to handle a fourfold increase in the number of users on its network.
Created by developers at Ericsson two decades ago to run its telephone exchanges, Erlang is a relatively simple programming language designed for large, high-availability applications. It garnered little attention outside of the telecoms industry since the mid-nineties, but has grown in prominence in recent years, with WhatsApp using it to handle billions of messages sent across its network each day.
According to Bet365’s R&D manager, Dan Macklin, the language is also well suited to reducing the complexity of its low-latency systems, and is now being used in production to drive its InPlay betting service, pushing live odds of sporting events to millions of customers in near real-time.
“The reason we chose Erlang was largely that we could see the software we had was getting increasingly complicated, and it was getting more and more difficult to scale,” says Macklin.
“The complexity meant it takes us much longer to get releases out, while the software becomes much more difficult to test, and it made it harder to innovate.
“Erlang has the functionality for scalability built into it, but it is also coupled in a very small language, so it allowed us to scale our system as the number of lines of code was drastically reduced. It meant that we could get our software out there faster, and we could test it quicker, and from that we could increase the productivity of our developers.”
Research and development
Macklin heads up a small team of developers as part of an R&D function created at Bet365 three years ago. The team's aim is to trial technologies that could give the company an advantage in the highly competitive online betting market.
“Our main remit is to challenge conventions so we try to look at different ways of solving problems, but ultimately with a focus on using different technologies to simplify the way that we do things,” he says.
The team began to look at how NoSQL and alternative programming languages can be used to solve the problem of growing complexity of its systems, which were unable to cope with growing customer demand and the need to develop software for modern servers which rely on multiple processor cores to improve performance. This meant that developers were forced to spend too much time maintaining existing systems to cope with bigger workloads, rather than focusing on creating new services.
“A big problem we have come across is the scale and innovation dilemma: Bet365 wants our development team to be building leading-edge products that really excite customers, but, because of the growth in the business, we get into this spiral where most of the development time is spent re-engineering existing systems to cope with a load that is coming,” Macklin says.
“Scaling problems are getting increasingly complicated in a multi-core world, and, when you do get time to innovate, you are worried that innovation may not run at the scale we need it to run at.”
The company decided to adopt Erlang to make its Java-based InPlay betting service, which handles up to two million concurrent users at peak times. Macklin says that the idea of using Erlang three years ago was unusual, but made sense for an application that required so much parallel processing - a similar issue to that of Ericsson's developers.
“The problems that the telephone switch providers had to deal with in the nineties are very similar to the kind of problems we have today: reliability and scalability. Back in the day with telephone switches you had lots of message routing going on, which is really an embarrassingly parallel problem [a workload comprising of many identical elements that can be separated],” says Macklin.
He adds that, to scale systems effectively in this manner, a simplified code base is required - and Erlang is tailored to this task. "One of our biggest successes was to apply that to the InPlay system,” he says.
The firm can now manage sharp increases in demand during busy periods - such as major sporting events - without slowing service to customers.
“Our code base is vastly simplified now, and we have less issues with reliability. We are an incredibly peaky business and we find that at the peak of trading we are able to provide our customer with a much more predictable quality product using Erlang," Macklin says.
"On a busy Saturday when we're pushing lots of data, we can maintain a consistent speed of updates no matter what load is placed on the system."
The success of the project led to the language being used to provide the middleware for an entirely new system: the creation of Bet365's Cash Out service. This allows customers to close their position early and take a return before an event - such as a football match - has finished.
“The problem with that is that the same trading data that we have in the InPlay systems that we need to get to our customers at very low latency, we need to apply it to a user’s open bet to generate a price,” he explains.
“That is quite hard when you have got as many open bets and data changes as we have. Again it was about how we could leverage Erlang into that world.”
Moving from R&D to the rest of the business
Aside from the technical hurdles, one of the main challenges R&D faced was creating the skills base within the wider developer teams to enable the move to Erlang. “It is one of those things where there is a bit hump to get into it. You look at the whole functional programming, the syntax and so forth, and it definitely requires an effort to get into it. But once you get people through that initial pain barrier, the developers tend to really, really like it," Macklin insists.
“It has the right amount of pragmatism to enable people to deliver quickly, with the right amount of geekiness to make it fun to work with. It is important to make it fun and intriguing, so developers can be passionate about it.”
After initially engaging with a service provider, Erlang Solutions, Bet365 quickly built its own expertise and now it trains its own developers. Around 10 percent of the 300 IT staff at the company are now proficient with the language, and Macklin expects it will be used more widely in applications in future. However, the firm will continue to rely on other languages, such as Java and .NET, which are better suited to certain tasks.
“Erlang is not useful for absolutely everything, but we have a roadmap of what we will be using it for in the next year. It has a very large applicability for some of our large-scale use cases,” he says.
Macklin adds that, despite not having the same profile as Scala or other emerging languages, it is likely to continue to gain traction with other businesses.
“Erlang may have been around for a while but that isn’t to its detriment – it is actually one of its strengths and it shows the pedigree,” he says.
"The big turning point – although we were looking at it before this – was the success of WhatsApp, and the scalability of what they get out of their platform and what they are able to deliver with such a small development team. If you wanted a poster child for an organistion that has used Erlang and has been massively successful then they would be one.
“It might not be as big as any of the other main players, so to speak. But the effect that it has had on our business has been really good, and it has been very effective in the problems we have used it for.”