Comic Relief’s chief technology officer goes to bed dreaming of load tests. It’s hardly surprising as Red Nose Day, unlike any other retailer promotion or ticketing operation, has just one 24-hour window in which people won’t return if they can’t donate on the first instance.
To counteract this worst case scenario, “you have to build for failure, because something will go wrong”, says Zenon Hannick, who experienced his first Red Nose Day as CTO on Friday.
During previous years Comic Relief has processed 250 donations per second throughout the event. While the charity couldn’t reveal the exact figures behind traffic and donations, it did confirm it broke its own fundraising record over the weekend.
Preparation is crucial for supporting such a spike in demand. And building for failure, on Comic Relief terms, is a combination of game-day rehearsals and reliance on a host of suppliers who can switch between availability zone within the datacentre in a flash.
The fifty-strong in-house technology team build these “multiple layers of redundancy” all year round. Thanks to their hard work, Hannick says the likelihood of the IT for Friday night’s donation spree falling over, was as probable as a “breakdown of the internet as a whole.”
How high are the stakes?
Unlike any other business, the UK public will not return to the rednoseday.com to donate if they are met with a 404 error notice. Unlike sports venues, or Glastonbury, for example, donators won’t refresh or join a queue to give their money away.
Because of this, Hannick and his team go to sleep sweating in the run up to Red Nose Day. If Comic Relief was unable to take the donations from the great British public, the hard work the charities and the BBC does to support those in need, would be in vain, he says.
The heart of Comic Relief’s estate, its donations platform, is built in partnership with Armakuni on Cloudfoundry and hosted on three stacks, two from AWS and one instance on hosting firm Carrenza, which has donated 3 million pounds worth of infrastructure services and consultancy to Comic Relief and Sport Relief since 2008.
Within that platform it uses a variety of payment firms including Worldpay, Stripe and Paypal. It also links to call centre support from over 140 different firms one of the largest donators of which is BT MyDonate. BT gives access to its platform and call plan to manages 10,000 lines, but if too many calls come in, there is aN IVR as a spillover.
“You can never have enough redundancy”, Hannick enthuses.
Being creative with code
Aside from the high availability stakes, Comic Relief’s technology team need think creatively to overcome challenges specific to the charities fairly unorthodox business model.
Hannick says: “I don’t know how many businesses where Dermot O’Leary comes and tells you he is planning on dancing for 24 hours for donations, and that he needs technology to support that…essentially, we don’t know what we are going to have to do until it happens, and we need creative people to support that.”
Creativity, he hastens to add, does not just mean a beautiful UX design or graphics, but a true understanding of what a user wants.
“Many people say code isn’t creative, but it’s one of the most creative processes – you are building something from scratch.”
Prior to his role as chief technology officer, Hannick has worked within Comic Relief managing Agile projects to build its donation platform since 2012.
This year, the charity’s flagship website rednoseday.com was delivered using Agile methodologies, which he describes as “challenging… and continuously challenging.”
“The team is in transition, but change is the nature of Agile. And that transition is hard work. Agile gives us the tools to tackle the hard things."
Like many charities, Comic Relief is at risk from thieves who are looking to ‘cardwash’ which is when thieves run a large number of stolen credit card details through online payment services to see which ones authenticate.
To bolster the fundraising platform’s security and anti-fraud capabilities, Hannick turned to NCC group, headed by security expert Dave Litchfield Senior, for penetration testing.
Time for a good night’s sleep?
Despite the fact that Red Nose Day passed without any nightmare situations, Hannick might not be able to get a good night’s sleep yet. The team spends the year “preparing constantly”, and have already begun planning for the next project – Sport Relief 2016.