We've all had the infuriating experience of using apps that freeze or malfunction. For Facebook, keeping this experience to a minimum for its 1.5 billion daily users is a business imperative.
Facebook's source control, which is the central repository that controls all of the ways in which developers makes changes to the software, has roughly one million commands sent to it every single day. This translates into over 100,000 changes made to software each week. At this scale, errors are bound to slip through.
To minimise the chance of this, the company scans its code continuously, including pre-testing, internal testing and at the point where software actually reaches end users. Techworld heard from Mark Harman, an engineering manager at Facebook and part time professor of software engineering at University College London, about developing a tool focused at the very early stages of code generation.
Named Sapienz, this tool automatically scans code for certain classes of bugs and is increasingly finding intelligent ways to suggest fixes too.
Bug fixes at this internal stage (there's a whole other department which deals with spotting bugs that have made it into the end product) can take minutes up to hours to solve, with the worst potentially eating into weeks of a developer's time.
The Sapienz tool relies on something called search based software engineering. This is the combination of search based optimisation and software engineering. "We mean search in the sense that we're searching the search space of tests to find tests which are good at revealing faults," says Harman. This is an area which has received ample interest and research in academic circles, says Harman, and which is currently gaining more traction in the industry.
"The number of gestures I could do in a simple app, the sequence of those gestures, the number of possible such sequences is larger than the number of stars in the known universe," he says. "We think of the problem of testing as this huge search space where tests live and the problem of automating testing is to find those sequences which are going to be good at revealing bugs. We can't possibly hope to cover all of them - that's infeasible because of the size of the scalability - but what we hope to do is sample intelligently."
At the moment, humans generate software tests, but it's a slow and boring process. "If we're honest with ourselves, it's probably one of the things that most testers regard as relatively boring compared to the more creative things they could be doing," says Harman. "For this reason, it often gets regarded by many companies as not explicitly unimportant, but implicitly unimportant." Until, of course, the day when a serious fault is discovered.
They hope to use the search based approach to develop tests, meaning that developers don't have to concern themselves with this anymore.
"The starting point is that we have to deploy into a continuous integration system," says Harman. Once a developer has submitted code, he explains, they want to automatically generate test cases and find issues with the software. The developer will then be informed about the problems, and this software will become a release build.
They deploy the tool on the back of Facebook Learner, which is the company's machine learning infrastructure, and allows this to process to be automated. Harman explains that the workflow for Sapienz looks a little like this: "The mobile app gets build, we run the Sapienz engine to search for test cases, we record crashes we find and those that we've been able to triage to particular developers, and this is running continuously with a recurring operator," he says.
Right now, the tool is geared towards automatically discovering faults, but in time, Harman hopes it will increasingly be able to automatically able to fix them, too.
"Now this is very much bleeding edge and it's also very current hot topic in the research community internationally, but what we wanted to do was take all of this technology and the unique position we find ourselves in with both static and dynamic analysis and see, can we combine all these techniques to automatically fix some of the bugs we're finding?"
The automated fix rate is currently at around 75%, according to Harman. The most common bug they find is the null pointer. He says the kind of bugs found at Facebook are actually very similar to most other apps. This is a boon to other companies, given that they plan to release Sapienz as an open source tool at some point.
Harman was initially working on automatic software testing and verification in a research group, which became a launchpad for a startup consisting of himself and two others, called Magica. In 2017, the company was acquired by Facebook, and the group moved within the hallowed walls of the software giant in the same year.
Meanwhile, another startup, Monoidics, which was experimenting with automated verification techniques, and was acquired by Facebook in 2013. This startup and Harman's grew into two complimentary software tools, Infer and Sapienz, respectively.
Infer offers a static analysis technique, whereas Sapienz offers a dynamic analysis technique. "The goal of infer is to take the source code, but not to run it, not to execute the programme and see can I find places which are likely to have bugs in," says Harman. "Sapienz's job is to actually run the code in a realistic environment and see if it can cause a failure in practise. Does it crash? Does it go out of memory? Does it just hang and give you that awful wheel that frustrates you?"
In this way, Sapienz finds the problem, and Infer discovers the likely cause.
Infer has already been released to the open source community and is currently being used by AWS, Mozilla, Spotify, Uber, JD.com, Marks and Spencer.
Does all of this mean that the human developer will soon be out of a job? Harman says that suggested fixes will still be directed to the developer. "We have this fix process, but it's the developer who finally reviews and decides whether this fix can go into the code base," he says, emphasising that in relation to developers, this tool is designed with collaboration, rather than replacement in mind.