Why browser benchmarks have nothing to do with real life
Let's face reality: While browser speeds are still a theoretically valid criteria in testing browsers, the latest browser generations have improved on web performance so drastically that some of these benchmarks hardly translate to real-world scenarios anymore. In fact, I'd argue that even IT pros don't realize how scores on these speed tests translate to effects on their day-to-day browsing.
And all users fall for it: from tech journalists to IT pros to the beginner. Again, scripting and rendering speeds may still be a valid criteria to web developers or in certain scenarios (I'm thinking browser-based automation), but it's starting to reach a point where the differences can't be made out by the user. Let's take things a bit further: What really determines how fast (or slow) a website is displayed on your screen is much more than the browser. In fact, the browser is just one link in a chain of many technologies, applications, and devices that determine how fast ITworld or Facebook appears on your screen. Among those are:
- Your GPU and graphics driver. Web browsers use GPU acceleration to display content on screen, and the addressed GPUs vary widely in speed -- you can't compare the rendering speed of a three-year-old onboard GPU with a GeForce GTX 590.
- Current CPU load.
- AV tools and firewalls.
- Number of computers sharing the same bandwidth at the same time.
- Network stack and filtering tools.
- 3rd party tools accessing the web and transferring data, or background software updates/checks.
- The connection from your PC to your router to your modem to your ISP switch.
- Web Server speed, traffic, and nodes.
- 3rd party content on websites and unnecessary/badly written scripts.
I'm sure I forgot half a dozen other things that lurk between the web server and your eyes!
A website's loading time is determined by all of these factors. If one or several links in the chain fail to deliver in time, there's no way to determine the speed of a browser. Therefore, judging a browser by its scripting performance or overall website display speed is - in my opinion - an extremely theoretical undertaking with little to no effect on any real-world scenario. And even if browser speed made a visible difference, you'd have to set up a local web server on your own PC and measure browsing speeds for local websites in order to avoid most (not all!) of the factors described above.
Then there's the factor of browser behavior: how do power users actually use a browser? Well, maybe we browse to a website and use a middle- or Ctrl-click to open one or many links in new tabs while we keep reading the current website. In the background, another website is loading, and will likely be ready and waiting for us once we're done reading the active page. That's just one of many different user routines. We're not all robots who type in the URL of a website and wait for it to fully load until we're able to read it. Of course, I don't want to make superficial statements about everyone's browsing habits, but you need to take a good look on how you actually surf the web before choosing your browser based on precise speeds.
Let's put the issue of classic speed tests aside. What (most) of the popular web browser comparisons neglect is that today's browser generation has become more resource intensive than ever. Operating systems like Windows 8 have become increasingly "slimmer" in order to fit on lower end or low-powered mobile devices (tablets, first-generation ultrabooks) or thin clients. One the other side, the browser, which is arguably the most important application on any PC, has become more bloated. And while a big memory footprint on high-end systems may only result in some annoying lag or crashes, it affects the performance of the aforementioned devices, including battery life and overall responsiveness, drastically. And that has a visible effect on your web browsing experience.
PC footprint larger than ever: Why it still matters
Running 15 tabs in your browser isn't an uncommon scenario, right? Have you ever taken a good look at your Task Manager while doing so? On one of my test beds (a 3 GHz Core 2 Duo machine with 4 GB of memory running Windows 7 SP1), Internet Explorer 9 with 15 tabs open takes up a whopping 1,056 MB of RAM. That's a gig of RAM that's being taken up just by a seemingly small application. Plus, these 15 tabs caused IE9 to create nine iexplore.exe processes, each taking up more threads (basic process units to which the OS allocates processor time) and handles (the amount of OS resources being accessed) than any other application or service that's running on most systems. Again, on faster machines with tons of memory, this resource hogging will manifest as reliability issues -- some lags and crashes. On netbooks, laptops, older desktops, first-generation ultrabooks, and tablets, it will cause noticeable performance problems.
- Increased RAM usage leads to Windows 7 paging out other applications out of memory, even if there's sufficient memory left. Your OS and applications speeds suffer. This isn't so much an issue on second-generation Core i7 (or higher-end Bulldozer) CPUs with tons of RAM, but will be on all the lower-specced devices that still make up the bulk of consumer and enterprise hardware out there.
- Windows 7's SuperFetch tries to populate all of your available memory with frequently used memory pages. The result: your favorite applications launch and work faster. If your browsers process increases in size, Windows pushes SuperFetch data out of memory - and performance of other applications suffer even more. This happens on all machines to a certain degree, no matter the spec.
- Large garbage collector (CG) and cycle collector (CC) pauses cause decreased overall CPU performance.
- There are more chances of leaks, in which the browser takes up memory that it is unable to free up even if the process is closed. It may take a restart of Windows to clear this up.
- You may see reliability problems caused by too much caching, poorer responsiveness from individual tabs, and stuttering video or audio.
Again, the more memory your browser takes up, the slower it will go, and it will affect the performance of your other applications as well. Plus, keeping your hardware busy means a decrease in battery life on notebooks. Keep in mind that Chrome, Firefox, and IE9 are built as basic 32-bit applications. This restricts them to 2 GB of memory, which increases the chances of reliability issues and freezes. (You could use the 64-bit versions, but these are incompatible with many add-ons.)
Among aspects like security, usability, stability, and compatibility I'd rank memory usage as one of the more important factors when deciding between browsers, as it affects not just browsing speed but reliability as well as overall PC performance.
Who's the slimmest?
To determine the memory usage of browsers, I prepared the following scenario:
- A total of three test PCs, which will help avoid larger discrepancies. Two of the test beds are running 32-bit Windows 7 SP1, while one is running 64-bit Windows 7 SP1. The end results on all three PCs are averaged.
- All browsers are run with basic third-party add-ons, such as Adobe Flash. No additional browser helpers or plug-ins are installed.
- The system has no third-party party tools or services running at all.
To test the memory scaling of browsers, I staged three different test runs for each browsers:
- 1 tab: Google.
- 5 tabs: Google, Facebook (static profile), South Park Studios video player (video clip -- paused), Lifehacker (article), and Nvidia (driver page).
- 15 tabs: Google, Facebook (static profile), South Park Studios video player (video clip -- paused), Lifehacker (article), Nvidia (driver page), Autoblog (static article), Engadget (static article), ITworld (static article), The Escapist (video clip -- paused), Joystiq (static article), YouTube (video clip -- paused), Cinemassacre (homepage), Screen Rant (static article) and Twitter (static profile).
All tabs were opened in intervals of 5 seconds. The websites I used represent a multitude of content types, such as static HTML articles, search engines, video content, massive dynamic content and social media sites. I made sure that all websites are made up of mostly static content. For example, I did not use the start page of ITworld but rather the printer-friendly version of an article; I also used the Facebook profile of an inactive user rather than my front page. By doing so I eliminate drastic fluctuations in my memory tests.
I used Process Explorer to determine total memory consumption and to keep an eye on excessive utilization of Windows resources such as handles, CPU time, or threads. After each test, I rebooted the machine to restore all locked memory.
I tested the following browser versions, comparing IE 9 with not just the stable versions of competing browsers but some beta and developer versions as well:
- Internet Explorer 9.0.8112 (final)
- Firefox 10.0.1 (final), 11 (beta)
- Google Chrome 17.0.963.46 (final), 18.0.1025.11 (beta), 19.0.1036.7 (dev)
- Opera 11.61 (final) 12.00 (alpha)
So, how'd they do?
In their official stable versions, IE9 and Chrome used the most memory in the 15-tab scenario. (Expect that number increase drastically if you're the type who switches between 40 or more tabs.) Firefox 10, however, beat them all: It consumed around half (!) of the memory that the others needed in this scenario. Although its basic resource usage ran at 68 MB, it scaled quite well up to 218 MB when loading five tabs.
How does this translate to real-world performance? Both IE9 and Chrome felt sluggish, with noticeable delays in switching between tabs. Also, overall system performance and responsiveness degraded. Firefox had no discernible effect on performance whatsoever.
Now let's move on to the beta releases of Opera, Firefox, and Chrome.
Firefox's beta did not achieve a significant improvement over the production version when opening one or or five tabs, tough it did shave another 70 MB off on the 15-tab test. Google Chrome, too, improved things a tad when only a handful of tabs were running, but it failed miserably with a usage of 1,084 MB at just 15 tabs -- unnacceptable! The beta version of Opera had a difficulty with one of our websites as its memory increased to 789 MB at five tabs. At 15 tabs, over 1,112 MB of memory was consumed, which shows that the beta version needs to go a long, long way to match the memory consumption of version 10. All browsers, with the exception of Firefox, showed visible lags -- it was no fun surfing the web or trying to work with other programs.
Looking at the dev versions of the browsers, Firefox again took the crown, improving its memory usage still further. It's unbelievable. Even though the browser froze up for five seconds during our 15-tab test, it still achieved the best overall performance.
Chrome came in the heaviest this time yet again, with a combined memory utilization of 1,118 MB, followed closely by the alpha version of Opera 12. The screenshot below shows Chrome eating up memory in Process Explorer.
Now, I completely realize that these benchmarks do not necessarily represent your real-life usage. Everyone has their own browsing behavior, which may or may not exceed what I've tested here. Also, factors such as third-party plug-ins, operating systems, background tasks, and many of the other variables mentioned above may result a slightly different picture. But this test should at least give you a general picture of the browsers that tend to use the least and most memory.
If you never open more than five or ten content-heavy websites at the same time, you should never base your browser choice on memory consumption. It simply doesn't matter a lot if your browser consumes 50, 200, or 400 MB of RAM, even on lower-end machines. However, heavy tab multitaskers should steer clear of IE9, Chrome (in all its incarnations), and Opera (Beta). Just go with Firefox. With RAM levels quickly reaching 1 GB with just 15 tabs, you'll encounter sluggishness even on faster machines. Again, it simply doesn't matter if you've got 4, 8, or 16 GB of RAM -- a browser taking up 1 to 1.5 GB of RAM quickly reaches the limits of both the Windows' and its own memory management capabilities. Threads and handles run wild, paging starts kicking in, and overall reliability goes down with responsiveness.
Firefox is the clear winner of the bunch. It was the only browser that did not slow things down and I recommended it for both lower-end mobile devices and high-end desktops.