To anyone with a base level of computer knowledge, the concept of a single system with 48 hard drives seems insane. To IT folks, it’s even more outrageous; most disk arrays are limited to 15 drives per shelf and certainly aren’t mounted in a server chassis. To Sun, 48 drives in 4U of space is just the newest entry in its line of x64 servers.
The Sun Fire X4500, nicknamed Thumper, is exactly that: a dual-Opteron server with 48 hot-swappable SATA drives in a single 4U chassis, achieving a raw total of 24TB with 500GB SATA drives. You could fit the entire catalog of the Library of Congress on a single X4500 — 2.5 times over. Armed with four gigabit NICs, 4GB of RAM, and a default installation of Solaris x86, the X4500 is a server in a class all its own.
But what’s it really for? Based on the performance levels I saw during my review, I'd say anything that requires lots of fast storage.
The X4500's internal architecture is as clean as that of any of the other Galaxy servers in Sun’s line: 48 top-loading SATA drives cooled by four enormous fans in the front. The server mainboard is a split-level design taking up relatively little space near the rear of the chassis, with two usable PCI-X slots, four gigabit NICs, and Sun’s N1 lights-out management processor attached in a special dual-purpose video/management card.
Power is delivered via two huge power supplies situated above the server mainboard. There’s a third slot in the rear for another power supply, but the default configuration doesn’t require it. I speculate that this bay is reserved to provide additional power should drives even more power-hungry hit the market, or possibly as a space for a local battery to ensure that the X4500 doesn’t suffer unnecessarily from an abrupt power outage.
The drives are driven by six SATA controllers, each responsible for eight disks. The first two drives in the chassis are reserved as OS drives, seen as bootable devices in the BIOS, and can be configured in a software mirror. The other 46 drives are up for grabs. Obviously there’s no way to get 48 drives into a 4U chassis without top-mounting them, but this configuration makes live drive swaps difficult.
It’s also worth noting that simply rack-mounting the 130-pound-plus X4500 required three sizable guys and a lot of sweating. I’m a little worried about the rails shipped with the server — they didn’t seem up to the task. In a production environment, I would be sure to place the X4500 at the bottom of a rack or above other well-secured hardware.
What is it good for?
h4>Sun is marketing the X4500 as an enormous storage repository for applications with enormous storage needs (such as IP surveillance) and as a single-point storage server for apps running on other servers. Unlike most storage devices, however, the server runs a standard build of Solaris, Red Hat Enterprise Linux, or Windows, and each drive is independent, without any hardware RAID.
For Linux and Windows, this is a problem, as the default file systems on those OSes aren’t really equipped to deal with this number of drives in a single system, nor with the file system sizes you can achieve with the X4500. Those operating systems can make use of the X4500, but with limited I/O speeds, higher CPU utilization, and the requirement that the 48 drives be cut into smaller logical segments. For instance, the default file system on RHEL4, ext3, is limited to 8TB, and it’s not possible to create a software RAID with more than 20 devices.
By leveraging Sun's ZFS (Zettabyte File System) under Solaris, however, the playing field changes drastically. ZFS can address all the disks in a single logical array, and can easily handle a 24TB file system -- and even a 48TB file system when X4500 is certified to run with 1TB SATA drives. (Check out our analysis of ZFS and our screencast demo for more on this file system.)
One of the issues I encountered with the X4500 was very simple: how to find one drive in the sea of disks inside the box. For Solaris and Linux, a simple utility called "hd" creates an ASCII map of the internal drive layout, showing the device address of each physical drive. Without this utility, locating a single disk would be maddening.
File system exercises
I first ran tests under ZFS, running OpenSolaris b57 on the X4500. Creating a file system of all 46 data drives was the work of a few seconds; creating mountpoints, iSCSI targets, and NFS and Samba shares took a few more minutes. All told, the process of turning the X4500 from a standard Solaris server into a very high-capacity NAS was as quick as that of any canned NAS solution on the market, but without built-in extras such as replication (although ZFS can be configured to perform this task).
By contrast, when running Red Hat Enterprise Linux, simply creating a software RAID of 16 drives took well over four hours between file system creation and RAID initialization. Only 16 drives were used in order to stay under the ext3's 8TB limit. Performance under ZFS was also higher, with reads well over 600MBps and writes in the 150MBps range as measured with Bonnie++ running on the local system. Under Linux, the performance was high but couldn’t match those numbers.
As an NAS device, the X4500’s performance was limited by protocol and Ethernet bandwidth limits, with NFS tests showing reads in the 90MBps range, with writes about 85MBps. iSCSI numbers were better, nearing the 120MBps limit of a single gigabit link for reads and 100MBps streaming writes at a fixed block size. As with all I/O benchmarks, changing the block size results in different numbers, so actual performance will be specific to the application used.
ZFS is key to top performance
Suffice it to say that Thumper is aptly named and is a truly unique product from a company that seems to be pulling away from a faltering reputation in the server market. Recent studies have shown that within a few short years, the world will generate more data than it can store. It would seem that Sun is doing its part to bridge that gap.
ZFS and the X4500 go hand in hand, seemingly created for each other in a love story rivaling anything that’s come out of Hollywood in the past 10 years. The speed of file system creation and raw I/O possible with ZFS surpasses that of any other file system available today, and truly makes the X4500 usable in enterprise settings.
On the other hand, using the X4500 with anything but Solaris and ZFS really isn't a viable option at this point. It will work, but it'll be problematic. Once FreeBSD has a stable ZFS implementation, it will fit right in, but for now, the message is clear: Although it’s possible to run the X4500 under other operating systems, Solaris will give you much more bang for the buck.
If you need a large amount of fast storage for applications then the X4500 should be on your list.