HSBC Bank plc has just completed a consolidation exercise that's seen almost 1,000 new VMs commissioned, of which almost 300 were converted from physical to virtual (P2V) machines. The bank's enterprise systems support manager John Gibson said that HSBC's consolidation exercise had seen almost 600 servers demised and thrown away, 330 re-used and a mighty 900 machines whose purchase was avoided. It's also saved a shedload of money -- more on this later.
Traditionally, HSBC had always operated a one-to-one policy of application to server. However, the resulting server sprawl was costly in terms of consumption, server under-utilisation, capital expenditure and staff. So, in 2004, the bank decided to investigate virtualisation as a means of consolidating its server sprawl.
The IT staff's evaluations and proof of concept work found that, at the time, VMware's ESX Server was the only viable option and so it embarked on a virtualisation migration project to get 1,300 physical servers down to 150.
The bank engaged VMware's professional services division to help with the project and put in place a development framework. It took a year from the decision being taken in March 2004 to the first contingency servers being deployed, followed by the first trial production machines. "The time taken wasn't because of technology but because of getting management buy-in," said Gibson. "We also used VMware's professional services to help with negotiations with management."
Having verified the concept in the real world, the next step was to identify which servers could be migrated to VMs.
Migration tools used include PlateSpin's PowerConvert and VMware P2V Assistant. The systems were divided into tiers one and two, with the mission- and performance-critical applications in the first tier. Tier 1 consists of about 120 four-way clustered servers connected to SANs. About 10 hosts live in each cluster or farm, sharing 6TB of disk and supporting 250 VM guests per farm, for an average VM to Gigabit-connected host ratio of about 20:1.
The second tier consists of two-way blade servers with about five to 10 VMs per blade. These are good for single CPU VMs, according to Gibson, they run on a single VLAN, and use local storage only. "We get a 7:1 VM-to-host ratio," he said. "We re-use blades -- not least because we then don't have to change anything such as the name and IP address."
What's perhaps surprising is the small amount of memory each VM receives. Less than 512MB is standard, with most running in 384MB. "I'm aggressive about the amount of RAM I give a VM," said Gibson. "If they struggle for RAM we can increase it straight away but we need strict criteria for allocating resources. We tend to minimise RAM and disk and see how it runs, since VirtualCenter helps us understand how much RAM is actually being used. In practice, we don't have too many spikes that demand more resources. We also only create data volumes of an appropriate size to ensure that we get a good ratio of VMs per host."
Utilisation rates in both tiers runs at between 50 to 80 per cent.
In terms of management and issues such as naming, Gibson said that VMs are treated just as if they were physical devices. This means that each VM is logically separate from the others and has its own IP address and name, and they all have BMC Patrol, anti-virus and backup agents installed locally.
And when a business unit asks for a server, that's treated just as it was before since there's still a requirement for physical resource under each VM. Requestors are required to complete a form to ensure that all required build and billing information is captured. The IT team validates it and passes to the team in India to implement the VM. The difference comes in the time it takes the IT organisation to respond. "Request processing and turn-round are less than three days for a VM guest, and it's usually more like three hours," said Gibson.
On broader issues, Gibson said that some might question why the organisation didn't decide to go for fewer, bigger servers, on the grounds that management time could be saved. He said that the risk of relying on a small number of boxes supporting dozens or maybe hundreds of VMs was too great. The size of the IT team hasn't been an issue anyway, said Gibson, with only two individuals required. "Any problems we've had haven't been related to ESX Server, but more with the underlying hardware," he said.
On the lessons learned, Gibson said the key was getting buy-in from management, along with high levels of communications and tailored training. The biggest challenge right now was the pressure to adopt more of the new technology, which places demands on the VI team to scale up the systems. Items to be incorporated into the virtual infrastructure (VI) include the DMZ, raw LUNs, virtual desktops, and larger guests. "We also want to avoid a virtual sprawl and balance new infrastructure with reuse," he said.
In terms of benefits, Gibson said that the environment gained. There is now a 141kW power saving from server demise and from avoiding the installation of new servers, minus a spend of 80kW in ESX infrastructure. The new VI has however provided capacity for 2,500 additional VMs and provided 51 racks of space.
Capital cost avoidance includes £1.65 million in re-use of physical servers and updating, £3.22 million saved on the purchase of new servers and £160,000 shaved off hardware maintenance costs from demised servers. In addition, there's a reduced time to market and lower rates of Window server recharge to the business. Gibson said the bank had gained improved physical server utilisation, operability, automation and manageability. And the VI also meant that the IT team in India was able to shoulder some of the load and enable the release of expensive UK-based contract staff.
Lower costs, better for the environment and faster turn-round -- only hardware vendors might quibble with that.