While mainstream enterprise IT professionals still regard grid computing as a nascent technology, the financial services vertical market has been using grid and the open-source Globus Toolkit in the production environment for years. In financial services, the question isn't so much who uses grid, but who doesn't.

This month, I spoke with Deborah Williams, who leads the capital markets practice at industry research firm Financial Insights, an IDC company (owned by Techworld's parent company, IDG). Williams has had a great vantage point on the evolution of IT in institutional trading, retail brokerages and risk management. We asked why grid has been such a natural fit in financial services.

Q: Tell us a bit about the computing evolution in financial services. It's the same compute evolution that everyone else has followed.
A: The first step was the monolithic mainframe with lots of distributed dumb terminals. There was a lot of distributed access, and then came the development of client/server and LAN-based systems that distributed actual application logic. Then you had the data distributed through N-tier architectures. Chunks of code that ran more slowly or [were] compute-intensive were offloaded onto another tier -- that evolution has been happening for a while.

Q: Where does grid fit in financial services?
A: Financial services has a long history of doing most of its heavy lifting in overnight batch cycles. Most of what drives the need for distributed computing is whatever they need on an ad hoc basis. In capital markets, it might be pricing or scenario analysis -- those types of activities that help traders monitor whether they should be buying or selling.

Where you see grid applied most often is in areas like risk management or middle-office kinds of functions, where even an eight-hour batch cycle wasn't enough to run re-evaluations on a whole portfolio. Not too long ago, banks were running 22-hour batch cycles, but that became pretty useless. The dealing room is open a significant portion of that time, so by the time you got that report 48 hours later, it was already outdated. As the instruments and the methodology for looking at risk scenarios [like Monte Carlo simulations] became more complex, it became clear that they needed to do a better job of keeping up.

Monte Carlo simulations are used in financial services to do randomised scenario analysis. It's become very popular for risk management. You can say, "We know the prices over the last x years look like this, so let's throw at this thing all the possible parameters." You end up with a distribution of prices.

If you assume a bell-shaped distribution curve, the traders care about the down-side tail. They care about whether it's flat, or tails back up -- so they're looking the shape of that tail for extreme risks. Monte Carlo analysis allows you to get a much better sense of what the tail looks like, rather than just predetermined scenarios.

In order to do this correctly, one must run ideally a minimum of 10,000 different re-evaluations to get enough points on the curve to project what the curve looks like. With the compute power available today, many organisations might run 500 to 600, because they just don't have enough time to run the 10,000.

Monte Carlo is a perfect kind of application for grid, because it's inherently parallel. You're taking the same calculation and doing it over and over with different parameters. It's really easy to chunk up individual calculations and throw them out on to different processors. Because it's inherently parallel in nature, anything that makes use of a Monte Carlo analysis has been a big area that you see grid applied to in the capital markets.

Q: What percent of these brokers are using Monte Carlo?
A: Just about all of them in some way, shape or form. It's a very common technique in the middle office at the enterprise level. Grid has become a very well-known, well-explored technique for exploiting these type of simulations. Institutions like Wachovia, Merrill Lynch, Morgan Stanley -- they all use grid to do some part of their derivatives pricing and revaluations and for risk management.

Q:To what extent are interoperability and integration issues important in the financial services industry vertical?
A: We have huge integration issues in our industry, I think because of the natural best-of-breed predilection of most of the firms. Over the last 20 years, we've picked the best solutions for the problems -- especially in capital markets and in banking, insurance not as much.

The IT shop throws up their hands and says, "Well our standard database is Sybase, and this runs Oracle," but the business guys don't care. They just want the best solution for the problem. So you end up with a very disparate, wide-ranging, siloed kind of architecture. And this over time has led to huge integration issues.

We know the old CRM story. People were trying to manage customer relationships, but couldn't see the customer anywhere. They could see their checking account, their brokerage account, their savings account, they're CD -- but they couldn't see it all in one place. That's because each of the business lines or product groups grew up their own IT infrastructure and created an independent silo of systems. So the integration problems have been huge. That's the natural environment that they're struggling with already. If you add to that technology integration problems, not only does it add to the cost and maintenance burden, it also further accentuates an integration challenge that they had to start with.

It is interesting that these are the same kind of integration issues we were seeing from the scientific environments we first grid-enabled. One of the unique characteristics of grid computing and the Globus Toolkit in particular is its ability to reuse existing infrastructures and tie them together in system-agnostic aggregations. Perhaps we will see the same in the financial services market as use of these technologies become more commonplace.

Grid pioneer Ian Foster is a board member at the Globus Consortium, a vendor-neutral, non-profit organization promoting the open-source Globus Toolkit in the enterprise. He can be reached at [email protected].