David Brown is worried. As managing director of the IT transformation group at Bank of New York Mellon, he is responsible for the health and welfare of 112,500 Cobol programs, 343 million lines of code, that run core banking and other operations. But many of the people who built that code base, some of which goes back to the early days of Cobol in the 1960s, will be retiring over the next several years.

"We have people we will be losing who have a lot of business knowledge. That scares me," Brown says. He's concerned about finding new Cobol programmers, who are expected to be in short supply in the next five to ten years. But what really keeps him up at night is the thought that he may not be able to transfer the deep understanding of the business logic embedded within the bank's programs before it walks out the door with the employees who are retiring.

More than 50 years after Cobol came on the scene, the language is alive and well in the world's largest corporations, where it excels at executing large scale batch and transaction processing operations on mainframes. The language is known for its scalability, performance and mathematical accuracy. But as the Boomer generation prepares to check out of the workforce, IT executives are taking a fresh look at their options.

In a Computerworld survey of 357 IT professionals conducted recently, 46% said they are already noticing a Cobol programmer shortage in the market, while 50% said the average age of their Cobol staff is 45 or older; 22% said the age is 55 or older.

"Organisations are trying not to get backed into a corner because of the skills issue," says Paul Vallely, mainframe sales director at software vendor Compuware. "I haven't seen companies move off mainframes due to the Cobol skills shortage yet, but it's looming."

For Bank of New York (BNY) Mellon, which bought its first mainframe in 1955, keeping the core Cobol applications that run the business on the mainframe makes sense. Modernisation efforts have made those programs more accessible through the use of web services and more up-to-date user interfaces.

But for other types of non-core applications, and for smaller workloads, organisations have been gradually migrating off of mainframes and away from Cobol. In some cases those Cobol programs are simply re-hosted on Linux or Windows servers; in other cases they're rewritten in other, object oriented languages and some programs are being replaced with packaged software.

"Over the past five years there has been an acceleration of businesses moving off host platforms," says Adam Burden, global application modernisation lead at Accenture. Often that means leaving Cobol behind by either rewriting it for J2EE or .Net or moving to packaged software.

Over the last few years, Gartner estimates that the world has seen about a 5% decline in total Cobol code, says analyst Dale Vecchio. Much of that involved migrations by small and medium-sized mainframe shops that move off what they see as a legacy language when they retire the hardware.

It's declining because the functions can be developed by some other building block. "Cobol is no longer needed," Vecchio says. "There are alternatives."

While rehosting can get code off the mainframe quickly, Micro Focus sells a platform that will support it on a Microsoft Azure cloud, that is often seen as an intermediate step. It can be used as a way to get Cobol off the mainframe quickly, before eventually completely modernising and transforming those legacy programs.

Cobol's image problem

Cobol, as a procedural language, is not perceived to be as agile as object-oriented languages for modern programming needs such as mobile apps and the web. And despite the availability of state-of-the-art Cobol development environments, including IBM's Enterprise Cobol on the mainframe and Micro Focus' Visual Cobol, which integrates well with Microsoft's Visual Studio development suite for .Net, Cobol is widely viewed as a legacy language.

Nearly half (49%) of survey respondents whose organisations don't use Cobol say the reason is that the language is simply outdated.

Not everyone agrees, of course. "Cobol has had lasting value, and it's not broken," says Kevin Stoodley, IBM Fellow and chief technology officer of enterprise modernisation tools, compilers and security at IBM.

Some 64% of readers concur, and say their organisations still use Cobol, more than any modern language except for Java/JavaScript and Visual Basic. The last time Computerworld ran a Cobol survey, back in 2006, some 62% of respondents said they still used Cobol at that point.

In the more recent survey, over 50% of respondents say Cobol represents more than half of all internal business application code.

"There has been no renaissance for Cobol," says Accenture's Burden. "There's not a whole lot of new development going on. But our clients are enhancing their core applications and continue to maintain them." Indeed, 53% of readers say they're still building at least some new business applications in Cobol. The vast majority of that code is still being written for the mainframe.

But IT organisations also don't have much choice. Migrating large scale systems built in Cobol is just too costly and risky a proposition. "They might want something more flexible but they just can't do it. They're captive to Cobol," Burden says.

The down economy has helped put off the inevitable, Compuware's Vallely says. "Economic issues provided everyone with a hall pass because not as many folks were looking to retire," he says. But as the economy improves, retirement plans may pick up too. "Organisations are trying to be more proactive," he predicts.

"No other language has seen as big an impact from changes in the demographics of the workforce as has Cobol," Vecchio says. Going forward it will become more difficult to maintain a Cobol portfolio. "The inflexion point will come when enough Cobol programmers have retired that an organisation can no longer tolerate the risk," he says. At that point, most of those programs will migrate, but not all.

Rightsizing Cobol

For BNY Mellon, those Cobol batch and transaction processing programs on the mainframe represent an enormous investment. And while Gartner says it's technically possible to move mainframe workloads of up to 3,000 MIPS, the workload at the bank, which relies heavily on Cobol, consumes 52,000 MIPS of processing horsepower, spans nine mainframes and is growing at a 10% clip each year.

"The business wants us to make investments in programming that buys them new revenue. Rewriting an application doesn't buy them any value-add," Brown says.

Instead, the strategy is to "rightsize" some non-core applications off the mainframe where there's a business benefit, try to keep mainframe MIPS growth under 5%, and stay the course with the bank's core Cobol applications by passing on the business knowledge to younger programmers the bank will need to recruit and train.

Other functions, such as general ledger and reporting, are moving onto distributed computing platforms, where they are either replaced by packaged software or reengineered into Java or .Net applications.

But Brown still needs Cobol programmers to replace those expected to retire, and the learning curve can extend out for a year or more. That means adding staff and having a period of overlap as Cobol's secrets get passed on to the next generation. "I'm trying to get those people onboard and do the knowledge transfer sooner rather than later," Brown says.

But that kind of proactive approach, and the extra costs it incurs, can be a hard sell. "We haven't gotten to the point of feeling the pain yet. When we do, it will happen," he says.

Brown wouldn't specify the number of people he's hoping to hire, but said that the "real heavy need" will happen in the next five to 10 years, as the original mainframe programmers are expected to retire en force during that period. BNY Mellon currently has "a few hundred" Cobol programmers on staff, Brown says.

Brown's concerns are well placed, says David Garza, president and CEO of Trinity Millennium Group, a software engineering firm that has handled code transformations for large businesses and government organisations. "Almost every job we get has Cobol in it," he says, and most of the calls come from organisations that have already lost their collective knowledge of the business logic. At that point, he says, "it's a big risk."

Closing the Cobol talent gap

Where do you find Cobol programmers these days? College graduates with training in Cobol are in short supply. In Michigan, for example, state schools that offer Cobol programming have cancelled classes due to a lack of interest.

"They can't get anyone to enroll," says Jonathan Miller, director of Saginaw County Information Systems and Services.

But some colleges are still providing Cobol training, with help from IBM. The mainframe vendor has developed curricula in association with more than 80 colleges and universities ranging from Brigham Young to Texas A&M. "We donate hardware and software, help with the curriculum, and they graduate hundreds of people every year," says Kevin Stoodley, IBM fellow and chief technology officer.

The Guardian Life Insurance company has recruited Cobol programmers from Workforce Opportunity Services, a non-profit that collaborates with business clients and local colleges to train economically disadvantaged students to fill less popular technology disciplines such as Cobol programming. "They take kids from disadvantaged neighborhoods and provide them as consultants," says former Guardian CIO Frank Wander, who now has his own consultancy, IT Excellence Institute.

"It's sort of a work/study programme. We have over 200 consultants today in five states, and we're expanding," says Workforce founder Art Langer.

BNY Mellon and many other organisations also increasingly rely on outsourcing firms to pick up maintenance and support duties. But for many customers with mission-critical applications, offshore is not the place to keep the institutional knowledge of the business rules that is behind that code. David Brown, managing director of BNY Mellon's IT transformation group, says the bank wants those skills in house.

Fortunately, it's not all that difficult to cross-train programmers in Cobol. "Right now it's pretty easy to hire programmers, and if they understand Java I can bring them back to procedural languages like Cobol," Brown says. The trick, he says, is to develop a curriculum that teaches not just Cobol, but the business rules behind the code that runs the company. "We need to make sure we can roll that forward."

The cost of waiting

Trinity Millennium Group and other vendors like it have established processes for analysing and extracting the business rules embedded between the lines of Cobol code. "The solutions have come a long way in terms of the ability to extract logic and rules," says Burden.

But the process is time consuming and costly. One Millennium client recently spent $1 million (£600,000) to have its Cobol programs analysed and business logic reconstructed as part of a migration project off the mainframe.

"If they had the legacy programmers there and we had done the exercise with them, it would have cost $200,000 and taken one tenth of the time," Garza says. If you wait until that institutional knowledge is gone, he warns, the costs can be as much as ten times higher than it would have been beforehand.

Compounding the loss of skills and business knowledge is the fact that, for some organisations, decades of changes have created a convoluted mess of spaghetti code that even the most experienced programmers can't figure out. "Some systems are snarled so badly that programmers aren't allowed to change the code at all," Garza says. "It's simply too risky to change it. They're frozen solid."

Package deal

That's the situation faced by Jim Gwinn, chief information officer for the USDA's Farm Service Agency. The USDA's System/36 and AS/400 systems run Cobol programs that process $25 billion (£16 billion) in farm loans and programmes.

"We have millions of lines of Cobol and there's a long history of it being rewritten," he says. "It has become increasingly difficult to change the code because of the complexity and the attrition of the knowledge base that wrote it." That's a big problem because laws that govern farm programmes change every year, driving a need to update the code to reflect those changes.

Gwinn hired consultants from IBM, who concluded that rewriting the programs in a different language or rehosting them on a distributed computing platform would be complicated and costly. But the System/36 hardware had to go, so Gwinn decided to bite the bullet: FSA will move off of its end-of-life mainframe systems by rewriting some of the code in Java and replacing the rest with packaged software from SAP.

But Gwinn says he'll miss Cobol. "It has been very stable and consistent, with little breakage due to code changes, which you see with Java-based changes," he says. "And in a distributed environment you have to balance your workloads a little more carefully."

Going for a rewrite

The anticipated exit of institutional knowledge and shortage of Cobol programmers were also primary drivers behind NYSE Euronext's decision to reengineer one million lines of Cobol on a mainframe that ran the stock exchange's post-trade systems. While Cobol was dependable, it wasn't viewed as maintainable in the long run.

Steven Hirsch, chief architect and chief data officer, cites the need to make changes very rapidly as another key reason the NYSE abandoned Cobol. "Ultimately, the code was not easily changeable in terms of what the business needed to move forward. We were pushing the envelope of what it took to scale the Cobol environment," he says.

So NYSE rewrote Cobol programs that run its post-trade systems for Ab Initio, a parallel-processing platform that runs under Linux on high-end HP DL580 servers. The new environment allows for more rapid development, and the rewrite also eliminated a substantial amount of unnecessary and redundant code that had crept into the original Cobol programs over the years.

If the business' Cobol code doesn't need to change much, and many batch and transaction processing programs don't, it can be maintained on or off of the mainframe indefinitely. But that philosophy wouldn't work for NYSE. "We are a rapidly changing business and we needed to move faster than our legacy code," Hirsch says.

As for trading, that's all proprietary NYSE Euronext software. "There's no Big Iron or Cobol," Hirsch explains. "There's been no use of mainframes in the trading environment for many years."

Rehosting: Lift and shift

When it comes to hiring new Cobol programmers, Jonathan J. Miller, director of information systems and services for Saginaw County, is struggling. "We've lost our systems programming staff," he says. And like many government IT organisations that have suffered from budget cuts, he doesn't have much to offer those in-demand Cobol programmers.

Generous government benefits used to attract candidates even when salaries were lower than in private business. Now, he says, "Our pay hasn't increased in eight years and benefits are diminished."

To fill in the gap, the county has been forced to contract with retired employees and outsource Cobol maintenance and support to a third party, something that just 18% of Computerworld readers say they are doing today.

The Cobol brain drain is starting to become critical for many government organisations, says Garza. "It's a high risk problem in many countries we are doing work in. The people have retired. Even the managers are gone. There's no one to talk to."

Saginaw County found itself hemmed in by the complexity of its Cobol infrastructure. It has four million lines of highly integrated Cobol programs that run everything from the prosecutor's office to payroll on a 46 MIPS Z9 series mainframe that is at end of life. With mainframe maintenance costs rising 10% to 20% each year, the county needs to get off the platform quickly.

But commercial software packages lack the level of integration users expect, and Miller's team doesn't have the time and resources to do a lot of integration work or re-engineer all of the program code for another platform.

So the county is starting a multi-phase project to recompile the code with Micro Focus Visual Cobol and re-host it on Windows servers. An associated VSAM database will also be migrated to SQL Server. Miller hopes that the more modern graphical development suite will make the Cobol programming position, which has gone unfilled for two years, more attractive to prospective programmers. But he acknowledges that finding talent will still be an uphill battle.

A legacy continues

Is there a role for Cobol off the mainframe? "I don't believe there is. Cobol and the mainframe run well together, and that's where I want to keep it," BNY Mellon's Brown says. But the bank is still creating new Cobol components on the mainframe, and will continue to do so.

That's a common sentiment among Accenture's large corporate customers, says Burden. Cobol will continue its gradual decline as midrange systems are retired and businesses continue to modernize legacy Cobol code or move to packaged software. Today Cobol is no longer the strategic language on which the business builds new applications. But it still represents the "family jewels" of business, Burden says. "They're enhancing existing applications and adding functionality to them. I've seen no slowdown in those activities."

If companies can't find talent to keep that infrastructure going, outsourcing firms such as Accenture are ready, says Burden. The scale of Accenture's support operation is large enough to provide a career track for Cobol programmers, and he says it's easy to cross-train programmers on the language. "We can turn out new programmers quickly. So if clients can't support Cobol, we will."

"People make too much of that trend that we're not graduating enough Cobol programmers," says IBM's Stoodley. Preserving the institutional knowledge is what's critical. "You can make a problem for yourself if you don't keep your team vibrant," he says. But as long as there's a demand for it, "businesses will find people willing to work on Cobol."

"You have to respect the architecture of Cobol," Burden says. It may have been created for simpler times in application development, but it remains the bedrock of many IT infrastructures. "I don't see that changing for another 10 years, or even longer."

Find your next job with techworld jobs