Recent Council on Competitiveness¹ studies have found that nearly all businesses that exploit high performance computing (HPC) consider it indispensable for rapid innovation, competitive advantage and corporate survival. But Council on Competitiveness research has also identified barriers to the broader use of HPC in the business sector. The companion article on high performance computing (HPC) in this issue of CTWatch Quarterly² examined one of these barriers—the difficulty of determining the return-on-investment for HPC hardware systems (also known as supercomputers)—and proposed a methodology for addressing this issue. Another important barrier preventing greater HPC use is the scarcity of application software capable of fully exploiting current and planned HPC hardware systems.

U.S. businesses rely on a diverse range of commercially available software from independent software vendors (ISVs). At the same time, experienced HPC business users want to exploit the problem-solving power of contemporary HPC hardware systems with hundreds, thousands or (soon) tens of thousands of processors to boost innovation and competitive advantage. Yet few ISV applications today can exploit ("scale to") even 100 processors, and many of the most-popular applications scale to only a few processors in practice.

Market forces and technical challenges in recent years have caused the ISVs to pull away from creating new and innovative HPC applications, and no other source has arisen to satisfy this market need. For business reasons, ISVs focus primarily on the desktop computing markets, which are much larger and therefore promise a better return on R&D investments. ISVs can sometimes afford to make modest enhancements to their application software so that it can run faster on HPC systems, but substantially revising existing applications or creating new ones does not pay off. As a result, the software that is available for HPC systems is often outdated and incapable of scaling to the level needed to meet industry's needs for boosting problem-solving performance. In some cases, the applications that companies want simply do not exist.

This need for production-quality application software and middleware has become a soft spot in the U.S. competitiveness armor; a pacing item in the private sector’s ability to harness the full potential of HPC. Without the necessary application software, American companies are losing their ability to aggressively use HPC to solve their most challenging problems and risk ceding leadership in the global marketplace.

To better understand why U.S businesses are having difficulty obtaining the high-performance, production-quality application software they need, the Council on Competitiveness convened an HPC Users Conference, followed by a day-long software workshop held in collaboration with the Ohio Supercomputer Center (OSC). Participants³ discussed the effectiveness of current business models for HPC application software, as well as technical and resource barriers preventing the development of more effective software. They produced a roadmap of actions ISVs, universities and government research establishments could follow, individually and collectively, to address these barriers and ensure that our country has the application software needed to solve our most important competitive problems. The major discussion themes and recommendations are highlighted in the following sections.

A variety of market and resource barriers are retarding development of more scalable, production-quality application software.

Despite the rise in global competition and HPC's proven ability to accelerate innovation and increase productivity, HPC remains a niche market with much smaller revenue opportunities than the larger commercial computing market. This lowers the research and development capital available to develop next-generation HPC application software and lessens the opportunity for ISVs to obtain investor funding for HPC R&D. It also heightens the risk that ISVs could be “bought out” by companies, which might restrict or eliminate access to their software by U.S. commercial and national security organizations that depend on it.

As for-profit companies, many of which are public, the ISVs often must pursue the broader commercial computing market with its larger revenue opportunity. In addition, according to the Council’s 2005 Study of the ISVs Serving the High Performance Computing Market, Part A: Current Market Dynamics,⁴ more than a third of the ISVs qualify as small businesses, earning less than $5 million annually. Even if these companies were to invest 10 percent of their revenue in research and development, this would be sufficient to add only a few new features to existing HPC applications. Fundamental rewrites of application software or the creation of new software for the HPC niche market require far more investment than this.

The niche status of the HPC market also reduces the potential for outside investor funding for the ISVs that serve the HPC community, eliminating another possible source of research and development resources. Additionally, unlike software developers who serve the broad commercial computing market and can keep their prices low by amortizing their research and development costs across a large user base, the ISVs that serve the HPC market do not have a correspondingly large customer set against which they can amortize their expenses. As a result, the ISVs must shoulder the cost of a major software redesign or spread it across a small user community, significantly increasing the price of the HPC application software.

In today’s market, ISVs typically sell unlimited annual-use licenses, often on a per-user or per-processor basis. Users often find that they cannot afford the additional software licensing fees that apply when they want to run their problems over more processors. This discourages them from running larger, more complex problems on their current systems or adding additional processors. As a result, users don’t exploit the applications to the desired and needed degree.

In addition, annual software license fees can be prohibitively expensive for the small and mid-sized companies that dominate the U.S business landscape – the same companies that represent the potential growth market for HPC. Most cannot afford HPC systems and software, nor do they want to deal with the complexity of this technology. Many only want access to HPC systems, software and expertise on a periodic basis. “The analogy,” explained Paul Bemis, vice president for marketing at Fluent Inc., “is you’re going to rent a car for the weekend. You go to Hertz and they say, sorry, all we offer are annual leases. But you only need the car for the weekend. Sorry, they say again, all we offer are annual leases. That is the ISV community today. It needs to change. We need to be asking, ‘How much do you need and how long do you need it?’ ”
.
An interesting service model has evolved in the oil and gas industry that may provide an analog. This industry is dominated by small, independent businesses that are the primary producers of oil and gas in this country. These small companies rely on service firms to perform their complex seismic computations and reservoir simulations. The service firms, populated with experienced engineers and scientists, have bundled their knowledge of the industry, their ability to create and use HPC software, and access to HPC systems. With these, they have created very profitable consulting businesses. This business model produces much higher margins than software development alone, because it delivers what the small oil and gas companies value most: access to industry expertise and advanced HPC tools in order to make decisions, without having to invest in the underlying technology. This permits even the smallest oil and gas companies to match the technical capabilities of larger competitors.

The small size of many ISVs severely limits their ability to acquire even one large parallel system, much less the full variety of HPC systems from different vendors. Yet access to these systems is essential for developing ISV software applications that will run compatibly and fast on the range of hardware products available in the HPC market. Hardware vendors that once donated machines to ISVs for software development, testing and support can no longer afford to do so because of sharply reduced profit margins. Instead, ISVs must either set aside significant funds to acquire systems or, more typically, find a way to access HPC systems without acquiring them.

Limited funds also make it difficult for many ISVs to support multiple types of HPC computer platforms. They therefore tend to focus on one type of HPC system: the so-called "clusters" built from commodity components, because these lower-priced systems are the most pervasive computer products in the HPC market. This has alarmed business users whose problems are better suited to other types of HPC systems. These users believe their crucial, competitive problems are being ignored.

A close examination of the current education pipeline presents a troubling picture regarding the availability of next-generation scientists and engineers. American-born students are choosing to pursue computational science and engineering less frequently than in the past. Research institutions face an added challenge; the development schedule for HPC application software may exceed the time that graduate students spend at their facilities. “The people we need to architect these codes are unique individuals,” explained John Morrison, division leader for computing, communications & networking, Los Alamos National Laboratory. “They have to understand the science domain, computational science, computer science, and then figure out how to make these big applications run on a parallel computer and to scale.”

On a positive note, trends in desktop computing may bring new talent to bear on this problem. Multi-core processors – technology that allows multiple processors to work together on a single chip—already are finding their way to the desktop computers of many workers and could stimulate development of more parallel software. As this technology proliferates in the future, every desktop computer is likely to become a parallel computer, and the broader commercial software development community will be forced to address the same challenge that the HPC community has struggled with for years: creating applications that work in a parallel environment. This should bring extraordinary energy, money and talent to the software development process, resulting in new tools and new solutions for HPC users, too.

In addition to market and resource barriers, HPC users and software developers also face technical barriers that are impeding the development of more advanced HPC application software and underlying middleware.

Businesses can increase dramatically the value of their HPC systems when usage is expanded beyond the Ph.D. researchers and integrated into the workflow of the organization. This means that HPC applications and tools must be distributed at the desktop to “non-experts,” i.e., designers, engineers, analysts and others who are more directly involved in the production of an organization’s product or service. The key to putting HPC applications and tools into the hands of the non-Ph.D.s at the desktop is better user interfaces that are easy to operate.

Integration of HPC into workflow practices becomes even more important as competitive pressures drive companies to shorten product cycles. As little as a six-month delay in product introduction can allow competitors to take market share and substantially cut into profit margins. Yet, because of the primitive user interfaces in use today, it can take longer to set up a problem than to run it, particularly as the number of processors increases in HPC systems. In this environment, even one day’s delay in setting up and running an HPC problem can significantly impact the workflow. When users are under time-to-market pressures, they tend to run smaller problems whose solutions may not provide the needed competitive boost.

For many ISVs, the requirement to maintain and add features to "legacy codes" is tying up limited development dollars that could be spent creating new software with greater parallelism, or at a minimum substantially revising the legacy software. Given the limited R&D resources of most ISVs and the requirement that publicly traded companies must report quarterly earnings, ISVs invest first in projects customers will pay for today, such as modest revisions to existing legacy software. Only then can they consider higher risk, more costly efforts such as developing new solutions from scratch or undertaking the “hundred man-year investment” that is needed to correctly re-configure a complex software application like MSC.Software’s NASTRAN to run on hundreds or thousands of processors.

Integrating HPC applications into the business workflow can increase productivity, but it can also slow development of new software. Industrial HPC users often are reluctant to make changes to hardware or critical software after it is integrated into key business processes. Re-coding legacy calculations to achieve greater parallelism can change the answers by several percent. For many long-time HPC users, consistency in results can be even more important than improved performance.

Further, the business process reengineering required to make the switch to newer, more efficient applications frequently is too time consuming and risky when companies are under market pressures to deliver products to customers on time and within specified design and quality parameters. Dr. Donald Paul, vice president and chief technology officer for Chevron, explained that competitive advantage often comes from “how you integrate all of the applications that you use into major workflows, and how well those are integrated into and drive the way the business works. The business process structures that can be built around the technology are as important as the technology, and are in fact dependent on the technology.” While rapidly changing software or systems might be acceptable or even desirable in a research environment, Dr. Paul explained that it is often intolerable in a business production environment, where a single application must be integrated into a complex stream of business processes.

While national laboratories and universities spend hundreds of millions of dollars every year on fundamental HPC software research, this software tends to be difficult to use in a corporate production environment. Research software typically is developed by Ph.D.s for use by highly experienced scientists who are willing to put in extraordinary time and effort to make it work. But industry wants fast desktop tools that do not require a Ph.D. to use. The world of technical computing has evolved from one that was managed by a few highly skilled and educated engineers to one in which computing resources are spread throughout an enterprise to a wide variety of people. Research organizations do not develop their applications with these needs in mind.

Further compounding the problem is a shrinking federal budget that has reduced the amount of government funding available to support joint programs between national labs and industry. In this environment, it is unlikely that the laboratories will reallocate significant portions of their budgets to focus on joint endeavors with ISVs. The labs could be criticized for misspending funds, since there is an assumption that they have been given “just enough” to fulfill their missions.

Additionally, research software developed within the universities is often an abstraction of a real problem, and may be licensed as open source software. Therefore, it has not been subjected to the rigorous testing demanded by industry, and may not allow for proprietary extensions to the code that companies require. Consequently, the software developed by these research organizations, while providing an important foundation for commercial software, is often inadequate for immediate use by industry.

Most HPC research codes are produced by the national laboratories or universities and licensed as open source, i.e., the source code is freely available and may be copied or altered without permission or payment. However, the terms of some open source license models discourage software developers from improving and commercializing these codes, thereby hindering software innovation.

For example, the General Public License (GPL) requires that derivative works also bear the GPL designation, so they automatically become open source as well. This makes ISVs and other software developers reluctant to invest in development, because they cannot keep their improvements proprietary or derive competitive advantage from them. David Turek, vice president of Deep Computing at IBM, shared the frustration of many software developers: “It’s very difficult when we make major investments to try to produce differentiated technology, and somebody says to me ‘Really nice. Now can you put it on everybody else’s platform in the world and make it cheap or free?’ What we have to do, given this circumstance, is to work with other platform vendors on new business models that satisfy the needs of the end users while also making our own technology investments economically viable.”

On the other hand, the Berkeley Software Distribution (BSD) license allows developers to incorporate the open source code into derivative works that are not open source and can become proprietary. Much successful HPC software that has transitioned from government and universities to industry bear BSD-type open source licenses.

A vibrant, growing HPC market is the best long-term guarantee that production quality HPC application software will be available. Participants in the Council-OSC software workshop also agreed that wider usage of HPC is crucial for future U.S. economic growth and the nation’s ability to maintain a preeminent position in the global marketplace. Participants made recommendations for stimulating long-term HPC market growth through near-term accomplishments.

1. Consider pricing changes or moving to a service- based model in order to provide users with software, expertise and access to appropriate HPC hardware.
2. Partner with companies that use HPC to run demonstration projects that establish the business value of using HPC.
3. Develop easy-to-use interfaces that encourage HPC applications to be integrated into organizational workflows.

1. Enhance their educational programs in computational science at the undergraduate and graduate levels to meet the needs for skilled technical workers.
2. Enhance their understanding of the ISV community's requirements so that they can better leverage their own software research and education agendas to help meet the ISVs' needs where appropriate.

1. Modify research support practices to provide sustained (multi-year) funding for research teams to develop mature research software and algorithms. Related to this, encourage commercialization of suitable software.
2. Where open source HPC research codes are being developed, terms of government grants and contracts should more seriously consider BSD model licenses, to enable ISVs to build commercial products based on the codes without jeopardizing the ISVs’ privately created intellectual property.

Implementing these recommendations entails considerable risk to the ISVs, the users, the research community, the government, and investors, but failure to take action may inhibit competitive advances by U.S. companies and place them in jeopardy should other countries or companies capitalize first on the potential for expanded use of HPC.

By leveraging HPC more fully to solve complex, crucial problems, America can unleash a new era of innovation-driven growth, creating new industries and markets, fueling wealth creation and profits, and generating high-value, higher-paying jobs that will raise the standard of living for all citizens.