Here is a step by step approach to a normalized productivity figure of merit based on workflow productivity measurements:

1. Define a standard reference system for comparison, a “canonical cluster”. Productivity estimates are made on this system and on the new system under evaluation and reported as ratios.
2. Define the set of working environments (application areas, project units, unique program activities) in terms of the resource emphasis (FLOPs, GUPs, bytes, …) and standard personnel rates.
3. For a few (usually just one) activity classes in each environment, describe the workflow as the effort fraction of standard workflow elements. Select a set of representative applications for attained performance (E_job) measurements. Design job-level productivity measurements for the workflow elements and measure the productivity term, ¹¹.
4. Define a typical job mix and the overall administrative workflow, and design administrative-level productivity measurements. Measure ¹³.
5. Define subjective scoring criteria for for each environment, and score the systems with regard to project success rate, accessibility, ease of use effectiveness, …
6. Vendors provide information to identify the relevant resource capability ratios of the systems, such as peak performance () or GUPs, etc. Vendors also provide information on availability of the new system. Reference system availability should be obtained from experience data.
7. Combine factors into the relative productivity figure of merit,
8. If absolute Productivity, P, is of interest, costs must be included and a utility constant (U_sys) defined. Costs include project and administrative personnel budgets, and system costs including initial costs and running costs over a defined lifetime, T.

We have tried to demonstrate that the productivity figure of merit we have described here is really much simpler than it may have first appeared. It is no more than a way of getting to a single number that combines what we know or can guess about a system configured for a particular environment into something that approximates a measure of total productivity. It can be used in HPCS design comparisons, subsequent budget justifications, and ultimately, we hope, in procurements.

Many of the numbers needed as input are traditional cost and performance variables. The hard parts, of course, are the benchmarks needed to measure the productivity of humans when confronted with these new systems. We see these productivity benchmarks as simply measuring curves of efficiency of job resource utilization, or system utility optimization, vs. the cost in time of the human effort. These curves should clearly indicate an obvious point of diminishing returns, an operating point.

We recognize that we have made use of words like simply, clearly, obviously, and this may be unfair. We know that getting a number that approaches being a true measure of productivity for a given system is going to be difficult. We need goals for the productivity benchmarking efforts to aim at, and we think this framework provides them.

We have developed a spreadsheet, which may be obtained from the authors, that allows one to gain some intuition into how system-wide productivity comes together from the many components. The first six sheets of the spreadsheet are entry sheets to be completed by the different entities that may be responsible for the information: Vendor Cost, Host Environment and Values, Performance, Job-level Productivity, Administration Productivity, and Subjective Productivity Evaluation. The last sheet summarizes the calculated result.

We see the process of obtaining an informative figure of merit as being incremental. One may start by entering guesses, policies, and goals, and then progress through preliminary measurements and even post-mortem analysis. For each entry, the spreadsheet has a quality descriptor which may be selected from a list, presently including: canonical, policy, wild guess, informed guess, measured 100%, measured 30%, measured 10%, measured 1%, post mortem.

It is instructive to play with the spreadsheet, changing performance, productivity, and cost variables and seeing their effect on the figure of merit and its components on the last sheet of the spreadsheet. In this way, one can learn quickly in a particular environment what matters and what doesn't for the life-cycle productivity, our figure of merit. Even with the simplifications,¹⁵ and possible biases,¹⁶ built into this approach, we believe it goes furthest towards allowing a real understanding of how best to reach the goal of maximizing overall productivity. An example of the use of the spreadsheet to compare the productivity of two systems can be found in [²].