Establishing a single, reasonably objective and quantitative framework to compare competing high productivity computing systems has been difficult to accomplish. There are many reasons for this, not the least of which is the inevitable subjective component of the concept of productivity. Compounding the difficulty, there are many elements that make up productivity and these are weighted and interrelated differently in the wide range of contexts into which a computer may be placed. Significantly improved productivity for high performance government and scientific computing is the key goal of the High Productivity Computing Systems (HPCS) program. Evaluating this critical characteristic across these contexts is clearly essential to attaining and confirming this goal.

This is not entirely a new phenomenon. Anyone who has driven a large scale computing budget request and procurement has had to address the problem of turning a set of preferences and criteria, newly defined by management, into a budget justification and a procurement figure of merit that will pass muster with agency (and OMB) auditors. The process of creating such a procurement figure of merit helps to focus the mind and cut through the complexity of competing user demands and computing options.

Imagining that we are initiating a procurement where Productivity = Utility/Cost will be the criteria, we have developed a total productivity figure of merit. This framework includes such system measurables as machine performance and reliability, developer productivity, and administration overhead and effectiveness of resource allocation. This is all in the context of information from the particular computing environment that may be proposing and procuring an HPCS computer. We note that this framework is applicable across the broad range of environments represented by HPCS mission partners and others with science and enterprise missions that are candidates for such systems.

The value of each variable in our figure of merit is intended to come from a single source, either from one of the HPCS R&D areas or from a mission organization that may procure a system. While we identify the potential source of each value, we do recognize that some of these numbers will not be easy to obtain, particularly those involving the impact of system design on human productivity. Nonetheless, we believe that, at the least, this framework will identify the individual metrics that these efforts should strive to measure. In the end, we will all have to admit that some combination of measurements, informed guesses, and subjective evaluations will be needed to arrive at a figure of merit number.

We also recognize that there is coupling between some of the variables we treat as independent. For example, a user's productivity is impacted by the way jobs are allocated. To deal with this, we suggest assuming an environment in which a particular variable is determined. This means that values for all the other variables, their “operating point,” must be specified for each measurement of a variable. ¹ For measurable variables, these operating points come from measurements and studies. One could iterate to a final answer, but we argue that this is unnecessary because the effect of any such coupling, with reasonable operating point guesses, is far smaller than the precision (such as it might be) of any of the measured variables involved.

Not surprisingly, our figure of merit has much in common with ideas expressed in earlier HPCS work. ³ However, we extend beyond this in the following ways:

In a well-balanced HPCS, significant costs will be incurred for resources other than just the CPU cycles that dominate thinking in the commodity cluster architectures. In particular, memory and bandwidth resources will have cost as much or more than CPU, and efficient programs and job allocation will have to optimize use of memory and bandwidth resources as much as CPU. Our framework allows for the inclusion of any set of significantly costly resources.

A single job may be highly optimized, and those in the project it comes from will inevitably believe its utility (or value) approaches infinity. However, a computer center must optimize the total workload, given its organization's evaluation of relative priority (utility or value) for each project and job. The overall utility of the total output of the computer depends on the degree to which the allocation of system resources reflects the institution's priorities and determination of value (or utility). Further, the productivity of the administration staff depends on a system's administration environment and tools, and on its stability. ⁵

The remainder of this article is organized as follows. In Section 2 we define the productivity figure of merit as a combination of factors, where each factor represents a different aspect of productivity that can be evaluated relatively independently of the others. In Section 3 we show how the figure of merit captures the perspectives on productivity of people in different institutional roles and we show how productivity ratios can be used to simplify the evaluation process. In Section 4 we discuss the need for productivity benchmarking and propose the use of operating points to narrow the scope of the analysis needed for a productivity evaluation. We provide a recipe for using the figure of merit in Section 5. We conclude and describe our experience applying the figure of merit using a simple spreadsheet (available from the authors) in Section 6.

In factoring out the productivity contributors, we take an approach that, like the blind men and the elephant, focuses on what can be measured at defined parts of the beast and builds to a picture of the whole productivity equation for a system in a particular environment. This assumes that our elephant is reasonably smooth and predictable between those points we can feel with our measurements. ⁷

We start with

Productivity = Utility/Cost

We expand the utility into system level and job level components

(1)

As a convention, we use the letters U, E, A, R, C to denote variables of utility, efficiency, availability, resources, and cost, respectively. The subscripts indicate the variables that address system level (including administrative and utility) and job level factors.

We recognize that some aspects of the system level efficiency will never be amenable to measurement and will always require subjective evaluation. Only subjective evaluation processes can address the first two variables in the utility numerator. In principle one can measure the last four variables and the HPCS research program is addressing such measurements.

We have been emphasizing that this is to be a figure of merit, including estimates and evaluations of what we expect the productivity output of an installation to be. For clarity, in explaining this formulation, we will start by talking as if we know what happened over the lifetime, as if we are doing a post-mortem. We will mention in passing how the components relate to ongoing work on productivity estimators. We will return to discuss these estimators in more detail in Section 4.

The goal of those optimizing utility at the job level is to maximize resources they can effectively apply to their problem. This will enable them to bring their project to a successful conclusion with higher utility (larger scale or finer granularity solutions or higher throughput of data intensive problems) or more rapidly (allowing more problems of similar utility to be accomplished within the lifetime of the resources). It is “not in their job description” to address the relative utility of their problem compared to others (though they may be inclined to do so). So, we consider utility at the job level, U_job, to be just the cost ($) of the resources that they have effectively used, and the job level efficiency E_job = U_job/C_sys, with C_sys the total lifetime system cost. As defined below and in the Appendix, U_job and E_job are averaged over all jobs.

The government, or the owners or stockholders, establish how activities are to be valued by the institution management. Management valuation appears in the largely subjective U_sys variable, which includes, for example, what constitutes “success” for a computing activity. They may well value a successfully completed activity higher than the cost of the resources used. (It would be unusual for them not to do so at budget proposal time.) U_sys assumes, as managers might often wish, that all resources are available and fully allocated at all times during the life of their expensive system.⁸ We define it as a multiplier to the peak system resources in CPU units. System utilization efficiency and project level effectiveness is included in E_adm and E_proj as described below.

Once a system is delivered, the system administrators (and vendor) strive to meet management expectations for availability, A_sys and system level resource utilization efficiency (including allocation), E_adm.

More detailed descriptions of the individual variables follow and they are expanded further in the Appendix. We indicate dimensional units in brackets.

P, the productivity, is dimensionless, as required by the common economic interpretation of the productivity concept we use, output/input, [$]/[$].

C is total cost of ownership, including all costs for developing software and running it on the computer, over a defined lifetime of the system (T). This is expanded into components in the Appendix. [$]

T is the lifetime of the system as defined in the typical budget approval process. It does not show up explicitly in the top-level equation, but it is important to the definitions of the variables as well as for the measurements. It will depend on considerations specific to each environment, including, for example, whether the procurement and justification (and cost) involve continuing upgrades. Those responsible for budget submissions at the proposing institution are the source of this variable. [yrs]

E_job is the ratio of U_job to the total system cost C_sys. U_job is the total productively used resources over the lifetime by all individual jobs in all project activities, normalized to the assumption of 100% availability and 100% resource allocation. To a large extent, this efficiency measures how well programs use parallel resources. We include other costly resources (e.g., memory, bandwidth, I/O) besides CPU. In order not to favor one class of resources over another, we weight the resources by their relative costs.⁹ We say “productively used” so that we only include resources that would be consumed in direct support of maximizing utility for the specified problem. Estimators for E_job and for , the corresponding average total project personnel time (including development and production) per project, are what Job-level Productivity Benchmarks should aim to measure for different environments and systems. More detailed definitions of these can be found in.² [Dimensionless]

A_sys is the availability; the fraction of total resource time available to the jobs making up R_job. This accounts for all planned and unplanned downtimes, but does not include job restarts due to failures, which is included in E_sys below. Note this is not fraction of system time, but fraction of resource time, so that it accounts for portions of the large system being available when other portions are down. This parameter is based on vendor estimates of MTTF, maintenance requirements, and other RAS considerations. [Dimensionless]

The product E_sys≡E_admE_proj is the effectiveness of the workload resource allocation at meeting the institution's priorities. System architecture, software, and administrative tools can make a big difference to this important metric. This is where utility (or value) enters the picture and we go beyond just looking at job-level optimization. We assume that the institution will have a process for establishing the priority or the utility or the value of the individual projects and jobs. E_sys is the ratio of the actual total value of the jobs run over the lifetime to the optimal total that would come from maximally efficient management of a reference HPCS platform,¹⁰ for a given job mix, a given installed system configuration, and a given time pattern of resource downtime. As shown in [²], we allow for time dependent values, since allocation efficiency that completes a job or project long after it is needed has little or no value. We recognize explicitly (see further discussion in [²]) that not all the factors that go into E_sys are amenable, even in principle, to before or after the fact quantitative measurement. Some will need to be evaluated subjectively, and primarily have to do with the effectiveness with which projects (management assigned and valued tasks) are able to accomplish management goals and schedules. This is why we have written E_sys as a product with E_adm as the measurable part and E_proj the project effectiveness part requiring subjective evaluation. E_adm is the estimator that, along with the cost of administration, is what System-level and Administration Productivity Benchmarks should aim to measure, for different systems, configurations, and user environments. E_proj represents other considerations that result in reduced efficiency at delivering utility. [Dimensionless]

U_sys is the summed utility (or value) of all projects or activities, per unit of total available resources, R, that would complete successfully during the lifetime T if the system administrators given the systems tools and capabilities of the HPCS Reference Platform were able to attain the optimum level of resource utilization and the system was perfectly available. The values may be assigned by whatever process the institution uses to prioritize its projects and justify its budget proposals. This variable only includes the local institution's evaluation of the value of the projects it intends for the installation. In some environments, it may be ignored and set to a constant such as C_sys. We convert all resource units to CPU operations, weighting by their relative costs. [$/ops]

Individuals with different professional responsibilities look at productivity with naturally different perspectives. These include, for example, acquisition decision makers, project managers, individual programmers (“users”), researchers, system administrators, service engineers, operators, vendors, and system designers. It is useful to focus on two of these, project managers, and decision makers.

In principle, project manager perspectives are aligned with their institutional management, the decision makers. In practice, they differ because at least our stereotype of a project manager is only concerned with project personnel costs and not either machine operating or capital costs. And similarly the project manager can only address certain terms in the utility numerator of the productivity ratio. So, the project manager's perspective on productivity is a subset of the decision maker in Eqs.1 and 2,

(2)

The decision maker's (acquisition) productivity we developed in earlier sections is then

(3)

Here E_org=E_admA_sys is the organization's and system's multi-project resource utilization efficiency.

We can get considerably more simplification and, perhaps, also more insight, if we now think in terms of comparing the new, next generation (HPCS) system that we are evaluating to a standard reference machine. This reference can be a defined traditional MPI cluster configuration for which there is some level of understanding and experience regarding productivity in its environment. All measurements and terms in Eq. 8 can be ratios of the new system to the reference. In most organizations, for budget and institutional reasons, the ratio of project (development personnel) costs to machine and operations costs is typically a constant. In this situation we conclude that

(4)

The normalization to the reference system is indicated by bars. We assume that management evaluation of utility obtainable per effectively utilized resource is identical for the two systems (=1). The not-surprising conclusion is that we can consider the relative improvement in productivity to be just the product of productivity measurables for the new system normalized to the old.

We could get an accurate figure of merit as part of a post-mortem – after the life cycle is over. At that point, we could have access to real experience. But that's not what we want; we need to predict and estimate in advance. So where do our productivity estimators come from? We have been assuming there are two classes of productivity benchmarking activities: 1) at the job- and project-level, measuring development and production activities; and, 2) at the system-level, measuring administration activities and the effect of the differences in system designs and configuration options on administration overhead and system-level resource utilization.

Development, job- and project-level, productivity benchmarks would aim to measure - for the problem mix of a specific environment - the development time required to attain different levels of productive resource utilization. The simplest example is how much programming time it takes to attain various levels of speedup. Curves of productive resource utilization vs. development and other project personnel time will increase, probably like step functions and will usually present an obvious point of diminishing returns. This can be taken as the “operating point.” Averaged over the work flow and job mix, it provides an estimator of and E_CPU, E_mem, E_BW, and E_IO.

Similarly, administration, system-level, benchmarks can aim to measure E_adm. For aspects involving effective scheduling, this could be accomplished, specific to a given environment and system configuration, by creating a simulated scheduling environment and allowing working administrators to attempt to optimize the allocation of a list of prioritized (value assigned) jobs for some simulated time period. An ideal allocation would be defined for each environment as attaining the full value of each project including time considerations. The ratio of the measured to the ideal would give an estimator for this aspect of E_sys. Just as for the development benchmarks, this can be treated as a curve where better allocation, more efficient use of resources, and a general increase in output value, uses more administrator time, and a reasonable operating point of diminishing returns is selected. The cost of the administrator time at this operating point would be included as all or part of c_adm (depending on how complete a benchmark simulation is used).

Here is a step by step approach to a normalized productivity figure of merit based on workflow productivity measurements:

1. Define a standard reference system for comparison, a “canonical cluster”. Productivity estimates are made on this system and on the new system under evaluation and reported as ratios.
2. Define the set of working environments (application areas, project units, unique program activities) in terms of the resource emphasis (FLOPs, GUPs, bytes, …) and standard personnel rates.
3. For a few (usually just one) activity classes in each environment, describe the workflow as the effort fraction of standard workflow elements. Select a set of representative applications for attained performance (E_job) measurements. Design job-level productivity measurements for the workflow elements and measure the productivity term, ¹¹.
4. Define a typical job mix and the overall administrative workflow, and design administrative-level productivity measurements. Measure ¹³.
5. Define subjective scoring criteria for for each environment, and score the systems with regard to project success rate, accessibility, ease of use effectiveness, …
6. Vendors provide information to identify the relevant resource capability ratios of the systems, such as peak performance () or GUPs, etc. Vendors also provide information on availability of the new system. Reference system availability should be obtained from experience data.
7. Combine factors into the relative productivity figure of merit,
8. If absolute Productivity, P, is of interest, costs must be included and a utility constant (U_sys) defined. Costs include project and administrative personnel budgets, and system costs including initial costs and running costs over a defined lifetime, T.

We have tried to demonstrate that the productivity figure of merit we have described here is really much simpler than it may have first appeared. It is no more than a way of getting to a single number that combines what we know or can guess about a system configured for a particular environment into something that approximates a measure of total productivity. It can be used in HPCS design comparisons, subsequent budget justifications, and ultimately, we hope, in procurements.

Many of the numbers needed as input are traditional cost and performance variables. The hard parts, of course, are the benchmarks needed to measure the productivity of humans when confronted with these new systems. We see these productivity benchmarks as simply measuring curves of efficiency of job resource utilization, or system utility optimization, vs. the cost in time of the human effort. These curves should clearly indicate an obvious point of diminishing returns, an operating point.

We recognize that we have made use of words like simply, clearly, obviously, and this may be unfair. We know that getting a number that approaches being a true measure of productivity for a given system is going to be difficult. We need goals for the productivity benchmarking efforts to aim at, and we think this framework provides them.

We have developed a spreadsheet, which may be obtained from the authors, that allows one to gain some intuition into how system-wide productivity comes together from the many components. The first six sheets of the spreadsheet are entry sheets to be completed by the different entities that may be responsible for the information: Vendor Cost, Host Environment and Values, Performance, Job-level Productivity, Administration Productivity, and Subjective Productivity Evaluation. The last sheet summarizes the calculated result.

We see the process of obtaining an informative figure of merit as being incremental. One may start by entering guesses, policies, and goals, and then progress through preliminary measurements and even post-mortem analysis. For each entry, the spreadsheet has a quality descriptor which may be selected from a list, presently including: canonical, policy, wild guess, informed guess, measured 100%, measured 30%, measured 10%, measured 1%, post mortem.

It is instructive to play with the spreadsheet, changing performance, productivity, and cost variables and seeing their effect on the figure of merit and its components on the last sheet of the spreadsheet. In this way, one can learn quickly in a particular environment what matters and what doesn't for the life-cycle productivity, our figure of merit. Even with the simplifications,¹⁵ and possible biases,¹⁶ built into this approach, we believe it goes furthest towards allowing a real understanding of how best to reach the goal of maximizing overall productivity. An example of the use of the spreadsheet to compare the productivity of two systems can be found in [²].

Here we expand the high level terms of Eq.1:

A. Cost

(5)

Here,

(6)

is the total life cycle cost of the system. [$]

It includes the following costs:

One Time Costs [$] Initial hardware Initial software (system and purchased application software) Non-standard or specialized facility equipment (e.g., water cooling) Facility preparation (electrical and other building modification) Installation

Recurring Costs [$/yr] Hardware maintenance (repair and preventive) Software maintenance (repair and preventive) Facility maintenance Hardware and software upgrades (as required by contracted goals) Electrical costs (for system and cooling)17 Space and standard facility equipment (GSA or other rate per sq-ft per year)

is the total lifetime administration costs.

n_adm is average number of personnel at the local institution supporting system maintenance and management, user assistance, and system workload optimization. c_adm is the cost per person-year [$/person-year] of this support. These costs are in large measure affected by the workload environment as well as the system and its administration tools. n_adm is the personnel load that System-level and Administration Productivity Benchmarks can aim to measure along with E_adm. n_adm can be broken into two parts, one that is a baseline cost in the host environment for managing a system of this magnitude, and a part that corresponds to the effort to optimize utilization and other activities where the administrators' productivity can be affected by administrative tools, . [Persons]

N is the total number, over the lifetime T, of unique program activities (projects with the goal of creating an appropriately efficient program and a correct and useful result). The definition of program activities and an estimate of N is very dependent on the local environment and should come from the institution for which systems are being evaluated. [Dimensionless]

is the average total project personnel time for one of the unique program activities counted by N. This includes, in addition to the software development effort, personnel costs for any production related effort (e.g., job submission, bookkeeping, etc.). This, along with E_job, is what Job-level Productivity Benchmarks should aim to measure for different environments and systems. [Person-years]

c_proj is the average cost per person-year of all project-related personnel. [$/Person-year]

Alternatively, we can write project costs in terms of the average total number of project personnel, n_proj, over the lifetime, T, so that the total project cost is .

B. Productively Used Resources at the Job Level

As noted before we define the job level utility as the cost of the resources productively used

U_job=c_CPUE_CPUR_CPU+c_memE_memR_mem+c_BWE_BWR_BW+c₁₀E₁₀R₁₀ (7)

and the job level efficiency as

E_job=U_job/C_sys (8)

Here, the subscripts refer to the resource types, CPU, memory, inter-processor bandwidth, and I/O bandwidth resources, respectively. This can obviously be generalized to include other resources with significant costs.

The c_r are the total costs attributable to each resource per unit of that resource. The total life cost is C_sys described above,

C_sys=c_CPUR_CPU+c_memR_mem+c_BWR_BW+c₁₀R₁₀ (9)

The R_r are the total lifetime resources of type r used by all of the N project activities.¹⁸ The resources are assumed for the purposes of this job-level variable to have been 100% allocated and the time to be 100% available for one of the N activities.¹⁹ Note that costs only come into Eq.5 to provide a relative weight for the different resources.

The R_r are based on performance measurements and a choice of configuration options. Remember that the R_r include the lifetime, so that the units for CPU, memory, bandwidth, and IO are [ops] (not ops/sec), [byte-years], [bytes], and [bytes], respectively. We weight the resources by their relative costs, so U_job, is just the cost ($) of the resources that the project teams could have used if they were perfectly efficient. The c_r provide the conversion from resource units to utility as cost in $ [$/ops], [$/byte-years], [$/bytes], [$/bytes].

The E_r are the fraction of the total of each resource productively utilized on average by all the jobs in the N project activities. The variables E_r are efficiency estimators that Job-level Productivity Benchmark efforts can aim to measure (for specified workflows and job mixes) along with the average effort per activity, .

By “productively,” we mean that resources used only to support parallelization are not counted, and that the single processor algorithm being used is not wasteful of resources. This can either be a protocol rule in the benchmarking or the benchmarks can down-rate the utilization fractions E_r for resource usage that is not in direct support of the task or algorithms that are less than optimal.²⁰

The E_r are [dimensionless].

C. Project and System Level Efficiency

E_adm, the measurable part of system efficiency, may be understood as the effectiveness of the administrative staff in allocating resources efficiently given the tools and the real environment of their system and the time that they have available, as included in the cost c_adm. An estimator for this traditional measure of system utilization, E_adm, is what System-level and Administration Productivity Benchmarks can aim to measure, as discussed in Section 4.

E_proj is the subjective component of the figure of merit to allow for evaluation of issues, which might be attributed to system hardware or software, such as project failures or delays and the accessibility of the computing system environment to staff with different levels of computing skills. In general, it includes system or configuration factors that impact the effectiveness of programming teams at accomplishing their goals. This is where utility vs. time considerations may be included, as discussed in [¹].

E_adm and E_proj are [dimensionless].

¹ This concept is described a bit more rigorously in [2].
² Murphy, D., Nash, T., Votta, L. Jr., “A System-wide Productivity Figure of Merit,” Sun Labs Technical Report, TR-2006-154, Sun Microsystems Laboratory (2006). http://sunlabs.com/techrep/2006/abstract-154.html
³ In particular, the important aspects that address user work productivity at the job-level reduce identically to Sterling's w-model of productivity [4].
⁴ Sterling, T., “Productivity Metrics and Models for High Performance Computing,” Journal High Performance Computing and Applications 18(4):433-440 (2004).
⁵ These extensions may be seen as a response to Kepner's remark “that all necessary ensemble averaging over users, applications and systems can be performed without loss of generality.” In fact, we show in [2] that our figure of merit follows from ensemble averaging in this way, starting essentially at the synthesis expression in Section 2.2 of Kepner's paper [6].
⁶ Kepner, J. “HPC Productivity Model Synthesis,” Journal of High Performance Computing and Applications 18(4):505-523 (2004).
⁷ One can also come to the same formula by looking at the elephant as if we were the committee that designed it, with a completely parameterized model. From this complete model, by calling out assumptions and simplifications, we can reduce to our productivity figure of merit. This approach is pursued in an appendix of [2] and the methodology can be used as a basis to derive a custom figure of merit in cases where our assumptions and simplifications are not applicable to a site's needs.
⁸ However, a system approaching 100% allocation will have long queues that would have a deleterious effect on the total utility coming out of the computer. Effects like long queues should be included as part of the environment conditions (the “operating point”) in which job level productivity measurements of E_job and , defined below, are made.
⁹ For example c_CPU is that fraction of the total lifetime cost, C_sys, that is attributable to CPU, in $/ops. For CPU, U_job can be understood (in Sterling's notation [4]) as S x T x c_CPU/C_sys summed over all jobs during the lifetime T, with S the attained performance [ops/sec].
¹⁰ Since E_sys is intended to compare the system level resource utilization efficiency for different system designs, vendors, and configurations, it needs to be a ratio to some absolute standard. A reasonable definition that we will use is the “ideal” one, which delivers the sum of the values of all the projects.
¹¹ The sustained systems performance benchmark (SSP)[12] developed at NERSC is the result of this type of analysis. It is used in practice to assist in procurement decisions.
¹² Kramer, W., Shalf, J. Strohmaier, E. “The NERSC Sustained System Performance (SSP) Metric,” Lawrence Berkeley National Laboratory. Paper LBNL-58868 (2005). http://repositories.cdlib.org/lbnl/LBNL-58868
¹³ The Effective System Performance benchmark (ESP)[14] developed at NERSC is an important step toward this type of evaluation.
¹⁴ Wong, A., Oliker, L., Dramer, W., Kaltz, T., Bailey, D. “ESP: A system utilization benchmark,” In Proceedings of the Supercomputing 2000 Conference (2000).
¹⁵ In the likely event that host environments wish to specify time-dependent values for projects, these may be parameterized functions or tables. In either case, the spreadsheet will have to be revised to accommodate time-dependent values and system level efficiencies since the present examples do not take time into account.
¹⁶ Value judgments may be implicit in some of our simplifications. If these are not consistent with those of a particular environment, they can obviously be adjusted using the basic framework of this approach and the spreadsheet. What is important is to attempt to include as many cost, performance, and productivity components as possible into the best possible single productivity figure of merit.
¹⁷ Note that electrical running costs may be dependent upon the job mix.
¹⁸ Sums over the jobs in a workflow and the projects at a computing center are shown explicitly in [2].
¹⁹ This assumption includes a simplification that the different resources are all available and allocated in the same proportions. It normalizes U_job so that for perfect job level utility optimization U_job = C_sys.
²⁰ It may be best to define “productive use of resources” with examples. If we replicate memory just to reduce latency by keeping data close to the CPUs in a parallelized problem, this use of the memory resource does not count as productive, although the CPU utilization may increase and be counted as a result. But if the parallelization allows us to expand the problem space (larger matrices, reduced granularity, etc.), or increase the throughput in data intensive situations, and increase the utility of a successful result, this memory use does count. Similarly, bandwidth to access non-local memory only counts if utility is increased as a result, and not just if it is replicated and “indirect” support of parallelization. The excess usage of a resource wasteful algorithm (for example, one that burns resources unnecessarily on a single processor) or of wasteful localization of a scaled up problem (for example, data placed so unnecessary bandwidth is used) does not count as productive use. In the end, to paraphrase Justice Potter Stewart in the context of something else difficult to define, we know “productive resource utilization” when we see it.