Jeffrey C. Carver, Mississippi State University
Lorin M. Hochstein, University of Nebraska, Lincoln
Richard P. Kendall, Information Sciences Institute, University of Southern California
Taiga Nakamura, University of Maryland, College Park
Marvin V. Zelkowitz, University of Maryland, College Park; Fraunhofer Center for Experimental Software Engineering
Victor R. Basili, University of Maryland, College Park; Fraunhofer Center for Experimental Software Engineering
Douglass E. Post, DoD High Performance Computing Modernization Office
2
2. Goals & methodologies
While the ultimate goal for the case studies of the ASC codes and the MP codes is the same (to improve the productivity of computational scientists and engineers), we have used a different approach for each type of code. The object of study in the ASC codes has been the individual programmer (e.g., “Where does the computational scientist spend her time?”), while the object of study in the MP studies has been the project (e.g. “Which factors determine project success?”). Table 2 shows a comparison of the goals of the two types of case studies.
| ASC Codes |
MP Codes |
- Characterize which scientific programming activities are time-consuming and problematic
- Characterize the common problems encountered by programmers
- Characterize the impact of technologies on developer effort
|
- Identify project success factors
- Identify ways that successful projects manage risk
- Identify productivity barriers that should be addressed by vendors
- Develop a reference body of case studies
|
Table 2. Case study goals.
Table 3 provides an overview of the methodology used for each type of case study. In general, the approach for the MP codes was more comprehensive (longer questionnaire, on-site interviews, multiple subjects interviewed independently), and the approach for the ASC codes was more lightweight, which permitted quicker turnaround time when running the studies. Furthermore, each type of study collected different types of information. The focus in the ASC was lower level (i.e., more details about fewer things), while the focus in the MP codes was higher level (i.e., less details about more things).
|
ASC Codes |
MP Codes |
| Type |
Ongoing |
Retrospective |
| Interviewees |
Technical leads |
Projects leads, project staff |
| Overview |
- Pre-interview questionnaire
- Telephone interview
- Generate summary document
- Send summary document for approval/comments
- Generate synthesis report across all projects
- Send synthesis report to all centers for approval/ comments
|
- Identify project and sponsors
- Negotiate case study participation
- Pre-interview questionnaire
- On-site interview
- Initial list of findings
- Follow-up
- Write report
|
| Focus |
- Product: attributes, machine target, history
- Project organization: structure, staff, configuration management
- Development activities: adding new features, testing, tuning, debugging, porting, effort distribution, bottlenecks, achieving performance
- Programming models and productivity: choice of model, adoption of language, productivity measures
|
- Goals, requirements, deliverables
- Project characteristics, structure, organization and risks
- Code Characteristics
- Staffing
- Workflow Management
- V&V, Testing
- Success Measures
- Lessons Learned
|
Table 3. Methodology.
