We believe that programmers go through an identifiable, repeated process when developing programs, which can be characterized by a directed graph model such as timed Markov models. We successfully gathered data and fit it to timed Markov models twice, in our 2004 and 2006 classroom experiments. The replay of programmer experience offline was one of the most important aspects of the data gathering process. The timed Markov models clearly indicate the most time intensive parts of the development cycle, quantitatively confirming our intuition – programmers spend most of their time debugging and, once they get a correct program, tuning it for performance is even more difficult. Our data also suggests, in this context, that programmers may introduce slightly fewer bugs in UPC programs and find it easier to optimize them, as compared to C/MPI programs.
Clearly, this is only the beginning. A lot more data needs to be collected before languages can be compared in a meaningful way. Towards this end, we are building various tools for the community at large. These tools will provide a general framework for data collection and model construction to study programmer productivity in a variety of ways.
2Hochstein, L., Basili, V. R. "An empirical study to compare two parallel programming models," In SPAA ’06, Proceedings of the eighteenth annual ACM symposium on Parallelism in algorithms and architectures, pages 114–114, New York, NY, USA, 2006. ACM Press.
3Post, D., Kendall, R. "Software project management and quality engineering, practices for complex, coupled multi-physics, massively parallel computational simulations: Lessons learned from asci," Los Alamos National Laboratory, 2003.
4Dongarra, J. J., Otto, S. W., Snir, M., Walker, D. "A message passing standard for MPP and workstations," Commun. ACM, 39(7):84–90, 1996.
5The UPC consortium. Upc language specifications v1.2. May 2005. upc.lbl.gov/docs/user/upc_spec_1.2.pdf
6Howard, R. Dynamic probabilistic systems. John Wiley, New York, 1971.
7Johnson, E. W., Brockman, J. B. "Measurement and analysis of sequential design processes," ACM Transactions on Design Automation of Electronic Systems, 3(1):1–20, 1998.
8Smith, R. P., Eppinger, S. D. "Identifying controlling features of engineering design iteration," Management Science, 43(3):276–293, Mar 1997.
9Smith, B., Mizell, D., Gilbert, J. R., Shah, V. "Towards a timed markov process model of softare development," Second International Workshop on Software Engineering for High Performance Computing System Applications, 2005.
10Shah, V., Gilbert, J. R., Aggarwal, V. "Productivity measurement in the classroom: Experiments and analysis," In preparation.
11Zelkowitz, M., Basili, B., Asgari, S., Hochstein, L., Hollingsworth, J., Nakamura, T. "Measuring productivity on high performance computers," In METRICS ’05, Proceedings of the 11th IEEE International Software Metrics Symposium (METRICS’05), page 6, Washington, DC, USA, 2005. IEEE Computer Society.
12Hochstein, L., Basili, V. R., Zelkowitz, M. V., Hollingsworth, J. K., Carver, J. "Combining self-reported and automatic data to improve programming effort measurement," In ESEC/FSE-13, Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering, pages 356–365, New York, NY, USA, 2005. ACM Press.
13 McCreary, C., Barowski, L. A. "VGJ: Visualizing graphs through java," In Graph Drawing, pages 454–455, 1998.
14Raitner, M. The gml file format. 2001. www.infosun.fmi.unipassau.de/Graphlet/GML/index.html
