We have realized in-situ visualization for a terascale earthquake simulation.9 This work also won the HPC Analytics Challenges at the SC 2006 Conference10 because of the scalability and interactive volume visualization we demonstrated. Over a wide-area network, we were able to interactively change view angles, adjust sampling steps, edit the color and opacity transfer function, and zoom in and out for visually monitoring the simulation running on 2048 processors of a supercomputer at the Pittsburgh Supercomputing Center. We were able to achieve high parallel efficiency exactly because we made the visualization calculations, i.e., direct volume rendering, to use the data structures used by simulation code, which removes the need to reorganize the simulation output and replicate data. Rendering is done in-situ using the same data partitioning made by the simulation, and thus no data movement is needed among processors. Similar to the traditional parallel volume rendering algorithms, our parallel in-situ rendering pipeline consists of two stages: parallel rendering and parallel image compositing. In the rendering stage, each processor renders its local data using software ray-casting. Note that this stage may not be balanced given a set of visualization parameters and the transfer function used. In the image compositing stage, a new algorithm is designed to build a communication schedule in parallel on the fly. The basic idea is to balance the overall visualization workload by carefully distributing the compositing calculations. This is possible because parallel image compositing uses only the data generated by the rendering stage and is thus completely independent of the simulation.
For implementation of in-situ visualization, no significant change is needed for the earthquake simulation code for the integration. The only requirement for the simulation is to provide APIs for the access of the simulation internal data structure, which does not require much effort in practice. Furthermore, because all the access is a read operation, the simulation context is not affected by the visualization calculations. The advantage of our approach is obvious. Scientists do not need to change their code to incorporate in-situ visualization. They only need to provide an interface for the visualization code to access their data, as everything else is taken care of by the visualization part. This approach is certainly the most acceptable by scientists.
We are not too far from peta- and exa-scale computing. Will we have the adequate tools for possibly extracting meaning from the data sets generated by such extreme-scale simulations? The investment made by the DOE SciDAC program in ultra-scale visualization [2] is timely and ensures that challenges will be addressed. In this article, we point out the grand challenges facing extreme-scale data analysis and visualization, and present several key technologies for gaining insights in ultra-scale simulations. While we have had some success in deploying some of these technologies, further research and experimental studies are still needed to make these new technologies benefit the scientific supercomputing community at large.
This work is supported in part by the DOE SciDAC program and NSF ITR program. The images displayed in this article were made by members of the Ultravis Institute and the VIDI research group at University of California at Davis. The supernova data set was provided by Dr. John Blondin at North Carolina State University. The turbulent combustion data set was provided by Dr. Jackie Chen at Sandia National Laboratory.
2 Ma, K. –L., Ross, R., Huang, J., Humphreys, G., Max, N., Moreland, K., Owens, J. D., Shen, H.-W. “Ultra-scale visualization: research and education,” Journal of Physics, Vol. 78. (also Proceedings of SciDAC 2007 Conference, 24-28 June, 2007, Boston, Massachusetts).
3 Scientific Discovery through Advanced Computing, Office of Science, Department of Energy - www.scidac.gov/
4 Yu, H., Wang, C., Ma, K.-L. “Parallel Hierarchical Visualization of Large 3D Time-Varying Vector Fields,” in Proceedings of the ACM/IEEE Supercomputing 2007 Conference (SC ’07).
5 Ma, K.-L. “Visualizing Visualizations: User Interfaces for Managing and Exploring Scientific Visualization Data,” IEEE Computer Graphics and Applications, Vol. 20, Number 5, 2000, pp. 16-19.
6 K.-L. Ma. “Machine Learning to Boost the Next Generation of Visualization Technology,” IEEE Computer Graphics and Applications, Volume 27, Number 5, 2007, pp. 6-9.
7 Akiba, H., Ma, K.-L. “A Tri-Space Visualization Interface for Analyzing Time-Varying Multivariate Volume Data,” In Proceedings of Eurographics/IEEE VGTC Symposium on Visualization, 2007, pp. 115-122.
8 Jones, C., Ma, K.-L., Sanderson, A., Myers Jr., L. R. “Visual Interrogation of Gyrokinetic Particle Simulation,” Journal of Physics, Vol. 78. (also Proceedings of SciDAC 2007 Conference, 24-28 June, 2007, Boston, Massachusetts).
9 Tu, T., Yu, H., Ramirez-Guzman, L., Bielak, J., Ghattas, O., Ma, K.-L., O’Hallaron, D. R. “From mesh generation to scientific visualization: an end-to-end approach to parallel supercomputing,” in Proceedings of ACM/IEEE Supercomputing 2006 Conference (SC ’06).
10 Yu, H., Tu, T., Bielak, J., Ghattas, O., Lopez, J. C., Ma, K.-L. O’Hallaron, D. R. Ramirezguzman, L., Stone, N., Taborda-Rios, R., Urbanic, J. “Remote runtime steering of integrated terascale simulation and visualization,” HPC Analytics Challenge, ACM/IEEE Supercomputing 2006 Conference (SC ’06).
