Goal 4, Priority 1: Enhancing Capability and Capacity of NCAR Supercomputing
The hallmark of supercomputing at NCAR has been provision of robust, reliable, effective, and efficient production computing with state-of-the-art storage, data analysis, data visualization services and tools for the user community. CISL increases the computational capacity available to our scientific community on a regular basis. Peripheral resources that complement and supplement the high-end computing environment are upgraded and enhanced as appropriate to match the growth in compute capacity.
FY2007 Accomplishments
In January 2007, NCAR selected the University of Wyoming and the State of Wyoming as foundational partners for creation of the expanded NCAR Supercomputing Center (NSC). Leading up to this event, CISL managed all aspects of the project discovery phase and prepared the broader community to support a management decision to re-establish the NSC in a new location. In particular, CISL facilitated the process of exploring partnerships to enable construction of an expanded NCAR supercomputing facility by conducting discussions with potential facility partners, performing technical and financial feasibility studies of partner proposals, and supporting NCAR and UCAR management in discussions of the project with members of the community, the UCAR Board of Trustees, and NSF.
Click to enlarge. This is a visualization of a coronal mass ejection impacting the Earth's magnetosphere. These phenomena can have significant impacts on satellites, astronauts, and systems ranging from GPS to power grids. Significant challenges in this field that will benefit from petascale computing include first-principles modeling of solar convection and its contribution to the 22-year solar cycle, and crucially, modeling the emergence of magnetic flux from the solar convection zone and the conditions that lead to solar flares and coronal mass ejections. (Image courtesy of Michael Wiltberger, NCAR HAO.)
Subsequent to hosting the NSF-funded High Performance Computing in the Geosciences Workshop at NCAR in September 2006, CISL provided geosciences researchers with an additional interactive forum to discuss and debate community high-performance computing requirements during a Town Hall session at the Fall 2006 AGU meeting. CISL personnel also presented their ideas and plans to University Corporation for Atmospheric Research members–which include representatives from each of UCAR’s member universities–and its Board of Trustees, to inform and get input from both groups. CISL staff also assisted various University of Wyoming and state officials with the preparation of support materials and conducting the question-and-answer sessions with Wyoming legislators–required for passage of the project-funding legislation. An NSC Project Office was created within CISL, and the staff assigned to this office is working with University of Wyoming personnel on a joint proposal to NSF to construct and operate the NSC.
Supercomputers–and supercomputing centers–are required to investigate fully today’s complex geoscience questions. Maximizing the efficiency of these centers requires developing the concept of a Grid that can federate computational resources, data, and skill sets, forming a whole that is greater than the sum of its parts. Because of the strategic importance of grid technology, NSF persuaded NCAR to join the TeraGrid in FY2006.
As of FY2007, deployment of the NCAR TeraGrid node is complete, and focus has shifted to preparing NCAR for its role as a TeraGrid resource provider. NCAR’s TeraGrid node consists of the IBM Blue Gene/L supercomputer, a 100-TB storage cluster, an Earth System Grid data server, and a visualization node, all connected to a 10-Gbps hub switch that is connected to the TeraGrid wide area network.
Click to enlarge. Frost is the 2,048-processor IBM Blue Gene/L system at NCAR. One quarter of this resource has been allocated to the TeraGrid, amounting to 4.5 million CPU hours of computer time annually. The system has been in production as a TeraGird resource since August 1, 2007. Frost is attached to a storage cluster, a visualization node, and has access to the multi-petabyte NCAR Mass Storage System. As a TeraGrid Resource Provider, NCAR is committed to offering a network of computational, data, and knowledge resources to multidisciplinary groups of researchers, students, educators, policy makers, and impact and assessment communities around the world.
In FY2007, CISL deployed the first phase of the Integrated Computing Environment for Scientific Simulation (ICESS), in which the IBM POWER4 computer was replaced by an IBM POWER5+ system. This increased the total computing capacity at NCAR by more than a factor of two beyond FY2006 levels. The increased computing capacity has also increased the average rate at which data are archived on the Mass Storage System (MSS), from 50 to 75 TB/month. Associated data analysis services are being upgraded to create a comprehensive environment capable of supporting low-end data processing through high-end visualization. A high-bandwidth, high-capacity shared file system has been installed to provide users with a large local storage area that can be used for advanced analysis and visualization needs.
CISL staff organized and hosted the eighth biennial session of Computing in Atmospheric Sciences (CAS2K7). Computer experts, scientists, and industry leaders gathered in Annecy, France during September 9-13, 2007, to discuss the status and future of high-performance computing for weather prediction and climate modeling. The conference provides the opportunity for supercomputing industry leaders to hear about their customers’ needs and present their product roadmaps, for scientists to exchange ideas and share experiences about computing resources and applications, and for data experts to discuss innovative methods for accessing, distributing, and archiving data.
Click to enlarge. MMM's Hurricane WRF model output in netCDF format in ArcMap overlaid with demographic and infrastructure data.
Also, a GIS Strategic Initiative, led by ISSE project scientist Olga Wilhelmi, is an interdisciplinary effort that integrates Earth system, environmental and social sciences through spatial analysis and interoperability of georeferenced information. The initiative promotes and supports the use of GIS as both an analysis and an infrastructure tool in atmospheric research and to address broader issues of geoinformatics and spatial data management. In FY2007, the GIS Initiative staff worked toward interoperability between weather and climate models and GIS analysis tools.
FY2008 Plans for Strategic Priority 1
During FY2008, the IBM POWER5+ system will be replaced by an IBM POWER6 system that is expected to provide at least a 2.5-fold increase over FY2007 performance levels. This will keep CISL at or above the compute capacity goal outlined its Strategic Plan. However, when installation of the POWER6 is complete, the NCAR Mesa Lab machine room will be within 10 percent of its operational power and cooling limits.
The increased computational capacity at NCAR, as well as the demand from NCAR scientists to import data from external computational facilities and/or observational field projects creates an increase in the demand for archival storage. Detailed plans will be made to carefully map the MSS needs and costs through the transition to the NSC. CISL plans to carefully examine MSS user policies and requirements, and then engage storage vendors to propose an overall solution to the MSS growth issues for 2008-2012 and perhaps beyond. At a minimum, another library must be added to the MSS during both FY2008 and FY2009 to increase the total archival storage capacity available at NCAR. Finally, CISL will deploy a large storage area network within the supercomputing environment to provide users with the most effective and efficient methods of data management across the machines.
An allocation model for TeraGrid users–with quotas–has been established for the NCAR MSS. CISL’s TeraGrid supercomputing resource, the Blue Gene/L, was acquired with a Major Research Infrastructure (MRI) grant from the NSF in spring 2005, and will be three years old in April 2008. It has been used to scale models up to high levels of parallelism, but as Blue Gene/L ages, a replacement system needs to be acquired to maintain a credible supercomputing presence on the TeraGrid. CISL is therefore planning to acquire a follow-on system in the FY2008 timeframe. This will allow CISL to better serve the supercomputing needs of UCAR, the TeraGrid, and partner institutions in the proposed Wyoming-based NSC.
In FY2008, the GIS Initiative will continue to work on improving compatibility, accessibility and accuracy of weather (i.e., WRF) models with GIS environment. We plan to investigate conversion mechanisms between sphere-based models and ellipsoid-based GIS data. Through participation in “The Potentials of and Limits to Adaptation in Norway” (PLAN) project, we will work on integrating quantitative and qualitative information in a GIS environment and, thus, improving methodologies for assessing societal vulnerability and adaptive capacity to climate change.
Related Lab Annual Report Sections:
Goal 4, Priority 1