Very large computational experiments: Accelerating science

This figure was produced in FY2008 from computational experiments carried out as part of FY2007's Breakthrough Science initiative and shows volume rendering of the amount of alignment between magnetic field and velocity in a 1536³ direct numerical simulation of magnetohydrodynamic (MHD) turbulence. Alignment is strong (blue-to-hot-pink) in the both rolled-up (and surrounding) sheets, but weak in between (salmon). This property helps explain the formation of such structures as the roll-up in turbulent MHD flows, and may lead to an understanding of the effect of these small-scale structures on the large-scale dynamics. (Figure courtesy Annick Pouquet, NCAR and Pablo Mininni, NCAR). The success of FY2007's Breakthrough Science initiative in providing increased opportunities for very large computational experiments has encouraged CISL to expand this program with the installation of bluefire in FY2008. Mechanisms were put in place to increase the size of computational experiments that could be accomplished with CISL supercomputing resources by the university, NCAR, and Climate Simulation Laboratory (CSL) communities. These mechanisms included:

Solicit proposals for the Accelerated Scientific Discovery (ASD) initiative from universities, the NSF, and NCAR to use almost 3 million bluefire processor hours during September, October, and November 2008.
Encourage universities to submit very large proposals to the October CISL HPC Panel (CHAP) by emphasizing the very large requests that the CHAP approved in April 2008 and the increased availability of HPC resources to the university community.
Encourage NCAR research based on very large computational experiments by reserving and allocating some computing resources through the NCAR Executive Committee.
Establish a minimum allocation award size for the Climate Simulation Laboratory (CSL) during the next round of CSL proposals and publicize it in the FY2008 announcement of opportunity to encourage larger proposals.

The focus on very large computational experiments supports NCAR's strategic goal to "Provide robust, accessible, and innovative information services and tools" to the university and NCAR communities. In addition, the research performed supports NCAR's strategic goal to "Improve understanding of the atmosphere, the Earth System, and the Sun."

FY2008 accomplishments

Accelerated Scientific Discovery:

In April 2008 CISL announced the 2008 Accelerated Scientific Discovery at NCAR program using the IBM POWER6, bluefire, to be installed in the summer of 2008.
Applications were accepted for research requiring more than 200,000 bluefire processor hours from universities with active NSF awards in the atmospheric or closely related sciences. Applications were also accepted from NCAR for research requiring more than 200,000 processor hours.
The CHAP reviewed and selected three university projects and a back-up project in case one of the university projects did not start on time. The NSF selected two projects. NCAR selected three projects and a back-up project.
Applicants were notified in early June, so selected projects would have sufficient time to set-up their code for the proposed computational experiments.
All selected and back-up projects were required to provide benchmarks showing the efficiency of their production code on bluefire by August 8, 2008 or risk losing 80% of their allocation. Each project was assigned a person from CISL's Consulting Services to assist them in tuning their code so they could make efficient use of bluefire. With this assistance, all projects met the deadline.
All selected projects also met the deadline to begin computations in September 2008.
CISL has contacted each ASD project on their data analysis and visualization needs and has already provided large amounts of disk space for those who need it for data analysis on CISL's dedicated data analysis computers.

CHAP proposals:

The NSF-supported university community in the atmospheric and related sciences was encouraged to submit very large proposals for the October 2008 CHAP meeting by noting the largest proposals that had been approved by the CHAP in April 2008 and noting the large increase in resources available to the university community on bluefire. A reminder was sent this year to university researchers approximately one week before the deadline highlighting this opportunity to propose large computational experiments.
The university community responded in September 2008 with an unexpectedly large number of proposals, more than double the usual number of panel requests. The largest proposals were similar in size to the ASD proposals at 400,000 bluefire processor hours, although they will be accomplished over a longer time period. Most university proposals envisioned completing their large computational experiments over 12 months.

NCAR allocations:

The supercomputing resources available to NCAR researchers has been under pressure for a number of years resulting in too much fragmentation of computing resources. In September 2008 NCAR management expanded the resources for very large computational experiments performed by NCAR researchers by expanding the NCAR Capacity Computing (NCC) pool of resources in proportion to the increase in computing delivered by bluefire.

CSL allocations:

During 2007 the CSL Allocation Panel, an NSF-appointed panel, suggested that some of the CSL proposals were requesting too few processor hours.
In 2008 a new, minimum request size was announced as part of the 2009 CSL Announcement of Opportunity issued by CISL.

FY2009 plans

Accelerated Scientific Discovery:

ASD projects will complete their computations on November 30, 2008. CISL staff will continue to provide a high level of service to these projects throughout their computing campaigns.
CISL will provide extensive assistance to the ASD projects with their data analysis and visualization needs. This effort is expected to continue through May 2009.

CHAP proposals:

The CHAP will meet on October 9, 2008 to review and make allocations to 45 university projects requesting large computing allocations.
Based on CISL's work with the ASD projects, CISL will implement a new code efficiency program for the university projects receiving the largest CHAP allocations. Each project receiving more than 150,000 bluefire processors hours will be asked to provide benchmarks of their production code showing good utilization of bluefire, prior to receiving their full allocation. CISL support staff will be assigned to each of these very large university projects to assist in tuning and optimization of the production code. These projects will be given a medium-size testing allocation to set up their production codes and run benchmarks. When the projects submit a satisfactory benchmark, the full allocation recommended by the CHAP will be provided.

NCAR allocations:

On December 1, 2008, NCAR's NCC program for very large computational experiments will begin. The first two projects have been selected from ASD projects submitted in 2008 by NCAR researchers. NCC projects are run one at a time for several months until completion to accelerate the scientific research process.

CSL allocations:

Thirteen projects were awarded large allocations in 2007, representing 45% of CISL's total supercomputing resources. The CSL projects will receive a 2.4-times increase in their monthly allocation on December 1, 2008 because of the additional resources provided by bluefire.
The CSL Allocation Panel will meet in January 2009 to select the CSL projects for the next 18 months with allocations beginning June 2009. The new minimum award size will ensure that the CSL provides resources for very large computational experiments in support of the U.S. Climate Change Science program.

Sponsorship

The ASD initiative is sponsored by the National Science Foundation through the provision of computational and support resources. The university researchers using these resources are funded by NSF awards from ATM and OCE. NCAR researchers utilizing the ASD computational resources were also sponsored by the NSF.

The Climate Simulation Laboratory (CSL) is a multiagency computing facility established in 1995 and is dedicated to climate modeling in support of the U.S. Climate Change Science Program. The CSL is administered by the National Science Foundation and hosted by NCAR.

Director's Message