- Director's Message
- Table of Contents
- Strategic Goals: Science, Facilities & Technology
- 1. Exploring atmospheric, Earth system, and solar processes, variability, and change
- 2. Investigating the interaction of the atmosphere, the broader Earth system, and human society
- 3. Improving prediction of weather, climate, and other atmospheric phenomena
- 4. Developing community models
- 5. Enabling innovative field experiments and measurement campaigns
- 6. Developing new instrumentation
- Research Catalog
CGD 2008 Profiles in Science: CCSM Software Engineering
CCSM software infrastructure
CSEG infrastructure improvement efforts have continued to focus on producing a robust, scalable, efficient and easily extensible and portable model system. Efforts to improve CCSM robustness were concentrated in the areas of continued expansion of the CCSM scripts and testing framework.
CSEG has continued to prioritize the porting of both the release and development versions of CCSM to new platforms. Both the release and development CCSM code base were successfully ported to the upgraded Cray XT4 at ORNL and NERSC. We have also ported the development version of CCSM to the new Cray XT4 at the University of Tennessee. The CCSM Software Engineering Group has worked closely with our partners at the DOE Oak Ridge National Lab and Cray to carry out these efforts. CCSM portability efforts have been greatly streamlined by the creation of a single executable version of the development code base. Substantial work was done to optimize performance on the ORNL XT4, leading to throughput rates of over 7 model-years/day for running a .5° atmosphere resolution with a 1° ocean resolution. CSEG also helped carry out numerous multi-century simulations, both at ORNL, NERSC and at NCAR, as part of CCSM4 development efforts.
CSEG has continued to support the development of an experimental run database that contains a web interface for both the input and query of model information, in order to provide both science and software developers the ability to easily duplicate and archive details of CCSM experiments. In the past, meta-data detailing various model runs were stored in multiple places, thereby making it difficult to easily determine details of past runs. A central, searchable database now provides the necessary information to easily reproduce a particular run and serve as a single placeholder for pointers to diagnostics and other model output. This database has provided CCSM developers and scientists with the capability to easily track experiments done both at NCAR and at remote sites. It has proven valuable in tracking model changes that are occurring as part of the creation of CCSM4.
CAM Development
CAM development efforts have focused on incorporating the new physical parameterizations that will become part of CAM4. These include schemes for cloud microphysics, boundary layer, shallow convection, radiation, cloud and aerosol optics. A significant refactoring of the code that determines which species are radiatively active and what their optical properties are has added tremendous flexibility which facilitates exploring questions related to radiative forcings.
Improvements to CAM's software infrastructure now allow better scaling to large numbers of processors as well as better memory scaling. By removing constraints that required both the CAM dynamics and physics to run on the same set of processes, the code is now capable of scaling far past the limits that previously were determined by the CAM dynamical core. In addition, scaling bottlenecks imposed by the reproducibility requirements on global sums have been overcome with new algorithms for computing global sums in a distributed, but reproducible way. Additional parallelism has also being exploited in the dynamical core in order to improve the performance when large numbers of tracer fields are utilized. This will be particularly beneficial for incorporating new chemistry parameterizations.
Finally, the longstanding goal of having the standalone version of CAM use the same surface components and the same driver code as the "standalone CAM" configuration of the full CCSM has finally been achieved.
CICE Development
CICE software engineering efforts have continued to focus on the incorporation of the Los Alamos National Laboratory sea ice model (CICE) as the standard CCSM sea ice component. CICE4, which has been released as a standalone model, is now the official CCSM sea ice component. This code base adds new improved physics options and numerical and computational enhancements. A new improved treatment of snow and ice has also been incorporated such that snow and ice albedos are more representative of limited observations. Development is currently underway to accommodate aerosols, such as black carbon and dust, as tracers in the sea ice and snow. This effort will also be coordinated with the Biogeochemistry working group. Other software engineering efforts had lead to the successful incorporation of CICE into the new CCSM coupling architecture (CPL7) and the stand-alone CAM systems, and the investigation of running CICE at 0.1-degree globally on peta-scale architectures.
CLM Development
Numerous improvements were made to the CLM model over the last year. The old CLM offline mode was removed in favor of using the new CCSM coupling architecture, CPL7 (see below). This will facilitate scientific consistency and code reuse in the CCSM system. As part of this process archiving of files is now performed by a separate offline archiving scripts rather than during model runtime. The point version of the CLM urban model was given global capability and brought into the CLM development trunk. Numerous improvements were also made to the CLM offline support scripts. The tool to interpolate initial conditions was upgraded so that it would work with high resolution CLM initial datasets containing CN biogeochemistry data. The CLM configuration scripts were significantly improved in order to allow error checking and automatically document of CLM configurations and input namelists. In addition there were upgrades to the CCSM data atmosphere model that allowed CLM to have a better representation of the diurnal cycle of solar input. The changes also permit different input fields to be interpolated in time independently and as scientists see fit rather than requiring a linear interpolation in time for all data. These improvements will help ensure the extensibility of the CLM code base as major new modifications are added as part of the creation of CLM4.
POP Development
POP development efforts focused on the addition of new scientific parameterizations, the addition of high-resolution support, and the development of new tools to support model-correctness testing. Scientific improvements included the zenith-angle-dependent diurnal cycle of solar forcing, sub-mesoscale parameterization, horizontally varying background vertical diffusivity, overflow parameterization, and the addition of a run-time option for generating topography from bathymetry data. Software engineering improvements to the model included the integration of the new LANL pop2 infrastructure code and the addition of code which facilitate porting to low-memory and reduced-instruction-set machines. New tools were also created for supporting correctness testing, including new support for testing various model decompositions. Finally, the CCSM POP2 scripts were enhanced in order to support running POP2 at ultra-high eddy resolving resolutions.
Nested Regional Modeling
Significant progress has been made over the last year in the Nested Regional Climate Modeling effort involving nesting the ROMS model inside POP. We have succeeded in creating a composite ocean model that is capable of two-way nesting. In this model, atmospheric surface states and lateral 3-D ocean boundary data are now passed from CCSM to ROMS to drive the regional model. After each coupling interval, the sea surface temperatures from ROMS are merged with those from POP and passed back to the CCSM coupler for use in calculating atmosphere-ocean fluxes for use in the system. Three different algorithms were implemented in order to blend the ROMS and POP SST's, using regridding weights generated by a modified version of SCRIP. ROMS output and restart files are now driven by this composite model as well. We are currently testing the model with the first set of longer runs, both in an active ocean only mode as well as inside the full CCSM system with all active models, in order to study the effect of the increased resolution.
Creation of New CCSM Coupling Architecture (CPL7)
Projected peta-scale architectures will require the ability for each CCSM component to run on much higher resolution configurations utilizing processor counts that are one to two orders of magnitude greater than present settings. CSEG has created a new coupling architecture, CPL7, that will provide the required flexibility to achieve this goal.
CPL7 contains backwards compatibility with the concurrent CCSM3 system (where each component runs on a unique set or processors, not utilized by any other component), provides "plug and play" capability of data and active components, and has been scientifically validated in long model simulations. This new coupling architecture now permits extensive code reuse and removes the existence of stand-alone component configurations that have a different code base than those used in coupled simulations. This has resulted in decreased code maintenance, consistent science across components and increased system robustness. In particular, the stand-alone CAM and CLM configurations (where some of the components only provide data forcing) use the same code base and coupling architecture as is used in simulations where all the components are fully active. All current CCSM4 development efforts are utilizing the CPL7 architecture.
The DOE Grand-Challenge simulation is utilizing the CLP7 coupling architecture. This simulation will be the first ever U.S. multi-decadal global climate simulation with eddy resolving ocean and high-resolution atmosphere (.25° atmosphere/land and .1° ocean/ice).
Although the current CPL7 system currently utilizes MCT (the CCSM3 coupling framework) as a coupling framework, ESMF is also being examined as an alternative framework. The ESMF Stage 1 Evaluation Plan was completed successfully in June 2007 and CSEG is currently involved in a joint effort with ESMF to create and evaluate an alterative ESMF-based CPL7 system via the ESMF Stage 2 Evaluation project.
Finally, the incorporation of parallel I/O throughout the entire CCSM model system has become a very high priority for the CSEG group. Parallel I/O has already been incorporated to some degree in CAM, CLM and POP. We hope to have it fully incorporated in each CCSM component within the next year.