Mass Storage System (MSS) improvement: AMSTAR
This chart of the past 12 years of NCAR
MSS growth shows unique and total (includes duplicate copies) bytes.
This exponential growth of the NCAR MSS is expected to continue with
the annual growth rate exceeding 5 petabytes by 2012.
The NCAR Mass Storage System (MSS) has been a high performance, reliable,
and scalable system over the past 20 years. It has demonstrated high
availability and accessibility and proven its cost effectiveness.
The MSS in many ways has been what has distinguished computing at NCAR from computing elsewhere, and the system continues to be highly regarded by users and by peers at other centers. MSS maintenance and development supports the NCAR strategic priority of "Enhancing capability and capacity of NCAR supercomputing."
Today, the NCAR MSS remains one of the most capacious archives. At the end of FY2008, it surpassed 6 petabytes (PB) of total data (4 PB of unique data), transfers more than 10 terabytes (TB) of data per day in response to user requests, and transfers another 10 TB of data per day for internal data migration and data movement to new media. The MSS is growing at a net rate of more than 200 TB per month.
The future development and deployment of the NCAR Mass Storage System is constrained by both the need to continue providing traditional file storage and access to Mass Storage services in the context of a full 7 by 24 production environment, as well as adopting new roles as an integrated component of larger UCAR-wide data management efforts. The MSS's traditional role will continue: to reliably preserve UCAR, NCAR, and university community data while serving a wide client base on multiple levels.
This chart projects the unprecedented
MSS growth estimated over the next four fiscal years FY2009-FY2012. These
estimates were used as the capacity increase requirement for AMSTAR.
The MSS must scale up to meet an ever-growing demand for secure, reliable
data storage and high-performance access. This requires the constant
evaluation and periodic deployment of the latest, highest-performance, and
most cost-effective hardware and software technologies available.
The completion of the CISL Integrated Computing Environment for Scientific Simulation (ICESS) Phase 2 deployment more than quadrupled NCAR's peak computing capacity. With the ICESS supercomputing upgrade combined with the planned growth of CISL's Data Support Section's scientific data holdings and the upcoming Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5), CISL is expecting an unprecedented increase in the capacity of the MSS over the next four years. Additionally, the tape library robotics currently in production are nearing end-of-life. The vendor of those tape libraries has set an end-of-service date of December 2010. This allows only two years to deploy replacement tape technology and migrate more than six petabytes of data to the new technology.
To accomplish this upgrade, CISL initiated a Request For Proposal (RFP) at the beginning of FY2008 named "Augmentation of the Mass Storage Tape Archive Resources" (AMSTAR). Released in late November 2008, AMSTAR is a four-year subcontract awarded to replace the MSS StorageTek Powderhorn tape libraries (silos) and 9940x tape drives and media with new robotic tape libraries, tape drives, and media that will increase MSS data storage capacity to beyond 30 petabytes. The AMSTAR offeror proposals were evaluated in mid-FY2008, and a competitive range was determined. The offerors in the competitive range provided equipment for an on-site Live-Test-Demonstration (LTD) where CISL staff evaluated the equipment. A final recommendation was made to the CISL Directorate based on the proposals and LTD results, approval to negotiate with the offeror was obtained, and contract negotiations were completed at the end of August 2008.
Production deployment of the new technology is expected in early FY2009. To complement the new AMSTAR tape technology and support the increased ICESS computing capacity, the following MSS enhancements occurred in FY2008:
- The MSS disk cache capacity was doubled from 50 terabytes to 100 terabytes.
- The MSS data connection to the new ICESS supercomputer, bluefire, was upgraded from 1 Gigabit Ethernet to 10 Gbps.
- Enterprise-class tape technologies were evaluated in preparation for AMSTAR.
FY2009 MSS plans include the production deployment of AMSTAR and the start of the two-year data migration to the new tape technology. Additional MSS server capacity upgrades will be deployed to support the increased data transfer and metadata workload required by bluefire.
The NCAR MSS is managed by CISL under the UCAR/NSF Cooperative Agreement and is supported by NSF Core funds including CSL funding.