2011 Workshop: DBs in CEDAR

The Madrigal and CEDAR databases have been around a long time, but have undergone a number of recent developments. Madrigal was originally developed at MIT Haystack Observatory (http://madrigal.haystack.mit.edu/madrigal/) to allow access to upper atmospheric realtime or archived data, but for the next release Jicamarca Radio Observatory is a co-developer. Madrigal is a local database that automatically shares metadata, allowing each site to act like a virtual observatory. The CEDAR DB (http://cedarweb.hao.ucar.edu) was developed at NCAR for incoherent scatter radar data as well as optical and other data, and stores all data locally, allowing central archiving. Both the CEDAR DB and Madrigal presently share the CEDAR database format based on 16 bit integers. Madrigal was installed at NCAR in 2009 (http://madrigal.hao.ucar.edu), with the capability of automatically downloading and archiving data from all other Madrigal sites, making it unnecessary to submit data separately to Madrigal and the CEDAR DB. Madrigal delivers data as ASCII flat files or in any variant of the CEDAR database format via a web interface or via application program interfaces (API's) in python, Matlab, or IDL. The CEDAR data are delivered via the on-line Virtual Solar Terrestrial Observatory (VSTO) system. The data can be retrieved in ASCII files, in netCDFv3 files, and in binary form using the openDAP protocol.

This workshop will feature discussions of future directions in community database design and database formats. Madrigal is exploring HDFv5 while VSTO is exploring netCDFv4 (.nc) with the goal of ingesting and serving NCAR model output (TIE-GCM, TIME-GCM, WACCM) in their native netCDF format. Both formats should be able to be supported especially since netCDFv4 uses HDFv5 as a storage layer. Our goal is to merge the Madrigal and CEDAR systems while allowing access via Madrigal or VSTO. We also plan to showcase examples of how new instruments' data are put into either system, and accessed via either system, or across systems. Attendees are encouraged to discuss ways in which the utility of these systems can be enhanced.