/pub: Operational Data
Above is a clickable image map of the /pub directory structure. This is mounted
on the RAID and accessible to all machines in the CDAAC cluster. The leaves of this tree point to
definitions of all CDAAC data formats and file types.
File Naming Conventions
These rules are generalizations of the naming
conventions that were used for GPS/MET which worked well for us:
- File names are composed of 3 or 4 sections separated by underscores.
- A six character id which uniquely identifies the file type.
- A file identifier which tells which instance of this file type.
- The subtype and version (1b and above)
- The format type: pl for perl format, nc for netCDF files, bnx for
BINEX files, etc.
- Files must be identified by year and day of year YYYY.DDD in the name.
This is necessary to ensure correct daily archival of files.
- Files that are associated with downloads from LEO satellites are
further identified by 'dump id': YYYY.DDD.SSS.NN where SSS is the LEO constellation id (always
'001' if there is only one satellite in the constellation) and NN is the dump number that day.
If there is a data downlink (dump) every orbit (say 100 minutes) then there will be around 14 dumps
per day. Dump ids are assigned based on the day they ended for midnight crossing dumps.
- All times are in GPS time.
- Files that have time intervals not associated with LEO data downloads are idendified as follows:
YYYY.DDD.HH.MM.DDDD where HH.MM.DDDD are the hour of day, the minute of the hour and the
duration of the file in minutes.
- All files of level 1b and above have a subtype and version (SSSS.VVVV) attached to them.
This is a version number which allows tracking of which program options and which code version are
used to generate any given file. The subtype gives information on program options, and the version
gives information on code versions. This information applies not only to the most recent data
processing step (ie, the program that turned an atmPhs file into an atmPrf file) but to all
previous steps as well. So, given the subtype and version of an atmPrf file, one can (with the
help of the code version database and CVS) determine the exact code version and options used for
all previous processing steps.
How to update the fileFormats area
Information on how to update this area can be found in the fileFormats howto guide.
Other file conventions
Files for CDAAC exist in many formats including many ASCII Bernese formats,
GPS data and netCDF for
higher level data products.
In order to work with the CDAAC database interface
netCDF files should follow these conventions:
All table fields containing latitudes must have the string 'lat' in
the field name. The latitude must be in degrees from -90 (South) to 90 (North).
- All table fields containing longitudes must have the string 'lon' in
the field name. The longitude must be in degrees from -180 (East) to 180 (West).
- All table fields containing dates must be in GPS seconds. The string 'time'
must appear in the field name.
- Any database fields containing the string 'filename' should contain the name
of a generated file available via /pub. These files should if possible be
netCDF and abide by certain basic conventions, including:
- One dimensional variables should have a dimension name and a matching
dimension variable. These dimension variables (ie MSL_alt) should be
consistent across files to allow statistical comparisons between files.
- Each variable should have 'long_name' and 'units' attributes to allow
display of help text and units.