Last modified 4 years ago Last modified on 2016-08-25 11:33:35

General Notes: netCDF usage, EPIC conventions, and other topics

Time series data collected the the CMG is stored and served in EPIC compliant netCDF files at http://stellwagen.er.usgs.gov. In some cases our usage stretches the definition a bit.

Our group's usage of the global attribute WATER_DEPTH is used to indicate how deep the water is. This value may be a nominal value or quite precise. In some cases, a pressure based WATER_DEPTH has been inserted, but this is unusual. In the most general terms, it will reflect the correct "z" location of the platform, but is NOT time varying. The depth variable is "measurement depth", but again is NOT time varying, so doesn't account for any scouring in that may have occurred. More detail is provided here.

In addition, it is important not to have NaN, _FillValue or inf in attributes: NetCDF, OPeNDAP (Hyrax) and THREDDS treat NaN and Inf in the variable attributes differently. NetCDF files can have NaN in the variable minimum and maximum attributes, and it will be displayed as a NaN if viewed with OPeNDAP at http://stellwagen.er.usgs.gov/opendap/MBAY_LT/8051adc-a.nc.html.

BUT if THREDDS is brokering the OPeNDAP access, and there’s a NAN, it generates an error like this:

At:http://coast-enviro.er.usgs.gov/thredds/dodsC/DATAFILES/MBAY_LT/8051adc-a.nc.html

Error {
   code = 500;
   message = "Illegal Numeric Value for Attribute Value for P_4.P_4_dods_errors.minimum";
};

I found in this case all, the P_4 data was 1e35, and the min and max attributes were then set to NaN (possibly by me in the post-editing). Nothing seemed like a problem until I tried to access the OPeNDAP data with THREDDS. Serving the same file directly through THREDDS ends up with a 0 in the attribute field, which is wrong, but doesn’t cause the service to crash.

I used NCO/ncatted to replace the NaN’s in the attributes, and now THREDDS/OPeNDAP can access the file correctly. If a file correctly contains all fill values to indicate that we tried to collect data, but the sensor didn't work, the min and max attributes should both be set to the same value, usually 0. An additional comment or note attribute should be added to indicate why there is no data for that variable.

In normal files, we use a fill value to indicate missing data or data that has been removed. In the variable, there must be a "_FillValue" attribute. It cannot be "fillvalue" or "FillValue_" or any other variant! Then clients that know how to deal with netCDF replace any data points matching _FillValue with NaN (or their equivalent missing value term). NOTE: In Matlab, ncread replaces data with _FillValue (1e35 mostly) with NaN during the read, so, even if there are really 1e35 in the .nc file, they won't be in the variable data read. Pretty slick, but can be confusing. To verify, ncdump may be used to output data from a variable to a .txt file. If ncdump finds a point that contains _FillValue, it replaces it with '_'. Here's an example command: C:\Users\emontgomery\progs\netcdf\ncdump -v Turb 9522Aecn-a.nc > test_952turb.txt

Things that would be nice to have in all kinds of time series data.

This list of Do's and Don'ts will get you started on the uniformity issues.

  • include a time.datum, Time (UTC) in USGS convention: 2440000 = 0000 h on May 23, 1968
  • Change long_name text and capitalization where doesn’t interfere with EPIC
  • use UDUNITS for units attribute where doesn’t interfere with EPIC
  • update or remove FORTRAN_format from variables
  • use a consistent source of variables and attributes instead of defining them with the processing program. ep_standard.nc is made by mk_epstandard.m, and could be the single place where variables are instantiated and modified. Variable names for each instrument would then be compared and plucked from ep_standard.m

*If declination is applied or removed, make sure it is done to everything in a file that contains direction. If one rotates directional data such as U, V, wave direction, and heading is included in the file, then heading should also be changed.

I believe some but not all of these have been implemented.

Be very careful of where on your MATLABPATH the functions come from!! There are many versions of functions like add_FillValue(), and they're not all the same, and they're not all correct. Use "which function_name" to see the path to function_name currently being used.