Last modified 2 years ago Last modified on 2017-10-04 10:43:34

List of specific dos and don’ts to help maintain consistency

Specific pieces of advice, rules, and digital legends we should follow in out netCDF files.

* File names should not have spaces in them.

"Test file.dat" is bad; "TestFile.dat" or "Test_file.dat" are good.
  • All units should comply with UDUNITS, a listing of which is here http://www.unidata.ucar.edu/software/udunits/udunits.txt (CRS 8/4/09) now also here (internal access only)https://wh-int.er.usgs.gov/forum/?page_id=337 (MM 10/19/09)
  • How we follow EPIC conventions
    • If it is an EPIC variable, don't mess with it. (CRS 8/4/09)
    • To the extent that some of our variables are not totally in synch with the master EPIC database, I would assign fixing that as a low priority until we know the side-effects of messing with EPIC. (CRS 8/4/09)
    • When making new EPIC variables, make sure we have described the variables we have correctly and consistently (CRS 8/4/09)
    • The complete EPIC key (variable name) listing can be found ftp://ftp.epic.noaa.gov/epic/epic.key here.
    • more info is available about EPIC compliance is also available on a separate page.
  • Our dimension variables (time, Lat, Lon and depth) should have a datum
    • time datum is “Time (UTC) in USGS convention: 2440000 = 0000 h on May 23, 1968″
    • depth datum should be referenced to MLLW etc or based on NAVD88 (ETM 8/6/09)or use a depth_source attribute set to something like 'shipboard echosounder', 'measuring staff'
    • lat and lon datum should be NAD83,(ETM 8/6/09). unless the GPS used says something different.
  • dimension variables are not allowed to have a _FillValue
  • dimension variables should not have minimun and maximum attributes (particularly if they're lat or lon (singlteons)).
  • bindist should contain an array of distances to the center of each bin from the transducer head
  • notes can be long and contain spaces, so don’t need to pepper_with_underscores
  • don’t abbreviate: use Transducer not xducer or ‘ducer
  • always add what was done to the file to the history attribute
  • make sure there are no NaN in data or attributes
  • a variable's "maximum" attribute should not be 1e35 if that is also the _FillValue.
  • if a variable has a "valid_range" attribute is should have 2 reasonable values [0 360]. It should not be " ".
  • some depth variable attributes also need to be GLOBAL:
    • blanking_distance,
    • center_first_bin,
    • bin_size
  • numeric format having way more digits than is appropriate in some attributes
    • ADCP toolbox, add_minmaxvalues is being cast to ncfloat, but min and max still have extra digits
  • NOTES (our convention is that NOTES are Caps)
    • use descriptive NOTES attributes to clarify content
    • in NOTES, ensure that references to variable names actually include the complete name, capitalized as it is in the file.
  • Only one WATER_DEPTH is needed, and it should be the CAPS one because it’s an EPIC element. (ETM 8/17/09)
  • Each file’s global attributes should include the following USGS and EPIC information, more explanation is available here at the USGS time series database page and in this wiki
    • USGS specific attributes
      • platform_type
      • Deployment_date
      • Recovery_date
      • history
      • latitude
      • latitude_units = “degree_north”
      • longitude
      • longitude_units = “degree_east”
      • serial_number='43B7' where the sensor SN is char
      • modification_date
      • initial_instrument_height (in meters)
      • initial_instrument_height_note = “height in meters above bottom: accurate for tripod mounted instruments”
      • nominal_sensor_depth = 26.419815
      • nominal_sensor_depth_note = “inst_depth = (water_depth – inst_height); nominal depth below surface, meters”
      • magnetic_variation_at_site = -15.63
      • magnetic_variation_applied = -15.63
      • start_time = “19-May-2004 02:42:30″
      • stop_time = “24-May-2004 07:42:30″
      • IF RTK survey was done
        • NAVD88_elevation_ref = 4.3 (some number from the survey)
        • sensor_offset_ref = 0.26 (the measured offset from the RTK'd point to the sensor (m))
    • EPIC attributes (see http://www.epic.noaa.gov/epic/document/conv_time.htm
      • CREATION_DATE
      • MOORING
      • INST_TYPE
      • WATER_DEPTH
      • WATER_DEPTH_source (in meters) % optional, but desirable
      • WATER_DEPTH_datum % optional, but desirable
      • DATA_TYPE
      • DATA_SUB_TYPE
      • COORD_SYSTEM
      • WATER_MASS
      • POS_CONST
      • DEPTH_CONST
      • DRIFTER
      • VAR_FILL = 1.0E35
      • DATA_ORIGIN = 'Woods Hole Coastal and Marine Science Center'
      • EXPERIMENT = 'Massachusetts Bay long-term Observations
      • PROJECT = 'USGS Coastal Marine Geology Program'
      • DESCRIPT = B buoy
      • DATA_CMNT = 'Anything pertinent to the data'
      • FILL_FLAG
      • COMPOSITE
      • DELTA_T (integer seconds)
  • Each variable should have the following (in old netcdf syntax- the idea is the same):
    • For example % 1294:P:ADCP Pressure:pres:deca-pascals:F15.2:ADCP Pressure at Transducer Head
    • G{‘P_1294′} = ncfloat(‘time’, ‘lat’, ‘lon’); % *Not* a function of depth since this ADCP’s pressure does not change with depth, i.e. is not a profile
    • ncobj = G{‘P_1294′}; % where an EPIC variable name = name_code
    • ncobj.name = ncchar(‘P’);
    • ncobj.long_name = ncchar(‘ADCP Pressure at Transducer Head’);
    • ncobj.generic_name = ncchar(‘pres’);
    • ncobj.units = ncchar(‘deca-pascals’);
    • ncobj.epic_code = nclong(1294);
    • ncobj.sensor_type = B.INST_TYPE(:);
    • ncobj.sensor_depth = ncfloat(wdepth – xducer_off); % use different computation if RTK'd
    • ncobj.initial_sensor_height = ncfloat(B{‘D’}.xducer_offset_from_bottom (1));
      • Surface moorings may have initial_sensor_depth
    • ncobj.height_depth_units = ncchar(‘m’);
    • ncobj.serial_number = nclong(serial);
    • ncobj.minimum = ncfloat(0); %cannot be NaN
    • ncobj.maximum = ncfloat(0); %cannot be NaN, and shouldn't be 1e35, if that's the theFillValue
    • ncobj.valid_range = B{‘Pressure’}.valid_range(:); % optional, but needs to be set correctly for the varia ble.
    • ncobj._FillValue = theFillValue; % theFillValue must match the type(single double, short) of the variable. In this case, be sure it is cast to 'single' before inserting!
  • “initial_sensor_height”, “serial_number”, height_depth_units” and “sensor_type” may be omitted for the variables if the instrument only samples at one elevation, and all are present in the global attributes. An ECO PAR or YSI EXO sensor would not need these attributes, but ADVs and DCAL data would, since the “primary” measurement (velocity) comes from one height and the transmissometer data comes from a different sensor at another height.
  • “comment” or “note” attributes are encouraged to document any data issues, like the conductivity cell fouling too much to provide good data after a certain date.
  • Note, in the conversion to CF, we change in the minimum and maximum attributes to include “actual_”, allowing them to follow the better practice in the more up-to date version.