Last modified 13 months ago Last modified on 2017-09-28 09:43:08

Aggregating data


More and more data we collect is big, and may be saved in sequential chunks by the instrument, or may take the form of A,B,C, sections where instruments are refurbished and turned around between the parts.

There are a couple of ways to combine the files. NCO's ncrcat can be used, syntax combining 3 ysi files to an _all is something like this:

\nco\ncrcat 9531ysiA_trm.cdf 9531ysiB_trm.cdf 9531ysiC_trm.cdf 9531ysi_all.cdf

There are more complicated options for variable selection, if all variables are not in all files.

Another option is combining using THREDDS. An example is here.

And you can also do it with ncml. I did a fairly complicated project to aggregate salinity and several other key parameters from all 30 years at Boston that involved xls files, and doing creative stuff in Matlab that does the heavy lifting to implement the above example on multiple selected files. The code is in trunk/whfc2ge- start with do_aggtsCF_native.m and go from there. Some documentation of the larger scope is here.