Motivation for the package

[jwblin]

Let me start out with my thoughts about motivation (in no particular order):

(1) There are many meteorological, climatological, and oceanographic quantities that have standardized definitions. This includes constants (e.g., dry gas constant) as well as derived quantities (e.g., saturation vapor pressure, isentropic potential vorticity, potential temperature). It doesn't make sense that every person/research group writes their own routines to make such calculations as that leads to duplicated effort, routines whose results are not invertible between one another (e.g., routines using different numbers of significant figures for constants), and uneven levels of testing of each group's individual routines.

(2) A number of existing and developing projects (e.g., cdms, PyClimate, Iris) provide well-tested routines to calculate some of these quantities but not many of them. Creating a standardized suite would also mean the still-developing projects would not have to write their own version if they did not want to.

(3) While existing projects provide very powerful OOP structured interfaces and data structures that leverage metadata, there is still a need for a suite of procedures that primarily act on just arrays. For many if not most of the standardized quanities described, metadata does not play a substantial role. Additionally, these algorithms can
be readily adapted to other planetary environments if they are just quantities that are calculated from arrays. Finally, such a suite of array-based procedures would be a halfway-house for people making the transition from Matlab/IDL to OOP. For such users, using an OOP-centric package with many dependencies can be cognitively daunting, when all they want to do is calculate potential temperature for an array.

(4) Many of these routines exist in Fortran form and, while not having been tested formally using software engineering best-practices, nonetheless have been substantially tested via use over several decades (e.g., the AWIPS I routines). Making use of these routines in some way would be an easy first-cut at creating a standardized library.

Thoughts? Responses? What motivations do y'all think should be included? I figure with a list of motivations, this will give us a goals list.

Towards that end, let me make one comment regarding goals/scope: My own view of this package is that it should calculate basically any AOS quantity you would want, and in that way is very broad, but that it should do little more than that, and in that way is very narrow. I don't think it should duplicate what cdms, Iris, pandas, or other more sophisticated data/object structures are capable of doing as someone who wants that capability, it seems to me, would be better served using cdms, Iris, etc. utilities. It also introduces more dependencies to the package. I think it's fine for routines in the package to accept cdms, Iris, pandas, etc. objects to operate on, but the output would still be an array (or maybe a pandas dataframe). This is just my $0.02; am happy to be convinced otherwise :). One problem I see is what about quantities that require manipulation of the time axis? Do we accept only input where time has been adequately processed? Or not calculate those quantities and leave those to cdms, Iris, etc.?

[rhattersley]

There are many meteorological, climatological, and oceanographic quantities that have standardized definitions.

there is still a need for a suite of procedures that primarily act on just arrays. For many if not most of the standardized quanities described, metadata does not play a substantial role

Have you considered including the units of measure? Expressing values using a package such as quantiities would allow some dimensional analysis and could help save time, or even avoid embarrassing mistakes.

But from what I can see, none of the main scientific Python distributions include a units package - that seems a strange omission.

[jwblin]

I think it's a great idea to include units, or, at the very least, make sure all the routines we write strictly adhere to SI or a units system that makes sense for our field (e.g., hPa for atmospheric pressure). The quanities package looks interesting but I'm worried that it's still a little early in its development, as well as concerned about adding another dependency. Is there a way we could code things to make dropping in quantities relatively easy to do later down the line? Or code in such a way that we use quantities but in an optional way? What do others think?

[doutriaux1]

Johnny,

udunits works pretty well for units convertion. For time I would recommend cdtime, I have yet to see something better at handling all various calendar out there (in the modeling world at least)

[jwblin]

Are udunits and cdtime available for separate install? How easy are they to install and are they available as a Debian package? Are they part of EPD and/or Anaconda CE?

[doutriaux1]

cdtime relies on cdms (so we could stick with udunits if absent). udunits can be installed as a stand alone fairly easily (udunits is now part of netcdf4 and is easy to install). Again this could be used only if available.

[david-ian-brown]

The NetCDF library and now NCL contain a standalone C version of cdtime. If we are not going to require cdms, it would be easy to just write some wrappers for this code and include it directlly. It has 1179 lines of code. Here is the comment at the top of the NCL version code. Just last week I wrote some aggregation code that made use of these functions. I agree with Charles that they do the best job for handling time for climate models and any data that uses CF compliant time units.

*********************************************************************/

/*

• The "nctime.c" file was extracted from the NetCDF-4.1.1 source code
• and permission was granted for inclusion in NCL.
  *
• I changed CU_VERBOSE and CU_FATAL to 0 so error messages would get printed, and
• then the program will exit.
  *
• This code was originally extracted with permission from the CDMS
• time conversion and arithmetic routines developed by Bob Drach,
• Program for Climate Model Diagnosis and Intercomparison (PCMDI) at
• Lawrence Livermore National Laboratory as part of the cdtime
• library. Russ Rew of the UCAR Unidata Program made changes and
• additions to support the "-t" option of the netCDF ncdump utility,
• including a 366-day climate calendar.
  *
• For the complete time conversion and climate calendar facilities of
• the CDMS library, get the original sources from LLNL.
  */

On Feb 21, 2013, at 10:31 AM, Charles wrote:

cdtime relies on cdms (so we could stick with udunits if absent). udunits can be installed as a stand alone fairly easily (udunits is now part of netcdf4 and is easy to install). Again this could be used only if available.

—
Reply to this email directly or view it on GitHub.

[jwblin]

Including a wrapper for cdtime sounds good to me as long as code we write that uses it first checks to see that a separate install of cdtime doesn't already exist and that we have unit tests that test both to see if we get the same results whether we use our version of cdtime or the stand-alone version (e.g., using virtualenv for an installation with and without cdtime). I get nervous about hidden dependencies we build ourselves that branches from a package separately developed and distributed.

[doutriaux1]

The wrappers do exist already, it's the cdtime module in UV-CDAT