NetCDF4 #181

ssolson · 2022-07-25T15:36:56Z

This PR enables MHKiT to work with the latest version of NetCDF4 (#177).

To fully pass the test on python 3.7, 3.8, 3.9 we use a conda environment (windows and py 3.7 were the only version combinations that could not find the hdf5 path). I have added separate test for pip versions 3.8, 3.9. This expands our tests to check for behavior when users use both pip and conda. This addition is in line with the testing consultation on MHKiT summarized in #183.

Separately on netcdf 1.6.0 there is a new compression kwarg. For Dolfyn I have removed the compression key on the io test. This test will be fixed in #186.

into Develop

ssolson · 2022-07-26T14:14:55Z

Hey James I have been looking into this more and it appears that it is actually the Dolfyn xarray .to_netcdf method which is breaking the UNIX tests for MHKiT which you can checkout on #181.
Based on our last conversation/ review of Dolfyn is this PR ready to go?

@ssolson Apparently recreating the test file fixes that issue #181

Originally posted by @jmcvey3 in #169 (comment)

So @jmcvey3 here you are saying we need to recreate the netcdf files which are being read in?
The UNIX tests here are failing on a `to_netcdf`` call that dolfyn is making using xarray.

see here: https://github.com/MHKiT-Software/MHKiT-Python/runs/7504333747?check_suite_focus=true

======================================================================
ERROR: test_save (mhkit.tests.dolfyn.test_read_io.io_testcase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/MHKiT-Python/MHKiT-Python/mhkit/tests/dolfyn/test_read_io.py", line 19, in test_save
    save_netcdf(ds, 'test_save', compression=True)
  File "/home/runner/work/MHKiT-Python/MHKiT-Python/mhkit/tests/dolfyn/base.py", line 54, in save_netcdf
    io.save(data, rfnm(name), *args, **kwargs)
  File "/home/runner/work/MHKiT-Python/MHKiT-Python/mhkit/dolfyn/io/api.py", line 145, in save
    ds.to_netcdf(filename, format=format, engine=engine, **kwargs)
  File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/xarray/core/dataset.py", line 1912, in to_netcdf
    invalid_netcdf=invalid_netcdf,
  File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/xarray/backends/api.py", line 1073, in to_netcdf
    dataset, store, writer, encoding=encoding, unlimited_dims=unlimited_dims
  File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/xarray/backends/api.py", line 1119, in dump_to_store
    store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
  File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/xarray/backends/common.py", line [266](https://github.com/MHKiT-Software/MHKiT-Python/runs/7504333747?check_suite_focus=true#step:6:267), in store
    variables, check_encoding_set, writer, unlimited_dims=unlimited_dims
  File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/xarray/backends/common.py", line 304, in set_variables
    name, v, check, unlimited_dims=unlimited_dims
  File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/xarray/backends/netCDF4_.py", line 497, in prepare_variable
    fill_value=fill_value,
  File "src/netCDF4/_netCDF4.pyx", line 2838, in netCDF4._netCDF4.Dataset.createVariable
  File "src/netCDF4/_netCDF4.pyx", line 4003, in netCDF4._netCDF4.Variable.__init__
  File "src/netCDF4/_netCDF4.pyx", line 1965, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Filter error: bad id or parameters or duplicate filter

jmcvey3 · 2022-07-26T16:07:58Z

Hey James I have been looking into this more and it appears that it is actually the Dolfyn xarray .to_netcdf method which is breaking the UNIX tests for MHKiT which you can checkout on #181.
Based on our last conversation/ review of Dolfyn is this PR ready to go?

@ssolson Apparently recreating the test file fixes that issue #181

Originally posted by @jmcvey3 in #169 (comment)

So @jmcvey3 here you are saying we need to recreate the netcdf files which are being read in? The UNIX tests here are failing on a `to_netcdf`` call that dolfyn is making using xarray.

see here: https://github.com/MHKiT-Software/MHKiT-Python/runs/7504333747?check_suite_focus=true

======================================================================
ERROR: test_save (mhkit.tests.dolfyn.test_read_io.io_testcase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/MHKiT-Python/MHKiT-Python/mhkit/tests/dolfyn/test_read_io.py", line 19, in test_save
    save_netcdf(ds, 'test_save', compression=True)
  File "/home/runner/work/MHKiT-Python/MHKiT-Python/mhkit/tests/dolfyn/base.py", line 54, in save_netcdf
    io.save(data, rfnm(name), *args, **kwargs)
  File "/home/runner/work/MHKiT-Python/MHKiT-Python/mhkit/dolfyn/io/api.py", line 145, in save
    ds.to_netcdf(filename, format=format, engine=engine, **kwargs)
  File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/xarray/core/dataset.py", line 1912, in to_netcdf
    invalid_netcdf=invalid_netcdf,
  File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/xarray/backends/api.py", line 1073, in to_netcdf
    dataset, store, writer, encoding=encoding, unlimited_dims=unlimited_dims
  File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/xarray/backends/api.py", line 1119, in dump_to_store
    store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
  File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/xarray/backends/common.py", line [266](https://github.com/MHKiT-Software/MHKiT-Python/runs/7504333747?check_suite_focus=true#step:6:267), in store
    variables, check_encoding_set, writer, unlimited_dims=unlimited_dims
  File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/xarray/backends/common.py", line 304, in set_variables
    name, v, check, unlimited_dims=unlimited_dims
  File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/xarray/backends/netCDF4_.py", line 497, in prepare_variable
    fill_value=fill_value,
  File "src/netCDF4/_netCDF4.pyx", line 2838, in netCDF4._netCDF4.Dataset.createVariable
  File "src/netCDF4/_netCDF4.pyx", line 4003, in netCDF4._netCDF4.Variable.__init__
  File "src/netCDF4/_netCDF4.pyx", line 1965, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Filter error: bad id or parameters or duplicate filter

@ssolson Yes, rebuilding and reuploading that test file fixed it on my branch. If it's still an issue let me know.

ssolson · 2022-07-26T17:48:25Z

@jmcvey3 I am almost certain what fixed it is you merged my changes to the master branch which required the previous version of netCDF. E.g. if your requirements file says netCDF4<=1.5.8 then that is what is going on.

I discussed this short-term fix in Issue #177 which this PR is attempting to close.

Let me know if this is not the case.

jmcvey3 · 2022-07-26T18:05:59Z

@jmcvey3 I am almost certain what fixed it is you merged my changes to the master branch which required the previous version of netCDF. E.g. if your requirements file says netCDF4<=1.5.8 then that is what is going on.

I discussed this short-term fix in Issue #177 which this PR is attempting to close.

Let me know if this is not the case.

Oh I see. I thought I'd already updated my branch, but apparently not.

ssolson · 2022-07-26T18:11:19Z

Okay cool. Yeah lmk if you have any ideas about the dolfyn to_netcdf issue above and I can take a stab at it.

Right now I am focused on the Windows issue with Py 3.7 (Windows 3.8, 3.9 both work currently)... getting closer to just dropping 3.6 support here 😄 and moving to 3.10.

ssolson · 2022-08-11T15:32:08Z

Removing compression from netcdf save in dolfyn fixed the Linux builds see commit 7f05e9f.

@jmcvey3 do you think you could take a look at what is going on with the new version of netcdf and compression?

ssolson · 2022-08-16T17:36:54Z

Using a conda environment we can get them all to pass. Testing this way would require us to change our guidance around how we ask users to install mhkit. E.g. we want to test the same way we recommend. Currently we recommend pip for everything. This would be moving towards a conda centric approach... part of our testing suggestions is test both pip and conda... and get mhkit on conda install.

jmcvey3 · 2022-08-16T19:23:10Z

Ah. It's in the 1.6.0 changelog. dolfyn uses "zlib=True" if compression is set to true, so that needs to be updated

 * add 'compression' kwarg to createVariable to enable new compression
   functionality in netcdf-c 4.9.0.  'None','zlib','szip','zstd','bzip2'
   'blosc_lz','blosc_lz4','blosc_lz4hc','blosc_zlib' and 'blosc_zstd'
   are currently supported. 'blosc_shuffle',
   'szip_mask' and 'szip_pixels_per_block' kwargs also added.
   compression='zlib' is equivalent to (the now deprecated) zlib=True.
   If the environment variable NETCDF_PLUGIN_DIR is set to point to the
   directory with the compression plugin lib__nc* files, then the compression plugins will
   be installed within the package and be automatically available (the binary
   wheels have this).  Otherwise, the environment variable HDF5_PLUGIN_PATH
   needs to be set at runtime  to point to plugins in order to use the new compression
   options.

ssolson · 2022-08-16T19:24:58Z

Thanks, James my question to you was specific to the compression. I was working and have fixed (albeit with the discussion I posted above) the mac 2.7 and windows py3.7 issues by using a conda environment. Going to sleep on what exactly I want to do and maybe try a couple more things before making a decision.

I was going to try adding compression back in tomorrow. I will follow up then.

jmcvey3 · 2022-08-16T19:26:03Z

Using a conda environment we can get them all to pass. Testing this way would require us to change our guidance around how we ask users to install mhkit. E.g. we want to test the same way we recommend. Currently we recommend pip for everything. This would be moving towards a conda centric approach... part of our testing suggestions is test both pip and conda... and get mhkit on conda install.

Possibly? I've found it's just much easier to use conda in github actions than venv.

ssolson · 2022-08-16T19:28:10Z

Ah. It's in the 1.6.0 changelog. dolfyn uses "zlib=True" if compression is set to true, so that needs to be updated

 * add 'compression' kwarg to createVariable to enable new compression
   functionality in netcdf-c 4.9.0.  'None','zlib','szip','zstd','bzip2'
   'blosc_lz','blosc_lz4','blosc_lz4hc','blosc_zlib' and 'blosc_zstd'
   are currently supported. 'blosc_shuffle',
   'szip_mask' and 'szip_pixels_per_block' kwargs also added.
   compression='zlib' is equivalent to (the now deprecated) zlib=True.
   If the environment variable NETCDF_PLUGIN_DIR is set to point to the
   directory with the compression plugin lib__nc* files, then the compression plugins will
   be installed within the package and be automatically available (the binary
   wheels have this).  Otherwise, the environment variable HDF5_PLUGIN_PATH
   needs to be set at runtime  to point to plugins in order to use the new compression
   options.

Awesome! Super helpful James. I can do keyword search for zlib in Dolfyn or if you know where that is set and point me to it that would be helpful.

ssolson · 2022-08-16T19:30:54Z

Using a conda environment we can get them all to pass. Testing this way would require us to change our guidance around how we ask users to install mhkit. E.g. we want to test the same way we recommend. Currently we recommend pip for everything. This would be moving towards a conda centric approach... part of our testing suggestions is test both pip and conda... and get mhkit on conda install.

Possibly? I've found it's just much easier to use conda in github actions than venv.

So this was a suggestion from the testing team. Essentially the testing mindset should be that you test what you tell your users to do. Which is what we do currently but using the conda env would deviate from our doementation as it stands. But yes working in the GitHub actions environment has been a bit of a headache and using a conda env ultimately fixed all the problems so it is certainly easier.

jmcvey3 · 2022-08-16T19:48:44Z

Did we not merge (or want to merge) the rest of the dolfyn source code? There's only an older version of the IO code in mhkit right now.

ssolson · 2022-08-16T19:53:43Z

Did we not merge (or want to merge) the rest of the dolfyn source code? There's only an older version of the IO code in mhkit right now.

I think you are referring to the master branch? The last Dolfyn merge went into the Development branch. Which I will state again that I disagree with this setup and that we should just push things into the master branch and delete the Development branch bc there is a whole tab for releases (https://github.com/MHKiT-Software/MHKiT-Python/releases) or we could add a release branch if we wanted to have this messier 2 branch system that we are currently on.

Let me know if I misunderstood your question.

jmcvey3 · 2022-08-16T19:59:05Z

Oh! Right. Yes so long as each branch doesn't interfere with another, but if there's unavoidable co-development on multiple branches, I could see the "Develop" branch being a good idea.

jmcvey3 · 2022-08-16T20:09:51Z

I created a new pull request for Develop off the same dolfyn branch with this fix and several others in it.

ssolson · 2022-08-17T16:14:12Z

@akeeste would you mind taking a look/ providing review/approval on these testing changes?

I have summarized the changes in the intro box.

akeeste · 2022-08-17T21:32:29Z

@ssolson

I've been keeping up with this and the Dolfyn thread at a high level. Can you confirm that I'm understanding correctly:

pip install is still being tested, but now as a different job in Actions than the conda install job.
You added conda install tests in anticipation of openly supporting/recommending that to users going forward
[Windows + conda install + python 3.7] is the only conda installation that doesn't currently work

The netcdf and dolfyn compression changes seem straightforward now that they're figured out. I can look into windows + conda + python 3.7, but am probably less familiar with this than you are.

ssolson · 2022-08-18T14:58:33Z

@ssolson

I've been keeping up with this and the Dolfyn thread at a high level. Can you confirm that I'm understanding correctly:

pip install is still being tested, but now as a different job in Actions than the conda install job.

Pip install is still being tested as before but I have removed the py 3.7 test because it fails on windows. I believe this is a github actions environment issue not an issue with our codebase. I plan to revisit testing soon to address other parts of #183 and will continue to work on this issue. At this point, I am approaching a month on this issue and I need to focus on other parts of MHKiT.

You added conda install tests in anticipation of openly supporting/recommending that to users going forward
Using a conda environment we are running all combinations of 3.7,8,9 and mac, windows, and linux.

[Windows + conda install + python 3.7] is the only conda installation that doesn't currently work

No. It would be [Windows + pip install + python 3.7], But I believe its a github actions environment issue.

The netcdf and dolfyn compression changes seem straightforward now that they're figured out. I can look into windows + conda + python 3.7, but am probably less familiar with this than you are.

I'm really just looking for a review. My suggested plan of action is to get these actions into the Develop branch, and add a comment to the testing issue #183 referencing the py3.7/windows issue. Integrating these changes will allow the code base to work with the newest version of Netcdf 1.6.0 and I can focus on the number of other MHKiT PRs which are in need of review.

akeeste · 2022-08-18T15:08:58Z

@ssolson Thanks for clarifying. I can't tag myself as reviewer, but have completed my review and approve of the changes. let's merge.

* Merge Master into Develop (#179) * Include last day of year (#160) Include the last day of the year when using years parameter. Fixes #154. * Timezone Bug fix: remove `.replace()` for `.astimezone()` (#161) * Bug fix: remove timezone replace for astimezone * add pytz to required packages * Fix: Pandas latest (#159) * Plot each col in DataFrame individually * Remove numpy and pandas version requirements * WDRT (#141) * Working version of Gaussian Copula * Create dedicated function for iso probs and quantiles * Contour plots worsk for single and multiple contours * Gumbel Copula. * Clayton Copula * Rosenblatt Copula * Guass and gumbel general copula * Add support for nonparametric gaussian, clayton, and gumbel copulas * Require statsmodel for nonparametric KDE copulas * Small changes * Fix bug in KDE log transformation * Adding docstrings * All doc strings updated * Add markers option to contours plotting * Add Copula tests comparing to WDRTresults * Fix x1, x2 pts bug in KDE contour, clean up * Example showing the calculation of all copla methods and comparing to WDRT results * Simplifications and lanuguage cleanup * Add statsmodel * Add testing data files * In plot envrionmental contors convert x1,x2 to values if a Series is passed * Docstring typo corrections * Corrected notebook description typos. * module and example for short-term extreme distributions * docstrings for short-term extreme functions * fixes #140 Speedup surface elevation calculation. * correction to surface_elevation * elevation * minor fixes based on review comments * add environmental_contours to init * Remove WDRT functionality * All WDRT contour functionality can be called from one function * Rework tests for new contour functions setup and location * Move all contour example into this file * Update to work with new structure. * Remove all commented out wdrt functions from resource * No changes made. Reverting back. * Have PCA method use general fit. Adds PCAmethod to fit method but can also accept the PCA dictionary. General Docstring Cleanup * Update discussion around the use of the copula method. * Remove copula stand alone notebook * minor formmating changes * Remove test bugs created from merge with origin * Import env contours into resource module and adjust test, examples, ect to function with tthe new structure * Uncommented tests * Remove reference to import the env contours module * fixing minor typo * Move env contours to _file and adjust package to handle new structure * Minor formatting cleanup * Cleanup unsed packages and variables * intial mler upload * formalize functions * add tests and example * sampling rate to averaging period * rename MLER example * working * merging * WDRT functions and examples finalized. Missing tests. * integrate mler into extreme * fixes tests? * fix tests * fix tests * fix full sea state example? * test contour samples * test sample full seastate * random seed * test long term extreme * expand MLER functionality * test short-term extremes * fix mler tests * MLER test error * mler fix 2 * update formats and asserts * further cleanup of countours.py for consistency, typos, and pep8 * run tests? * fix test? * change lists to np.array so they pass the assert statements * clean extreme.py for consistency, pep8 * points_per_interval should be int * typing in examples * full sea state example: make it easier to follow, add more explanation, and use consistent naming * contour example: make it easier to follow, add more explanation, and use consistent naming * Alias ste function and minor formatting. * naming convention * rename example * ste example * allow weights to be np.array * allow weights to be np.ndarray Co-authored-by: ssolson <[email protected]> Co-authored-by: ssolson <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: hivanov-nrel <[email protected]> * DOLfYN IO (#126) * input output files + testing * Code update * Test reformatting * Testing paths * dependency switch from h5netcdf to netcdf4 * Code cleanup * Test import fix * Minor docstring edits * Add dolfyn test data * Test fixes for mhkit * Removing uneeded test files, improving consistency * Not sure why these files didn't upload * Organizational changes and clarifications. Request for additional clarification * Dolfyn codebase updates * Dolfyn example notebooks update * Testing updates * Update example datafile * Latest dolfyn v0.13.0 updates * Name change * input output files + testing * Code update * Organizational changes and clarifications. Request for additional clarification * Dolfyn codebase updates * fix bug and make faster. Un-hard-code default seed/phase. * fix tests * delete commented out tests * retrigger checks * Checkpoint push * New dolfyn data files * Reorganize dolfyn testing * pytest install for warnings tests * Final update for IO * Remove future updates * Ensure file compression runs * Update numpy dependency * trying numpy v1.22 * changing numpy requirements * Remake unittest test cases * Save format options * Add dolfyn view to notebooks * Decode remaining binary ad2cp variables * DOLfYN 1.0.0 dependency Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: ssolson <[email protected]> Co-authored-by: Michelen <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: rpauly18 <[email protected]> * regenerate WDRT examples (#163) * updating version number in package * fixing dolfyn imports from pip * updating version number for bug fix * NDBC Metocean data (#152) * catch extra header in ndbc data * NDBC cwind example * update unit catch in request_data * update gust plot to show more data * add tests to read cwind with and without units * move graphics import to ndbc module and fix capitalization * use resample, add gridlines, rename variables, replace NDBC nans up front * rename wind_example to metocean_example * adding "Develop" branch to PR Co-authored-by: Carlos A. Michelén Ströfer <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: hivanov-nrel <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: Adam Keester <[email protected]> * Test Suite Restructure (#174) * test_wave file to folder with individual feature files * Move io tests into io folder * Send plots to a plots folder * Move resource characterizations plot tests into resource metrics file * move load tests to folder * move power tests to folder * move river tests to folder * move tidal tests to folder * move utils tests to folder * add inits so that tests pick up the tests in subfolders * Require previous netCDF4 release. * FIx data directory path * Fix data directory path * Fix load data directory path * Remove relpath call to see if it fixes Windows test suite issues * Remove relpath from the rest of io tests in wave * DOLfYN source code (#169) * Initial push * Latest bugfixes * Remove phase-in text * Update docstrings * Readability, imports * Make functions public * Docstring updates * Fix test oversight * Fix docstring * Missing timestamp fix for classic Nortek * Docstring updates * Renaming functions per mhkit standards * Single array vs binned array functions * Reorganizing analysis code * Update invalid orientation matrix warning * Add config details to dataset attributes * dolfyn bugfixes Co-authored-by: jmcvey3 <[email protected]> * Delft 3D Timestep to Seconds Function (#168) * fixing file history * fixing file history * fixing file history * made revisions from ssolson review * changed time_stamp to timestamp * updated varible names * updated variable names * fixed syntax line 63 * umdated variable name * fixed typos in doc strings * ssolson review 5-26-2022 * deleted excessive code in get_all_data * chaged TI to a % from a fraction * adjusted TI test to be a % * fixed doc string for TI function * Docstring adjustments * updated doc strings * Make convert_time a hidden func, make 2 functions to access it * Minor changes to get_lay_data docstring and formatting * small changes * remove print from create_points * example notebook clean up * fixed merge confilics * updated after merge * final developmet barnach merge updates * deleated space * Delete test_river.py * added coverage to test * updated min and max plot labels Co-authored-by: Browning <[email protected]> Co-authored-by: ssolson <[email protected]> * NetCDF4 (#181) * Use conda TEST env on Py 3.7, 3.8, 3.9 * Use pip install with Py 3.8 and 3.9. * Updates for WEC-Sim v5.0 (#185) * update read_wecsim for WEC-Sim v5.0 * add cable class * update notebook with results * add try-except for Morrison (v4.1) vs Morison (v4.2+) * add cable check and dataset to wave tests * update cable dataset * Fix pip install tests (#194) * Require previous version of NetCDF4. * Use original pre install for environemnt in pip tests * Only test pip * No loading shell PATH * Do not upgrade h5py * Set NETCDF to 1.5.8 * pytest and coverage * yaml syntax fix * Change need to pip-build * Control coverage and test via rc files * Move configuration file to workflows directory * Move configuration file to workflows directory * Specify the location of configuration files in coverage/pytest call * run conda & pip, pass both to coveralls * conda bash -l {0} * Add hindcast build which runs serial * configuration files for hindcast run * Specify hindcast configuration for hindcast build * Remove version specification from hindcast calls * Specify configuration files to include/omit hindcast accordingly * Change job name * Fix configuration to hindcast file not test * Use all OS and py versions in tests * pandas <=1.5.0 * xarray <=2022.9.0 * Only hindcast * Clean up package install * Fixing hypy and h5pyd to previous versions * Run limited other tests * No hindcast tests * remove hindcast comments and coveralls call * remove py3.8 from pip build * requirements.txt netCDF <= 1.5.8 * Add function to compute value at a given return period (#193) * Run CI on push to Develop (#200) * Dolfyn general updates (#186) * Initial push * Latest bugfixes * Remove phase-in text * Update docstrings * Readability, imports * Make functions public * Docstring updates * Fix test oversight * Fix docstring * Missing timestamp fix for classic Nortek * Docstring updates * Renaming functions per mhkit standards * Single array vs binned array functions * Reorganizing analysis code * Update invalid orientation matrix warning * Add config details to dataset attributes * dolfyn bugfixes * ADV updates : * Update compression option for netcdf4 * Remove compression options * Fix clean function * Set functions as private * Move stress functions to ADV dir * Add bottleneck to reqs * Minor changes * Remove old functions * Code simplification * Update examples * Make functions public * Add bottleneck to dependencies * Fix notebook * Bugfix for beam vars Co-authored-by: jmcvey3 <[email protected]> * Delft3D z calculation (#190) * updated variable names * updated variable names * updated variable names * fixed a few docstring typos * updated s1 to water level * updated s1 to waterlevel * updated s1 to waterlevel * updated s1 to water level * updated s1 to waterlever * fixed typo depth to waterdepth * added edges = nearest example * added edges = nearest option in variable interpolate * updated z to waterdepth * added edges= nearest example Co-authored-by: Browning <[email protected]> * Directional NDBC (#197) * working on directional NDBC * clean up functions. Write docstrings. Write assert statements. Write tests. Create Tutorial. * docstring and asserts for plotting function * Bug Fix: Averaging histogram bin and wave energy (#205) * Fix averaging bug * Add outline to bin counts text for better contrast * Update plotting function and plot in example notebook * Fix: update variable name to remove reference before assignment error (#208) * update wave.contours.samples_contour to not get variable referenced before assignment error Co-authored-by: Graham Penrose <[email protected]> * HSDS (#211) Adds hindcast tests back into the test suite. The NREL HSDS API issues were resolved by creating multiple calls to the API for direction wave spectrum requests. Additionally, an exponential back-off time was implemented to retry calls with an increased wait time between calls. Co-authored-by: Adam Keester <[email protected]> * Provide function to convert from Te to Tp using ITTC approximation (#210) * Provide function to convert from Te to Tp using ITTC approximation * Apply suggestions from code review Co-authored-by: Adam Keester <[email protected]> * Metocean module - WIND Toolkit (#187) * initial script * update wind_toolkit with 4 regions and 1-hr or 5-min data * update parameters of wind_toolkit functions * compare NDBC and WIND metocean data * fix typo in wpto hindcast example * finish metocean example and add results * add wind_toolkit to wave/io/__init__.py * update WIND Toolkit parameter list * initial test structure for WIND toolkit * finish metocean example * add function to plot each region * wind toolkit tests and test data * add tests for misc wind_toolkit MHKiT functions and cases * fix wind toolkit tests * add elevation_to_string utility function * misc cleanup * clarify available parameters * add users lat_lon to plot_region visualization * update example with new functions * add numpy dependency back in * Ingnore the new hindcast folder * Include only the new hindcast folder * Move hindcast tests to hindcast folder * Specify exclusion of hindcast from standard coverage * update hindcast coverage * remove unnecessary header * move hindcast and wind_toolkit to wave/io/hindcast * update .coveragerc files * update paths for new wave.io.hindcast directory Co-authored-by: ssolson <[email protected]> * Dolfyn updates (#212) * RDI reader, logging, duty cycle motion correction * Updates * Force float32 datatype * Docstring formatting * Change 'default' docstring default * Remove old comments * Trasect comparison (#199) * Require previous version of NetCDF4. * updated variable names * updated variable names * updated variable names * fixed a few docstring typos * updated s1 to water level * updated s1 to waterlevel * updated s1 to waterlevel * updated s1 to water level * updated s1 to waterlever * fixed typo depth to waterdepth * D3d Tanana Transect Example * added optional edge interpolation with nearest * updated z to water depth * RDI reader, logging, duty cycle motion correction * add dolfyn, currenly will not run on the branch need jmcvey3 trdi_5beam * found river bottom and removed data below * Updates * Force float32 datatype * added downsampelind comparison and tried contourf * D3D tanana data * tanana transect 2 * tanana transect 3 * Docstring formatting * Change 'default' docstring default * Remove old comments * updated code to match Energies paper * Review * Boat transect image * updated variable names * updated variable names * updated variable names added to discriptions * Clean up and suggestions * added to the examples * TRTS study edits * code and discription updates * Edits and TODOs * moved files, updated discriptions * readding Delft3D_example notebook * added USGS discharge * updated doc strings * example updates and move data files * updated doc strings and Error equations * updated doc strings * pulled down development branch --------- Co-authored-by: ssolson <[email protected]> Co-authored-by: Browning <[email protected]> Co-authored-by: jmcvey3 <[email protected]> --------- Co-authored-by: Carlos A. Michelén Ströfer <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: hivanov-nrel <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: Adam Keester <[email protected]> Co-authored-by: Emily Browning <[email protected]> Co-authored-by: Browning <[email protected]> Co-authored-by: Mark Bruggemann <[email protected]> Co-authored-by: Graham Penrose <[email protected]>

* Require previous version of NetCDF4. * Merge Master into Develop (#179) * Include last day of year (#160) Include the last day of the year when using years parameter. Fixes #154. * Timezone Bug fix: remove `.replace()` for `.astimezone()` (#161) * Bug fix: remove timezone replace for astimezone * add pytz to required packages * Fix: Pandas latest (#159) * Plot each col in DataFrame individually * Remove numpy and pandas version requirements * WDRT (#141) * Working version of Gaussian Copula * Create dedicated function for iso probs and quantiles * Contour plots worsk for single and multiple contours * Gumbel Copula. * Clayton Copula * Rosenblatt Copula * Guass and gumbel general copula * Add support for nonparametric gaussian, clayton, and gumbel copulas * Require statsmodel for nonparametric KDE copulas * Small changes * Fix bug in KDE log transformation * Adding docstrings * All doc strings updated * Add markers option to contours plotting * Add Copula tests comparing to WDRTresults * Fix x1, x2 pts bug in KDE contour, clean up * Example showing the calculation of all copla methods and comparing to WDRT results * Simplifications and lanuguage cleanup * Add statsmodel * Add testing data files * In plot envrionmental contors convert x1,x2 to values if a Series is passed * Docstring typo corrections * Corrected notebook description typos. * module and example for short-term extreme distributions * docstrings for short-term extreme functions * fixes #140 Speedup surface elevation calculation. * correction to surface_elevation * elevation * minor fixes based on review comments * add environmental_contours to init * Remove WDRT functionality * All WDRT contour functionality can be called from one function * Rework tests for new contour functions setup and location * Move all contour example into this file * Update to work with new structure. * Remove all commented out wdrt functions from resource * No changes made. Reverting back. * Have PCA method use general fit. Adds PCAmethod to fit method but can also accept the PCA dictionary. General Docstring Cleanup * Update discussion around the use of the copula method. * Remove copula stand alone notebook * minor formmating changes * Remove test bugs created from merge with origin * Import env contours into resource module and adjust test, examples, ect to function with tthe new structure * Uncommented tests * Remove reference to import the env contours module * fixing minor typo * Move env contours to _file and adjust package to handle new structure * Minor formatting cleanup * Cleanup unsed packages and variables * intial mler upload * formalize functions * add tests and example * sampling rate to averaging period * rename MLER example * working * merging * WDRT functions and examples finalized. Missing tests. * integrate mler into extreme * fixes tests? * fix tests * fix tests * fix full sea state example? * test contour samples * test sample full seastate * random seed * test long term extreme * expand MLER functionality * test short-term extremes * fix mler tests * MLER test error * mler fix 2 * update formats and asserts * further cleanup of countours.py for consistency, typos, and pep8 * run tests? * fix test? * change lists to np.array so they pass the assert statements * clean extreme.py for consistency, pep8 * points_per_interval should be int * typing in examples * full sea state example: make it easier to follow, add more explanation, and use consistent naming * contour example: make it easier to follow, add more explanation, and use consistent naming * Alias ste function and minor formatting. * naming convention * rename example * ste example * allow weights to be np.array * allow weights to be np.ndarray Co-authored-by: ssolson <[email protected]> Co-authored-by: ssolson <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: hivanov-nrel <[email protected]> * DOLfYN IO (#126) * input output files + testing * Code update * Test reformatting * Testing paths * dependency switch from h5netcdf to netcdf4 * Code cleanup * Test import fix * Minor docstring edits * Add dolfyn test data * Test fixes for mhkit * Removing uneeded test files, improving consistency * Not sure why these files didn't upload * Organizational changes and clarifications. Request for additional clarification * Dolfyn codebase updates * Dolfyn example notebooks update * Testing updates * Update example datafile * Latest dolfyn v0.13.0 updates * Name change * input output files + testing * Code update * Organizational changes and clarifications. Request for additional clarification * Dolfyn codebase updates * fix bug and make faster. Un-hard-code default seed/phase. * fix tests * delete commented out tests * retrigger checks * Checkpoint push * New dolfyn data files * Reorganize dolfyn testing * pytest install for warnings tests * Final update for IO * Remove future updates * Ensure file compression runs * Update numpy dependency * trying numpy v1.22 * changing numpy requirements * Remake unittest test cases * Save format options * Add dolfyn view to notebooks * Decode remaining binary ad2cp variables * DOLfYN 1.0.0 dependency Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: ssolson <[email protected]> Co-authored-by: Michelen <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: rpauly18 <[email protected]> * regenerate WDRT examples (#163) * updating version number in package * fixing dolfyn imports from pip * updating version number for bug fix * NDBC Metocean data (#152) * catch extra header in ndbc data * NDBC cwind example * update unit catch in request_data * update gust plot to show more data * add tests to read cwind with and without units * move graphics import to ndbc module and fix capitalization * use resample, add gridlines, rename variables, replace NDBC nans up front * rename wind_example to metocean_example * adding "Develop" branch to PR Co-authored-by: Carlos A. Michelén Ströfer <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: hivanov-nrel <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: Adam Keester <[email protected]> * Test Suite Restructure (#174) * test_wave file to folder with individual feature files * Move io tests into io folder * Send plots to a plots folder * Move resource characterizations plot tests into resource metrics file * move load tests to folder * move power tests to folder * move river tests to folder * move tidal tests to folder * move utils tests to folder * add inits so that tests pick up the tests in subfolders * Require previous netCDF4 release. * FIx data directory path * Fix data directory path * Fix load data directory path * Remove relpath call to see if it fixes Windows test suite issues * Remove relpath from the rest of io tests in wave * DOLfYN source code (#169) * Initial push * Latest bugfixes * Remove phase-in text * Update docstrings * Readability, imports * Make functions public * Docstring updates * Fix test oversight * Fix docstring * Missing timestamp fix for classic Nortek * Docstring updates * Renaming functions per mhkit standards * Single array vs binned array functions * Reorganizing analysis code * Update invalid orientation matrix warning * Add config details to dataset attributes * dolfyn bugfixes Co-authored-by: jmcvey3 <[email protected]> * Delft 3D Timestep to Seconds Function (#168) * fixing file history * fixing file history * fixing file history * made revisions from ssolson review * changed time_stamp to timestamp * updated varible names * updated variable names * fixed syntax line 63 * umdated variable name * fixed typos in doc strings * ssolson review 5-26-2022 * deleted excessive code in get_all_data * chaged TI to a % from a fraction * adjusted TI test to be a % * fixed doc string for TI function * Docstring adjustments * updated doc strings * Make convert_time a hidden func, make 2 functions to access it * Minor changes to get_lay_data docstring and formatting * small changes * remove print from create_points * example notebook clean up * fixed merge confilics * updated after merge * final developmet barnach merge updates * deleated space * Delete test_river.py * added coverage to test * updated min and max plot labels Co-authored-by: Browning <[email protected]> Co-authored-by: ssolson <[email protected]> * NetCDF4 (#181) * Use conda TEST env on Py 3.7, 3.8, 3.9 * Use pip install with Py 3.8 and 3.9. * Updates for WEC-Sim v5.0 (#185) * update read_wecsim for WEC-Sim v5.0 * add cable class * update notebook with results * add try-except for Morrison (v4.1) vs Morison (v4.2+) * add cable check and dataset to wave tests * update cable dataset * updated variable names * updated variable names * updated variable names * fixed a few docstring typos * updated s1 to water level * updated s1 to waterlevel * updated s1 to waterlevel * updated s1 to water level * updated s1 to waterlever * fixed typo depth to waterdepth * D3d Tanana Transect Example * Fix pip install tests (#194) * Require previous version of NetCDF4. * Use original pre install for environemnt in pip tests * Only test pip * No loading shell PATH * Do not upgrade h5py * Set NETCDF to 1.5.8 * pytest and coverage * yaml syntax fix * Change need to pip-build * Control coverage and test via rc files * Move configuration file to workflows directory * Move configuration file to workflows directory * Specify the location of configuration files in coverage/pytest call * run conda & pip, pass both to coveralls * conda bash -l {0} * Add hindcast build which runs serial * configuration files for hindcast run * Specify hindcast configuration for hindcast build * Remove version specification from hindcast calls * Specify configuration files to include/omit hindcast accordingly * Change job name * Fix configuration to hindcast file not test * Use all OS and py versions in tests * pandas <=1.5.0 * xarray <=2022.9.0 * Only hindcast * Clean up package install * Fixing hypy and h5pyd to previous versions * Run limited other tests * No hindcast tests * remove hindcast comments and coveralls call * remove py3.8 from pip build * requirements.txt netCDF <= 1.5.8 * Add function to compute value at a given return period (#193) * Run CI on push to Develop (#200) * added optional edge interpolation with nearest * updated z to water depth * Dolfyn general updates (#186) * Initial push * Latest bugfixes * Remove phase-in text * Update docstrings * Readability, imports * Make functions public * Docstring updates * Fix test oversight * Fix docstring * Missing timestamp fix for classic Nortek * Docstring updates * Renaming functions per mhkit standards * Single array vs binned array functions * Reorganizing analysis code * Update invalid orientation matrix warning * Add config details to dataset attributes * dolfyn bugfixes * ADV updates : * Update compression option for netcdf4 * Remove compression options * Fix clean function * Set functions as private * Move stress functions to ADV dir * Add bottleneck to reqs * Minor changes * Remove old functions * Code simplification * Update examples * Make functions public * Add bottleneck to dependencies * Fix notebook * Bugfix for beam vars Co-authored-by: jmcvey3 <[email protected]> * RDI reader, logging, duty cycle motion correction * Delft3D z calculation (#190) * updated variable names * updated variable names * updated variable names * fixed a few docstring typos * updated s1 to water level * updated s1 to waterlevel * updated s1 to waterlevel * updated s1 to water level * updated s1 to waterlever * fixed typo depth to waterdepth * added edges = nearest example * added edges = nearest option in variable interpolate * updated z to waterdepth * added edges= nearest example Co-authored-by: Browning <[email protected]> * Directional NDBC (#197) * working on directional NDBC * clean up functions. Write docstrings. Write assert statements. Write tests. Create Tutorial. * docstring and asserts for plotting function * add dolfyn, currenly will not run on the branch need jmcvey3 trdi_5beam * found river bottom and removed data below * Bug Fix: Averaging histogram bin and wave energy (#205) * Fix averaging bug * Add outline to bin counts text for better contrast * Update plotting function and plot in example notebook * Fix: update variable name to remove reference before assignment error (#208) * update wave.contours.samples_contour to not get variable referenced before assignment error Co-authored-by: Graham Penrose <[email protected]> * HSDS (#211) Adds hindcast tests back into the test suite. The NREL HSDS API issues were resolved by creating multiple calls to the API for direction wave spectrum requests. Additionally, an exponential back-off time was implemented to retry calls with an increased wait time between calls. Co-authored-by: Adam Keester <[email protected]> * Provide function to convert from Te to Tp using ITTC approximation (#210) * Provide function to convert from Te to Tp using ITTC approximation * Apply suggestions from code review Co-authored-by: Adam Keester <[email protected]> * Updates * Force float32 datatype * added downsampelind comparison and tried contourf * D3D tanana data * tanana transect 2 * tanana transect 3 * Docstring formatting * Metocean module - WIND Toolkit (#187) * initial script * update wind_toolkit with 4 regions and 1-hr or 5-min data * update parameters of wind_toolkit functions * compare NDBC and WIND metocean data * fix typo in wpto hindcast example * finish metocean example and add results * add wind_toolkit to wave/io/__init__.py * update WIND Toolkit parameter list * initial test structure for WIND toolkit * finish metocean example * add function to plot each region * wind toolkit tests and test data * add tests for misc wind_toolkit MHKiT functions and cases * fix wind toolkit tests * add elevation_to_string utility function * misc cleanup * clarify available parameters * add users lat_lon to plot_region visualization * update example with new functions * add numpy dependency back in * Ingnore the new hindcast folder * Include only the new hindcast folder * Move hindcast tests to hindcast folder * Specify exclusion of hindcast from standard coverage * update hindcast coverage * remove unnecessary header * move hindcast and wind_toolkit to wave/io/hindcast * update .coveragerc files * update paths for new wave.io.hindcast directory Co-authored-by: ssolson <[email protected]> * Change 'default' docstring default * Remove old comments * updated code to match Energies paper * Review * Boat transect image * updated variable names * Dolfyn updates (#212) * RDI reader, logging, duty cycle motion correction * Updates * Force float32 datatype * Docstring formatting * Change 'default' docstring default * Remove old comments * updated variable names * updated variable names added to discriptions * Clean up and suggestions * added to the examples * TRTS study edits * code and discription updates * Edits and TODOs * moved files, updated discriptions * readding Delft3D_example notebook * added USGS discharge * updated doc strings * example updates and move data files * updated doc strings and Error equations * updated doc strings * pulled down development branch * updated figure visibility * updated figure at L2 equation * restor old version * updated L2 eqation and intro fig * updated L2 eqation and intro fig * updated L2 equation * pulled in Develope branch * updated figure formate --------- Co-authored-by: ssolson <[email protected]> Co-authored-by: ssolson <[email protected]> Co-authored-by: Carlos A. Michelén Ströfer <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: hivanov-nrel <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: jmcvey3 <[email protected]> Co-authored-by: rpauly18 <[email protected]> Co-authored-by: Adam Keester <[email protected]> Co-authored-by: Browning <[email protected]> Co-authored-by: Mark Bruggemann <[email protected]> Co-authored-by: Graham Penrose <[email protected]>

ssolson added 6 commits July 12, 2022 09:46

Merge branch 'Develop' of https://github.com/MHKiT-Software/MHKiT-Python

57cca22

into Develop

Require previous version of NetCDF4.

f81a6c4

Merge branch 'Develop' of https://github.com/MHKiT-Software/MHKiT-Python

3e8e11a

into Develop

Merge branch 'Develop' of https://github.com/MHKiT-Software/MHKiT-Python

5b880da

into Develop

Merge branch 'Develop' of https://github.com/MHKiT-Software/MHKiT-Python

7a8abb2

into Develop

Remove netcdf version requirement

194d501

ssolson added the Clean Up Improve code consistency and readability label Jul 25, 2022

ssolson self-assigned this Jul 25, 2022

ssolson linked an issue Jul 25, 2022 that may be closed by this pull request

NETCDF 1.6.0 Breaking Tests #177

Closed

ssolson added 2 commits July 25, 2022 09:50

Add h5py to requirements

d13f1bb

Move h5py to be before NetCDF in rquirements file

e6de067

ssolson mentioned this pull request Jul 25, 2022

DOLfYN source code #169

Merged

ssolson added 2 commits July 26, 2022 08:33

Add h5py to actions workflow.

7b7497b

force h5py install

c276a73

ssolson added 7 commits August 4, 2022 11:29

checkout v3, setup-python v4, no miniconda

d980a53

Load path before each run (shell: bash -l {0})

0e352c6

Add name to fix yml syntax error

54b9b5d

Remove bash call before python installation

50ceacf

conda and cython

dc7fe8f

Remove compressions from netcdf save

7f05e9f

Use classic mhkit miniconda install in actions

9cbcbe4

ssolson added 3 commits August 11, 2022 09:55

Use classic miniconda, actions-setup-python, and checkout"

a0fc84d

Remove bash -l call because it adds py2.7 to PATH

45a96c8

Remove bash -l call because it adds py2.7 to PATH

c75cab8

ssolson added 2 commits August 16, 2022 09:09

Add coveralls to conda environment

12d9e7b

activate TEST env on coveralls call

1130699

ssolson added 6 commits August 17, 2022 08:02

Test with compression and set compression to zlib

970b07d

Remove compression

025fe79

Remove compression

af115e6

Add pip build with 3.8 and 3.9 support

478f80f

rename jobs

d4112a5

Change job name in needs

9f23046

ssolson marked this pull request as ready for review August 17, 2022 16:12

ssolson merged commit eadd8b9 into MHKiT-Software:Develop Aug 18, 2022

ssolson mentioned this pull request Sep 18, 2022

h5py error when installing MHKiT on clean install #192

Closed

ssolson mentioned this pull request Oct 3, 2022

Metocean module - WIND Toolkit #187

Merged

4 tasks

ssolson deleted the netcdf branch February 1, 2023 15:34

ssolson mentioned this pull request Feb 10, 2023

MHKiT v0.6.0 #222

Merged

NetCDF4 #181

NetCDF4 #181

Uh oh!

Conversation

ssolson commented Jul 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ssolson commented Jul 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmcvey3 commented Jul 26, 2022

Uh oh!

ssolson commented Jul 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmcvey3 commented Jul 26, 2022

Uh oh!

ssolson commented Jul 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ssolson commented Aug 11, 2022

Uh oh!

ssolson commented Aug 16, 2022

Uh oh!

jmcvey3 commented Aug 16, 2022

Uh oh!

ssolson commented Aug 16, 2022

Uh oh!

jmcvey3 commented Aug 16, 2022

Uh oh!

ssolson commented Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ssolson commented Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmcvey3 commented Aug 16, 2022

Uh oh!

ssolson commented Aug 16, 2022

Uh oh!

jmcvey3 commented Aug 16, 2022

Uh oh!

jmcvey3 commented Aug 16, 2022

Uh oh!

ssolson commented Aug 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akeeste commented Aug 17, 2022

Uh oh!

ssolson commented Aug 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akeeste commented Aug 18, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ssolson commented Jul 25, 2022 •

edited

Loading

ssolson commented Jul 26, 2022 •

edited

Loading

ssolson commented Jul 26, 2022 •

edited

Loading

ssolson commented Jul 26, 2022 •

edited

Loading

ssolson commented Aug 16, 2022 •

edited

Loading

ssolson commented Aug 16, 2022 •

edited

Loading

ssolson commented Aug 17, 2022 •

edited

Loading

ssolson commented Aug 18, 2022 •

edited

Loading