-
Notifications
You must be signed in to change notification settings - Fork 623
Netcdf parallel output extensions and bug repairs #1476
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Netcdf parallel output extensions and bug repairs #1476
Conversation
@kestonsmith-noaa Can you change this PR to point to dev/ufs-weather-model instead of develop? |
Done. PR is now pointing to the correct branch.
…On Mon, Jul 28, 2025 at 8:30 AM Jessica Meixner ***@***.***> wrote:
*JessicaMeixner-NOAA* left a comment (NOAA-EMC/WW3#1476)
<#1476 (comment)>
@kestonsmith-noaa <https://github.com/kestonsmith-noaa> Can you change
this PR to point to dev/ufs-weather-model instead of develop?
—
Reply to this email directly, view it on GitHub
<#1476 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AZUY35FYD3OPCG6KTDBQSJD3KYJUZAVCNFSM6AAAAACCMWEP5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCMRXGAYTCMBZHA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Note this PR replaced #1446 |
model/src/wav_history_mod.F90
Outdated
|
||
else if (trim(outvars(n)%dims) == 'nk') then ! freq + 1 axis for wavenumber | ||
var3d => var3dnk | ||
if(vname .eq. 'WN') call write_var3d_transpose(iodesc3dnk, vname, wn (1:len_nk ,1:nseal_cpl) ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kestonsmith-noaa Did you try simply sending transpose(wn)
instead of creating a new routine etc?
Hi Denise, As I recall, I did start there and ran into some difficulty. I
will check later this afternoon and get back to you with a clearer answer
on why I abandoned that path. Best, Keston
…On Mon, Aug 18, 2025 at 10:02 AM Denise Worthen ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In model/src/wav_history_mod.F90
<#1476 (comment)>:
> @@ -317,10 +376,16 @@ subroutine write_history ( timen )
if (vname .eq. 'USSPX') call write_var3d(iodesc3dp, vname, ussp (1:nseal_cpl, 1:usspf(2)) )
if (vname .eq. 'USSPY') call write_var3d(iodesc3dp, vname, ussp (1:nseal_cpl,nk+1:nk+usspf(2)) )
+ else if (trim(outvars(n)%dims) == 'nk') then ! freq + 1 axis for wavenumber
+ var3d => var3dnk
+ if(vname .eq. 'WN') call write_var3d_transpose(iodesc3dnk, vname, wn (1:len_nk ,1:nseal_cpl) )
@kestonsmith-noaa <https://github.com/kestonsmith-noaa> Did you try
simply sending transpose(wn) instead of creating a new routine etc?
—
Reply to this email directly, view it on GitHub
<#1476 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AZUY35D43Z4PUSEIVL33OC33OHMBLAVCNFSM6AAAAACCMWEP5KVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZTCMRYGUZDCNZYGM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
call handle_err(ierr,'def_freq') | ||
ierr = pio_put_att(pioid, varid, 'units', 's-1') | ||
end if | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please fix the indenting on these lines (2-space indents for if-statements).
Also, is there a reason they are PIO_DOUBLE? They're just real(kind=4) values, correct?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kestonsmith-noaa Why are these axis values required to be double precision?
@kestonsmith-noaa Have you compiled and run in debug mode w/ the additional variable
|
Thanks Denise, I had not run this in debug mode with wavenumber output.
Unless you already see it I will try to find the issue tonight. Best,
Keston
…On Mon, Aug 18, 2025 at 3:50 PM Denise Worthen ***@***.***> wrote:
*DeniseWorthen* left a comment (NOAA-EMC/WW3#1476)
<#1476 (comment)>
@kestonsmith-noaa <https://github.com/kestonsmith-noaa> Have you compiled
and run in debug mode w/ the additional variable WN included in the
output? I just did that test and it failed.
193: forrtl: severe (408): fort: (2): Subscript #2 of the array VAR has value 3674 which is greater than the upper bound of 3662
193:
193: Image PC Routine Line Source
193: fv3_s2sw.ks.db.ex 000000000146BD66 wav_history_mod_m 569 wav_history_mod.F90
193: fv3_s2sw.ks.db.ex 0000000001450B0E wav_history_mod_m 381 wav_history_mod.F90
193: fv3_s2sw.ks.db.ex 0000000001F5862C w3wavemd_mp_w3wav 2380 w3wavemd.F90
193: fv3_s2sw.ks.db.ex 0000000001426E83 wav_comp_nuopc_mp 1159 wav_comp_nuopc.F90
—
Reply to this email directly, view it on GitHub
<#1476 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AZUY35FBQ4CB7Q76D6S3CN33OIU7ZAVCNFSM6AAAAACCMWEP5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCOJYGIYDEMZUG4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I think the issue is that WN is global, and there doesn't seem to be an "extended array"
Once you fix the array size issue, please also try (in debug mode, with |
I tried handling the wavenumber output with write_var3d on transpose(wn)... but the output is spatially incoherent- as if the indexing is randomized. This is true for both the structured and unstructured case. The spatial noise in this output is different than the noise I mentioned in issue 1401(#1401). I understand regarding WN being a global variable, but the indexing doesn't seem to be consistent with this (at least the way the other global variables are indexed)- I'm sorry I don't have a more complete answer here. I never dug deep enough into the binary output routines and ww3_ounf to determine where the problem is for the wn output there either- it also looks like an indexing issue, but not everywhere. I've made a change to write_var3d_transpose to address the bug you found. I am testing this presently. |
@kestonsmith-noaa I've been able to get what appears to be a sensible This is 1e3*wn(k=1). ![]() Output is here: |
Great! This looks consistent with the output I have. Makes sense, we just need your "global" flag in var3d. How would you suggest we reconcile this? |
I would definitely prefer just using the existing var3d routine, transpose and the optional global flag. Could you just grab those changes from my branch ? One weird thing is that I can't find your fork. If I try to compare my branch to yours, it doesn't give me your fork in the possible list of forks. |
I agree regarding jettisoning var3d_transpose. I will set up my branch as
you suggest.I'm not sure what the issue is with the fork. If it doesn't
mess up the PR I can make another fork off of dev/ufs-weather-model.
…On Wed, Aug 20, 2025 at 3:34 PM Denise Worthen ***@***.***> wrote:
*DeniseWorthen* left a comment (NOAA-EMC/WW3#1476)
<#1476 (comment)>
I would definitely prefer just using the existing var3d routine, transpose
and the optional global flag. Could you just grab those changes from my
branch ?
One weird thing is that I can't find your fork. If I try to compare my
branch to yours, it doesn't give me your fork in the possible list of forks.
—
Reply to this email directly, view it on GitHub
<#1476 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AZUY35F2PX4VGNONUYUPOQL3OTEUTAVCNFSM6AAAAACCMWEP5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMBXHAYDQOJVGA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Don't worry about the fork issue right now, it's just weird. Regarding the binary, I suspect that the issue is in |
@kestonsmith-noaa Do you know if |
I suspect wn will not be an output in GFSv17 but that question should go to
@JessicaMeixner-NOAA for the definitive answer.
…On Thu, Aug 21, 2025 at 8:08 AM Denise Worthen ***@***.***> wrote:
*DeniseWorthen* left a comment (NOAA-EMC/WW3#1476)
<#1476 (comment)>
@kestonsmith-noaa <https://github.com/kestonsmith-noaa> Do you know if wn
will be an output field for the GFSv17? If yes, then I think this needs to
be tested with that mesh and on wcoss2 to ensure no unexpected behaviour.
—
Reply to this email directly, view it on GitHub
<#1476 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AZUY35F4SP7S4J2HJ33RSD33OWZERAVCNFSM6AAAAACCMWEP5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMJQGMZTIOBSG4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
@kestonsmith-noaa @DeniseWorthen - The output variables for GFSv17 are https://github.com/NOAA-EMC/global-workflow/blob/develop/dev/parm/config/gfs/config.wave#L92 |
I replaced my write_var3d routine with Dennise's updated version and tested
the wn output with write_var3d (... transpose (wn...). The result is the
same as I was getting previously and I believe it is correct. Please
remember that these fields are inconsistent with the binary wn output.
I still need to modify the new write_var3d to prevent mapping NaN
directional values to incorrect field values rather than NaN, this is the
result of behavior of the mod function in:
varloc(:) = mod(630. - rade*varloc(:), 360.)
On Thu, Aug 21, 2025 at 8:24 AM Keston Smith - NOAA Affiliate <
***@***.***> wrote:
… I suspect wn will not be an output in GFSv17 but that question should go
to @JessicaMeixner-NOAA for the definitive answer.
On Thu, Aug 21, 2025 at 8:08 AM Denise Worthen ***@***.***>
wrote:
> *DeniseWorthen* left a comment (NOAA-EMC/WW3#1476)
> <#1476 (comment)>
>
> @kestonsmith-noaa <https://github.com/kestonsmith-noaa> Do you know if wn
> will be an output field for the GFSv17? If yes, then I think this needs to
> be tested with that mesh and on wcoss2 to ensure no unexpected behaviour.
>
> —
> Reply to this email directly, view it on GitHub
> <#1476 (comment)>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AZUY35F4SP7S4J2HJ33RSD33OWZERAVCNFSM6AAAAACCMWEP5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMJQGMZTIOBSG4>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
from radians to transform to degrees
model/src/wav_history_mod.F90
Outdated
|
||
deallocate(varloc) | ||
end subroutine write_var3d | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like this routine got swapped in order w/ the var2d routine (ie, original order is routines var2d, then var3d), so it is appearing as a large difference. Please reorder var2d and var3d so it is clear what changed in the var3d routine.
if (vname .eq. 'USTY') call write_var2d(vname, ust (1:nseal_cpl)*asf(1:nseal_cpl), dir=sin(ustdir(1:nseal_cpl)), usemask='true') | ||
if (vname .eq. 'USTX') call write_var2d(vname, ust (1:nsea)*asf(1:nsea), dir=cos(ustdir(1:nsea)), init0='false', global='true') | ||
if (vname .eq. 'USTY') call write_var2d(vname, ust (1:nsea)*asf(1:nsea), dir=sin(ustdir(1:nsea)), init0='false', global='true') | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have you run in debug mode and verified all the changed variables (ustx,usty, ths,psi) work as expected?
Done.
…On Thu, Aug 21, 2025 at 9:57 AM Denise Worthen ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In model/src/wav_history_mod.F90
<#1476 (comment)>:
> + do k=1,size(varloc,1)
+ if (varloc(k).ne.undef) varloc(k)=mod( 630.-rade*varloc(k), 360.)
+ enddo
+ end if
+ end if
+ var3d(jsea,:) = varloc(:)
+ end do
+
+ ierr = pio_inq_varid(pioid, trim(vname), varid)
+ call handle_err(ierr, 'inquire variable '//trim(vname))
+ call pio_setframe(pioid, varid, int(1,kind=PIO_OFFSET_KIND))
+ call pio_write_darray(pioid, varid, iodesc, var3d, ierr)
+
+ deallocate(varloc)
+ end subroutine write_var3d
+
It looks like this routine got swapped in order w/ the var2d routine (ie,
original order is routines var2d, then var3d), so it is appearing as a
large difference. Please reorder var2d and var3d so it is clear what
changed in the var3d routine.
—
Reply to this email directly, view it on GitHub
<#1476 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AZUY35BMAVUCNFAPDSGM3HL3OXF3BAVCNFSM6AAAAACCMWEP5KVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZTCNBQHAYTMMZYG4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I corrected the handling of nan's for directional fields in write_var3d in
NetcdfNcPIOfixes. This has been tested with partitioned direction output
(pdir) and passed.
On Thu, Aug 21, 2025 at 10:19 AM Keston Smith - NOAA Affiliate <
***@***.***> wrote:
… Done.
On Thu, Aug 21, 2025 at 9:57 AM Denise Worthen ***@***.***>
wrote:
> ***@***.**** commented on this pull request.
> ------------------------------
>
> In model/src/wav_history_mod.F90
> <#1476 (comment)>:
>
> > + do k=1,size(varloc,1)
> + if (varloc(k).ne.undef) varloc(k)=mod( 630.-rade*varloc(k), 360.)
> + enddo
> + end if
> + end if
> + var3d(jsea,:) = varloc(:)
> + end do
> +
> + ierr = pio_inq_varid(pioid, trim(vname), varid)
> + call handle_err(ierr, 'inquire variable '//trim(vname))
> + call pio_setframe(pioid, varid, int(1,kind=PIO_OFFSET_KIND))
> + call pio_write_darray(pioid, varid, iodesc, var3d, ierr)
> +
> + deallocate(varloc)
> + end subroutine write_var3d
> +
>
> It looks like this routine got swapped in order w/ the var2d routine (ie,
> original order is routines var2d, then var3d), so it is appearing as a
> large difference. Please reorder var2d and var3d so it is clear what
> changed in the var3d routine.
>
> —
> Reply to this email directly, view it on GitHub
> <#1476 (review)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AZUY35BMAVUCNFAPDSGM3HL3OXF3BAVCNFSM6AAAAACCMWEP5KVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZTCNBQHAYTMMZYG4>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
Yes, there is no need for that. I think I was following the time dimension
output (which should be double). I will switch those to real as well as
related mpi types if you would prefer.
…On Thu, Aug 21, 2025 at 2:34 PM Denise Worthen ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In model/src/wav_history_mod.F90
<#1476 (comment)>:
> @@ -207,6 +230,18 @@ subroutine write_history ( timen )
ierr = pio_put_att(pioid, varid, 'long_name', 'node connectivity')
end if
+ ! define the frequency axis variables for wavenumber(wn) and spectra(ef)
+ if (k_axis) then
+ ierr = pio_def_var(pioid, 'freq_ef', PIO_DOUBLE, (/ktid/), varid)
+ call handle_err(ierr,'def_freq')
+ ierr = pio_put_att(pioid, varid, 'units', 's-1')
+ end if
+ if (nk_axis) then
+ ierr = pio_def_var(pioid, 'freq_wn', PIO_DOUBLE, (/nktid/), varid)
+ call handle_err(ierr,'def_freq')
+ ierr = pio_put_att(pioid, varid, 'units', 's-1')
+ end if
+
@kestonsmith-noaa <https://github.com/kestonsmith-noaa> Why are these
axis values required to be double precision?
—
Reply to this email directly, view it on GitHub
<#1476 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AZUY35FZJELXZPULAD7OFE33OYGMFAVCNFSM6AAAAACCMWEP5KVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZTCNBRHA3DIMBQGE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Unless they need to be double, I would use single. |
This reverts commit 1c736cd.
@kestonsmith-noaa Are you able to remove the tabs and the trailing whitespace in wav_history_mod.F90? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kestonsmith-noaa Thanks for removing the tabs; I think the whitespace might be harder, depending on which editor you're using.
Yes, Should be all cleaned up now. I think I've addressed all the issues
you raised and tested the netcdf vs binary output where relevant. Please
let me know if you need anything else from my end. Best, Keston
…On Mon, Aug 25, 2025 at 10:42 AM Denise Worthen ***@***.***> wrote:
*DeniseWorthen* left a comment (NOAA-EMC/WW3#1476)
<#1476 (comment)>
@kestonsmith-noaa <https://github.com/kestonsmith-noaa> Are you able to
remove the tabs and the trailing whitespace in wav_history_mod.F90?
—
Reply to this email directly, view it on GitHub
<#1476 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AZUY35GY4YIY3OHAK7OG3UD3PMOFDAVCNFSM6AAAAACCMWEP5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMRQGU3TQNBWGU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
@kestonsmith-noaa Is there any reason to be concerned about the failing checks on this subcomponent PR? I see it's been approved but just wanted to be sure before we proceed with WM PR #2826. |
@gspetro-NOAA - are you talking about the CI tests in WW3? If so, then no - those are not concerning. The CI has been broken for a while and does not reflect anything related to @kestonsmith-noaa's code. I'll try to get that turned off if that's causing confusion. |
Are we bumping WW3 to the latest develop or keeping this hash specifically? |
WW3's "develop" for UFS is dev/ufs-waether-model. This PR should be a branch on top of dev/ufs-weather-model. I will confirm now. |
Yes, thanks! I realized that too late |
Confirmed this branch is up to date w/dev/ufs-weather-model ... Always good to double check! Thanks @BrianCurtis-NOAA |
Testing has successfully completed for WM PR #2444, so we can begin the merging process for this subcomponent PR. |
91c2038
into
NOAA-EMC:dev/ufs-weather-model
Pull Request Summary
Repair bugs in netcdf parallel output and extend netcdf output capacity beyond original implementation.
Description
The proposed changes repair bugs in netcdf parallel output affecting partitioned direction(PDIR), friction velocity(UST), mean directional spread (THS), partitioned directional spread (PSI), and other variables. Units are corrected for energy flux (CGE) and dynamic time step (DTDYN).
Support is added for outputting wave number (WN).
Frequency dimensions are added to the output as well, where output variables depend on frequency (EF and WN).
All changes are within file wav_history_mod.F90
Changes only affect coupled runs within ufs-weather-model at present.
Jessica Meixner
bug, enhancement
Issue(s) addressed
Addresses:
#1366
Also has relevance to:
#1401
Commit Message
Fix bugs in netcdf parallel output and adds support for wave number and frequency dimensions in history output.
Check list
Testing
Changes were tested by comparing netcdf parallel output with binary output converted to netcdf with ww3_ounf.
No, regression tests which work with netcdf parallel output are needed to test the changes.
No
None