Skip to content

Conversation

kestonsmith-noaa
Copy link
Collaborator

@kestonsmith-noaa kestonsmith-noaa commented Jul 25, 2025

Pull Request Summary

Repair bugs in netcdf parallel output and extend netcdf output capacity beyond original implementation.

Description

The proposed changes repair bugs in netcdf parallel output affecting partitioned direction(PDIR), friction velocity(UST), mean directional spread (THS), partitioned directional spread (PSI), and other variables. Units are corrected for energy flux (CGE) and dynamic time step (DTDYN).

Support is added for outputting wave number (WN).

Frequency dimensions are added to the output as well, where output variables depend on frequency (EF and WN).

All changes are within file wav_history_mod.F90

Changes only affect coupled runs within ufs-weather-model at present.

  • Add any suggestions for a reviewer

Jessica Meixner

  • Mention any labels that should be added: bug, documentation, enhancement, new feature
    bug, enhancement

Issue(s) addressed

Addresses:
#1366

Also has relevance to:
#1401

Commit Message

Fix bugs in netcdf parallel output and adds support for wave number and frequency dimensions in history output.

Check list

Testing

  • How were these changes tested?
    Changes were tested by comparing netcdf parallel output with binary output converted to netcdf with ww3_ounf.
  • Are the changes covered by regression tests? (If not, why? Do new tests need to be added?)
    No, regression tests which work with netcdf parallel output are needed to test the changes.
  • Have the matrix regression tests been run (if yes, please note HPC and compiler)?
    No
  • Please indicate the expected changes in the regression test output, (Note the list of known non-identical tests.)
    None
  • Please provide the summary output of matrix.comp (matrix.Diff.txt, matrixCompFull.txt and matrixCompSummary.txt):

@JessicaMeixner-NOAA
Copy link
Collaborator

@kestonsmith-noaa Can you change this PR to point to dev/ufs-weather-model instead of develop?

@kestonsmith-noaa kestonsmith-noaa changed the base branch from develop to dev/ufs-weather-model July 28, 2025 14:20
@kestonsmith-noaa
Copy link
Collaborator Author

kestonsmith-noaa commented Jul 28, 2025 via email

@JessicaMeixner-NOAA
Copy link
Collaborator

Note this PR replaced #1446


else if (trim(outvars(n)%dims) == 'nk') then ! freq + 1 axis for wavenumber
var3d => var3dnk
if(vname .eq. 'WN') call write_var3d_transpose(iodesc3dnk, vname, wn (1:len_nk ,1:nseal_cpl) )
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kestonsmith-noaa Did you try simply sending transpose(wn) instead of creating a new routine etc?

@kestonsmith-noaa
Copy link
Collaborator Author

kestonsmith-noaa commented Aug 18, 2025 via email

call handle_err(ierr,'def_freq')
ierr = pio_put_att(pioid, varid, 'units', 's-1')
end if

Copy link
Contributor

@DeniseWorthen DeniseWorthen Aug 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix the indenting on these lines (2-space indents for if-statements).

Also, is there a reason they are PIO_DOUBLE? They're just real(kind=4) values, correct?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kestonsmith-noaa Why are these axis values required to be double precision?

@DeniseWorthen
Copy link
Contributor

@kestonsmith-noaa Have you compiled and run in debug mode w/ the additional variable WN included in the output? I just did that test and it failed.

193: forrtl: severe (408): fort: (2): Subscript #2 of the array VAR has value 3674 which is greater than the upper bound of 3662
193:
193: Image              PC                Routine            Line        Source
193: fv3_s2sw.ks.db.ex  000000000146BD66  wav_history_mod_m         569  wav_history_mod.F90
193: fv3_s2sw.ks.db.ex  0000000001450B0E  wav_history_mod_m         381  wav_history_mod.F90
193: fv3_s2sw.ks.db.ex  0000000001F5862C  w3wavemd_mp_w3wav        2380  w3wavemd.F90
193: fv3_s2sw.ks.db.ex  0000000001426E83  wav_comp_nuopc_mp        1159  wav_comp_nuopc.F90

@kestonsmith-noaa
Copy link
Collaborator Author

kestonsmith-noaa commented Aug 18, 2025 via email

@DeniseWorthen
Copy link
Contributor

DeniseWorthen commented Aug 18, 2025

I think the issue is that WN is global, and there doesn't seem to be an "extended array"

5:w3adatmd.F90:1326:    ALLOCATE ( WADATS(IMOD)%WN(0:NK+1,0:NSEA), STAT=ISTAT )

Once you fix the array size issue, please also try (in debug mode, with WN output included) and determine if you can use the existing var3d write routines if you call with transpose(wn(1:nk,1:nsea))

@kestonsmith-noaa
Copy link
Collaborator Author

I tried handling the wavenumber output with write_var3d on transpose(wn)... but the output is spatially incoherent- as if the indexing is randomized. This is true for both the structured and unstructured case. The spatial noise in this output is different than the noise I mentioned in issue 1401(#1401). I understand regarding WN being a global variable, but the indexing doesn't seem to be consistent with this (at least the way the other global variables are indexed)- I'm sorry I don't have a more complete answer here. I never dug deep enough into the binary output routines and ww3_ounf to determine where the problem is for the wn output there either- it also looks like an indexing issue, but not everywhere. I've made a change to write_var3d_transpose to address the bug you found. I am testing this presently.

@DeniseWorthen
Copy link
Contributor

DeniseWorthen commented Aug 20, 2025

@kestonsmith-noaa I've been able to get what appears to be a sensible wn output field using this branch using the existing var3d routine but sending w/ the transpose of wn. Note that branch also contains a couple of fixes I had been working on for the netcdf output.

This is 1e3*wn(k=1).

Screenshot 2025-08-20 at 2 54 00 PM

Output is here:
/scratch4/NCEPDEV/stmp/Denise.Worthen/RT_RUNDIRS/Denise.Worthen/FV3_RT/rt_1795368/pdlib.rundir

@kestonsmith-noaa
Copy link
Collaborator Author

Great! This looks consistent with the output I have. Makes sense, we just need your "global" flag in var3d. How would you suggest we reconcile this?

@DeniseWorthen
Copy link
Contributor

I would definitely prefer just using the existing var3d routine, transpose and the optional global flag. Could you just grab those changes from my branch ?

One weird thing is that I can't find your fork. If I try to compare my branch to yours, it doesn't give me your fork in the possible list of forks.

@kestonsmith-noaa
Copy link
Collaborator Author

kestonsmith-noaa commented Aug 20, 2025 via email

@DeniseWorthen
Copy link
Contributor

DeniseWorthen commented Aug 20, 2025

Don't worry about the fork issue right now, it's just weird.

Regarding the binary, I suspect that the issue is in pdlib_field_vec and the do_output_exchanges. I think there needs to be added code for the wn arrays.

@DeniseWorthen
Copy link
Contributor

@kestonsmith-noaa Do you know if wn will be an output field for the GFSv17? If yes, then I think this needs to be tested with that mesh and on wcoss2 to ensure no unexpected behaviour.

@kestonsmith-noaa
Copy link
Collaborator Author

kestonsmith-noaa commented Aug 21, 2025 via email

@JessicaMeixner-NOAA
Copy link
Collaborator

@kestonsmith-noaa
Copy link
Collaborator Author

kestonsmith-noaa commented Aug 21, 2025 via email


deallocate(varloc)
end subroutine write_var3d

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this routine got swapped in order w/ the var2d routine (ie, original order is routines var2d, then var3d), so it is appearing as a large difference. Please reorder var2d and var3d so it is clear what changed in the var3d routine.

if (vname .eq. 'USTY') call write_var2d(vname, ust (1:nseal_cpl)*asf(1:nseal_cpl), dir=sin(ustdir(1:nseal_cpl)), usemask='true')
if (vname .eq. 'USTX') call write_var2d(vname, ust (1:nsea)*asf(1:nsea), dir=cos(ustdir(1:nsea)), init0='false', global='true')
if (vname .eq. 'USTY') call write_var2d(vname, ust (1:nsea)*asf(1:nsea), dir=sin(ustdir(1:nsea)), init0='false', global='true')

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you run in debug mode and verified all the changed variables (ustx,usty, ths,psi) work as expected?

@kestonsmith-noaa
Copy link
Collaborator Author

kestonsmith-noaa commented Aug 21, 2025 via email

@kestonsmith-noaa
Copy link
Collaborator Author

kestonsmith-noaa commented Aug 21, 2025 via email

@kestonsmith-noaa
Copy link
Collaborator Author

kestonsmith-noaa commented Aug 21, 2025 via email

@DeniseWorthen
Copy link
Contributor

Unless they need to be double, I would use single.

@DeniseWorthen
Copy link
Contributor

DeniseWorthen commented Aug 25, 2025

@kestonsmith-noaa Are you able to remove the tabs and the trailing whitespace in wav_history_mod.F90?

Copy link
Contributor

@DeniseWorthen DeniseWorthen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kestonsmith-noaa Thanks for removing the tabs; I think the whitespace might be harder, depending on which editor you're using.

@kestonsmith-noaa
Copy link
Collaborator Author

kestonsmith-noaa commented Aug 25, 2025 via email

@gspetro-NOAA
Copy link

@kestonsmith-noaa Is there any reason to be concerned about the failing checks on this subcomponent PR? I see it's been approved but just wanted to be sure before we proceed with WM PR #2826.

@JessicaMeixner-NOAA
Copy link
Collaborator

@gspetro-NOAA - are you talking about the CI tests in WW3?

If so, then no - those are not concerning. The CI has been broken for a while and does not reflect anything related to @kestonsmith-noaa's code. I'll try to get that turned off if that's causing confusion.

@BrianCurtis-NOAA
Copy link
Contributor

Are we bumping WW3 to the latest develop or keeping this hash specifically?

@JessicaMeixner-NOAA
Copy link
Collaborator

WW3's "develop" for UFS is dev/ufs-waether-model. This PR should be a branch on top of dev/ufs-weather-model. I will confirm now.

@BrianCurtis-NOAA
Copy link
Contributor

WW3's "develop" for UFS is dev/ufs-waether-model. This PR should be a branch on top of dev/ufs-weather-model. I will confirm now.

Yes, thanks! I realized that too late

@JessicaMeixner-NOAA
Copy link
Collaborator

Confirmed this branch is up to date w/dev/ufs-weather-model ... Always good to double check! Thanks @BrianCurtis-NOAA

@gspetro-NOAA
Copy link

Testing has successfully completed for WM PR #2444, so we can begin the merging process for this subcomponent PR.

@JessicaMeixner-NOAA JessicaMeixner-NOAA merged commit 91c2038 into NOAA-EMC:dev/ufs-weather-model Oct 1, 2025
3 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Netcdf parallel output inconsistencies with WW3 binary output
5 participants