Add tracing instrumentation to nuopc driver

[DusanJovic-NOAA]

For detailed information about submitting Pull Requests (PRs) to the CICE-Consortium,
please refer to: https://github.com/CICE-Consortium/About-Us/wiki/Resource-Index#information-for-developers

PR checklist

• Short (1 sentence) summary of your PR:
  Add tracing instrumentation to nuopc driver
• Developer(s):
  ENTER INFORMATION HERE
• Suggest PR reviewers from list in the column to the right.
• Please copy the PR test results link or provide a summary of testing completed below.
  ENTER INFORMATION HERE
• How much do the PR code changes differ from the unmodified code?
  • bit for bit
  • different at roundoff level
  • more substantial
• Does this PR create or have dependencies on Icepack or any other models?
  • Yes
  • No
• Does this PR update the Icepack submodule? If so, the Icepack submodule must point to a hash on Icepack's main branch.
  • Yes
  • No
• Does this PR add any new test cases?
  • Yes
  • No
• Is the documentation being updated? ("Documentation" includes information on the wiki or in the .rst files from doc/source/, which are used to create the online technical docs at https://readthedocs.org/projects/cice-consortium-cice/. A test build of the technical docs will be performed as part of the PR testing.)
  • Yes
  • No, does the documentation need to be updated at a later time?
    • Yes
    • No
• Please document the changes in detail, including why the changes are made. This will become part of the PR commit log.

This PR adds calls to ufs tracing routines that will create a trace file which can then be visualized, which is found to be useful in identifying various performance issues.

See ufs-weather-model issue ufs-community/ufs-weather-model#2883

Added calls are not used by default, unless build option (-DUFS_TRACING=ON) is specified when building the ufs-weather-model.

[NickSzapiro-NOAA]

@DusanJovic-NOAA CICE has a my_task == master_task construct instead of the mype==0 you have here. Does that still work for you?

[DusanJovic-NOAA]

@DusanJovic-NOAA CICE has a my_task == master_task construct instead of the mype==0 you have here. Does that still work for you?

my_task and master_task are available only after call init_communicate(lmpicom) in InitializeAdvertise. Which is too late, I wanted to start tracing as early as possible in SetServices. SetServices and InitializeP0 are probably not very expensive, but just to be as consistent as possible with other components I wanted to start tracing in all component at the same time, the beginning of SetServices.

[DeniseWorthen]

@DusanJovic-NOAA Same question as for WW3---could you put this in the wrapper mod and avoid the ifdefs in the cap?

[DusanJovic-NOAA]

@DusanJovic-NOAA Same question as for WW3---could you put this in the wrapper mod and avoid the ifdefs in the cap?

OK, I tried adding the stub version of the 'ufs_trace_mod' in cice_wrapper_mod.F90, like this:

diff --git a/cicecore/drivers/nuopc/cmeps/cice_wrapper_mod.F90 b/cicecore/drivers/nuopc/cmeps/cice_wrapper_mod.F90
index db0140b..7532791 100644
--- a/cicecore/drivers/nuopc/cmeps/cice_wrapper_mod.F90
+++ b/cicecore/drivers/nuopc/cmeps/cice_wrapper_mod.F90
@@ -101,3 +101,28 @@ end subroutine t_barrierf
 #endif

 end module cice_wrapper_mod
+
+#ifndef UFS_TRACING
+
+module ufs_trace_mod
+
+  implicit none
+  public ufs_trace_init
+  public ufs_trace
+  public ufs_trace_finalize
+
+contains
+
+  subroutine ufs_trace_init()
+  end subroutine ufs_trace_init
+
+  subroutine ufs_trace(component, routine, ph)
+    character(len=*), intent(in) :: component, routine, ph
+  end subroutine ufs_trace
+
+  subroutine ufs_trace_finalize()
+  end subroutine ufs_trace_finalize
+
+end module ufs_trace_mod
+
+#endif

And removed all #ifdef UFS_TRACING from ice_comp_nuopc.F90.

Compilation, actually linking, fails with "'multiple definition of `ufs_trace_mod._'" error.

ld: CICE-interface/libcice.a(cice_wrapper_mod.F90.o): in function `ufs_trace_mod_mp_ufs_trace_finalize_':
/scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/ufs/ufs_tracing/ufs-weather-model/CICE-interface/CICE/cicecore/drivers/nuopc/cmeps/cice_wrapper_mod.F90:124: multiple definition of `ufs_trace_mod_mp_ufs_trace_finalize_'; MOM6-interface/libmom6.a(ufs_trace_mod.F90.o):/scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/ufs/ufs_tracing/ufs-weather-model/MOM6-interface/MOM6/config_src/external/ufs_tracing/ufs_trace_mod.F90:18: first defined here
ld: CICE-interface/libcice.a(cice_wrapper_mod.F90.o): in function `ufs_trace_mod_mp_ufs_trace_':
/scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/ufs/ufs_tracing/ufs-weather-model/CICE-interface/CICE/cicecore/drivers/nuopc/cmeps/cice_wrapper_mod.F90:121: multiple definition of `ufs_trace_mod_mp_ufs_trace_'; MOM6-interface/libmom6.a(ufs_trace_mod.F90.o):/scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/ufs/ufs_tracing/ufs-weather-model/MOM6-interface/MOM6/config_src/external/ufs_tracing/ufs_trace_mod.F90:15: first defined here
ld: CICE-interface/libcice.a(cice_wrapper_mod.F90.o): in function `ufs_trace_mod_mp_ufs_trace_init_':
/scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/ufs/ufs_tracing/ufs-weather-model/CICE-interface/CICE/cicecore/drivers/nuopc/cmeps/cice_wrapper_mod.F90:117: multiple definition of `ufs_trace_mod_mp_ufs_trace_init_'; MOM6-interface/libmom6.a(ufs_trace_mod.F90.o):/scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/ufs/ufs_tracing/ufs-weather-model/MOM6-interface/MOM6/config_src/external/ufs_tracing/ufs_trace_mod.F90:11: first defined here
ld: CICE-interface/libcice.a(cice_wrapper_mod.F90.o): in function `ufs_trace_mod._':
/scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/ufs/ufs_tracing/ufs-weather-model/CICE-interface/CICE/cicecore/drivers/nuopc/cmeps/cice_wrapper_mod.F90:107: multiple definition of `ufs_trace_mod._'; MOM6-interface/libmom6.a(ufs_trace_mod.F90.o):/scratch3/NCEPDEV/fv3-cam/Dusan.Jovic/ufs/ufs_tracing/ufs-weather-model/MOM6-interface/MOM6/config_src/external/ufs_tracing/ufs_trace_mod.F90:1: first defined here
make[2]: *** [CMakeFiles/ufs_model.dir/build.make:163: ufs_model] Error 1
make[1]: *** [CMakeFiles/Makefile2:796: CMakeFiles/ufs_model.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

As I expected, see my comment in NOAA-EMC/WW3#1494 (comment)

[DusanJovic-NOAA]

Instead of adding the stub module in the cap, as I tried above, we could add a wrapper module, as we did in CMEPS. The wrapper module will still have ifdefs, while the cap will just call the wrapper routines without ifdefs.