Visibility (scope) of dimensions in a group hierarchy

[ethanrd]

The visibility/scope of dimensions in a group hierarchy changed in early 2021 from being only visible in the local group and child groups to now being visible globally within the netCDF dataset. This change was made in the netCDF-C library and the NUG documentation but not in the netCDF-Java library.

It looks like the netCDF-C change was made in PR #2012 "Regularize the scoping of dimensions". PR #2012 references a bug fix PR related to user-defined types, PR #1959 "Regularize the scoping of types". User-defined types, unlike dimensions, were always defined as globally visible in the NUG.

The change to the NUG was made with PR #39 "Clarify scope rules of dimensions and types". The text changes can be seen in the associated commit, db6fb2.

I have not found any discussion of this conceptual (NUG, maybe data model) change. @DennisHeimbigner, do you remember if there was other discussion around this? Was it done simply to have symmetry between the code dealing with user-defined types and that dealing with dimensions or were there other reasons? Do you remember any discussion of doing this in netCDF-Java as well?

Does anyone know the rationale around the NUG (pre this change) limiting the scope of dimensions to child groups. Perhaps just simplicity? @JohnLCaron, any thoughts? @czender, I know you've done a lot with groups. Any thoughts?

[ethanrd]

Here's an example CDL with a variable with a dimension not defined in the variables group or an ancestor group, /g_b/var_b_2(/g_a/dim_a_1):

netcdf DimScope_example_simple {
  dimensions:
    dim_1 = 1 ;

  group: g_a {
    dimensions:
      dim_a_1 = 1;
  }

  group: g_b {
    dimensions:
      dim_b_1 = 1;
    variables:
      float var_b_1(dim_1);
      float var_b_2(/g_a/dim_a_1);
  }
}

Running it through ncgen successfully produces a nc4 dataset.

NetCDF-Java Problem

As mentioned above, this dimension scoping change did not get propogated to netCDF-Java. As a result, netCDF-Java fails to open a datasets where a variable has a dimension outside of that variables self-ancestor groups. Trying to open the example dataset throws:

java.lang.IllegalArgumentException: Dimension dim_a_1 does not exit

ncgen and ncdump problem

While ncgen can generate example datasets from CDL, running the resulting dataset through ncdump results in CDL with incorrect dimension fully qualified names (FQName). Instead of var_b_2(/g_a/dim_a_1), it generates var_b_2(/dim_a_1).

There is no "/dim_a_1" so running this generated CDL through ncgen again fails with

Undefined or forward referenced dimension: dim_a_1

[czender]

The scope change back in 2021 snuck in without my noticing. This is the first I've heard about it. I do not recall any discussion about it at the time. I am also ignorant as to why dimensions initially only had scope in descendent groups, even though their IDs were global. @edhartnett might recall. This old convention made sense to me because then a file was like a tree of independent branches. You could prune it by removing a branch and that would never affect other branches (siblings or cousins or ...). With the newer convention that does not seem to be true. If you prune a branch where a dimension was defined then you might leave behind another branch that makes use of that dimension, and so you would have to define the dimension you were pruning somewhere in the remant file so dimensions would not be orphaned.

[JohnLCaron]

The CDM chose the simpler tree approach, motivated by the complexity of HDF5 network model, where any given object could have multiple "names" aka paths, to reach it. It also meant that dimensions didnt need to have a "full name". I'm not sure what netcdf-java does, but more recent libraries of mine simply boost the dimensions that are the same up to the common parent. I _think_ that solves the problem, at the cost of the cdl looking different than the C library. CF conventions allow coordinate references to be fully qualified (i.e. be in a different branch), and there may be some devils there.

…

On Thu, Aug 21, 2025 at 8:16 AM Charlie Zender ***@***.***> wrote: The scope change back in 2021 snuck in without my noticing. This is the first I've heard about it. I do not recall any discussion about it at the time. I am also ignorant as to why dimensions initially only had scope in descendent groups, even though their IDs were global. @edhartnett <https://github.com/edhartnett> might recall. This old convention made sense to me because then a file was like a tree of independent branches. You could prune it by removing a branch and that would never affect other branches (siblings or cousins or ...). With the newer convention that does not seem to be true. If you prune a branch where a dimension was defined then you might leave behind another branch that makes use of that dimension, and so you would have to define the dimension you were pruning somewhere in the remant file so dimensions would not be orphaned. — Reply to this email directly, view it on GitHub <#3157 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEVZQD4BM2CLHXHWUYBWQT3OXID5AVCNFSM6AAAAACENLJIIGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTIMJXHE3DEMY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[DennisHeimbigner]

I am not sure what is being discussed.
If you are using the netcdf library API, then there should be no notion of scope at all (with one minor exception).
This is because the API assumes you will reference dimensions (and types for that matter) via a unique ID.
This ID is obtained by the client and can be any dimension in the tree.
Once the client has the dimension id, it can be used to define any variable.

If you are referring to a text representation (e.g. CDL) of a netcdf dataset, then the CDL compiler (i.e. ncgen) has a notion of scope for dimensions that is as described above.
I should note that ncgen also supports fully qualified names for dimensions, so as with the netcdf library, it is possible to define a variable with any specific desired dimension.

[ethanrd]

Hi @DennisHeimbigner - Yes, I'm talking about what dimensions the netCDF-C library allows to be given to a particular variable when the dimension and variable are in different groups.

I agree that global scope for dimensions seems to be the intent of the netCDF-C libraries current behavior (though there are some edge cases that don't seem to behave this way). My question is about why the change was made in 2021 to move dimensions to a global scope away from what the NUG at the time spelled out, "dimensions are scoped such that they are visible to all child groups".

I suspect the nc_inq_dimids() function is a remnant of the old dimension scoping rule given it's include_parents argument allows retrieving either the dimensions in the given group or
the dimensions in the given group and all parent groups.

[JohnLCaron]

Is there an example file where the problem occurs?

…

On Thu, Aug 21, 2025 at 1:17 PM Dennis Heimbigner ***@***.***> wrote: I am not sure what is being discussed. If you are using the netcdf library API, then there should be no notion of scope at all (with one minor exception). This is because the API assumes you will reference dimensions (and types for that matter) via a unique ID. This ID is obtained by the client and can be any dimension in the tree. Once the client has the dimension id, it can be used to define any variable. If you are referring to a text representation (e.g. CDL) of a netcdf dataset, then the CDL compiler (i.e. ncgen) has a notion of scope for dimensions that is as described above. I should note that ncgen also supports fully qualified names for dimensions, so as with the netcdf library, it is possible to define a variable with any specific desired dimension. — Reply to this email directly, view it on GitHub <#3157 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEVZQHMNJSOFKO6TNPGEPL3OYLNPAVCNFSM6AAAAACENLJIIGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTIMJYGIZTIMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ethanrd]

I created an example file using ncgen on the following CDL:

netcdf DimScope_example_SuperSimple {
  dimensions:
    dim_1 = 1 ;

  group: g_a {
    dimensions:
      dim_a_1 = 1;
  }

  group: g_b {
    dimensions:
      dim_b_1 = 1;
    variables:
      float var_b_1(dim_1);
      float var_b_2(/g_a/dim_a_1);
  }
}

ToolsUI fails to open the resulting file throwing IllegalArgumentException: Dimension dim_a_1 does not exist.
The output from ncdump on the resulting file gets the FQName for the dimension in var_b_2 wrong.

[ethanrd]

Here's the resulting .nc4 file, zipped.
DimScope_example_SuperSimple.zip

[JohnLCaron]

I was wrong, my library (netchdf) is barfing on the file also. I see that it's the typedefs I'm pushing up to the common group parent, not the Dimensions. Hmmm. I think the HDF5 group has made a lot of dubious choices over the years, they seem to love complexity for its own sake. Meanwhile the people who understand the api, much less the code, grow older (present company excepted, of course ;^)

…

On Fri, Aug 22, 2025 at 10:50 AM Ethan Davis ***@***.***> wrote: Here's the resulting .nc4 file, zipped. DimScope_example_SuperSimple.zip <https://github.com/user-attachments/files/21941838/DimScope_example_SuperSimple.zip> — Reply to this email directly, view it on GitHub <#3157 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEVZQDMJ33VYL3D4YXEPLT3O5C4JAVCNFSM6AAAAACENLJIIGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTIMJZGE4DSMA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[DennisHeimbigner]

My question is about why the change was made in 2021 to move dimensions to a global scope away from what the NUG at the time spelled out, "dimensions are scoped such that they are visible to all child groups".

I agree with you about the edge case -- nc_inq_dimids. It selects dimensions corresponding to the group+parents search algorithm, But it does not affect the general rule that any dimension in the tree can be used to define any variable in the tree.

Personally, I doubt that it was the case that the netcdf-c library ever enforced the group+parents rule; it would be costly to verify that every dimension used to define a variable was properly scoped to conform to that rule.
The PR #2012 appears to be concerned only with FQNs.
I can, however, believe that the documentation was wrong.

As for the CDL file above, that is probably a bug in ncgen. I will look and fix.

[ethanrd]

Hi @DennisHeimbigner - I was not thinking of nc_inq_dimids() as an edge cases. Here's a simple edge case I found ...

I can write C code to build the following dataset:

netcdf DimScope_SameDimName {
  dimensions:
    dim_1 = 1 ;

  group: g_a {
    dimensions:
      dim_a_1 = 1;
  }

  group: g_b {
    dimensions:
      dim_b_1 = 1;
  }
}

It successfully writes a valid .nc4 file.

If I then add code to create a /var_1(/dim_1) variable, it also successfully writes a valid .nc4 file.

However, if I then add code to create a `/var_2(/g_a/dim_a_1)' variable, it fails.

Here's a gist with my sample code. And here's the failure message:

line 71 of Write_SimpleEdgeCase.c.main: NetCDF: Problem with HDF5 dimscales.

I've tried this with more levels in the group hierarchy. It seems to fail whenever the dimension is at a level in the hierarchy deeper than the variable being defined with that dimension.

[DennisHeimbigner]

I was able to process that .cdl file correctly using the current main.

[DennisHeimbigner]

Dimscales strikes again :-)
I never considered what rules govern
HDF5 dimscales. Have you looked at the HDF5
documentation to see if this is documented?

[lesserwhirls]

Check out this presentation from 2005: NetCDF-4: A New Data Model, Programming
Interface, and Format Using HDF5: Final Project Review. Based on this, it seems pretty clear that from the data model point of view, a deliberate design decision was made such that "Groups are containers that provide hierarchical scopes for variables, dimensions, attributes, and other Groups", a decision perhaps motivated a goal of simplicity (as others have mentioned). The example @ethanrd shows indicates a breakage at the data model level.

[edhartnett]

This has nothing to do with DIMSCALES IIRC. Dims were expressly designed such that dimensions would be visible only in the group they are defined in, and child groups. However, this is simply syntactic sugar - in the data file, each dimension got a unique file-wide ID, and that was the dimid used by the netCDF API. So dimensions were really global but we (intended to) restrict var creation to only use dims in parent groups. However, this restriction was apparently removed, with little harm done, I guess. Thanks for drudging up this 20 year old presentation! I particularly like my timeline: [image: image.png]

…

On Thu, Aug 28, 2025 at 2:03 PM Sean Arms ***@***.***> wrote: Check out this presentation from 2005: NetCDF-4: A New Data Model, Programming Interface, and Format Using HDF5: Final Project Review <https://www.unidata.ucar.edu/presentations/Rew/netcdf-hdf-final.pdf>. Based on this, it seems pretty clear that from the data model point of view, a deliberate design decision was made such that "Groups are containers that provide hierarchical scopes for variables, dimensions, attributes, and other Groups", a decision perhaps motivated a goal of simplicity (as others have mentioned). The example @ethanrd <https://github.com/ethanrd> shows <#3157 (comment)> indicates a breakage at the data model level. — Reply to this email directly, view it on GitHub <#3157 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCSXXBDTBOJURLOHE6XZ333P5N7JAVCNFSM6AAAAACENLJIIGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTIMRUHEYTSMI> . You are receiving this because you were mentioned.Message ID: ***@***.***>