Add distributed mesh support and fake neighbor handling in BreakMeshByBlock #31636

ChengHauYang · 2025-09-30T15:59:24Z

closes #12033
closes #21161

Reason

Currently, BreakMeshByBlock (bmbb) uses a hacky approach to prepare the mesh.
When nodes are duplicated by bmbb, their former neighbors are no longer neighbors (since neighborhood is based on shared nodes). As a result:

We cannot call prepare_for_use() at the end of bmbb, because it would erase the discontinuous neighbor information.
To work around this, the framework has special hacks for bmbb (e.g., skipping mesh validity checks in debug mode when bmbb is present).

Another limitation is that bmbb cannot run with distributed meshes. This PR addresses that.

Design

Preserving discontinuous neighbors
- Since prepare_for_use() wipes out discontinuous neighbor data, we introduce a new data structure to store this information (referred to as fake neighbors).
- These fake neighbors remain accessible to interface kernels that require them (e.g., CZM).
Support for distributed meshes
Following Roy’s suggestions:
- Unique node IDs: In replicated meshes, new nodes were created with
```
Node::build(*current_node, mesh->n_nodes()).release();
```
  This fails in distributed meshes.
  -> Solution: use communication between processors and assign deterministic IDs based on max_node_id strides.
- Ghost node handling:
  - Phase 0: Each rank computes locally connected blocks for its owned nodes and sends this info to the node owner.
  - Phase 1: Node owners aggregate connected block info and broadcast the global set to all ranks.
- Boundary ID assignment: Updated in line with Roy’s fix (commit link).

Impact

bmbb now supports distributed meshes.
prepare_for_use() can be safely called at the end of bmbb.
Removes the need for special hacks in the MOOSE framework for bmbb.

hugary1995

Some early comments.

hugary1995 · 2025-09-30T21:26:05Z

framework/include/loops/ThreadedElementLoopBase.h

+          const Elem * neighbor_ptr = elem->neighbor_ptr(side);
+          const Elem * neighbor =
+              (neighbor_ptr) ? neighbor_ptr : _mesh.neighbor_fake_ptr(elem, side);


Suggested change

const Elem * neighbor_ptr = elem->neighbor_ptr(side);

const Elem * neighbor =

(neighbor_ptr) ? neighbor_ptr : _mesh.neighbor_fake_ptr(elem, side);

const auto * neighbor_ptr = elem->neighbor_ptr(side) || _mesh.neighbor_fake_ptr(elem, side);

Might have to add an assertion below to make sure it's nullptr.

Thank you for pointing this out.

Using || in this context does not work because in C++ it always evaluates to a bool, not a pointer.

As a result, I go with

const auto * neighbor = elem->neighbor_ptr(side) ? elem->neighbor_ptr(side) : _mesh.neighbor_fake_ptr(elem, side);

hugary1995 · 2025-09-30T21:41:58Z

framework/include/mesh/MooseMesh.h

+  void
+  setElemSideToFakeNeighborElemSideMap(std::unordered_map<std::pair<dof_id_type, unsigned int>,
+                                                          std::pair<dof_id_type, unsigned int>> map)
+  {
+    _elemid_side_to_fake_neighbor_elemid_side = std::move(map);
+  }


We need a more fine-grained API to modify the map.

Thanks for the suggestion! I’ve extended the interface to provide a more fine-grained API:

addFakeNeighbor: add or update a single (elem, side) entry

removeFakeNeighbor: remove a single entry if it exists

hasFakeNeighbor: check whether a mapping exists

getFakeNeighborMap: accessor for read-only access to the whole map

setFakeNeighborMap: still available for bulk replacement

framework/include/mesh/MooseMesh.h

hugary1995 · 2025-09-30T21:43:16Z

framework/include/mesh/MooseMesh.h

+    _elemid_side_to_fake_neighbor_elemid_side.clear();
+  }
+
+  const Elem * neighbor_fake_ptr(const Elem * elem, unsigned int side) const


Suggested change

const Elem * neighbor_fake_ptr(const Elem * elem, unsigned int side) const

const Elem * fake_neighbor_ptr(const Elem * elem, unsigned int side) const

Also, perhaps we should come up with a better name than "fake". Perhaps "additional_neighbor_ptr"? I'll take suggestions from @roystgnr and @lindsayad

I currently just rename it to be fake_neighbor_ptr . Let's wait for the suggestions.

hugary1995 · 2025-09-30T21:43:41Z

framework/include/mesh/MooseMesh.h

+      return nullptr;
+  }
+
+  int neighbor_fake_side(const Elem * elem, unsigned int side) const


similar comment on naming.

Change it to be fake_neighbor_side

hugary1995 · 2025-09-30T21:55:27Z

framework/include/meshgenerators/BreakMeshByBlockGenerator.h

+  std::unordered_map<ElemIDSidePair, ElemIDSidePair> _elemid_side_to_fake_neighbor_elemid_side;
+
+  /// generate the new boundary interface
+  void addInterfaceBoundary(MeshBase & mesh);


I suggest renaming this to addInterface.

hugary1995 · 2025-09-30T21:55:37Z

framework/include/meshgenerators/BreakMeshByBlockGenerator.h

+  std::map<dof_id_type, std::set<subdomain_id_type>> _nodeid_to_connected_blocks;
+
+  /// @brief a set of pairs of block ids between which new boundary sides are created
+  std::set<std::pair<subdomain_id_type, subdomain_id_type>> _new_boundary_sides_list;


I prefer the old name _neighboring_block_list. Why was the renaming, is it now serving different purposes?

This name I grabbed from Roy's implementation: roystgnr@2583950. I will use _neighboring_block_list to replace this.

hugary1995 · 2025-09-30T22:04:01Z

framework/include/meshgenerators/BreakMeshByBlockGeneratorBase.h

+  void findBoundaryName(MeshBase & mesh,
+                        const subdomain_id_type & /*primaryBlockID*/,
+                        const subdomain_id_type & /*secondaryBlockID*/,
+                        std::string & /*boundaryName*/,
+                        const boundary_id_type & /*boundaryID*/,
+                        BoundaryInfo & /*boundary_info*/);


Suggested change

void findBoundaryName(MeshBase & mesh,

const subdomain_id_type & /*primaryBlockID*/,

const subdomain_id_type & /*secondaryBlockID*/,

std::string & /*boundaryName*/,

const boundary_id_type & /*boundaryID*/,

BoundaryInfo & /*boundary_info*/);

BoundaryID getBoundaryIDFromBlockPair(MeshBase & mesh,

const subdomain_id_type & /*primaryBlockID*/,

const subdomain_id_type & /*secondaryBlockID*/);

hugary1995 · 2025-09-30T22:09:00Z

framework/include/utils/MooseMeshUtils.h

 // Used to temporarily store information about which lower-dimensional
 // sides to add and what subdomain id to use for the added sides.
-struct ElemSidePair
+struct ElemIDSidePair


See if you are reinventing BndElement...

I actually wrongly changed here. This is the original implementation inside MOOSE. I will revert this back, and on my side, I will try to use BndElement .

hugary1995 · 2025-09-30T22:09:20Z

framework/src/base/MeshGeneratorSystem.C

-  if (type == "BreakMeshByBlockGenerator")
-    _has_bmbb = true;


moosebuild · 2025-10-02T00:55:03Z

Job Documentation, step Docs: sync website on 9f29e32 wanted to post the following:

View the site here

This comment will be updated on new commits.

…hbor RM 1. Distributed mesh fixes - Node ID assignment: using `mesh->n_nodes()` fails for distributed meshes (IDs are sparse). Switch to `max_node_id` with a stride. - Ghost node handling: - *Phase 0*: Each rank computes locally connected blocks for its owned nodes and sends them to the node owner. - *Phase 1*: Node owners aggregate connected block info and broadcast the global set to all ranks. - Boundary ID assignment aligned with Roy’s fix ([commit link](roystgnr@2583950)). 2. New FakeNeighbor datastructure - Introduce a RelationshipManager for ghosted fake neighbors (CZM discontinuous neighbors, not topological). - Robust initialization: - Override `set_mesh` to ensure `_functor` is created JIT on first use. - Rebuild `_functor` in `internalInitWithMesh` if the neighbor map was populated after an early empty init. - Handles mesh cloning (`SetupMeshAction`) cases where `set_mesh` is called before initialization. 3. Simplifications - Since we now call `prepare_for_use()` at the end of BMBb, remove BreakMeshByBlockGenerator-specific hacks. - Simplify node ID handling. - Drop `MeshGeneratorSystem::_has_bmbb` flag and `hasBreakMeshByBlockGenerator()` API. # Conflicts: # framework/src/base/MeshGeneratorSystem.C

Fixes crashes from map::at when nodes had no valid blocks. Each node now gets an entry (even empty), keeping the map consistent with node_to_elem_map.

… nodes

…ributed mode. The crash was caused by an incorrect decision on node duplication, which stemmed from using incomplete, processor-local data (`node_to_elem_map`). The fix ensures the decision is based on globally synchronized data (`_nodeid_to_connected_blocks`), guaranteeing a consistent and correct outcome across all processors.

… node is in a junction with more than two blocks that share the node. It only happens that the node is in the interface of two blocks and has the block IDs equal to what the user set.

…estriction for bmbb tests (2) regold for the bmbb tests

Refactors the disconnected neighbor mechanism to be robust and safe under parallel (MPI) execution. Previously, the implementation stored raw `Elem*` pointers in a map, which was fundamentally unsafe in a distributed environment. Pointers are only valid on the local rank and become meaningless across MPI processes, leading to segmentation faults in multi-process runs. This commit resolves the issue by transitioning to an ID-based design: - Core map (`_disconnected_neighbors_by_id`) now stores `dof_id_type` pairs instead of raw `ConstBndElement` pointers, ensuring portability and serializability across ranks. - The `disconnectedNeighbor` accessor now returns a `std::optional<ConstBndElement>`, replacing unsafe raw pointers with a safer C++ interface that naturally handles missing neighbors. **Key Learning:** For parallel-safe design, always **store globally unique IDs** and resolve them into rank-local pointers, which avoids pointer validity issues and ensures correctness across distributed executions.

- Replaced raw Elem* usage with ID-based ElemSide pairs for safer parallel handling - Added cache (_cached_disconnected_neighbors) to speed up repeated lookups - Modified neighbor queries across loops (NonlinearThread, ComputeUserObjectsThread, ComputeMaterialsObjectThread, FEProblemBase, etc.) to use disconnectedNeighborPtr

1. MooseMesh: Introduced `_has_incomplete_interface_pairs` flag with getter/setter, initialized in the constructor to detect missing reverse boundary pairs. 2. BreakMeshByBlockGenerator: Set `add_interface_on_two_sides = true` and `split_interface = true` by default; detects incomplete boundary pairs and outputs a warning if the user attempts to run CZM with such configurations. 3. CohesiveZoneAction & Tests: Added runtime error for incomplete CZM interfaces; updated test inputs to use explicit block-pair boundaries (replacing the unsupported single `interface` setup) and added the `boundary_pair_not_found` failure test

moosebuild · 2025-10-30T20:07:51Z

Job Precheck, step Clang format on 040fb48 wanted to post the following:

Your code requires style changes.

A patch was auto generated and copied here
You can directly apply the patch by running, in the top level of your repository:

curl -s https://mooseframework.inl.gov/docs/PRs/31636/clang_format/style.patch | git apply -v

Alternatively, with your repository up to date and in the top level of your repository:

git clang-format d12cd78c8f6e9e740dc0feb87483d83d8471f4e7

moosebuild · 2025-10-30T20:07:52Z

libmesh

@@ -1 +1 @@
-Subproject commit 910736a3a20aded528c3ba99371ce65980dc8e2c
+Subproject commit f956256a7b2b96dc0df2852e4f54f506ef20b062


Caution! This contains a submodule update

moosebuild · 2025-10-30T20:07:53Z

Job Precheck, step Versioner verify on 040fb48 wanted to post the following:

Versioner templates

Found 14 templates, 0 failed

Versioner influential files

Found 58 influential files, 1 changed, 0 added, 0 removed

package	status	file
libmesh	CHANGE	libmesh

Versioner versions

Found 9 packages, 2 changed, 2 failed

package	status	hash	to hash	version	to version
libmesh	NEED BUMP	e987a26	948e63c	2025.10.15 build 1	2025.10.15 build 1
moose-dev	NEED BUMP	ed380db	071c6cd	2025.10.27	2025.10.27

Verification failed.

hugary1995 requested changes Sep 30, 2025

View reviewed changes

ChengHauYang force-pushed the break_mesh branch from 690fb02 to b7d4c86 Compare October 1, 2025 21:52

GiudGiud self-assigned this Oct 2, 2025

ChengHauYang force-pushed the break_mesh branch from 0ddb2cf to 9f29e32 Compare October 2, 2025 18:27

hugary1995 mentioned this pull request Oct 14, 2025

Improve Mortar-based cohesive zone modeling capability #31697

Open

ChengHauYang and others added 24 commits October 30, 2025 13:44

Always create map entries for nodes in _nodeid_to_connected_blocks

d2b7a7b

Fixes crashes from map::at when nodes had no valid blocks. Each node now gets an entry (even empty), keeping the map consistent with node_to_elem_map.

revert some changes so that the we do not create unwanted dulplicated…

60a4cee

… nodes

If we set block_pairs in bmbb, the node should not duplicate when the…

d285a7a

… node is in a junction with more than two blocks that share the node. It only happens that the node is in the interface of two blocks and has the block IDs equal to what the user set.

closes idaholab#12033 and idaholab#21161 (1) remove REPLICATED mode r…

abe6ecb

…estriction for bmbb tests (2) regold for the bmbb tests

[WIP] change the neighbor ptr implementation to one line

d716a9f

[WIP] API for fake neighbor

b86cf02

slightly changed the API in MooseMesh related to disconnected neighbors

6c2fcb2

clean the code

be22da9

change data structure

ce6d298

remove the place the do not need disconnectedNeighbor

b1a3564

change libmesh side to reach disconnected neighbor through neighbor_ptr

3d609a9

works for opt mode (not for dbg mode)

c678094

update datastructure in libmesh

27d8842

support distributed meshes

23454e6

update libmesh to use disconnected boundary (libmesh commit: bee28f5)

4b54de1

czm tests do not limit to replicated mesh only

950786a

checkout libmesh

363e382

update unit tests for new framework

ad6d432

update libmesh

040fb48

ChengHauYang force-pushed the break_mesh branch from 9f29e32 to 040fb48 Compare October 30, 2025 19:59

moosebuild reviewed Oct 30, 2025

View reviewed changes

	const Elem * neighbor_fake_ptr(const Elem * elem, unsigned int side) const
	const Elem * fake_neighbor_ptr(const Elem * elem, unsigned int side) const

		@@ -1 +1 @@
		Subproject commit 910736a3a20aded528c3ba99371ce65980dc8e2c
		Subproject commit f956256a7b2b96dc0df2852e4f54f506ef20b062

Add distributed mesh support and fake neighbor handling in BreakMeshByBlock #31636

Are you sure you want to change the base?

Add distributed mesh support and fake neighbor handling in BreakMeshByBlock #31636

Conversation

ChengHauYang commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reason

Design

Impact

Uh oh!

hugary1995 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

moosebuild commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

moosebuild commented Oct 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

moosebuild commented Oct 30, 2025

Versioner templates

Versioner influential files

Versioner versions

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ChengHauYang commented Sep 30, 2025 •

edited

Loading

moosebuild commented Oct 2, 2025 •

edited

Loading