[Question] Question about getting multi rigid contact data to simulate robot skin

[ZhangYi1999]

Hello, everyone. I am currently investigate how to simulate a robot skin module in isaac lab. The robot skin module is consist of multi small contact sensor cell and connected with each other to become a grid or hive shape. Like this,

In this case, each contact cell will have its own force sensor data. According to my understand about isaac sim and isaac lab, I think there are following possible way to simulate it properly:

RigidContactView

Create a omni.isaac.core.prims.RigidBody for every cell, connected them with UsdPhysics.FixedJoint, and use a omni.isaac.core.prims.RigidContactView to query all cell in all environment at the same time. This is the same idea as ContactSensor from omni.isaac.lab.

However, this method doesn't work because when the amount is getting higher, it gives this error message.

2024-08-22 12:25:15 [25,841ms] [Error] [omni.physx.plugin] PhysX error: PhysX Internal CUDA error. Simulation cannot continue! Error code 700!
, FILE /builds/omniverse/physics/physx/source/physx/src/NpScene.cpp, LINE 3059
2024-08-22 12:25:15 [25,841ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2024-08-22 12:25:15 [25,841ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../extensions/runtime/source/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 152
2024-08-22 12:25:15 [25,841ms] [Error] [omni.physx.tensors.plugin] CUDA context validation failed
2024-08-22 12:25:15 [25,841ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../extensions/runtime/source/omni.physx.tensors/plugins/gpu/GpuSimulationData.cpp: 199
2024-08-22 12:25:15 [25,841ms] [Error] [omni.physx.tensors.plugin] Failed to initialize GPU simulation data
Traceback (most recent call last):
  File "/home/yi/Program/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/assets/asset_base.py", line 92, in <lambda>
    lambda event, obj=weakref.proxy(self): obj._initialize_callback(event),
  File "/home/yi/Program/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/assets/asset_base.py", line 263, in _initialize_callback
    self._initialize_impl()
  File "/home/yi/Program/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/assets/articulation/articulation.py", line 946, in _initialize_impl
    self._physics_sim_view = physx.create_simulation_view(self._backend)
  File "/home/yi/.local/share/ov/pkg/isaac-sim-4.1.0/extsPhysics/omni.physics.tensors/omni/physics/tensors/impl/api.py", line 12, in create_simulation_view
    raise Exception("Failed to create simulation view backend")
Exception: Failed to create simulation view backend
Traceback (most recent call last):
  File "/home/yi/Program/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sensors/sensor_base.py", line 64, in <lambda>
    lambda event, obj=weakref.proxy(self): obj._initialize_callback(event),
  File "/home/yi/Program/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sensors/sensor_base.py", line 271, in _initialize_callback
    self._initialize_impl()
  File "/home/yi/Program/isaac_skin_simulation/exts/isaac_skin/isaac_skin/sensors/skin/skin.py", line 191, in _initialize_impl
    super()._initialize_impl()
  File "/home/yi/Program/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sensors/sensor_base.py", line 225, in _initialize_impl
    self._is_outdated = torch.ones(self._num_envs, dtype=torch.bool, device=self._device)
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2024-08-22 12:25:21 [31,781ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-08-22 12:25:21 [31,781ms] [Error] [carb.graphics-vulkan.plugin] vkCreateRayTracingPipelinesKHR failed.
2024-08-22 12:25:21 [31,781ms] [Error] [carb.graphics-vulkan.plugin] No Collection/Library can be nullptr with eCollection as compileType.
2024-08-22 12:25:21 [31,781ms] [Error] [gpu.foundation.plugin] Failed to create the raytracing pipeline.

I think it could be the issue of creating too many rigid body in an inefficient way.

Callback function

Instead of creat a RigidBody for each cell, we can also create only one RigidBody for the whole group and consider each cell as an individual Collider under the overall rigid body.

Use a customed callback function, which can get the detailed collider information like the following code. This code comes from the example in ContactReportDemo and ContactReportDirectAPIDemo from omni.physx.demos

    def update(self, stage, dt, viewport, physxIFace):
        if not self._contact_report_sub:
            if self.use_full_report:
                self._contact_report_sub = get_physx_simulation_interface().subscribe_full_contact_report_events(self._on_full_contact_report_event)
            else:
                self._contact_report_sub = get_physx_simulation_interface().subscribe_contact_report_events(self._on_contact_report_event)
        
    def _on_contact_report_event(self, contact_headers, contact_data):
        self.print_contacts(contact_headers, contact_data, None)
                        
    def _on_full_contact_report_event(self, contact_headers, contact_data, friction_anchors):
        self.print_contacts(contact_headers, contact_data, friction_anchors)

    def print_contacts(self, contact_headers, contact_data, friction_anchors):
        for contact_header in contact_headers:
            print("Got contact header type: " + str(contact_header.type))
            print("Actor0: " + str(PhysicsSchemaTools.intToSdfPath(contact_header.actor0)))
            print("Actor1: " + str(PhysicsSchemaTools.intToSdfPath(contact_header.actor1)))
            print("Collider0: " + str(PhysicsSchemaTools.intToSdfPath(contact_header.collider0)))
            print("Collider1: " + str(PhysicsSchemaTools.intToSdfPath(contact_header.collider1)))
            print("StageId: " + str(contact_header.stage_id))
            print("Number of contacts: " + str(contact_header.num_contact_data))
            
            contact_data_offset = contact_header.contact_data_offset
            num_contact_data = contact_header.num_contact_data
            
            for index in range(contact_data_offset, contact_data_offset + num_contact_data, 1):
                print("Contact:")
                print("Contact position: " + str(contact_data[index].position))
                print("Contact normal: " + str(contact_data[index].normal))
                print("Contact impulse: " + str(contact_data[index].impulse))
                print("Contact separation: " + str(contact_data[index].separation))
                print("Contact faceIndex0: " + str(contact_data[index].face_index0))
                print("Contact faceIndex1: " + str(contact_data[index].face_index1))
                print("Contact material0: " + str(PhysicsSchemaTools.intToSdfPath(contact_data[index].material0)))
                print("Contact material1: " + str(PhysicsSchemaTools.intToSdfPath(contact_data[index].material1)))

            if friction_anchors is not None:
                friction_anchors_data_offset = contact_header.friction_anchors_offset
                num_friction_anchor_data = contact_header.num_friction_anchors_data

                for index in range(friction_anchors_data_offset, friction_anchors_data_offset + num_friction_anchor_data, 1):
                    print("Friction anchor:")
                    print("Friction position: " + str(friction_anchors[index].position))
                    print("Friction impulse: " + str(friction_anchors[index].impulse))

However, this way doesn't make much sense since it should query the information one by one, which means if you have 10 env and each env has 100 cell, you need to run this call function 1000 times. What's more, I am not sure if this call function runs on cpu or gpu, and will it effect the preformance of RL or not.

Calculate the cell assignment manually

Since RigidContactView can also report contact position, it will be possible to use the contact position to determine which cell is collided. However, this could increase the computation complexity.

Question

My question is, is there a better way to do this or can we use any of the above idea but do it efficiently. Btw, is it possible that the get_contact_force_data function in RigidContactView return collider information as well?
If anyone could give same advice, I will be super grateful. Thanks in advance!

[RandomOakForest]

Thanks for posting this. Our apologies we haven't been able to reply earlier. I'll move this to a discussion for the team to follow up upon.

[YuHoChau]

I have the same question here. Any updates so far? Thanks.