想问一下rootless服务器能运行auto data collection吗

[Dwl2021]

尝试使用rootless docker，但是脚本似乎依然有sudo的需求

[Dwl2021]

请求管理员把所有文件夹的权限都开了，并且curobo确定是已经跑起来了，但是在采集数据的时候遇到了，server端一系列报错，以至于只保存下了全空的mcap文件。具体报错如下：

2026-03-23T15:27:56Z [1,713ms] [Fatal] [carb.crashreporter-breakpad.plugin] 030: _carb.cpython-311-x86_64-linux-gnu.so!std::__detail::_Map_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, void*>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, void*> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&)+0xdb19 (??:?)
2026-03-23T15:27:56Z [1,793ms] [Fatal] [carb.crashreporter-breakpad.plugin] 031: libpython3.11.so.1.0!_PyObject_MakeTpCall+0x95 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/./Include/internal/pycore_ceval.h:123)
2026-03-23T15:27:57Z [1,968ms] [Fatal] [carb.crashreporter-breakpad.plugin] 032: libpython3.11.so.1.0!_PyEval_EvalFrameDefault+0x6cae (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Python/ceval.c:4769)
2026-03-23T15:27:57Z [2,160ms] [Fatal] [carb.crashreporter-breakpad.plugin] 033: libpython3.11.so.1.0!_PyEval_Vector+0xba (/builds/omniverse/externals/python-build/_build/Python-3.11.13/./Include/internal/pycore_ceval.h:73)
2026-03-23T15:27:57Z [2,254ms] [Fatal] [carb.crashreporter-breakpad.plugin] 034: libpython3.11.so.1.0!_PyObject_FastCallDictTstate+0xb9 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Objects/call.c:154)
2026-03-23T15:27:57Z [2,346ms] [Fatal] [carb.crashreporter-breakpad.plugin] 035: libpython3.11.so.1.0!_PyObject_Call_Prepend+0xe4 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Objects/call.c:489)
2026-03-23T15:27:57Z [2,498ms] [Fatal] [carb.crashreporter-breakpad.plugin] 036: libpython3.11.so.1.0!slot_tp_init+0xcc (/builds/omniverse/externals/python-build/_build/Python-3.11.13/./Include/object.h:537)
2026-03-23T15:27:57Z [2,652ms] [Fatal] [carb.crashreporter-breakpad.plugin] 037: libpython3.11.so.1.0!type_call+0x91 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Objects/typeobject.c:1104)
2026-03-23T15:27:57Z [2,745ms] [Fatal] [carb.crashreporter-breakpad.plugin] 038: libpython3.11.so.1.0!_PyObject_MakeTpCall+0x95 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/./Include/internal/pycore_ceval.h:123)
2026-03-23T15:27:57Z [2,933ms] [Fatal] [carb.crashreporter-breakpad.plugin] 039: libpython3.11.so.1.0!_PyEval_EvalFrameDefault+0x6cae (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Python/ceval.c:4769)
2026-03-23T15:27:58Z [3,122ms] [Fatal] [carb.crashreporter-breakpad.plugin] 040: libpython3.11.so.1.0!PyEval_EvalCode+0x4ba (/builds/omniverse/externals/python-build/_build/Python-3.11.13/./Include/internal/pycore_ceval.h:73)
2026-03-23T15:27:58Z [3,343ms] [Fatal] [carb.crashreporter-breakpad.plugin] 041: libpython3.11.so.1.0!run_mod+0xa5 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Python/pythonrun.c:1742)
2026-03-23T15:27:58Z [3,571ms] [Fatal] [carb.crashreporter-breakpad.plugin] 042: libpython3.11.so.1.0!_PyRun_SimpleFileObject+0x130 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Python/pythonrun.c:1662)
2026-03-23T15:27:58Z [3,795ms] [Fatal] [carb.crashreporter-breakpad.plugin] 043: libpython3.11.so.1.0!_PyRun_AnyFileObject+0x3c (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Python/pythonrun.c:79)
2026-03-23T15:27:59Z [4,042ms] [Fatal] [carb.crashreporter-breakpad.plugin] 044: libpython3.11.so.1.0!Py_RunMain+0x793 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Modules/main.c:361)
2026-03-23T15:27:59Z [4,296ms] [Fatal] [carb.crashreporter-breakpad.plugin] 045: libpython3.11.so.1.0!Py_BytesMain+0x43 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Modules/main.c:739)
2026-03-23T15:27:59Z [4,349ms] [Fatal] [carb.crashreporter-breakpad.plugin] 046: libc.so.6!__libc_init_first+0x8a (??:?)
2026-03-23T15:27:59Z [4,383ms] [Fatal] [carb.crashreporter-breakpad.plugin] 047: libc.so.6!__libc_start_main+0x8b (??:?)
2026-03-23T15:27:59Z [4,389ms] [Fatal] [carb.crashreporter-breakpad.plugin] 048: python3!_start+0x29 (??:?)
/isaac-sim/python.sh: line 73:  9057 Aborted                 (core dumped) $python_exe "${filtered_args[@]}" $args

[MiaoMieu]

这个看起来是isaacsim 报错了 可以给出更多日志吗？ 或者先确定容器内直接启动isaacsim 会不会报错？

[CookieYang]

2026-03-26T09:00:40Z [271,871ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2026-03-26T09:00:40Z [271,871ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2026-03-26T09:00:40Z [271,871ms] [Error] [gpu.foundation.plugin] Failed to allocate memory for resource allocator Foundation Resource Allocator
2026-03-26T09:00:40Z [271,871ms] [Error] [gpu.foundation.plugin] Failed to create a heap for pool Foundation Resource Allocator memory pool
2026-03-26T09:00:40Z [271,871ms] [Error] [gpu.foundation.plugin] Releasing memory that isn't owned by the sub-allocator Foundation Resource Allocator
2026-03-26T09:00:40Z [271,871ms] [Error] [carb.graphics-vulkan.plugin] Out of GPU memory allocating resource 'Foundation Resource Allocator' [size: unknown]
2026-03-26T09:00:40Z [271,871ms] [Error] [carb.graphics-vulkan.plugin] Failure injector rule to repro:
{
debugName="Foundation Resource Allocator",
}
2026-03-26T09:00:40Z [271,871ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2026-03-26T09:00:40Z [271,871ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2026-03-26T09:00:40Z [271,871ms] [Error] [gpu.foundation.plugin] Failed to allocate memory for resource allocator Foundation Resource Allocator
2026-03-26T09:00:40Z [271,871ms] [Error] [gpu.foundation.plugin] Failed to create a heap for pool Foundation Resource Allocator memory pool
2026-03-26T09:00:40Z [271,871ms] [Error] [gpu.foundation.plugin] Releasing memory that isn't owned by the sub-allocator Foundation Resource Allocator
2026-03-26T09:00:41Z [0ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] A crash has occurred. If a debugger should be attached, please set the '/crashreporter/debuggerAttachTimeoutMs' setting to a timeout in milliseconds. This can be used to allow the crash reporter to wait for up to that long for a debugger to attach before processing or sending the crash report.
2026-03-26T09:00:41Z [72ms] [Error] [carb.crashreporter-breakpad.plugin] [crash] Wrote dump file '/isaac-sim/kit/data/Kit/Isaac-Sim Python/5.1/300b4464-ea22-4199-5c9207a2-8c564852.dmp'
2026-03-26T09:00:41Z [76ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] preventing upload of minidump due to user opt-out: '/isaac-sim/kit/data/Kit/Isaac-Sim Python/5.1/300b4464-ea22-4199-5c9207a2-8c564852.dmp'
2026-03-26T09:00:41Z [79ms] [Error] [carb.crashreporter-breakpad.plugin] [crash] dump file size is 6356360 bytes, file is readable.
2026-03-26T09:00:41Z [89ms] [Fatal] [carb.crashreporter-breakpad.plugin] [crash] Crash detected in pid 139 thread 139
2026-03-26T09:00:41Z [92ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] Crash metadata:
2026-03-26T09:00:41Z [96ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] BuildGitlabJobID = '209507876'
2026-03-26T09:00:41Z [99ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] CarbSdkVersion = '206.6+release.9587.07f17b1b.gl'
2026-03-26T09:00:41Z [102ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] CrashTime = 'Thu Mar 26 09:00:41 2026 GMT'
2026-03-26T09:00:41Z [106ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] DriverShaderCacheWrapper = 'disabled'
2026-03-26T09:00:41Z [109ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] DumpId = '300b4464-ea22-4199-5c9207a2-8c564852'
2026-03-26T09:00:41Z [112ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] LastUploadStatus = '0'
2026-03-26T09:00:41Z [116ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] ProductName = 'OmniverseKit'
2026-03-26T09:00:41Z [119ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] PythonTracebackStatus = ''py-spy' successfully wrote info to '/isaac-sim/kit/data/Kit/Isaac-Sim Python/5.1/300b4464-ea22-4199-5c9207a2-8c564852.py.txt''
2026-03-26T09:00:41Z [123ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] RetryCount = '0'
2026-03-26T09:00:41Z [126ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] StartupTime = '1774515368'
2026-03-26T09:00:41Z [129ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] UploadSuccessful = '0'
2026-03-26T09:00:41Z [133ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] UptimeSeconds = '273'
2026-03-26T09:00:41Z [136ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] UserStory = ''
2026-03-26T09:00:41Z [139ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] UserStoryStatus = 'Running in headless mode; not gathering user story'
2026-03-26T09:00:41Z [143ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] Version = '107.3.3+production.229672.69cbf6ad.gl'
2026-03-26T09:00:41Z [146ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] aftermath_enabled = '1'
2026-03-26T09:00:41Z [149ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] aftermath_flags = '3'
2026-03-26T09:00:41Z [153ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] aftermath_status = 'auto-enabled'
2026-03-26T09:00:41Z [156ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] aftermath_version = '2024.1'
2026-03-26T09:00:41Z [159ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] appName = 'Isaac-Sim Python'
2026-03-26T09:00:41Z [162ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] appState = 'started'
2026-03-26T09:00:41Z [165ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] appVersion = '5.1.0'
2026-03-26T09:00:41Z [168ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] autoloadExts = ''
2026-03-26T09:00:41Z [171ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildBranch = 'production'
2026-03-26T09:00:41Z [174ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildCi = 'gl'
2026-03-26T09:00:41Z [176ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildConfig = 'release'
2026-03-26T09:00:41Z [179ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildGitlabJobName = 'kit-build-release-linux-x86_64'
2026-03-26T09:00:41Z [182ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildGitlabJobStage = 'kit-build'
2026-03-26T09:00:41Z [185ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildGitlabPipelineID = '34875094'
2026-03-26T09:00:41Z [188ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildHash = '69cbf6ad'
2026-03-26T09:00:41Z [191ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildMajor = '107'
2026-03-26T09:00:41Z [194ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildMinor = '3'
2026-03-26T09:00:41Z [197ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildNumber = '229672'
2026-03-26T09:00:41Z [200ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildPatch = '3'
2026-03-26T09:00:41Z [203ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildVersion = '107.3.3'
2026-03-26T09:00:41Z [206ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] carboniteFrameworkVersion = '206.6+release.9587.07f17b1b.gl'
2026-03-26T09:00:41Z [209ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] carboniteSdkVersion = '206.6+release.9587.07f17b1b.gl'
2026-03-26T09:00:41Z [212ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] commandLine = '/isaac-sim/kit/python/bin/python3 scripts/data_collector_server.py --enable_physics --enable_curobo --headless --publish_ros'
2026-03-26T09:00:41Z [215ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] cpuCoreLimited = '64'
2026-03-26T09:00:41Z [218ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] cpuId = 'Intel64 Family 6 Model 207 Stepping 2'
2026-03-26T09:00:41Z [221ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] cpuLogicalCoresBareMetal = '64'
2026-03-26T09:00:41Z [224ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] cpuName = 'INTEL(R) XEON(R) GOLD 6526Y'
2026-03-26T09:00:41Z [227ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] cpuPhysicalCoresBareMetal = '32'
2026-03-26T09:00:41Z [230ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] cpuUsageQuota = '-1.000000'
2026-03-26T09:00:41Z [233ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] cpuVendor = 'GenuineIntel'
2026-03-26T09:00:41Z [236ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] environmentName = 'Individual'
2026-03-26T09:00:41Z [239ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] externalBuild = '1'
2026-03-26T09:00:41Z [242ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] extraExts = 'isaacsim.ros2.bridge-4.12.4'
2026-03-26T09:00:41Z [245ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] gpuDriver_0 = '580.126'
2026-03-26T09:00:41Z [248ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] gpuOutOfMemory = 'oom'
2026-03-26T09:00:41Z [251ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] gpuVRAM_0 = '26015170560'
2026-03-26T09:00:41Z [254ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] gpu_0 = 'NVIDIA GeForce RTX 4090 D'
2026-03-26T09:00:41Z [258ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] kitRendererDriverVersion = '580.126'
2026-03-26T09:00:41Z [261ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lastCommand = 'CreateNodeCommand(graph=?,node_path=/Replicator/SDGPipeline/OgnWriteSemantics_10,node_type=omni.replicator.core.OgnWriteSemantics,create_usd=True)'
2026-03-26T09:00:41Z [265ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lastCommands = 'CreateAttrCommand(node=?,attr_name=state:sdp_intergraph_downstream_node_handles,attr_type=?,attr_port=?,attr_default=?,attr_extended_type=?),CreateNodeCommand(graph=?,node_path=/Replicator/SDGPipeline/OgnGetPrims_08,node_type=omni.replicator.core.OgnGetPrims,create_usd=True),CreateNodeCommand(graph=?,node_path=/Replicator/SDGPipeline/OnImpulseEvent_08,node_type=omni.graph.action.OnImpulseEvent,create_usd=True),CreateNodeCommand(graph=?,node_path=/Replicator/SDGPipeline/OgnWriteSemantics_08,node_type=omni.replicator.core.OgnWriteSemantics,create_usd=True),CreateNodeCommand(graph=?,node_path=/Replicator/SDGPipeline/OgnGetPrims_09,node_type=omni.replicator.core.OgnGetPrims,create_usd=True),CreateNodeCommand(graph=?,node_path=/Replicator/SDGPipeline/OnImpulseEvent_09,node_type=omni.graph.action.OnImpulseEvent,create_usd=True),CreateNodeCommand(graph=?,node_path=/Replicator/SDGPipeline/OgnWriteSemantics_09,node_type=omni.replicator.core.OgnWriteSemantics,create_usd=True),CreateNodeCommand(graph=?,node_path=/Replicator/SDGPipeline/OgnGetPrims_10,node_type=omni.replicator.core.OgnGetPrims,create_usd=True),CreateNodeCommand(graph=?,node_path=/Replicator/SDGPipeline/OnImpulseEvent_10,node_type=omni.graph.action.OnImpulseEvent,create_usd=True),CreateNodeCommand(graph=?,node_path=/Replicator/SDGPipeline/OgnWriteSemantics_10,node_type=omni.replicator.core.OgnWriteSemantics,create_usd=True)'
2026-03-26T09:00:41Z [268ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_isaacSim_buildBranch = 'release'
2026-03-26T09:00:41Z [272ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_isaacSim_buildDate = 'Fri Oct 17 03:55:10 2025 +0000'
2026-03-26T09:00:41Z [275ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_isaacSim_buildHash = '9c81211'
2026-03-26T09:00:41Z [278ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_isaacSim_buildRepo = 'https://gitlab-master.nvidia.com/omniverse/isaac/omni_isaac_sim.git'
2026-03-26T09:00:41Z [282ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_isaacSim_buildVersion = '5.1.0-rc.19'
2026-03-26T09:00:41Z [285ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_physx_buildBranch = 'HEAD'
2026-03-26T09:00:41Z [289ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_physx_buildDate = 'Oct-03-2025'
2026-03-26T09:00:41Z [292ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_physx_buildHash = '3a61992'
2026-03-26T09:00:41Z [295ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_physx_buildRepo = 'gitlab-master.nvidia.com/omniverse/physics'
2026-03-26T09:00:41Z [299ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_physx_buildVersion = '107.3.26'
2026-03-26T09:00:41Z [302ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] memoryStats = '(avail/total) RAM: 96.259/125.563GB, Swap: 1.99975/2GB, VM: 1.71799e+10/1.71799e+10GB'
2026-03-26T09:00:41Z [305ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] nvidia-smi = '
==============NVSMI LOG==============

Timestamp : Thu Mar 26 08:56:09 2026
Driver Version : 580.126.09
CUDA Version : 13.0

Attached GPUs : 1
GPU 00000000:3D:00.0
Product Name : NVIDIA GeForce RTX 4090 D
Product Brand : GeForce
Product Architecture : Ada Lovelace
Display Mode : Requested functionality has been deprecated
Display Attached : No
Display Active : Disabled
Persistence Mode : Disabled
Addressing Mode : None
MIG Mode
Current : N/A
Pending : N/A
Accounting Mode : Disabled
Accounting Mode Buffer Size : 4000
Driver Model
Current : N/A
Pending : N/A
Serial Number : N/A
GPU UUID : GPU-ca498c7b-431b-d498-a669-abb22dcf4358
GPU PDI : 0xe3d1da28eb4d283b
Minor Number : 0
VBIOS Version : 95.02.74.00.0C
MultiGPU Board : No
Board ID : 0x3d00
Board Part Number : N/A
GPU Part Number : 2685-250-A1
FRU Part Number : N/A
Platform Info
Chassis Serial Number : N/A
Slot Number : N/A
Tray Index : N/A
Host ID : N/A
Peer Type : N/A
Module Id : 1
GPU Fabric GUID : N/A
Inforom Version
Image Version : G002.0000.00.03
OEM Object : 2.0
ECC Object : 6.16
Power Management Object : N/A
Inforom BBX Object Flush
Latest Timestamp : N/A
Latest Duration : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU C2C Mode : N/A
GPU Virtualization Mode
Virtualization Mode : None
Host VGPU Mode : N/A
vGPU Heterogeneous Mode : N/A
GPU Recovery Action : None
GSP Firmware Version : 580.126.09
IBMNPU
Relaxed Ordering Mode : N/A
PCI
Bus : 0x3D
Device : 0x00
Domain : 0x0000
Base Classcode : 0x3
Sub Classcode : 0x0
Device Id : 0x268510DE
Bus Id : 00000000:3D:00.0
Sub System Id : 0x169710DE
GPU Link Info
PCIe Generation
Max : 4
Current : 4
Device Current : 4
Device Max : 4
Host Max : 5
Link Width
Max : 16x
Current : 8x
Bridge Chip
Type : N/A
Firmware : N/A
Replays Since Reset : 0
Replay Number Rollovers : 0
Tx Throughput : 341 KB/s
Rx Throughput : 341 KB/s
Atomic Caps Outbound : N/A
Atomic Caps Inbound : N/A
Fan Speed : 30 %
Performance State : P2
Clocks Event Reasons
Idle : Not Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
HW Thermal Slowdown : Not Active
HW Power Brake Slowdown : Not Active
Sync Boost : Not Active
SW Thermal Slowdown : Not Active
Display Clock Setting : Not Active
Clocks Event Reasons Counters
SW Power Capping : 34205 us
Sync Boost : 0 us
SW Thermal Slowdown : 0 us
HW Thermal Slowdown : 0 us
HW Power Braking : 0 us
Sparse Operation Mode : N/A
FB Memory Usage
Total : 24564 MiB
Reserved : 484 MiB
Used : 5340 MiB
Free : 18742 MiB
BAR1 Memory Usage
Total : 256 MiB
Used : 7 MiB
Free : 249 MiB
Conf Compute Protected Memory Usage
Total : 0 MiB
Used : 0 MiB
Free : 0 MiB
Compute Mode : Default
Utilization
GPU : 42 %
Memory : 15 %
Encoder : 0 %
Decoder : 0 %
JPEG : 0 %
OFA : 0 %
Encoder Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
FBC Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
DRAM Encryption Mode
Current : N/A
Pending : N/A
ECC Mode
Current : Disabled
Pending : Disabled
ECC Errors
Volatile
SRAM Correctable : N/A
SRAM Uncorrectable Parity : N/A
SRAM Uncorrectable SEC-DED : N/A
DRAM Correctable : N/A
DRAM Uncorrectable : N/A
Aggregate
SRAM Correctable : N/A
SRAM Uncorrectable Parity : N/A
SRAM Uncorrectable SEC-DED : N/A
DRAM Correctable : N/A
DRAM Uncorrectable : N/A
SRAM Threshold Exceeded : N/A
Aggregate Uncorrectable SRAM Sources
SRAM L2 : N/A
SRAM SM : N/A
SRAM Microcontroller : N/A
SRAM PCIE : N/A
SRAM Other : N/A
Channel Repair Pending : No
TPC Repair Pending : No
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending Page Blacklist : N/A
Remapped Rows
Correctable Error : 0
Uncorrectable Error : 0
Pending : No
Remapping Failure Occurred : No
Bank Remap Availability Histogram
Max : 192 bank(s)
High : 0 bank(s)
Partial : 0 bank(s)
Low : 0 bank(s)
None : 0 bank(s)
Temperature
GPU Current Temp : 51 C
GPU T.Limit Temp : 33 C
GPU Shutdown T.Limit Temp : -7 C
GPU Slowdown T.Limit Temp : -2 C
GPU Max Operating T.Limit Temp : 0 C
GPU Target Temperature : 84 C
Memory Current Temp : N/A
Memory Max Operating T.Limit Temp : N/A
GPU Power Readings
Average Power Draw : 80.23 W
Instantaneous Power Draw : 123.61 W
Current Power Limit : 425.00 W
Requested Power Limit : 425.00 W
Default Power Limit : 425.00 W
Min Power Limit : 150.00 W
Max Power Limit : 425.00 W
GPU Memory Power Readings
Average Power Draw : N/A
Instantaneous Power Draw : N/A
Module Power Readings
Average Power Draw : N/A
Instantaneous Power Draw : N/A
Current Power Limit : N/A
Requested Power Limit : N/A
Default Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
Power Smoothing : N/A
Workload Power Profiles
Requested Profiles : N/A
Enforced Profiles : N/A
Clocks
Graphics : 2520 MHz
SM : 2520 MHz
Memory : 10251 MHz
Video : 1965 MHz
Applications Clocks
Graphics : N/A
Memory : N/A
Default Applications Clocks
Graphics : N/A
Memory : N/A
Deferred Clocks
Memory : N/A
Max Clocks
Graphics : 3105 MHz
SM : 3105 MHz
Memory : 10501 MHz
Video : 2415 MHz
Max Customer Boost Clocks
Graphics : N/A
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Fabric
State : N/A
Status : N/A
CliqueId : N/A
ClusterUUID : N/A
Health
Summary : N/A
Bandwidth : N/A
Route Recovery in progress : N/A
Route Unhealthy : N/A
Access Timeout Recovery : N/A
Incorrect Configuration : N/A
Partition Assigned : N/A
Processes
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 1745
Type : G
Name :
Used GPU Memory : 4 MiB
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 78150
Type : C+G
Name :
Used GPU Memory : 5285 MiB
Capabilities
EGM : disabled

'
2026-03-26T09:00:41Z [309ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] osDistro = 'ubuntu'
2026-03-26T09:00:41Z [312ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] osName = '24.04.2 LTS (Noble Numbat)'
2026-03-26T09:00:41Z [316ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] osVersion = '24.04.2'
2026-03-26T09:00:41Z [319ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] portableMode = '1'
2026-03-26T09:00:41Z [322ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] runEnvironment = 'Individual'
2026-03-26T09:00:41Z [326ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] runningInContainer = '1'
2026-03-26T09:00:41Z [329ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] shaderdb_debugSymbols = '0'
2026-03-26T09:00:41Z [333ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] shaderdb_dumpIncludeOverrides = '0'
2026-03-26T09:00:41Z [336ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] shaderdb_dumpIntermediates = '0'
2026-03-26T09:00:41Z [340ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] shaderdb_obfuscateCode = '1'
2026-03-26T09:00:41Z [343ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] shaderdb_optimizationLevel = '1'

2026-03-26T09:00:41Z [347ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] systemInfo = '
Driver Version: 580.126.09
=============================================================================================
GPU
---------------------------------------------------------------------------------------------
0
=============================================================================================
OS: 24.04.2 LTS (Noble Numbat) ubuntu, Version: 24.04.2, Kernel: 6.8.0-106-generic
Processor: INTEL(R) XEON(R) GOLD 6526Y
Cores: 32
---------------------------------------------------------------------------------------------
Total Memory (MB): 128576
Total Page/Swap (MB): 2047
---------------------------------------------------------------------------------------------
'
2026-03-26T09:00:41Z [350ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] telemetrySessionId = '9059868531960409619'
2026-03-26T09:00:41Z [353ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] terminatedByAbort = '0'
2026-03-26T09:00:41Z [356ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] totalRamBareMetalMB = '128576'
2026-03-26T09:00:41Z [359ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] totalRamLimitedMB = '128576'
2026-03-26T09:00:41Z [362ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] totalSwapBareMetalMB = '2047'
2026-03-26T09:00:41Z [365ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] totalSwapLimitedMB = '2047'
2026-03-26T09:00:41Z [368ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] userId = 'default'
2026-03-26T09:00:41Z [371ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] workingDirectory = '/geniesim/main/data_collection'
2026-03-26T09:00:41Z [374ms] [Fatal] [carb.crashreporter-breakpad.plugin] [crash] Thread 139 backtrace follows:
2026-03-26T09:00:41Z [394ms] [Fatal] [carb.crashreporter-breakpad.plugin] 000: libc.so.6!__sigaction+0x50 (??:?)
2026-03-26T09:00:41Z [411ms] [Fatal] [carb.crashreporter-breakpad.plugin] 001: libgpu.foundation.plugin.so!void std::vector<std::tuple<char const*, float, char const*, unsigned int>, std::allocator<std::tuple<char const*, float, char const*, unsigned int> > >::_M_realloc_insert<std::tuple<char const*, float, char const*, unsigned int> >(__gnu_cxx::__normal_iterator<std::tuple<char const*, float, char const*, unsigned int>, std::vector<std::tuple<char const, float, char const*, unsigned int>, std::allocator<std::tuple<char const*, float, char const*, unsigned int> > > >, std::tuple<char const*, float, char const*, unsigned int>&&)+0x7017 (??:?)
2026-03-26T09:00:41Z [427ms] [Fatal] [carb.crashreporter-breakpad.plugin] 002: libgpu.foundation.plugin.so!std::vector<unsigned char, std::allocator >::_M_default_append(unsigned long)+0x112cb (??:?)
2026-03-26T09:00:41Z [442ms] [Fatal] [carb.crashreporter-breakpad.plugin] 003: librtx.raytracing.plugin.so!std::_Rb_tree<unsigned int, std::pair<unsigned int const, unsigned int>, std::_Select1st<std::pair<unsigned int const, unsigned int> >, std::less, std::allocator<std::pair<unsigned int const, unsigned int> > >::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_iterator<std::pair<unsigned int const, unsigned int> >, unsigned int const&)+0x122d2 (??:?)
2026-03-26T09:00:41Z [454ms] [Fatal] [carb.crashreporter-breakpad.plugin] 004: libcarb.scenerenderer-rtx.plugin.so!std::unordered_map<unsigned int, unsigned int, std::hash, std::equal_to, std::allocator<std::pair<unsigned int const, unsigned int> > >::~unordered_map()+0x50a3f (??:?)
2026-03-26T09:00:41Z [467ms] [Fatal] [carb.crashreporter-breakpad.plugin] 005: libcarb.scenerenderer-rtx.plugin.so!std::unordered_map<unsigned int, unsigned int, std::hash, std::equal_to, std::allocator<std::pair<unsigned int const, unsigned int> > >::~unordered_map()+0x55e46 (??:?)
2026-03-26T09:00:41Z [479ms] [Fatal] [carb.crashreporter-breakpad.plugin] 006: libcarb.scenerenderer-rtx.plugin.so!std::unordered_map<unsigned int, unsigned int, std::hash, std::equal_to, std::allocator<std::pair<unsigned int const, unsigned int> > >::~unordered_map()+0x59aaf (??:?)
2026-03-26T09:00:41Z [492ms] [Fatal] [carb.crashreporter-breakpad.plugin] 007: libcarb.scenerenderer-rtx.plugin.so!std::_Hashtable<unsigned int, unsigned int, std::allocator, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<unsigned int, false>*, unsigned long)+0xb8fc (??:?)
2026-03-26T09:00:41Z [503ms] [Fatal] [carb.crashreporter-breakpad.plugin] 008: librtx.hydra.so!std::unique_lockstd::shared_mutex::unlock()+0xf299 (??:?)
2026-03-26T09:00:41Z [516ms] [Fatal] [carb.crashreporter-breakpad.plugin] 009: librtx.hydra.so!std::unique_lockstd::shared_mutex::unlock()+0x1487c (??:?)
2026-03-26T09:00:41Z [529ms] [Fatal] [carb.crashreporter-breakpad.plugin] 010: libomni.usd.so!std::_Hashtable<unsigned int, unsigned int, std::allocator, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_rehash(unsigned long, unsigned long const&)+0xe6bf (??:?)
2026-03-26T09:00:41Z [543ms] [Fatal] [carb.crashreporter-breakpad.plugin] 011: libomni.usd.so!std::_Hashtable<unsigned int, unsigned int, std::allocator, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_rehash(unsigned long, unsigned long const&)+0x120c8 (??:?)
2026-03-26T09:00:41Z [555ms] [Fatal] [carb.crashreporter-breakpad.plugin] 012: libomni.usd.so!std::_Hashtable<unsigned int, unsigned int, std::allocator, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_rehash(unsigned long, unsigned long const&)+0x14502 (??:?)
2026-03-26T09:00:41Z [568ms] [Fatal] [carb.crashreporter-breakpad.plugin] 013: libomni.usd.so!std::_Hashtable<unsigned int, unsigned int, std::allocator, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_rehash(unsigned long, unsigned long const&)+0x158fc (??:?)
2026-03-26T09:00:41Z [581ms] [Fatal] [carb.crashreporter-breakpad.plugin] 014: libomni.usd.so!std::_Hashtable<unsigned int, unsigned int, std::allocator, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_rehash(unsigned long, unsigned long const&)+0x16b56 (??:?)
2026-03-26T09:00:41Z [596ms] [Fatal] [carb.crashreporter-breakpad.plugin] 015: libcarb.eventdispatcher.plugin.so!carbOnPluginRegisterEx+0x3441 (??:?)
2026-03-26T09:00:41Z [611ms] [Fatal] [carb.crashreporter-breakpad.plugin] 016: libomni.kit.loop-isaac.plugin.so!void std::__detail::__to_chars_10_impl(char*, unsigned int, unsigned int)+0x1a48d (??:?)
2026-03-26T09:00:42Z [623ms] [Fatal] [carb.crashreporter-breakpad.plugin] 017: libomni.kit.app.plugin.so!+0x235e1
2026-03-26T09:00:42Z [638ms] [Fatal] [carb.crashreporter-breakpad.plugin] 018: libomni.kit.app.plugin.so!carbOnPluginPreStartup+0x1fa0 (??:?)
2026-03-26T09:00:42Z [653ms] [Fatal] [carb.crashreporter-breakpad.plugin] 019: _app.cpython-311-x86_64-linux-gnu.so!std::__detail::_Map_base<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, void*>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, void*> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::hash<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](std::__cxx11::basic_string<char, std::char_traits, std::allocator >&&)+0xcfd7 (??:?)
2026-03-26T09:00:42Z [669ms] [Fatal] [carb.crashreporter-breakpad.plugin] 020: _app.cpython-311-x86_64-linux-gnu.so!std::__detail::_Map_base<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, void*>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, void*> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::hash<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](std::__cxx11::basic_string<char, std::char_traits, std::allocator >&&)+0x88ae (??:?)
2026-03-26T09:00:42Z [768ms] [Fatal] [carb.crashreporter-breakpad.plugin] 021: libpython3.11.so.1.0!cfunction_call+0x37 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Objects/methodobject.c:542)
2026-03-26T09:00:42Z [829ms] [Fatal] [carb.crashreporter-breakpad.plugin] 022: libpython3.11.so.1.0!_PyObject_MakeTpCall+0x95 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/./Include/internal/pycore_ceval.h:123)
2026-03-26T09:00:42Z [1,000ms] [Fatal] [carb.crashreporter-breakpad.plugin] 023: libpython3.11.so.1.0!_PyEval_EvalFrameDefault+0x8b8c (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Python/ceval.c:7285)
2026-03-26T09:00:42Z [1,162ms] [Fatal] [carb.crashreporter-breakpad.plugin] 024: libpython3.11.so.1.0!PyEval_EvalCode+0x4ba (/builds/omniverse/externals/python-build/_build/Python-3.11.13/./Include/internal/pycore_ceval.h:73)
2026-03-26T09:00:42Z [1,362ms] [Fatal] [carb.crashreporter-breakpad.plugin] 025: libpython3.11.so.1.0!run_mod+0xa5 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Python/pythonrun.c:1742)
2026-03-26T09:00:42Z [1,563ms] [Fatal] [carb.crashreporter-breakpad.plugin] 026: libpython3.11.so.1.0!_PyRun_SimpleFileObject+0x130 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Python/pythonrun.c:1662)
2026-03-26T09:00:43Z [1,758ms] [Fatal] [carb.crashreporter-breakpad.plugin] 027: libpython3.11.so.1.0!_PyRun_AnyFileObject+0x3c (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Python/pythonrun.c:79)
2026-03-26T09:00:43Z [1,974ms] [Fatal] [carb.crashreporter-breakpad.plugin] 028: libpython3.11.so.1.0!Py_RunMain+0x793 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Modules/main.c:361)
2026-03-26T09:00:43Z [2,195ms] [Fatal] [carb.crashreporter-breakpad.plugin] 029: libpython3.11.so.1.0!Py_BytesMain+0x43 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Modules/main.c:739)
2026-03-26T09:00:43Z [2,216ms] [Fatal] [carb.crashreporter-breakpad.plugin] 030: libc.so.6!__libc_init_first+0x8a (??:?)
2026-03-26T09:00:43Z [2,233ms] [Fatal] [carb.crashreporter-breakpad.plugin] 031: libc.so.6!__libc_start_main+0x8b (??:?)
2026-03-26T09:00:43Z [2,239ms] [Fatal] [carb.crashreporter-breakpad.plugin] 032: python3!_start+0x29 (??:?)
/isaac-sim/python.sh: line 73: 139 Segmentation fault (core dumped) $python_exe "${filtered_args[@]}" $args
There was an error running python
[2026-03-26 09:00:48] Error: data_collector_server.py process died
Last 30 lines of log:
2026-03-26T09:00:41Z [467ms] [Fatal] [carb.crashreporter-breakpad.plugin] 005: libcarb.scenerenderer-rtx.plugin.so!std::unordered_map<unsigned int, unsigned int, std::hash, std::equal_to, std::allocator<std::pair<unsigned int const, unsigned int> > >::~unordered_map()+0x55e46 (??:?)
2026-03-26T09:00:41Z [479ms] [Fatal] [carb.crashreporter-breakpad.plugin] 006: libcarb.scenerenderer-rtx.plugin.so!std::unordered_map<unsigned int, unsigned int, std::hash, std::equal_to, std::allocator<std::pair<unsigned int const, unsigned int> > >::~unordered_map()+0x59aaf (??:?)
2026-03-26T09:00:41Z [492ms] [Fatal] [carb.crashreporter-breakpad.plugin] 007: libcarb.scenerenderer-rtx.plugin.so!std::_Hashtable<unsigned int, unsigned int, std::allocator, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<unsigned int, false>*, unsigned long)+0xb8fc (??:?)
2026-03-26T09:00:41Z [503ms] [Fatal] [carb.crashreporter-breakpad.plugin] 008: librtx.hydra.so!std::unique_lockstd::shared_mutex::unlock()+0xf299 (??:?)
2026-03-26T09:00:41Z [516ms] [Fatal] [carb.crashreporter-breakpad.plugin] 009: librtx.hydra.so!std::unique_lockstd::shared_mutex::unlock()+0x1487c (??:?)
2026-03-26T09:00:41Z [529ms] [Fatal] [carb.crashreporter-breakpad.plugin] 010: libomni.usd.so!std::_Hashtable<unsigned int, unsigned int, std::allocator, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_rehash(unsigned long, unsigned long const&)+0xe6bf (??:?)
2026-03-26T09:00:41Z [543ms] [Fatal] [carb.crashreporter-breakpad.plugin] 011: libomni.usd.so!std::_Hashtable<unsigned int, unsigned int, std::allocator, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_rehash(unsigned long, unsigned long const&)+0x120c8 (??:?)
2026-03-26T09:00:41Z [555ms] [Fatal] [carb.crashreporter-breakpad.plugin] 012: libomni.usd.so!std::_Hashtable<unsigned int, unsigned int, std::allocator, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_rehash(unsigned long, unsigned long const&)+0x14502 (??:?)
2026-03-26T09:00:41Z [568ms] [Fatal] [carb.crashreporter-breakpad.plugin] 013: libomni.usd.so!std::_Hashtable<unsigned int, unsigned int, std::allocator, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_rehash(unsigned long, unsigned long const&)+0x158fc (??:?)
2026-03-26T09:00:41Z [581ms] [Fatal] [carb.crashreporter-breakpad.plugin] 014: libomni.usd.so!std::_Hashtable<unsigned int, unsigned int, std::allocator, std::__detail::_Identity, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, true, true> >::_M_rehash(unsigned long, unsigned long const&)+0x16b56 (??:?)
2026-03-26T09:00:41Z [596ms] [Fatal] [carb.crashreporter-breakpad.plugin] 015: libcarb.eventdispatcher.plugin.so!carbOnPluginRegisterEx+0x3441 (??:?)
2026-03-26T09:00:41Z [611ms] [Fatal] [carb.crashreporter-breakpad.plugin] 016: libomni.kit.loop-isaac.plugin.so!void std::__detail::__to_chars_10_impl(char*, unsigned int, unsigned int)+0x1a48d (??:?)
2026-03-26T09:00:42Z [623ms] [Fatal] [carb.crashreporter-breakpad.plugin] 017: libomni.kit.app.plugin.so!+0x235e1
2026-03-26T09:00:42Z [638ms] [Fatal] [carb.crashreporter-breakpad.plugin] 018: libomni.kit.app.plugin.so!carbOnPluginPreStartup+0x1fa0 (??:?)
2026-03-26T09:00:42Z [653ms] [Fatal] [carb.crashreporter-breakpad.plugin] 019: _app.cpython-311-x86_64-linux-gnu.so!std::__detail::_Map_base<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, void*>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, void*> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::hash<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](std::__cxx11::basic_string<char, std::char_traits, std::allocator >&&)+0xcfd7 (??:?)
2026-03-26T09:00:42Z [669ms] [Fatal] [carb.crashreporter-breakpad.plugin] 020: _app.cpython-311-x86_64-linux-gnu.so!std::__detail::_Map_base<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, void*>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, void*> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::hash<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](std::__cxx11::basic_string<char, std::char_traits, std::allocator >&&)+0x88ae (??:?)
2026-03-26T09:00:42Z [768ms] [Fatal] [carb.crashreporter-breakpad.plugin] 021: libpython3.11.so.1.0!cfunction_call+0x37 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Objects/methodobject.c:542)
2026-03-26T09:00:42Z [829ms] [Fatal] [carb.crashreporter-breakpad.plugin] 022: libpython3.11.so.1.0!_PyObject_MakeTpCall+0x95 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/./Include/internal/pycore_ceval.h:123)
2026-03-26T09:00:42Z [1,000ms] [Fatal] [carb.crashreporter-breakpad.plugin] 023: libpython3.11.so.1.0!_PyEval_EvalFrameDefault+0x8b8c (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Python/ceval.c:7285)
2026-03-26T09:00:42Z [1,162ms] [Fatal] [carb.crashreporter-breakpad.plugin] 024: libpython3.11.so.1.0!PyEval_EvalCode+0x4ba (/builds/omniverse/externals/python-build/_build/Python-3.11.13/./Include/internal/pycore_ceval.h:73)
2026-03-26T09:00:42Z [1,362ms] [Fatal] [carb.crashreporter-breakpad.plugin] 025: libpython3.11.so.1.0!run_mod+0xa5 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Python/pythonrun.c:1742)
2026-03-26T09:00:42Z [1,563ms] [Fatal] [carb.crashreporter-breakpad.plugin] 026: libpython3.11.so.1.0!_PyRun_SimpleFileObject+0x130 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Python/pythonrun.c:1662)
2026-03-26T09:00:43Z [1,758ms] [Fatal] [carb.crashreporter-breakpad.plugin] 027: libpython3.11.so.1.0!_PyRun_AnyFileObject+0x3c (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Python/pythonrun.c:79)
2026-03-26T09:00:43Z [1,974ms] [Fatal] [carb.crashreporter-breakpad.plugin] 028: libpython3.11.so.1.0!Py_RunMain+0x793 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Modules/main.c:361)
2026-03-26T09:00:43Z [2,195ms] [Fatal] [carb.crashreporter-breakpad.plugin] 029: libpython3.11.so.1.0!Py_BytesMain+0x43 (/builds/omniverse/externals/python-build/_build/Python-3.11.13/Modules/main.c:739)
2026-03-26T09:00:43Z [2,216ms] [Fatal] [carb.crashreporter-breakpad.plugin] 030: libc.so.6!__libc_init_first+0x8a (??:?)
2026-03-26T09:00:43Z [2,233ms] [Fatal] [carb.crashreporter-breakpad.plugin] 031: libc.so.6!__libc_start_main+0x8b (??:?)
2026-03-26T09:00:43Z [2,239ms] [Fatal] [carb.crashreporter-breakpad.plugin] 032: python3!_start+0x29 (??:?)
/isaac-sim/python.sh: line 73: 139 Segmentation fault (core dumped) $python_exe "${filtered_args[@]}" $args
There was an error running python

==========================================
Cleaning up...

Stopping data_collector_server...
Stopping run_data_collection.py...
Cleanup complete
2026-03-26 17:00:50 - Container stopped
2026-03-26 17:00:50 -
2026-03-26 17:00:50 - ==========================================
2026-03-26 17:00:50 - Data collection completed
2026-03-26 17:00:50 - ==========================================
2026-03-26 17:00:50 - Logs saved to: /code/genie_sim/source/data_collection/logs/sort_the_fruit_into_the_box_apple_g2
2026-03-26 17:00:50 - - run_data_collection_sh.log (script output)
2026-03-26 17:00:50 - - container.log (container logs)
2026-03-26 17:00:50 - - data_collector_server.log (if available)
2026-03-26 17:00:50 - - run_data_collection.log (if available)
2026-03-26 17:00:50 -
2026-03-26 17:00:50 - Exiting (container will be cleaned up automatically)...
2026-03-26 17:00:50 - Cleaning up container...
2026-03-26 17:00:50 - Stopping container: data_collection_open_source
类似的问题，机房的服务器会报segmetation fault，我个人的PC就不会。启动脚本是：./scripts/run_data_collection.sh --headless --task tasks/geniesim_2025/sort_fruit/g2/sort_the_fruit_into_the_box_apple_g2.json。

Log 前面很多VkResult: ERROR_OUT_OF_DEVICE_MEMORY的错误，但实际上显存完全够用。

容器环境下单独启动isaac 是没问题的。

@MiaoMieu

[MiaoMieu]

这里看log是 isaacsim渲染时报错了， 我们之前遇到过一次类似的报错是多环境使用了同一个资产，不知道对你有没有帮助

[MiaoMieu]

可以先尝试下 关闭 publish_ros 和 关闭 client 的 recording 看会不会报错