EXC_BAD_ACCESS in jl_object_id__cold on macOS via (py)juliacall

[mkitti]

(python3.13) (base) kittisopikulm@kittisopikulm-wm1 python3.13 % lldb python
(lldb) target create "python"
Current executable set to '/Users/kittisopikulm/.julia/dev/PythonCall/python3.13/.pixi/envs/default/bin/python' (arm64).
(lldb) run
Process 6498 launched: '/Users/kittisopikulm/.julia/dev/PythonCall/python3.13/.pixi/envs/default/bin/python' (arm64)
Python 3.13.3 | packaged by conda-forge | (main, Apr 14 2025, 20:44:30) [Clang 18.1.8 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import juliacall
warning: (arm64) /Users/kittisopikulm/.julia/juliaup/julia-1.11.5+0.aarch64.apple.darwin14/share/julia/compiled/v1.11/Printf/3FQLY_RnzCd.dylib empty dSYM file detected, dSYM was created with an executable with no debug info.
warning: (arm64) /Users/kittisopikulm/.julia/juliaup/julia-1.11.5+0.aarch64.apple.darwin14/share/julia/compiled/v1.11/UUIDs/SIw1t_RnzCd.dylib empty dSYM file detected, dSYM was created with an executable with no debug info.
warning: (arm64) /Users/kittisopikulm/.julia/juliaup/julia-1.11.5+0.aarch64.apple.darwin14/share/julia/compiled/v1.11/Logging/PWFjL_RnzCd.dylib empty dSYM file detected, dSYM was created with an executable with no debug info.
warning: (arm64) /Users/kittisopikulm/.julia/compiled/v1.11/UnsafePointers/FMCLb_LIt2D.dylib empty dSYM file detected, dSYM was created with an executable with no debug info.
warning: (arm64) /Users/kittisopikulm/.julia/juliaup/julia-1.11.5+0.aarch64.apple.darwin14/share/julia/compiled/v1.11/Mmap/GDYy0_RnzCd.dylib empty dSYM file detected, dSYM was created with an executable with no debug info.
warning: (arm64) /Users/kittisopikulm/.julia/compiled/v1.11/Pidfile/wlmRx_LIt2D.dylib empty dSYM file detected, dSYM was created with an executable with no debug info.
warning: (arm64) /Users/kittisopikulm/.julia/compiled/v1.11/Scratch/ICI1U_LIt2D.dylib empty dSYM file detected, dSYM was created with an executable with no debug info.
warning: (arm64) /Users/kittisopikulm/.julia/juliaup/julia-1.11.5+0.aarch64.apple.darwin14/share/julia/compiled/v1.11/LazyArtifacts/MRP8l_RnzCd.dylib empty dSYM file detected, dSYM was created with an executable with no debug info.
warning: (arm64) /Users/kittisopikulm/.julia/compiled/v1.11/DataValueInterfaces/9Lpkp_LIt2D.dylib empty dSYM file detected, dSYM was created with an executable with no debug info.
warning: (arm64) /Users/kittisopikulm/.julia/compiled/v1.11/DataAPI/3a8mN_LIt2D.dylib empty dSYM file detected, dSYM was created with an executable with no debug info.
warning: (arm64) /Users/kittisopikulm/.julia/compiled/v1.11/IteratorInterfaceExtensions/N0h8q_LIt2D.dylib empty dSYM file detected, dSYM was created with an executable with no debug info.
warning: (arm64) /Users/kittisopikulm/.julia/compiled/v1.11/TableTraits/I6SaN_LIt2D.dylib empty dSYM file detected, dSYM was created with an executable with no debug info.
Process 6498 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x100a0)
    frame #0: 0x00000001061636a0 libjulia-internal.1.11.5.dylib`jl_object_id__cold(tv=<unavailable>, v=0x000000015da15840) at builtins.c:441:13 [opt]
Target 0: (python) stopped.
warning: libjulia-internal.1.11.5.dylib was compiled with optimization - stepping may behave oddly; variables may not be available.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x100a0)
  * frame #0: 0x00000001061636a0 libjulia-internal.1.11.5.dylib`jl_object_id__cold(tv=<unavailable>, v=0x000000015da15840) at builtins.c:441:13 [opt]
    frame #1: 0x000000010615cc70 libjulia-internal.1.11.5.dylib`smallintset_rehash(a=0x000000014fea6a50, hash=(libjulia-internal.1.11.5.dylib`idset_hash [inlined] jl_genericmemory_ptr_ref at julia.h:1192:12
libjulia-internal.1.11.5.dylib`idset_hash at idset.c:6:21), data=0x0000000141bd2bd0, newsz=32768, np=<unavailable>) at smallintset.c:218:48 [opt]
    frame #2: 0x000000010615c854 libjulia-internal.1.11.5.dylib`jl_smallintset_insert(pcache=0x000000016fdf9cf8, parent=0x0000000000000000, hash=(libjulia-internal.1.11.5.dylib`idset_hash [inlined] jl_genericmemory_ptr_ref at julia.h:1192:12
libjulia-internal.1.11.5.dylib`idset_hash at idset.c:6:21), val=7115, data=0x0000000141bd2bd0) at smallintset.c:197:13 [opt]
    frame #3: 0x0000000106164468 libjulia-internal.1.11.5.dylib`jl_idset_put_idx(keys=0x0000000141bd2bd0, idxs=0x000000014fea6a50, idx=7115) at idset.c:104:9 [opt]
    frame #4: 0x000000010617de8c libjulia-internal.1.11.5.dylib`jl_as_global_root(val=0x000000015d9bf280, insert=1) at staticdata.c:2548:34 [opt]
    frame #5: 0x00000001061ee43c libjulia-internal.1.11.5.dylib`jl_encode_as_indexed_root(s=0x000000016fdf9fb0, v=0x000000015d9bf280) at ircode.c:108:13 [opt]
    frame #6: 0x00000001061eabf0 libjulia-internal.1.11.5.dylib`jl_encode_value_(s=<unavailable>, v=0x0000000141cd1e50, as_literal=0) at ircode.c:273:13 [opt]
    frame #7: 0x00000001061ee7dc libjulia-internal.1.11.5.dylib`jl_encode_memory_slice(s=0x000000016fdf9fb0, mem=0x000000013fb6f4f0, offset=0, len=<unavailable>) at ircode.c:135:13 [opt]
    frame #8: 0x00000001061eb378 libjulia-internal.1.11.5.dylib`jl_encode_value_(s=<unavailable>, v=0x0000000141d1c250, as_literal=1) at ircode.c:403:9 [opt]
    frame #9: 0x00000001061e9fe0 libjulia-internal.1.11.5.dylib`ijl_compress_ir(m=0x000000015d401f60, code=0x0000000142461600) at ircode.c:866:9 [opt]
    frame #10: 0x000000014bee9f24 sys.dylib`julia_maybe_compress_codeinfo_39899.1 at typeinfer.jl:394
    frame #11: 0x000000014bb7ab00 sys.dylib`julia_CodeInstance_26618.1 at typeinfer.jl:408
    frame #12: 0x000000014c5fd6cc sys.dylib`julia_cache_resultNOT._39908.1 at typeinfer.jl:435
    frame #13: 0x000000014c5fd7a0 sys.dylib`jfptr_cache_resultNOT._39909.1 + 60
    frame #14: 0x000000014bbaccd4 sys.dylib`julia_finish_nocycle_39879.1 at typeinfer.jl:269
    frame #15: 0x000000014bbfb484 sys.dylib`julia__typeinf_39867.1 at typeinfer.jl:249
    frame #16: 0x000000014c5f9594 sys.dylib`julia_typeinf_39819.1 at typeinfer.jl:215
    frame #17: 0x000000014c67bfc4 sys.dylib`julia_typeinf_edge_39988.1 at typeinfer.jl:923
    frame #18: 0x000000014c68a2ac sys.dylib`julia_abstract_call_method_38697.1 at abstractinterpretation.jl:660
    frame #19: 0x000000014bedcc4c sys.dylib`julia_abstract_call_gf_by_type_38516.1 at abstractinterpretation.jl:102
    frame #20: 0x000000014bf6cfa0 sys.dylib`julia_abstract_call_known_39231.1 at abstractinterpretation.jl:2200
    frame #21: 0x000000014c2eb160 sys.dylib`julia_abstract_call_39387.1 at abstractinterpretation.jl:2282
    frame #22: 0x000000014bb7c08c sys.dylib`julia_abstract_call_39402.1 at abstractinterpretation.jl:2275
    frame #23: 0x000000014bf53a54 sys.dylib`julia_abstract_call_39380.1 at abstractinterpretation.jl:2423
    frame #24: 0x000000014c6713a8 sys.dylib`julia_abstract_eval_call_39486.1 at abstractinterpretation.jl:2438
    frame #25: 0x000000014bbe8734 sys.dylib`julia_abstract_eval_statement_expr_39501.1 at abstractinterpretation.jl:2454
    frame #26: 0x000000014c674594 sys.dylib`julia_abstract_eval_statement_39625.1 at abstractinterpretation.jl:2752
    frame #27: 0x000000014c276aa0 sys.dylib`julia_abstract_eval_basic_statement_39691.1 at abstractinterpretation.jl:3068
    frame #28: 0x000000014bf55eb8 sys.dylib`julia_typeinf_local_39762.1 at abstractinterpretation.jl:3331
    frame #29: 0x000000014bbacd40 sys.dylib`julia_typeinf_nocycle_39812.1 at abstractinterpretation.jl:3413
    frame #30: 0x000000014bbfb35c sys.dylib`julia__typeinf_39867.1 at typeinfer.jl:244
    frame #31: 0x000000014c5f9594 sys.dylib`julia_typeinf_39819.1 at typeinfer.jl:215
    frame #32: 0x000000014bbfb9a0 sys.dylib`julia_typeinf_ext_40029.1 at typeinfer.jl:1101
    frame #33: 0x000000014bee143c sys.dylib`julia_typeinf_ext_toplevel_40061.1 at typeinfer.jl:1139
    frame #34: 0x000000014c5f94f0 sys.dylib`julia_typeinf_ext_toplevel_40056.1 at typeinfer.jl:1135
    frame #35: 0x000000014bb76c10 sys.dylib`jfptr_typeinf_ext_toplevel_40057.1 + 64
    frame #36: 0x000000010614d9b4 libjulia-internal.1.11.5.dylib`jl_type_infer [inlined] jl_apply(args=<unavailable>, nargs=3) at julia.h:2157:12 [opt]
    frame #37: 0x000000010614d9a4 libjulia-internal.1.11.5.dylib`jl_type_infer(mi=0x0000000141531b10, world=26727, force=<unavailable>) at gf.c:390:32 [opt]
    frame #38: 0x0000000105ef0aa0 libjulia-codegen.1.11.5.dylib`jl_generate_fptr_impl(mi=0x0000000141531b10, world=26727, did_compile=0x000000016fdfd08c) at jitlayers.cpp:519:19 [opt]
    frame #39: 0x0000000106152b5c libjulia-internal.1.11.5.dylib`jl_compile_method_internal(mi=0x0000000141531b10, world=26727) at gf.c:2536:16 [opt]
    frame #40: 0x0000000106156028 libjulia-internal.1.11.5.dylib`ijl_apply_generic [inlined] _jl_invoke(F=0x000000015d4682a0, args=0x000000016fdfd2f0, nargs=2, mfunc=0x0000000141531b10, world=26727) at gf.c:2940:16 [opt]
    frame #41: 0x0000000106155f84 libjulia-internal.1.11.5.dylib`ijl_apply_generic(F=0x000000015d4682a0, args=0x000000016fdfd2f0, nargs=2) at gf.c:3125:12 [opt]
    frame #42: 0x000000015d1f388c WdXsa_LIt2D.dylib`julia__pyjl_callmethod_9729 at base.jl:67
    frame #43: 0x000000015d1df970 WdXsa_LIt2D.dylib`julia__pyjl_callmethod_9722 at C.jl:63
    frame #44: 0x000000015d229730 WdXsa_LIt2D.dylib`jfptr__pyjl_callmethod_9723 + 108
    frame #45: 0x000000015d22a4d4 WdXsa_LIt2D.dylib`jlcapi__pyjl_callmethod_9817 + 232
    frame #46: 0x00000001000edb10 python`cfunction_call + 352
    frame #47: 0x000000010007ce40 python`_PyObject_MakeTpCall + 340
    frame #48: 0x00000001001fad94 python`_PyEval_EvalFrameDefault + 11428
    frame #49: 0x000000010007dbd4 python`PyObject_Vectorcall + 92
    frame #50: 0x0000000100129314 python`_Py_slot_tp_getattr_hook + 896
    frame #51: 0x00000001000f7388 python`PyObject_GetAttr + 88
    frame #52: 0x0000000100202768 python`_PyEval_EvalFrameDefault + 42616
    frame #53: 0x00000001001f6eac python`PyEval_EvalCode + 308
    frame #54: 0x00000001001f2320 python`builtin_exec + 1756
    frame #55: 0x00000001000ee8bc python`cfunction_vectorcall_FASTCALL_KEYWORDS + 140
    frame #56: 0x00000001001fc064 python`_PyEval_EvalFrameDefault + 16244
    frame #57: 0x000000010007f99c python`object_vacall + 400
    frame #58: 0x000000010007f708 python`PyObject_CallMethodObjArgs + 124
    frame #59: 0x0000000100252828 python`PyImport_ImportModuleLevelObject + 2408
    frame #60: 0x0000000100200f84 python`_PyEval_EvalFrameDefault + 36500
    frame #61: 0x00000001001f6eac python`PyEval_EvalCode + 308
    frame #62: 0x00000001001f2320 python`builtin_exec + 1756
    frame #63: 0x00000001000ee8bc python`cfunction_vectorcall_FASTCALL_KEYWORDS + 140
    frame #64: 0x000000010007dbd4 python`PyObject_Vectorcall + 92
    frame #65: 0x00000001001fad94 python`_PyEval_EvalFrameDefault + 11428
    frame #66: 0x00000001001f6eac python`PyEval_EvalCode + 308
    frame #67: 0x00000001001f2320 python`builtin_exec + 1756
    frame #68: 0x00000001000ee8bc python`cfunction_vectorcall_FASTCALL_KEYWORDS + 140
    frame #69: 0x000000010007dbd4 python`PyObject_Vectorcall + 92
    frame #70: 0x00000001001fad94 python`_PyEval_EvalFrameDefault + 11428
    frame #71: 0x00000001002b7980 python`pymain_run_module + 288
    frame #72: 0x00000001002b8514 python`pymain_run_stdin + 908
    frame #73: 0x00000001002b774c python`Py_RunMain + 2880
    frame #74: 0x00000001002b9164 python`pymain_main + 716
    frame #75: 0x00000001000047c8 python`main + 56
    frame #76: 0x000000018132ffd8 dyld`start + 2412

[mkitti]

Reproduction steps:

1. Use pixi to install Python 3.13 and pyjuliacall from conda-forge
2. Run Python under lldb
3. In Python 3.13, run import juliacall

Pixi details:

pixi init
pixi add python=3.13.3 pyjuliacall
pixi shell

Pixi.toml:

[project]
authors = ["Mark Kittisopikul <kittisopikulm@janelia.hhmi.org>"]
channels = ["conda-forge"]
description = "Add a short description here"
name = "python3.13"
platforms = ["osx-arm64"]
version = "0.1.0"

[tasks]

[dependencies]
python = ">=3.13.3,<3.14"
pyjuliacall = ">=0.9.24,<0.10"

[vchuravy]

Does not reproduce under x86 / Archlinux.

[MilesCranmer]

It's useful that you can repro this locally. Perhaps this issue is due to the same root cause as #58171 (also macOS-only, although I only see it in GitHub action runners)

[MilesCranmer]

For whatever reason I'm also unable to reproduce this locally on macOS M1/arm64. I am quite confused about the conditions required to repro these issues. Do you happen to be running this on a low memory machine @mkitti? (Like my GitHub action)

[MilesCranmer]

@vchuravy What does this block do?

julia/src/smallintset.c

Lines 212 to 223 in 760b2e5

	while (1) {
	jl_genericmemory_t *newa = jl_alloc_int_1d(np + 1, newsz);
	JL_GC_PUSH1(&newa);
	for (i = 0; i < sz; i += 1) {
	size_t val1 = jl_intref(a, i);
	if (val1 != 0) {
	if (!smallintset_insert_(newa, hash(val1 - 1, data), val1)) {
	break;
	}
	}
	}
	JL_GC_POP();

Is the missing root likely located here, or is it just surfacing a missing root from elsewhere?

I'm just reading some of the docs around this, like

// NOTE: it is the caller's responsibility to make sure arguments are
// rooted such that the gc can see them on the stack.
// `foo(f(), g())` is not safe,
// since the result of `f()` is not rooted during the call to `g()`,
// and the arguments to foo are not gc-protected during the call to foo.
// foo can't do anything about it, so the caller must do:
// jl_value_t *x=NULL, *y=NULL; JL_GC_PUSH2(&x, &y);
// x = f(); y = g(); foo(x, y)

Just so I understand, why isn't smallintset_rehash written as follows?

    JL_GC_PUSH3(&newa, &data, &a)

i.e., why is it only

    JL_GC_PUSH1(&newa)

It looks like data comes (via jl_smallintset_insert) from here:

julia/src/module.c

Line 721 in 760b2e5

jl_svec_t *bindings = jl_atomic_load_relaxed(&m->bindings);

but doesn't look like there's a JL_GC_PUSH there either.

(@vtjnash?)

[JeffBezanson]

Just so I understand, why isn't smallintset_rehash written as follows?

As the comment says, our convention is that it's the caller's job to ensure arguments are rooted (one way or another). So if there's a rooting problem here it's in the caller.

[MilesCranmer]

Thanks

[MilesCranmer]

I guess I'm not sure when this is/is not valid. I'll just throw some more out there... Are any of these potentially missing a JL_GC_PUSH*? Or are all of these fine? They seem to go against the comment's recommendation.

Here's perhaps the most relevant one to @mkitti's stack trace:

julia/src/ircode.c

Lines 339 to 341 in 760b2e5

	write_uint8(s->s, TAG_QUOTENODE);
	jl_value_t *inner = jl_quotenode_value(v);
	// we might need to return this exact object at run time, therefore codegen might

Here are some others in ircode.c from scrolling around. (I'm assuming these are mostly fine for some reasons?)

julia/src/ircode.c

Lines 212 to 213 in 4b90899

	jl_value_t *e = jl_genericmemory_ptr_ref(mem, offset + i);
	jl_encode_value(s, e);

(^Does the flags.arrayelem_isboxed mean its fine or something?)

julia/src/ircode.c

Lines 865 to 867 in 760b2e5

	}
	jl_encode_value_(&s, jl_get_nth_field((jl_value_t*)code, i), copy);
	}

julia/src/ircode.c

Lines 875 to 878 in 760b2e5

	// can't optimize generated functions
	jl_encode_value_(&s, (jl_value_t*)jl_compress_argnames(code->slotnames), 1);
	else
	jl_encode_value(&s, jl_nothing);

julia/src/ircode.c

Lines 489 to 491 in 760b2e5

	jl_svec_t *sv = jl_alloc_svec_uninit(len);
	jl_value_t **data = jl_svec_data(sv);
	for (i = 0; i < len; i++) {

julia/src/ircode.c

Lines 504 to 506 in 760b2e5

	for (i = 0; i < numel; i++) {
	data[i] = jl_decode_value(s);
	}

julia/src/ircode.c

Lines 552 to 554 in 760b2e5

	}
	jl_value_t *aty = jl_decode_value(s);
	jl_array_t *a = jl_alloc_array_nd(aty, dims, ndims);

There's a lot more at this part of the file; will skip. I'm assuming they don't need to be GC'd or something? I wasn't sure because the comment suggests even jl_value_t *x = NULL should be pushed to the GC

[MilesCranmer]

Ah, I just saw the JL_GC_DISABLED. Assuming that has something to do with it!

[vchuravy]

So the issue with GC things is that the effect is often far from the cause.

E.g. what sometimes happens:

• Create a root and derive a raw pointer from it
• Lose root, and GC frees the object
• Julia reuses memory with some other data structure
• Raw pointer is used to write data
• New object is now corrupted and at some point it will blow up in your face.

And the crash report is from the unrelated use of a now corrupted object, but it is not the place of the bug.

So it is paramount to find out:
What object used to be stored here? Which write corrupted it? Who was responsible to hang onto the root.

This is relatively doable with rr and an absolutely pain without. A reliable reproducer helps a lot.

[MilesCranmer]

We can't reproduce on Linux though, so rr isn't available. Is there an equivalent to 'chaos mode' for the garbage collector to try to generate the same bug on Linux? Or something like rr on macOS?

[vchuravy]

Maybe you can try the following:

1. Build Julia in debug mode
2. It looks like lldb will disable address-randomization (ASLR), so when you run the same program in the same session the address you are faulting on ought to be the same.
3. If that is the case you can set a watch point at that memory address and note all the places it was written from.

That's a poor man's rr.

You should also set ENABLE_GDB_LISTENER=1 to get backtraces in lldb that show you which Julia functions are involved.

There are these GC debug options

julia/Make.inc

Lines 80 to 82 in 6f52a98

	# GC debugging options
	WITH_GC_VERIFY := 0
	WITH_GC_DEBUG_ENV := 0

and in theory you should be able to set

julia/src/gc-stock.h

Line 614 in 6f52a98

extern jl_value_t *lostval;

to a pointer and have GC help you find it, but I never really used that methodology.

[MilesCranmer]

I'm afraid I won’t have time to dive into this considering how long it will take me to get familiar with the lower level GC stuff. However, this issue is really worrying to me. With Python 3.13 adoption ramping up, more users will encounter this crash, which will impact the hundreds of packages dependent on the PythonCall.jl bridge, including widely-used libraries like diffeqpy (→ DifferentialEquations.jl), pysr (→ SymbolicRegression.jl), pydata’s sparse (→ Finch.jl), or PyXtal.

It appears likely that there’s some underlying Julia GC or memory issue simply exposed by PythonCall.jl, rather than something unique to their interaction. If anyone is able to try the debugging steps above, it would be great for preventing further disruption in the ecosystem. I’m happy to help test or verify fixes a bit later in the year.

Also - if you are unable to reproduce locally, one option is using https://github.com/mxschmitt/action-tmate in a small test action. It lets you SSH into a GitHub action after a crash. So since the GitHub action macos-latest runners seem to reliably hit the jl_object_id__cold error, it seems a free and simple option for non-macOS users would be to create a small GitHub action and ssh in after the crash, and debug from there.