BUG: Data race in stringdtype_finalize_descr under free-threading #28813

vfdev-5 · 2025-04-24T07:16:46Z

Describe the issue:

We see the following data race report by TSAN in JAX CI 3.13 free-threading:

Report

2025-04-24T05:33:44.6129397Z ==================
2025-04-24T05:33:44.6129802Z WARNING: ThreadSanitizer: data race (pid=52002)
2025-04-24T05:33:44.6130382Z   Read of size 1 at 0x7d8a50380f73 by thread T7 (mutexes: read M0):
2025-04-24T05:33:44.6132017Z     #0 stringdtype_finalize_descr /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/stringdtype/dtype.c:636:17 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x31053c) (BuildId: 598fa1985950a4073a6de7e028dbc8ea4211493e)
2025-04-24T05:33:44.6135116Z     #1 PyArray_NewFromDescr_int /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/ctors.c:665:9 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x24e193) (BuildId: 598fa1985950a4073a6de7e028dbc8ea4211493e)
2025-04-24T05:33:44.6137739Z     #2 PyArray_NewFromDescrAndBase /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/ctors.c:1022:12 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x24fd07) (BuildId: 598fa1985950a4073a6de7e028dbc8ea4211493e)
2025-04-24T05:33:44.6140185Z     #3 PyArray_NewFromDescr /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/ctors.c:1007:12 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x24fd07)
2025-04-24T05:33:44.6142210Z     #4 PyArray_NewLikeArrayWithShape /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/ctors.c (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x24fd07)
2025-04-24T05:33:44.6144619Z     #5 PyArray_NewLikeArray /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/ctors.c:1184:12 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x253f97) (BuildId: 598fa1985950a4073a6de7e028dbc8ea4211493e)
2025-04-24T05:33:44.6147209Z     #6 PyArray_FromArray /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/ctors.c:1950:32 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x253f97)
2025-04-24T05:33:44.6149636Z     #7 PyArray_FromAny_int /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/ctors.c:1605:25 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x25374f) (BuildId: 598fa1985950a4073a6de7e028dbc8ea4211493e)
2025-04-24T05:33:44.6152178Z     #8 PyArray_CheckFromAny_int /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/ctors.c:1830:11 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x2548e7) (BuildId: 598fa1985950a4073a6de7e028dbc8ea4211493e)
2025-04-24T05:33:44.6155030Z     #9 _array_fromobject_generic /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/multiarraymodule.c:1672:28 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x2dbb8d) (BuildId: 598fa1985950a4073a6de7e028dbc8ea4211493e)
2025-04-24T05:33:44.6157699Z     #10 array_asarray /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/multiarraymodule.c:1784:21 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x2d11cf) (BuildId: 598fa1985950a4073a6de7e028dbc8ea4211493e)
2025-04-24T05:33:44.6160179Z     #11 cfunction_vectorcall_FASTCALL_KEYWORDS /__w/jax/jax/cpython/Objects/methodobject.c:441:24 (python3.13+0x28a720) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6162075Z     #12 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eb1fa) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6163532Z     #13 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eb1fa)
2025-04-24T05:33:44.6165038Z     #14 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e3c6b) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6166795Z     #15 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfd9a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6168102Z     #16 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfd9a)
2025-04-24T05:33:44.6169505Z     #17 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb86f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6171312Z     #18 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef785) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6172869Z     #19 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:62:18 (python3.13+0x1ef785)
2025-04-24T05:33:44.6174497Z     #20 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:285:24 (python3.13+0x1eb436) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6175699Z     #21 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb436)
2025-04-24T05:33:44.6176873Z     #22 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb565) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6178606Z     #23 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e5e52) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6180409Z     #24 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfd9a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6181829Z     #25 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfd9a)
2025-04-24T05:33:44.6183054Z     #26 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb86f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6184784Z     #27 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef785) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6186439Z     #28 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:62:18 (python3.13+0x1ef785)
2025-04-24T05:33:44.6187729Z     #29 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:285:24 (python3.13+0x1eb436) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6189052Z     #30 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb436)
2025-04-24T05:33:44.6190235Z     #31 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb565) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6191766Z     #32 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e5e52) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6193488Z     #33 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfd9a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6194906Z     #34 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfd9a)
2025-04-24T05:33:44.6196192Z     #35 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb86f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6197917Z     #36 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef8f0) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6199505Z     #37 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:92:18 (python3.13+0x1ef8f0)
2025-04-24T05:33:44.6200821Z     #38 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb4e3) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6202035Z     #39 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb4e3)
2025-04-24T05:33:44.6203217Z     #40 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb565) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6204898Z     #41 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e5e52) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6206727Z     #42 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfd9a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6208091Z     #43 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfd9a)
2025-04-24T05:33:44.6209445Z     #44 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb86f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6211092Z     #45 _PyObject_VectorcallDictTstate /__w/jax/jax/cpython/Objects/call.c:135:15 (python3.13+0x1ea3ed) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6212970Z     #46 _PyObject_Call_Prepend /__w/jax/jax/cpython/Objects/call.c:504:24 (python3.13+0x1ebee7) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6214592Z     #47 slot_tp_call /__w/jax/jax/cpython/Objects/typeobject.c:9557:15 (python3.13+0x2f98e8) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6216173Z     #48 _PyObject_MakeTpCall /__w/jax/jax/cpython/Objects/call.c:242:18 (python3.13+0x1ea65c) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6217911Z     #49 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eb2b8) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6219456Z     #50 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eb2b8)
2025-04-24T05:33:44.6220804Z     #51 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e3c6b) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6222494Z     #52 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfd9a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6223983Z     #53 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfd9a)
2025-04-24T05:33:44.6225256Z     #54 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb86f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6226847Z     #55 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef83f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6228323Z     #56 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef83f)
2025-04-24T05:33:44.6229754Z     #57 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb4e3) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6231222Z     #58 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb4e3)
2025-04-24T05:33:44.6232370Z     #59 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb565) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6233839Z     #60 thread_run /__w/jax/jax/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x5695b2) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6235454Z     #61 pythread_wrapper /__w/jax/jax/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4c24c7) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6236350Z 
2025-04-24T05:33:44.6236677Z   Previous write of size 1 at 0x7d8a50380f73 by thread T1 (mutexes: read M0):
2025-04-24T05:33:44.6238598Z     #0 stringdtype_finalize_descr /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/stringdtype/dtype.c:637:29 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x31057e) (BuildId: 598fa1985950a4073a6de7e028dbc8ea4211493e)
2025-04-24T05:33:44.6241329Z     #1 PyArray_NewFromDescr_int /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/ctors.c:665:9 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x24e193) (BuildId: 598fa1985950a4073a6de7e028dbc8ea4211493e)
2025-04-24T05:33:44.6243859Z     #2 PyArray_NewFromDescrAndBase /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/ctors.c:1022:12 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x24fd07) (BuildId: 598fa1985950a4073a6de7e028dbc8ea4211493e)
2025-04-24T05:33:44.6246217Z     #3 PyArray_NewFromDescr /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/ctors.c:1007:12 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x24fd07)
2025-04-24T05:33:44.6248191Z     #4 PyArray_NewLikeArrayWithShape /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/ctors.c (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x24fd07)
2025-04-24T05:33:44.6250866Z     #5 PyArray_NewLikeArray /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/ctors.c:1184:12 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x253f97) (BuildId: 598fa1985950a4073a6de7e028dbc8ea4211493e)
2025-04-24T05:33:44.6253077Z     #6 PyArray_FromArray /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/ctors.c:1950:32 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x253f97)
2025-04-24T05:33:44.6255346Z     #7 PyArray_FromAny_int /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/ctors.c:1605:25 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x25374f) (BuildId: 598fa1985950a4073a6de7e028dbc8ea4211493e)
2025-04-24T05:33:44.6257852Z     #8 PyArray_CheckFromAny_int /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/ctors.c:1830:11 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x2548e7) (BuildId: 598fa1985950a4073a6de7e028dbc8ea4211493e)
2025-04-24T05:33:44.6260642Z     #9 _array_fromobject_generic /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/multiarraymodule.c:1672:28 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x2dbb8d) (BuildId: 598fa1985950a4073a6de7e028dbc8ea4211493e)
2025-04-24T05:33:44.6263502Z     #10 array_asarray /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/multiarraymodule.c:1784:21 (_multiarray_umath.cpython-313t-x86_64-linux-gnu.so+0x2d11cf) (BuildId: 598fa1985950a4073a6de7e028dbc8ea4211493e)
2025-04-24T05:33:44.6265868Z     #11 cfunction_vectorcall_FASTCALL_KEYWORDS /__w/jax/jax/cpython/Objects/methodobject.c:441:24 (python3.13+0x28a720) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6267791Z     #12 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eb1fa) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6269326Z     #13 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eb1fa)
2025-04-24T05:33:44.6270752Z     #14 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e3c6b) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6272464Z     #15 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfd9a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6273815Z     #16 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfd9a)
2025-04-24T05:33:44.6275139Z     #17 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb86f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6276871Z     #18 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef785) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6278323Z     #19 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:62:18 (python3.13+0x1ef785)
2025-04-24T05:33:44.6279926Z     #20 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:285:24 (python3.13+0x1eb436) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6281148Z     #21 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb436)
2025-04-24T05:33:44.6282343Z     #22 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb565) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6284008Z     #23 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e5e52) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6285850Z     #24 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfd9a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6287184Z     #25 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfd9a)
2025-04-24T05:33:44.6288528Z     #26 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb86f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6290393Z     #27 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef785) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6291843Z     #28 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:62:18 (python3.13+0x1ef785)
2025-04-24T05:33:44.6293102Z     #29 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:285:24 (python3.13+0x1eb436) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6294296Z     #30 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb436)
2025-04-24T05:33:44.6295526Z     #31 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb565) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6297154Z     #32 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e5e52) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6327066Z     #33 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfd9a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6329305Z     #34 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfd9a)
2025-04-24T05:33:44.6331054Z     #35 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb86f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6332755Z     #36 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef8f0) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6334189Z     #37 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:92:18 (python3.13+0x1ef8f0)
2025-04-24T05:33:44.6335377Z     #38 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb4e3) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6336659Z     #39 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb4e3)
2025-04-24T05:33:44.6337985Z     #40 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb565) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6339879Z     #41 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e5e52) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6341702Z     #42 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfd9a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6343047Z     #43 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfd9a)
2025-04-24T05:33:44.6344367Z     #44 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb86f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6346405Z     #45 _PyObject_VectorcallDictTstate /__w/jax/jax/cpython/Objects/call.c:135:15 (python3.13+0x1ea3ed) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6348090Z     #46 _PyObject_Call_Prepend /__w/jax/jax/cpython/Objects/call.c:504:24 (python3.13+0x1ebee7) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6349825Z     #47 slot_tp_call /__w/jax/jax/cpython/Objects/typeobject.c:9557:15 (python3.13+0x2f98e8) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6351398Z     #48 _PyObject_MakeTpCall /__w/jax/jax/cpython/Objects/call.c:242:18 (python3.13+0x1ea65c) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6353108Z     #49 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eb2b8) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6354590Z     #50 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eb2b8)
2025-04-24T05:33:44.6356082Z     #51 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e3c6b) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6358204Z     #52 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfd9a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6359771Z     #53 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfd9a)
2025-04-24T05:33:44.6361042Z     #54 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb86f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6362755Z     #55 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef83f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6364263Z     #56 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef83f)
2025-04-24T05:33:44.6365633Z     #57 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb4e3) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6366870Z     #58 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb4e3)
2025-04-24T05:33:44.6368096Z     #59 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb565) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6370006Z     #60 thread_run /__w/jax/jax/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x5695b2) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6371593Z     #61 pythread_wrapper /__w/jax/jax/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4c24c7) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6372468Z 
2025-04-24T05:33:44.6372630Z   Mutex M0 (0x7d8a4d258908) created at:
2025-04-24T05:33:44.6373380Z     #0 __tsan_mutex_create <null> (python3.13+0x130623) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6382813Z     #1 absl::lts_20230802::Mutex::Mutex() /proc/self/cwd/external/com_google_absl/absl/synchronization/mutex.h:1022:3 (libjax_common.so+0xfb04455) (BuildId: 8b2f41ae7ee286b8fad5de461fc7db05574249cb)
2025-04-24T05:33:44.6386011Z     #2 void nanobind::init<>::execute<nanobind::class_<absl::lts_20230802::Mutex>>(nanobind::class_<absl::lts_20230802::Mutex>&)::'lambda'(nanobind::pointer_and_handle<absl::lts_20230802::Mutex>)::operator()(nanobind::pointer_and_handle<absl::lts_20230802::Mutex>) const /proc/self/cwd/external/nanobind/include/nanobind/nb_class.h:362:36 (libjax_common.so+0xfb04455)
2025-04-24T05:33:44.6393343Z     #3 _object* nanobind::detail::func_create<false, true, void nanobind::init<>::execute<nanobind::class_<absl::lts_20230802::Mutex>>(nanobind::class_<absl::lts_20230802::Mutex>&)::'lambda'(nanobind::pointer_and_handle<absl::lts_20230802::Mutex>), void, nanobind::pointer_and_handle<absl::lts_20230802::Mutex>, 0ul, nanobind::scope, nanobind::name, nanobind::is_method>(void nanobind::init<>::execute<nanobind::class_<absl::lts_20230802::Mutex>>(nanobind::class_<absl::lts_20230802::Mutex>&)::'lambda'(nanobind::pointer_and_handle<absl::lts_20230802::Mutex>)&&, void (*)(nanobind::pointer_and_handle<absl::lts_20230802::Mutex>), std::integer_sequence<unsigned long, 0ul>, nanobind::scope const&, nanobind::name const&, nanobind::is_method const&)::'lambda'(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*)::operator()(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*) const /proc/self/cwd/external/nanobind/include/nanobind/nb_func.h:264:13 (libjax_common.so+0xfb04455)
2025-04-24T05:33:44.6404010Z     #4 _object* nanobind::detail::func_create<false, true, void nanobind::init<>::execute<nanobind::class_<absl::lts_20230802::Mutex>>(nanobind::class_<absl::lts_20230802::Mutex>&)::'lambda'(nanobind::pointer_and_handle<absl::lts_20230802::Mutex>), void, nanobind::pointer_and_handle<absl::lts_20230802::Mutex>, 0ul, nanobind::scope, nanobind::name, nanobind::is_method>(void nanobind::init<>::execute<nanobind::class_<absl::lts_20230802::Mutex>>(nanobind::class_<absl::lts_20230802::Mutex>&)::'lambda'(nanobind::pointer_and_handle<absl::lts_20230802::Mutex>)&&, void (*)(nanobind::pointer_and_handle<absl::lts_20230802::Mutex>), std::integer_sequence<unsigned long, 0ul>, nanobind::scope const&, nanobind::name const&, nanobind::is_method const&)::'lambda'(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*)::__invoke(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*) /proc/self/cwd/external/nanobind/include/nanobind/nb_func.h:219:14 (libjax_common.so+0xfb04455)
2025-04-24T05:33:44.6411643Z     #5 nanobind::detail::nb_func_vectorcall_simple(_object*, _object* const*, unsigned long, _object*) /proc/self/cwd/external/nanobind/src/nb_func.cpp:915:26 (libjax_common.so+0xca5f7d2) (BuildId: 8b2f41ae7ee286b8fad5de461fc7db05574249cb)
2025-04-24T05:33:44.6414248Z     #6 nanobind::detail::nb_type_vectorcall(_object*, _object* const*, unsigned long, _object*) /proc/self/cwd/external/nanobind/src/nb_type.cpp:1030:9 (libjax_common.so+0xca6fc5e) (BuildId: 8b2f41ae7ee286b8fad5de461fc7db05574249cb)
2025-04-24T05:33:44.6416601Z     #7 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eb1fa) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6418242Z     #8 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eb1fa)
2025-04-24T05:33:44.6419783Z     #9 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e3c6b) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6421659Z     #10 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfa72) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6423052Z     #11 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfa72)
2025-04-24T05:33:44.6423959Z     #12 PyEval_EvalCode /__w/jax/jax/cpython/Python/ceval.c:604:21 (python3.13+0x3dfa72)
2025-04-24T05:33:44.6425281Z     #13 builtin_exec_impl /__w/jax/jax/cpython/Python/bltinmodule.c:1143:17 (python3.13+0x3d9ea9) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6426635Z     #14 builtin_exec /__w/jax/jax/cpython/Python/clinic/bltinmodule.c.h:556:20 (python3.13+0x3d9ea9)
2025-04-24T05:33:44.6428148Z     #15 cfunction_vectorcall_FASTCALL_KEYWORDS /__w/jax/jax/cpython/Objects/methodobject.c:441:24 (python3.13+0x28a720) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6430015Z     #16 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb4e3) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6431421Z     #17 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb4e3)
2025-04-24T05:33:44.6432578Z     #18 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb565) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6434197Z     #19 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e5e52) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6436039Z     #20 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfd9a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6437365Z     #21 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfd9a)
2025-04-24T05:33:44.6438716Z     #22 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb86f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6440413Z     #23 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ed8a0) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6441795Z     #24 object_vacall /__w/jax/jax/cpython/Objects/call.c:819:14 (python3.13+0x1ed8a0)
2025-04-24T05:33:44.6443105Z     #25 PyObject_CallMethodObjArgs /__w/jax/jax/cpython/Objects/call.c:880:24 (python3.13+0x1ed54b) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6444972Z     #26 import_find_and_load /__w/jax/jax/cpython/Python/import.c:3704:11 (python3.13+0x4626b2) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6446374Z     #27 PyImport_ImportModuleLevelObject /__w/jax/jax/cpython/Python/import.c:3786:15 (python3.13+0x4626b2)
2025-04-24T05:33:44.6447708Z     #28 import_name /__w/jax/jax/cpython/Python/ceval.c:2731:16 (python3.13+0x3edb87) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6449178Z     #29 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:3205:19 (python3.13+0x3edb87)
2025-04-24T05:33:44.6450688Z     #30 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfa72) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6452008Z     #31 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfa72)
2025-04-24T05:33:44.6452962Z     #32 PyEval_EvalCode /__w/jax/jax/cpython/Python/ceval.c:604:21 (python3.13+0x3dfa72)
2025-04-24T05:33:44.6454299Z     #33 builtin_exec_impl /__w/jax/jax/cpython/Python/bltinmodule.c:1143:17 (python3.13+0x3d9ea9) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6455875Z     #34 builtin_exec /__w/jax/jax/cpython/Python/clinic/bltinmodule.c.h:556:20 (python3.13+0x3d9ea9)
2025-04-24T05:33:44.6457398Z     #35 cfunction_vectorcall_FASTCALL_KEYWORDS /__w/jax/jax/cpython/Objects/methodobject.c:441:24 (python3.13+0x28a720) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6459223Z     #36 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb4e3) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6460408Z     #37 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb4e3)
2025-04-24T05:33:44.6461572Z     #38 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb565) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6463180Z     #39 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e5e52) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6465023Z     #40 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfd9a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6466334Z     #41 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfd9a)
2025-04-24T05:33:44.6467529Z     #42 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb86f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6469258Z     #43 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ed8a0) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6470588Z     #44 object_vacall /__w/jax/jax/cpython/Objects/call.c:819:14 (python3.13+0x1ed8a0)
2025-04-24T05:33:44.6471821Z     #45 PyObject_CallMethodObjArgs /__w/jax/jax/cpython/Objects/call.c:880:24 (python3.13+0x1ed54b) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6473521Z     #46 import_find_and_load /__w/jax/jax/cpython/Python/import.c:3704:11 (python3.13+0x4626b2) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6475079Z     #47 PyImport_ImportModuleLevelObject /__w/jax/jax/cpython/Python/import.c:3786:15 (python3.13+0x4626b2)
2025-04-24T05:33:44.6476507Z     #48 builtin___import___impl /__w/jax/jax/cpython/Python/bltinmodule.c:277:12 (python3.13+0x3d81f4) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6477908Z     #49 builtin___import__ /__w/jax/jax/cpython/Python/clinic/bltinmodule.c.h:107:20 (python3.13+0x3d81f4)
2025-04-24T05:33:44.6479559Z     #50 cfunction_vectorcall_FASTCALL_KEYWORDS /__w/jax/jax/cpython/Objects/methodobject.c:441:24 (python3.13+0x28a720) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6481495Z     #51 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb4e3) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6482676Z     #52 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb4e3)
2025-04-24T05:33:44.6483854Z     #53 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb565) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6485538Z     #54 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e5e52) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6487298Z     #55 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfd9a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6488723Z     #56 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfd9a)
2025-04-24T05:33:44.6489940Z     #57 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb86f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6491652Z     #58 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ed8a0) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6493178Z     #59 object_vacall /__w/jax/jax/cpython/Objects/call.c:819:14 (python3.13+0x1ed8a0)
2025-04-24T05:33:44.6494533Z     #60 PyObject_CallMethodObjArgs /__w/jax/jax/cpython/Objects/call.c:880:24 (python3.13+0x1ed54b) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6496266Z     #61 PyImport_ImportModuleLevelObject /__w/jax/jax/cpython/Python/import.c:3854:25 (python3.13+0x461db2) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6497875Z     #62 import_name /__w/jax/jax/cpython/Python/ceval.c:2731:16 (python3.13+0x3edb87) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6499392Z     #63 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:3205:19 (python3.13+0x3edb87)
2025-04-24T05:33:44.6500868Z     #64 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfa72) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6502170Z     #65 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfa72)
2025-04-24T05:33:44.6503079Z     #66 PyEval_EvalCode /__w/jax/jax/cpython/Python/ceval.c:604:21 (python3.13+0x3dfa72)
2025-04-24T05:33:44.6504364Z     #67 run_eval_code_obj /__w/jax/jax/cpython/Python/pythonrun.c:1381:9 (python3.13+0x4a424e) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6505872Z     #68 run_mod /__w/jax/jax/cpython/Python/pythonrun.c:1466:19 (python3.13+0x4a3975) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6507369Z     #69 pyrun_file /__w/jax/jax/cpython/Python/pythonrun.c:1295:15 (python3.13+0x49f485) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6508779Z     #70 _PyRun_SimpleFileObject /__w/jax/jax/cpython/Python/pythonrun.c:517:13 (python3.13+0x49f485)
2025-04-24T05:33:44.6510142Z     #71 _PyRun_AnyFileObject /__w/jax/jax/cpython/Python/pythonrun.c:77:15 (python3.13+0x49ebd8) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6511727Z     #72 pymain_run_file_obj /__w/jax/jax/cpython/Modules/main.c:410:15 (python3.13+0x4dc7bf) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6512924Z     #73 pymain_run_file /__w/jax/jax/cpython/Modules/main.c:429:15 (python3.13+0x4dc7bf)
2025-04-24T05:33:44.6514132Z     #74 pymain_run_python /__w/jax/jax/cpython/Modules/main.c:696:21 (python3.13+0x4dba0c) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6515373Z     #75 Py_RunMain /__w/jax/jax/cpython/Modules/main.c:775:5 (python3.13+0x4dba0c)
2025-04-24T05:33:44.6516494Z     #76 pymain_main /__w/jax/jax/cpython/Modules/main.c:805:12 (python3.13+0x4dbe48) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6518170Z     #77 Py_BytesMain /__w/jax/jax/cpython/Modules/main.c:829:12 (python3.13+0x4dbecb) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6519864Z     #78 main /__w/jax/jax/cpython/./Programs/python.c:15:12 (python3.13+0x15c7eb) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6520636Z 
2025-04-24T05:33:44.6520886Z   Thread T7 (tid=56695, running) created by main thread at:
2025-04-24T05:33:44.6521713Z     #0 pthread_create <null> (python3.13+0xde1df) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6523066Z     #1 do_start_joinable_thread /__w/jax/jax/cpython/Python/thread_pthread.h:290:14 (python3.13+0x4c1378) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6524841Z     #2 PyThread_start_joinable_thread /__w/jax/jax/cpython/Python/thread_pthread.h:332:9 (python3.13+0x4c119a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6526541Z     #3 ThreadHandle_start /__w/jax/jax/cpython/./Modules/_threadmodule.c:422:9 (python3.13+0x569147) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6527861Z     #4 do_start_new_thread /__w/jax/jax/cpython/./Modules/_threadmodule.c:1849:9 (python3.13+0x569147)
2025-04-24T05:33:44.6529596Z     #5 thread_PyThread_start_joinable_thread /__w/jax/jax/cpython/./Modules/_threadmodule.c:1972:14 (python3.13+0x568241) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6531351Z     #6 cfunction_call /__w/jax/jax/cpython/Objects/methodobject.c:540:18 (python3.13+0x28b3b7) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6532904Z     #7 _PyObject_MakeTpCall /__w/jax/jax/cpython/Objects/call.c:242:18 (python3.13+0x1ea65c) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6534644Z     #8 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eb2b8) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6536040Z     #9 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eb2b8)
2025-04-24T05:33:44.6537425Z     #10 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1502:19 (python3.13+0x3e68ed) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6539317Z     #11 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfd9a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6540625Z     #12 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfd9a)
2025-04-24T05:33:44.6541849Z     #13 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb86f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6543528Z     #14 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef8f0) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6545029Z     #15 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:92:18 (python3.13+0x1ef8f0)
2025-04-24T05:33:44.6546323Z     #16 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb4e3) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6547537Z     #17 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb4e3)
2025-04-24T05:33:44.6548826Z     #18 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb565) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6550454Z     #19 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e5e52) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6552198Z     #20 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfd9a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6553513Z     #21 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfd9a)
2025-04-24T05:33:44.6554955Z     #22 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb86f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6556559Z     #23 _PyObject_VectorcallDictTstate /__w/jax/jax/cpython/Objects/call.c:135:15 (python3.13+0x1ea3ed) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6558172Z     #24 _PyObject_Call_Prepend /__w/jax/jax/cpython/Objects/call.c:504:24 (python3.13+0x1ebee7) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6559776Z     #25 slot_tp_call /__w/jax/jax/cpython/Objects/typeobject.c:9557:15 (python3.13+0x2f98e8) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6561318Z     #26 _PyObject_MakeTpCall /__w/jax/jax/cpython/Objects/call.c:242:18 (python3.13+0x1ea65c) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6563010Z     #27 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eb2b8) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6564498Z     #28 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eb2b8)
2025-04-24T05:33:44.6566091Z     #29 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e3c6b) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6567872Z     #30 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfd9a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6569337Z     #31 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfd9a)
2025-04-24T05:33:44.6570586Z     #32 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb86f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6572199Z     #33 _PyObject_VectorcallDictTstate /__w/jax/jax/cpython/Objects/call.c:146:15 (python3.13+0x1ea3b0) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6573832Z     #34 _PyObject_Call_Prepend /__w/jax/jax/cpython/Objects/call.c:504:24 (python3.13+0x1ebee7) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6575466Z     #35 slot_tp_init /__w/jax/jax/cpython/Objects/typeobject.c:9803:15 (python3.13+0x2fb0ac) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6576993Z     #36 type_call /__w/jax/jax/cpython/Objects/typeobject.c:1997:19 (python3.13+0x2ebf4c) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6578567Z     #37 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:361:18 (python3.13+0x1eb49b) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6580016Z     #38 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb565) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6581616Z     #39 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e5e52) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6583339Z     #40 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfa72) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6584722Z     #41 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfa72)
2025-04-24T05:33:44.6585631Z     #42 PyEval_EvalCode /__w/jax/jax/cpython/Python/ceval.c:604:21 (python3.13+0x3dfa72)
2025-04-24T05:33:44.6586887Z     #43 run_eval_code_obj /__w/jax/jax/cpython/Python/pythonrun.c:1381:9 (python3.13+0x4a424e) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6588513Z     #44 run_mod /__w/jax/jax/cpython/Python/pythonrun.c:1466:19 (python3.13+0x4a3975) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6589994Z     #45 pyrun_file /__w/jax/jax/cpython/Python/pythonrun.c:1295:15 (python3.13+0x49f485) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6591464Z     #46 _PyRun_SimpleFileObject /__w/jax/jax/cpython/Python/pythonrun.c:517:13 (python3.13+0x49f485)
2025-04-24T05:33:44.6592812Z     #47 _PyRun_AnyFileObject /__w/jax/jax/cpython/Python/pythonrun.c:77:15 (python3.13+0x49ebd8) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6594422Z     #48 pymain_run_file_obj /__w/jax/jax/cpython/Modules/main.c:410:15 (python3.13+0x4dc7bf) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6595629Z     #49 pymain_run_file /__w/jax/jax/cpython/Modules/main.c:429:15 (python3.13+0x4dc7bf)
2025-04-24T05:33:44.6596827Z     #50 pymain_run_python /__w/jax/jax/cpython/Modules/main.c:696:21 (python3.13+0x4dba0c) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6598001Z     #51 Py_RunMain /__w/jax/jax/cpython/Modules/main.c:775:5 (python3.13+0x4dba0c)
2025-04-24T05:33:44.6599247Z     #52 pymain_main /__w/jax/jax/cpython/Modules/main.c:805:12 (python3.13+0x4dbe48) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6600696Z     #53 Py_BytesMain /__w/jax/jax/cpython/Modules/main.c:829:12 (python3.13+0x4dbecb) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6602092Z     #54 main /__w/jax/jax/cpython/./Programs/python.c:15:12 (python3.13+0x15c7eb) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6603028Z 
2025-04-24T05:33:44.6603291Z   Thread T1 (tid=56689, running) created by main thread at:
2025-04-24T05:33:44.6604105Z     #0 pthread_create <null> (python3.13+0xde1df) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6605569Z     #1 do_start_joinable_thread /__w/jax/jax/cpython/Python/thread_pthread.h:290:14 (python3.13+0x4c1378) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6607285Z     #2 PyThread_start_joinable_thread /__w/jax/jax/cpython/Python/thread_pthread.h:332:9 (python3.13+0x4c119a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6609003Z     #3 ThreadHandle_start /__w/jax/jax/cpython/./Modules/_threadmodule.c:422:9 (python3.13+0x569147) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6610963Z     #4 do_start_new_thread /__w/jax/jax/cpython/./Modules/_threadmodule.c:1849:9 (python3.13+0x569147)
2025-04-24T05:33:44.6612479Z     #5 thread_PyThread_start_joinable_thread /__w/jax/jax/cpython/./Modules/_threadmodule.c:1972:14 (python3.13+0x568241) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6614391Z     #6 cfunction_call /__w/jax/jax/cpython/Objects/methodobject.c:540:18 (python3.13+0x28b3b7) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6615986Z     #7 _PyObject_MakeTpCall /__w/jax/jax/cpython/Objects/call.c:242:18 (python3.13+0x1ea65c) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6618092Z     #8 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eb2b8) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6619644Z     #9 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eb2b8)
2025-04-24T05:33:44.6621055Z     #10 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1502:19 (python3.13+0x3e68ed) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6622810Z     #11 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfd9a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6624125Z     #12 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfd9a)
2025-04-24T05:33:44.6625452Z     #13 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb86f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6627150Z     #14 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef8f0) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6628968Z     #15 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:92:18 (python3.13+0x1ef8f0)
2025-04-24T05:33:44.6630276Z     #16 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb4e3) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6631694Z     #17 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb4e3)
2025-04-24T05:33:44.6632845Z     #18 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb565) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6634519Z     #19 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e5e52) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6636236Z     #20 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfd9a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6637560Z     #21 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfd9a)
2025-04-24T05:33:44.6638918Z     #22 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb86f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6640714Z     #23 _PyObject_VectorcallDictTstate /__w/jax/jax/cpython/Objects/call.c:135:15 (python3.13+0x1ea3ed) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6642374Z     #24 _PyObject_Call_Prepend /__w/jax/jax/cpython/Objects/call.c:504:24 (python3.13+0x1ebee7) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6643961Z     #25 slot_tp_call /__w/jax/jax/cpython/Objects/typeobject.c:9557:15 (python3.13+0x2f98e8) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6645579Z     #26 _PyObject_MakeTpCall /__w/jax/jax/cpython/Objects/call.c:242:18 (python3.13+0x1ea65c) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6647347Z     #27 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eb2b8) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6648899Z     #28 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eb2b8)
2025-04-24T05:33:44.6650307Z     #29 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e3c6b) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6652038Z     #30 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfd9a) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6653364Z     #31 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfd9a)
2025-04-24T05:33:44.6654657Z     #32 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb86f) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6656290Z     #33 _PyObject_VectorcallDictTstate /__w/jax/jax/cpython/Objects/call.c:146:15 (python3.13+0x1ea3b0) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6657951Z     #34 _PyObject_Call_Prepend /__w/jax/jax/cpython/Objects/call.c:504:24 (python3.13+0x1ebee7) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6659677Z     #35 slot_tp_init /__w/jax/jax/cpython/Objects/typeobject.c:9803:15 (python3.13+0x2fb0ac) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6661210Z     #36 type_call /__w/jax/jax/cpython/Objects/typeobject.c:1997:19 (python3.13+0x2ebf4c) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6662650Z     #37 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:361:18 (python3.13+0x1eb49b) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6664064Z     #38 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb565) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6665754Z     #39 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e5e52) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6667745Z     #40 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dfa72) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6669186Z     #41 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1816:12 (python3.13+0x3dfa72)
2025-04-24T05:33:44.6670097Z     #42 PyEval_EvalCode /__w/jax/jax/cpython/Python/ceval.c:604:21 (python3.13+0x3dfa72)
2025-04-24T05:33:44.6671368Z     #43 run_eval_code_obj /__w/jax/jax/cpython/Python/pythonrun.c:1381:9 (python3.13+0x4a424e) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6672861Z     #44 run_mod /__w/jax/jax/cpython/Python/pythonrun.c:1466:19 (python3.13+0x4a3975) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6674433Z     #45 pyrun_file /__w/jax/jax/cpython/Python/pythonrun.c:1295:15 (python3.13+0x49f485) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6675768Z     #46 _PyRun_SimpleFileObject /__w/jax/jax/cpython/Python/pythonrun.c:517:13 (python3.13+0x49f485)
2025-04-24T05:33:44.6677148Z     #47 _PyRun_AnyFileObject /__w/jax/jax/cpython/Python/pythonrun.c:77:15 (python3.13+0x49ebd8) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6678958Z     #48 pymain_run_file_obj /__w/jax/jax/cpython/Modules/main.c:410:15 (python3.13+0x4dc7bf) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6680180Z     #49 pymain_run_file /__w/jax/jax/cpython/Modules/main.c:429:15 (python3.13+0x4dc7bf)
2025-04-24T05:33:44.6681404Z     #50 pymain_run_python /__w/jax/jax/cpython/Modules/main.c:696:21 (python3.13+0x4dba0c) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6682547Z     #51 Py_RunMain /__w/jax/jax/cpython/Modules/main.c:775:5 (python3.13+0x4dba0c)
2025-04-24T05:33:44.6683687Z     #52 pymain_main /__w/jax/jax/cpython/Modules/main.c:805:12 (python3.13+0x4dbe48) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6685195Z     #53 Py_BytesMain /__w/jax/jax/cpython/Modules/main.c:829:12 (python3.13+0x4dbecb) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6686533Z     #54 main /__w/jax/jax/cpython/./Programs/python.c:15:12 (python3.13+0x15c7eb) (BuildId: 85ee8e4444c760229a8aeea440c1c91756ac668b)
2025-04-24T05:33:44.6687263Z 
2025-04-24T05:33:44.6688085Z SUMMARY: ThreadSanitizer: data race /__w/jax/jax/numpy/.mesonpy-5c3ku1pk/../numpy/_core/src/multiarray/stringdtype/dtype.c:636:17 in stringdtype_finalize_descr
2025-04-24T05:33:44.6689263Z ==================

cc @hawkinsp

Reproduce the code example:

To be provided

Error message:

Python and NumPy Versions:

Python 3.13.3+ experimental free-threading build (heads/3.13:5ab628f, Apr 20 2025, 05:24:54) [Clang 18.1.3 (1ubuntu1)]
Numpy: from source, 9ead596

Runtime Environment:

No response

Context for the issue:

JAX CI Link: https://github.com/jax-ml/jax/actions/runs/14633862367/job/41061013541

The text was updated successfully, but these errors were encountered:

seberg · 2025-04-24T08:52:34Z

Do you think this is most likely a recent regression or just randomly showing up now?

vfdev-5 · 2025-04-24T09:03:47Z

it is the first time I see this race in JAX CI

ngoldbaum · 2025-04-24T15:45:44Z

Weird, I can't think of a way for two threads to get a handle on the same newly created StringDType instance from the array creation logic.

If you have any idea which test triggered this that would probably help.

vfdev-5 · 2025-04-29T09:40:18Z

I'm trying to get a reproducer.
Essentially, JAX tests reporting the race are doing something like:

    numpy_string_array = np.array(
        ["abcdefghijklmnopqrstuvwxyz", "cba"], dtype=np.dtypes.StringDType()  # type: ignore
    )
    jax_string_array = jnp.asarray(numpy_string_array, device=cpu_devices[0])
    # or
    # jax_string_array = jax.device_put(numpy_string_array, device=cpu_devices[0])
    # or
    # jax_string_array = jnp.asarray(
    #     numpy_string_array,
    #     device=cpu_devices[0],
    #     dtype=np.dtypes.StringDType(),
    # )

and debugging a bit the numpy code when creating such a string array, dtype is set up as following:

numpy/numpy/_core/src/multiarray/arraytypes.c.src

Line 4305 in e05ec39

ret = (PyArray_Descr *)_builtin_descrs[num];

Seems like dtype is fetched from a global structure _builtin_descrs and maybe then this code where the data race is happened is modifying the global object?

numpy/numpy/_core/src/multiarray/stringdtype/dtype.c

Lines 636 to 637 in e05ec39

    
           if (sdtype->array_owned == 0) { 
        
               sdtype->array_owned = 1;

seberg · 2025-04-29T13:44:02Z

Hmmm, @ngoldbaum maybe the way things were we set the global singleton exactly once from array_owned=0 to array_owned=1?
Although... even if not the global one, this code does look like it may be racy if the same dtype instance was used in multiple threads.

ngoldbaum · 2025-04-29T13:48:52Z

Well, we have a lock hanging off the descriptor. Might as well use it...

Thanks for tracking this down @vfdev-5.

ngoldbaum · 2025-04-29T14:11:20Z

and debugging a bit the numpy code when creating such a string array, dtype is set up as following:

I'm confused by this - how is execution accessing _builtin_descrs?

If you look slightly higher up in the implementation of PyArray_DescrFromType, you'll see there's a check for stringdtype:

numpy/numpy/_core/src/multiarray/arraytypes.c.src

Lines 4250 to 4267 in 80d5607

    
           PyArray_DescrFromType(int type) 
        
           { 
        
               PyArray_Descr *ret = NULL; 
        
               npy_bool is_stringdtype = (type == NPY_VSTRING || type == NPY_VSTRINGLTR); 
        
               if (type < 0) { 
        
                   /* 
        
                    * It's not valid for type to be less than 0. 
        
                    * If that happens, then no other branch of 
        
                    * this if/else chain should be followed. 
        
                    * This is effectively a no-op that ensures 
        
                    * the default error is raised. 
        
                    */ 
        
                   ret = NULL; 
        
               } 
        
               else if  (is_stringdtype) { 
        
                   ret = (PyArray_Descr *)new_stringdtype_instance(NULL, 1); 
        
               }

So there's probably something else going on in the case that races for Jax.

One thing I can do is set the array_owned flag on the singleton instance. Maybe there's a code path that somehow associates that instance with an array?

I can also use locking as discussed above.

seberg · 2025-04-29T14:27:03Z

Maybe there's a code path that somehow associates that instance with an array?

If array_owned=0 for the singleton instance, I suppose it is very likely that we end up using the singleton somewhere and then attach it basically on the first usage ever. So changing that sounds like a good idea to me either way (maybe orthogonal to the locking, which is probably still needed?).

ngoldbaum · 2025-04-29T14:46:47Z

I created #28862

vfdev-5 added the 00 - Bug label Apr 24, 2025

tylerjereddy added the 39 - free-threading PRs and issues related to support for free-threading CPython (a.k.a. no-GIL, PEP 703) label Apr 24, 2025

ngoldbaum mentioned this issue Apr 29, 2025

BUG: fix stringdtype singleton thread safety #28862

Merged

seberg closed this as completed in #28862 Apr 29, 2025

charris mentioned this issue Apr 29, 2025

BUG: fix stringdtype singleton thread safety #28864

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

BUG: Data race in stringdtype_finalize_descr under free-threading #28813

BUG: Data race in stringdtype_finalize_descr under free-threading #28813

vfdev-5 commented Apr 24, 2025 •

edited

Loading

seberg commented Apr 24, 2025

Uh oh!

vfdev-5 commented Apr 24, 2025

Uh oh!

ngoldbaum commented Apr 24, 2025

Uh oh!

vfdev-5 commented Apr 29, 2025

Uh oh!

seberg commented Apr 29, 2025

Uh oh!

ngoldbaum commented Apr 29, 2025

Uh oh!

ngoldbaum commented Apr 29, 2025

Uh oh!

seberg commented Apr 29, 2025

Uh oh!

ngoldbaum commented Apr 29, 2025

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Uh oh!

BUG: Data race in stringdtype_finalize_descr under free-threading #28813

BUG: Data race in stringdtype_finalize_descr under free-threading #28813

Comments

vfdev-5 commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the issue:

Reproduce the code example:

Error message:

Python and NumPy Versions:

Runtime Environment:

Context for the issue:

seberg commented Apr 24, 2025

Uh oh!

vfdev-5 commented Apr 24, 2025

Uh oh!

ngoldbaum commented Apr 24, 2025

Uh oh!

vfdev-5 commented Apr 29, 2025

Uh oh!

seberg commented Apr 29, 2025

Uh oh!

ngoldbaum commented Apr 29, 2025

Uh oh!

ngoldbaum commented Apr 29, 2025

Uh oh!

seberg commented Apr 29, 2025

Uh oh!

ngoldbaum commented Apr 29, 2025

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

vfdev-5 commented Apr 24, 2025 •

edited

Loading