You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Crash when concurrently writing with print and concurrently modifying sys.stdout #130163
Playing with the code from #130148, I stumbled on a segfault in a free-threaded debug build with the following (seems to trigger faster/more consistently when pasted in the REPL):
Introduce a _PySys_GetAttrRef that returns a new reference instead of a borrowed reference. Use that instead.
We should audit all the uses of _PySys_GetAttr and PySys_GetObject. I expect most of them will need to be replaced with functions that return new references, but it doesn't all have to be in a single PR.
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
changed the title [-]Segfault in a freethreaded JIT build from using `contextlib.redirect_stdout` in threads[/-][+]Segfault in a freethreaded build from using `contextlib.redirect_stdout` in threads[/+]on Feb 15, 2025
) from T2 decrements shared ref_cnt and got 0 and do _Py_Dealloc
Py_DECREF from T1 decrements local ref_cnt, got 0 and go to _Py_MergeZeroLocalRefcount, see that shared ref_cnt is 0 and go to fast path and do _Py_Dealloc.
This is double free and so segfault.
The easiest way to fix this is probably to protect PyFile_WriteObject with a critical section. However, this might slightly affect the performance of single-threaded apps.
The best solution might be to update the documentation and point out this potential issue with redirect_*.
The easiest way to fix this is probably to protect PyFile_WriteObject with a critical section. However, this might slightly affect the performance of single-threaded apps.
Would making the presence of the critical section dependent on whether the build is freethreaded (and/or the GIL is disabled) avoid the performance impact for gilfull builds? Or would it be a bad idea to have this conditional difference in behavior?
The best solution might be to update the documentation and point out this potential issue with redirect_*.
ISTM that fixing the segfault is necessary, even if a note is added to docs, because users may hit the segfault without using redirect_*.
I can get a "segfault" error when I use sys_displayhook, but not where it's used.
For sys_write, sys_format - I have checked all the places where they called, and most of the locations are safe. However, a few places use %s specifier, in such places it is very tricky to get segfault.
get_warnoptions, PySys_ResetWarnOptions, _PySys_AddXOptionWithError - are deprecated, but it is a very tricky to get segfault too.
flush_io_stream, PyErr_Display, pyrun_one_parse_ast, print_expr, write_unraisable_exc, thread_excepthook - very tight - They use a borrowed reference right after getting. It might be possible to reproduce it under heavy contention, but I wasn't able to do so.
_PyRun_InteractiveLoopObject only checks presence of ps1 and ps2.
flush_std_files used from _Py_Finalize and from fatal_error. For _Py_Finalize I can repro, for fatal_error - no.
whichmodule, _Py_FindSourceFile iterate borrowed reference, but I wasn't able to reproduce segfault.
Keep or not before raised exception - in the most places _PySys_GetAttr called without an exception set.
Also, can you point me out what to do next? I can replace _PySys_GetAttr with _PySys_GetAttrRef, but in the most places it is interleave with @serhiy-storchaka PRs. I'm not sure what the correct way to deal with this situation is.
#130503 is an adaptation of #111035 or #129736 that does not introduce new public API and does not change the current behavior (except fixing crashes, mainly). It will make minimal difference with #111035 or #129736, whatever will be accepted for future Python version.
I added also tests for print(), input() and sys.displayhook() (they are relatively simple). Reproducers for other crashes are harder, so I do not think that this is worth hassle.
Thank you for your tests, @sergey-miryanov, but sorry, I have not used them. They are high quality, but too large and expensive. It is not worth to keep them forever after fixing this issue.
@serhiy-storchaka I simplified it a bit yesterday, not sure if you saw the latest changes. Maybe it will be useful, but it is all up to you. Anyway, it was very interesting.
Activity
[-]Segfault in a freethreaded JIT build from using `contextlib.redirect_stdout` in threads[/-][+]Segfault in a freethreaded build from using `contextlib.redirect_stdout` in threads[/+]picnixz commentedon Feb 15, 2025
The issue could be with
StringIO()orsys.stdout/sys.stderror with context managers in general. I don't know where to start to investigate this =/tomasr8 commentedon Feb 15, 2025
Here's the backtrace:
The immediate cause seems to be related to the printing itself, but I didn't dig into it too much
devdanzin commentedon Feb 15, 2025
Substituting
StringIOwith the following stops crashing, would it point to it being the culprit?picnixz commentedon Feb 15, 2025
I suspect it has something to do with the
PyFile_*interface.tomasr8 commentedon Feb 15, 2025
Might be something to do with
redirect_stdout, I'm getting a segfault even with this simplified reproducer:edit:
Both crashes seem to be caused by
Py_TYPEreturningNULL. Here's the backtrace for this repro:picnixz commentedon Feb 15, 2025
Have you tried removing the usage to
StringIO()? because it appears that we crash inPy_DECREFof a file object.sergey-miryanov commentedon Feb 16, 2025
Yes, it is caused by
redirect_stdout.PyObject_GetAttr(cpython/Objects/fileobject.c
Line 113 in 798f8d3
Py_DECREF(cpython/Objects/fileobject.c
Line 127 in 798f8d3
_Py_DeallocPy_DECREFfrom T1 decrements local ref_cnt, got 0 and go to_Py_MergeZeroLocalRefcount, see that shared ref_cnt is 0 and go to fast path and do_Py_Dealloc.The easiest way to fix this is probably to protect
PyFile_WriteObjectwith a critical section. However, this might slightly affect the performance of single-threaded apps.The best solution might be to update the documentation and point out this potential issue with
redirect_*.devdanzin commentedon Feb 16, 2025
Using
open("/dev/null", "w")instead ofStringIOstill segfaults.But any program that does
redirect_stdoutdoes will also segfault:Would making the presence of the critical section dependent on whether the build is freethreaded (and/or the GIL is disabled) avoid the performance impact for gilfull builds? Or would it be a bad idea to have this conditional difference in behavior?
ISTM that fixing the segfault is necessary, even if a note is added to docs, because users may hit the segfault without using
redirect_*.30 remaining items
sergey-miryanov commentedon Feb 17, 2025
Thanks, I will try.
sergey-miryanov commentedon Feb 23, 2025
I've looked at how
_PySys_GetAttris used. I can reproduce thesegfaulterror in some places where it's used. I've written tests for these places.@colesbury @ZeroIntensity, @serhiy-storchaka please take a look.
Here's a copy from my PR:
_pickle.c/whichmodule_threadmodule.c/thread_excepthookfaulthandler.c/faulthandler_get_filenobltinmodule.c/builtin_print_implbltinmodule.c/builtin_input_implerrors.c/write_unraisable_excerrors.c/format_unraisable_v_warnings.c/show_warningintrinsics.c/print_exprpylifecycle.c/flush_std_filespythonrun.c/_PyRun_InteractiveLoopObjectpythonrun.c/pyrun_one_parse_astpythonrun.c/_Py_HandleSystemExitAndKeyboardInterruptpythonrun.c/_PyErr_PrintExpythonrun.c/PyErr_Displaypythonrun.c/flush_io_streamsysmodule.c/sys_displayhooksysmodule.c/get_warnoptionssysmodule.c/PySys_ResetWarnOptionssysmodule.c/PySys_HasWarnOptionssysmodule.c/_PySys_AddXOptionWithErrorsysmodule.c/PySys_SetArgvExsysmodule.c/sys_writesysmodule.c/sys_formattraceback.c/_Py_FindSourceFilesys_displayhook, but not where it's used.sys_write,sys_format- I have checked all the places where they called, and most of the locations are safe. However, a few places use%sspecifier, in such places it is very tricky to getsegfault.get_warnoptions,PySys_ResetWarnOptions,_PySys_AddXOptionWithError- are deprecated, but it is a very tricky to getsegfaulttoo.flush_io_stream,PyErr_Display,pyrun_one_parse_ast,print_expr,write_unraisable_exc,thread_excepthook- very tight - They use a borrowed reference right after getting. It might be possible to reproduce it under heavy contention, but I wasn't able to do so._PyRun_InteractiveLoopObjectonly checks presence ofps1andps2.flush_std_filesused from_Py_Finalizeand fromfatal_error. For_Py_FinalizeI can repro, forfatal_error- no.whichmodule,_Py_FindSourceFileiterate borrowed reference, but I wasn't able to reproducesegfault._PySys_GetAttrcalled without an exception set.Also, can you point me out what to do next? I can replace
_PySys_GetAttrwith_PySys_GetAttrRef, but in the most places it is interleave with @serhiy-storchaka PRs. I'm not sure what the correct way to deal with this situation is.serhiy-storchaka commentedon Feb 25, 2025
#130503 is an adaptation of #111035 or #129736 that does not introduce new public API and does not change the current behavior (except fixing crashes, mainly). It will make minimal difference with #111035 or #129736, whatever will be accepted for future Python version.
I added also tests for
print(),input()andsys.displayhook()(they are relatively simple). Reproducers for other crashes are harder, so I do not think that this is worth hassle.Thank you for your tests, @sergey-miryanov, but sorry, I have not used them. They are high quality, but too large and expensive. It is not worth to keep them forever after fixing this issue.
sergey-miryanov commentedon Feb 25, 2025
@serhiy-storchaka I simplified it a bit yesterday, not sure if you saw the latest changes. Maybe it will be useful, but it is all up to you. Anyway, it was very interesting.
gh-130163: Fix crashes related to PySys_GetObject() (GH-130503)
[3.13] pythongh-130163: Fix crashes related to PySys_GetObject() (pyt…
[3.13] gh-130163: Fix crashes related to PySys_GetObject() (GH-130503) (
[3.13] pythongh-130163: Fix a leak in _pickle.c after backporting
[3.13] gh-130163: Fix a leak in _pickle.c after backporting (GH-130568)
[3.12] pythongh-130163: Fix crashes related to PySys_GetObject() (pyt…
[3.13] pythongh-130163: Fix a leak in _pickle.c after backporting (py…
[3.12] gh-130163: Fix crashes related to PySys_GetObject() (GH-130503) (
vstinner commentedon Feb 27, 2025
I cannot reproduce the crash using #130163 (comment) reproducer. Can we close the issue?
serhiy-storchaka commentedon Feb 27, 2025
It has already been fixed.