Skip to content

gh-127041: Acquire PEP 788 finalization guard in _PyXI_Enter - #162004

Open
utam-1 wants to merge 5 commits into
python:mainfrom
utam-1:fix-127041-pep788-guard
Open

utam-1 wants to merge 5 commits into
python:mainfrom
utam-1:fix-127041-pep788-guard

Conversation

@utam-1

@utam-1 utam-1 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Based on the feedback I have utilized the PEP 788 finalization guard API (_PyInterpreterGuard_TryAcquire / _PyInterpreterGuard_Release, added in #128640) inside _PyXI_session, as follows:

  • Before _enter_session(): Call _PyInterpreterGuard_TryAcquire(). If destruction has already started, fail cleanly with InterpreterError("interpreter is being destroyed").

  • At the end of _exit_session(): Release the guard after thread-state cleanup and swap-back, so Py_EndInterpreter() waits for any in-flight session to exit before finalizing.

Comment thread Python/crossinterp.c Outdated

// Acquire PEP 788 finalization guard before switching or running.
// If finalization has already started, TryAcquire will fail safely.
if (_PyInterpreterGuard_TryAcquire(interp, &session->guard) < 0) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should try to use the public API (PyInterpreterGuard_FromCurrent) and store the result as a pointer. That would also remove the need for the extra has_guard field, because NULL could act as has_guard = 0.

@utam-1 utam-1 Nov 15, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried using the PyInterpreterGuard_FromCurrent() but that resulted in a deadlock in _PyThreadState_Attach, reason could be the behaviour of _enter_session ( Not super sure though).

Instead, since PyInterpreterGuard wraps PyInterpreterState *interp (NULL when zero-initialized), I removed has_guard by checking session->guard.interp != NULL while keeping _PyInterpreterGuard_TryAcquire before _enter_session().

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please determine the source of the deadlock. I think it's a bad idea to rely on guard internals here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the Python and C stack trace (I symbolised it a little bit because it's easier to follow this way):

test_concurrent_run_and_destroy (test.test__interpreters.DestroyTests.test_concurrent_run_and_destroy) ...
=== C Stack Trace on Freed tstate->interp ===
  #01 _PyThreadState_CheckConsistency (Python\pystate.c:3371)
  #02 take_gil (Python\ceval_gil.c:411)
  #03 _PyEval_AcquireLock (Python\ceval_gil.c:593)
  #04 _PyThreadState_Attach (Python\pystate.c:2405)
  #05 _PyThreadState_Swap (Python\pystate.c:2801)
  #06 PyThreadState_Swap (Python\pystate.c:2810)
  #07 _enter_session (Python\crossinterp.c:2603)
  #08 _PyXI_Enter (Python\crossinterp.c:2731)
  #09 _run_in_interpreter (Modules\_interpretersmodule.c:676)
  #10 _interpreters_run_string_impl (Modules\_interpretersmodule.c:1199)
  #11 _interpreters_run_string (Modules\clinic\_interpretersmodule.c.h:583)

Assertion failed: !_PyMem_IsPtrFreed(tstate->interp), file Python\pystate.c, line 3371
Fatal Python error: Aborted

Current thread 0x00003e8c [Thread-1 (runner)] (most recent call first):
  File "Lib\test\test__interpreters.py", line 478 in runner
  File "Lib\threading.py", line 1161 in run
  File "Lib\threading.py", line 1219 in _bootstrap_inner
  File "Lib\threading.py", line 1181 in _bootstrap

Based on the further analysis I found two things related to public API:

  • It takes a void argument, that leads to it guarding the caller's interpreter not the sub-interpreter we require.
  • Calling it after _enter_session() leaves PyThreadState_Swap() unguarded—so a concurrent destroy() frees interp or marks the new thread state SHUTTING_DOWN, hanging in _PyThreadState_HangThread() inside take_gil().

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's an abort, not a deadlock. Can you show the diff?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry for mixing it up, I'm sharing the diff:

diff --git a/Python/crossinterp.c b/Python/crossinterp.c
index b8dd5f11c2a..e998fb627da 100644
--- a/Python/crossinterp.c
+++ b/Python/crossinterp.c
@@ -2546,6 +2546,8 @@ struct xi_session {
     // once the session exits.  Do not access this directly; use
     // _PyXI_Preserve() and _PyXI_GetPreserved() instead;
     PyObject *_preserved;
+
+    PyInterpreterGuard *guard;
 };
 
 _PyXI_session *
@@ -2641,6 +2643,11 @@ _exit_session(_PyXI_session *session)
         assert(!session->own_init_tstate);
     }
 
+    if (session->guard != NULL) {
+        PyInterpreterGuard_Close(session->guard);
+        session->guard = NULL;
+    }
+
     *session = (_PyXI_session){0};
 }
 
@@ -2717,6 +2724,11 @@ _PyXI_Enter(_PyXI_session *session,
     // Switch to the requested interpreter (if necessary).
     _enter_session(session, interp);
     _PyXI_failure override = XI_FAILURE_INIT;
     override.code = _PyXI_ERR_UNCAUGHT_EXCEPTION;
 #ifndef NDEBUG
     tstate = _PyThreadState_GET();
 #endif
 
+    session->guard = PyInterpreterGuard_FromCurrent();
+    if (session->guard == NULL) {
+        goto error;
+    }
+
     // Ensure this thread owns __main__.
     if (_PyInterpreterState_SetRunningMain(interp) < 0) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right, we need to attach to an interpreter to use PyInterpreterGuard_FromCurrent, so that won't work

But, I think the current approach is thread-unsafe as currently written, because there's nothing stopping an interpreter from being deleted during the call to _PyInterpreterGuard_TryAcquire. PEP 788 comes with interpreter views for this exact use-case, so let's try those. Something like this should work:

  1. Switch the PyInterpreterState *interp parameter of _PyXI_Enter to a PyInterpreterView *view.
  2. Call PyInterpreterGuard_FromView before trying to attach.
  3. Release the guard after attaching to the interpreter (we don't want to block finalization for all of the subinterpreter's execution; just the part where it can be potentially deleted by another thread).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated _PyXI_Enter to take PyInterpreterView *view, storing PyInterpreterGuard_FromView(view) on session->guard and closing it via PyInterpreterGuard_Close(session->guard).

Closing the guard after _exit_session() seemed plausible, as doing so right after _enter_session() made the below stress testing script hang forever.

.\PCbuild\amd64t\python_d.exe -X faulthandler -c "from test import support; import _interpreters, threading
for _ in range(200):
    interp = _interpreters.create()
    b = threading.Barrier(2)
    def runner():
        b.wait()
        try: _interpreters.run_string(interp, 'x = 1')
        except _interpreters.InterpreterError: pass
    def destroyer():
        b.wait()
        try: _interpreters.destroy(interp)
        except _interpreters.InterpreterError: pass
    ts = [threading.Thread(target=runner), threading.Thread(target=destroyer)]
    for t in ts: t.start()
    for t in ts: t.join()
    try: _interpreters.destroy(interp)
    except _interpreters.InterpreterError: pass
print('FREE-THREADED 200x DONE')"

Comment thread Modules/_interpretersmodule.c Outdated
Comment on lines +676 to +677
PyInterpreterView view = { .id = PyInterpreterState_GetID(interp) };
if (_PyXI_Enter(session, &view, shareables, &result) < 0) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't a good idea either. The idea is that we should refactor most of the crossinterpreter APIs to use PyInterpreterView, so this function should just take a PyInterpreterView *view argument instead of PyInterpreterState *interp.

Comment thread Python/crossinterp.c Outdated
if (sharedns != NULL) {
_destroy_sharedns(sharedns);
}
PyErr_SetString(PyExc_InterpreterError, "interpreter is being destroyed");

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be a better idea to use PythonFinalizationError here.

@utam-1
utam-1 requested a review from FFY00 as a code owner November 19, 2026 12:52
Comment thread Modules/_interpretersmodule.c Outdated
Comment on lines +1157 to +1158
PyInterpreterView *view =
_PyInterpreterView_FromID(PyInterpreterState_GetID(interp));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still don't think this is thread-safe. The interpreter can be finalized and deleted in another thread during the call to PyInterpreterState_GetID.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added resolve_interp_view() for addressing this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants