You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
(Mostly) Stop Special-casing the Main Interpreter #109857
We make a bunch of accommodations for the "main" (initial) interpreter, especially during initialization and finalization. This adds complexity to the runtime. We should consider making the main interpreter (mostly) not special. At most we need to specifically special-case relative to the main thread.
The main thread of a Python program uses the initial thread state of the main interpreter. That thread is special for the following reasons, at least for the Python executable (and for some embedding cases):
where the main code of the app executes
where the REPL runs
where signal handlers run
where the main (initial) interpreter is always created
(usually) where runtime finalization happens
Thus the initial thread state of the main interpreter plays a critical role. The code reflects this, especially in Python/pylifecycle.c. However, a lot of the special treatment of the main interpreter is unnecessary. 1
The main interpreter is the one created during runtime initialization and used during initialization. This is relevant for any non-immortal, non-static objects that we store on _PyRuntimeState, which we should strenuously avoid doing anyway. It is likely also relevant for embedders.
There are also a couple of places where the "main" interpreter needs to be factored in (which we currently isn't), but doesn't necessarily need to be addressed with this issue:
runtime finalization should happen in the main thread, with the main interpreter?
os.fork() should update the main thread state and the main interpreter (or only be allowed in the main thread of the main interpreter)?
FYI, we also optimize the code somewhat using the fact that the initial thread state of the main interpreter is likely to be the active one (e.g. that thread state and interpreter state are statically allocated and initialized). However, I'm not sure we need to change much about this. ↩
It might be useful to think about this hypothetical future API:
Runtime/interpreter init/teardown for use in “plugins” that embed Python without any support from the enclosing “application”.
init:
if Python is not initialized yet, starts a new thread with a dummy main interpreter (for signal handlers and such) and calls Py_Initialize
calls Py_NewInterpreter
teardown:
calls Py_EndInterpreter
if it's the last interpreter that was started this way and Python was initialized this way, schedules Py_Finalize in the main interpreter
(If the “application” uses Py_Initialize/Py_Finalize itself, this should still work, if all “plugins” are initialized before, and finalized after, Python itself.)
I like Petr's proposal, at least the way I'm reading it. Making Py_Initialize set up the runtime and the main thread stuff, then Py_NewInterpreter becomes an explicit step,1 and also hopefully the step where we provide configuration.
So an embedder could create lightweight interpreters (i.e. no imports) and totally ignore search paths and all that stuff.2 That would be a great benefit and open up a lot of possibilities.
So yeah, making the first interpreter into just another interpreter would be great. My only concern would be how it interacts with fork, but that's mostly out of sympathy for those who have to actually worry about that :)
Footnotes
And we update the main existing APIs to implicitly create a default one for compat. ↩
Or... hypothetically... create a lightweight interpreter to calculate the search paths and then create a "full" interpreter ↩
(I'm going to write down some thoughts here, but I should probably open a DPO thread.)
FWIW, I've also been thinking a lot about the distinct phases of init/fini and how making the boundaries actually structural in the code might be helpful. There's some similarity with PEP 432.
-> API "A"
init core runtime -> "AB"
init core main interpreter -> "ABC" (incl. builtin types/objects)
enable ceval -> "ABCD" (run Python code)
enable frozen imports -> "ABCDE"
init more (e.g. OS services) -> "ABCDEF"
enable external imports -> "ABCDEFG"
init almost everything else -> "ABCDEFGH"
enable threading -> full API
...and the finalization phases are mostly the mirror of that. Naturally there would be various sorts of config wedged in there too. Note that I'm not suggesting that level of granularity necessarily.
I abandoned my PR #108577 which adds PyInterpreterState_IsMain(). I'm no longer sure if this function is the right approach to handle sub-interpreters.
Activity
encukou commentedon Sep 26, 2023
It might be useful to think about this hypothetical future API:
Runtime/interpreter init/teardown for use in “plugins” that embed Python without any support from the enclosing “application”.
Py_InitializePy_NewInterpreterPy_EndInterpreterPy_Finalizein the main interpreter(If the “application” uses
Py_Initialize/Py_Finalizeitself, this should still work, if all “plugins” are initialized before, and finalized after, Python itself.)zooba commentedon Sep 26, 2023
I like Petr's proposal, at least the way I'm reading it. Making
Py_Initializeset up the runtime and the main thread stuff, thenPy_NewInterpreterbecomes an explicit step,1 and also hopefully the step where we provide configuration.So an embedder could create lightweight interpreters (i.e. no imports) and totally ignore search paths and all that stuff.2 That would be a great benefit and open up a lot of possibilities.
So yeah, making the first interpreter into just another interpreter would be great. My only concern would be how it interacts with
fork, but that's mostly out of sympathy for those who have to actually worry about that :)Footnotes
And we update the main existing APIs to implicitly create a default one for compat. ↩
Or... hypothetically... create a lightweight interpreter to calculate the search paths and then create a "full" interpreter ↩
ericsnowcurrently commentedon Sep 26, 2023
Yeah, making
Py_NewInterpreter()a separate step is right in line with what I've been thinking about. :)ericsnowcurrently commentedon Sep 27, 2023
(I'm going to write down some thoughts here, but I should probably open a DPO thread.)
FWIW, I've also been thinking a lot about the distinct phases of init/fini and how making the boundaries actually structural in the code might be helpful. There's some similarity with PEP 432.
...and the finalization phases are mostly the mirror of that. Naturally there would be various sorts of config wedged in there too. Note that I'm not suggesting that level of granularity necessarily.
vstinner commentedon Sep 27, 2023
PEP 587 added a half-baken private provisional API: https://docs.python.org/dev/c-api/init_config.html#multi-phase-initialization-private-provisional-api
vstinner commentedon Oct 3, 2023
I abandoned my PR #108577 which adds PyInterpreterState_IsMain(). I'm no longer sure if this function is the right approach to handle sub-interpreters.