Skip to content

gh-161941: Freeze the encodings modules imported at startup - #161951

Open
eendebakpt wants to merge 1 commit into
python:mainfrom
eendebakpt:startup-freeze-encodings-rebased
Open

eendebakpt wants to merge 1 commit into
python:mainfrom
eendebakpt:startup-freeze-encodings-rebased

Conversation

@eendebakpt

@eendebakpt eendebakpt commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Freeze encodings, encodings.aliases, encodings.utf_8 and the platform codec modules again. FrozenImporter now checks that the directory exists. If it does not, the location of a standard library package is looked up on sys.path with PathFinder, so its non-frozen submodules are found wherever the standard library is, including inside a zip file. Frozen packages outside the standard library are not affected. Importing encodings from disk was a large part of interpreter startup on Windows.

Benchmark on the issue.

Freeze encodings, encodings.aliases, encodings.utf_8 and the platform
codec modules again. They were unfrozen because a partially frozen
package broke the import of its non-frozen submodules with a zipped
standard library: FrozenImporter set the package's __path__ to the
package directory in sys._stdlib_dir, which does not exist in that
layout (sys._stdlib_dir is still set, to a missing directory), so
encodings.oem and friends could not be found.

FrozenImporter now checks that the directory exists. If it does not,
the location of a standard library package is looked up on sys.path
with PathFinder, so its non-frozen submodules are found wherever the
standard library is, including inside a zip file. Frozen packages
outside the standard library are not affected.

Importing encodings from disk was the single largest item of
interpreter startup on Windows: it needs the sys.path directory
listings and four .pyc files before the first codec lookup.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@read-the-docs-community

Copy link
Copy Markdown

Documentation build overview

📚 cpython-previews | 🛠️ Build #35876902 | 📁 Comparing 67c66c4 against main (2c2d801)

  🔍 Preview build  

2 files changed
± whatsnew/3.16.html
± whatsnew/changelog.html

@FFY00

FFY00 commented Nov 20, 2026

Copy link
Copy Markdown
Member

Can you explain a bit better the import system changes?

@eendebakpt

Copy link
Copy Markdown
Contributor Author

Can you explain a bit better the import system changes?

Background: a frozen package gets __path__ = [<stdlib_dir>/<pkg>] so that its non-frozen submodules can still be imported from disk. That path is computed from sys._stdlib_dir without checking that it exists. With a zipped stdlib (gh-148750) the directory isn't there, so encodings.__path__ pointed at a directory that does not exist and every codec that isn't frozen failed to import. encodings is special in the sense that it is the only frozen stdlib package, and only some of its submodules are frozen.

To allow encodings and a few submodules to be frozen (and let the remaining submodules be normal, either zipped or not) we make changes to only FrozenImporter:

  • We add a new _package_dir(fullname, pkgdir): if pkgdir exists it is returned unchanged, so the normal case behaves as before. Otherwise, for a stdlib package, it asks PathFinder.find_spec(fullname) and uses that spec's first submodule_search_locations entry. This is what finds stdlib.zip/encodings through zipimport. If nothing is found, pkgdir is returned as before.
  • _package_dir is only called for packages, not for plain modules (both calls are inside an if ispkg: block). Currently that is only encodings, plus the __phello__ test packages.
  • It is called from the two places that fill in submodule_search_locations: find_spec() and _fix_up_module().
  • The isdir check and the PathFinder lookup in _package_dir are limited to names in sys.stdlib_module_names, so a frozen package outside the stdlib never picks up a same-named directory from sys.path.
  • The two asserts removed from _fix_up_module() checked that __path__ equals the package directory returned by _resolve_filename(), which no longer holds when the fallback is used. E.g. for a zipped stdlib we have __path__ == ['<dir>/stdlib.zip/encodings'], while _resolve_filename() returns <stdlib_dir>/encodings.
  • Only __path__ is affected. encodings itself is still loaded from the frozen data, but the change to __path__ allows the zipped submodules to be found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting core review type-feature A feature request or enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants