Skip to content

Fix reference cycle in IndexFile._raise_exc and remove gc.collect() - #2350

Open
jenshnielsen wants to merge 3 commits into
gitpython-developers:mainfrom
jenshnielsen:jenshnielsen/agents/gitpython-repo-close-gc-optimization
Open

jenshnielsen wants to merge 3 commits into
gitpython-developers:mainfrom
jenshnielsen:jenshnielsen/agents/gitpython-repo-close-gc-optimization

Conversation

@jenshnielsen

Copy link
Copy Markdown

Disclaimer: The changes are AI written but the issue is real and I have tested the fix

Follow up to #686

This breaks the reference cycle such that a Repo close on longer requires a gc.collect to release all file handles held by the process. This significantly improves performance on Windows.

jenshnielsen and others added 2 commits November 20, 2026 12:59
`IndexFile._iter_expand_paths()` passes `os.walk()` an `onerror` callback
that re-raises the error it receives. `os.walk()` fails for every file path,
so each `index.add()` of a file raised through `raise_exc()` while its frame
local `e` still referenced the exception. That forms the cycle
exception -> `__traceback__` -> `raise_exc` frame -> exception, and the
traceback also retains the `_iter_expand_paths` generator frame, whose `self`
references the `IndexFile` and its `Repo`.

A `Repo` used to stage a file was therefore only released by the cyclic garbage
collector instead of by reference counting. Until then its persistent
`git cat-file` processes stayed alive and held repository files open, which on
Windows prevents deleting the repository. `Repo.close()` hid this by forcing
`gc.collect()`.

Delete the local in a `finally` block before the exception propagates, which
breaks the cycle without changing which errors are raised or handled.

The new test stages a file with the garbage collector disabled and checks that
dropping the `Repo` frees it. It fails before this change and passes after it
with both the CLI and Gix backends on Windows (Python 3.14).

Agent-Model: unknown

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`Repo.close()`, used by `__exit__` and `__del__`, called `gc.collect()` twice
on Windows to release handles held by unreachable objects. Each call walks the
whole Python heap. With 3 million live objects, closing a repository cost about
1.4 seconds in an application. A dropped `Repo` is freed by reference counting,
so `__del__` paid this cost outside any automatic collection.

The handles came from objects that GitPython left in reference cycles. Running
the Windows test suite without any `gc.collect()` calls, and recording every
failed temporary directory removal, found these sources:

- The `IndexFile.add()` exception cycle fixed in the previous commit kept the
  repository and its `git cat-file` processes alive. It caused the observed
  library leaks and the file lock that `test_root_module` is marked xfail for.
- `GitCmdObjectDB` reads objects only through Git, so no smmap windows or
  memory maps remain; `gitdb.util.mman.collect()` is kept because it is cheap.
- The remaining leaks were test artifacts: `MagicMock` call records and logged
  exceptions retained by pytest. The test harness keeps its own collections.

`close()` now only stops the persistent Git processes and releases unused gitdb
memory maps. `Submodule.remove()` closes the module repository before deleting
its checkout, as its other deletion paths already do, instead of collecting.
The changelog documents that callers must close or release their own objects
that are only reachable through reference cycles.

With 3 million live objects on Windows and Python 3.14, explicit `close()`
and dropping a `Repo` take about 2 ms (CLI backend) or under 0.1 ms (Gix
backend) instead of about 225 ms.

Without any collection in `Repo.close()` or the test harness, the full Windows
suite passed with the CLI backend (2127 passed). With this change, the affected
index, documentation and submodule tests pass, except for an order-dependent
`test_git_submodule_compatibility` failure that also occurs on `main`.

Agent-Model: unknown

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 20, 2026 13:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The iterator check confirms cycle removal, but Windows repository deletion and non-forced submodule removal still require runtime validation after forced GC was removed.

No open issues 🎉

What changed in this PR

This PR addresses the reference cycle behind delayed repository cleanup, removing the costly garbage-collection workaround introduced for Windows in #686.

Changes:

  • Break the exception reference cycle during index path expansion.
  • Remove forced garbage collection from repository cleanup and explicitly close submodules before deletion.
  • Add a repository-lifetime regression test and document the cleanup behavior.
File Description
test/​test_index.py Verify staging does not keep a dropped repository alive with garbage collection disabled.
git/​repo/​base.py Remove forced Windows garbage collections from Repo.close().
git/​objects/​submodule/​base.py Explicitly close the submodule repository before removing its working tree.
git/​index/​base.py Delete the exception reference after raising to break the reference cycle.
doc/​source/​changes.rst Document cleanup changes and callers’ resource-management responsibilities.

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The test disables automatic GC but does not assert that repository cleanup avoids explicit gc.collect() calls, leaving the Windows performance change unprotected against regression.

1 open issue

Review effort: Max

Comment thread test/test_index.py Resolved
Refactor garbage collection test to mock gc.collect and ensure it is not called when repo is dropped.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Copilot AI max review requested due to automatic review settings November 20, 2026 14:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Missing regression coverage leaves reader termination, pipe closure, and strict checkout deletion unverified after repository close or disposal and during recursive submodule removal with cyclic GC disabled.

No open issues 🎉

2 previously missed issues

In code that hasn't changed since last review

Medium severity Test reader cleanup during submodule removal with GC disabled

git/​objects/​submodule/​base.py:1555

GitHub Copilot (AI): Existing non-forced configuration=False removal tests do not check resource release with cyclic GC disabled, so they cannot catch handles surviving until collection. Add a clean-submodule regression that opens a reader on the child Repo used by remove() and checks reader termination and strict checkout deletion before re-enabling GC. Cover recursive removal too.

Medium severity Test cat-file reader cleanup on close and repository drop

test/​test_index.py:73

GitHub Copilot (AI): Staging this file never opens either persistent cat-file reader, so the test cannot catch a reader or pipe remaining open after the Windows collection workaround is removed. Force the CLI backend, open both readers, and check that explicit close() and dropping the repository terminate them, close their pipes, and allow strict checkout removal with GC disabled.

1 resolved since last review

Review effort: Max

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants