Repository navigation
Conversation
| } \ | ||
| } while (0) | ||
|
|
||
| #define CHECK_TUPLE_ITEMS(obj, n) do { \ |
There was a problem hiding this comment.
Truth be told, we reinvent similar check over and over:
cpython/Objects/memoryobject.c
Lines 217 to 222 in 9a75080
Lines 26 to 33 in 9a75080
Sometimes without a macro:
cpython/Modules/_io/stringio.c
Lines 922 to 930 in 9a75080
Lines 450 to 465 in 9a75080
| Py_ssize_t num_interpreters = PyList_GET_SIZE(stack_frames); | ||
| for (Py_ssize_t i = 0; i < num_interpreters; i++) { | ||
| PyObject *interp_info = PyList_GET_ITEM(stack_frames, i); | ||
| CHECK_TUPLE_ITEMS(interp_info, 2); |
There was a problem hiding this comment.
We validate as we go, so a bad thread in the middle leaves the earlier threads already written and total_samples bumped even though write_sample raised. Should we walk and validate the whole structure first so this is all-or-nothing? Otherwise the caller has no way to recover from the TypeError.
|
|
||
| PyObject *interp_id_obj = PyStructSequence_GET_ITEM(interp_info, 0); | ||
| PyObject *threads = PyStructSequence_GET_ITEM(interp_info, 1); | ||
| CHECK_LIST(threads); |
There was a problem hiding this comment.
This narrows the hole but does not close it. PyLong_AsLong(status_obj) goes through _PyNumber_Index, so an __index__ that mutates the same list re-enters us while we still hold a cached size and a borrowed thread_info:
threads = []
class Evil:
def __index__(self):
del threads[:]
return 0
threads.extend([(1, Evil(), []), (2, 0, [])])
_remote_debugging.BinaryWriter("/tmp/o.bin", 1000, 0, compression=0).write_sample([(0, threads)], 2000)This still segfaults on PyList_GET_ITEM(threads, 1) because del threads[:] frees ob_item. writer_intern_string has the same problem via PyObject_Hash. If we want to claim write_sample is safe against arbitrary input we need strong references to the containers (or to re-read the sizes), not only a type check up front. Happy to take this as a follow-up, but then let's not close the issue with this PR.
| int32_t end_column = LOCATION_NOT_AVAILABLE; | ||
|
|
||
| if (location != Py_None) { | ||
| CHECK_TUPLE_ITEMS(location, 4); |
There was a problem hiding this comment.
normalize_location() in Lib/profiling/sampling/collector.py accepts an int location (a bare lineno) and binary_collector.py hands stack_frames straight to write_sample, so this now raises where the other collectors work. Do we want a PyLong_Check(location) case here too, or is that branch in normalize_location() dead and we should drop it instead?
There was a problem hiding this comment.
@pablogsal I still need some time, but this is a dead path, excercised only by tests. To put aside this PR, do you want me to open a separate PR that removes these?
There was a problem hiding this comment.
Yes, please, a separate PR for the integer-location branches and their tests makes sense. Let's keep this one focused on container validation. Thanks!
|
|
||
| PyObject *interp_id_obj = PyStructSequence_GET_ITEM(interp_info, 0); | ||
| PyObject *threads = PyStructSequence_GET_ITEM(interp_info, 1); | ||
| CHECK_LIST(threads); |
There was a problem hiding this comment.
The __index__ repro above is still open: we iterate threads with a cached size and a borrowed thread_info while PyLong_AsLong(status_obj) can run arbitrary code. No rush, but can you tell me if you want to fix it here or in a separate issue? In the second case let's not close gh-157227 with this PR.
| filename, 1000, 0, compression=0 | ||
| ) as writer: | ||
| with self.assertRaisesRegex(TypeError, field): | ||
| writer.write_sample(sample, 2000) |
There was a problem hiding this comment.
Nit, feel free to ignore: if breaking the writer is intended, can we assert that finalize() raises ValueError after the TypeError?
|
|
||
| PyObject *interp_id_obj = PyStructSequence_GET_ITEM(interp_info, 0); | ||
| PyObject *threads = PyStructSequence_GET_ITEM(interp_info, 1); | ||
| CHECK_LIST(threads); |
There was a problem hiding this comment.
This still crashes when status.index clears threads: the next iteration uses the cached size and reads freed storage. We need mutation-safe traversal and a regression test.
| process_thread_sample(BinaryWriter *writer, PyObject *thread_info, | ||
| uint32_t interpreter_id, uint64_t timestamp_us) | ||
| { | ||
| CHECK_TUPLE_ITEMS(thread_info, 3); |
There was a problem hiding this comment.
A malformed second thread still counts the first and breaks the writer, so earlier valid samples cannot be finalized. Can we validate before committing the sample?
| int32_t end_column = LOCATION_NOT_AVAILABLE; | ||
|
|
||
| if (location != Py_None) { | ||
| CHECK_TUPLE_ITEMS(location, 4); |
There was a problem hiding this comment.
Yes, please, a separate PR for the integer-location branches and their tests makes sense. Let's keep this one focused on container validation. Thanks!
pablogsal
left a comment
There was a problem hiding this comment.
Thanks for the updates! Two things are still open, see the comments.
|
|
||
| PyObject *interp_id_obj = PyStructSequence_GET_ITEM(interp_info, 0); | ||
| PyObject *threads = PyStructSequence_GET_ITEM(interp_info, 1); | ||
| CHECK_LIST(threads); |
There was a problem hiding this comment.
This still crashes on the current head. PyLong_AsLong(status_obj) can run an __index__ that clears threads, and the next iteration reads freed storage. The same happens through string_hash_func with a str subclass filename whose __hash__ clears the frame list.
Can we check the leaves too? PyLong_Check for status, opcode and the location items, and PyUnicode_CheckExact for filename and funcname. Then no Python code runs while we walk the lists and the borrowed references are fine. We need a regression test for this as well.
| process_thread_sample(BinaryWriter *writer, PyObject *thread_info, | ||
| uint32_t interpreter_id, uint64_t timestamp_us) | ||
| { | ||
| CHECK_TUPLE_ITEMS(thread_info, 3); |
There was a problem hiding this comment.
The new test case only checks that we raise. I tried one good sample followed by a sample with a bad second thread: total_samples is bumped for the first thread, the writer is broken and finalize() raises ValueError, so the earlier samples are gone.
Can we validate the whole sample before committing anything? If breaking the writer is intended, let's assert that in the test.
|
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase |
See #157227 for more details
Basically,
BinaryWriter.write_sample()only checks whetherstack_framesis a list, and doesPyList_GET_ITEM()/PyStructSequence_GET_ITEM()all the way. The PR adds a check for these containers.Now it's graceful:
instead of
segmentation fault._remote_debugging:BinaryWriter.write_sample()trivial segfault #157227