1

Closed

pykd 0.3.2.0 crashes python 3.6.0 on unload

description

Initialializing pykd (e.g. by starting and quitting and interactive session immediately) and then unloading it again will crash windbg.

The point of the crash is here: https://github.com/python/cpython/blob/3.6/Python/ceval_gil.h#L175

So something funky must be happening with the GIL.

Python version: Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 07:18:10) [MSC v.1900 32 bit (Intel)] on win32

Stack trace of the crash:
 # ChildEBP RetAddr  Args to Child              
00 06fecf98 77e3bb6b 00720064 0070006f 0067005f ucrtbase!abort+0x4b
01 06fecfec 77e138e6 77f4e3f4 084fd298 77e0145e python36!Py_FatalError+0xfb [c:\build\cpython36\python\pylifecycle.c @ 1457]
02 06fecff8 77e0145e 00000000 086da238 00000002 python36!drop_gil+0x16 [c:\build\cpython36\python\ceval_gil.h @ 175]
03 (Inline) -------- -------- -------- -------- python36!PyEval_SaveThread+0x34 [c:\build\cpython36\python\ceval.c @ 355]
04 06fed014 77dfffc4 08763288 00000000 00020019 python36!winreg_OpenKey_impl+0x3e [c:\build\cpython36\pc\winreg.c @ 1317]
05 06fed038 77db35d5 086d0d50 08695488 00000002 python36!winreg_OpenKey+0x54 [c:\build\cpython36\pc\clinic\winreg.c.h @ 645]
06 06fed068 77db370e 77dfff70 08695488 00000002 python36!_PyCFunction_FastCallDict+0x1a5 [c:\build\cpython36\objects\methodobject.c @ 251]
07 06fed08c 77e19407 086da238 08695488 00000002 python36!_PyCFunction_FastCallKeywords+0x3e [c:\build\cpython36\objects\methodobject.c @ 295]
08 06fed0bc 77e14d10 00000000 08695340 086d64dc python36!call_function+0xf7 [c:\build\cpython36\python\ceval.c @ 4788]
09 06fed13c 77e1951a 08695340 00000000 00000002 python36!_PyEval_EvalFrameDefault+0xae0 [c:\build\cpython36\python\ceval.c @ 3277]
0a (Inline) -------- -------- -------- -------- python36!PyEval_EvalFrameEx+0x10 [c:\build\cpython36\python\ceval.c @ 718]
0b 06fed158 77e19591 00000002 086af480 086c5030 python36!_PyFunction_FastCall+0x5a [c:\build\cpython36\python\ceval.c @ 4872]
0c 06fed17c 77e19467 00000002 00000000 00000083 python36!fast_function+0x51 [c:\build\cpython36\python\ceval.c @ 4905]
0d 06fed1a4 77e14d10 00000000 086d6380 086d633c python36!call_function+0x157 [c:\build\cpython36\python\ceval.c @ 4809]
0e 06fed224 77e1951a 086d6380 00000000 00000002 python36!_PyEval_EvalFrameDefault+0xae0 [c:\build\cpython36\python\ceval.c @ 3277]
0f (Inline) -------- -------- -------- -------- python36!PyEval_EvalFrameEx+0x10 [c:\build\cpython36\python\ceval.c @ 718]
10 06fed240 77e19591 00000002 086af480 086c5078 python36!_PyFunction_FastCall+0x5a [c:\build\cpython36\python\ceval.c @ 4872]
11 06fed264 77e19467 00000002 00000000 00000083 python36!fast_function+0x51 [c:\build\cpython36\python\ceval.c @ 4905]
12 06fed28c 77e14d10 00000000 086b3808 086d61d8 python36!call_function+0x157 [c:\build\cpython36\python\ceval.c @ 4809]
13 06fed308 77e185a5 086d61d8 00000000 086c50c0 python36!_PyEval_EvalFrameDefault+0xae0 [c:\build\cpython36\python\ceval.c @ 3277]
14 (Inline) -------- -------- -------- -------- python36!PyEval_EvalFrameEx+0x13 [c:\build\cpython36\python\ceval.c @ 718]
15 06fed354 77e19614 00000000 0865d56c 00000004 python36!_PyEval_EvalCodeWithName+0x735 [c:\build\cpython36\python\ceval.c @ 4119]
16 06fed3a4 77e19467 00000004 00000002 00000083 python36!fast_function+0xd4 [c:\build\cpython36\python\ceval.c @ 4929]
17 06fed3cc 77e14d10 00000000 0869ef98 0865d400 python36!call_function+0x157 [c:\build\cpython36\python\ceval.c @ 4809]
18 06fed448 77e185a5 0865d400 00000000 086a9db0 python36!_PyEval_EvalFrameDefault+0xae0 [c:\build\cpython36\python\ceval.c @ 3277]
19 (Inline) -------- -------- -------- -------- python36!PyEval_EvalFrameEx+0x13 [c:\build\cpython36\python\ceval.c @ 718]
1a 06fed494 77e19614 00000000 08701670 00000002 python36!_PyEval_EvalCodeWithName+0x735 [c:\build\cpython36\python\ceval.c @ 4119]
1b 06fed4e4 77e19467 00000002 00000001 00000083 python36!fast_function+0xd4 [c:\build\cpython36\python\ceval.c @ 4929]
1c 06fed50c 77e14d10 00000000 08701510 08701324 python36!call_function+0x157 [c:\build\cpython36\python\ceval.c @ 4809]
1d 06fed58c 77e1951a 08701510 00000000 00000002 python36!_PyEval_EvalFrameDefault+0xae0 [c:\build\cpython36\python\ceval.c @ 3277]
1e (Inline) -------- -------- -------- -------- python36!PyEval_EvalFrameEx+0x10 [c:\build\cpython36\python\ceval.c @ 718]
1f 06fed5a8 77e19591 00000002 086a2420 086a9e40 python36!_PyFunction_FastCall+0x5a [c:\build\cpython36\python\ceval.c @ 4872]
20 06fed5cc 77e19467 00000002 00000000 00000083 python36!fast_function+0x51 [c:\build\cpython36\python\ceval.c @ 4905]
21 06fed5f4 77e14d10 00000000 087011d0 06fed718 python36!call_function+0x157 [c:\build\cpython36\python\ceval.c @ 4809]
22 06fed670 77e1951a 087011d0 00000000 086a9e88 python36!_PyEval_EvalFrameDefault+0xae0 [c:\build\cpython36\python\ceval.c @ 3277]
23 (Inline) -------- -------- -------- -------- python36!PyEval_EvalFrameEx+0x10 [c:\build\cpython36\python\ceval.c @ 718]
24 06fed68c 77e1969d 00000002 086a2420 086a9e88 python36!_PyFunction_FastCall+0x5a [c:\build\cpython36\python\ceval.c @ 4872]
25 06fed6c4 77d772d1 086a9e88 06fed710 00000002 python36!_PyFunction_FastCallDict+0x5d [c:\build\cpython36\python\ceval.c @ 4972]
26 06fed6e8 77d77d53 086a9e88 06fed710 00000002 python36!_PyObject_FastCallDict+0x61 [c:\build\cpython36\objects\abstract.c @ 2295]
27 06fed72c 77e32d86 086a23f0 77fbacc4 086dc020 python36!_PyObject_CallMethodIdObjArgs+0xa3 [c:\build\cpython36\objects\abstract.c @ 2780]
28 06fed76c 77e1018a 086dc020 00000000 00000000 python36!PyImport_ImportModuleLevelObject+0x236 [c:\build\cpython36\python\import.c @ 1592]
29 06fed79c 77db3596 0867c480 087118d0 00000000 python36!builtin___import__+0x6a [c:\build\cpython36\python\bltinmodule.c @ 231]
2a 06fed7c8 77db370e 77e10120 086a7c30 00000001 python36!_PyCFunction_FastCallDict+0x166 [c:\build\cpython36\objects\methodobject.c @ 231]
2b 06fed7ec 77e19407 08678b20 086a7c30 00000001 python36!_PyCFunction_FastCallKeywords+0x3e [c:\build\cpython36\objects\methodobject.c @ 295]
2c 06fed81c 77e14d10 00000000 08763498 086a7af0 python36!call_function+0xf7 [c:\build\cpython36\python\ceval.c @ 4788]
2d 06fed898 77e185a5 086a7af0 00000000 08763498 python36!_PyEval_EvalFrameDefault+0xae0 [c:\build\cpython36\python\ceval.c @ 3277]
2e (Inline) -------- -------- -------- -------- python36!PyEval_EvalFrameEx+0x13 [c:\build\cpython36\python\ceval.c @ 718]
2f 06fed8e4 77e52ac6 086db960 00000000 00000000 python36!_PyEval_EvalCodeWithName+0x735 [c:\build\cpython36\python\ceval.c @ 4119]
30 (Inline) -------- -------- -------- -------- python36!PyEval_EvalCodeEx+0x25 [c:\build\cpython36\python\ceval.c @ 4140]
31 (Inline) -------- -------- -------- -------- python36!PyEval_EvalCode+0x25 [c:\build\cpython36\python\ceval.c @ 695]
32 06fed928 77e528a7 086db960 086db960 00000000 python36!run_mod+0x46 [c:\build\cpython36\python\pythonrun.c @ 980]
33 06fed94c 77e53956 51e8bae4 00000101 086db960 python36!PyRun_StringFlags+0xa7 [c:\build\cpython36\python\pythonrun.c @ 904]
*** WARNING: Unable to verify checksum for C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\winext\pykd.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\winext\pykd.dll - 
34 06fed968 51e5c007 51e8bae4 00000101 086db960 python36!PyRun_String+0x16 [c:\build\cpython36\python\pythonrun.c @ 1510]
WARNING: Stack unwind information not available. Following frames may be wrong.
35 06fed9a0 51e5ae2e 51e62410 08407cd8 00000000 pykd+0xc007
36 06fed9c0 775d65be 00000000 08407cd8 00000000 pykd+0xae2e
37 06fed9d8 775d6fa6 00000000 00000001 08407cd8 dbgeng!ExtensionInfo::Unload+0x77
38 06fed9f4 775d558f 00000001 169b75fe 04733b10 dbgeng!ExtensionInfo::Delete+0x1f
39 06fede50 7760e8d3 00000000 00000000 00000000 dbgeng!ParseBangCmd+0x3a1
3a 06fedecc 7760f712 169b7482 04733b10 00000001 dbgeng!ProcessCommands+0x816
3b 06fedf2c 7755e0cf 00000000 169b483a 00000000 dbgeng!ProcessCommandsAndCatch+0xad
3c 06fee394 7755e2a8 0000000a 00000000 169b4876 dbgeng!Execute+0x247
3d 06fee3d8 00134bd7 04733b18 00000001 06fee7b8 dbgeng!DebugClient::ExecuteWide+0x68
3e 06fee794 00135068 ffffffff 00000008 ffffff00 windbg!ProcessCommand+0x12f
3f 06fef7b0 001372d1 65386ba8 00136d60 00136d60 windbg!ProcessEngineCommands+0xd0
40 06fef7ec 75938744 00000000 75938720 96b05a2a windbg!EngineLoop+0x571
41 06fef800 76f22de6 00000000 54905a86 00000000 KERNEL32!BaseThreadInitThunk+0x24
42 06fef848 76f22db6 ffffffff 76f44749 00000000 ntdll!__RtlUserThreadStart+0x2f
43 06fef858 00000000 00136d60 00000000 00000000 ntdll!_RtlUserThreadStart+0x1b
Closed Mar 9 at 4:18 PM by kernelnet

comments

ussrhero wrote Mar 8 at 5:27 PM

Do you use pykd_bootstrapper (http://pykd.codeplex.com/releases/view/624814) to run python ?
Is the bug reproduced with the last pykd_bootstrapper version ?

poizan42 wrote Mar 9 at 10:20 AM

Seems like my last comment disappeared. Yes it's pykd_bootstrapper, and the bug seems to be that PythonSingleton::stop() destroys the global interpreter which destroys the global state, and then it tries to switch to the thread state it just destroyed which obviously doesn't go over so well.

Actually the code in PythonSingleton::stop() seems weird. Shouldn't it just deinitialize the pykd module and then call Py_FinalizeEx? That should destroy all the subintepreters as well.

poizan42 wrote Mar 9 at 10:58 AM

Uhm is pykd_bootstrapper version 2.0 in the pykd_ext dir in the repo and not pykd_bootstrapper? The organization is really confusing. In that case my analysis doesn't apply. But I get the crash using the compiled PYKD BOOTSTRAPPER 2.0.

poizan42 wrote Mar 9 at 12:43 PM

Ah, so there was a new version released a week ago. It was 2.0.0.8 that I had that problem with, it is indeed fixed in 2.0.0.10.

Feel free to close this issue.

kernelnet wrote Mar 9 at 4:11 PM

sorry for inconvenient, it is my fault. I've uploaded to public some unstable versions of pykd_bootstrapper ((

wrote Mar 9 at 4:18 PM