A process serving application pool exceeded time limits during shu
A process serving application pool exceeded time limits during shu
am 16.11.2007 14:24:01 von Savas
Hi,
I have a website that has been hanging up its application pool since this
morning. It had done the same thing couple of months ago but it stopped doing
it by itself.
Now it started again. winDBg has not been any help. On september 7th when it
did the same thing I had some logs by iisstate. Can somebody please look at
it and tell me what the cause is?
None of these logs make any sense to me at all.
Opened log file 'C:\iisstate\output\IISState-1228.log'
***********************
Starting new log output
IISState version 3.3.1
Fri Sep 07 09:53:49 2007
OS = Windows 2003 Server
Executable: w3wp.exe
PID = 1228
Note: Thread times are formatted as HH:MM:SS.ms
***********************
Thread ID: 0
System Thread ID: 930
Kernel Time: 0:0:0.15
User Time: 0:0:0.78
Thread Status: Thread is in a WAIT state.
Thread Type: HTTP Compression Thread
# ChildEBP RetAddr
00 0006fc08 7c827d0b ntdll!KiFastSystemCallRet
01 0006fc0c 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 0006fc7c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 0006fc90 5a364662 kernel32!WaitForSingleObject+0x12
04 0006fca0 5a366e3f w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
05 0006fca8 5a3af42d w3dt!UlAtqStartListen+0x2d
06 0006fcb8 5a3bc335 w3core!W3_SERVER::StartListen+0xbd
07 0006ff0c 0100187c w3core!UlW3Start+0x26e
08 0006ff44 01001a27 w3wp!wmain+0x22a
09 0006ffc0 77e6f23b w3wp!wmainCRTStartup+0x12f
0a 0006fff0 00000000 kernel32!BaseProcessStart+0x23
Thread ID: 1
System Thread ID: 1204
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 006afea0 7c827cfb ntdll!KiFastSystemCallRet
01 006afea4 7c80e5bb ntdll!NtWaitForMultipleObjects+0xc
02 006aff48 7c80e4a2 ntdll!EtwpWaitForMultipleObjectsEx+0xf7
03 006affb8 77e64829 ntdll!EtwpEventPump+0x27f
04 006affec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 2
System Thread ID: 1618
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 00d3ff9c 7c826f4b ntdll!KiFastSystemCallRet
01 00d3ffa0 7c83d424 ntdll!NtDelayExecution+0xc
02 00d3ffb8 77e64829 ntdll!RtlpTimerThread+0x47
03 00d3ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 3
System Thread ID: 11a0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 00d7ff70 7c8277db ntdll!KiFastSystemCallRet
01 00d7ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
02 00d7ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
03 00d7ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 4
System Thread ID: 374
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: Unable to locate ASP page
No remote call being made
# ChildEBP RetAddr
00 00dffe18 7c82783b ntdll!KiFastSystemCallRet
01 00dffe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
02 00dfff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
03 00dfff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
04 00dfffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
05 00dfffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
06 00dfffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 5
System Thread ID: 8b0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 00e3ff08 7c826f4b ntdll!KiFastSystemCallRet
01 00e3ff0c 77e41ed1 ntdll!NtDelayExecution+0xc
02 00e3ff74 77e424ed kernel32!SleepEx+0x68
03 00e3ff84 776bbb0f kernel32!Sleep+0xf
04 00e3ff90 776bbab4 ole32!CROIDTable::WorkerThreadLoop+0x14
05 00e3ffac 776b1704 ole32!CRpcThread::WorkerLoop+0x26
06 00e3ffb8 77e64829 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x20
07 00e3ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 6
System Thread ID: ea4
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 00ffff24 7c8277db ntdll!KiFastSystemCallRet
01 00ffff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
02 00ffff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
03 00ffff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
04 00ffffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
05 00ffffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
06 00ffffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 7
System Thread ID: eac
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 0104ff24 7c8277db ntdll!KiFastSystemCallRet
01 0104ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
02 0104ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
03 0104ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
04 0104ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
05 0104ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
06 0104ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 8
System Thread ID: 1750
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 0108ff24 7c8277db ntdll!KiFastSystemCallRet
01 0108ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
02 0108ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
03 0108ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
04 0108ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
05 0108ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
06 0108ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 9
System Thread ID: ec8
Kernel Time: 0:0:0.15
User Time: 0:0:0.0
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 010cff24 7c8277db ntdll!KiFastSystemCallRet
01 010cff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
02 010cff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
03 010cff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
04 010cffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
05 010cffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
06 010cffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 10
System Thread ID: cb8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 0118fcec 7c827cfb ntdll!KiFastSystemCallRet
01 0118fcf0 7c83c78e ntdll!NtWaitForMultipleObjects+0xc
02 0118ffb8 77e64829 ntdll!RtlpWaitThread+0x161
03 0118ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 11
System Thread ID: 904
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
*** WARNING: Unable to verify checksum for
C:\WINDOWS\system32\inetsrv\IISGuardFilter.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for
C:\WINDOWS\system32\inetsrv\IISGuardFilter.dll -
Thread Type: Other
# ChildEBP RetAddr
00 0123ff08 7739c811 ntdll!KiFastSystemCallRet
01 0123ff28 10005ee0 USER32!NtUserGetMessage+0xc
WARNING: Stack unwind information not available. Following frames may be
wrong.
02 7739be68 645653ec IISGuardFilter!HttpFilterProc+0x4b10
03 8b55ff8b 00000000 0x645653ec
Thread ID: 12
System Thread ID: 37c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: HTTP Compression Thread
# ChildEBP RetAddr
00 01bafa84 7c827d0b ntdll!KiFastSystemCallRet
01 01bafa88 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 01bafaf8 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 01bafb0c 5a3b820f kernel32!WaitForSingleObject+0x12
04 01baffb8 77e64829 w3core!HTTP_COMPRESSION::CompressionThread+0x126
05 01baffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 13
System Thread ID: 1174
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: Unable to locate ASP page
No remote call being made
# ChildEBP RetAddr
00 01befe18 7c82783b ntdll!KiFastSystemCallRet
01 01befe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
02 01beff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
03 01beff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
04 01beffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
05 01beffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
06 01beffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 14
System Thread ID: 174c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 01c2ff8c 7c826f4b ntdll!KiFastSystemCallRet
01 01c2ff90 7c81943a ntdll!NtDelayExecution+0xc
02 01c2ffb8 77e64829 ntdll!RtlpIOWorkerThread+0x3f
03 01c2ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 15
System Thread ID: ee8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: ASP
Executing Page: Unable to locate ASP page
# ChildEBP RetAddr
00 0291ff0c 7c827d0b ntdll!KiFastSystemCallRet
01 0291ff10 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 0291ff80 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 0291ff94 709fc803 kernel32!WaitForSingleObject+0x12
04 0291ffb4 709fc7db asp!CApplnCleanupMgr::ApplnCleanupDoWork+0x1e
05 0291ffb8 77e64829 asp!CApplnCleanupMgr::ApplnCleanupThread+0xa
06 0291ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 16
System Thread ID: 974
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: ASP
Executing Page: Unable to locate ASP page
# ChildEBP RetAddr
00 029dfe1c 7c827cfb ntdll!KiFastSystemCallRet
01 029dfe20 77e6202c ntdll!NtWaitForMultipleObjects+0xc
02 029dfec8 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
03 029dff24 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
04 029dff40 709fb8be USER32!MsgWaitForMultipleObjects+0x1f
05 029dff84 77bcb530 asp!CMTACallbackThread::Thread+0x4f
06 029dffb8 77e64829 msvcrt!_endthreadex+0xa3
07 029dffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 17
System Thread ID: cac
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: ASP
Executing Page: Unable to locate ASP page
# ChildEBP RetAddr
00 02a1fe1c 7c827cfb ntdll!KiFastSystemCallRet
01 02a1fe20 77e6202c ntdll!NtWaitForMultipleObjects+0xc
02 02a1fec8 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
03 02a1ff24 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
04 02a1ff40 70a017e9 USER32!MsgWaitForMultipleObjects+0x1f
05 02a1ff84 77bcb3ca asp!Debugger+0x9e
06 02a1ffb8 77e64829 msvcrt!_endthread+0xab
07 02a1ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 18
System Thread ID: 127c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: PDM (Debugger) Thread.
# ChildEBP RetAddr
00 02a6fd80 7c827cfb ntdll!KiFastSystemCallRet
01 02a6fd84 77e6202c ntdll!NtWaitForMultipleObjects+0xc
02 02a6fe2c 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
03 02a6fe88 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
04 02a6fea4 54b21868 USER32!MsgWaitForMultipleObjects+0x1f
05 02a6ff68 776bc906 pdm+0x1868
06 02a6ff88 54b31012 ole32!CoInitializeEx+0x295
07 02a6ffb0 54b30fa7 pdm+0x11012
08 02a6ffb8 77e64829 pdm+0x10fa7
09 02a6ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 19
System Thread ID: 1678
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: ASP
Executing Page: Unable to locate ASP page
# ChildEBP RetAddr
00 02b4fef4 7c827d0b ntdll!KiFastSystemCallRet
01 02b4fef8 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 02b4ff68 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 02b4ff7c 709fe8ef kernel32!WaitForSingleObject+0x12
04 02b4ffb8 77e64829 asp!CViperReqManager::WatchThread+0x68
05 02b4ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 20
System Thread ID: 12f8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Idle ASP thread
# ChildEBP RetAddr
00 02bafee8 7c827d0b ntdll!KiFastSystemCallRet
01 02bafeec 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 02baff5c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 02baff70 4a77921b kernel32!WaitForSingleObject+0x12
04 02baff84 77bcb530 comsvcs!CSTAThreadPool::LoadBalanceThreadControlLoop+0x29
05 02baffb8 77e64829 msvcrt!_endthreadex+0xa3
06 02baffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 21
System Thread ID: 16b8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Idle ASP thread
# ChildEBP RetAddr
00 02befee0 7c827d0b ntdll!KiFastSystemCallRet
01 02befee4 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 02beff54 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 02beff68 4a778cb6 kernel32!WaitForSingleObject+0x12
04 02beff84 77bcb530 comsvcs!CSTAThreadPool::KillThreadControlLoop+0x25
05 02beffb8 77e64829 msvcrt!_endthreadex+0xa3
06 02beffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 22
System Thread ID: 1748
Kernel Time: 0:0:0.171
User Time: 0:0:1.953
Thread Status: Thread is in a WAIT state.
Thread Type: Idle ASP thread
# ChildEBP RetAddr
00 02c2fdcc 7c827cfb ntdll!KiFastSystemCallRet
01 02c2fdd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
02 02c2fe78 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
03 02c2fed4 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
04 02c2fef0 4a77cb28 USER32!MsgWaitForMultipleObjects+0x1f
05 02c2ff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x1f9
06 02c2ffb8 77e64829 msvcrt!_endthreadex+0xa3
07 02c2ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 23
System Thread ID: 1610
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Idle ASP thread
# ChildEBP RetAddr
00 02c6fdcc 7c827cfb ntdll!KiFastSystemCallRet
01 02c6fdd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
02 02c6fe78 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
03 02c6fed4 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
04 02c6fef0 4a77cb28 USER32!MsgWaitForMultipleObjects+0x1f
05 02c6ff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x1f9
06 02c6ffb8 77e64829 msvcrt!_endthreadex+0xa3
07 02c6ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 24
System Thread ID: bf0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Idle ASP thread
# ChildEBP RetAddr
00 02cafdcc 7c827cfb ntdll!KiFastSystemCallRet
01 02cafdd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
02 02cafe78 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
03 02cafed4 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
04 02cafef0 4a77cb28 USER32!MsgWaitForMultipleObjects+0x1f
05 02caff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x1f9
06 02caffb8 77e64829 msvcrt!_endthreadex+0xa3
07 02caffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 25
System Thread ID: 146c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Idle ASP thread
# ChildEBP RetAddr
00 02cefdcc 7c827cfb ntdll!KiFastSystemCallRet
01 02cefdd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
02 02cefe78 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
03 02cefed4 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
04 02cefef0 4a77cb28 USER32!MsgWaitForMultipleObjects+0x1f
05 02ceff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x1f9
06 02ceffb8 77e64829 msvcrt!_endthreadex+0xa3
07 02ceffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 26
System Thread ID: d6c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Idle ASP thread
# ChildEBP RetAddr
00 03d0fd00 7c827d0b ntdll!KiFastSystemCallRet
01 03d0fd04 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 03d0fd74 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 03d0fd88 4a756c3b kernel32!WaitForSingleObject+0x12
04 03d0ffb8 77e64829 comsvcs!PingThread+0xf6
05 03d0ffec 00000000 kernel32!BaseThreadStart+0x34
Thread ID: 27
System Thread ID: 1450
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 08b4ff70 7c8277db ntdll!KiFastSystemCallRet
01 08b4ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
02 08b4ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
03 08b4ffec 00000000 kernel32!BaseThreadStart+0x34
*****
Dump name is formatted as: PID-Timestamp.dmp
Creating C:\iisstate\output\1228-1189148079.dmp - mini user dump
*****
Closing open log file C:\iisstate\output\IISState-1228.log
Re: A process serving application pool exceeded time limits during
am 16.11.2007 14:48:02 von David Wang
This worker process is pretty idle. Nothing is going on.
None of the tools like IIS State or WINDBG is going to just run and
tell you what's wrong. They need to be running at the moment when
something has gone wrong to take a stack trace. Otherwise, you get
nothing. However, the stack backtrace and memory dump is more useful
than anything you can assemble together because their information
represents what is actually wrong, assuming one can decipher the
system.
And if tools can automatically decipher bugs, we wouldn't need humans
writing/fixing software, would we?
Your event log entry indicates that something is taking longer than
expected during shutdown. That shouldn't hang anything because IIS
will simply kill the worker process after waiting for its designated
amount of time.
If you don't want to hang during shutdown, then you want to take a
stack backtrace during shutdown. You can configure tools like WINDBG
to do that, but it won't do it nor figure it out by default. You'll
have to do figure it out for the tool.
//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//
On Nov 16, 5:24 am, Savas wrote:
> Hi,
>
> I have a website that has been hanging up its application pool since this
> morning. It had done the same thing couple of months ago but it stopped doing
> it by itself.
>
> Now it started again. winDBg has not been any help. On september 7th when it
> did the same thing I had some logs by iisstate. Can somebody please look at
> it and tell me what the cause is?
>
> None of these logs make any sense to me at all.
>
> Opened log file 'C:\iisstate\output\IISState-1228.log'
>
> ***********************
> Starting new log output
> IISState version 3.3.1
>
> Fri Sep 07 09:53:49 2007
>
> OS = Windows 2003 Server
> Executable: w3wp.exe
> PID = 1228
>
> Note: Thread times are formatted as HH:MM:SS.ms
>
> ***********************
>
> Thread ID: 0
> System Thread ID: 930
> Kernel Time: 0:0:0.15
> User Time: 0:0:0.78
> Thread Status: Thread is in a WAIT state.
> Thread Type: HTTP Compression Thread
> # ChildEBP RetAddr
> 00 0006fc08 7c827d0b ntdll!KiFastSystemCallRet
> 01 0006fc0c 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 0006fc7c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> 03 0006fc90 5a364662 kernel32!WaitForSingleObject+0x12
> 04 0006fca0 5a366e3f w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
> 05 0006fca8 5a3af42d w3dt!UlAtqStartListen+0x2d
> 06 0006fcb8 5a3bc335 w3core!W3_SERVER::StartListen+0xbd
> 07 0006ff0c 0100187c w3core!UlW3Start+0x26e
> 08 0006ff44 01001a27 w3wp!wmain+0x22a
> 09 0006ffc0 77e6f23b w3wp!wmainCRTStartup+0x12f
> 0a 0006fff0 00000000 kernel32!BaseProcessStart+0x23
>
> Thread ID: 1
> System Thread ID: 1204
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 006afea0 7c827cfb ntdll!KiFastSystemCallRet
> 01 006afea4 7c80e5bb ntdll!NtWaitForMultipleObjects+0xc
> 02 006aff48 7c80e4a2 ntdll!EtwpWaitForMultipleObjectsEx+0xf7
> 03 006affb8 77e64829 ntdll!EtwpEventPump+0x27f
> 04 006affec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 2
> System Thread ID: 1618
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Other
> # ChildEBP RetAddr
> 00 00d3ff9c 7c826f4b ntdll!KiFastSystemCallRet
> 01 00d3ffa0 7c83d424 ntdll!NtDelayExecution+0xc
> 02 00d3ffb8 77e64829 ntdll!RtlpTimerThread+0x47
> 03 00d3ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 3
> System Thread ID: 11a0
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Other
> # ChildEBP RetAddr
> 00 00d7ff70 7c8277db ntdll!KiFastSystemCallRet
> 01 00d7ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
> 02 00d7ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
> 03 00d7ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 4
> System Thread ID: 374
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Possible ASP page. Possible DCOM activity
> Executing Page: Unable to locate ASP page
>
> No remote call being made
>
> # ChildEBP RetAddr
> 00 00dffe18 7c82783b ntdll!KiFastSystemCallRet
> 01 00dffe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
> 02 00dfff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
> 03 00dfff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
> 04 00dfffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
> 05 00dfffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
> 06 00dfffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 5
> System Thread ID: 8b0
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Other
> # ChildEBP RetAddr
> 00 00e3ff08 7c826f4b ntdll!KiFastSystemCallRet
> 01 00e3ff0c 77e41ed1 ntdll!NtDelayExecution+0xc
> 02 00e3ff74 77e424ed kernel32!SleepEx+0x68
> 03 00e3ff84 776bbb0f kernel32!Sleep+0xf
> 04 00e3ff90 776bbab4 ole32!CROIDTable::WorkerThreadLoop+0x14
> 05 00e3ffac 776b1704 ole32!CRpcThread::WorkerLoop+0x26
> 06 00e3ffb8 77e64829 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x20
> 07 00e3ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 6
> System Thread ID: ea4
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: HTTP Listener
> # ChildEBP RetAddr
> 00 00ffff24 7c8277db ntdll!KiFastSystemCallRet
> 01 00ffff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> 02 00ffff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
> 03 00ffff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
> 04 00ffffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
> 05 00ffffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
> 06 00ffffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 7
> System Thread ID: eac
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: HTTP Listener
> # ChildEBP RetAddr
> 00 0104ff24 7c8277db ntdll!KiFastSystemCallRet
> 01 0104ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> 02 0104ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
> 03 0104ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
> 04 0104ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
> 05 0104ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
> 06 0104ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 8
> System Thread ID: 1750
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: HTTP Listener
> # ChildEBP RetAddr
> 00 0108ff24 7c8277db ntdll!KiFastSystemCallRet
> 01 0108ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> 02 0108ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
> 03 0108ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
> 04 0108ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
> 05 0108ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
> 06 0108ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 9
> System Thread ID: ec8
> Kernel Time: 0:0:0.15
> User Time: 0:0:0.0
> Thread Type: HTTP Listener
> # ChildEBP RetAddr
> 00 010cff24 7c8277db ntdll!KiFastSystemCallRet
> 01 010cff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> 02 010cff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
> 03 010cff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
> 04 010cffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
> 05 010cffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
> 06 010cffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 10
> System Thread ID: cb8
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 0118fcec 7c827cfb ntdll!KiFastSystemCallRet
> 01 0118fcf0 7c83c78e ntdll!NtWaitForMultipleObjects+0xc
> 02 0118ffb8 77e64829 ntdll!RtlpWaitThread+0x161
> 03 0118ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 11
> System Thread ID: 904
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> *** WARNING: Unable to verify checksum for
> C:\WINDOWS\system32\inetsrv\IISGuardFilter.dll
> *** ERROR: Symbol file could not be found. Defaulted to export symbols for
> C:\WINDOWS\system32\inetsrv\IISGuardFilter.dll -
> Thread Type: Other
> # ChildEBP RetAddr
> 00 0123ff08 7739c811 ntdll!KiFastSystemCallRet
> 01 0123ff28 10005ee0 USER32!NtUserGetMessage+0xc
> WARNING: Stack unwind information not available. Following frames may be
> wrong.
> 02 7739be68 645653ec IISGuardFilter!HttpFilterProc+0x4b10
> 03 8b55ff8b 00000000 0x645653ec
>
> Thread ID: 12
> System Thread ID: 37c
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: HTTP Compression Thread
> # ChildEBP RetAddr
> 00 01bafa84 7c827d0b ntdll!KiFastSystemCallRet
> 01 01bafa88 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 01bafaf8 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> 03 01bafb0c 5a3b820f kernel32!WaitForSingleObject+0x12
> 04 01baffb8 77e64829 w3core!HTTP_COMPRESSION::CompressionThread+0x126
> 05 01baffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 13
> System Thread ID: 1174
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Possible ASP page. Possible DCOM activity
> Executing Page: Unable to locate ASP page
>
> No remote call being made
>
> # ChildEBP RetAddr
> 00 01befe18 7c82783b ntdll!KiFastSystemCallRet
> 01 01befe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
> 02 01beff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
> 03 01beff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
> 04 01beffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
> 05 01beffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
> 06 01beffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 14
> System Thread ID: 174c
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Other
> # ChildEBP RetAddr
> 00 01c2ff8c 7c826f4b ntdll!KiFastSystemCallRet
> 01 01c2ff90 7c81943a ntdll!NtDelayExecution+0xc
> 02 01c2ffb8 77e64829 ntdll!RtlpIOWorkerThread+0x3f
> 03 01c2ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 15
> System Thread ID: ee8
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: ASP
> Executing Page: Unable to locate ASP page
>
> # ChildEBP RetAddr
> 00 0291ff0c 7c827d0b ntdll!KiFastSystemCallRet
> 01 0291ff10 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 0291ff80 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> 03 0291ff94 709fc803 kernel32!WaitForSingleObject+0x12
> 04 0291ffb4 709fc7db asp!CApplnCleanupMgr::ApplnCleanupDoWork+0x1e
> 05 0291ffb8 77e64829 asp!CApplnCleanupMgr::ApplnCleanupThread+0xa
> 06 0291ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 16
> System Thread ID: 974
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: ASP
> Executing Page: Unable to locate ASP page
>
> # ChildEBP RetAddr
> 00 029dfe1c 7c827cfb ntdll!KiFastSystemCallRet
> 01 029dfe20 77e6202c ntdll!NtWaitForMultipleObjects+0xc
> 02 029dfec8 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
> 03 029dff24 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
> 04 029dff40 709fb8be USER32!MsgWaitForMultipleObjects+0x1f
> 05 029dff84 77bcb530 asp!CMTACallbackThread::Thread+0x4f
> 06 029dffb8 77e64829 msvcrt!_endthreadex+0xa3
> 07 029dffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 17
> System Thread ID: cac
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: ASP
> Executing Page: Unable to locate ASP page
>
> # ChildEBP RetAddr
> 00 02a1fe1c 7c827cfb ntdll!KiFastSystemCallRet
> 01 02a1fe20 77e6202c ntdll!NtWaitForMultipleObjects+0xc
> 02 02a1fec8 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
> 03 02a1ff24 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
> 04 02a1ff40 70a017e9 USER32!MsgWaitForMultipleObjects+0x1f
> 05 02a1ff84 77bcb3ca asp!Debugger+0x9e
> 06 02a1ffb8 77e64829 msvcrt!_endthread+0xab
> 07 02a1ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 18
> System Thread ID: 127c
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: PDM (Debugger) Thread.
> # ChildEBP RetAddr
> 00 02a6fd80 7c827cfb ntdll!KiFastSystemCallRet
> 01 02a6fd84 77e6202c ntdll!NtWaitForMultipleObjects+0xc
> 02 02a6fe2c 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
> 03 02a6fe88 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
> 04 02a6fea4 54b21868 USER32!MsgWaitForMultipleObjects+0x1f
> 05 02a6ff68 776bc906 pdm+0x1868
> 06 02a6ff88 54b31012 ole32!CoInitializeEx+0x295
> 07 02a6ffb0 54b30fa7 pdm+0x11012
> 08 02a6ffb8 77e64829 pdm+0x10fa7
> 09 02a6ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 19
> System Thread ID: 1678
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: ASP
> Executing Page: Unable to locate ASP page
>
> # ChildEBP RetAddr
> 00 02b4fef4 7c827d0b ntdll!KiFastSystemCallRet
> 01 02b4fef8 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 02b4ff68 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> 03 02b4ff7c 709fe8ef kernel32!WaitForSingleObject+0x12
> 04 02b4ffb8 77e64829 asp!CViperReqManager::WatchThread+0x68
> 05 02b4ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 20
> System Thread ID: 12f8
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: Idle ASP thread
> # ChildEBP RetAddr
> 00 02bafee8 7c827d0b ntdll!KiFastSystemCallRet
> 01 02bafeec 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 02baff5c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> 03 02baff70 4a77921b kernel32!WaitForSingleObject+0x12
> 04 02baff84 77bcb530 comsvcs!CSTAThreadPool::LoadBalanceThreadControlLoop+0x29
> 05 02baffb8 77e64829 msvcrt!_endthreadex+0xa3
> 06 02baffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 21
> System Thread ID: 16b8
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: Idle ASP thread
> # ChildEBP RetAddr
> 00 02befee0 7c827d0b ntdll!KiFastSystemCallRet
> 01 02befee4 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 02beff54 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> 03 02beff68 4a778cb6 kernel32!WaitForSingleObject+0x12
> 04 02beff84 77bcb530 comsvcs!CSTAThreadPool::KillThreadControlLoop+0x25
> 05 02beffb8 77e64829 msvcrt!_endthreadex+0xa3
> 06 02beffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 22
> System Thread ID: 1748
> Kernel Time: 0:0:0.171
> User Time: 0:0:1.953
> Thread Status: Thread is in a WAIT state.
> Thread Type: Idle ASP thread
> # ChildEBP RetAddr
> 00 02c2fdcc 7c827cfb ntdll!KiFastSystemCallRet
> 01 02c2fdd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
> 02 02c2fe78 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
> 03 02c2fed4 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
> 04 02c2fef0 4a77cb28 USER32!MsgWaitForMultipleObjects+0x1f
> 05 02c2ff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x1f9
> 06 02c2ffb8 77e64829 msvcrt!_endthreadex+0xa3
> 07 02c2ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 23
> System Thread ID: 1610
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: Idle ASP thread
> # ChildEBP RetAddr
> 00 02c6fdcc 7c827cfb ntdll!KiFastSystemCallRet
> 01 02c6fdd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
> 02 02c6fe78 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
> 03 02c6fed4 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
> 04 02c6fef0 4a77cb28 USER32!MsgWaitForMultipleObjects+0x1f
> 05 02c6ff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x1f9
> 06 02c6ffb8 77e64829 msvcrt!_endthreadex+0xa3
> 07 02c6ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 24
> System Thread ID: bf0
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: Idle ASP thread
> # ChildEBP RetAddr
> 00 02cafdcc 7c827cfb ntdll!KiFastSystemCallRet
> 01 02cafdd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
> 02 02cafe78 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
> 03 02cafed4 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
> 04 02cafef0 4a77cb28 USER32!MsgWaitForMultipleObjects+0x1f
> 05 02caff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x1f9
> 06 02caffb8 77e64829 msvcrt!_endthreadex+0xa3
> 07 02caffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 25
> System Thread ID: 146c
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: Idle ASP thread
> # ChildEBP RetAddr
> 00 02cefdcc 7c827cfb ntdll!KiFastSystemCallRet
> 01 02cefdd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
> 02 02cefe78 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
> 03 02cefed4 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
> 04 02cefef0 4a77cb28 USER32!MsgWaitForMultipleObjects+0x1f
> 05 02ceff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x1f9
> 06 02ceffb8 77e64829 msvcrt!_endthreadex+0xa3
> 07 02ceffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 26
> System Thread ID: d6c
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: Idle ASP thread
> # ChildEBP RetAddr
> 00 03d0fd00 7c827d0b ntdll!KiFastSystemCallRet
> 01 03d0fd04 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 03d0fd74 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> 03 03d0fd88 4a756c3b kernel32!WaitForSingleObject+0x12
> 04 03d0ffb8 77e64829 comsvcs!PingThread+0xf6
> 05 03d0ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 27
> System Thread ID: 1450
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Other
> # ChildEBP RetAddr
> 00 08b4ff70 7c8277db ntdll!KiFastSystemCallRet
> 01 08b4ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
> 02 08b4ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
> 03 08b4ffec 00000000 kernel32!BaseThreadStart+0x34
>
> *****
>
> Dump name is formatted as: PID-Timestamp.dmp
>
> Creating C:\iisstate\output\1228-1189148079.dmp - mini user dump
>
> *****
>
> Closing open log file C:\iisstate\output\IISState-1228.log
Re: A process serving application pool exceeded time limits during shu
am 16.11.2007 15:05:31 von Savas
Many thanks for your fast response David.
Unfortunately I lack the expertise to deal with this kind of situations.
Please, tell me if I am doing the right thing.
I find out the pid of the applicaiton pool with iisapp.vbs
Say it is 2345
The I open WinDBG and attach it to this process
then I am getting things like this:
ModLoad: 08180000 08186000 C:\Program Files\Common
Files\System\ado\msader15.dll
(da4.c88): Break instruction exception - code 80000003 (first chance)
eax=7ffdc000 ebx=00000001 ecx=00000002 edx=00000003 esi=00000004
edi=00000005
eip=7c81a3e1 esp=08eaffcc ebp=08eafff4 iopl=0 nv up ei pl zr na
pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000
efl=00000246
ntdll!DbgBreakPoint:
7c81a3e1 cc int 3
Then I reload the symbols
and press F5 and start debugging.
Is this the right thing to do?
During this debug, the same hangup happened but all I got on WinDBG were
these three lines:
ModLoad: 4bbe0000 4bc72000 C:\Program Files\Common
Files\System\ado\msado15.dll
ModLoad: 4bbe0000 4bc72000 C:\Program Files\Common
Files\System\ado\msado15.dll
ModLoad: 00670000 00676000 C:\Program Files\Common
Files\System\ado\msader15.dll
You said I need to take a stack backtrace during shutdown. Can you
please tell me how to do it?
David Wang wrote:
> This worker process is pretty idle. Nothing is going on.
>
> None of the tools like IIS State or WINDBG is going to just run and
> tell you what's wrong. They need to be running at the moment when
> something has gone wrong to take a stack trace. Otherwise, you get
> nothing. However, the stack backtrace and memory dump is more useful
> than anything you can assemble together because their information
> represents what is actually wrong, assuming one can decipher the
> system.
>
> And if tools can automatically decipher bugs, we wouldn't need humans
> writing/fixing software, would we?
>
> Your event log entry indicates that something is taking longer than
> expected during shutdown. That shouldn't hang anything because IIS
> will simply kill the worker process after waiting for its designated
> amount of time.
>
> If you don't want to hang during shutdown, then you want to take a
> stack backtrace during shutdown. You can configure tools like WINDBG
> to do that, but it won't do it nor figure it out by default. You'll
> have to do figure it out for the tool.
>
>
>
> //David
> http://w3-4u.blogspot.com
> http://blogs.msdn.com/David.Wang
> //
>
>
>
>
> On Nov 16, 5:24 am, Savas wrote:
>> Hi,
>>
>> I have a website that has been hanging up its application pool since this
>> morning. It had done the same thing couple of months ago but it stopped doing
>> it by itself.
>>
>> Now it started again. winDBg has not been any help. On september 7th when it
>> did the same thing I had some logs by iisstate. Can somebody please look at
>> it and tell me what the cause is?
>>
>> None of these logs make any sense to me at all.
>>
>> Opened log file 'C:\iisstate\output\IISState-1228.log'
>>
>> ***********************
>> Starting new log output
>> IISState version 3.3.1
>>
>> Fri Sep 07 09:53:49 2007
>>
>> OS = Windows 2003 Server
>> Executable: w3wp.exe
>> PID = 1228
>>
>> Note: Thread times are formatted as HH:MM:SS.ms
>>
>> ***********************
>>
>> Thread ID: 0
>> System Thread ID: 930
>> Kernel Time: 0:0:0.15
>> User Time: 0:0:0.78
>> Thread Status: Thread is in a WAIT state.
>> Thread Type: HTTP Compression Thread
>> # ChildEBP RetAddr
>> 00 0006fc08 7c827d0b ntdll!KiFastSystemCallRet
>> 01 0006fc0c 77e61d1e ntdll!NtWaitForSingleObject+0xc
>> 02 0006fc7c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
>> 03 0006fc90 5a364662 kernel32!WaitForSingleObject+0x12
>> 04 0006fca0 5a366e3f w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
>> 05 0006fca8 5a3af42d w3dt!UlAtqStartListen+0x2d
>> 06 0006fcb8 5a3bc335 w3core!W3_SERVER::StartListen+0xbd
>> 07 0006ff0c 0100187c w3core!UlW3Start+0x26e
>> 08 0006ff44 01001a27 w3wp!wmain+0x22a
>> 09 0006ffc0 77e6f23b w3wp!wmainCRTStartup+0x12f
>> 0a 0006fff0 00000000 kernel32!BaseProcessStart+0x23
>>
>> Thread ID: 1
>> System Thread ID: 1204
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Status: Thread is in a WAIT state.
>> Thread Type: Other
>> # ChildEBP RetAddr
>> 00 006afea0 7c827cfb ntdll!KiFastSystemCallRet
>> 01 006afea4 7c80e5bb ntdll!NtWaitForMultipleObjects+0xc
>> 02 006aff48 7c80e4a2 ntdll!EtwpWaitForMultipleObjectsEx+0xf7
>> 03 006affb8 77e64829 ntdll!EtwpEventPump+0x27f
>> 04 006affec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 2
>> System Thread ID: 1618
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Type: Other
>> # ChildEBP RetAddr
>> 00 00d3ff9c 7c826f4b ntdll!KiFastSystemCallRet
>> 01 00d3ffa0 7c83d424 ntdll!NtDelayExecution+0xc
>> 02 00d3ffb8 77e64829 ntdll!RtlpTimerThread+0x47
>> 03 00d3ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 3
>> System Thread ID: 11a0
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Type: Other
>> # ChildEBP RetAddr
>> 00 00d7ff70 7c8277db ntdll!KiFastSystemCallRet
>> 01 00d7ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
>> 02 00d7ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
>> 03 00d7ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 4
>> System Thread ID: 374
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Type: Possible ASP page. Possible DCOM activity
>> Executing Page: Unable to locate ASP page
>>
>> No remote call being made
>>
>> # ChildEBP RetAddr
>> 00 00dffe18 7c82783b ntdll!KiFastSystemCallRet
>> 01 00dffe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
>> 02 00dfff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
>> 03 00dfff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
>> 04 00dfffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
>> 05 00dfffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
>> 06 00dfffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 5
>> System Thread ID: 8b0
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Type: Other
>> # ChildEBP RetAddr
>> 00 00e3ff08 7c826f4b ntdll!KiFastSystemCallRet
>> 01 00e3ff0c 77e41ed1 ntdll!NtDelayExecution+0xc
>> 02 00e3ff74 77e424ed kernel32!SleepEx+0x68
>> 03 00e3ff84 776bbb0f kernel32!Sleep+0xf
>> 04 00e3ff90 776bbab4 ole32!CROIDTable::WorkerThreadLoop+0x14
>> 05 00e3ffac 776b1704 ole32!CRpcThread::WorkerLoop+0x26
>> 06 00e3ffb8 77e64829 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x20
>> 07 00e3ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 6
>> System Thread ID: ea4
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Type: HTTP Listener
>> # ChildEBP RetAddr
>> 00 00ffff24 7c8277db ntdll!KiFastSystemCallRet
>> 01 00ffff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
>> 02 00ffff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
>> 03 00ffff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
>> 04 00ffffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
>> 05 00ffffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
>> 06 00ffffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 7
>> System Thread ID: eac
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Type: HTTP Listener
>> # ChildEBP RetAddr
>> 00 0104ff24 7c8277db ntdll!KiFastSystemCallRet
>> 01 0104ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
>> 02 0104ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
>> 03 0104ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
>> 04 0104ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
>> 05 0104ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
>> 06 0104ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 8
>> System Thread ID: 1750
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Type: HTTP Listener
>> # ChildEBP RetAddr
>> 00 0108ff24 7c8277db ntdll!KiFastSystemCallRet
>> 01 0108ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
>> 02 0108ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
>> 03 0108ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
>> 04 0108ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
>> 05 0108ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
>> 06 0108ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 9
>> System Thread ID: ec8
>> Kernel Time: 0:0:0.15
>> User Time: 0:0:0.0
>> Thread Type: HTTP Listener
>> # ChildEBP RetAddr
>> 00 010cff24 7c8277db ntdll!KiFastSystemCallRet
>> 01 010cff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
>> 02 010cff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
>> 03 010cff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
>> 04 010cffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
>> 05 010cffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
>> 06 010cffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 10
>> System Thread ID: cb8
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Status: Thread is in a WAIT state.
>> Thread Type: Other
>> # ChildEBP RetAddr
>> 00 0118fcec 7c827cfb ntdll!KiFastSystemCallRet
>> 01 0118fcf0 7c83c78e ntdll!NtWaitForMultipleObjects+0xc
>> 02 0118ffb8 77e64829 ntdll!RtlpWaitThread+0x161
>> 03 0118ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 11
>> System Thread ID: 904
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> *** WARNING: Unable to verify checksum for
>> C:\WINDOWS\system32\inetsrv\IISGuardFilter.dll
>> *** ERROR: Symbol file could not be found. Defaulted to export symbols for
>> C:\WINDOWS\system32\inetsrv\IISGuardFilter.dll -
>> Thread Type: Other
>> # ChildEBP RetAddr
>> 00 0123ff08 7739c811 ntdll!KiFastSystemCallRet
>> 01 0123ff28 10005ee0 USER32!NtUserGetMessage+0xc
>> WARNING: Stack unwind information not available. Following frames may be
>> wrong.
>> 02 7739be68 645653ec IISGuardFilter!HttpFilterProc+0x4b10
>> 03 8b55ff8b 00000000 0x645653ec
>>
>> Thread ID: 12
>> System Thread ID: 37c
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Status: Thread is in a WAIT state.
>> Thread Type: HTTP Compression Thread
>> # ChildEBP RetAddr
>> 00 01bafa84 7c827d0b ntdll!KiFastSystemCallRet
>> 01 01bafa88 77e61d1e ntdll!NtWaitForSingleObject+0xc
>> 02 01bafaf8 77e61c8d kernel32!WaitForSingleObjectEx+0xac
>> 03 01bafb0c 5a3b820f kernel32!WaitForSingleObject+0x12
>> 04 01baffb8 77e64829 w3core!HTTP_COMPRESSION::CompressionThread+0x126
>> 05 01baffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 13
>> System Thread ID: 1174
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Type: Possible ASP page. Possible DCOM activity
>> Executing Page: Unable to locate ASP page
>>
>> No remote call being made
>>
>> # ChildEBP RetAddr
>> 00 01befe18 7c82783b ntdll!KiFastSystemCallRet
>> 01 01befe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
>> 02 01beff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
>> 03 01beff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
>> 04 01beffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
>> 05 01beffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
>> 06 01beffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 14
>> System Thread ID: 174c
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Type: Other
>> # ChildEBP RetAddr
>> 00 01c2ff8c 7c826f4b ntdll!KiFastSystemCallRet
>> 01 01c2ff90 7c81943a ntdll!NtDelayExecution+0xc
>> 02 01c2ffb8 77e64829 ntdll!RtlpIOWorkerThread+0x3f
>> 03 01c2ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 15
>> System Thread ID: ee8
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Status: Thread is in a WAIT state.
>> Thread Type: ASP
>> Executing Page: Unable to locate ASP page
>>
>> # ChildEBP RetAddr
>> 00 0291ff0c 7c827d0b ntdll!KiFastSystemCallRet
>> 01 0291ff10 77e61d1e ntdll!NtWaitForSingleObject+0xc
>> 02 0291ff80 77e61c8d kernel32!WaitForSingleObjectEx+0xac
>> 03 0291ff94 709fc803 kernel32!WaitForSingleObject+0x12
>> 04 0291ffb4 709fc7db asp!CApplnCleanupMgr::ApplnCleanupDoWork+0x1e
>> 05 0291ffb8 77e64829 asp!CApplnCleanupMgr::ApplnCleanupThread+0xa
>> 06 0291ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 16
>> System Thread ID: 974
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Status: Thread is in a WAIT state.
>> Thread Type: ASP
>> Executing Page: Unable to locate ASP page
>>
>> # ChildEBP RetAddr
>> 00 029dfe1c 7c827cfb ntdll!KiFastSystemCallRet
>> 01 029dfe20 77e6202c ntdll!NtWaitForMultipleObjects+0xc
>> 02 029dfec8 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
>> 03 029dff24 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
>> 04 029dff40 709fb8be USER32!MsgWaitForMultipleObjects+0x1f
>> 05 029dff84 77bcb530 asp!CMTACallbackThread::Thread+0x4f
>> 06 029dffb8 77e64829 msvcrt!_endthreadex+0xa3
>> 07 029dffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 17
>> System Thread ID: cac
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Status: Thread is in a WAIT state.
>> Thread Type: ASP
>> Executing Page: Unable to locate ASP page
>>
>> # ChildEBP RetAddr
>> 00 02a1fe1c 7c827cfb ntdll!KiFastSystemCallRet
>> 01 02a1fe20 77e6202c ntdll!NtWaitForMultipleObjects+0xc
>> 02 02a1fec8 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
>> 03 02a1ff24 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
>> 04 02a1ff40 70a017e9 USER32!MsgWaitForMultipleObjects+0x1f
>> 05 02a1ff84 77bcb3ca asp!Debugger+0x9e
>> 06 02a1ffb8 77e64829 msvcrt!_endthread+0xab
>> 07 02a1ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 18
>> System Thread ID: 127c
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Status: Thread is in a WAIT state.
>> Thread Type: PDM (Debugger) Thread.
>> # ChildEBP RetAddr
>> 00 02a6fd80 7c827cfb ntdll!KiFastSystemCallRet
>> 01 02a6fd84 77e6202c ntdll!NtWaitForMultipleObjects+0xc
>> 02 02a6fe2c 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
>> 03 02a6fe88 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
>> 04 02a6fea4 54b21868 USER32!MsgWaitForMultipleObjects+0x1f
>> 05 02a6ff68 776bc906 pdm+0x1868
>> 06 02a6ff88 54b31012 ole32!CoInitializeEx+0x295
>> 07 02a6ffb0 54b30fa7 pdm+0x11012
>> 08 02a6ffb8 77e64829 pdm+0x10fa7
>> 09 02a6ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 19
>> System Thread ID: 1678
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Status: Thread is in a WAIT state.
>> Thread Type: ASP
>> Executing Page: Unable to locate ASP page
>>
>> # ChildEBP RetAddr
>> 00 02b4fef4 7c827d0b ntdll!KiFastSystemCallRet
>> 01 02b4fef8 77e61d1e ntdll!NtWaitForSingleObject+0xc
>> 02 02b4ff68 77e61c8d kernel32!WaitForSingleObjectEx+0xac
>> 03 02b4ff7c 709fe8ef kernel32!WaitForSingleObject+0x12
>> 04 02b4ffb8 77e64829 asp!CViperReqManager::WatchThread+0x68
>> 05 02b4ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 20
>> System Thread ID: 12f8
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Status: Thread is in a WAIT state.
>> Thread Type: Idle ASP thread
>> # ChildEBP RetAddr
>> 00 02bafee8 7c827d0b ntdll!KiFastSystemCallRet
>> 01 02bafeec 77e61d1e ntdll!NtWaitForSingleObject+0xc
>> 02 02baff5c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
>> 03 02baff70 4a77921b kernel32!WaitForSingleObject+0x12
>> 04 02baff84 77bcb530 comsvcs!CSTAThreadPool::LoadBalanceThreadControlLoop+0x29
>> 05 02baffb8 77e64829 msvcrt!_endthreadex+0xa3
>> 06 02baffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 21
>> System Thread ID: 16b8
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Status: Thread is in a WAIT state.
>> Thread Type: Idle ASP thread
>> # ChildEBP RetAddr
>> 00 02befee0 7c827d0b ntdll!KiFastSystemCallRet
>> 01 02befee4 77e61d1e ntdll!NtWaitForSingleObject+0xc
>> 02 02beff54 77e61c8d kernel32!WaitForSingleObjectEx+0xac
>> 03 02beff68 4a778cb6 kernel32!WaitForSingleObject+0x12
>> 04 02beff84 77bcb530 comsvcs!CSTAThreadPool::KillThreadControlLoop+0x25
>> 05 02beffb8 77e64829 msvcrt!_endthreadex+0xa3
>> 06 02beffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 22
>> System Thread ID: 1748
>> Kernel Time: 0:0:0.171
>> User Time: 0:0:1.953
>> Thread Status: Thread is in a WAIT state.
>> Thread Type: Idle ASP thread
>> # ChildEBP RetAddr
>> 00 02c2fdcc 7c827cfb ntdll!KiFastSystemCallRet
>> 01 02c2fdd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
>> 02 02c2fe78 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
>> 03 02c2fed4 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
>> 04 02c2fef0 4a77cb28 USER32!MsgWaitForMultipleObjects+0x1f
>> 05 02c2ff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x1f9
>> 06 02c2ffb8 77e64829 msvcrt!_endthreadex+0xa3
>> 07 02c2ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 23
>> System Thread ID: 1610
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Status: Thread is in a WAIT state.
>> Thread Type: Idle ASP thread
>> # ChildEBP RetAddr
>> 00 02c6fdcc 7c827cfb ntdll!KiFastSystemCallRet
>> 01 02c6fdd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
>> 02 02c6fe78 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
>> 03 02c6fed4 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
>> 04 02c6fef0 4a77cb28 USER32!MsgWaitForMultipleObjects+0x1f
>> 05 02c6ff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x1f9
>> 06 02c6ffb8 77e64829 msvcrt!_endthreadex+0xa3
>> 07 02c6ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 24
>> System Thread ID: bf0
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Status: Thread is in a WAIT state.
>> Thread Type: Idle ASP thread
>> # ChildEBP RetAddr
>> 00 02cafdcc 7c827cfb ntdll!KiFastSystemCallRet
>> 01 02cafdd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
>> 02 02cafe78 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
>> 03 02cafed4 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
>> 04 02cafef0 4a77cb28 USER32!MsgWaitForMultipleObjects+0x1f
>> 05 02caff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x1f9
>> 06 02caffb8 77e64829 msvcrt!_endthreadex+0xa3
>> 07 02caffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 25
>> System Thread ID: 146c
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Status: Thread is in a WAIT state.
>> Thread Type: Idle ASP thread
>> # ChildEBP RetAddr
>> 00 02cefdcc 7c827cfb ntdll!KiFastSystemCallRet
>> 01 02cefdd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
>> 02 02cefe78 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
>> 03 02cefed4 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
>> 04 02cefef0 4a77cb28 USER32!MsgWaitForMultipleObjects+0x1f
>> 05 02ceff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x1f9
>> 06 02ceffb8 77e64829 msvcrt!_endthreadex+0xa3
>> 07 02ceffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 26
>> System Thread ID: d6c
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Status: Thread is in a WAIT state.
>> Thread Type: Idle ASP thread
>> # ChildEBP RetAddr
>> 00 03d0fd00 7c827d0b ntdll!KiFastSystemCallRet
>> 01 03d0fd04 77e61d1e ntdll!NtWaitForSingleObject+0xc
>> 02 03d0fd74 77e61c8d kernel32!WaitForSingleObjectEx+0xac
>> 03 03d0fd88 4a756c3b kernel32!WaitForSingleObject+0x12
>> 04 03d0ffb8 77e64829 comsvcs!PingThread+0xf6
>> 05 03d0ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> Thread ID: 27
>> System Thread ID: 1450
>> Kernel Time: 0:0:0.0
>> User Time: 0:0:0.0
>> Thread Type: Other
>> # ChildEBP RetAddr
>> 00 08b4ff70 7c8277db ntdll!KiFastSystemCallRet
>> 01 08b4ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
>> 02 08b4ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
>> 03 08b4ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> *****
>>
>> Dump name is formatted as: PID-Timestamp.dmp
>>
>> Creating C:\iisstate\output\1228-1189148079.dmp - mini user dump
>>
>> *****
>>
>> Closing open log file C:\iisstate\output\IISState-1228.log
>
Re: A process serving application pool exceeded time limits during
am 17.11.2007 12:39:28 von David Wang
I recommend that you contact debugging professionals, such as
Microsoft PSS, for debugging support. I have no problems helping you
understand what is going on and using the tools, but I cannot take
responsibility to debug for you or hand-hold you step-by-step.
All of the following commands (and more) are documented within
debugger.chm that comes with the Microsoft Debugging Toolkit. I
suggest that you break into the w3wp.exe process during the hang up
and run the following command to capture a memory dump:
..dump /ma
Then, you can load that memory dump file with debuggers like windbg
and debug at leisure.
~*kb50
with proper symbols should get you started. When you have stack
backtrace you may notice something strange or hanging with some of the
stacks. Depending on the situation, there are other debugging commands
to get more information, but they generally require Windows Systems
experience beyond the scope of this newsgroup.
//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//
On Nov 16, 6:05 am, "M. Savas Zorlu" wrote:
> Many thanks for your fast response David.
>
> Unfortunately I lack the expertise to deal with this kind of situations.
>
> Please, tell me if I am doing the right thing.
>
> I find out the pid of the applicaiton pool with iisapp.vbs
>
> Say it is 2345
>
> The I open WinDBG and attach it to this process
>
> then I am getting things like this:
>
> ModLoad: 08180000 08186000 C:\Program Files\Common
> Files\System\ado\msader15.dll
> (da4.c88): Break instruction exception - code 80000003 (first chance)
> eax=7ffdc000 ebx=00000001 ecx=00000002 edx=00000003 esi=00000004
> edi=00000005
> eip=7c81a3e1 esp=08eaffcc ebp=08eafff4 iopl=0 nv up ei pl zr na
> pe nc
> cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000
> efl=00000246
> ntdll!DbgBreakPoint:
> 7c81a3e1 cc int 3
>
> Then I reload the symbols
>
> and press F5 and start debugging.
>
> Is this the right thing to do?
>
> During this debug, the same hangup happened but all I got on WinDBG were
> these three lines:
>
> ModLoad: 4bbe0000 4bc72000 C:\Program Files\Common
> Files\System\ado\msado15.dll
> ModLoad: 4bbe0000 4bc72000 C:\Program Files\Common
> Files\System\ado\msado15.dll
> ModLoad: 00670000 00676000 C:\Program Files\Common
> Files\System\ado\msader15.dll
>
> You said I need to take a stack backtrace during shutdown. Can you
> please tell me how to do it?
>
>
>
> David Wang wrote:
> > This worker process is pretty idle. Nothing is going on.
>
> > None of the tools like IIS State or WINDBG is going to just run and
> > tell you what's wrong. They need to be running at the moment when
> > something has gone wrong to take a stack trace. Otherwise, you get
> > nothing. However, the stack backtrace and memory dump is more useful
> > than anything you can assemble together because their information
> > represents what is actually wrong, assuming one can decipher the
> > system.
>
> > And if tools can automatically decipher bugs, we wouldn't need humans
> > writing/fixing software, would we?
>
> > Your event log entry indicates that something is taking longer than
> > expected during shutdown. That shouldn't hang anything because IIS
> > will simply kill the worker process after waiting for its designated
> > amount of time.
>
> > If you don't want to hang during shutdown, then you want to take a
> > stack backtrace during shutdown. You can configure tools like WINDBG
> > to do that, but it won't do it nor figure it out by default. You'll
> > have to do figure it out for the tool.
>
> > //David
> >http://w3-4u.blogspot.com
> >http://blogs.msdn.com/David.Wang
> > //
>
> > On Nov 16, 5:24 am, Savas wrote:
> >> Hi,
>
> >> I have a website that has been hanging up its application pool since this
> >> morning. It had done the same thing couple of months ago but it stopped doing
> >> it by itself.
>
> >> Now it started again. winDBg has not been any help. On september 7th when it
> >> did the same thing I had some logs by iisstate. Can somebody please look at
> >> it and tell me what the cause is?
>
> >> None of these logs make any sense to me at all.
>
> >> Opened log file 'C:\iisstate\output\IISState-1228.log'
>
> >> ***********************
> >> Starting new log output
> >> IISState version 3.3.1
>
> >> Fri Sep 07 09:53:49 2007
>
> >> OS = Windows 2003 Server
> >> Executable: w3wp.exe
> >> PID = 1228
>
> >> Note: Thread times are formatted as HH:MM:SS.ms
>
> >> ***********************
>
> >> Thread ID: 0
> >> System Thread ID: 930
> >> Kernel Time: 0:0:0.15
> >> User Time: 0:0:0.78
> >> Thread Status: Thread is in a WAIT state.
> >> Thread Type: HTTP Compression Thread
> >> # ChildEBP RetAddr
> >> 00 0006fc08 7c827d0b ntdll!KiFastSystemCallRet
> >> 01 0006fc0c 77e61d1e ntdll!NtWaitForSingleObject+0xc
> >> 02 0006fc7c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> >> 03 0006fc90 5a364662 kernel32!WaitForSingleObject+0x12
> >> 04 0006fca0 5a366e3f w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
> >> 05 0006fca8 5a3af42d w3dt!UlAtqStartListen+0x2d
> >> 06 0006fcb8 5a3bc335 w3core!W3_SERVER::StartListen+0xbd
> >> 07 0006ff0c 0100187c w3core!UlW3Start+0x26e
> >> 08 0006ff44 01001a27 w3wp!wmain+0x22a
> >> 09 0006ffc0 77e6f23b w3wp!wmainCRTStartup+0x12f
> >> 0a 0006fff0 00000000 kernel32!BaseProcessStart+0x23
>
> >> Thread ID: 1
> >> System Thread ID: 1204
> >> Kernel Time: 0:0:0.0
> >> User Time: 0:0:0.0
> >> Thread Status: Thread is in a WAIT state.
> >> Thread Type: Other
> >> # ChildEBP RetAddr
> >> 00 006afea0 7c827cfb ntdll!KiFastSystemCallRet
> >> 01 006afea4 7c80e5bb ntdll!NtWaitForMultipleObjects+0xc
> >> 02 006aff48 7c80e4a2 ntdll!EtwpWaitForMultipleObjectsEx+0xf7
> >> 03 006affb8 77e64829 ntdll!EtwpEventPump+0x27f
> >> 04 006affec 00000000 kernel32!BaseThreadStart+0x34
>
> >> Thread ID: 2
> >> System Thread ID: 1618
> >> Kernel Time: 0:0:0.0
> >> User Time: 0:0:0.0
> >> Thread Type: Other
> >> # ChildEBP RetAddr
> >> 00 00d3ff9c 7c826f4b ntdll!KiFastSystemCallRet
> >> 01 00d3ffa0 7c83d424 ntdll!NtDelayExecution+0xc
> >> 02 00d3ffb8 77e64829 ntdll!RtlpTimerThread+0x47
> >> 03 00d3ffec 00000000 kernel32!BaseThreadStart+0x34
>
> >> Thread ID: 3
> >> System Thread ID: 11a0
> >> Kernel Time: 0:0:0.0
> >> User Time: 0:0:0.0
> >> Thread Type: Other
> >> # ChildEBP RetAddr
> >> 00 00d7ff70 7c8277db ntdll!KiFastSystemCallRet
> >> 01 00d7ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
> >> 02 00d7ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
> >> 03 00d7ffec 00000000 kernel32!BaseThreadStart+0x34
>
> >> Thread ID: 4
> >> System Thread ID: 374
> >> Kernel Time: 0:0:0.0
> >> User Time: 0:0:0.0
> >> Thread Type: Possible ASP page. Possible DCOM activity
> >> Executing Page: Unable to locate ASP page
>
> >> No remote call being made
>
> >> # ChildEBP RetAddr
> >> 00 00dffe18 7c82783b ntdll!KiFastSystemCallRet
> >> 01 00dffe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
> >> 02 00dfff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
> >> 03 00dfff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
> >> 04 00dfffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
> >> 05 00dfffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
> >> 06 00dfffec 00000000 kernel32!BaseThreadStart+0x34
>
> >> Thread ID: 5
> >> System Thread ID: 8b0
> >> Kernel Time: 0:0:0.0
> >> User Time: 0:0:0.0
> >> Thread Type: Other
> >> # ChildEBP RetAddr
> >> 00 00e3ff08 7c826f4b ntdll!KiFastSystemCallRet
> >> 01 00e3ff0c 77e41ed1 ntdll!NtDelayExecution+0xc
> >> 02 00e3ff74 77e424ed kernel32!SleepEx+0x68
> >> 03 00e3ff84 776bbb0f kernel32!Sleep+0xf
> >> 04 00e3ff90 776bbab4 ole32!CROIDTable::WorkerThreadLoop+0x14
> >> 05 00e3ffac 776b1704 ole32!CRpcThread::WorkerLoop+0x26
> >> 06 00e3ffb8 77e64829 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x20
> >> 07 00e3ffec 00000000 kernel32!BaseThreadStart+0x34
>
> >> Thread ID: 6
> >> System Thread ID: ea4
> >> Kernel Time: 0:0:0.0
> >> User Time: 0:0:0.0
> >> Thread Type: HTTP Listener
> >> # ChildEBP RetAddr
> >> 00 00ffff24 7c8277db ntdll!KiFastSystemCallRet
> >> 01 00ffff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> >> 02 00ffff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
> >> 03 00ffff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
> >> 04 00ffffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
> >> 05 00ffffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
> >> 06 00ffffec 00000000 kernel32!BaseThreadStart+0x34
>
> >> Thread ID: 7
> >> System Thread ID: eac
> >> Kernel Time: 0:0:0.0
> >> User Time: 0:0:0.0
> >> Thread Type: HTTP Listener
> >> # ChildEBP RetAddr
> >> 00 0104ff24 7c8277db ntdll!KiFastSystemCallRet
> >> 01 0104ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> >> 02 0104ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
> >> 03 0104ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
> >> 04 0104ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
> >> 05 0104ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
> >> 06 0104ffec 00000000 kernel32!BaseThreadStart+0x34
>
> >> Thread ID: 8
> >> System Thread ID: 1750
> >> Kernel Time: 0:0:0.0
> >> User Time: 0:0:0.0
> >> Thread Type: HTTP Listener
> >> # ChildEBP RetAddr
> >> 00 0108ff24 7c8277db ntdll!KiFastSystemCallRet
> >> 01 0108ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> >> 02 0108ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
> >> 03 0108ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
> >> 04 0108ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
> >> 05 0108ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
> >> 06 0108ffec 00000000 kernel32!BaseThreadStart+0x34
>
> >> Thread ID: 9
> >> System Thread ID: ec8
> >> Kernel Time: 0:0:0.15
> >> User Time: 0:0:0.0
> >> Thread Type: HTTP Listener
> >> # ChildEBP RetAddr
> >> 00 010cff24 7c8277db ntdll!KiFastSystemCallRet
> >> 01 010cff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> >> 02 010cff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
> >> 03 010cff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
> >> 04 010cffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
> >> 05 010cffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
> >> 06 010cffec 00000000 kernel32!BaseThreadStart+0x34
>
> >> Thread ID: 10
> >> System Thread ID: cb8
> >> Kernel Time: 0:0:0.0
> >> User Time: 0:0:0.0
> >> Thread Status: Thread is in a WAIT state.
> >> Thread Type: Other
> >> # ChildEBP RetAddr
> >> 00 0118fcec 7c827cfb ntdll!KiFastSystemCallRet
> >> 01 0118fcf0 7c83c78e ntdll!NtWaitForMultipleObjects+0xc
> >> 02 0118ffb8 77e64829 ntdll!RtlpWaitThread+0x161
> >> 03 0118ffec 00000000 kernel32!BaseThreadStart+0x34
>
> >> Thread ID: 11
> >> System Thread ID: 904
> >> Kernel Time: 0:0:0.0
> >> User Time: 0:0:0.0
> >> *** WARNING: Unable to verify checksum for
> >> C:\WINDOWS\system32\inetsrv\IISGuardFilter.dll
> >> *** ERROR: Symbol file could not be found. Defaulted to export symbols for
> >> C:\WINDOWS\system32\inetsrv\IISGuardFilter.dll -
> >> Thread Type: Other
> >> # ChildEBP RetAddr
> >> 00 0123ff08 7739c811 ntdll!KiFastSystemCallRet
> >> 01 0123ff28 10005ee0 USER32!NtUserGetMessage+0xc
> >> WARNING: Stack unwind information not available. Following frames may be
> >> wrong.
> >> 02 7739be68 645653ec IISGuardFilter!HttpFilterProc+0x4b10
> >> 03 8b55ff8b 00000000 0x645653ec
>
> >> Thread ID: 12
> >> System Thread ID: 37c
> >> Kernel Time: 0:0:0.0
> >> User Time: 0:0:0.0
> >> Thread Status: Thread is in a
>
> ...
>
> read more >>- Hide quoted text -
>
> - Show quoted text -