How the heck can you effectively debug these "process serving app

Hello there, in an attempt to retain the few dozen hairs remaining on
the top of my head before I yank them all out in angst, I'm completely
stuck....HEEEELP!

We are always getting these errors in our System event log:

A process serving application pool 'poolName' suffered a fatal
communication error with the World Wide Web Publishing Service. The
process id was '1234'.

We have IIS6 on Windows Server 2003. These errors are all coming from
our intranet website hosted on this server. The intranet website runs
on .NET 2.0. We also have many other websites hosted on the server,
most .NET, but some classic ASP, but this particular error only is
reported for the intranet website. All other ASP.NET applications are
hosted within their own application pools.

The error happens A LOT (like, every few minutes). It's a huge issue
because it seems to cause large file uploads through our Intranet to
crap out when the issue happens (perhaps due to application pool
recycling?).

So to start a debugging session I ran:
cscript \windows\system32\iisapp.vbs

Which lists all app pools and processes. In this case at the time I
ran iisapp.vbs, the application pool named "poolName" is hosted by
process 9900. So I ran this:

iisstate -p 9900 -hc -d

In an attempt to log the next hard crash. When the hard crash happens,
I just can't figure out how to make effective use of the log
information. I've looked at several logs but none seem to make me go
"AHA!". I understand that Microsoft recommends running this tool, but
I don't think that I can interpret any of it...please, MSFT or anyone,
your help is greatly appreciated to interpret the crash log and
provide recommendations!!! What follows is the log, through to the end
of this post....Thx!!!

----------------------------------------

Opened log file 'E:\Program Files\IIS Resources\IISState\output
\IISState.log'

***********************
Starting new log output
IISState version 3.0

Thu Nov 15 14:46:28 2007

OS = Windows 2003 Server
Executable: w3wp.exe
PID = 6976

Note: Thread times are formatted as HH:MM:SS.ms

***********************


IIS has crashed...
Beginning Analysis
DLL (!FunctionName) that failed:




Thread ID: 30
System Thread ID: 1a90
Kernel Time: 0:0:0.375
User Time: 0:0:2.140
Thread Type: Other
# ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be
wrong.
00 1bf8f118 1db03f0e 0x1db03f5f
01 00000000 00000000 0x1db03f0e
Closing open log file E:\Program Files\IIS Resources\IISState\output
\IISState.log
Opened log file 'E:\Program Files\IIS Resources\IISState\output
\IISState.log'

***********************
Starting new log output
IISState version 3.0

Thu Nov 15 15:59:00 2007

OS = Windows 2003 Server
Executable: w3wp.exe
PID = 9900

Note: Thread times are formatted as HH:MM:SS.ms

***********************


IIS has crashed...
Beginning Analysis
DLL (!FunctionName) that failed: kernel32!RaiseException




Thread ID: 27
System Thread ID: 1ea0
Kernel Time: 0:0:0.468
User Time: 0:0:4.359
Thread Type: Other
# ChildEBP RetAddr
00 1acaf898 79f55b05 kernel32!RaiseException+0x53
01 1acaf8f8 7a056a79 mscorwks!RaiseTheExceptionInternalOnly+0x226
02 1acaf90c 7a056af2 mscorwks!RaiseTheException+0x4d
03 1acaf934 7a056b30 mscorwks!RaiseTheException+0xbf
04 1acaf960 7a056b41 mscorwks!RealCOMPlusThrow+0x38
05 1acaf96c 7a0e38fa mscorwks!RealCOMPlusThrow+0xb
06 1acafa9c 7a0e4b97 mscorwks!Thread::RaiseCrossContextException+0x3ac
07 1acafb50 7a0e5a07 mscorwks!Thread::DoADCallBack+0x26c
08 1acafb68 79ed89ca mscorwks!Thread::UserResumeThread+0xe1
09 1acafbfc 79ed88f1 mscorwks!Thread::DoADCallBack+0x355
0a 1acafc38 7a0e4d96 mscorwks!Thread::DoADCallBack+0x541
0b 1acafc60 7a0e4dc3 mscorwks!Thread::DoADCallBack+0x575
0c 1acafc74 7a077528 mscorwks!ManagedThreadBase::ThreadPool+0x13
0d 1acafcbc 7a077556 mscorwks!AddTimerCallbackEx+0x77
0e 1acafccc 7a1158ae mscorwks!AddTimerCallback+0xf
0f 1acafd08 79f8b7a5 mscorwks!
ThreadpoolMgr::AsyncTimerCallbackCompletion+0x64
10 1acafd20 79f8b341 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0x40
11 1acafd94 79ed8e36 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x225
12 1acaffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
13 1acaffec 00000000 kernel32!BaseThreadStart+0x34
Closing open log file E:\Program Files\IIS Resources\IISState\output
\IISState.log
Opened log file 'E:\Program Files\IIS Resources\IISState\output
\IISState.log'

***********************
Starting new log output
IISState version 3.0

Thu Nov 15 15:59:01 2007

OS = Windows 2003 Server
Executable: w3wp.exe
PID = 9900

Note: Thread times are formatted as HH:MM:SS.ms

***********************




Thread ID: 0
System Thread ID: 26d4
Kernel Time: 0:0:0.78
User Time: 0:0:0.171
Thread Status: Thread is in a WAIT state.
Thread Type: HTTP Compression Thread
# ChildEBP RetAddr
00 0006fc08 7c827d0b ntdll!KiFastSystemCallRet
01 0006fc0c 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 0006fc7c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 0006fc90 5a364662 kernel32!WaitForSingleObject+0x12
04 0006fca0 5a366e3f w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
05 0006fca8 5a3af42d w3dt!UlAtqStartListen+0x2d
06 0006fcb8 5a3bc335 w3core!W3_SERVER::StartListen+0xbd
07 0006ff0c 0100187c w3core!UlW3Start+0x26e
08 0006ff44 01001a27 w3wp!wmain+0x22a
09 0006ffc0 77e6f23b w3wp!wmainCRTStartup+0x12f
0a 0006fff0 00000000 kernel32!BaseProcessStart+0x23




Thread ID: 1
System Thread ID: 1290
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 008bfea0 7c827cfb ntdll!KiFastSystemCallRet
01 008bfea4 7c80e5bb ntdll!NtWaitForMultipleObjects+0xc
02 008bff48 7c80e4a2 ntdll!EtwpWaitForMultipleObjectsEx+0xf7
03 008bffb8 77e64829 ntdll!EtwpEventPump+0x27f
04 008bffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 2
System Thread ID: 234c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 00f4ff9c 7c826f4b ntdll!KiFastSystemCallRet
01 00f4ffa0 7c83d424 ntdll!NtDelayExecution+0xc
02 00f4ffb8 77e64829 ntdll!RtlpTimerThread+0x47
03 00f4ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 3
System Thread ID: c6c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 00f8ff70 7c8277db ntdll!KiFastSystemCallRet
01 00f8ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
02 00f8ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
03 00f8ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 4
System Thread ID: 27ec
Kernel Time: 0:0:0.15
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.

No remote call being made

# ChildEBP RetAddr
00 00fffe18 7c82783b ntdll!KiFastSystemCallRet
01 00fffe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
02 00ffff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
03 00ffff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
04 00ffffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
05 00ffffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
06 00ffffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 5
System Thread ID: 5f8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 1e47ff70 7c8277db ntdll!KiFastSystemCallRet
01 1e47ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
02 1e47ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
03 1e47ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 6
System Thread ID: 1070
Kernel Time: 0:0:0.93
User Time: 0:0:0.265
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 0120ff24 7c8277db ntdll!KiFastSystemCallRet
01 0120ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
02 0120ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
03 0120ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
04 0120ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
05 0120ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
06 0120ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 7
System Thread ID: 2734
Kernel Time: 0:0:0.125
User Time: 0:0:0.171
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 0124ff24 7c8277db ntdll!KiFastSystemCallRet
01 0124ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
02 0124ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
03 0124ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
04 0124ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
05 0124ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
06 0124ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 8
System Thread ID: bb4
Kernel Time: 0:0:0.140
User Time: 0:0:0.593
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 0128ff24 7c8277db ntdll!KiFastSystemCallRet
01 0128ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
02 0128ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
03 0128ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
04 0128ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
05 0128ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
06 0128ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 9
System Thread ID: fc0
Kernel Time: 0:0:0.140
User Time: 0:0:0.359
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 012cff24 7c8277db ntdll!KiFastSystemCallRet
01 012cff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
02 012cff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
03 012cff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
04 012cffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
05 012cffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
06 012cffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 10
System Thread ID: 1ca4
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 0138fcec 7c827cfb ntdll!KiFastSystemCallRet
01 0138fcf0 7c83c78e ntdll!NtWaitForMultipleObjects+0xc
02 0138ffb8 77e64829 ntdll!RtlpWaitThread+0x161
03 0138ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 11
System Thread ID: 2774
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: HTTP Compression Thread
# ChildEBP RetAddr
00 01d5fa84 7c827d0b ntdll!KiFastSystemCallRet
01 01d5fa88 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 01d5faf8 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 01d5fb0c 5a3b820f kernel32!WaitForSingleObject+0x12
04 01d5ffb8 77e64829 w3core!HTTP_COMPRESSION::CompressionThread+0x126
05 01d5ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 12
System Thread ID: 17dc
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.

No remote call being made

# ChildEBP RetAddr
00 01d9fe18 7c82783b ntdll!KiFastSystemCallRet
01 01d9fe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
02 01d9ff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
03 01d9ff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
04 01d9ffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
05 01d9ffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
06 01d9ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 13
System Thread ID: 27dc
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 01ddff8c 7c826f4b ntdll!KiFastSystemCallRet
01 01ddff90 7c81943a ntdll!NtDelayExecution+0xc
02 01ddffb8 77e64829 ntdll!RtlpIOWorkerThread+0x3f
03 01ddffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 14
System Thread ID: 26e8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 0211fe78 7c8277db ntdll!KiFastSystemCallRet
01 0211fe7c 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
02 0211fea8 79f798a6 kernel32!GetQueuedCompletionStatus+0x29
03 0211ff14 79f79c4f mscorwks!ThreadpoolMgr::CompletionPortThreadStart
+0x11a
04 0211ffb8 77e64829 mscorwks!ThreadpoolMgr::intermediateThreadProc
+0x49
05 0211ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 15
System Thread ID: 1c10
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 0215fdb4 7c826f4b ntdll!KiFastSystemCallRet
01 0215fdb8 77e41ed1 ntdll!NtDelayExecution+0xc
02 0215fe20 79f56e05 kernel32!SleepEx+0x68
03 0215fe54 79f56dd5 mscorwks!EESleepEx+0xa3
04 0215fe88 79f409c5 mscorwks!__DangerousSwitchToThread+0x70
05 0215fe94 79f56ca7 mscorwks!__SwitchToThread+0xb
06 0215ffb8 77e64829 mscorwks!ThreadpoolMgr::GateThreadStart+0xa1
07 0215ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 16
System Thread ID: c68
Kernel Time: 0:0:0.750
User Time: 0:0:5.281
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 0219fc80 7c827d0b ntdll!KiFastSystemCallRet
01 0219fc84 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 0219fcf4 79e718fd kernel32!WaitForSingleObjectEx+0xac
03 0219fd38 79e718c6 mscorwks!PEImage::LoadImage+0x199
04 0219fd88 79e7187c mscorwks!CLREvent::WaitEx+0x117
05 0219fd98 79f5e7db mscorwks!CLREvent::Wait+0x17
06 0219fe18 79f8b495 mscorwks!ThreadpoolMgr::SafeWait+0x73
07 0219fe94 79f79c4f mscorwks!ThreadpoolMgr::WorkerThreadStart+0xf5
08 0219ffb8 77e64829 mscorwks!ThreadpoolMgr::intermediateThreadProc
+0x49
09 0219ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 17
System Thread ID: cac
Kernel Time: 0:0:0.906
User Time: 0:0:6.953
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 021ffbe4 7c827d0b ntdll!KiFastSystemCallRet
01 021ffbe8 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 021ffc58 79e718fd kernel32!WaitForSingleObjectEx+0xac
03 021ffc9c 79e718c6 mscorwks!PEImage::LoadImage+0x199
04 021ffcec 79e7187c mscorwks!CLREvent::WaitEx+0x117
05 021ffcfc 79f5e7db mscorwks!CLREvent::Wait+0x17
06 021ffd7c 7a1153d9 mscorwks!ThreadpoolMgr::SafeWait+0x73
07 021ffda4 79f8b4e0 mscorwks!ThreadpoolMgr::EnterRetirement+0x8e
08 021ffe14 79f79c4f mscorwks!ThreadpoolMgr::WorkerThreadStart+0x393
09 021fffb8 77e64829 mscorwks!ThreadpoolMgr::intermediateThreadProc
+0x49
0a 021fffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 18
System Thread ID: 1770
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 022efe30 7c827cfb ntdll!KiFastSystemCallRet
01 022efe34 77e6202c ntdll!NtWaitForMultipleObjects+0xc
02 022efedc 77e62fbe kernel32!WaitForMultipleObjectsEx+0x11a
03 022efef8 79f0cf96 kernel32!WaitForMultipleObjects+0x18
04 022eff58 79f0cef3 mscorwks!DebuggerRCThread::MainLoop+0xcf
05 022eff88 79f0ce36 mscorwks!DebuggerRCThread::ThreadProc+0xca
06 022effb8 77e64829 mscorwks!DebuggerRCThread::ThreadProcStatic+0x82
07 022effec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 19
System Thread ID: 914
Kernel Time: 0:0:0.15
User Time: 0:0:0.578
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 0275fe68 7c827d0b ntdll!KiFastSystemCallRet
01 0275fe6c 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 0275fedc 79e718fd kernel32!WaitForSingleObjectEx+0xac
03 0275ff20 79e718c6 mscorwks!PEImage::LoadImage+0x199
04 0275ff70 79e7187c mscorwks!CLREvent::WaitEx+0x117
05 0275ff80 7a0d866f mscorwks!CLREvent::Wait+0x17
06 0275ffa8 7a0d8987 mscorwks!SVR::gc_heap::gc_thread_function+0x2f
07 0275ffb8 77e64829 mscorwks!SVR::gc_heap::gc_thread_stub+0x9b
08 0275ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 20
System Thread ID: 238c
Kernel Time: 0:0:0.0
User Time: 0:0:0.609
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 0279fd68 7c827d0b ntdll!KiFastSystemCallRet
01 0279fd6c 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 0279fddc 79e718fd kernel32!WaitForSingleObjectEx+0xac
03 0279fe20 79e718c6 mscorwks!PEImage::LoadImage+0x199
04 0279fe70 79e7187c mscorwks!CLREvent::WaitEx+0x117
05 0279fe80 7a0d8898 mscorwks!CLREvent::Wait+0x17
06 0279fea8 7a0d8987 mscorwks!SVR::gc_heap::gc_thread_function+0x58
07 0279ffb8 77e64829 mscorwks!SVR::gc_heap::gc_thread_stub+0x9b
08 0279ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 21
System Thread ID: e20
Kernel Time: 0:0:0.15
User Time: 0:0:0.578
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 027dfc68 7c827d0b ntdll!KiFastSystemCallRet
01 027dfc6c 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 027dfcdc 79e718fd kernel32!WaitForSingleObjectEx+0xac
03 027dfd20 79e718c6 mscorwks!PEImage::LoadImage+0x199
04 027dfd70 79e7187c mscorwks!CLREvent::WaitEx+0x117
05 027dfd80 7a0d8898 mscorwks!CLREvent::Wait+0x17
06 027dfda8 7a0d8987 mscorwks!SVR::gc_heap::gc_thread_function+0x58
07 027dffb8 77e64829 mscorwks!SVR::gc_heap::gc_thread_stub+0x9b
08 027dffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 22
System Thread ID: 144c
Kernel Time: 0:0:0.15
User Time: 0:0:0.578
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 0281fb68 7c827d0b ntdll!KiFastSystemCallRet
01 0281fb6c 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 0281fbdc 79e718fd kernel32!WaitForSingleObjectEx+0xac
03 0281fc20 79e718c6 mscorwks!PEImage::LoadImage+0x199
04 0281fc70 79e7187c mscorwks!CLREvent::WaitEx+0x117
05 0281fc80 7a0d8898 mscorwks!CLREvent::Wait+0x17
06 0281fca8 7a0d8987 mscorwks!SVR::gc_heap::gc_thread_function+0x58
07 0281ffb8 77e64829 mscorwks!SVR::gc_heap::gc_thread_stub+0x9b
08 0281ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 23
System Thread ID: 18f8
Kernel Time: 0:0:0.31
User Time: 0:0:0.31
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 0285fccc 7c827cfb ntdll!KiFastSystemCallRet
01 0285fcd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
02 0285fd78 77e62fbe kernel32!WaitForMultipleObjectsEx+0x11a
03 0285fd94 7a085b65 kernel32!WaitForMultipleObjects+0x18
04 0285fdb4 7a08648f mscorwks!SVR::WaitForFinalizerEvent+0x7a
05 0285fdc8 79ed8a2c mscorwks!SVR::GCHeap::FinalizerThreadWorker+0x75
06 0285fdd8 79ed89ca mscorwks!Thread::UserResumeThread+0xfb
07 0285fe6c 79ed88f1 mscorwks!Thread::DoADCallBack+0x355
08 0285fea8 79ed988c mscorwks!Thread::DoADCallBack+0x541
09 0285fed0 79ed9857 mscorwks!ManagedThreadBase_NoADTransition+0x32
0a 0285fedc 7a086638 mscorwks!ManagedThreadBase::FinalizerBase+0xb
0b 0285ff14 79ed8e36 mscorwks!SVR::GCHeap::FinalizerThreadStart+0xbb
0c 0285ffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
0d 0285ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 24
System Thread ID: 1a88
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 0289fcb8 7c827d0b ntdll!KiFastSystemCallRet
01 0289fcbc 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 0289fd2c 79e718fd kernel32!WaitForSingleObjectEx+0xac
03 0289fd70 79e718c6 mscorwks!PEImage::LoadImage+0x199
04 0289fdc0 79e7187c mscorwks!CLREvent::WaitEx+0x117
05 0289fdd0 79f79049 mscorwks!CLREvent::Wait+0x17
06 0289fe94 79ed8e36 mscorwks!AppDomain::ADUnloadThreadStart+0x2fb
07 0289ffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
08 0289ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 25
System Thread ID: 227c
Kernel Time: 0:0:0.15
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 0294fec8 7c826f4b ntdll!KiFastSystemCallRet
01 0294fecc 77e41ed1 ntdll!NtDelayExecution+0xc
02 0294ff34 79f66a87 kernel32!SleepEx+0x68
03 0294ffb0 79f66a18 mscorwks!ThreadpoolMgr::TimerThreadFire+0x6d
04 0294ffb8 77e64829 mscorwks!ThreadpoolMgr::TimerThreadStart+0x54
05 0294ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 26
System Thread ID: 2454
Kernel Time: 0:0:1.0
User Time: 0:0:8.625
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 1ac6fc00 7c827d0b ntdll!KiFastSystemCallRet
01 1ac6fc04 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 1ac6fc74 79e718fd kernel32!WaitForSingleObjectEx+0xac
03 1ac6fcb8 79e718c6 mscorwks!PEImage::LoadImage+0x199
04 1ac6fd08 79e7187c mscorwks!CLREvent::WaitEx+0x117
05 1ac6fd18 79f5e7db mscorwks!CLREvent::Wait+0x17
06 1ac6fd98 79f8b495 mscorwks!ThreadpoolMgr::SafeWait+0x73
07 1ac6fe14 79ed8e36 mscorwks!ThreadpoolMgr::WorkerThreadStart+0xf5
08 1ac6ffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
09 1ac6ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 27
System Thread ID: 1ea0
Kernel Time: 0:0:0.468
User Time: 0:0:4.359
Thread Type: Other
# ChildEBP RetAddr
00 1acaf898 79f55b05 kernel32!RaiseException+0x53
01 1acaf8f8 7a056a79 mscorwks!RaiseTheExceptionInternalOnly+0x226
02 1acaf90c 7a056af2 mscorwks!RaiseTheException+0x4d
03 1acaf934 7a056b30 mscorwks!RaiseTheException+0xbf
04 1acaf960 7a056b41 mscorwks!RealCOMPlusThrow+0x38
05 1acaf96c 7a0e38fa mscorwks!RealCOMPlusThrow+0xb
06 1acafa9c 7a0e4b97 mscorwks!Thread::RaiseCrossContextException+0x3ac
07 1acafb50 7a0e5a07 mscorwks!Thread::DoADCallBack+0x26c
08 1acafb68 79ed89ca mscorwks!Thread::UserResumeThread+0xe1
09 1acafbfc 79ed88f1 mscorwks!Thread::DoADCallBack+0x355
0a 1acafc38 7a0e4d96 mscorwks!Thread::DoADCallBack+0x541
0b 1acafc60 7a0e4dc3 mscorwks!Thread::DoADCallBack+0x575
0c 1acafc74 7a077528 mscorwks!ManagedThreadBase::ThreadPool+0x13
0d 1acafcbc 7a077556 mscorwks!AddTimerCallbackEx+0x77
0e 1acafccc 7a1158ae mscorwks!AddTimerCallback+0xf
0f 1acafd08 79f8b7a5 mscorwks!
ThreadpoolMgr::AsyncTimerCallbackCompletion+0x64
10 1acafd20 79f8b341 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0x40
11 1acafd94 79ed8e36 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x225
12 1acaffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
13 1acaffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 28
System Thread ID: 878
Kernel Time: 0:0:0.359
User Time: 0:0:1.328
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 1acefae4 7c827d0b ntdll!KiFastSystemCallRet
01 1acefae8 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 1acefb58 79e718fd kernel32!WaitForSingleObjectEx+0xac
03 1acefb9c 79e718c6 mscorwks!PEImage::LoadImage+0x199
04 1acefbec 79e7187c mscorwks!CLREvent::WaitEx+0x117
05 1acefbfc 79f5e7db mscorwks!CLREvent::Wait+0x17
06 1acefc7c 7a1153d9 mscorwks!ThreadpoolMgr::SafeWait+0x73
07 1acefca4 79f8b4e0 mscorwks!ThreadpoolMgr::EnterRetirement+0x8e
08 1acefd14 79ed8e36 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x393
09 1aceffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
0a 1aceffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 29
System Thread ID: 1824
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 1b5cff54 7c8277db ntdll!KiFastSystemCallRet
01 1b5cff58 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
02 1b5cff84 65001897 kernel32!GetQueuedCompletionStatus+0x29
03 1b5cffb8 77e64829 System_Data!SNIAsyncWait+0x6c
04 1b5cffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 30
System Thread ID: 9b4
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 1be8fec0 7c827cfb ntdll!KiFastSystemCallRet
01 1be8fec4 77e6202c ntdll!NtWaitForMultipleObjects+0xc
02 1be8ff6c 77e62fbe kernel32!WaitForMultipleObjectsEx+0x11a
03 1be8ff88 76929e35 kernel32!WaitForMultipleObjects+0x18
04 1be8ffb8 77e64829 USERENV!NotificationThread+0x5f
05 1be8ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 31
System Thread ID: 15c4
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
*** WARNING: Unable to verify checksum for E:\WINDOWS\assembly
\NativeImages_v2.0.50727_32\mscorlib
\16d71085704686243fafcd1e8c89bda2\mscorlib.ni.dll
Thread Type: Other
# ChildEBP RetAddr
00 1c1ff34c 7c826f4b ntdll!KiFastSystemCallRet
01 1c1ff350 77e41ed1 ntdll!NtDelayExecution+0xc
02 1c1ff3b8 79f56e05 kernel32!SleepEx+0x68
03 1c1ff3ec 7a04cd47 mscorwks!EESleepEx+0xa3
04 1c1ff3f8 7a12f1c1 mscorwks!CExecutionEngine::ClrSleepEx+0xd
05 1c1ff408 7a0e44c1 mscorwks!ClrSleepEx+0x13
06 1c1ff444 7a079218 mscorwks!Thread::UserSleep+0x63
07 1c1ff4e4 1afb1b15 mscorwks!ThreadNative::Sleep+0xce
WARNING: Frame IP not in any known module. Following frames may be
wrong.
08 1c1ff50c 793d70fb 0x1afb1b15
09 00000000 00000000 mscorlib_ni+0x3170fb




Thread ID: 32
System Thread ID: 1f68
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 1c29f2cc 7c826f4b ntdll!KiFastSystemCallRet
01 1c29f2d0 77e41ed1 ntdll!NtDelayExecution+0xc
02 1c29f338 79f56e05 kernel32!SleepEx+0x68
03 1c29f36c 7a04cd47 mscorwks!EESleepEx+0xa3
04 1c29f378 7a12f1c1 mscorwks!CExecutionEngine::ClrSleepEx+0xd
05 1c29f388 7a0e44c1 mscorwks!ClrSleepEx+0x13
06 1c29f3c4 7a079218 mscorwks!Thread::UserSleep+0x63
07 1c29f464 1afb1b15 mscorwks!ThreadNative::Sleep+0xce
WARNING: Frame IP not in any known module. Following frames may be
wrong.
08 1c29f48c 793d70fb 0x1afb1b15
09 00000000 00000000 mscorlib_ni+0x3170fb




Thread ID: 33
System Thread ID: ef0
Kernel Time: 0:0:0.15
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 1d5bfe48 7c827cfb ntdll!KiFastSystemCallRet
01 1d5bfe4c 77e6202c ntdll!NtWaitForMultipleObjects+0xc
02 1d5bfef4 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
03 1d5bff50 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
04 1d5bff6c 4dd668ab USER32!MsgWaitForMultipleObjects+0x1f
05 1d5bffb8 77e64829 gdiplus!BackgroundThreadProc+0x59
06 1d5bffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 34
System Thread ID: 9d8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 1e63ff70 7c8277db ntdll!KiFastSystemCallRet
01 1e63ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
02 1e63ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
03 1e63ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 35
System Thread ID: 1f18
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 1e67ff70 7c8277db ntdll!KiFastSystemCallRet
01 1e67ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
02 1e67ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
03 1e67ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 36
System Thread ID: 1720
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 1e6bff70 7c8277db ntdll!KiFastSystemCallRet
01 1e6bff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
02 1e6bffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
03 1e6bffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 37
System Thread ID: 1720
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 1e6bff70 7c8277db ntdll!KiFastSystemCallRet
01 1e6bff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
02 1e6bffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
03 1e6bffec 00000000 kernel32!BaseThreadStart+0x34

*****

Dump name is formatted as: PID-Timestamp.dmp

Creating E:\Program Files\IIS Resources\IISState\output
\9900-1195160343.dmp - user full dump
Dump successfully written

*****

Closing open log file E:\Program Files\IIS Resources\IISState\output
\IISState.log

Re: How the heck can you effectively debug these "process serving app pool suffered...blah blah

Try
Procmon utility , If you haven't tried yet.
http://www.microsoft.com/technet/sysinternals/ProcessesAndTh reads/Portmon.mspx

"Shan Plourde" wrote in message
news:ec423693-9008-414c-a8c7-02d9b6ef1cde@d50g2000hsf.google groups.com...
> Hello there, in an attempt to retain the few dozen hairs remaining on
> the top of my head before I yank them all out in angst, I'm completely
> stuck....HEEEELP!
>
> We are always getting these errors in our System event log:
>
> A process serving application pool 'poolName' suffered a fatal
> communication error with the World Wide Web Publishing Service. The
> process id was '1234'.
>
> We have IIS6 on Windows Server 2003. These errors are all coming from
> our intranet website hosted on this server. The intranet website runs
> on .NET 2.0. We also have many other websites hosted on the server,
> most .NET, but some classic ASP, but this particular error only is
> reported for the intranet website. All other ASP.NET applications are
> hosted within their own application pools.
>
> The error happens A LOT (like, every few minutes). It's a huge issue
> because it seems to cause large file uploads through our Intranet to
> crap out when the issue happens (perhaps due to application pool
> recycling?).
>
> So to start a debugging session I ran:
> cscript \windows\system32\iisapp.vbs
>
> Which lists all app pools and processes. In this case at the time I
> ran iisapp.vbs, the application pool named "poolName" is hosted by
> process 9900. So I ran this:
>
> iisstate -p 9900 -hc -d
>
> In an attempt to log the next hard crash. When the hard crash happens,
> I just can't figure out how to make effective use of the log
> information. I've looked at several logs but none seem to make me go
> "AHA!". I understand that Microsoft recommends running this tool, but
> I don't think that I can interpret any of it...please, MSFT or anyone,
> your help is greatly appreciated to interpret the crash log and
> provide recommendations!!! What follows is the log, through to the end
> of this post....Thx!!!
>
> ----------------------------------------
>
> Opened log file 'E:\Program Files\IIS Resources\IISState\output
> \IISState.log'
>
> ***********************
> Starting new log output
> IISState version 3.0
>
> Thu Nov 15 14:46:28 2007
>
> OS = Windows 2003 Server
> Executable: w3wp.exe
> PID = 6976
>
> Note: Thread times are formatted as HH:MM:SS.ms
>
> ***********************
>
>
> IIS has crashed...
> Beginning Analysis
> DLL (!FunctionName) that failed:
>
>
>
>
> Thread ID: 30
> System Thread ID: 1a90
> Kernel Time: 0:0:0.375
> User Time: 0:0:2.140
> Thread Type: Other
> # ChildEBP RetAddr
> WARNING: Frame IP not in any known module. Following frames may be
> wrong.
> 00 1bf8f118 1db03f0e 0x1db03f5f
> 01 00000000 00000000 0x1db03f0e
> Closing open log file E:\Program Files\IIS Resources\IISState\output
> \IISState.log
> Opened log file 'E:\Program Files\IIS Resources\IISState\output
> \IISState.log'
>
> ***********************
> Starting new log output
> IISState version 3.0
>
> Thu Nov 15 15:59:00 2007
>
> OS = Windows 2003 Server
> Executable: w3wp.exe
> PID = 9900
>
> Note: Thread times are formatted as HH:MM:SS.ms
>
> ***********************
>
>
> IIS has crashed...
> Beginning Analysis
> DLL (!FunctionName) that failed: kernel32!RaiseException
>
>
>
>
> Thread ID: 27
> System Thread ID: 1ea0
> Kernel Time: 0:0:0.468
> User Time: 0:0:4.359
> Thread Type: Other
> # ChildEBP RetAddr
> 00 1acaf898 79f55b05 kernel32!RaiseException+0x53
> 01 1acaf8f8 7a056a79 mscorwks!RaiseTheExceptionInternalOnly+0x226
> 02 1acaf90c 7a056af2 mscorwks!RaiseTheException+0x4d
> 03 1acaf934 7a056b30 mscorwks!RaiseTheException+0xbf
> 04 1acaf960 7a056b41 mscorwks!RealCOMPlusThrow+0x38
> 05 1acaf96c 7a0e38fa mscorwks!RealCOMPlusThrow+0xb
> 06 1acafa9c 7a0e4b97 mscorwks!Thread::RaiseCrossContextException+0x3ac
> 07 1acafb50 7a0e5a07 mscorwks!Thread::DoADCallBack+0x26c
> 08 1acafb68 79ed89ca mscorwks!Thread::UserResumeThread+0xe1
> 09 1acafbfc 79ed88f1 mscorwks!Thread::DoADCallBack+0x355
> 0a 1acafc38 7a0e4d96 mscorwks!Thread::DoADCallBack+0x541
> 0b 1acafc60 7a0e4dc3 mscorwks!Thread::DoADCallBack+0x575
> 0c 1acafc74 7a077528 mscorwks!ManagedThreadBase::ThreadPool+0x13
> 0d 1acafcbc 7a077556 mscorwks!AddTimerCallbackEx+0x77
> 0e 1acafccc 7a1158ae mscorwks!AddTimerCallback+0xf
> 0f 1acafd08 79f8b7a5 mscorwks!
> ThreadpoolMgr::AsyncTimerCallbackCompletion+0x64
> 10 1acafd20 79f8b341 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0x40
> 11 1acafd94 79ed8e36 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x225
> 12 1acaffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
> 13 1acaffec 00000000 kernel32!BaseThreadStart+0x34
> Closing open log file E:\Program Files\IIS Resources\IISState\output
> \IISState.log
> Opened log file 'E:\Program Files\IIS Resources\IISState\output
> \IISState.log'
>
> ***********************
> Starting new log output
> IISState version 3.0
>
> Thu Nov 15 15:59:01 2007
>
> OS = Windows 2003 Server
> Executable: w3wp.exe
> PID = 9900
>
> Note: Thread times are formatted as HH:MM:SS.ms
>
> ***********************
>
>
>
>
> Thread ID: 0
> System Thread ID: 26d4
> Kernel Time: 0:0:0.78
> User Time: 0:0:0.171
> Thread Status: Thread is in a WAIT state.
> Thread Type: HTTP Compression Thread
> # ChildEBP RetAddr
> 00 0006fc08 7c827d0b ntdll!KiFastSystemCallRet
> 01 0006fc0c 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 0006fc7c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> 03 0006fc90 5a364662 kernel32!WaitForSingleObject+0x12
> 04 0006fca0 5a366e3f w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
> 05 0006fca8 5a3af42d w3dt!UlAtqStartListen+0x2d
> 06 0006fcb8 5a3bc335 w3core!W3_SERVER::StartListen+0xbd
> 07 0006ff0c 0100187c w3core!UlW3Start+0x26e
> 08 0006ff44 01001a27 w3wp!wmain+0x22a
> 09 0006ffc0 77e6f23b w3wp!wmainCRTStartup+0x12f
> 0a 0006fff0 00000000 kernel32!BaseProcessStart+0x23
>
>
>
>
> Thread ID: 1
> System Thread ID: 1290
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 008bfea0 7c827cfb ntdll!KiFastSystemCallRet
> 01 008bfea4 7c80e5bb ntdll!NtWaitForMultipleObjects+0xc
> 02 008bff48 7c80e4a2 ntdll!EtwpWaitForMultipleObjectsEx+0xf7
> 03 008bffb8 77e64829 ntdll!EtwpEventPump+0x27f
> 04 008bffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 2
> System Thread ID: 234c
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Other
> # ChildEBP RetAddr
> 00 00f4ff9c 7c826f4b ntdll!KiFastSystemCallRet
> 01 00f4ffa0 7c83d424 ntdll!NtDelayExecution+0xc
> 02 00f4ffb8 77e64829 ntdll!RtlpTimerThread+0x47
> 03 00f4ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 3
> System Thread ID: c6c
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Other
> # ChildEBP RetAddr
> 00 00f8ff70 7c8277db ntdll!KiFastSystemCallRet
> 01 00f8ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
> 02 00f8ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
> 03 00f8ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 4
> System Thread ID: 27ec
> Kernel Time: 0:0:0.15
> User Time: 0:0:0.0
> Thread Type: Possible ASP page. Possible DCOM activity
> Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
> Continuing with other analysis.
>
> No remote call being made
>
> # ChildEBP RetAddr
> 00 00fffe18 7c82783b ntdll!KiFastSystemCallRet
> 01 00fffe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
> 02 00ffff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
> 03 00ffff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
> 04 00ffffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
> 05 00ffffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
> 06 00ffffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 5
> System Thread ID: 5f8
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Other
> # ChildEBP RetAddr
> 00 1e47ff70 7c8277db ntdll!KiFastSystemCallRet
> 01 1e47ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
> 02 1e47ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
> 03 1e47ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 6
> System Thread ID: 1070
> Kernel Time: 0:0:0.93
> User Time: 0:0:0.265
> Thread Type: HTTP Listener
> # ChildEBP RetAddr
> 00 0120ff24 7c8277db ntdll!KiFastSystemCallRet
> 01 0120ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> 02 0120ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
> 03 0120ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
> 04 0120ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
> 05 0120ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
> 06 0120ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 7
> System Thread ID: 2734
> Kernel Time: 0:0:0.125
> User Time: 0:0:0.171
> Thread Type: HTTP Listener
> # ChildEBP RetAddr
> 00 0124ff24 7c8277db ntdll!KiFastSystemCallRet
> 01 0124ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> 02 0124ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
> 03 0124ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
> 04 0124ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
> 05 0124ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
> 06 0124ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 8
> System Thread ID: bb4
> Kernel Time: 0:0:0.140
> User Time: 0:0:0.593
> Thread Type: HTTP Listener
> # ChildEBP RetAddr
> 00 0128ff24 7c8277db ntdll!KiFastSystemCallRet
> 01 0128ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> 02 0128ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
> 03 0128ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
> 04 0128ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
> 05 0128ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
> 06 0128ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 9
> System Thread ID: fc0
> Kernel Time: 0:0:0.140
> User Time: 0:0:0.359
> Thread Type: HTTP Listener
> # ChildEBP RetAddr
> 00 012cff24 7c8277db ntdll!KiFastSystemCallRet
> 01 012cff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> 02 012cff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
> 03 012cff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
> 04 012cffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
> 05 012cffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
> 06 012cffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 10
> System Thread ID: 1ca4
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 0138fcec 7c827cfb ntdll!KiFastSystemCallRet
> 01 0138fcf0 7c83c78e ntdll!NtWaitForMultipleObjects+0xc
> 02 0138ffb8 77e64829 ntdll!RtlpWaitThread+0x161
> 03 0138ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 11
> System Thread ID: 2774
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: HTTP Compression Thread
> # ChildEBP RetAddr
> 00 01d5fa84 7c827d0b ntdll!KiFastSystemCallRet
> 01 01d5fa88 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 01d5faf8 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> 03 01d5fb0c 5a3b820f kernel32!WaitForSingleObject+0x12
> 04 01d5ffb8 77e64829 w3core!HTTP_COMPRESSION::CompressionThread+0x126
> 05 01d5ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 12
> System Thread ID: 17dc
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Possible ASP page. Possible DCOM activity
> Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
> Continuing with other analysis.
>
> No remote call being made
>
> # ChildEBP RetAddr
> 00 01d9fe18 7c82783b ntdll!KiFastSystemCallRet
> 01 01d9fe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
> 02 01d9ff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
> 03 01d9ff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
> 04 01d9ffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
> 05 01d9ffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
> 06 01d9ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 13
> System Thread ID: 27dc
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Other
> # ChildEBP RetAddr
> 00 01ddff8c 7c826f4b ntdll!KiFastSystemCallRet
> 01 01ddff90 7c81943a ntdll!NtDelayExecution+0xc
> 02 01ddffb8 77e64829 ntdll!RtlpIOWorkerThread+0x3f
> 03 01ddffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 14
> System Thread ID: 26e8
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Other
> # ChildEBP RetAddr
> 00 0211fe78 7c8277db ntdll!KiFastSystemCallRet
> 01 0211fe7c 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> 02 0211fea8 79f798a6 kernel32!GetQueuedCompletionStatus+0x29
> 03 0211ff14 79f79c4f mscorwks!ThreadpoolMgr::CompletionPortThreadStart
> +0x11a
> 04 0211ffb8 77e64829 mscorwks!ThreadpoolMgr::intermediateThreadProc
> +0x49
> 05 0211ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 15
> System Thread ID: 1c10
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Other
> # ChildEBP RetAddr
> 00 0215fdb4 7c826f4b ntdll!KiFastSystemCallRet
> 01 0215fdb8 77e41ed1 ntdll!NtDelayExecution+0xc
> 02 0215fe20 79f56e05 kernel32!SleepEx+0x68
> 03 0215fe54 79f56dd5 mscorwks!EESleepEx+0xa3
> 04 0215fe88 79f409c5 mscorwks!__DangerousSwitchToThread+0x70
> 05 0215fe94 79f56ca7 mscorwks!__SwitchToThread+0xb
> 06 0215ffb8 77e64829 mscorwks!ThreadpoolMgr::GateThreadStart+0xa1
> 07 0215ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 16
> System Thread ID: c68
> Kernel Time: 0:0:0.750
> User Time: 0:0:5.281
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 0219fc80 7c827d0b ntdll!KiFastSystemCallRet
> 01 0219fc84 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 0219fcf4 79e718fd kernel32!WaitForSingleObjectEx+0xac
> 03 0219fd38 79e718c6 mscorwks!PEImage::LoadImage+0x199
> 04 0219fd88 79e7187c mscorwks!CLREvent::WaitEx+0x117
> 05 0219fd98 79f5e7db mscorwks!CLREvent::Wait+0x17
> 06 0219fe18 79f8b495 mscorwks!ThreadpoolMgr::SafeWait+0x73
> 07 0219fe94 79f79c4f mscorwks!ThreadpoolMgr::WorkerThreadStart+0xf5
> 08 0219ffb8 77e64829 mscorwks!ThreadpoolMgr::intermediateThreadProc
> +0x49
> 09 0219ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 17
> System Thread ID: cac
> Kernel Time: 0:0:0.906
> User Time: 0:0:6.953
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 021ffbe4 7c827d0b ntdll!KiFastSystemCallRet
> 01 021ffbe8 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 021ffc58 79e718fd kernel32!WaitForSingleObjectEx+0xac
> 03 021ffc9c 79e718c6 mscorwks!PEImage::LoadImage+0x199
> 04 021ffcec 79e7187c mscorwks!CLREvent::WaitEx+0x117
> 05 021ffcfc 79f5e7db mscorwks!CLREvent::Wait+0x17
> 06 021ffd7c 7a1153d9 mscorwks!ThreadpoolMgr::SafeWait+0x73
> 07 021ffda4 79f8b4e0 mscorwks!ThreadpoolMgr::EnterRetirement+0x8e
> 08 021ffe14 79f79c4f mscorwks!ThreadpoolMgr::WorkerThreadStart+0x393
> 09 021fffb8 77e64829 mscorwks!ThreadpoolMgr::intermediateThreadProc
> +0x49
> 0a 021fffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 18
> System Thread ID: 1770
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 022efe30 7c827cfb ntdll!KiFastSystemCallRet
> 01 022efe34 77e6202c ntdll!NtWaitForMultipleObjects+0xc
> 02 022efedc 77e62fbe kernel32!WaitForMultipleObjectsEx+0x11a
> 03 022efef8 79f0cf96 kernel32!WaitForMultipleObjects+0x18
> 04 022eff58 79f0cef3 mscorwks!DebuggerRCThread::MainLoop+0xcf
> 05 022eff88 79f0ce36 mscorwks!DebuggerRCThread::ThreadProc+0xca
> 06 022effb8 77e64829 mscorwks!DebuggerRCThread::ThreadProcStatic+0x82
> 07 022effec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 19
> System Thread ID: 914
> Kernel Time: 0:0:0.15
> User Time: 0:0:0.578
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 0275fe68 7c827d0b ntdll!KiFastSystemCallRet
> 01 0275fe6c 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 0275fedc 79e718fd kernel32!WaitForSingleObjectEx+0xac
> 03 0275ff20 79e718c6 mscorwks!PEImage::LoadImage+0x199
> 04 0275ff70 79e7187c mscorwks!CLREvent::WaitEx+0x117
> 05 0275ff80 7a0d866f mscorwks!CLREvent::Wait+0x17
> 06 0275ffa8 7a0d8987 mscorwks!SVR::gc_heap::gc_thread_function+0x2f
> 07 0275ffb8 77e64829 mscorwks!SVR::gc_heap::gc_thread_stub+0x9b
> 08 0275ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 20
> System Thread ID: 238c
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.609
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 0279fd68 7c827d0b ntdll!KiFastSystemCallRet
> 01 0279fd6c 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 0279fddc 79e718fd kernel32!WaitForSingleObjectEx+0xac
> 03 0279fe20 79e718c6 mscorwks!PEImage::LoadImage+0x199
> 04 0279fe70 79e7187c mscorwks!CLREvent::WaitEx+0x117
> 05 0279fe80 7a0d8898 mscorwks!CLREvent::Wait+0x17
> 06 0279fea8 7a0d8987 mscorwks!SVR::gc_heap::gc_thread_function+0x58
> 07 0279ffb8 77e64829 mscorwks!SVR::gc_heap::gc_thread_stub+0x9b
> 08 0279ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 21
> System Thread ID: e20
> Kernel Time: 0:0:0.15
> User Time: 0:0:0.578
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 027dfc68 7c827d0b ntdll!KiFastSystemCallRet
> 01 027dfc6c 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 027dfcdc 79e718fd kernel32!WaitForSingleObjectEx+0xac
> 03 027dfd20 79e718c6 mscorwks!PEImage::LoadImage+0x199
> 04 027dfd70 79e7187c mscorwks!CLREvent::WaitEx+0x117
> 05 027dfd80 7a0d8898 mscorwks!CLREvent::Wait+0x17
> 06 027dfda8 7a0d8987 mscorwks!SVR::gc_heap::gc_thread_function+0x58
> 07 027dffb8 77e64829 mscorwks!SVR::gc_heap::gc_thread_stub+0x9b
> 08 027dffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 22
> System Thread ID: 144c
> Kernel Time: 0:0:0.15
> User Time: 0:0:0.578
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 0281fb68 7c827d0b ntdll!KiFastSystemCallRet
> 01 0281fb6c 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 0281fbdc 79e718fd kernel32!WaitForSingleObjectEx+0xac
> 03 0281fc20 79e718c6 mscorwks!PEImage::LoadImage+0x199
> 04 0281fc70 79e7187c mscorwks!CLREvent::WaitEx+0x117
> 05 0281fc80 7a0d8898 mscorwks!CLREvent::Wait+0x17
> 06 0281fca8 7a0d8987 mscorwks!SVR::gc_heap::gc_thread_function+0x58
> 07 0281ffb8 77e64829 mscorwks!SVR::gc_heap::gc_thread_stub+0x9b
> 08 0281ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 23
> System Thread ID: 18f8
> Kernel Time: 0:0:0.31
> User Time: 0:0:0.31
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 0285fccc 7c827cfb ntdll!KiFastSystemCallRet
> 01 0285fcd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
> 02 0285fd78 77e62fbe kernel32!WaitForMultipleObjectsEx+0x11a
> 03 0285fd94 7a085b65 kernel32!WaitForMultipleObjects+0x18
> 04 0285fdb4 7a08648f mscorwks!SVR::WaitForFinalizerEvent+0x7a
> 05 0285fdc8 79ed8a2c mscorwks!SVR::GCHeap::FinalizerThreadWorker+0x75
> 06 0285fdd8 79ed89ca mscorwks!Thread::UserResumeThread+0xfb
> 07 0285fe6c 79ed88f1 mscorwks!Thread::DoADCallBack+0x355
> 08 0285fea8 79ed988c mscorwks!Thread::DoADCallBack+0x541
> 09 0285fed0 79ed9857 mscorwks!ManagedThreadBase_NoADTransition+0x32
> 0a 0285fedc 7a086638 mscorwks!ManagedThreadBase::FinalizerBase+0xb
> 0b 0285ff14 79ed8e36 mscorwks!SVR::GCHeap::FinalizerThreadStart+0xbb
> 0c 0285ffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
> 0d 0285ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 24
> System Thread ID: 1a88
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 0289fcb8 7c827d0b ntdll!KiFastSystemCallRet
> 01 0289fcbc 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 0289fd2c 79e718fd kernel32!WaitForSingleObjectEx+0xac
> 03 0289fd70 79e718c6 mscorwks!PEImage::LoadImage+0x199
> 04 0289fdc0 79e7187c mscorwks!CLREvent::WaitEx+0x117
> 05 0289fdd0 79f79049 mscorwks!CLREvent::Wait+0x17
> 06 0289fe94 79ed8e36 mscorwks!AppDomain::ADUnloadThreadStart+0x2fb
> 07 0289ffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
> 08 0289ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 25
> System Thread ID: 227c
> Kernel Time: 0:0:0.15
> User Time: 0:0:0.0
> Thread Type: Other
> # ChildEBP RetAddr
> 00 0294fec8 7c826f4b ntdll!KiFastSystemCallRet
> 01 0294fecc 77e41ed1 ntdll!NtDelayExecution+0xc
> 02 0294ff34 79f66a87 kernel32!SleepEx+0x68
> 03 0294ffb0 79f66a18 mscorwks!ThreadpoolMgr::TimerThreadFire+0x6d
> 04 0294ffb8 77e64829 mscorwks!ThreadpoolMgr::TimerThreadStart+0x54
> 05 0294ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 26
> System Thread ID: 2454
> Kernel Time: 0:0:1.0
> User Time: 0:0:8.625
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 1ac6fc00 7c827d0b ntdll!KiFastSystemCallRet
> 01 1ac6fc04 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 1ac6fc74 79e718fd kernel32!WaitForSingleObjectEx+0xac
> 03 1ac6fcb8 79e718c6 mscorwks!PEImage::LoadImage+0x199
> 04 1ac6fd08 79e7187c mscorwks!CLREvent::WaitEx+0x117
> 05 1ac6fd18 79f5e7db mscorwks!CLREvent::Wait+0x17
> 06 1ac6fd98 79f8b495 mscorwks!ThreadpoolMgr::SafeWait+0x73
> 07 1ac6fe14 79ed8e36 mscorwks!ThreadpoolMgr::WorkerThreadStart+0xf5
> 08 1ac6ffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
> 09 1ac6ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 27
> System Thread ID: 1ea0
> Kernel Time: 0:0:0.468
> User Time: 0:0:4.359
> Thread Type: Other
> # ChildEBP RetAddr
> 00 1acaf898 79f55b05 kernel32!RaiseException+0x53
> 01 1acaf8f8 7a056a79 mscorwks!RaiseTheExceptionInternalOnly+0x226
> 02 1acaf90c 7a056af2 mscorwks!RaiseTheException+0x4d
> 03 1acaf934 7a056b30 mscorwks!RaiseTheException+0xbf
> 04 1acaf960 7a056b41 mscorwks!RealCOMPlusThrow+0x38
> 05 1acaf96c 7a0e38fa mscorwks!RealCOMPlusThrow+0xb
> 06 1acafa9c 7a0e4b97 mscorwks!Thread::RaiseCrossContextException+0x3ac
> 07 1acafb50 7a0e5a07 mscorwks!Thread::DoADCallBack+0x26c
> 08 1acafb68 79ed89ca mscorwks!Thread::UserResumeThread+0xe1
> 09 1acafbfc 79ed88f1 mscorwks!Thread::DoADCallBack+0x355
> 0a 1acafc38 7a0e4d96 mscorwks!Thread::DoADCallBack+0x541
> 0b 1acafc60 7a0e4dc3 mscorwks!Thread::DoADCallBack+0x575
> 0c 1acafc74 7a077528 mscorwks!ManagedThreadBase::ThreadPool+0x13
> 0d 1acafcbc 7a077556 mscorwks!AddTimerCallbackEx+0x77
> 0e 1acafccc 7a1158ae mscorwks!AddTimerCallback+0xf
> 0f 1acafd08 79f8b7a5 mscorwks!
> ThreadpoolMgr::AsyncTimerCallbackCompletion+0x64
> 10 1acafd20 79f8b341 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0x40
> 11 1acafd94 79ed8e36 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x225
> 12 1acaffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
> 13 1acaffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 28
> System Thread ID: 878
> Kernel Time: 0:0:0.359
> User Time: 0:0:1.328
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 1acefae4 7c827d0b ntdll!KiFastSystemCallRet
> 01 1acefae8 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 1acefb58 79e718fd kernel32!WaitForSingleObjectEx+0xac
> 03 1acefb9c 79e718c6 mscorwks!PEImage::LoadImage+0x199
> 04 1acefbec 79e7187c mscorwks!CLREvent::WaitEx+0x117
> 05 1acefbfc 79f5e7db mscorwks!CLREvent::Wait+0x17
> 06 1acefc7c 7a1153d9 mscorwks!ThreadpoolMgr::SafeWait+0x73
> 07 1acefca4 79f8b4e0 mscorwks!ThreadpoolMgr::EnterRetirement+0x8e
> 08 1acefd14 79ed8e36 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x393
> 09 1aceffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
> 0a 1aceffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 29
> System Thread ID: 1824
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Other
> # ChildEBP RetAddr
> 00 1b5cff54 7c8277db ntdll!KiFastSystemCallRet
> 01 1b5cff58 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> 02 1b5cff84 65001897 kernel32!GetQueuedCompletionStatus+0x29
> 03 1b5cffb8 77e64829 System_Data!SNIAsyncWait+0x6c
> 04 1b5cffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 30
> System Thread ID: 9b4
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 1be8fec0 7c827cfb ntdll!KiFastSystemCallRet
> 01 1be8fec4 77e6202c ntdll!NtWaitForMultipleObjects+0xc
> 02 1be8ff6c 77e62fbe kernel32!WaitForMultipleObjectsEx+0x11a
> 03 1be8ff88 76929e35 kernel32!WaitForMultipleObjects+0x18
> 04 1be8ffb8 77e64829 USERENV!NotificationThread+0x5f
> 05 1be8ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 31
> System Thread ID: 15c4
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> *** WARNING: Unable to verify checksum for E:\WINDOWS\assembly
> \NativeImages_v2.0.50727_32\mscorlib
> \16d71085704686243fafcd1e8c89bda2\mscorlib.ni.dll
> Thread Type: Other
> # ChildEBP RetAddr
> 00 1c1ff34c 7c826f4b ntdll!KiFastSystemCallRet
> 01 1c1ff350 77e41ed1 ntdll!NtDelayExecution+0xc
> 02 1c1ff3b8 79f56e05 kernel32!SleepEx+0x68
> 03 1c1ff3ec 7a04cd47 mscorwks!EESleepEx+0xa3
> 04 1c1ff3f8 7a12f1c1 mscorwks!CExecutionEngine::ClrSleepEx+0xd
> 05 1c1ff408 7a0e44c1 mscorwks!ClrSleepEx+0x13
> 06 1c1ff444 7a079218 mscorwks!Thread::UserSleep+0x63
> 07 1c1ff4e4 1afb1b15 mscorwks!ThreadNative::Sleep+0xce
> WARNING: Frame IP not in any known module. Following frames may be
> wrong.
> 08 1c1ff50c 793d70fb 0x1afb1b15
> 09 00000000 00000000 mscorlib_ni+0x3170fb
>
>
>
>
> Thread ID: 32
> System Thread ID: 1f68
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Other
> # ChildEBP RetAddr
> 00 1c29f2cc 7c826f4b ntdll!KiFastSystemCallRet
> 01 1c29f2d0 77e41ed1 ntdll!NtDelayExecution+0xc
> 02 1c29f338 79f56e05 kernel32!SleepEx+0x68
> 03 1c29f36c 7a04cd47 mscorwks!EESleepEx+0xa3
> 04 1c29f378 7a12f1c1 mscorwks!CExecutionEngine::ClrSleepEx+0xd
> 05 1c29f388 7a0e44c1 mscorwks!ClrSleepEx+0x13
> 06 1c29f3c4 7a079218 mscorwks!Thread::UserSleep+0x63
> 07 1c29f464 1afb1b15 mscorwks!ThreadNative::Sleep+0xce
> WARNING: Frame IP not in any known module. Following frames may be
> wrong.
> 08 1c29f48c 793d70fb 0x1afb1b15
> 09 00000000 00000000 mscorlib_ni+0x3170fb
>
>
>
>
> Thread ID: 33
> System Thread ID: ef0
> Kernel Time: 0:0:0.15
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 1d5bfe48 7c827cfb ntdll!KiFastSystemCallRet
> 01 1d5bfe4c 77e6202c ntdll!NtWaitForMultipleObjects+0xc
> 02 1d5bfef4 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
> 03 1d5bff50 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
> 04 1d5bff6c 4dd668ab USER32!MsgWaitForMultipleObjects+0x1f
> 05 1d5bffb8 77e64829 gdiplus!BackgroundThreadProc+0x59
> 06 1d5bffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 34
> System Thread ID: 9d8
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Other
> # ChildEBP RetAddr
> 00 1e63ff70 7c8277db ntdll!KiFastSystemCallRet
> 01 1e63ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
> 02 1e63ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
> 03 1e63ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 35
> System Thread ID: 1f18
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Other
> # ChildEBP RetAddr
> 00 1e67ff70 7c8277db ntdll!KiFastSystemCallRet
> 01 1e67ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
> 02 1e67ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
> 03 1e67ffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 36
> System Thread ID: 1720
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Other
> # ChildEBP RetAddr
> 00 1e6bff70 7c8277db ntdll!KiFastSystemCallRet
> 01 1e6bff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
> 02 1e6bffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
> 03 1e6bffec 00000000 kernel32!BaseThreadStart+0x34
>
>
>
>
> Thread ID: 37
> System Thread ID: 1720
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Other
> # ChildEBP RetAddr
> 00 1e6bff70 7c8277db ntdll!KiFastSystemCallRet
> 01 1e6bff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
> 02 1e6bffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
> 03 1e6bffec 00000000 kernel32!BaseThreadStart+0x34
>
> *****
>
> Dump name is formatted as: PID-Timestamp.dmp
>
> Creating E:\Program Files\IIS Resources\IISState\output
> \9900-1195160343.dmp - user full dump
> Dump successfully written
>
> *****
>
> Closing open log file E:\Program Files\IIS Resources\IISState\output
> \IISState.log
>

Re: How the heck can you effectively debug these "process serving app

Hi Spitfire, thank you for the suggestion! Your link pointed to
Portmon, is that the same thing as Procmon? Have you had practical
experience using it to diagnose and resolve this specific type of
error?

Thanks
Shan


On Nov 15, 4:34 pm, "spitfire" wrote:
> Try
> Procmon utility , If you haven't tried yet.http://www.microsoft.com/technet/sysinternals/ProcessesA ndThreads/Por...
>
> "Shan Plourde" wrote in message
>
> news:ec423693-9008-414c-a8c7-02d9b6ef1cde@d50g2000hsf.google groups.com...
>
> > Hello there, in an attempt to retain the few dozen hairs remaining on
> > the top of my head before I yank them all out in angst, I'm completely
> > stuck....HEEEELP!
>
> > We are always getting these errors in our System event log:
>
> > A process serving application pool 'poolName' suffered a fatal
> > communication error with the World Wide Web Publishing Service. The
> > process id was '1234'.
>
> > We have IIS6 on Windows Server 2003. These errors are all coming from
> > our intranet website hosted on this server. The intranet website runs
> > on .NET 2.0. We also have many other websites hosted on the server,
> > most .NET, but some classic ASP, but this particular error only is
> > reported for the intranet website. All other ASP.NET applications are
> > hosted within their own application pools.
>
> > The error happens A LOT (like, every few minutes). It's a huge issue
> > because it seems to cause large file uploads through our Intranet to
> > crap out when the issue happens (perhaps due to application pool
> > recycling?).
>
> > So to start a debugging session I ran:
> > cscript \windows\system32\iisapp.vbs
>
> > Which lists all app pools and processes. In this case at the time I
> > ran iisapp.vbs, the application pool named "poolName" is hosted by
> > process 9900. So I ran this:
>
> > iisstate -p 9900 -hc -d
>
> > In an attempt to log the next hard crash. When the hard crash happens,
> > I just can't figure out how to make effective use of the log
> > information. I've looked at several logs but none seem to make me go
> > "AHA!". I understand that Microsoft recommends running this tool, but
> > I don't think that I can interpret any of it...please, MSFT or anyone,
> > your help is greatly appreciated to interpret the crash log and
> > provide recommendations!!! What follows is the log, through to the end
> > of this post....Thx!!!
>
> > ----------------------------------------
>
> > Opened log file 'E:\Program Files\IIS Resources\IISState\output
> > \IISState.log'
>
> > ***********************
> > Starting new log output
> > IISState version 3.0
>
> > Thu Nov 15 14:46:28 2007
>
> > OS = Windows 2003 Server
> > Executable: w3wp.exe
> > PID = 6976
>
> > Note: Thread times are formatted as HH:MM:SS.ms
>
> > ***********************
>
> > IIS has crashed...
> > Beginning Analysis
> > DLL (!FunctionName) that failed:
>
> > Thread ID: 30
> > System Thread ID: 1a90
> > Kernel Time: 0:0:0.375
> > User Time: 0:0:2.140
> > Thread Type: Other
> > # ChildEBP RetAddr
> > WARNING: Frame IP not in any known module. Following frames may be
> > wrong.
> > 00 1bf8f118 1db03f0e 0x1db03f5f
> > 01 00000000 00000000 0x1db03f0e
> > Closing open log file E:\Program Files\IIS Resources\IISState\output
> > \IISState.log
> > Opened log file 'E:\Program Files\IIS Resources\IISState\output
> > \IISState.log'
>
> > ***********************
> > Starting new log output
> > IISState version 3.0
>
> > Thu Nov 15 15:59:00 2007
>
> > OS = Windows 2003 Server
> > Executable: w3wp.exe
> > PID = 9900
>
> > Note: Thread times are formatted as HH:MM:SS.ms
>
> > ***********************
>
> > IIS has crashed...
> > Beginning Analysis
> > DLL (!FunctionName) that failed: kernel32!RaiseException
>
> > Thread ID: 27
> > System Thread ID: 1ea0
> > Kernel Time: 0:0:0.468
> > User Time: 0:0:4.359
> > Thread Type: Other
> > # ChildEBP RetAddr
> > 00 1acaf898 79f55b05 kernel32!RaiseException+0x53
> > 01 1acaf8f8 7a056a79 mscorwks!RaiseTheExceptionInternalOnly+0x226
> > 02 1acaf90c 7a056af2 mscorwks!RaiseTheException+0x4d
> > 03 1acaf934 7a056b30 mscorwks!RaiseTheException+0xbf
> > 04 1acaf960 7a056b41 mscorwks!RealCOMPlusThrow+0x38
> > 05 1acaf96c 7a0e38fa mscorwks!RealCOMPlusThrow+0xb
> > 06 1acafa9c 7a0e4b97 mscorwks!Thread::RaiseCrossContextException+0x3ac
> > 07 1acafb50 7a0e5a07 mscorwks!Thread::DoADCallBack+0x26c
> > 08 1acafb68 79ed89ca mscorwks!Thread::UserResumeThread+0xe1
> > 09 1acafbfc 79ed88f1 mscorwks!Thread::DoADCallBack+0x355
> > 0a 1acafc38 7a0e4d96 mscorwks!Thread::DoADCallBack+0x541
> > 0b 1acafc60 7a0e4dc3 mscorwks!Thread::DoADCallBack+0x575
> > 0c 1acafc74 7a077528 mscorwks!ManagedThreadBase::ThreadPool+0x13
> > 0d 1acafcbc 7a077556 mscorwks!AddTimerCallbackEx+0x77
> > 0e 1acafccc 7a1158ae mscorwks!AddTimerCallback+0xf
> > 0f 1acafd08 79f8b7a5 mscorwks!
> > ThreadpoolMgr::AsyncTimerCallbackCompletion+0x64
> > 10 1acafd20 79f8b341 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0x40
> > 11 1acafd94 79ed8e36 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x225
> > 12 1acaffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
> > 13 1acaffec 00000000 kernel32!BaseThreadStart+0x34
> > Closing open log file E:\Program Files\IIS Resources\IISState\output
> > \IISState.log
> > Opened log file 'E:\Program Files\IIS Resources\IISState\output
> > \IISState.log'
>
> > ***********************
> > Starting new log output
> > IISState version 3.0
>
> > Thu Nov 15 15:59:01 2007
>
> > OS = Windows 2003 Server
> > Executable: w3wp.exe
> > PID = 9900
>
> > Note: Thread times are formatted as HH:MM:SS.ms
>
> > ***********************
>
> > Thread ID: 0
> > System Thread ID: 26d4
> > Kernel Time: 0:0:0.78
> > User Time: 0:0:0.171
> > Thread Status: Thread is in a WAIT state.
> > Thread Type: HTTP Compression Thread
> > # ChildEBP RetAddr
> > 00 0006fc08 7c827d0b ntdll!KiFastSystemCallRet
> > 01 0006fc0c 77e61d1e ntdll!NtWaitForSingleObject+0xc
> > 02 0006fc7c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> > 03 0006fc90 5a364662 kernel32!WaitForSingleObject+0x12
> > 04 0006fca0 5a366e3f w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
> > 05 0006fca8 5a3af42d w3dt!UlAtqStartListen+0x2d
> > 06 0006fcb8 5a3bc335 w3core!W3_SERVER::StartListen+0xbd
> > 07 0006ff0c 0100187c w3core!UlW3Start+0x26e
> > 08 0006ff44 01001a27 w3wp!wmain+0x22a
> > 09 0006ffc0 77e6f23b w3wp!wmainCRTStartup+0x12f
> > 0a 0006fff0 00000000 kernel32!BaseProcessStart+0x23
>
> > Thread ID: 1
> > System Thread ID: 1290
> > Kernel Time: 0:0:0.0
> > User Time: 0:0:0.0
> > Thread Status: Thread is in a WAIT state.
> > Thread Status: Thread is in a WAIT state.
> > Thread Type: Other
> > # ChildEBP RetAddr
> > 00 008bfea0 7c827cfb ntdll!KiFastSystemCallRet
> > 01 008bfea4 7c80e5bb ntdll!NtWaitForMultipleObjects+0xc
> > 02 008bff48 7c80e4a2 ntdll!EtwpWaitForMultipleObjectsEx+0xf7
> > 03 008bffb8 77e64829 ntdll!EtwpEventPump+0x27f
> > 04 008bffec 00000000 kernel32!BaseThreadStart+0x34
>
> > Thread ID: 2
> > System Thread ID: 234c
> > Kernel Time: 0:0:0.0
> > User Time: 0:0:0.0
> > Thread Type: Other
> > # ChildEBP RetAddr
> > 00 00f4ff9c 7c826f4b ntdll!KiFastSystemCallRet
> > 01 00f4ffa0 7c83d424 ntdll!NtDelayExecution+0xc
> > 02 00f4ffb8 77e64829 ntdll!RtlpTimerThread+0x47
> > 03 00f4ffec 00000000 kernel32!BaseThreadStart+0x34
>
> > Thread ID: 3
> > System Thread ID: c6c
> > Kernel Time: 0:0:0.0
> > User Time: 0:0:0.0
> > Thread Type: Other
> > # ChildEBP RetAddr
> > 00 00f8ff70 7c8277db ntdll!KiFastSystemCallRet
> > 01 00f8ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
> > 02 00f8ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
> > 03 00f8ffec 00000000 kernel32!BaseThreadStart+0x34
>
> > Thread ID: 4
> > System Thread ID: 27ec
> > Kernel Time: 0:0:0.15
> > User Time: 0:0:0.0
> > Thread Type: Possible ASP page. Possible DCOM activity
> > Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
> > Continuing with other analysis.
>
> > No remote call being made
>
> > # ChildEBP RetAddr
> > 00 00fffe18 7c82783b ntdll!KiFastSystemCallRet
> > 01 00fffe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
> > 02 00ffff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
> > 03 00ffff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
> > 04 00ffffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
> > 05 00ffffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
> > 06 00ffffec 00000000 kernel32!BaseThreadStart+0x34
>
> > Thread ID: 5
> > System Thread ID: 5f8
> > Kernel Time: 0:0:0.0
> > User Time: 0:0:0.0
> > Thread Type: Other
> > # ChildEBP RetAddr
> > 00 1e47ff70 7c8277db ntdll!KiFastSystemCallRet
> > 01 1e47ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
> > 02 1e47ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
> > 03 1e47ffec 00000000 kernel32!BaseThreadStart+0x34
>
> > Thread ID: 6
> > System Thread ID: 1070
> > Kernel Time: 0:0:0.93
> > User Time: 0:0:0.265
> > Thread Type: HTTP Listener
> > # ChildEBP RetAddr
> > 00 0120ff24 7c8277db ntdll!KiFastSystemCallRet
> > 01 0120ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> > 02 0120ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
> > 03 0120ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
> > 04 0120ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
> > 05 0120ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
> > 06 0120ffec 00000000 kernel32!BaseThreadStart+0x34
>
> > Thread ID: 7
> > System Thread ID: 2734
> > Kernel Time: 0:0:0.125
> > User Time: 0:0:0.171
> > Thread Type: HTTP Listener
> > # ChildEBP RetAddr
> > 00 0124ff24 7c8277db ntdll!KiFastSystemCallRet
> > 01 0124ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> > 02 0124ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
> > 03 0124ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
> > 04 0124ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
> > 05 0124ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
> > 06 0124ffec 00000000 kernel32!BaseThreadStart+0x34
>
> > Thread ID: 8
> > System Thread ID: bb4
> > Kernel Time: 0:0:0.140
> > User Time: 0:0:0.593
> > Thread Type: HTTP Listener
> > # ChildEBP RetAddr
> > 00 0128ff24 7c8277db ntdll!KiFastSystemCallRet
> > 01 0128ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> > 02 0128ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
> > 03 0128ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
> > 04 0128ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
> > 05 0128ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
> > 06 0128ffec 00000000 kernel32!BaseThreadStart+0x34
>
> > Thread ID: 9
> > System Thread ID: fc0
> > Kernel Time: 0:0:0.140
> > User Time: 0:0:0.359
> > Thread Type: HTTP Listener
> > # ChildEBP RetAddr
> > 00
>
> ...
>
> read more >>

Re: How the heck can you effectively debug these "process serving app pool suffered...blah blah

Yes, run the utility and create some filters just to display w3wp.exe
process(s).
You will be able to see each file/registry I/O performed by the process.
It helped me trace few problems in my system.
But it might be different in your case.

good luck!


"Shan Plourde" wrote in message
news:4883b6f6-dcd5-4346-b064-fc4d488879b6@d61g2000hsa.google groups.com...
> Hi Spitfire, thank you for the suggestion! Your link pointed to
> Portmon, is that the same thing as Procmon? Have you had practical
> experience using it to diagnose and resolve this specific type of
> error?
>
> Thanks
> Shan
>
>
> On Nov 15, 4:34 pm, "spitfire" wrote:
>> Try
>> Procmon utility , If you haven't tried
>> yet.http://www.microsoft.com/technet/sysinternals/ProcessesA ndThreads/Por...
>>
>> "Shan Plourde" wrote in message
>>
>> news:ec423693-9008-414c-a8c7-02d9b6ef1cde@d50g2000hsf.google groups.com...
>>
>> > Hello there, in an attempt to retain the few dozen hairs remaining on
>> > the top of my head before I yank them all out in angst, I'm completely
>> > stuck....HEEEELP!
>>
>> > We are always getting these errors in our System event log:
>>
>> > A process serving application pool 'poolName' suffered a fatal
>> > communication error with the World Wide Web Publishing Service. The
>> > process id was '1234'.
>>
>> > We have IIS6 on Windows Server 2003. These errors are all coming from
>> > our intranet website hosted on this server. The intranet website runs
>> > on .NET 2.0. We also have many other websites hosted on the server,
>> > most .NET, but some classic ASP, but this particular error only is
>> > reported for the intranet website. All other ASP.NET applications are
>> > hosted within their own application pools.
>>
>> > The error happens A LOT (like, every few minutes). It's a huge issue
>> > because it seems to cause large file uploads through our Intranet to
>> > crap out when the issue happens (perhaps due to application pool
>> > recycling?).
>>
>> > So to start a debugging session I ran:
>> > cscript \windows\system32\iisapp.vbs
>>
>> > Which lists all app pools and processes. In this case at the time I
>> > ran iisapp.vbs, the application pool named "poolName" is hosted by
>> > process 9900. So I ran this:
>>
>> > iisstate -p 9900 -hc -d
>>
>> > In an attempt to log the next hard crash. When the hard crash happens,
>> > I just can't figure out how to make effective use of the log
>> > information. I've looked at several logs but none seem to make me go
>> > "AHA!". I understand that Microsoft recommends running this tool, but
>> > I don't think that I can interpret any of it...please, MSFT or anyone,
>> > your help is greatly appreciated to interpret the crash log and
>> > provide recommendations!!! What follows is the log, through to the end
>> > of this post....Thx!!!
>>
>> > ----------------------------------------
>>
>> > Opened log file 'E:\Program Files\IIS Resources\IISState\output
>> > \IISState.log'
>>
>> > ***********************
>> > Starting new log output
>> > IISState version 3.0
>>
>> > Thu Nov 15 14:46:28 2007
>>
>> > OS = Windows 2003 Server
>> > Executable: w3wp.exe
>> > PID = 6976
>>
>> > Note: Thread times are formatted as HH:MM:SS.ms
>>
>> > ***********************
>>
>> > IIS has crashed...
>> > Beginning Analysis
>> > DLL (!FunctionName) that failed:
>>
>> > Thread ID: 30
>> > System Thread ID: 1a90
>> > Kernel Time: 0:0:0.375
>> > User Time: 0:0:2.140
>> > Thread Type: Other
>> > # ChildEBP RetAddr
>> > WARNING: Frame IP not in any known module. Following frames may be
>> > wrong.
>> > 00 1bf8f118 1db03f0e 0x1db03f5f
>> > 01 00000000 00000000 0x1db03f0e
>> > Closing open log file E:\Program Files\IIS Resources\IISState\output
>> > \IISState.log
>> > Opened log file 'E:\Program Files\IIS Resources\IISState\output
>> > \IISState.log'
>>
>> > ***********************
>> > Starting new log output
>> > IISState version 3.0
>>
>> > Thu Nov 15 15:59:00 2007
>>
>> > OS = Windows 2003 Server
>> > Executable: w3wp.exe
>> > PID = 9900
>>
>> > Note: Thread times are formatted as HH:MM:SS.ms
>>
>> > ***********************
>>
>> > IIS has crashed...
>> > Beginning Analysis
>> > DLL (!FunctionName) that failed: kernel32!RaiseException
>>
>> > Thread ID: 27
>> > System Thread ID: 1ea0
>> > Kernel Time: 0:0:0.468
>> > User Time: 0:0:4.359
>> > Thread Type: Other
>> > # ChildEBP RetAddr
>> > 00 1acaf898 79f55b05 kernel32!RaiseException+0x53
>> > 01 1acaf8f8 7a056a79 mscorwks!RaiseTheExceptionInternalOnly+0x226
>> > 02 1acaf90c 7a056af2 mscorwks!RaiseTheException+0x4d
>> > 03 1acaf934 7a056b30 mscorwks!RaiseTheException+0xbf
>> > 04 1acaf960 7a056b41 mscorwks!RealCOMPlusThrow+0x38
>> > 05 1acaf96c 7a0e38fa mscorwks!RealCOMPlusThrow+0xb
>> > 06 1acafa9c 7a0e4b97 mscorwks!Thread::RaiseCrossContextException+0x3ac
>> > 07 1acafb50 7a0e5a07 mscorwks!Thread::DoADCallBack+0x26c
>> > 08 1acafb68 79ed89ca mscorwks!Thread::UserResumeThread+0xe1
>> > 09 1acafbfc 79ed88f1 mscorwks!Thread::DoADCallBack+0x355
>> > 0a 1acafc38 7a0e4d96 mscorwks!Thread::DoADCallBack+0x541
>> > 0b 1acafc60 7a0e4dc3 mscorwks!Thread::DoADCallBack+0x575
>> > 0c 1acafc74 7a077528 mscorwks!ManagedThreadBase::ThreadPool+0x13
>> > 0d 1acafcbc 7a077556 mscorwks!AddTimerCallbackEx+0x77
>> > 0e 1acafccc 7a1158ae mscorwks!AddTimerCallback+0xf
>> > 0f 1acafd08 79f8b7a5 mscorwks!
>> > ThreadpoolMgr::AsyncTimerCallbackCompletion+0x64
>> > 10 1acafd20 79f8b341 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0x40
>> > 11 1acafd94 79ed8e36 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x225
>> > 12 1acaffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
>> > 13 1acaffec 00000000 kernel32!BaseThreadStart+0x34
>> > Closing open log file E:\Program Files\IIS Resources\IISState\output
>> > \IISState.log
>> > Opened log file 'E:\Program Files\IIS Resources\IISState\output
>> > \IISState.log'
>>
>> > ***********************
>> > Starting new log output
>> > IISState version 3.0
>>
>> > Thu Nov 15 15:59:01 2007
>>
>> > OS = Windows 2003 Server
>> > Executable: w3wp.exe
>> > PID = 9900
>>
>> > Note: Thread times are formatted as HH:MM:SS.ms
>>
>> > ***********************
>>
>> > Thread ID: 0
>> > System Thread ID: 26d4
>> > Kernel Time: 0:0:0.78
>> > User Time: 0:0:0.171
>> > Thread Status: Thread is in a WAIT state.
>> > Thread Type: HTTP Compression Thread
>> > # ChildEBP RetAddr
>> > 00 0006fc08 7c827d0b ntdll!KiFastSystemCallRet
>> > 01 0006fc0c 77e61d1e ntdll!NtWaitForSingleObject+0xc
>> > 02 0006fc7c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
>> > 03 0006fc90 5a364662 kernel32!WaitForSingleObject+0x12
>> > 04 0006fca0 5a366e3f w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
>> > 05 0006fca8 5a3af42d w3dt!UlAtqStartListen+0x2d
>> > 06 0006fcb8 5a3bc335 w3core!W3_SERVER::StartListen+0xbd
>> > 07 0006ff0c 0100187c w3core!UlW3Start+0x26e
>> > 08 0006ff44 01001a27 w3wp!wmain+0x22a
>> > 09 0006ffc0 77e6f23b w3wp!wmainCRTStartup+0x12f
>> > 0a 0006fff0 00000000 kernel32!BaseProcessStart+0x23
>>
>> > Thread ID: 1
>> > System Thread ID: 1290
>> > Kernel Time: 0:0:0.0
>> > User Time: 0:0:0.0
>> > Thread Status: Thread is in a WAIT state.
>> > Thread Status: Thread is in a WAIT state.
>> > Thread Type: Other
>> > # ChildEBP RetAddr
>> > 00 008bfea0 7c827cfb ntdll!KiFastSystemCallRet
>> > 01 008bfea4 7c80e5bb ntdll!NtWaitForMultipleObjects+0xc
>> > 02 008bff48 7c80e4a2 ntdll!EtwpWaitForMultipleObjectsEx+0xf7
>> > 03 008bffb8 77e64829 ntdll!EtwpEventPump+0x27f
>> > 04 008bffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> > Thread ID: 2
>> > System Thread ID: 234c
>> > Kernel Time: 0:0:0.0
>> > User Time: 0:0:0.0
>> > Thread Type: Other
>> > # ChildEBP RetAddr
>> > 00 00f4ff9c 7c826f4b ntdll!KiFastSystemCallRet
>> > 01 00f4ffa0 7c83d424 ntdll!NtDelayExecution+0xc
>> > 02 00f4ffb8 77e64829 ntdll!RtlpTimerThread+0x47
>> > 03 00f4ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> > Thread ID: 3
>> > System Thread ID: c6c
>> > Kernel Time: 0:0:0.0
>> > User Time: 0:0:0.0
>> > Thread Type: Other
>> > # ChildEBP RetAddr
>> > 00 00f8ff70 7c8277db ntdll!KiFastSystemCallRet
>> > 01 00f8ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
>> > 02 00f8ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
>> > 03 00f8ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> > Thread ID: 4
>> > System Thread ID: 27ec
>> > Kernel Time: 0:0:0.15
>> > User Time: 0:0:0.0
>> > Thread Type: Possible ASP page. Possible DCOM activity
>> > Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
>> > Continuing with other analysis.
>>
>> > No remote call being made
>>
>> > # ChildEBP RetAddr
>> > 00 00fffe18 7c82783b ntdll!KiFastSystemCallRet
>> > 01 00fffe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
>> > 02 00ffff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
>> > 03 00ffff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
>> > 04 00ffffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
>> > 05 00ffffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
>> > 06 00ffffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> > Thread ID: 5
>> > System Thread ID: 5f8
>> > Kernel Time: 0:0:0.0
>> > User Time: 0:0:0.0
>> > Thread Type: Other
>> > # ChildEBP RetAddr
>> > 00 1e47ff70 7c8277db ntdll!KiFastSystemCallRet
>> > 01 1e47ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
>> > 02 1e47ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
>> > 03 1e47ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> > Thread ID: 6
>> > System Thread ID: 1070
>> > Kernel Time: 0:0:0.93
>> > User Time: 0:0:0.265
>> > Thread Type: HTTP Listener
>> > # ChildEBP RetAddr
>> > 00 0120ff24 7c8277db ntdll!KiFastSystemCallRet
>> > 01 0120ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
>> > 02 0120ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
>> > 03 0120ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
>> > 04 0120ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
>> > 05 0120ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
>> > 06 0120ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> > Thread ID: 7
>> > System Thread ID: 2734
>> > Kernel Time: 0:0:0.125
>> > User Time: 0:0:0.171
>> > Thread Type: HTTP Listener
>> > # ChildEBP RetAddr
>> > 00 0124ff24 7c8277db ntdll!KiFastSystemCallRet
>> > 01 0124ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
>> > 02 0124ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
>> > 03 0124ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
>> > 04 0124ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
>> > 05 0124ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
>> > 06 0124ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> > Thread ID: 8
>> > System Thread ID: bb4
>> > Kernel Time: 0:0:0.140
>> > User Time: 0:0:0.593
>> > Thread Type: HTTP Listener
>> > # ChildEBP RetAddr
>> > 00 0128ff24 7c8277db ntdll!KiFastSystemCallRet
>> > 01 0128ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
>> > 02 0128ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
>> > 03 0128ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
>> > 04 0128ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
>> > 05 0128ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
>> > 06 0128ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> > Thread ID: 9
>> > System Thread ID: fc0
>> > Kernel Time: 0:0:0.140
>> > User Time: 0:0:0.359
>> > Thread Type: HTTP Listener
>> > # ChildEBP RetAddr
>> > 00
>>
>> ...
>>
>> read more >>
>

Re: How the heck can you effectively debug these "process serving app pool suffered...blah blah

Sorry, may be I gave you wrong link.
Try this one.
http://www.microsoft.com/technet/sysinternals/ProcessesAndTh reads/processmonitor.mspx

my bad :(



"Shan Plourde" wrote in message
news:4883b6f6-dcd5-4346-b064-fc4d488879b6@d61g2000hsa.google groups.com...
> Hi Spitfire, thank you for the suggestion! Your link pointed to
> Portmon, is that the same thing as Procmon? Have you had practical
> experience using it to diagnose and resolve this specific type of
> error?
>
> Thanks
> Shan
>
>
> On Nov 15, 4:34 pm, "spitfire" wrote:
>> Try
>> Procmon utility , If you haven't tried
>> yet.http://www.microsoft.com/technet/sysinternals/ProcessesA ndThreads/Por...
>>
>> "Shan Plourde" wrote in message
>>
>> news:ec423693-9008-414c-a8c7-02d9b6ef1cde@d50g2000hsf.google groups.com...
>>
>> > Hello there, in an attempt to retain the few dozen hairs remaining on
>> > the top of my head before I yank them all out in angst, I'm completely
>> > stuck....HEEEELP!
>>
>> > We are always getting these errors in our System event log:
>>
>> > A process serving application pool 'poolName' suffered a fatal
>> > communication error with the World Wide Web Publishing Service. The
>> > process id was '1234'.
>>
>> > We have IIS6 on Windows Server 2003. These errors are all coming from
>> > our intranet website hosted on this server. The intranet website runs
>> > on .NET 2.0. We also have many other websites hosted on the server,
>> > most .NET, but some classic ASP, but this particular error only is
>> > reported for the intranet website. All other ASP.NET applications are
>> > hosted within their own application pools.
>>
>> > The error happens A LOT (like, every few minutes). It's a huge issue
>> > because it seems to cause large file uploads through our Intranet to
>> > crap out when the issue happens (perhaps due to application pool
>> > recycling?).
>>
>> > So to start a debugging session I ran:
>> > cscript \windows\system32\iisapp.vbs
>>
>> > Which lists all app pools and processes. In this case at the time I
>> > ran iisapp.vbs, the application pool named "poolName" is hosted by
>> > process 9900. So I ran this:
>>
>> > iisstate -p 9900 -hc -d
>>
>> > In an attempt to log the next hard crash. When the hard crash happens,
>> > I just can't figure out how to make effective use of the log
>> > information. I've looked at several logs but none seem to make me go
>> > "AHA!". I understand that Microsoft recommends running this tool, but
>> > I don't think that I can interpret any of it...please, MSFT or anyone,
>> > your help is greatly appreciated to interpret the crash log and
>> > provide recommendations!!! What follows is the log, through to the end
>> > of this post....Thx!!!
>>
>> > ----------------------------------------
>>
>> > Opened log file 'E:\Program Files\IIS Resources\IISState\output
>> > \IISState.log'
>>
>> > ***********************
>> > Starting new log output
>> > IISState version 3.0
>>
>> > Thu Nov 15 14:46:28 2007
>>
>> > OS = Windows 2003 Server
>> > Executable: w3wp.exe
>> > PID = 6976
>>
>> > Note: Thread times are formatted as HH:MM:SS.ms
>>
>> > ***********************
>>
>> > IIS has crashed...
>> > Beginning Analysis
>> > DLL (!FunctionName) that failed:
>>
>> > Thread ID: 30
>> > System Thread ID: 1a90
>> > Kernel Time: 0:0:0.375
>> > User Time: 0:0:2.140
>> > Thread Type: Other
>> > # ChildEBP RetAddr
>> > WARNING: Frame IP not in any known module. Following frames may be
>> > wrong.
>> > 00 1bf8f118 1db03f0e 0x1db03f5f
>> > 01 00000000 00000000 0x1db03f0e
>> > Closing open log file E:\Program Files\IIS Resources\IISState\output
>> > \IISState.log
>> > Opened log file 'E:\Program Files\IIS Resources\IISState\output
>> > \IISState.log'
>>
>> > ***********************
>> > Starting new log output
>> > IISState version 3.0
>>
>> > Thu Nov 15 15:59:00 2007
>>
>> > OS = Windows 2003 Server
>> > Executable: w3wp.exe
>> > PID = 9900
>>
>> > Note: Thread times are formatted as HH:MM:SS.ms
>>
>> > ***********************
>>
>> > IIS has crashed...
>> > Beginning Analysis
>> > DLL (!FunctionName) that failed: kernel32!RaiseException
>>
>> > Thread ID: 27
>> > System Thread ID: 1ea0
>> > Kernel Time: 0:0:0.468
>> > User Time: 0:0:4.359
>> > Thread Type: Other
>> > # ChildEBP RetAddr
>> > 00 1acaf898 79f55b05 kernel32!RaiseException+0x53
>> > 01 1acaf8f8 7a056a79 mscorwks!RaiseTheExceptionInternalOnly+0x226
>> > 02 1acaf90c 7a056af2 mscorwks!RaiseTheException+0x4d
>> > 03 1acaf934 7a056b30 mscorwks!RaiseTheException+0xbf
>> > 04 1acaf960 7a056b41 mscorwks!RealCOMPlusThrow+0x38
>> > 05 1acaf96c 7a0e38fa mscorwks!RealCOMPlusThrow+0xb
>> > 06 1acafa9c 7a0e4b97 mscorwks!Thread::RaiseCrossContextException+0x3ac
>> > 07 1acafb50 7a0e5a07 mscorwks!Thread::DoADCallBack+0x26c
>> > 08 1acafb68 79ed89ca mscorwks!Thread::UserResumeThread+0xe1
>> > 09 1acafbfc 79ed88f1 mscorwks!Thread::DoADCallBack+0x355
>> > 0a 1acafc38 7a0e4d96 mscorwks!Thread::DoADCallBack+0x541
>> > 0b 1acafc60 7a0e4dc3 mscorwks!Thread::DoADCallBack+0x575
>> > 0c 1acafc74 7a077528 mscorwks!ManagedThreadBase::ThreadPool+0x13
>> > 0d 1acafcbc 7a077556 mscorwks!AddTimerCallbackEx+0x77
>> > 0e 1acafccc 7a1158ae mscorwks!AddTimerCallback+0xf
>> > 0f 1acafd08 79f8b7a5 mscorwks!
>> > ThreadpoolMgr::AsyncTimerCallbackCompletion+0x64
>> > 10 1acafd20 79f8b341 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0x40
>> > 11 1acafd94 79ed8e36 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x225
>> > 12 1acaffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
>> > 13 1acaffec 00000000 kernel32!BaseThreadStart+0x34
>> > Closing open log file E:\Program Files\IIS Resources\IISState\output
>> > \IISState.log
>> > Opened log file 'E:\Program Files\IIS Resources\IISState\output
>> > \IISState.log'
>>
>> > ***********************
>> > Starting new log output
>> > IISState version 3.0
>>
>> > Thu Nov 15 15:59:01 2007
>>
>> > OS = Windows 2003 Server
>> > Executable: w3wp.exe
>> > PID = 9900
>>
>> > Note: Thread times are formatted as HH:MM:SS.ms
>>
>> > ***********************
>>
>> > Thread ID: 0
>> > System Thread ID: 26d4
>> > Kernel Time: 0:0:0.78
>> > User Time: 0:0:0.171
>> > Thread Status: Thread is in a WAIT state.
>> > Thread Type: HTTP Compression Thread
>> > # ChildEBP RetAddr
>> > 00 0006fc08 7c827d0b ntdll!KiFastSystemCallRet
>> > 01 0006fc0c 77e61d1e ntdll!NtWaitForSingleObject+0xc
>> > 02 0006fc7c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
>> > 03 0006fc90 5a364662 kernel32!WaitForSingleObject+0x12
>> > 04 0006fca0 5a366e3f w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
>> > 05 0006fca8 5a3af42d w3dt!UlAtqStartListen+0x2d
>> > 06 0006fcb8 5a3bc335 w3core!W3_SERVER::StartListen+0xbd
>> > 07 0006ff0c 0100187c w3core!UlW3Start+0x26e
>> > 08 0006ff44 01001a27 w3wp!wmain+0x22a
>> > 09 0006ffc0 77e6f23b w3wp!wmainCRTStartup+0x12f
>> > 0a 0006fff0 00000000 kernel32!BaseProcessStart+0x23
>>
>> > Thread ID: 1
>> > System Thread ID: 1290
>> > Kernel Time: 0:0:0.0
>> > User Time: 0:0:0.0
>> > Thread Status: Thread is in a WAIT state.
>> > Thread Status: Thread is in a WAIT state.
>> > Thread Type: Other
>> > # ChildEBP RetAddr
>> > 00 008bfea0 7c827cfb ntdll!KiFastSystemCallRet
>> > 01 008bfea4 7c80e5bb ntdll!NtWaitForMultipleObjects+0xc
>> > 02 008bff48 7c80e4a2 ntdll!EtwpWaitForMultipleObjectsEx+0xf7
>> > 03 008bffb8 77e64829 ntdll!EtwpEventPump+0x27f
>> > 04 008bffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> > Thread ID: 2
>> > System Thread ID: 234c
>> > Kernel Time: 0:0:0.0
>> > User Time: 0:0:0.0
>> > Thread Type: Other
>> > # ChildEBP RetAddr
>> > 00 00f4ff9c 7c826f4b ntdll!KiFastSystemCallRet
>> > 01 00f4ffa0 7c83d424 ntdll!NtDelayExecution+0xc
>> > 02 00f4ffb8 77e64829 ntdll!RtlpTimerThread+0x47
>> > 03 00f4ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> > Thread ID: 3
>> > System Thread ID: c6c
>> > Kernel Time: 0:0:0.0
>> > User Time: 0:0:0.0
>> > Thread Type: Other
>> > # ChildEBP RetAddr
>> > 00 00f8ff70 7c8277db ntdll!KiFastSystemCallRet
>> > 01 00f8ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
>> > 02 00f8ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
>> > 03 00f8ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> > Thread ID: 4
>> > System Thread ID: 27ec
>> > Kernel Time: 0:0:0.15
>> > User Time: 0:0:0.0
>> > Thread Type: Possible ASP page. Possible DCOM activity
>> > Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
>> > Continuing with other analysis.
>>
>> > No remote call being made
>>
>> > # ChildEBP RetAddr
>> > 00 00fffe18 7c82783b ntdll!KiFastSystemCallRet
>> > 01 00fffe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
>> > 02 00ffff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
>> > 03 00ffff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
>> > 04 00ffffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
>> > 05 00ffffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
>> > 06 00ffffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> > Thread ID: 5
>> > System Thread ID: 5f8
>> > Kernel Time: 0:0:0.0
>> > User Time: 0:0:0.0
>> > Thread Type: Other
>> > # ChildEBP RetAddr
>> > 00 1e47ff70 7c8277db ntdll!KiFastSystemCallRet
>> > 01 1e47ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
>> > 02 1e47ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
>> > 03 1e47ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> > Thread ID: 6
>> > System Thread ID: 1070
>> > Kernel Time: 0:0:0.93
>> > User Time: 0:0:0.265
>> > Thread Type: HTTP Listener
>> > # ChildEBP RetAddr
>> > 00 0120ff24 7c8277db ntdll!KiFastSystemCallRet
>> > 01 0120ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
>> > 02 0120ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
>> > 03 0120ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
>> > 04 0120ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
>> > 05 0120ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
>> > 06 0120ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> > Thread ID: 7
>> > System Thread ID: 2734
>> > Kernel Time: 0:0:0.125
>> > User Time: 0:0:0.171
>> > Thread Type: HTTP Listener
>> > # ChildEBP RetAddr
>> > 00 0124ff24 7c8277db ntdll!KiFastSystemCallRet
>> > 01 0124ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
>> > 02 0124ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
>> > 03 0124ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
>> > 04 0124ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
>> > 05 0124ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
>> > 06 0124ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> > Thread ID: 8
>> > System Thread ID: bb4
>> > Kernel Time: 0:0:0.140
>> > User Time: 0:0:0.593
>> > Thread Type: HTTP Listener
>> > # ChildEBP RetAddr
>> > 00 0128ff24 7c8277db ntdll!KiFastSystemCallRet
>> > 01 0128ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
>> > 02 0128ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
>> > 03 0128ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
>> > 04 0128ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
>> > 05 0128ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
>> > 06 0128ffec 00000000 kernel32!BaseThreadStart+0x34
>>
>> > Thread ID: 9
>> > System Thread ID: fc0
>> > Kernel Time: 0:0:0.140
>> > User Time: 0:0:0.359
>> > Thread Type: HTTP Listener
>> > # ChildEBP RetAddr
>> > 00
>>
>> ...
>>
>> read more >>
>

Re: How the heck can you effectively debug these "process serving app

In this case, tools like procmon, filemon, and regmon are not useful
because there is a real crash happening in the worker process, not
really file/registry access issues.

You need to do *real* debugging to figure out what is wrong with your
web application. None of that contrived Visual Studio "Hit F5 and
debug" non-real-world stuff. What IIS State provides is but a taste of
what real, powerful debugging is like, and it requires a bit of
Windows systems knowledge to make sense of it all. Don't be afraid at
the log file; there is great benefit when one learns to understand
what the log file says.

Now, your log file actually indicates two different types of crash
patterns, telling me there's probably a couple of bad bugs within your
web application causing such frequent failures. The application pool
recycling is NOT the problem. Your crashing code is the problem.
Recycling is what brings the application back (sans in-process state)
and has nothing to do with your issue next to attempting to recover
from the crash.

Maybe the object you use to handle large file uploads is the problem.
What code is responsible for this functionality on your website? It
would not be the first time I see a file uploader have bugs.


//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//




On Nov 15, 2:10 pm, Shan Plourde wrote:
> Hi Spitfire, thank you for the suggestion! Your link pointed to
> Portmon, is that the same thing as Procmon? Have you had practical
> experience using it to diagnose and resolve this specific type of
> error?
>
> Thanks
> Shan
>
> On Nov 15, 4:34 pm, "spitfire" wrote:
>
>
>
> > Try
> > Procmon utility , If you haven't tried yet.http://www.microsoft.com/technet/sysinternals/ProcessesA ndThreads/Por...
>
> > "Shan Plourde" wrote in message
>
> >news:ec423693-9008-414c-a8c7-02d9b6ef1cde@d50g2000hsf.googl egroups.com...
>
> > > Hello there, in an attempt to retain the few dozen hairs remaining on
> > > the top of my head before I yank them all out in angst, I'm completely
> > > stuck....HEEEELP!
>
> > > We are always getting these errors in our System event log:
>
> > > A process serving application pool 'poolName' suffered a fatal
> > > communication error with the World Wide Web Publishing Service. The
> > > process id was '1234'.
>
> > > We have IIS6 on Windows Server 2003. These errors are all coming from
> > > our intranet website hosted on this server. The intranet website runs
> > > on .NET 2.0. We also have many other websites hosted on the server,
> > > most .NET, but some classic ASP, but this particular error only is
> > > reported for the intranet website. All other ASP.NET applications are
> > > hosted within their own application pools.
>
> > > The error happens A LOT (like, every few minutes). It's a huge issue
> > > because it seems to cause large file uploads through our Intranet to
> > > crap out when the issue happens (perhaps due to application pool
> > > recycling?).
>
> > > So to start a debugging session I ran:
> > > cscript \windows\system32\iisapp.vbs
>
> > > Which lists all app pools and processes. In this case at the time I
> > > ran iisapp.vbs, the application pool named "poolName" is hosted by
> > > process 9900. So I ran this:
>
> > > iisstate -p 9900 -hc -d
>
> > > In an attempt to log the next hard crash. When the hard crash happens,
> > > I just can't figure out how to make effective use of the log
> > > information. I've looked at several logs but none seem to make me go
> > > "AHA!". I understand that Microsoft recommends running this tool, but
> > > I don't think that I can interpret any of it...please, MSFT or anyone,
> > > your help is greatly appreciated to interpret the crash log and
> > > provide recommendations!!! What follows is the log, through to the end
> > > of this post....Thx!!!
>
> > > ----------------------------------------
>
> > > Opened log file 'E:\Program Files\IIS Resources\IISState\output
> > > \IISState.log'
>
> > > ***********************
> > > Starting new log output
> > > IISState version 3.0
>
> > > Thu Nov 15 14:46:28 2007
>
> > > OS = Windows 2003 Server
> > > Executable: w3wp.exe
> > > PID = 6976
>
> > > Note: Thread times are formatted as HH:MM:SS.ms
>
> > > ***********************
>
> > > IIS has crashed...
> > > Beginning Analysis
> > > DLL (!FunctionName) that failed:
>
> > > Thread ID: 30
> > > System Thread ID: 1a90
> > > Kernel Time: 0:0:0.375
> > > User Time: 0:0:2.140
> > > Thread Type: Other
> > > # ChildEBP RetAddr
> > > WARNING: Frame IP not in any known module. Following frames may be
> > > wrong.
> > > 00 1bf8f118 1db03f0e 0x1db03f5f
> > > 01 00000000 00000000 0x1db03f0e
> > > Closing open log file E:\Program Files\IIS Resources\IISState\output
> > > \IISState.log
> > > Opened log file 'E:\Program Files\IIS Resources\IISState\output
> > > \IISState.log'
>
> > > ***********************
> > > Starting new log output
> > > IISState version 3.0
>
> > > Thu Nov 15 15:59:00 2007
>
> > > OS = Windows 2003 Server
> > > Executable: w3wp.exe
> > > PID = 9900
>
> > > Note: Thread times are formatted as HH:MM:SS.ms
>
> > > ***********************
>
> > > IIS has crashed...
> > > Beginning Analysis
> > > DLL (!FunctionName) that failed: kernel32!RaiseException
>
> > > Thread ID: 27
> > > System Thread ID: 1ea0
> > > Kernel Time: 0:0:0.468
> > > User Time: 0:0:4.359
> > > Thread Type: Other
> > > # ChildEBP RetAddr
> > > 00 1acaf898 79f55b05 kernel32!RaiseException+0x53
> > > 01 1acaf8f8 7a056a79 mscorwks!RaiseTheExceptionInternalOnly+0x226
> > > 02 1acaf90c 7a056af2 mscorwks!RaiseTheException+0x4d
> > > 03 1acaf934 7a056b30 mscorwks!RaiseTheException+0xbf
> > > 04 1acaf960 7a056b41 mscorwks!RealCOMPlusThrow+0x38
> > > 05 1acaf96c 7a0e38fa mscorwks!RealCOMPlusThrow+0xb
> > > 06 1acafa9c 7a0e4b97 mscorwks!Thread::RaiseCrossContextException+0x3ac
> > > 07 1acafb50 7a0e5a07 mscorwks!Thread::DoADCallBack+0x26c
> > > 08 1acafb68 79ed89ca mscorwks!Thread::UserResumeThread+0xe1
> > > 09 1acafbfc 79ed88f1 mscorwks!Thread::DoADCallBack+0x355
> > > 0a 1acafc38 7a0e4d96 mscorwks!Thread::DoADCallBack+0x541
> > > 0b 1acafc60 7a0e4dc3 mscorwks!Thread::DoADCallBack+0x575
> > > 0c 1acafc74 7a077528 mscorwks!ManagedThreadBase::ThreadPool+0x13
> > > 0d 1acafcbc 7a077556 mscorwks!AddTimerCallbackEx+0x77
> > > 0e 1acafccc 7a1158ae mscorwks!AddTimerCallback+0xf
> > > 0f 1acafd08 79f8b7a5 mscorwks!
> > > ThreadpoolMgr::AsyncTimerCallbackCompletion+0x64
> > > 10 1acafd20 79f8b341 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0x40
> > > 11 1acafd94 79ed8e36 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x225
> > > 12 1acaffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
> > > 13 1acaffec 00000000 kernel32!BaseThreadStart+0x34
> > > Closing open log file E:\Program Files\IIS Resources\IISState\output
> > > \IISState.log
> > > Opened log file 'E:\Program Files\IIS Resources\IISState\output
> > > \IISState.log'
>
> > > ***********************
> > > Starting new log output
> > > IISState version 3.0
>
> > > Thu Nov 15 15:59:01 2007
>
> > > OS = Windows 2003 Server
> > > Executable: w3wp.exe
> > > PID = 9900
>
> > > Note: Thread times are formatted as HH:MM:SS.ms
>
> > > ***********************
>
> > > Thread ID: 0
> > > System Thread ID: 26d4
> > > Kernel Time: 0:0:0.78
> > > User Time: 0:0:0.171
> > > Thread Status: Thread is in a WAIT state.
> > > Thread Type: HTTP Compression Thread
> > > # ChildEBP RetAddr
> > > 00 0006fc08 7c827d0b ntdll!KiFastSystemCallRet
> > > 01 0006fc0c 77e61d1e ntdll!NtWaitForSingleObject+0xc
> > > 02 0006fc7c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> > > 03 0006fc90 5a364662 kernel32!WaitForSingleObject+0x12
> > > 04 0006fca0 5a366e3f w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
> > > 05 0006fca8 5a3af42d w3dt!UlAtqStartListen+0x2d
> > > 06 0006fcb8 5a3bc335 w3core!W3_SERVER::StartListen+0xbd
> > > 07 0006ff0c 0100187c w3core!UlW3Start+0x26e
> > > 08 0006ff44 01001a27 w3wp!wmain+0x22a
> > > 09 0006ffc0 77e6f23b w3wp!wmainCRTStartup+0x12f
> > > 0a 0006fff0 00000000 kernel32!BaseProcessStart+0x23
>
> > > Thread ID: 1
> > > System Thread ID: 1290
> > > Kernel Time: 0:0:0.0
> > > User Time: 0:0:0.0
> > > Thread Status: Thread is in a WAIT state.
> > > Thread Status: Thread is in a WAIT state.
> > > Thread Type: Other
> > > # ChildEBP RetAddr
> > > 00 008bfea0 7c827cfb ntdll!KiFastSystemCallRet
> > > 01 008bfea4 7c80e5bb ntdll!NtWaitForMultipleObjects+0xc
> > > 02 008bff48 7c80e4a2 ntdll!EtwpWaitForMultipleObjectsEx+0xf7
> > > 03 008bffb8 77e64829 ntdll!EtwpEventPump+0x27f
> > > 04 008bffec 00000000 kernel32!BaseThreadStart+0x34
>
> > > Thread ID: 2
> > > System Thread ID: 234c
> > > Kernel Time: 0:0:0.0
> > > User Time: 0:0:0.0
> > > Thread Type: Other
> > > # ChildEBP RetAddr
> > > 00 00f4ff9c 7c826f4b ntdll!KiFastSystemCallRet
> > > 01 00f4ffa0 7c83d424 ntdll!NtDelayExecution+0xc
> > > 02 00f4ffb8 77e64829 ntdll!RtlpTimerThread+0x47
> > > 03 00f4ffec 00000000 kernel32!BaseThreadStart+0x34
>
> > > Thread ID: 3
> > > System Thread ID: c6c
> > > Kernel Time: 0:0:0.0
> > > User Time: 0:0:0.0
> > > Thread Type: Other
> > > # ChildEBP RetAddr
> > > 00 00f8ff70 7c8277db ntdll!KiFastSystemCallRet
> > > 01 00f8ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
> > > 02 00f8ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
> > > 03 00f8ffec 00000000 kernel32!BaseThreadStart+0x34
>
> > > Thread ID: 4
> > > System Thread ID: 27ec
> > > Kernel Time: 0:0:0.15
> > > User Time: 0:0:0.0
> > > Thread Type: Possible ASP page. Possible DCOM activity
> > > Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
> > > Continuing with other analysis.
>
> > > No remote call being made
>
> > > # ChildEBP RetAddr
> > > 00 00fffe18 7c82783b ntdll!KiFastSystemCallRet
> > > 01 00fffe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
> > > 02 00ffff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
> > > 03 00ffff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
> > > 04 00ffffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
> > > 05 00ffffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
> > > 06 00ffffec 00000000 kernel32!BaseThreadStart+0x34
>
> > > Thread ID: 5
> > > System Thread ID: 5f8
> > > Kernel Time: 0:0:0.0
> > > User Time: 0:0:0.0
> > > Thread Type: Other
> > > # ChildEBP RetAddr
> > > 00 1e47ff70 7c8277db ntdll!KiFastSystemCallRet
> > > 01 1e47ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
> > > 02 1e47ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
> > > 03 1e47ffec 00000000 kernel32!BaseThreadStart+0x34
>
> > > Thread ID: 6
> > > System Thread ID: 1070
> > > Kernel Time: 0:0:0.93
> > > User Time: 0:0:0.265
> > > Thread Type: HTTP Listener
> > > # ChildEBP RetAddr
> > > 00 0120ff24 7c8277db ntdll!KiFastSystemCallRet
> > > 01 0120ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> > > 02 0120ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
> > > 03 0120ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
> > > 04 0120ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
> > > 05 0120ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
> > > 06 0120ffec 00000000 kernel32!BaseThreadStart+0x34
>
> > > Thread ID: 7
> > > System Thread ID: 2734
> > > Kernel Time: 0:0:0.125
> > > User Time: 0:0:0.171
> > > Thread Type: HTTP Listener
> > > # ChildEBP RetAddr
> > > 00 0124ff24 7c8277db ntdll!KiFastSystemCallRet
> > > 01 0124ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> > > 02 0124ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
> > > 03 0124ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
> > > 04 0124ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
> > > 05 0124ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
> > > 06 0124ffec 00000000 kernel32!BaseThreadStart+0x34
>
> > > Thread ID: 8
> > > System Thread ID: bb4
> > > Kernel Time:
>
> ...
>
> read more >>- Hide quoted text -
>
> - Show quoted text -

Re: How the heck can you effectively debug these "process serving app

Hi David, thank you for your insights, I absolutely believe that it is
an application specific issue causing the application pool to crash,
but I am unable to determine which part of the application is causing
the issue. I am very much interested in knowing how to interpret the
log file: have you determined from the log file that it is the
application's file upload functionality? If so, how were you able to
determine that?

This is an intranet application and recently I have made some changes
to support large file uploads on it. At this time there is a "multiple
file uploads" page that supports up to 5 uploads, and each upload
could be in the vicinity of several hundred KB to 30MB in size. I've
configured the standard httpRuntime configuration elements for the
application to allow larger uploads and to allow larger execution
times. The processing of the uploading on the streams on the server
side essentially goes from a .NET user control code behind to various
business logic and data model layer classes that persist the up to 5
streams.

I look forward to your responses to my questions, and thank you very
much for your help! In the meantime I will review the file uploading
logic and accompanying documentation for the APIs that I'm using to
see if there is anything that looks incorrect to me, and share those
findings as well.

Thanks again
Shan


On Nov 16, 7:43 am, David Wang wrote:
> In this case, tools like procmon, filemon, and regmon are not useful
> because there is a real crash happening in the worker process, not
> really file/registry access issues.
>
> You need to do *real* debugging to figure out what is wrong with your
> web application. None of that contrived Visual Studio "Hit F5 and
> debug" non-real-world stuff. What IIS State provides is but a taste of
> what real, powerful debugging is like, and it requires a bit of
> Windows systems knowledge to make sense of it all. Don't be afraid at
> the log file; there is great benefit when one learns to understand
> what the log file says.
>
> Now, your log file actually indicates two different types of crash
> patterns, telling me there's probably a couple of bad bugs within your
> web application causing such frequent failures. The application pool
> recycling is NOT the problem. Your crashing code is the problem.
> Recycling is what brings the application back (sans in-process state)
> and has nothing to do with your issue next to attempting to recover
> from the crash.
>
> Maybe the object you use to handle large file uploads is the problem.
> What code is responsible for this functionality on your website? It
> would not be the first time I see a file uploader have bugs.
>
> //Davidhttp://w3-4u.blogspot.comhttp://blogs.msdn.com/David. Wang
> //
>
> On Nov 15, 2:10 pm, Shan Plourde wrote:
>
> > Hi Spitfire, thank you for the suggestion! Your link pointed to
> > Portmon, is that the same thing as Procmon? Have you had practical
> > experience using it to diagnose and resolve this specific type of
> > error?
>
> > Thanks
> > Shan
>
> > On Nov 15, 4:34 pm, "spitfire" wrote:
>
> > > Try
> > > Procmon utility , If you haven't tried yet.http://www.microsoft.com/technet/sysinternals/ProcessesA ndThreads/Por...
>
> > > "Shan Plourde" wrote in message
>
> > >news:ec423693-9008-414c-a8c7-02d9b6ef1cde@d50g2000hsf.googl egroups.com...
>
> > > > Hello there, in an attempt to retain the few dozen hairs remaining on
> > > > the top of my head before I yank them all out in angst, I'm completely
> > > > stuck....HEEEELP!
>
> > > > We are always getting these errors in our System event log:
>
> > > > A process serving application pool 'poolName' suffered a fatal
> > > > communication error with the World Wide Web Publishing Service. The
> > > > process id was '1234'.
>
> > > > We have IIS6 on Windows Server 2003. These errors are all coming from
> > > > our intranet website hosted on this server. The intranet website runs
> > > > on .NET 2.0. We also have many other websites hosted on the server,
> > > > most .NET, but some classic ASP, but this particular error only is
> > > > reported for the intranet website. All other ASP.NET applications are
> > > > hosted within their own application pools.
>
> > > > The error happens A LOT (like, every few minutes). It's a huge issue
> > > > because it seems to cause large file uploads through our Intranet to
> > > > crap out when the issue happens (perhaps due to application pool
> > > > recycling?).
>
> > > > So to start a debugging session I ran:
> > > > cscript \windows\system32\iisapp.vbs
>
> > > > Which lists all app pools and processes. In this case at the time I
> > > > ran iisapp.vbs, the application pool named "poolName" is hosted by
> > > > process 9900. So I ran this:
>
> > > > iisstate -p 9900 -hc -d
>
> > > > In an attempt to log the next hard crash. When the hard crash happens,
> > > > I just can't figure out how to make effective use of the log
> > > > information. I've looked at several logs but none seem to make me go
> > > > "AHA!". I understand that Microsoft recommends running this tool, but
> > > > I don't think that I can interpret any of it...please, MSFT or anyone,
> > > > your help is greatly appreciated to interpret the crash log and
> > > > provide recommendations!!! What follows is the log, through to the end
> > > > of this post....Thx!!!
>
> > > > ----------------------------------------
>
> > > > Opened log file 'E:\Program Files\IIS Resources\IISState\output
> > > > \IISState.log'
>
> > > > ***********************
> > > > Starting new log output
> > > > IISState version 3.0
>
> > > > Thu Nov 15 14:46:28 2007
>
> > > > OS = Windows 2003 Server
> > > > Executable: w3wp.exe
> > > > PID = 6976
>
> > > > Note: Thread times are formatted as HH:MM:SS.ms
>
> > > > ***********************
>
> > > > IIS has crashed...
> > > > Beginning Analysis
> > > > DLL (!FunctionName) that failed:
>
> > > > Thread ID: 30
> > > > System Thread ID: 1a90
> > > > Kernel Time: 0:0:0.375
> > > > User Time: 0:0:2.140
> > > > Thread Type: Other
> > > > # ChildEBP RetAddr
> > > > WARNING: Frame IP not in any known module. Following frames may be
> > > > wrong.
> > > > 00 1bf8f118 1db03f0e 0x1db03f5f
> > > > 01 00000000 00000000 0x1db03f0e
> > > > Closing open log file E:\Program Files\IIS Resources\IISState\output
> > > > \IISState.log
> > > > Opened log file 'E:\Program Files\IIS Resources\IISState\output
> > > > \IISState.log'
>
> > > > ***********************
> > > > Starting new log output
> > > > IISState version 3.0
>
> > > > Thu Nov 15 15:59:00 2007
>
> > > > OS = Windows 2003 Server
> > > > Executable: w3wp.exe
> > > > PID = 9900
>
> > > > Note: Thread times are formatted as HH:MM:SS.ms
>
> > > > ***********************
>
> > > > IIS has crashed...
> > > > Beginning Analysis
> > > > DLL (!FunctionName) that failed: kernel32!RaiseException
>
> > > > Thread ID: 27
> > > > System Thread ID: 1ea0
> > > > Kernel Time: 0:0:0.468
> > > > User Time: 0:0:4.359
> > > > Thread Type: Other
> > > > # ChildEBP RetAddr
> > > > 00 1acaf898 79f55b05 kernel32!RaiseException+0x53
> > > > 01 1acaf8f8 7a056a79 mscorwks!RaiseTheExceptionInternalOnly+0x226
> > > > 02 1acaf90c 7a056af2 mscorwks!RaiseTheException+0x4d
> > > > 03 1acaf934 7a056b30 mscorwks!RaiseTheException+0xbf
> > > > 04 1acaf960 7a056b41 mscorwks!RealCOMPlusThrow+0x38
> > > > 05 1acaf96c 7a0e38fa mscorwks!RealCOMPlusThrow+0xb
> > > > 06 1acafa9c 7a0e4b97 mscorwks!Thread::RaiseCrossContextException+0x3ac
> > > > 07 1acafb50 7a0e5a07 mscorwks!Thread::DoADCallBack+0x26c
> > > > 08 1acafb68 79ed89ca mscorwks!Thread::UserResumeThread+0xe1
> > > > 09 1acafbfc 79ed88f1 mscorwks!Thread::DoADCallBack+0x355
> > > > 0a 1acafc38 7a0e4d96 mscorwks!Thread::DoADCallBack+0x541
> > > > 0b 1acafc60 7a0e4dc3 mscorwks!Thread::DoADCallBack+0x575
> > > > 0c 1acafc74 7a077528 mscorwks!ManagedThreadBase::ThreadPool+0x13
> > > > 0d 1acafcbc 7a077556 mscorwks!AddTimerCallbackEx+0x77
> > > > 0e 1acafccc 7a1158ae mscorwks!AddTimerCallback+0xf
> > > > 0f 1acafd08 79f8b7a5 mscorwks!
> > > > ThreadpoolMgr::AsyncTimerCallbackCompletion+0x64
> > > > 10 1acafd20 79f8b341 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0x40
> > > > 11 1acafd94 79ed8e36 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x225
> > > > 12 1acaffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
> > > > 13 1acaffec 00000000 kernel32!BaseThreadStart+0x34
> > > > Closing open log file E:\Program Files\IIS Resources\IISState\output
> > > > \IISState.log
> > > > Opened log file 'E:\Program Files\IIS Resources\IISState\output
> > > > \IISState.log'
>
> > > > ***********************
> > > > Starting new log output
> > > > IISState version 3.0
>
> > > > Thu Nov 15 15:59:01 2007
>
> > > > OS = Windows 2003 Server
> > > > Executable: w3wp.exe
> > > > PID = 9900
>
> > > > Note: Thread times are formatted as HH:MM:SS.ms
>
> > > > ***********************
>
> > > > Thread ID: 0
> > > > System Thread ID: 26d4
> > > > Kernel Time: 0:0:0.78
> > > > User Time: 0:0:0.171
> > > > Thread Status: Thread is in a WAIT state.
> > > > Thread Type: HTTP Compression Thread
> > > > # ChildEBP RetAddr
> > > > 00 0006fc08 7c827d0b ntdll!KiFastSystemCallRet
> > > > 01 0006fc0c 77e61d1e ntdll!NtWaitForSingleObject+0xc
> > > > 02 0006fc7c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> > > > 03 0006fc90 5a364662 kernel32!WaitForSingleObject+0x12
> > > > 04 0006fca0 5a366e3f w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
> > > > 05 0006fca8 5a3af42d w3dt!UlAtqStartListen+0x2d
> > > > 06 0006fcb8 5a3bc335 w3core!W3_SERVER::StartListen+0xbd
> > > > 07 0006ff0c 0100187c w3core!UlW3Start+0x26e
> > > > 08 0006ff44 01001a27 w3wp!wmain+0x22a
> > > > 09 0006ffc0 77e6f23b w3wp!wmainCRTStartup+0x12f
> > > > 0a 0006fff0 00000000 kernel32!BaseProcessStart+0x23
>
> > > > Thread ID: 1
> > > > System Thread ID: 1290
> > > > Kernel Time: 0:0:0.0
> > > > User Time: 0:0:0.0
> > > > Thread Status: Thread is in a WAIT state.
> > > > Thread Status: Thread is in a WAIT state.
> > > > Thread Type: Other
> > > > # ChildEBP RetAddr
> > > > 00 008bfea0 7c827cfb ntdll!KiFastSystemCallRet
> > > > 01 008bfea4 7c80e5bb ntdll!NtWaitForMultipleObjects+0xc
> > > > 02 008bff48 7c80e4a2 ntdll!EtwpWaitForMultipleObjectsEx+0xf7
> > > > 03 008bffb8 77e64829 ntdll!EtwpEventPump+0x27f
> > > > 04 008bffec 00000000 kernel32!BaseThreadStart+0x34
>
> > > > Thread ID: 2
> > > > System Thread ID: 234c
> > > > Kernel Time: 0:0:0.0
> > > > User Time: 0:0:0.0
> > > > Thread Type: Other
> > > > # ChildEBP RetAddr
> > > > 00 00f4ff9c 7c826f4b ntdll!KiFastSystemCallRet
> > > > 01 00f4ffa0 7c83d424 ntdll!NtDelayExecution+0xc
> > > > 02 00f4ffb8 77e64829 ntdll!RtlpTimerThread+0x47
> > > > 03 00f4ffec 00000000 kernel32!BaseThreadStart+0x34
>
> > > > Thread ID: 3
> > > > System Thread ID: c6c
> > > > Kernel Time: 0:0:0.0
> > > > User Time: 0:0:0.0
> > > > Thread Type: Other
> > > > # ChildEBP RetAddr
> > > > 00 00f8ff70 7c8277db ntdll!KiFastSystemCallRet
> > > > 01 00f8ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
> > > > 02 00f8ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
> > > > 03 00f8ffec 00000000 kernel32!BaseThreadStart+0x34
>
> > > > Thread ID: 4
> > > > System Thread ID: 27ec
> > > > Kernel Time: 0:0:0.15
> > > > User Time: 0:0:0.0
> > > > Thread Type: Possible ASP page. Possible DCOM activity
> > > > Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
> > > > Continuing with other analysis.
>
> > > > No remote call being made
>
> > > > # ChildEBP RetAddr
> > > > 00 00fffe18 7c82783b ntdll!KiFastSystemCallRet
> > > > 01 00fffe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
> > > > 02 00ffff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
> > > > 03 00ffff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
> > > > 04 00ffffac
>
> ...
>
> read more >>

Re: How the heck can you effectively debug these "process serving app

David, the other thing that I'd like to understand is what type of
application exceptions could cause an app pool to crash without the
exceptions bubbling up to the top of the application? Our exception
handling strategy for our applications is to simply let exceptions
bubble up to the top of the stack trace, and then email exceptions and
log them appropriately. Obviously when an app pool crashes it can't be
due to a simple exception like this.

I would assume that app pool crashes would either be things such as
infinite loops, stack overflows, thread contentions / deadlock
situations....appreciate hearing your insights on this as well!

Thanks
Shan

On Nov 16, 9:30 am, Shan Plourde wrote:
> Hi David, thank you for your insights, I absolutely believe that it is
> an application specific issue causing the application pool to crash,
> but I am unable to determine which part of the application is causing
> the issue. I am very much interested in knowing how to interpret the
> log file: have you determined from the log file that it is the
> application's file upload functionality? If so, how were you able to
> determine that?
>
> This is an intranet application and recently I have made some changes
> to support large file uploads on it. At this time there is a "multiple
> file uploads" page that supports up to 5 uploads, and each upload
> could be in the vicinity of several hundred KB to 30MB in size. I've
> configured the standard httpRuntime configuration elements for the
> application to allow larger uploads and to allow larger execution
> times. The processing of the uploading on the streams on the server
> side essentially goes from a .NET user control code behind to various
> business logic and data model layer classes that persist the up to 5
> streams.
>
> I look forward to your responses to my questions, and thank you very
> much for your help! In the meantime I will review the file uploading
> logic and accompanying documentation for the APIs that I'm using to
> see if there is anything that looks incorrect to me, and share those
> findings as well.
>
> Thanks again
> Shan
>
> On Nov 16, 7:43 am, David Wang wrote:
>
> > In this case, tools like procmon, filemon, and regmon are not useful
> > because there is a real crash happening in the worker process, not
> > really file/registry access issues.
>
> > You need to do *real* debugging to figure out what is wrong with your
> > web application. None of that contrived Visual Studio "Hit F5 and
> > debug" non-real-world stuff. What IIS State provides is but a taste of
> > what real, powerful debugging is like, and it requires a bit of
> > Windows systems knowledge to make sense of it all. Don't be afraid at
> > the log file; there is great benefit when one learns to understand
> > what the log file says.
>
> > Now, your log file actually indicates two different types of crash
> > patterns, telling me there's probably a couple of bad bugs within your
> > web application causing such frequent failures. The application pool
> > recycling is NOT the problem. Your crashing code is the problem.
> > Recycling is what brings the application back (sans in-process state)
> > and has nothing to do with your issue next to attempting to recover
> > from the crash.
>
> > Maybe the object you use to handle large file uploads is the problem.
> > What code is responsible for this functionality on your website? It
> > would not be the first time I see a file uploader have bugs.
>
> > //Davidhttp://w3-4u.blogspot.comhttp://blogs.msdn.com/David. Wang
> > //
>
> > On Nov 15, 2:10 pm, Shan Plourde wrote:
>
> > > Hi Spitfire, thank you for the suggestion! Your link pointed to
> > > Portmon, is that the same thing as Procmon? Have you had practical
> > > experience using it to diagnose and resolve this specific type of
> > > error?
>
> > > Thanks
> > > Shan
>
> > > On Nov 15, 4:34 pm, "spitfire" wrote:
>
> > > > Try
> > > > Procmon utility , If you haven't tried yet.http://www.microsoft.com/technet/sysinternals/ProcessesA ndThreads/Por...
>
> > > > "Shan Plourde" wrote in message
>
> > > >news:ec423693-9008-414c-a8c7-02d9b6ef1cde@d50g2000hsf.googl egroups.com...
>
> > > > > Hello there, in an attempt to retain the few dozen hairs remaining on
> > > > > the top of my head before I yank them all out in angst, I'm completely
> > > > > stuck....HEEEELP!
>
> > > > > We are always getting these errors in our System event log:
>
> > > > > A process serving application pool 'poolName' suffered a fatal
> > > > > communication error with the World Wide Web Publishing Service. The
> > > > > process id was '1234'.
>
> > > > > We have IIS6 on Windows Server 2003. These errors are all coming from
> > > > > our intranet website hosted on this server. The intranet website runs
> > > > > on .NET 2.0. We also have many other websites hosted on the server,
> > > > > most .NET, but some classic ASP, but this particular error only is
> > > > > reported for the intranet website. All other ASP.NET applications are
> > > > > hosted within their own application pools.
>
> > > > > The error happens A LOT (like, every few minutes). It's a huge issue
> > > > > because it seems to cause large file uploads through our Intranet to
> > > > > crap out when the issue happens (perhaps due to application pool
> > > > > recycling?).
>
> > > > > So to start a debugging session I ran:
> > > > > cscript \windows\system32\iisapp.vbs
>
> > > > > Which lists all app pools and processes. In this case at the time I
> > > > > ran iisapp.vbs, the application pool named "poolName" is hosted by
> > > > > process 9900. So I ran this:
>
> > > > > iisstate -p 9900 -hc -d
>
> > > > > In an attempt to log the next hard crash. When the hard crash happens,
> > > > > I just can't figure out how to make effective use of the log
> > > > > information. I've looked at several logs but none seem to make me go
> > > > > "AHA!". I understand that Microsoft recommends running this tool, but
> > > > > I don't think that I can interpret any of it...please, MSFT or anyone,
> > > > > your help is greatly appreciated to interpret the crash log and
> > > > > provide recommendations!!! What follows is the log, through to the end
> > > > > of this post....Thx!!!
>
> > > > > ----------------------------------------
>

Re: How the heck can you effectively debug these "process serving app

Hi David,
I reviewed the file upload logic. Essentially, an array of up to 5
HttpPostedFile instances are passed down the stack trace until they
are eventually saved. The way they are saved though is not using
HttpPostedFile.SaveAs() but rather the input stream for an upload is
acquired first via HttpPostedFile.InputStream. The input stream is
passed to another method in another assembly in the GAC. That save
method saves the input stream as follows. I can't remember what the
method behind the madness is for accesing the input stream versus
simply using HttpPostedFile.SaveAs() method. But could this be the
culprit? I may try temporarily using .SaveAs() and review this method
further.

Appreciate your insights!
Shan
-------------
FileStream fsOut = new FileStream(filePath,
FileMode.Create);

byte[] bytesToWrite = new byte[32768];

while (true)
{
int read = fileStream.Read(bytesToWrite, 0,
bytesToWrite.Length);
if (read <= 0)
{
break;
}
fsOut.Write(bytesToWrite, 0, read);
}

fsOut.Close();
fileStream.Close();
--------------------




On Nov 16, 7:43 am, David Wang wrote:
> In this case, tools like procmon, filemon, and regmon are not useful
> because there is a real crash happening in the worker process, not
> really file/registry access issues.
>
> You need to do *real* debugging to figure out what is wrong with your
> web application. None of that contrived Visual Studio "Hit F5 and
> debug" non-real-world stuff. What IIS State provides is but a taste of
> what real, powerful debugging is like, and it requires a bit of
> Windows systems knowledge to make sense of it all. Don't be afraid at
> the log file; there is great benefit when one learns to understand
> what the log file says.
>
> Now, your log file actually indicates two different types of crash
> patterns, telling me there's probably a couple of bad bugs within your
> web application causing such frequent failures. The application pool
> recycling is NOT the problem. Your crashing code is the problem.
> Recycling is what brings the application back (sans in-process state)
> and has nothing to do with your issue next to attempting to recover
> from the crash.
>
> Maybe the object you use to handle large file uploads is the problem.
> What code is responsible for this functionality on your website? It
> would not be the first time I see a file uploader have bugs.
>
> //Davidhttp://w3-4u.blogspot.comhttp://blogs.msdn.com/David. Wang
> //
>
> On Nov 15, 2:10 pm, Shan Plourde wrote:> Hi Spitfire, thank you for the suggestion! Your link pointed to
> > Portmon, is that the same thing as Procmon? Have you had practical
> > experience using it to diagnose and resolve this specific type of
> > error?
>
> > Thanks
> > Shan
>
> > On Nov 15, 4:34 pm, "spitfire" wrote:
>
> > > Try
> > > Procmon utility , If you haven't tried yet.http://www.microsoft.com/technet/sysinternals/ProcessesA ndThreads/Por...
>
> > > "Shan Plourde" wrote in message
>
> > >news:ec423693-9008-414c-a8c7-02d9b6ef1cde@d50g2000hsf.googl egroups.com...
>
> > > > Hello there, in an attempt to retain the few dozen hairs remaining on
> > > > the top of my head before I yank them all out in angst, I'm completely
> > > > stuck....HEEEELP!
>
> > > > We are always getting these errors in our System event log:
>
> > > > A process serving application pool 'poolName' suffered a fatal
> > > > communication error with the World Wide Web Publishing Service. The
> > > > process id was '1234'.
>
> > > > We have IIS6 on Windows Server 2003. These errors are all coming from
> > > > our intranet website hosted on this server. The intranet website runs
> > > > on .NET 2.0. We also have many other websites hosted on the server,
> > > > most .NET, but some classic ASP, but this particular error only is
> > > > reported for the intranet website. All other ASP.NET applications are
> > > > hosted within their own application pools.
>
> > > > The error happens A LOT (like, every few minutes). It's a huge issue
> > > > because it seems to cause large file uploads through our Intranet to
> > > > crap out when the issue happens (perhaps due to application pool
> > > > recycling?).
>
> > > > So to start a debugging session I ran:
> > > > cscript \windows\system32\iisapp.vbs
>
> > > > Which lists all app pools and processes. In this case at the time I
> > > > ran iisapp.vbs, the application pool named "poolName" is hosted by
> > > > process 9900. So I ran this:
>
> > > > iisstate -p 9900 -hc -d
>
> > > > In an attempt to log the next hard crash. When the hard crash happens,
> > > > I just can't figure out how to make effective use of the log
> > > > information. I've looked at several logs but none seem to make me go
> > > > "AHA!". I understand that Microsoft recommends running this tool, but
> > > > I don't think that I can interpret any of it...please, MSFT or anyone,
> > > > your help is greatly appreciated to interpret the crash log and
> > > > provide recommendations!!! What follows is the log, through to the end
> > > > of this post....Thx!!!
>
> > > > ----------------------------------------
>
> > > > Opened log file 'E:\Program Files\IIS Resources\IISState\output
> > > > \IISState.log'
>
> > > > ***********************
> > > > Starting new log output
> > > > IISState version 3.0
>
> > > > Thu Nov 15 14:46:28 2007
>
> > > > OS = Windows 2003 Server
> > > > Executable: w3wp.exe
> > > > PID = 6976
>
> > > > Note: Thread times are formatted as HH:MM:SS.ms
>
> > > > ***********************
>
> > > > IIS has crashed...
> > > > Beginning Analysis
> > > > DLL (!FunctionName) that failed:
>
> > > > Thread ID: 30
> > > > System Thread ID: 1a90
> > > > Kernel Time: 0:0:0.375
> > > > User Time: 0:0:2.140
> > > > Thread Type: Other
> > > > # ChildEBP RetAddr
> > > > WARNING: Frame IP not in any known module. Following frames may be
> > > > wrong.
> > > > 00 1bf8f118 1db03f0e 0x1db03f5f
> > > > 01 00000000 00000000 0x1db03f0e
> > > > Closing open log file E:\Program Files\IIS Resources\IISState\output
> > > > \IISState.log
> > > > Opened log file 'E:\Program Files\IIS Resources\IISState\output
> > > > \IISState.log'
>
> > > > ***********************
> > > > Starting new log output
> > > > IISState version 3.0
>
> > > > Thu Nov 15 15:59:00 2007
>
> > > > OS = Windows 2003 Server
> > > > Executable: w3wp.exe
> > > > PID = 9900
>
> > > > Note: Thread times are formatted as HH:MM:SS.ms
>
> > > > ***********************
>
> > > > IIS has crashed...
> > > > Beginning Analysis
> > > > DLL (!FunctionName) that failed: kernel32!RaiseException
>
> > > > Thread ID: 27
> > > > System Thread ID: 1ea0
> > > > Kernel Time: 0:0:0.468
> > > > User Time: 0:0:4.359
> > > > Thread Type: Other
> > > > # ChildEBP RetAddr
> > > > 00 1acaf898 79f55b05 kernel32!RaiseException+0x53
> > > > 01 1acaf8f8 7a056a79 mscorwks!RaiseTheExceptionInternalOnly+0x226
> > > > 02 1acaf90c 7a056af2 mscorwks!RaiseTheException+0x4d
> > > > 03 1acaf934 7a056b30 mscorwks!RaiseTheException+0xbf
> > > > 04 1acaf960 7a056b41 mscorwks!RealCOMPlusThrow+0x38
> > > > 05 1acaf96c 7a0e38fa mscorwks!RealCOMPlusThrow+0xb
> > > > 06 1acafa9c 7a0e4b97 mscorwks!Thread::RaiseCrossContextException+0x3ac
> > > > 07 1acafb50 7a0e5a07 mscorwks!Thread::DoADCallBack+0x26c
> > > > 08 1acafb68 79ed89ca mscorwks!Thread::UserResumeThread+0xe1
> > > > 09 1acafbfc 79ed88f1 mscorwks!Thread::DoADCallBack+0x355
> > > > 0a 1acafc38 7a0e4d96 mscorwks!Thread::DoADCallBack+0x541
> > > > 0b 1acafc60 7a0e4dc3 mscorwks!Thread::DoADCallBack+0x575
> > > > 0c 1acafc74 7a077528 mscorwks!ManagedThreadBase::ThreadPool+0x13
> > > > 0d 1acafcbc 7a077556 mscorwks!AddTimerCallbackEx+0x77
> > > > 0e 1acafccc 7a1158ae mscorwks!AddTimerCallback+0xf
> > > > 0f 1acafd08 79f8b7a5 mscorwks!
> > > > ThreadpoolMgr::AsyncTimerCallbackCompletion+0x64
> > > > 10 1acafd20 79f8b341 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0x40
> > > > 11 1acafd94 79ed8e36 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x225
> > > > 12 1acaffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
> > > > 13 1acaffec 00000000 kernel32!BaseThreadStart+0x34
> > > > Closing open log file E:\Program Files\IIS Resources\IISState\output
> > > > \IISState.log
> > > > Opened log file 'E:\Program Files\IIS Resources\IISState\output
> > > > \IISState.log'
>
> > > > ***********************
> > > > Starting new log output
> > > > IISState version 3.0
>
> > > > Thu Nov 15 15:59:01 2007
>
> > > > OS = Windows 2003 Server
> > > > Executable: w3wp.exe
> > > > PID = 9900
>
> > > > Note: Thread times are formatted as HH:MM:SS.ms
>
> > > > ***********************
>
> > > > Thread ID: 0
> > > > System Thread ID: 26d4
> > > > Kernel Time: 0:0:0.78
> > > > User Time: 0:0:0.171
> > > > Thread Status: Thread is in a WAIT state.
> > > > Thread Type: HTTP Compression Thread
> > > > # ChildEBP RetAddr
> > > > 00 0006fc08 7c827d0b ntdll!KiFastSystemCallRet
> > > > 01 0006fc0c 77e61d1e ntdll!NtWaitForSingleObject+0xc
> > > > 02 0006fc7c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> > > > 03 0006fc90 5a364662 kernel32!WaitForSingleObject+0x12
> > > > 04 0006fca0 5a366e3f w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
> > > > 05 0006fca8 5a3af42d w3dt!UlAtqStartListen+0x2d
> > > > 06 0006fcb8 5a3bc335 w3core!W3_SERVER::StartListen+0xbd
> > > > 07 0006ff0c 0100187c w3core!UlW3Start+0x26e
> > > > 08 0006ff44 01001a27 w3wp!wmain+0x22a
> > > > 09 0006ffc0 77e6f23b w3wp!wmainCRTStartup+0x12f
> > > > 0a 0006fff0 00000000 kernel32!BaseProcessStart+0x23
>
> > > > Thread ID: 1
> > > > System Thread ID: 1290
> > > > Kernel Time: 0:0:0.0
> > > > User Time: 0:0:0.0
> > > > Thread Status: Thread is in a WAIT state.
> > > > Thread Status: Thread is in a WAIT state.
> > > > Thread Type: Other
> > > > # ChildEBP RetAddr
> > > > 00 008bfea0 7c827cfb ntdll!KiFastSystemCallRet
> > > > 01 008bfea4 7c80e5bb ntdll!NtWaitForMultipleObjects+0xc
> > > > 02 008bff48 7c80e4a2 ntdll!EtwpWaitForMultipleObjectsEx+0xf7
> > > > 03 008bffb8 77e64829 ntdll!EtwpEventPump+0x27f
> > > > 04 008bffec 00000000 kernel32!BaseThreadStart+0x34
>
> > > > Thread ID: 2
> > > > System Thread ID: 234c
> > > > Kernel Time: 0:0:0.0
> > > > User Time: 0:0:0.0
> > > > Thread Type: Other
> > > > # ChildEBP RetAddr
> > > > 00 00f4ff9c 7c826f4b ntdll!KiFastSystemCallRet
> > > > 01 00f4ffa0 7c83d424 ntdll!NtDelayExecution+0xc
> > > > 02 00f4ffb8 77e64829 ntdll!RtlpTimerThread+0x47
> > > > 03 00f4ffec 00000000 kernel32!BaseThreadStart+0x34
>
> > > > Thread ID: 3
> > > > System Thread ID: c6c
> > > > Kernel Time: 0:0:0.0
> > > > User Time: 0:0:0.0
> > > > Thread Type: Other
> > > > # ChildEBP RetAddr
> > > > 00 00f8ff70 7c8277db ntdll!KiFastSystemCallRet
> > > > 01 00f8ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
> > > > 02 00f8ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
> > > > 03 00f8ffec 00000000 kernel32!BaseThreadStart+0x34
>
> > > > Thread ID: 4
> > > > System Thread ID: 27ec
> > > > Kernel Time: 0:0:0.15
> > > > User Time: 0:0:0.0
> > > > Thread Type: Possible ASP page. Possible DCOM activity
> > > > Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
> > > > Continuing with other analysis.
>
> > > > No remote call being made
>
> > > > # ChildEBP RetAddr
> > > > 00 00fffe18 7c82783b ntdll!KiFastSystemCallRet
> > > > 01 00fffe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
> > > > 02 00ffff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
> > > > 03 00ffff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
> > > > 04 00ffffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
> > > > 05 00ffffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
> > > > 06 00ffffec 00000000 kernel32!BaseThreadStart+0x34
>
> > > > Thread ID: 5
> > > > System Thread ID: 5f8
> > > > Kernel Time: 0:0:0.0
> > > > User Time: 0:0:0.0
> > > > Thread Type: Other
> > > > # ChildEBP RetAddr
> > > > 00 1e47ff70 7c8277db ntdll!KiFastSystemCallRet
> > > > 01 1e47ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
> > > > 02 1e47ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
> > > > 03 1e47ffec 00000000 kernel32!BaseThreadStart+0x34
>
> > > > Thread ID: 6
> > > > System Thread ID: 1070
> > > > Kernel Time: 0:0:0.93
> > > > User Time: 0:0:0.265
> > > > Thread Type: HTTP Listener
> > > > # ChildEBP RetAddr
> > > > 00 0120ff24 7c8277db ntdll!KiFastSystemCallRet
> > > > 01 0120ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> > > > 02 0120ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
> > > > 03 0120ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
> > > > 04 0120ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
> > > > 05 0120ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
> > > > 06 0120ffec 00000000 kernel32!BaseThreadStart+0x34
>
> > > > Thread ID: 7
> > > > System Thread ID: 2734
> > > > Kernel Time: 0:0:0.125
> > > > User Time: 0:0:0.171
> > > > Thread Type: HTTP Listener
> > > > # ChildEBP RetAddr
> > > > 00 0124ff24 7c8277db ntdll!KiFastSystemCallRet
> > > > 01 0124ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
> > > > 02 0124ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
> > > > 03 0124ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
> > > > 04 0124ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
> > > > 05 0124ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
> > > > 06 0124ffec 00000000 kernel32!BaseThreadStart+0x34
>
> > > > Thread ID: 8
> > > > System Thread ID: bb4
> > > > Kernel Time:
>
> > ...
>
> > read more >>- Hide quoted text -
>
> > - Show quoted text -

Re: How the heck can you effectively debug these "process serving app

David or others from MSFT...I am really at a dead end here and need
some help...any help would be much appreciated!
Thx
Shan


On Nov 16, 10:42 am, Shan Plourde wrote:
> Hi David,
> I reviewed the file upload logic. Essentially, an array of up to 5
> HttpPostedFile instances are passed down the stack trace until they
> are eventually saved. The way they are saved though is not using
> HttpPostedFile.SaveAs() but rather the input stream for an upload is
> acquired first via HttpPostedFile.InputStream. The input stream is
> passed to another method in another assembly in the GAC. That save
> method saves the input stream as follows. I can't remember what the
> method behind the madness is for accesing the input stream versus
> simply using HttpPostedFile.SaveAs() method. But could this be the
> culprit? I may try temporarily using .SaveAs() and review this method
> further.
>
> Appreciate your insights!
> Shan
> -------------
> FileStream fsOut = new FileStream(filePath,
> FileMode.Create);
>
> byte[] bytesToWrite = new byte[32768];
>
> while (true)
> {
> int read = fileStream.Read(bytesToWrite, 0,
> bytesToWrite.Length);
> if (read <= 0)
> {
> break;
> }
> fsOut.Write(bytesToWrite, 0, read);
> }
>
> fsOut.Close();
> fileStream.Close();
> --------------------
>
> On Nov 16, 7:43 am, David Wang wrote:
>
> > In this case, tools like procmon, filemon, and regmon are not useful
> > because there is a real crash happening in the worker process, not
> > really file/registry access issues.
>
> > You need to do *real* debugging to figure out what is wrong with your
> > web application. None of that contrived Visual Studio "Hit F5 and
> > debug" non-real-world stuff. What IIS State provides is but a taste of
> > what real, powerful debugging is like, and it requires a bit of
> > Windows systems knowledge to make sense of it all. Don't be afraid at
> > the log file; there is great benefit when one learns to understand
> > what the log file says.
>
> > Now, your log file actually indicates two different types of crash
> > patterns, telling me there's probably a couple of bad bugs within your
> > web application causing such frequent failures. The application pool
> > recycling is NOT the problem. Your crashing code is the problem.
> > Recycling is what brings the application back (sans in-process state)
> > and has nothing to do with your issue next to attempting to recover
> > from the crash.
>
> > Maybe the object you use to handle large file uploads is the problem.
> > What code is responsible for this functionality on your website? It
> > would not be the first time I see a file uploader have bugs.
>
> > //Davidhttp://w3-4u.blogspot.comhttp://blogs.msdn.com/David. Wang
> > //
>
> > On Nov 15, 2:10 pm, Shan Plourde wrote:> Hi Spitfire, thank you for the suggestion! Your link pointed to
> > > Portmon, is that the same thing as Procmon? Have you had practical
> > > experience using it to diagnose and resolve this specific type of
> > > error?
>
> > > Thanks
> > > Shan
>
> > > On Nov 15, 4:34 pm, "spitfire" wrote:
>
> > > > Try
> > > > Procmon utility , If you haven't tried yet.http://www.microsoft.com/technet/sysinternals/ProcessesA ndThreads/Por...
>
> > > > "Shan Plourde" wrote in message
>
> > > >news:ec423693-9008-414c-a8c7-02d9b6ef1cde@d50g2000hsf.googl egroups.com...
>
> > > > > Hello there, in an attempt to retain the few dozen hairs remaining on
> > > > > the top of my head before I yank them all out in angst, I'm completely
> > > > > stuck....HEEEELP!
>
> > > > > We are always getting these errors in our System event log:
>
> > > > > A process serving application pool 'poolName' suffered a fatal
> > > > > communication error with the World Wide Web Publishing Service. The
> > > > > process id was '1234'.
>
> > > > > We have IIS6 on Windows Server 2003. These errors are all coming from
> > > > > our intranet website hosted on this server. The intranet website runs
> > > > > on .NET 2.0. We also have many other websites hosted on the server,
> > > > > most .NET, but some classic ASP, but this particular error only is
> > > > > reported for the intranet website. All other ASP.NET applications are
> > > > > hosted within their own application pools.
>
> > > > > The error happens A LOT (like, every few minutes). It's a huge issue
> > > > > because it seems to cause large file uploads through our Intranet to
> > > > > crap out when the issue happens (perhaps due to application pool
> > > > > recycling?).
>
> > > > > So to start a debugging session I ran:
> > > > > cscript \windows\system32\iisapp.vbs
>
> > > > > Which lists all app pools and processes. In this case at the time I
> > > > > ran iisapp.vbs, the application pool named "poolName" is hosted by
> > > > > process 9900. So I ran this:
>
> > > > > iisstate -p 9900 -hc -d
>
> > > > > In an attempt to log the next hard crash. When the hard crash happens,
> > > > > I just can't figure out how to make effective use of the log
> > > > > information. I've looked at several logs but none seem to make me go
> > > > > "AHA!". I understand that Microsoft recommends running this tool, but
> > > > > I don't think that I can interpret any of it...please, MSFT or anyone,
> > > > > your help is greatly appreciated to interpret the crash log and
> > > > > provide recommendations!!! What follows is the log, through to the end
> > > > > of this post....Thx!!!
>
> > > > > ----------------------------------------
>
> > > > > Opened log file 'E:\Program Files\IIS Resources\IISState\output
> > > > > \IISState.log'
>
> > > > > ***********************
> > > > > Starting new log output
> > > > > IISState version 3.0
>
> > > > > Thu Nov 15 14:46:28 2007
>
> > > > > OS = Windows 2003 Server
> > > > > Executable: w3wp.exe
> > > > > PID = 6976
>
> > > > > Note: Thread times are formatted as HH:MM:SS.ms
>
> > > > > ***********************
>
> > > > > IIS has crashed...
> > > > > Beginning Analysis
> > > > > DLL (!FunctionName) that failed:
>
> > > > > Thread ID: 30
> > > > > System Thread ID: 1a90
> > > > > Kernel Time: 0:0:0.375
> > > > > User Time: 0:0:2.140
> > > > > Thread Type: Other
> > > > > # ChildEBP RetAddr
> > > > > WARNING: Frame IP not in any known module. Following frames may be
> > > > > wrong.
> > > > > 00 1bf8f118 1db03f0e 0x1db03f5f
> > > > > 01 00000000 00000000 0x1db03f0e
> > > > > Closing open log file E:\Program Files\IIS Resources\IISState\output
> > > > > \IISState.log
> > > > > Opened log file 'E:\Program Files\IIS Resources\IISState\output
> > > > > \IISState.log'
>
> > > > > ***********************
> > > > > Starting new log output
> > > > > IISState version 3.0
>
> > > > > Thu Nov 15 15:59:00 2007
>
> > > > > OS = Windows 2003 Server
> > > > > Executable: w3wp.exe
> > > > > PID = 9900
>
> > > > > Note: Thread times are formatted as HH:MM:SS.ms
>
> > > > > ***********************
>
> > > > > IIS has crashed...
> > > > > Beginning Analysis
> > > > > DLL (!FunctionName) that failed: kernel32!RaiseException
>
> > > > > Thread ID: 27
> > > > > System Thread ID: 1ea0
> > > > > Kernel Time: 0:0:0.468
> > > > > User Time: 0:0:4.359
> > > > > Thread Type: Other
> > > > > # ChildEBP RetAddr
> > > > > 00 1acaf898 79f55b05 kernel32!RaiseException+0x53
> > > > > 01 1acaf8f8 7a056a79 mscorwks!RaiseTheExceptionInternalOnly+0x226
> > > > > 02 1acaf90c 7a056af2 mscorwks!RaiseTheException+0x4d
> > > > > 03 1acaf934 7a056b30 mscorwks!RaiseTheException+0xbf
> > > > > 04 1acaf960 7a056b41 mscorwks!RealCOMPlusThrow+0x38
> > > > > 05 1acaf96c 7a0e38fa mscorwks!RealCOMPlusThrow+0xb
> > > > > 06 1acafa9c 7a0e4b97 mscorwks!Thread::RaiseCrossContextException+0x3ac
> > > > > 07 1acafb50 7a0e5a07 mscorwks!Thread::DoADCallBack+0x26c
> > > > > 08 1acafb68 79ed89ca mscorwks!Thread::UserResumeThread+0xe1
> > > > > 09 1acafbfc 79ed88f1 mscorwks!Thread::DoADCallBack+0x355
> > > > > 0a 1acafc38 7a0e4d96 mscorwks!Thread::DoADCallBack+0x541
> > > > > 0b 1acafc60 7a0e4dc3 mscorwks!Thread::DoADCallBack+0x575
> > > > > 0c 1acafc74 7a077528 mscorwks!ManagedThreadBase::ThreadPool+0x13
> > > > > 0d 1acafcbc 7a077556 mscorwks!AddTimerCallbackEx+0x77
> > > > > 0e 1acafccc 7a1158ae mscorwks!AddTimerCallback+0xf
> > > > > 0f 1acafd08 79f8b7a5 mscorwks!
> > > > > ThreadpoolMgr::AsyncTimerCallbackCompletion+0x64
> > > > > 10 1acafd20 79f8b341 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0x40
> > > > > 11 1acafd94 79ed8e36 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x225
> > > > > 12 1acaffb8 77e64829 mscorwks!Thread::intermediateThreadProc+0x49
> > > > > 13 1acaffec 00000000 kernel32!BaseThreadStart+0x34
> > > > > Closing open log file E:\Program Files\IIS Resources\IISState\output
> > > > > \IISState.log
> > > > > Opened log file 'E:\Program Files\IIS Resources\IISState\output
> > > > > \IISState.log'
>
> > > > > ***********************
> > > > > Starting new log output
> > > > > IISState version 3.0
>
> > > > > Thu Nov 15 15:59:01 2007
>
> > > > > OS = Windows 2003 Server
> > > > > Executable: w3wp.exe
> > > > > PID = 9900
>
> > > > > Note: Thread times are formatted as HH:MM:SS.ms
>
> > > > > ***********************
>
> > > > > Thread ID: 0
> > > > > System Thread ID: 26d4
> > > > > Kernel Time: 0:0:0.78
> > > > > User Time: 0:0:0.171
> > > > > Thread Status: Thread is in a WAIT state.
> > > > > Thread Type: HTTP Compression Thread
> > > > > # ChildEBP RetAddr
> > > > > 00 0006fc08 7c827d0b ntdll!KiFastSystemCallRet
> > > > > 01 0006fc0c 77e61d1e ntdll!NtWaitForSingleObject+0xc
> > > > > 02 0006fc7c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> > > > > 03 0006fc90 5a364662 kernel32!WaitForSingleObject+0x12
> > > > > 04 0006fca0 5a366e3f w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
> > > > > 05 0006fca8 5a3af42d w3dt!UlAtqStartListen+0x2d
> > > > > 06 0006fcb8 5a3bc335 w3core!W3_SERVER::StartListen+0xbd
> > > > > 07 0006ff0c 0100187c w3core!UlW3Start+0x26e
> > > > > 08 0006ff44 01001a27 w3wp!wmain+0x22a
> > > > > 09 0006ffc0 77e6f23b w3wp!wmainCRTStartup+0x12f
> > > > > 0a 0006fff0 00000000 kernel32!BaseProcessStart+0x23
>
> > > > > Thread ID: 1
> > > > > System Thread ID: 1290
> > > > > Kernel Time: 0:0:0.0
> > > > > User Time: 0:0:0.0
> > > > > Thread Status: Thread is in a WAIT state.
> > > > > Thread Status: Thread is in a WAIT state.
> > > > > Thread Type: Other
> > > > > # ChildEBP RetAddr
> > > > > 00
>
> ...
>
> read more >>

Re: How the heck can you effectively debug these "process serving app pool suffered...blah blah

There are all sorts of reasons your application may be crashing - it could
even be that not enough contiguous memory could be allocated within the
process' address space. Or that the process was running out of available
address space, or whatever.

You need to get a dump that you can then analyze using a debugging tool.
IISState is one option, or IIS Debug Diagnostics (this includes an analytics
piece), or just get a dump file some other way (adplus.vbs can also do
this - all these tools just manipulate the same Windows symbolic debug
engine) and then analyze using WinDBG + !SOS extension.

Alternatively, if these crashes don't happen to frequently, you can "orphan"
the crashed worker process (Windows doesn't shut down the crashed process
when it starts a new one), and you can attach a debugger to the orphaned
process. however if the crashes are happenign often, yo might end up with a
lot of orphaned processes consuming memory, which may cause other issues
with your server if you aren't able to get to the debugging phase quickly
enough :-)

Cheers
Ken

--
My IIS Blog: www.adOpenStatic.com/cs/blogs/ken

"Shan Plourde" wrote in message
news:28065dc6-2a7c-416e-892c-5100deb3c008@c29g2000hsa.google groups.com...
> David, the other thing that I'd like to understand is what type of
> application exceptions could cause an app pool to crash without the
> exceptions bubbling up to the top of the application? Our exception
> handling strategy for our applications is to simply let exceptions
> bubble up to the top of the stack trace, and then email exceptions and
> log them appropriately. Obviously when an app pool crashes it can't be
> due to a simple exception like this.
>
> I would assume that app pool crashes would either be things such as
> infinite loops, stack overflows, thread contentions / deadlock
> situations....appreciate hearing your insights on this as well!
>
> Thanks
> Shan
>
> On Nov 16, 9:30 am, Shan Plourde wrote:
>> Hi David, thank you for your insights, I absolutely believe that it is
>> an application specific issue causing the application pool to crash,
>> but I am unable to determine which part of the application is causing
>> the issue. I am very much interested in knowing how to interpret the
>> log file: have you determined from the log file that it is the
>> application's file upload functionality? If so, how were you able to
>> determine that?
>>
>> This is an intranet application and recently I have made some changes
>> to support large file uploads on it. At this time there is a "multiple
>> file uploads" page that supports up to 5 uploads, and each upload
>> could be in the vicinity of several hundred KB to 30MB in size. I've
>> configured the standard httpRuntime configuration elements for the
>> application to allow larger uploads and to allow larger execution
>> times. The processing of the uploading on the streams on the server
>> side essentially goes from a .NET user control code behind to various
>> business logic and data model layer classes that persist the up to 5
>> streams.
>>
>> I look forward to your responses to my questions, and thank you very
>> much for your help! In the meantime I will review the file uploading
>> logic and accompanying documentation for the APIs that I'm using to
>> see if there is anything that looks incorrect to me, and share those
>> findings as well.
>>
>> Thanks again
>> Shan
>>
>> On Nov 16, 7:43 am, David Wang wrote:
>>
>> > In this case, tools like procmon, filemon, and regmon are not useful
>> > because there is a real crash happening in the worker process, not
>> > really file/registry access issues.
>>
>> > You need to do *real* debugging to figure out what is wrong with your
>> > web application. None of that contrived Visual Studio "Hit F5 and
>> > debug" non-real-world stuff. What IIS State provides is but a taste of
>> > what real, powerful debugging is like, and it requires a bit of
>> > Windows systems knowledge to make sense of it all. Don't be afraid at
>> > the log file; there is great benefit when one learns to understand
>> > what the log file says.
>>
>> > Now, your log file actually indicates two different types of crash
>> > patterns, telling me there's probably a couple of bad bugs within your
>> > web application causing such frequent failures. The application pool
>> > recycling is NOT the problem. Your crashing code is the problem.
>> > Recycling is what brings the application back (sans in-process state)
>> > and has nothing to do with your issue next to attempting to recover
>> > from the crash.
>>
>> > Maybe the object you use to handle large file uploads is the problem.
>> > What code is responsible for this functionality on your website? It
>> > would not be the first time I see a file uploader have bugs.
>>
>> > //Davidhttp://w3-4u.blogspot.comhttp://blogs.msdn.com/David. Wang
>> > //
>>
>> > On Nov 15, 2:10 pm, Shan Plourde wrote:
>>
>> > > Hi Spitfire, thank you for the suggestion! Your link pointed to
>> > > Portmon, is that the same thing as Procmon? Have you had practical
>> > > experience using it to diagnose and resolve this specific type of
>> > > error?
>>
>> > > Thanks
>> > > Shan
>>
>> > > On Nov 15, 4:34 pm, "spitfire" wrote:
>>
>> > > > Try
>> > > > Procmon utility , If you haven't tried
>> > > > yet.http://www.microsoft.com/technet/sysinternals/ProcessesA ndThreads/Por...
>>
>> > > > "Shan Plourde" wrote in message
>>
>> > > >news:ec423693-9008-414c-a8c7-02d9b6ef1cde@d50g2000hsf.googl egroups.com...
>>
>> > > > > Hello there, in an attempt to retain the few dozen hairs
>> > > > > remaining on
>> > > > > the top of my head before I yank them all out in angst, I'm
>> > > > > completely
>> > > > > stuck....HEEEELP!
>>
>> > > > > We are always getting these errors in our System event log:
>>
>> > > > > A process serving application pool 'poolName' suffered a fatal
>> > > > > communication error with the World Wide Web Publishing Service.
>> > > > > The
>> > > > > process id was '1234'.
>>
>> > > > > We have IIS6 on Windows Server 2003. These errors are all coming
>> > > > > from
>> > > > > our intranet website hosted on this server. The intranet website
>> > > > > runs
>> > > > > on .NET 2.0. We also have many other websites hosted on the
>> > > > > server,
>> > > > > most .NET, but some classic ASP, but this particular error only
>> > > > > is
>> > > > > reported for the intranet website. All other ASP.NET applications
>> > > > > are
>> > > > > hosted within their own application pools.
>>
>> > > > > The error happens A LOT (like, every few minutes). It's a huge
>> > > > > issue
>> > > > > because it seems to cause large file uploads through our Intranet
>> > > > > to
>> > > > > crap out when the issue happens (perhaps due to application pool
>> > > > > recycling?).
>>
>> > > > > So to start a debugging session I ran:
>> > > > > cscript \windows\system32\iisapp.vbs
>>
>> > > > > Which lists all app pools and processes. In this case at the time
>> > > > > I
>> > > > > ran iisapp.vbs, the application pool named "poolName" is hosted
>> > > > > by
>> > > > > process 9900. So I ran this:
>>
>> > > > > iisstate -p 9900 -hc -d
>>
>> > > > > In an attempt to log the next hard crash. When the hard crash
>> > > > > happens,
>> > > > > I just can't figure out how to make effective use of the log
>> > > > > information. I've looked at several logs but none seem to make me
>> > > > > go
>> > > > > "AHA!". I understand that Microsoft recommends running this tool,
>> > > > > but
>> > > > > I don't think that I can interpret any of it...please, MSFT or
>> > > > > anyone,
>> > > > > your help is greatly appreciated to interpret the crash log and
>> > > > > provide recommendations!!! What follows is the log, through to
>> > > > > the end
>> > > > > of this post....Thx!!!
>>
>> > > > > ----------------------------------------
>>

Re: How the heck can you effectively debug these "process serving app

Thanks everyone for your kind help and suggestions!

Since the issue seemed impossible to debug with DebugDiag, IISState,
etc., on my own, we contacted Microsoft PSS. A helpful Microsoft
support engineer requested our debug logs. After I sent him our debug
logs, within 2 or so hours he emailed us a link to download a hotfix.
The hotfix is at http://support.microsoft.com/kb/942086. The issue
apparently looks like the ASP.NET worker process crashes when the .NET
deserializer looks for a default constructor and cannot find it.
Anyhow, it was a Microsoft in this case, so it was definitely worth
the time contacting them to resolve this.

I have definitely learned how to do more detailed debugging with
DebugDiag...hopefully in the future it will come in handy to debug
application specific issues.

Thanks again,
Shan

On Nov 20, 5:48 am, "Ken Schaefer"
wrote:
> There are all sorts of reasons your application may be crashing - it could
> even be that not enough contiguous memory could be allocated within the
> process' address space. Or that the process was running out of available
> address space, or whatever.
>
> You need to get a dump that you can then analyze using a debugging tool.
> IISState is one option, or IIS Debug Diagnostics (this includes an analytics
> piece), or just get a dump file some other way (adplus.vbs can also do
> this - all these tools just manipulate the same Windows symbolic debug
> engine) and then analyze using WinDBG + !SOS extension.
>
> Alternatively, if these crashes don't happen to frequently, you can "orphan"
> the crashed worker process (Windows doesn't shut down the crashed process
> when it starts a new one), and you can attach a debugger to the orphaned
> process. however if the crashes are happenign often, yo might end up with a
> lot of orphaned processes consuming memory, which may cause other issues
> with your server if you aren't able to get to the debugging phase quickly
> enough :-)
>
> Cheers
> Ken
>
> --
> My IIS Blog:www.adOpenStatic.com/cs/blogs/ken
>
> "Shan Plourde" wrote in message
>
> news:28065dc6-2a7c-416e-892c-5100deb3c008@c29g2000hsa.google groups.com...
>
> > David, the other thing that I'd like to understand is what type of
> > application exceptions could cause an app pool to crash without the
> > exceptions bubbling up to the top of the application? Our exception
> > handling strategy for our applications is to simply let exceptions
> > bubble up to the top of the stack trace, and then email exceptions and
> > log them appropriately. Obviously when an app pool crashes it can't be
> > due to a simple exception like this.
>
> > I would assume that app pool crashes would either be things such as
> > infinite loops, stack overflows, thread contentions / deadlock
> > situations....appreciate hearing your insights on this as well!
>
> > Thanks
> > Shan
>
> > On Nov 16, 9:30 am, Shan Plourde wrote:
> >> Hi David, thank you for your insights, I absolutely believe that it is
> >> an application specific issue causing the application pool to crash,
> >> but I am unable to determine which part of the application is causing
> >> the issue. I am very much interested in knowing how to interpret the
> >> log file: have you determined from the log file that it is the
> >> application's file upload functionality? If so, how were you able to
> >> determine that?
>
> >> This is an intranet application and recently I have made some changes
> >> to support large file uploads on it. At this time there is a "multiple
> >> file uploads" page that supports up to 5 uploads, and each upload
> >> could be in the vicinity of several hundred KB to 30MB in size. I've
> >> configured the standard httpRuntime configuration elements for the
> >> application to allow larger uploads and to allow larger execution
> >> times. The processing of the uploading on the streams on the server
> >> side essentially goes from a .NET user control code behind to various
> >> business logic and data model layer classes that persist the up to 5
> >> streams.
>
> >> I look forward to your responses to my questions, and thank you very
> >> much for your help! In the meantime I will review the file uploading
> >> logic and accompanying documentation for the APIs that I'm using to
> >> see if there is anything that looks incorrect to me, and share those
> >> findings as well.
>
> >> Thanks again
> >> Shan
>
> >> On Nov 16, 7:43 am, David Wang wrote:
>
> >> > In this case, tools like procmon, filemon, and regmon are not useful
> >> > because there is a real crash happening in the worker process, not
> >> > really file/registry access issues.
>
> >> > You need to do *real* debugging to figure out what is wrong with your
> >> > web application. None of that contrived Visual Studio "Hit F5 and
> >> > debug" non-real-world stuff. What IIS State provides is but a taste of
> >> > what real, powerful debugging is like, and it requires a bit of
> >> > Windows systems knowledge to make sense of it all. Don't be afraid at
> >> > the log file; there is great benefit when one learns to understand
> >> > what the log file says.
>
> >> > Now, your log file actually indicates two different types of crash
> >> > patterns, telling me there's probably a couple of bad bugs within your
> >> > web application causing such frequent failures. The application pool
> >> > recycling is NOT the problem. Your crashing code is the problem.
> >> > Recycling is what brings the application back (sans in-process state)
> >> > and has nothing to do with your issue next to attempting to recover
> >> > from the crash.
>
> >> > Maybe the object you use to handle large file uploads is the problem.
> >> > What code is responsible for this functionality on your website? It
> >> > would not be the first time I see a file uploader have bugs.
>
> >> > //Davidhttp://w3-4u.blogspot.comhttp://blogs.msdn.com/David. Wang
> >> > //
>
> >> > On Nov 15, 2:10 pm, Shan Plourde wrote:
>
> >> > > Hi Spitfire, thank you for the suggestion! Your link pointed to
> >> > > Portmon, is that the same thing as Procmon? Have you had practical
> >> > > experience using it to diagnose and resolve this specific type of
> >> > > error?
>
> >> > > Thanks
> >> > > Shan
>
> >> > > On Nov 15, 4:34 pm, "spitfire" wrote:
>
> >> > > > Try
> >> > > > Procmon utility , If you haven't tried
> >> > > > yet.http://www.microsoft.com/technet/sysinternals/ProcessesA ndThreads/Por...
>
> >> > > > "Shan Plourde" wrote in message
>
> >> > > >news:ec423693-9008-414c-a8c7-02d9b6ef1cde@d50g2000hsf.googl egroups.com...
>
> >> > > > > Hello there, in an attempt to retain the few dozen hairs
> >> > > > > remaining on
> >> > > > > the top of my head before I yank them all out in angst, I'm
> >> > > > > completely
> >> > > > > stuck....HEEEELP!
>
> >> > > > > We are always getting these errors in our System event log:
>
> >> > > > > A process serving application pool 'poolName' suffered a fatal
> >> > > > > communication error with the World Wide Web Publishing Service.
> >> > > > > The
> >> > > > > process id was '1234'.
>
> >> > > > > We have IIS6 on Windows Server 2003. These errors are all coming
> >> > > > > from
> >> > > > > our intranet website hosted on this server. The intranet website
> >> > > > > runs
> >> > > > > on .NET 2.0. We also have many other websites hosted on the
> >> > > > > server,
> >> > > > > most .NET, but some classic ASP, but this particular error only
> >> > > > > is
> >> > > > > reported for the intranet website. All other ASP.NET applications
> >> > > > > are
> >> > > > > hosted within their own application pools.
>
> >> > > > > The error happens A LOT (like, every few minutes). It's a huge
> >> > > > > issue
> >> > > > > because it seems to cause large file uploads through our Intranet
> >> > > > > to
> >> > > > > crap out when the issue happens (perhaps due to application pool
> >> > > > > recycling?).
>
> >> > > > > So to start a debugging session I ran:
> >> > > > > cscript \windows\system32\iisapp.vbs
>
> >> > > > > Which lists all app pools and processes. In this case at the time
> >> > > > > I
> >> > > > > ran iisapp.vbs, the application pool named "poolName" is hosted
> >> > > > > by
> >> > > > > process 9900. So I ran this:
>
> >> > > > > iisstate -p 9900 -hc -d
>
> >> > > > > In an attempt to log the next hard crash. When the hard crash
> >> > > > > happens,
> >> > > > > I just can't figure out how to make effective use of the log
> >> > > > > information. I've looked at several logs but none seem to make me
> >> > > > > go
> >> > > > > "AHA!". I understand that Microsoft recommends running this tool,
> >> > > > > but
> >> > > > > I don't think that I can interpret any of it...please, MSFT or
> >> > > > > anyone,
> >> > > > > your help is greatly appreciated to interpret the crash log and
> >> > > > > provide recommendations!!! What follows is the log, through to
> >> > > > > the end
> >> > > > > of this post....Thx!!!
>
> >> > > > > ----------------------------------------