App Pool Hang

We have recently upgraded our website to .Net 2.0. Since the upgrade the
servers running our site have been hanging up and not responding to client
requests. We have done a number of dumps while the worker process is hung
but can't see exactly what the problem is. We can see some long running file
uploads but the vendor says its not their problem. If anyone can see
something in the attached debug diag log that could prove what the problem is
I would appreciate the help.

Type of Analysis Performed Hang Analysis
Operating System Windows Server 2003
Number Of Processors 4
Process ID 9520
Process Image c:\WINDOWS\system32\inetsrv\w3wp.exe
System Up-Time 114 day(s) 12:42:44
Process Up-Time 04:49:31

Top 5 Threads by CPU time
Note - Times include both user mode and kernel mode for each thread Thread
ID: 46 Total CPU Time: 01:27:32.327 Entry Point for Thread:
mscorwks!Thread::intermediateThreadProc
Thread ID: 33 Total CPU Time: 00:08:22.874 Entry Point for Thread:
mscorwks!Thread::intermediateThreadProc
Thread ID: 41 Total CPU Time: 00:05:59.468 Entry Point for Thread:
mscorwks!Thread::intermediateThreadProc
Thread ID: 43 Total CPU Time: 00:04:49.593 Entry Point for Thread:
mscorwks!Thread::intermediateThreadProc
Thread ID: 36 Total CPU Time: 00:04:01.781 Entry Point for Thread:
mscorwks!Thread::intermediateThreadProc

Thread report

Thread 0 - System ID 8084
Entry point w3wp!wmainCRTStartup
Create time 9/12/2007 5:29:09 PM
Time spent in user mode 0 Days 00:00:00.156
Time spent in kernel mode 0 Days 00:00:00.531

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtDelayExecution+c
kernel32!SleepEx+68
kernel32!Sleep+b
w3dt!UL_NATIVE_REQUEST::ReleaseAllWorkerRequests+1d
w3dt!WP_CONTEXT::CleanupOutstandingRequests+1c
w3dt!UlAtqStartListen+51
w3core!W3_SERVER::StartListen+8e
w3core!UlW3Start+ea
w3wp!wmain+236
w3wp!wmainCRTStartup+12f
kernel32!BaseProcessStart+23

Thread 1 - System ID 10432
Entry point ntdll!RtlpTimerThread
Create time 9/12/2007 5:29:09 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.281

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtDelayExecution+c
ntdll!RtlpTimerThread+45
kernel32!BaseThreadStart+34

Thread 2 - System ID 8340
Create time 12:00:00 AM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.00

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtReplyWaitReceivePortEx+c
rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+193
rpcrt4!RecvLotsaCallsWrapper+9
rpcrt4!BaseCachedThreadRoutine+9c
rpcrt4!ThreadStartRoutine+17
kernel32!BaseThreadStart+34

Thread 3 - System ID 6216
Entry point w3tp!THREAD_MANAGER::ThreadManagerThread
Create time 9/12/2007 5:29:09 PM
Time spent in user mode 0 Days 00:00:02.031
Time spent in kernel mode 0 Days 00:00:00.703

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwRemoveIoCompletion+c
kernel32!GetQueuedCompletionStatus+27
w3tp!THREAD_POOL_DATA::ThreadPoolThread+31
w3tp!THREAD_POOL_DATA::ThreadPoolThread+21
w3tp!THREAD_MANAGER::ThreadManagerThread+35
kernel32!BaseThreadStart+34

Thread 4 - System ID 6824
Entry point w3tp!THREAD_MANAGER::ThreadManagerThread
Create time 9/12/2007 5:29:09 PM
Time spent in user mode 0 Days 00:00:02.156
Time spent in kernel mode 0 Days 00:00:00.734

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwRemoveIoCompletion+c
kernel32!GetQueuedCompletionStatus+27
w3tp!THREAD_POOL_DATA::ThreadPoolThread+31
w3tp!THREAD_POOL_DATA::ThreadPoolThread+21
w3tp!THREAD_MANAGER::ThreadManagerThread+35
kernel32!BaseThreadStart+34

Thread 5 - System ID 7640
Entry point w3tp!THREAD_MANAGER::ThreadManagerThread
Create time 9/12/2007 5:29:09 PM
Time spent in user mode 0 Days 00:00:02.015
Time spent in kernel mode 0 Days 00:00:00.703

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwRemoveIoCompletion+c
kernel32!GetQueuedCompletionStatus+27
w3tp!THREAD_POOL_DATA::ThreadPoolThread+31
w3tp!THREAD_POOL_DATA::ThreadPoolThread+21
w3tp!THREAD_MANAGER::ThreadManagerThread+35
kernel32!BaseThreadStart+34

Thread 6 - System ID 8604
Entry point w3tp!THREAD_MANAGER::ThreadManagerThread
Create time 9/12/2007 5:29:09 PM
Time spent in user mode 0 Days 00:00:01.906
Time spent in kernel mode 0 Days 00:00:00.734

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwRemoveIoCompletion+c
kernel32!GetQueuedCompletionStatus+27
w3tp!THREAD_POOL_DATA::ThreadPoolThread+31
w3tp!THREAD_POOL_DATA::ThreadPoolThread+21
w3tp!THREAD_MANAGER::ThreadManagerThread+35
kernel32!BaseThreadStart+34

Thread 7 - System ID 10044
Entry point ntdll!RtlpWaitThread
Create time 9/12/2007 5:29:09 PM
Time spent in user mode 0 Days 00:00:00.015
Time spent in kernel mode 0 Days 00:00:00.375

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
ntdll!RtlpWaitThread+158
kernel32!BaseThreadStart+34

Thread 8 - System ID 4816
Entry point msvcrt!_endthreadex+26
Create time 9/12/2007 5:29:09 PM
Time spent in user mode 0 Days 00:00:00.781
Time spent in kernel mode 0 Days 00:00:04.218

This thread is not fully resolved and may or may not be a problem. Further
analysis of these threads may be required.

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
user32!RealMsgWaitForMultipleObjectsEx+13f
user32!MsgWaitForMultipleObjects+1d
acsrtl!RemoveWorkItem+27b
msvcrt!free+c8
0x0026a5b0

Thread 9 - System ID 7652
Entry point msvcrt!_endthreadex+26
Create time 9/12/2007 5:29:09 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.406

This thread is not fully resolved and may or may not be a problem. Further
analysis of these threads may be required.

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
user32!RealMsgWaitForMultipleObjectsEx+13f
user32!MsgWaitForMultipleObjects+1d
acsrtl!RemoveWorkItem+27b
kernel32!BaseThreadStart+34

Thread 10 - System ID 10468
Entry point msvcrt!_endthreadex+26
Create time 9/12/2007 5:29:09 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.203

This thread is not fully resolved and may or may not be a problem. Further
analysis of these threads may be required.

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
user32!RealMsgWaitForMultipleObjectsEx+13f
user32!MsgWaitForMultipleObjects+1d
acsrtl!RemoveWorkItem+27b
kernel32!BaseThreadStart+34

Thread 11 - System ID 5512
Entry point msvcrt!_endthreadex+26
Create time 9/12/2007 5:29:09 PM
Time spent in user mode 0 Days 00:00:00.609
Time spent in kernel mode 0 Days 00:00:02.578

This thread is not fully resolved and may or may not be a problem. Further
analysis of these threads may be required.

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
user32!RealMsgWaitForMultipleObjectsEx+13f
user32!MsgWaitForMultipleObjects+1d
acsrtl!RemoveWorkItem+27b
msvcrt!free+c8
0x0026a7f0

Thread 12 - System ID 9900
Entry point msatq!AtqInitialize+1354
Create time 9/12/2007 5:29:09 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.531

This thread is not fully resolved and may or may not be a problem. Further
analysis of these threads may be required.

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwRemoveIoCompletion+c
kernel32!GetQueuedCompletionStatus+27
msatq!AtqInitialize+1392
kernel32!BaseThreadStart+34

Thread 13 - System ID 352
Entry point wmisink!CWMIEventSinkWrapper::SetupSubscription+30b
Create time 9/12/2007 5:29:09 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.312

This thread is not fully resolved and may or may not be a problem. Further
analysis of these threads may be required.

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
kernel32!WaitForMultipleObjects+17
wmisink!CWMIEventSinkWrapper::SetupSubscription+48e

Thread 14 - System ID 6108
Entry point RfFilExt!CMBData::~CMBData+c2
Create time 9/12/2007 5:29:09 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.375

This thread is calling an ISAPI Filter RfFilExt

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
kernel32!WaitForSingleObject+f
RfFilExt!CMBData::~CMBData+f6

Thread 15 - System ID 5992
Create time 12:00:00 AM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.00

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtReplyWaitReceivePortEx+c
rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+193
rpcrt4!RecvLotsaCallsWrapper+9
rpcrt4!BaseCachedThreadRoutine+9c
rpcrt4!ThreadStartRoutine+17
kernel32!BaseThreadStart+34

Thread 16 - System ID 7900
Create time 12:00:00 AM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.00

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtReplyWaitReceivePortEx+c
rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+193
rpcrt4!RecvLotsaCallsWrapper+9
rpcrt4!BaseCachedThreadRoutine+9c
rpcrt4!ThreadStartRoutine+17
kernel32!BaseThreadStart+34

Thread 17 - System ID 8108
Entry point w3core!HTTP_COMPRESSION::CompressionThread
Create time 9/12/2007 5:29:11 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.390

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
kernel32!WaitForSingleObject+f
w3core!HTTP_COMPRESSION::CompressionThread+85
kernel32!BaseThreadStart+34

Thread 18 - System ID 12228
Entry point mscorwks!ThreadpoolMgr::intermediateThreadProc
Create time 9/12/2007 5:29:15 PM
Time spent in user mode 0 Days 00:00:00.062
Time spent in kernel mode 0 Days 00:00:00.296

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
mscorwks!PEImage::LoadImage+199
mscorwks!CLREvent::WaitEx+117
mscorwks!CLREvent::Wait+17
mscorwks!ThreadpoolMgr::SafeWait+73
mscorwks!ThreadpoolMgr::CompletionPortThreadStart+24d
mscorwks!ThreadpoolMgr::intermediateThreadProc+49
kernel32!BaseThreadStart+34

Thread 19 - System ID 9820
Entry point mscorwks!ThreadpoolMgr::GateThreadStart
Create time 9/12/2007 5:29:15 PM
Time spent in user mode 0 Days 00:00:00.031
Time spent in kernel mode 0 Days 00:00:00.218

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtDelayExecution+c
kernel32!SleepEx+68
mscorwks!EESleepEx+a3
mscorwks!__DangerousSwitchToThread+70
mscorwks!__SwitchToThread+b
mscorwks!ThreadpoolMgr::GateThreadStart+a1
kernel32!BaseThreadStart+34

Thread 20 - System ID 6512
Entry point mscorwks!ThreadpoolMgr::intermediateThreadProc
Create time 9/12/2007 5:29:15 PM
Time spent in user mode 0 Days 00:01:54.015
Time spent in kernel mode 0 Days 00:00:08.812

This thread is not fully resolved and may or may not be a problem. Further
analysis of these threads may be required.

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+6f
mscorwks!Thread::DoAppropriateAptStateWait+3c
mscorwks!Thread::DoAppropriateWaitWorker+144
mscorwks!Thread::DoAppropriateWait+40
mscorwks!CLREvent::WaitEx+f7
mscorwks!CLREvent::Wait+17
mscorwks!SVR::GCHeap::FinalizerThreadWait+105
mscorwks!GCInterface::RunFinalizers+99
0x3198d4a4
System_Web_ni+20a5a4
System_Web_ni+20a5a4

Thread 21 - System ID 11708
Entry point mscorwks!DebuggerRCThread::ThreadProcStatic
Create time 9/12/2007 5:29:15 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.500

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
kernel32!WaitForMultipleObjects+17
mscorwks!DebuggerRCThread::MainLoop+cf
mscorwks!DebuggerRCThread::ThreadProc+ca
mscorwks!DebuggerRCThread::ThreadProcStatic+82
kernel32!BaseThreadStart+34

Thread 22 - System ID 3500
Entry point mscorwks!SVR::gc_heap::gc_thread_stub
Create time 9/12/2007 5:29:16 PM
Time spent in user mode 0 Days 00:03:02.031
Time spent in kernel mode 0 Days 00:00:02.421

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
mscorwks!PEImage::LoadImage+199
mscorwks!CLREvent::WaitEx+117
mscorwks!CLREvent::Wait+17
mscorwks!SVR::gc_heap::gc_thread_function+2e
mscorwks!SVR::gc_heap::gc_thread_stub+9b
kernel32!BaseThreadStart+34

Thread 23 - System ID 6900
Entry point mscorwks!SVR::gc_heap::gc_thread_stub
Create time 9/12/2007 5:29:16 PM
Time spent in user mode 0 Days 00:02:58.265
Time spent in kernel mode 0 Days 00:00:00.703

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
mscorwks!PEImage::LoadImage+199
mscorwks!CLREvent::WaitEx+117
mscorwks!CLREvent::Wait+17
mscorwks!SVR::gc_heap::gc_thread_function+57
mscorwks!SVR::gc_heap::gc_thread_stub+9b
kernel32!BaseThreadStart+34

Thread 24 - System ID 6392
Entry point mscorwks!SVR::gc_heap::gc_thread_stub
Create time 9/12/2007 5:29:16 PM
Time spent in user mode 0 Days 00:03:12.453
Time spent in kernel mode 0 Days 00:00:00.953

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
mscorwks!PEImage::LoadImage+199
mscorwks!CLREvent::WaitEx+117
mscorwks!CLREvent::Wait+17
mscorwks!SVR::gc_heap::gc_thread_function+57
mscorwks!SVR::gc_heap::gc_thread_stub+9b
kernel32!BaseThreadStart+34

Thread 25 - System ID 6916
Entry point mscorwks!SVR::gc_heap::gc_thread_stub
Create time 9/12/2007 5:29:16 PM
Time spent in user mode 0 Days 00:03:02.968
Time spent in kernel mode 0 Days 00:00:00.671

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
mscorwks!PEImage::LoadImage+199
mscorwks!CLREvent::WaitEx+117
mscorwks!CLREvent::Wait+17
mscorwks!SVR::gc_heap::gc_thread_function+57
mscorwks!SVR::gc_heap::gc_thread_stub+9b
kernel32!BaseThreadStart+34

Thread 26 - System ID 4144
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 5:29:16 PM
Time spent in user mode 0 Days 00:00:00.953
Time spent in kernel mode 0 Days 00:00:00.484

This thread is making a COM call to thread 20 within the same process

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
kernel32!WaitForSingleObject+f
ole32!GetToSTA+6d
ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+c8
ole32!CRpcChannelBuffer::SendReceive2+b6
ole32!CAptRpcChnl::SendReceive+ab
ole32!CCtxComChnl::SendReceive+8f
rpcrt4!NdrProxySendReceive+41
rpcrt4!NdrClientCall2+1bc
rpcrt4!ObjectStublessClient+89
rpcrt4!ObjectStubless+f
ole32!CObjectContext::InternalContextCallback+10b
ole32!CObjectContext::ContextCallback+6e
mscorwks!CtxEntry::EnterContextOle32BugAware+2b
mscorwks!CtxEntry::EnterContext+2db
mscorwks!RCWCleanupList::ReleaseRCWListInCorrectCtx+c4
mscorwks!RCWCleanupList::CleanupAllWrappers+77
mscorwks!SyncBlockCache::CleanupSyncBlocks+ec
mscorwks!Thread::DoExtraWorkForFinalizer+40
mscorwks!SVR::GCHeap::FinalizerThreadWorker+c6
mscorwks!Thread::UserResumeThread+fb
mscorwks!Thread::DoADCallBack+355
mscorwks!Thread::DoADCallBack+541
mscorwks!ManagedThreadBase_NoADTransition+32
mscorwks!ManagedThreadBase::FinalizerBase+b
mscorwks!SVR::GCHeap::FinalizerThreadStart+bb
mscorwks!Thread::intermediateThreadProc+49
kernel32!BaseThreadStart+34

Thread 27 - System ID 5544
Entry point mscorwks!ThreadpoolMgr::TimerThreadStart
Create time 9/12/2007 5:29:16 PM
Time spent in user mode 0 Days 00:00:00.046
Time spent in kernel mode 0 Days 00:00:00.312

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtDelayExecution+c
kernel32!SleepEx+68
mscorwks!ThreadpoolMgr::TimerThreadFire+6d
mscorwks!ThreadpoolMgr::TimerThreadStart+56
kernel32!BaseThreadStart+34

Thread 28 - System ID 9504
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 5:29:16 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.359

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+6f
mscorwks!Thread::DoAppropriateAptStateWait+3c
mscorwks!Thread::DoAppropriateWaitWorker+144
mscorwks!Thread::DoAppropriateWait+40
mscorwks!CLREvent::WaitEx+f7
mscorwks!CLREvent::Wait+17
mscorwks!SVR::GCHeap::FinalizerThreadWait+105
mscorwks!AppDomain::Exit+d9
mscorwks!AppDomain::Unload+1b0
mscorwks!AppDomain::ADUnloadWorkerHelper+76
mscorwks!AppDomain::DoADUnloadWork+6c
mscorwks!AppDomain::DoADUnloadWork+de
mscorwks!AppDomain::ADUnloadThreadStart+459
mscorwks!Thread::intermediateThreadProc+49
kernel32!BaseThreadStart+34

Thread 29 - System ID 10868
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 5:29:16 PM
Time spent in user mode 0 Days 00:00:00.140
Time spent in kernel mode 0 Days 00:00:00.421

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
mscorwks!PEImage::LoadImage+199
mscorwks!CLREvent::WaitEx+117
mscorwks!CLREvent::Wait+17
mscorwks!ThreadpoolMgr::SafeWait+73
mscorwks!ThreadpoolMgr::CompletionPortThreadStart+24d
mscorwks!Thread::intermediateThreadProc+49
kernel32!BaseThreadStart+34

Thread 30 - System ID 10840
Entry point rpcrt4!ThreadStartRoutine
Create time 9/12/2007 5:29:16 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.328

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwRemoveIoCompletion+c
kernel32!GetQueuedCompletionStatus+27
rpcrt4!COMMON_ProcessCalls+9f
rpcrt4!LOADABLE_TRANSPORT::ProcessIOEvents+115
rpcrt4!ProcessIOEventsWrapper+9
rpcrt4!BaseCachedThreadRoutine+9c
rpcrt4!ThreadStartRoutine+17
kernel32!BaseThreadStart+34

Thread 31 - System ID 6996
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 5:29:17 PM
Time spent in user mode 0 Days 00:01:50.937
Time spent in kernel mode 0 Days 00:00:14.156

This thread is not fully resolved and may or may not be a problem. Further
analysis of these threads may be required.

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+6f
mscorwks!Thread::DoAppropriateAptStateWait+3c
mscorwks!Thread::DoAppropriateWaitWorker+144
mscorwks!Thread::DoAppropriateWait+40
mscorwks!CLREvent::WaitEx+f7
mscorwks!CLREvent::Wait+17
mscorwks!SVR::GCHeap::FinalizerThreadWait+105
mscorwks!GCInterface::RunFinalizers+99
0x3198d4a4
mscorlib_ni+297171
System_Web_ni+20a5a4
System_Web_ni+20a5f0
System_Web_ni+20a644
System_Web_ni+20a644
System_Web_ni+20a644
System_Web_ni+21bbf6
System_Web_ni+21a38f
System_Web_ni+21b521
System_Web_ni+21b487
System_Web_ni+21b3a7
System_Web_ni+c88df
System_Web_ni+9a071
System_Web_ni+9a39b
System_Web_ni+c8305
System_Web_ni+3ea679
System_Web_ni+ca305
mscorlib_ni+319c1a
mscorwks!CallDescrWorker+33
mscorwks!CallDescrWorkerWithHandler+a3
mscorwks!MethodDesc::CallDescr+19c
mscorwks!MethodDesc::CallTargetWorker+20
mscorwks!MethodDescCallSite::Call+18
mscorwks!ExecuteCodeWithGuaranteedCleanupHelper+b2
mscorwks!ReflectionInvocation::ExecuteCodeWithGuaranteedClea nup+f9
mscorlib_ni+2a84fb
mscorlib_ni+2a83ee

RE: App Pool Hang

here is the rest of the log


Thread 32 - System ID 7328
Entry point userenv!NotificationThread
Create time 9/12/2007 5:29:18 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.421

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
kernel32!WaitForMultipleObjects+17
userenv!NotificationThread+5f
kernel32!BaseThreadStart+34

Thread 33 - System ID 2704
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 5:29:20 PM
Time spent in user mode 0 Days 00:07:50.468
Time spent in kernel mode 0 Days 00:00:32.406

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
mscorwks!PEImage::LoadImage+199
mscorwks!CLREvent::WaitEx+117
mscorwks!CLREvent::Wait+17
mscorwks!ThreadpoolMgr::SafeWait+73
mscorwks!ThreadpoolMgr::WorkerThreadStart+f1
mscorwks!Thread::intermediateThreadProc+49
kernel32!BaseThreadStart+34

Thread 34 - System ID 6160
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 5:29:20 PM
Time spent in user mode 0 Days 00:01:10.359
Time spent in kernel mode 0 Days 00:00:10.437

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+6f
mscorwks!Thread::DoAppropriateAptStateWait+3c
mscorwks!Thread::DoAppropriateWaitWorker+144
mscorwks!Thread::DoAppropriateWait+40
mscorwks!Thread::JoinEx+86
mscorwks!Thread::UserAbort+5b2
mscorwks!ThreadNative::Abort+e4
mscorlib_ni+317e50
System_Web_ni+f1573
System_Web_ni+3edef7
mscorlib_ni+319c1a
mscorlib_ni+2a83dd
mscorlib_ni+319c7f
mscorwks!CallDescrWorker+33
mscorwks!CallDescrWorkerWithHandler+a3
mscorwks!DispatchCallBody+1e
mscorwks!DispatchCallDebuggerWrapper+3d
mscorwks!DispatchCallNoEH+51
mscorwks!AddTimerCallback_Worker+81
mscorwks!Thread::UserResumeThread+fb
mscorwks!Thread::DoADCallBack+355
mscorwks!Thread::DoADCallBack+541
mscorwks!Thread::UserResumeThread+a6
mscorwks!Thread::DoADCallBack+d9
mscorwks!Thread::UserResumeThread+e1
mscorwks!Thread::DoADCallBack+355
mscorwks!Thread::DoADCallBack+541
mscorwks!Thread::DoADCallBack+575
mscorwks!ManagedThreadBase::ThreadPool+13
mscorwks!AddTimerCallbackEx+6e
mscorwks!AddTimerCallback+f
mscorwks!ThreadpoolMgr::AsyncTimerCallbackCompletion+5b
mscorwks!ThreadpoolMgr::ExecuteWorkRequest+40
mscorwks!ThreadpoolMgr::WorkerThreadStart+1f2
mscorwks!Thread::intermediateThreadProc+49
kernel32!BaseThreadStart+34

Thread 35 - System ID 8200
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 5:29:37 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.609

This thread is not fully resolved and may or may not be a problem. Further
analysis of these threads may be required.

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtDelayExecution+c
kernel32!SleepEx+68
mscorwks!EESleepEx+a3
mscorwks!CExecutionEngine::ClrSleepEx+d
mscorwks!ClrSleepEx+13
mscorwks!Thread::UserSleep+63
mscorwks!ThreadNative::Sleep+ce
0x058ff970
mscorlib_ni+57262a

Thread 36 - System ID 1324
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 5:29:37 PM
Time spent in user mode 0 Days 00:03:33.625
Time spent in kernel mode 0 Days 00:00:28.156

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+6f
mscorwks!Thread::DoAppropriateAptStateWait+3c
mscorwks!Thread::DoAppropriateWaitWorker+144
mscorwks!Thread::DoAppropriateWait+40
mscorwks!CLREvent::WaitEx+f7
mscorwks!CLREvent::Wait+17
mscorwks!Thread::Wait+1b
mscorwks!Thread::Block+17
mscorwks!SyncBlock::Wait+136
mscorwks!ObjHeader::Wait+29
mscorwks!ObjectNative::WaitTimeout+e5
mscorlib_ni+3163f0
mscorlib_ni+317a7b
mscorlib_ni+317a7b
mscorwks!CallDescrWorker+33
mscorwks!CallDescrWorkerWithHandler+a3
mscorwks!MethodDesc::CallDescr+19c
mscorwks!MethodDesc::CallTargetWorker+20
mscorwks!MethodDescCallSite::Call+18
mscorwks!ExecuteCodeWithGuaranteedCleanupHelper+b2
mscorwks!ReflectionInvocation::ExecuteCodeWithGuaranteedClea nup+f9
mscorlib_ni+2a84fb
mscorlib_ni+2a83ee
mscorlib_ni+317b5c
mscorwks!CallDescrWorker+33
mscorwks!CallDescrWorkerWithHandler+a3
mscorwks!MethodDesc::CallDescr+19c
mscorwks!MethodDesc::CallTargetWorker+20
mscorwks!MethodDescCallSite::Call+18
mscorwks!ThreadNative::KickOffThread_Worker+15d
mscorwks!Thread::UserResumeThread+fb
mscorwks!Thread::DoADCallBack+355
mscorwks!Thread::DoADCallBack+541
mscorwks!Thread::UserResumeThread+a6
mscorwks!Thread::DoADCallBack+d9
mscorwks!Thread::UserResumeThread+e1
mscorwks!Thread::DoADCallBack+355
mscorwks!Thread::DoADCallBack+541
mscorwks!Thread::DoADCallBack+575
mscorwks!ManagedThreadBase::KickOff+13
mscorwks!ThreadNative::KickOffThread+230
mscorwks!Thread::intermediateThreadProc+49
kernel32!BaseThreadStart+34

Thread 37 - System ID 8996
Entry point GdiPlus!BackgroundThreadProc
Create time 9/12/2007 5:29:44 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.375

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
user32!RealMsgWaitForMultipleObjectsEx+13f
user32!MsgWaitForMultipleObjects+1d
GdiPlus!BackgroundThreadProc+5b
ntdll!NtRegisterThreadTerminatePort+c
kernel32!BaseThreadStart+34

Thread 38 - System ID 7120
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 5:29:45 PM
Time spent in user mode 0 Days 00:00:00.125
Time spent in kernel mode 0 Days 00:00:00.515

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwRemoveIoCompletion+c
kernel32!GetQueuedCompletionStatus+27
mscorwks!ThreadpoolMgr::CompletionPortThreadStart+11c
mscorwks!Thread::intermediateThreadProc+49
kernel32!BaseThreadStart+34

Thread 39 - System ID 9436
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 5:29:51 PM
Time spent in user mode 0 Days 00:00:00.031
Time spent in kernel mode 0 Days 00:00:00.578

This thread is not fully resolved and may or may not be a problem. Further
analysis of these threads may be required.

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtDelayExecution+c
kernel32!SleepEx+68
mscorwks!EESleepEx+a3
mscorwks!CExecutionEngine::ClrSleepEx+d
mscorwks!ClrSleepEx+13
mscorwks!Thread::UserSleep+63
mscorwks!ThreadNative::Sleep+ce
0x0936c000
mscorlib_ni+57262a

Thread 40 - System ID 4116
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 5:29:51 PM
Time spent in user mode 0 Days 00:01:50.625
Time spent in kernel mode 0 Days 00:00:13.859

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+6f
mscorwks!Thread::DoAppropriateAptStateWait+3c
mscorwks!Thread::DoAppropriateWaitWorker+144
mscorwks!Thread::DoAppropriateWait+40
mscorwks!CLREvent::WaitEx+f7
mscorwks!CLREvent::Wait+17
mscorwks!Thread::Wait+1b
mscorwks!Thread::Block+17
mscorwks!SyncBlock::Wait+136
mscorwks!ObjHeader::Wait+29
mscorwks!ObjectNative::WaitTimeout+e5
mscorlib_ni+3163f0
mscorlib_ni+317a7b
mscorlib_ni+317a7b
mscorwks!CallDescrWorker+33
mscorwks!CallDescrWorkerWithHandler+a3
mscorwks!MethodDesc::CallDescr+19c
mscorwks!MethodDesc::CallTargetWorker+20
mscorwks!MethodDescCallSite::Call+18
mscorwks!ExecuteCodeWithGuaranteedCleanupHelper+b2
mscorwks!ReflectionInvocation::ExecuteCodeWithGuaranteedClea nup+f9
mscorlib_ni+2a84fb
mscorlib_ni+2a83ee
mscorlib_ni+317b5c
mscorwks!CallDescrWorker+33
mscorwks!CallDescrWorkerWithHandler+a3
mscorwks!MethodDesc::CallDescr+19c
mscorwks!MethodDesc::CallTargetWorker+20
mscorwks!MethodDescCallSite::Call+18
mscorwks!ThreadNative::KickOffThread_Worker+15d
mscorwks!Thread::UserResumeThread+fb
mscorwks!Thread::DoADCallBack+355
mscorwks!Thread::DoADCallBack+541
mscorwks!Thread::UserResumeThread+a6
mscorwks!Thread::DoADCallBack+d9
mscorwks!Thread::UserResumeThread+e1
mscorwks!Thread::DoADCallBack+355
mscorwks!Thread::DoADCallBack+541
mscorwks!Thread::DoADCallBack+575
mscorwks!ManagedThreadBase::KickOff+13
mscorwks!ThreadNative::KickOffThread+230
mscorwks!Thread::intermediateThreadProc+49
kernel32!BaseThreadStart+34

Thread 41 - System ID 11024
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 5:36:01 PM
Time spent in user mode 0 Days 00:05:29.218
Time spent in kernel mode 0 Days 00:00:30.250

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
mscorwks!PEImage::LoadImage+199
mscorwks!CLREvent::WaitEx+117
mscorwks!CLREvent::Wait+17
mscorwks!ThreadpoolMgr::SafeWait+73
mscorwks!ThreadpoolMgr::EnterRetirement+8e
mscorwks!ThreadpoolMgr::WorkerThreadStart+360
mscorwks!Thread::intermediateThreadProc+49
kernel32!BaseThreadStart+34

Thread 42 - System ID 11640
Entry point comsvcs!PingThread
Create time 9/12/2007 5:57:01 PM
Time spent in user mode 0 Days 00:00:00.015
Time spent in kernel mode 0 Days 00:00:00.453

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
kernel32!WaitForSingleObject+f
comsvcs!PingThread+ea
kernel32!BaseThreadStart+34

Thread 43 - System ID 7128
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 7:42:56 PM
Time spent in user mode 0 Days 00:04:41.453
Time spent in kernel mode 0 Days 00:00:08.140

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
mscorwks!PEImage::LoadImage+199
mscorwks!CLREvent::WaitEx+117
mscorwks!CLREvent::Wait+17
mscorwks!ThreadpoolMgr::SafeWait+73
mscorwks!ThreadpoolMgr::WorkerThreadStart+f1
mscorwks!Thread::intermediateThreadProc+49
kernel32!BaseThreadStart+34

Thread 44 - System ID 11940
Entry point ntdll!RtlpWorkerThread
Create time 9/12/2007 8:11:25 PM
Time spent in user mode 0 Days 00:00:00.046
Time spent in kernel mode 0 Days 00:00:00.062

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwRemoveIoCompletion+c
ntdll!RtlpWorkerThread+3b
kernel32!BaseThreadStart+34

Thread 45 - System ID 5728
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 8:15:55 PM
Time spent in user mode 0 Days 00:00:32.328
Time spent in kernel mode 0 Days 00:00:03.578

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+6f
mscorwks!Thread::DoAppropriateAptStateWait+3c
mscorwks!Thread::DoAppropriateWaitWorker+144
mscorwks!Thread::DoAppropriateWait+40
mscorwks!CLREvent::WaitEx+f7
mscorwks!ADUnloadSink::WaitUnloadCompletion+b
mscorwks!AppDomain::UnloadWait+67
mscorwks!AppDomain::UnloadById+126
mscorwks!AppDomainNative::Unload+a0
mscorlib_ni+2d9c8c
System_Web_ni+e14e4
mscorlib_ni+31912f
mscorlib_ni+2a83dd
mscorlib_ni+319218
mscorwks!CallDescrWorker+33
mscorwks!CallDescrWorkerWithHandler+a3
mscorwks!DispatchCallBody+1e
mscorwks!DispatchCallDebuggerWrapper+3d
mscorwks!DispatchCallNoEH+51
mscorwks!Holder *,&AcquireDelegateInfo,&ReleaseDelegateInfo,0,&CompareDefaul t *>,2>::~Holder *,&AcquireDelegateInfo,&ReleaseDelegateInfo,0,&CompareDefaul t *>,2>+bb
mscorwks!Thread::UserResumeThread+fb
mscorwks!Thread::DoADCallBack+355
mscorwks!Thread::DoADCallBack+541
mscorwks!Thread::UserResumeThread+a6
mscorwks!Thread::DoADCallBack+d9
mscorwks!Thread::UserResumeThread+e1
mscorwks!Thread::DoADCallBack+355
mscorwks!Thread::DoADCallBack+541
mscorwks!Thread::DoADCallBack+575
mscorwks!ManagedThreadBase::ThreadPool+13
mscorwks!QueueUserWorkItemCallback+9d
mscorwks!ThreadpoolMgr::ExecuteWorkRequest+40
mscorwks!ThreadpoolMgr::WorkerThreadStart+1f2
mscorwks!Thread::intermediateThreadProc+49
kernel32!BaseThreadStart+34

Thread 46 - System ID 11536
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 8:16:57 PM
Time spent in user mode 0 Days 00:51:01.062
Time spent in kernel mode 0 Days 00:36:31.265

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtDelayExecution+c
kernel32!SleepEx+68
mscorwks!EESleepEx+a3
mscorwks!CExecutionEngine::ClrSleepEx+d
mscorwks!ClrSleepEx+13
mscorwks!Thread::UserSleep+63
mscorwks!ThreadNative::Sleep+ce
System_Web_ni+f1198
mscorlib_ni+31912f
mscorlib_ni+2a83dd
mscorlib_ni+319218
mscorwks!CallDescrWorker+33
mscorwks!CallDescrWorkerWithHandler+a3
mscorwks!DispatchCallBody+1e
mscorwks!DispatchCallDebuggerWrapper+3d
mscorwks!DispatchCallNoEH+51
mscorwks!Holder *,&AcquireDelegateInfo,&ReleaseDelegateInfo,0,&CompareDefaul t *>,2>::~Holder *,&AcquireDelegateInfo,&ReleaseDelegateInfo,0,&CompareDefaul t *>,2>+bb
mscorwks!Thread::UserResumeThread+fb
mscorwks!Thread::DoADCallBack+355
mscorwks!Thread::DoADCallBack+541
mscorwks!Thread::UserResumeThread+a6
mscorwks!Thread::DoADCallBack+d9
mscorwks!Thread::UserResumeThread+e1
mscorwks!Thread::DoADCallBack+355
mscorwks!Thread::DoADCallBack+541
mscorwks!Thread::DoADCallBack+575
mscorwks!ManagedThreadBase::ThreadPool+13
mscorwks!QueueUserWorkItemCallback+9d
mscorwks!ThreadpoolMgr::ExecuteWorkRequest+40
mscorwks!ThreadpoolMgr::WorkerThreadStart+1f2
mscorwks!Thread::intermediateThreadProc+49
kernel32!BaseThreadStart+34

Thread 47 - System ID 8856
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 8:33:50 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.00

This thread is not fully resolved and may or may not be a problem. Further
analysis of these threads may be required.

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtDelayExecution+c
kernel32!SleepEx+68
mscorwks!EESleepEx+a3
mscorwks!CExecutionEngine::ClrSleepEx+d
mscorwks!ClrSleepEx+13
mscorwks!Thread::UserSleep+63
mscorwks!ThreadNative::Sleep+ce
0x3de3f270
mscorlib_ni+57262a

Thread 48 - System ID 8196
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 8:33:50 PM
Time spent in user mode 0 Days 00:00:00.859
Time spent in kernel mode 0 Days 00:00:00.093

Function Source
SharedUserData!SystemCallStub+4
ntdll!NtWaitForMultipleObjects+c
kernel32!WaitForMultipleObjectsEx+11a
mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+6f
mscorwks!Thread::DoAppropriateAptStateWait+3c
mscorwks!Thread::DoAppropriateWaitWorker+144
mscorwks!Thread::DoAppropriateWait+40
mscorwks!CLREvent::WaitEx+f7
mscorwks!CLREvent::Wait+17
mscorwks!Thread::Wait+1b
mscorwks!Thread::Block+17
mscorwks!SyncBlock::Wait+136
mscorwks!ObjHeader::Wait+29
mscorwks!ObjectNative::WaitTimeout+e5
mscorlib_ni+3163f0
mscorlib_ni+317a7b
mscorlib_ni+317a7b
mscorwks!CallDescrWorker+33
mscorwks!CallDescrWorkerWithHandler+a3
mscorwks!MethodDesc::CallDescr+19c
mscorwks!MethodDesc::CallTargetWorker+20
mscorwks!MethodDescCallSite::Call+18
mscorwks!ExecuteCodeWithGuaranteedCleanupHelper+b2
mscorwks!ReflectionInvocation::ExecuteCodeWithGuaranteedClea nup+f9
mscorlib_ni+2a84fb
mscorlib_ni+2a83ee
mscorlib_ni+317b5c
mscorwks!CallDescrWorker+33
mscorwks!CallDescrWorkerWithHandler+a3
mscorwks!MethodDesc::CallDescr+19c
mscorwks!MethodDesc::CallTargetWorker+20
mscorwks!MethodDescCallSite::Call+18
mscorwks!ThreadNative::KickOffThread_Worker+15d
mscorwks!Thread::UserResumeThread+fb
mscorwks!Thread::DoADCallBack+355
mscorwks!Thread::DoADCallBack+541
mscorwks!Thread::UserResumeThread+a6
mscorwks!Thread::DoADCallBack+d9
mscorwks!Thread::UserResumeThread+e1
mscorwks!Thread::DoADCallBack+355
mscorwks!Thread::DoADCallBack+541
mscorwks!Thread::DoADCallBack+575
mscorwks!ManagedThreadBase::KickOff+13
mscorwks!ThreadNative::KickOffThread+230
mscorwks!Thread::intermediateThreadProc+49
kernel32!BaseThreadStart+34

Thread 49 - System ID 9072
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 8:34:49 PM
Time spent in user mode 0 Days 00:00:03.156
Time spent in kernel mode 0 Days 00:00:00.406

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
mscorwks!PEImage::LoadImage+199
mscorwks!CLREvent::WaitEx+117
mscorwks!CLREvent::Wait+17
mscorwks!ThreadpoolMgr::SafeWait+73
mscorwks!ThreadpoolMgr::WorkerThreadStart+f1
mscorwks!Thread::intermediateThreadProc+49
kernel32!BaseThreadStart+34

Thread 50 - System ID 7892
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 8:34:51 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.00

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwRemoveIoCompletion+c
kernel32!GetQueuedCompletionStatus+27
mscorwks!ThreadpoolMgr::CompletionPortThreadStart+11c
mscorwks!Thread::intermediateThreadProc+49
kernel32!BaseThreadStart+34

Thread 51 - System ID 8316
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 8:34:59 PM
Time spent in user mode 0 Days 00:00:03.062
Time spent in kernel mode 0 Days 00:00:00.218

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
mscorwks!PEImage::LoadImage+199
mscorwks!CLREvent::WaitEx+117
mscorwks!CLREvent::Wait+17
mscorwks!ThreadpoolMgr::SafeWait+73
mscorwks!ThreadpoolMgr::EnterRetirement+8e
mscorwks!ThreadpoolMgr::WorkerThreadStart+360
mscorwks!Thread::intermediateThreadProc+49
kernel32!BaseThreadStart+34

Thread 52 - System ID 9488
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 8:35:00 PM
Time spent in user mode 0 Days 00:00:05.265
Time spent in kernel mode 0 Days 00:00:00.343

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
mscorwks!PEImage::LoadImage+199
mscorwks!CLREvent::WaitEx+117
mscorwks!CLREvent::Wait+17
mscorwks!ThreadpoolMgr::SafeWait+73
mscorwks!ThreadpoolMgr::WorkerThreadStart+f1
mscorwks!Thread::intermediateThreadProc+49
kernel32!BaseThreadStart+34

Thread 53 - System ID 8432
Entry point ole32!CRpcThreadCache::RpcWorkerThreadEntry
Create time 9/12/2007 8:35:05 PM
Time spent in user mode 0 Days 00:00:00.046
Time spent in kernel mode 0 Days 00:00:00.00

Function Source
SharedUserData!SystemCallStub+4
user32!NtUserGetMessage+c
user32!GetMessageW+31
ole32!CDllHost::STAWorkerLoop+70
ole32!CDllHost::WorkerThread+c6
ole32!DLLHostThreadEntry+9
ole32!CRpcThread::WorkerLoop+1e
ole32!CRpcThreadCache::RpcWorkerThreadEntry+1f
kernel32!BaseThreadStart+34

Thread 54 - System ID 8052
Entry point mscorwks!Thread::intermediateThreadProc
Create time 9/12/2007 8:40:00 PM
Time spent in user mode 0 Days 00:00:00.031
Time spent in kernel mode 0 Days 00:00:00.00

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwWaitForSingleObject+c
kernel32!WaitForSingleObjectEx+ac
mscorwks!PEImage::LoadImage+199
mscorwks!CLREvent::WaitEx+117
mscorwks!CLREvent::Wait+17
mscorwks!ThreadpoolMgr::SafeWait+73
mscorwks!ThreadpoolMgr::CompletionPortThreadStart+24d
mscorwks!Thread::intermediateThreadProc+49
kernel32!BaseThreadStart+34

Thread 55 - System ID 7564
Entry point ntdll!RtlpWorkerThread
Create time 9/12/2007 10:14:00 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.00

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwRemoveIoCompletion+c
ntdll!RtlpWorkerThread+3b
kernel32!BaseThreadStart+34

Thread 56 - System ID 9208
Entry point ntdll!RtlpWorkerThread
Create time 9/12/2007 10:16:00 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.00

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwRemoveIoCompletion+c
ntdll!RtlpWorkerThread+3b
kernel32!BaseThreadStart+34

Thread 57 - System ID 6580
Entry point ntdll!RtlpWorkerThread
Create time 9/12/2007 10:17:00 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.00

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwRemoveIoCompletion+c
ntdll!RtlpWorkerThread+3b
kernel32!BaseThreadStart+34

Thread 58 - System ID 11844
Entry point ntdll!RtlpWorkerThread
Create time 9/12/2007 10:18:00 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.00

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwRemoveIoCompletion+c
ntdll!RtlpWorkerThread+3b
kernel32!BaseThreadStart+34

Thread 59 - System ID 5652
Entry point ntdll!RtlpWorkerThread
Create time 9/12/2007 10:18:00 PM
Time spent in user mode 0 Days 00:00:00.00
Time spent in kernel mode 0 Days 00:00:00.00

Function Source
SharedUserData!SystemCallStub+4
ntdll!ZwRemoveIoCompletion+c
ntdll!RtlpWorkerThread+3b
kernel32!BaseThreadStart+34

Well-Known COM STA Threads Report
STA Name Thread ID Thread Status Call Status

Main STA 20 In-Call (bad symbols) not fully resolved and may or may
not be a problem. Further analysis of this thread may be required
Apartment-threaded host for MTA clients 53 In-Call (bad symbols)

HTTP report
IIS ATQ worker thread count 4 Thread(s)
Available ATQ worker thread count 4 Thread(s)
Active client connections 2 client connection(s)

Client Connections
Client connection from 12.14.115.254:47437 to 204.26.1.231:443
Host Header apps.imercer.com:443
POST request for /ePRISM/FileManager/UploadFilePopup.uplx
HTTP Version HTTP/1.1
SSL Request False
Time alive 04:30:26
QueryString
Request mapped to
HTTP Request State Unknown
Native Request State NREQ_STATE_PROCESS

Client connection from 198.29.191.149:50215 to 204.26.1.231:443
Host Header apps.imercer.com:443
POST request for /ePRISM/FileManager/UploadFilePopup.uplx
HTTP Version HTTP/1.1
SSL Request False
Time alive 04:08:06
QueryString
Request mapped to
HTTP Request State Unknown
Native Request State NREQ_STATE_PROCESS

Re: App Pool Hang

Damien,

Already gave a clue on another thread on using SOS to diagnose those
issues.
Please see:
http://groups.google.com/group/microsoft.public.inetserver.i is/browse_thread/thread/43db9307c23d5500/c9a9f17071adfed2#c9 a9f17071adfed2

Tiago Halm
http://www.codeplex.com/filterdotnet
http://www.codeplex.com/iisshield

Re: App Pool Hang

Good post on that tool but the dump looks almost the same as the debugdiag
tool's report. Any thoughts on what I should be looking for?

Re: App Pool Hang

1) You need symbols - if you have a dump file, WinDBG can do this for you.
Without them, you won't get clean thread stacks (incorrect functions,etc.)
making analysis difficult
2) Most likely the issue is either a deadlock (in which case you actually
are experiencing a hang) or severe serialization (which makes things go so
slow, that they appear hung). I would say you are probably hitting a
serialization issue. Where & How I cannot say w/out clean thread stacks.
3) You need good symbols for SOS to work (which will get you the managed
stacks).


Pat

"Damien Kalvar" wrote in message
news:ABC2B34F-8E53-4509-82AF-679D02F3B941@microsoft.com...
> Good post on that tool but the dump looks almost the same as the debugdiag
> tool's report. Any thoughts on what I should be looking for?

Re: App Pool Hang

Guys thanks for the suggestions, I found a good PDF explaining how to use
windbg and sos so I'll be able to use the tools now.

We hadn't been looking at a serialization issue but the site does use the
state server and the developers who worked on the site before me are throwing
a ton of information in session state. But its been running that way for a
while, are there any changes from 1.1 to 2.0 that might cause what we have
been doing to suddenly becomre a problem?

Re: App Pool Hang

Using the sos I was able to get these interesting thread stacks

OS Thread Id: 0x1970 (20)
ESP EIP
0281eefc 7ffe0304 [HelperMethodFrame: 0281eefc]
System.GC.WaitForPendingFinalizers()
0281ef4c 3198d4a4 SoftArtisans.Net.FileUpEe.Finalize()
0281ef78 3198d442 SoftArtisans.Net.FileUpEe.__dtor()
0281ef80 3198d415 SoftArtisans.Net.FileUpEe.Dispose()
0281ef88 0fddf14d
Mercer.ePRISM.Web.UI.FileManager.FileUploadUtility.ProcessUp load(System.String, System.String, System.Web.HttpRequest)
0281efdc 0fddd8e7
Mercer.ePRISM.Web.UI.FileManager.UserControls.UploadFileCont rol.Page_Load(System.Object, System.EventArgs)
0281f048 687ca5a4 System.Web.UI.Control.OnLoad(System.EventArgs).....


OS Thread Id: 0x1b54 (31)
ESP EIP
0417ed7c 7ffe0304 [HelperMethodFrame: 0417ed7c]
System.GC.WaitForPendingFinalizers()
0417edcc 3198d4a4 SoftArtisans.Net.FileUpEe.Finalize()
0417edf8 3198d442 SoftArtisans.Net.FileUpEe.__dtor()
0417ee00 3198d415 SoftArtisans.Net.FileUpEe.Dispose()
0417ee08 0fddf14d
Mercer.ePRISM.Web.UI.FileManager.FileUploadUtility.ProcessUp load(System.String, System.String, System.Web.HttpRequest)
0417ee5c 0fddd8e7
Mercer.ePRISM.Web.UI.FileManager.UserControls.UploadFileCont rol.Page_Load(System.Object, System.EventArgs)
0417eec8 687ca5a4 System.Web.UI.Control.OnLoad(System.EventArgs).......

These aren't the full stacks but does anyone have some suggestions on how to
find out what the WaitForPendingFinalizers funtions are waiting on?

Re: App Pool Hang

Google is your friend and MSDN too.

Look here:
http://msdn2.microsoft.com/en-us/library/system.gc.waitforpe ndingfinalizers.aspx

Use Reflector and inspect if upon calling
SoftArtisans.Net.FileUpEe.Dispose

the GC.SuppressFinalize method is called too
http://msdn2.microsoft.com/en-us/library/system.gc.suppressf inalize.aspx

It should be a good indicator of what is causing the AppPool to hang.

Tiago Halm

Re: App Pool Hang

yeah, I had seen all that already and used reflector to dig into the Fileup
code. I guess what I was asking for is if anyone could help out with how to
use the windbg tool to see what the waitforpendingfinalizers is waiting on.

Re: App Pool Hang

Those threads are blocked on GC - but you need to find the thread that is
actually doing the blocking (i.e. owns the lock).


Pat

"Damien Kalvar" wrote in message
news:31521D9C-0945-48D8-AACC-160A8B934B59@microsoft.com...
> Using the sos I was able to get these interesting thread stacks
>
> OS Thread Id: 0x1970 (20)
> ESP EIP
> 0281eefc 7ffe0304 [HelperMethodFrame: 0281eefc]
> System.GC.WaitForPendingFinalizers()
> 0281ef4c 3198d4a4 SoftArtisans.Net.FileUpEe.Finalize()
> 0281ef78 3198d442 SoftArtisans.Net.FileUpEe.__dtor()
> 0281ef80 3198d415 SoftArtisans.Net.FileUpEe.Dispose()
> 0281ef88 0fddf14d
> Mercer.ePRISM.Web.UI.FileManager.FileUploadUtility.ProcessUp load(System.String,
> System.String, System.Web.HttpRequest)
> 0281efdc 0fddd8e7
> Mercer.ePRISM.Web.UI.FileManager.UserControls.UploadFileCont rol.Page_Load(System.Object,
> System.EventArgs)
> 0281f048 687ca5a4 System.Web.UI.Control.OnLoad(System.EventArgs).....
>
>
> OS Thread Id: 0x1b54 (31)
> ESP EIP
> 0417ed7c 7ffe0304 [HelperMethodFrame: 0417ed7c]
> System.GC.WaitForPendingFinalizers()
> 0417edcc 3198d4a4 SoftArtisans.Net.FileUpEe.Finalize()
> 0417edf8 3198d442 SoftArtisans.Net.FileUpEe.__dtor()
> 0417ee00 3198d415 SoftArtisans.Net.FileUpEe.Dispose()
> 0417ee08 0fddf14d
> Mercer.ePRISM.Web.UI.FileManager.FileUploadUtility.ProcessUp load(System.String,
> System.String, System.Web.HttpRequest)
> 0417ee5c 0fddd8e7
> Mercer.ePRISM.Web.UI.FileManager.UserControls.UploadFileCont rol.Page_Load(System.Object,
> System.EventArgs)
> 0417eec8 687ca5a4 System.Web.UI.Control.OnLoad(System.EventArgs).......
>
> These aren't the full stacks but does anyone have some suggestions on how
> to
> find out what the WaitForPendingFinalizers funtions are waiting on?
>
>