IIS6 ASP requests increasing, recycle WP and sessions lost, 2262

This seems to be difficult to find information on!

I am getting event 2262 :
ISAPI 'C:\WINDOWS\system32\inetsrv\asp.dll' reported itself as
unhealthy for
the following reason: 'ASP unhealthy because 100% of executing
requests are
hung and 8% of the request queue is full.'.

After recently moving a website ( mainly classic ASP pages) from from
IIS6 running in IIS5 mode, to IIS6 native mode, I am getting sessions
lost and users needing to log in again due to Worker Process
recycling. This is happening several times during the day, and was not
a problem in iis5 mode.
Watching the requests in IIS6 Request Viewer, and the ASP request
Queued count in performance monitor, it is easy to see the requests
slowly queuing up, and not releasing. Sometimes a single client(IP
addr) shows as having multiple requests running against them for
different ASP pages, all with a status of processing.

Recycling happens anywhere between 40 and 140 queued requests, it is
hard to replicate the exact cause, but only seems to happen under
load. CPU and Memory usage of the Worker Process seems to be fairly
stable.

I have gone through the pages displayed as being processing in the IIS
request viewer. I have been through the code, and closed of recordsets
and database connections ( and set to nothing) and that have not been
explicitly closed. This has made no difference or reduced the
frequency of the WP recycling. These are common pages that all users
load.
I do have an occasional ASPEXEC call which gets stuck in Task Manager
from a report run, and there can sometime be multiple instances of
this "stuck" in task manager. However, when I have tried to recreate
this situation, the ASPEXEC call works correctly, and clears itself
from both task manager and the ASP Request queue as expected.
Pages that show as being stuck in the queue seem to work correctly
most of the time.

There is also an ISAPI filter to a mapping engine which shows in the
IIS6 ASP queues, but this does not enter into the ASP queues in
perfmon, and should not be related to the 2262 error message.
I have run IISState several time against the WP, sometimes it does not
complete, and each time it causes a recylce of the process, causing
another logoff, so I try to anticipate a recycle, and do it before the
recycle occurs (is it supposed to cause session loss and recycle of
WP?)

Several questions arise:
Is it possible to clear distinct requests from within the ASP queue in
the WP?
Does the Thread count settings for ASP.NEt affect ASP threads as well?
Should IISState cause a recycle?
How can I stop the queuing/recycling?

Attached is the last IISState dump, at the time, the ASP request Queue
in perfmon showed around 160 queued requests.

Opened log file 'C:\iisstate\output\IISState-7396.log'

***********************
Starting new log output
IISState version 3.3.1

Thu Nov 29 12:59:42 2007

OS = Windows 2003 Server
Executable: w3wp.exe
PID = 7396

Note: Thread times are formatted as HH:MM:SS.ms

***********************




Thread ID: 0
System Thread ID: 2b8
Kernel Time: 0:0:0.62
User Time: 0:0:0.62
Thread Status: Thread is in a WAIT state.
Thread Type: HTTP Compression Thread
# ChildEBP RetAddr
00 0007fc08 7c827d0b ntdll!KiFastSystemCallRet
01 0007fc0c 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 0007fc7c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 0007fc90 5a364662 kernel32!WaitForSingleObject+0x12
04 0007fca0 5a366e3f w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
05 0007fca8 5a3af42d w3dt!UlAtqStartListen+0x2d
06 0007fcb8 5a3bc335 w3core!W3_SERVER::StartListen+0xbd
07 0007ff0c 0100187c w3core!UlW3Start+0x26e
08 0007ff44 01001a27 w3wp!wmain+0x22a
09 0007ffc0 77e6f23b w3wp!wmainCRTStartup+0x12f
0a 0007fff0 00000000 kernel32!BaseProcessStart+0x23




Thread ID: 1
System Thread ID: 2d0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 00ccff9c 7c826f4b ntdll!KiFastSystemCallRet
01 00ccffa0 7c83d424 ntdll!NtDelayExecution+0xc
02 00ccffb8 77e64829 ntdll!RtlpTimerThread+0x47
03 00ccffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 2
System Thread ID: 1990
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: Unable to locate ASP page

No remote call being made

# ChildEBP RetAddr
00 00d7fe18 7c82783b ntdll!KiFastSystemCallRet
01 00d7fe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
02 00d7ff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
03 00d7ff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
04 00d7ffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
05 00d7ffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
06 00d7ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 3
System Thread ID: dcc
Kernel Time: 0:0:0.390
User Time: 0:0:0.359
*** WARNING: Unable to verify checksum for
*** ERROR: Symbol file could not be found. Defaulted to export
symbols for -
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 00f7eb80 7c827d0b ntdll!KiFastSystemCallRet
01 00f7eb84 71b21af5 ntdll!NtWaitForSingleObject+0xc
02 00f7ebc0 71b2c517 mswsock!SockWaitForSingleObject+0x19d
03 00f7ec38 71c02fee mswsock!WSPRecv+0x203
04 00f7ec80 6a6b8cce WS2_32!recv+0x83
WARNING: Stack unwind information not available. Following frames may
be wrong.
05 00f7ed3c 01908568 isapi_redirect+0x8cce
06 01908550 06062ff0 0x1908568
07 01908534 00000058 0x6062ff0




Thread ID: 4
System Thread ID: 1d38
Kernel Time: 0:0:0.125
User Time: 0:0:0.171
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 00fbeb80 7c827d0b ntdll!KiFastSystemCallRet
01 00fbeb84 71b21af5 ntdll!NtWaitForSingleObject+0xc
02 00fbebc0 71b2c517 mswsock!SockWaitForSingleObject+0x19d
03 00fbec38 71c02fee mswsock!WSPRecv+0x203
04 00fbec80 6a6b8cce WS2_32!recv+0x83
WARNING: Stack unwind information not available. Following frames may
be wrong.
05 00fbed3c 019044c8 isapi_redirect+0x8cce
06 019044b0 06075258 0x19044c8
07 01904494 00000058 0x6075258




Thread ID: 5
System Thread ID: 1ffc
Kernel Time: 0:0:0.968
User Time: 0:0:0.968
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 00ffeb80 7c827d0b ntdll!KiFastSystemCallRet
01 00ffeb84 71b21af5 ntdll!NtWaitForSingleObject+0xc
02 00ffebc0 71b2c517 mswsock!SockWaitForSingleObject+0x19d
03 00ffec38 71c02fee mswsock!WSPRecv+0x203
04 00ffec80 6a6b8cce WS2_32!recv+0x83
WARNING: Stack unwind information not available. Following frames may
be wrong.
05 00ffed3c 019126f8 isapi_redirect+0x8cce
06 019126e0 06185e98 0x19126f8
07 019126c4 00000048 0x6185e98




Thread ID: 6
System Thread ID: f20
Kernel Time: 0:0:0.593
User Time: 0:0:0.328
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 0104eb80 7c827d0b ntdll!KiFastSystemCallRet
01 0104eb84 71b21af5 ntdll!NtWaitForSingleObject+0xc
02 0104ebc0 71b2c517 mswsock!SockWaitForSingleObject+0x19d
03 0104ec38 71c02fee mswsock!WSPRecv+0x203
04 0104ec80 6a6b8cce WS2_32!recv+0x83
WARNING: Stack unwind information not available. Following frames may
be wrong.
05 0104ed3c 0190c608 isapi_redirect+0x8cce
06 0190c5f0 06095010 0x190c608
07 0190c5d4 00000058 0x6095010




Thread ID: 7
System Thread ID: 174
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 0110fcec 7c827cfb ntdll!KiFastSystemCallRet
01 0110fcf0 7c83c78e ntdll!NtWaitForMultipleObjects+0xc
02 0110ffb8 77e64829 ntdll!RtlpWaitThread+0x161
03 0110ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 8
System Thread ID: bf8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: HTTP Compression Thread
# ChildEBP RetAddr
00 01adfa84 7c827d0b ntdll!KiFastSystemCallRet
01 01adfa88 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 01adfaf8 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 01adfb0c 5a3b820f kernel32!WaitForSingleObject+0x12
04 01adffb8 77e64829 w3core!HTTP_COMPRESSION::CompressionThread+0x126
05 01adffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 9
System Thread ID: 2cc
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: ASP
Executing Page: Unable to locate ASP page

# ChildEBP RetAddr
00 0293ff0c 7c827d0b ntdll!KiFastSystemCallRet
01 0293ff10 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 0293ff80 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 0293ff94 709fc803 kernel32!WaitForSingleObject+0x12
04 0293ffb4 709fc7db asp!CApplnCleanupMgr::ApplnCleanupDoWork+0x1e
05 0293ffb8 77e64829 asp!CApplnCleanupMgr::ApplnCleanupThread+0xa
06 0293ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 10
System Thread ID: 1af4
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: ASP
Executing Page: Unable to locate ASP page

# ChildEBP RetAddr
00 029ffe1c 7c827cfb ntdll!KiFastSystemCallRet
01 029ffe20 77e6202c ntdll!NtWaitForMultipleObjects+0xc
02 029ffec8 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
03 029fff24 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
04 029fff40 709fb8be USER32!MsgWaitForMultipleObjects+0x1f
05 029fff84 77bcb530 asp!CMTACallbackThread::Thread+0x4f
06 029fffb8 77e64829 msvcrt!_endthreadex+0xa3
07 029fffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 11
System Thread ID: f64
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: ASP
Executing Page: Unable to locate ASP page

# ChildEBP RetAddr
00 02a3fef4 7c827d0b ntdll!KiFastSystemCallRet
01 02a3fef8 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 02a3ff68 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 02a3ff7c 709fe8ef kernel32!WaitForSingleObject+0x12
04 02a3ffb8 77e64829 asp!CViperReqManager::WatchThread+0x68
05 02a3ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 12
System Thread ID: 1858
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Idle ASP thread
# ChildEBP RetAddr
00 02a9fee8 7c827d0b ntdll!KiFastSystemCallRet
01 02a9feec 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 02a9ff5c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 02a9ff70 4a77921b kernel32!WaitForSingleObject+0x12
04 02a9ff84 77bcb530 comsvcs!
CSTAThreadPool::LoadBalanceThreadControlLoop+0x29
05 02a9ffb8 77e64829 msvcrt!_endthreadex+0xa3
06 02a9ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 13
System Thread ID: 18c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Idle ASP thread
# ChildEBP RetAddr
00 02adfee0 7c827d0b ntdll!KiFastSystemCallRet
01 02adfee4 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 02adff54 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 02adff68 4a778cb6 kernel32!WaitForSingleObject+0x12
04 02adff84 77bcb530 comsvcs!CSTAThreadPool::KillThreadControlLoop
+0x25
05 02adffb8 77e64829 msvcrt!_endthreadex+0xa3
06 02adffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 14
System Thread ID: 1744
Kernel Time: 0:0:1.562
User Time: 0:0:2.78
Thread Status: Thread is in a WAIT state.
Thread Type: Idle ASP thread
# ChildEBP RetAddr
00 02b1fdcc 7c827cfb ntdll!KiFastSystemCallRet
01 02b1fdd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
02 02b1fe78 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
03 02b1fed4 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
04 02b1fef0 4a77cb28 USER32!MsgWaitForMultipleObjects+0x1f
05 02b1ff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x1f9
06 02b1ffb8 77e64829 msvcrt!_endthreadex+0xa3
07 02b1ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 15
System Thread ID: 1bd8
Kernel Time: 0:0:2.937
User Time: 0:0:2.296
Thread Status: Thread is in a WAIT state.
Thread Type: Idle ASP thread
# ChildEBP RetAddr
00 02b5fdcc 7c827cfb ntdll!KiFastSystemCallRet
01 02b5fdd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
02 02b5fe78 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
03 02b5fed4 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
04 02b5fef0 4a77cb28 USER32!MsgWaitForMultipleObjects+0x1f
05 02b5ff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x1f9
06 02b5ffb8 77e64829 msvcrt!_endthreadex+0xa3
07 02b5ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 16
System Thread ID: 1760
Kernel Time: 0:0:2.640
User Time: 0:0:3.140
Thread Status: Thread is in a WAIT state.
Thread Type: Idle ASP thread
# ChildEBP RetAddr
00 02b9fdcc 7c827cfb ntdll!KiFastSystemCallRet
01 02b9fdd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
02 02b9fe78 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
03 02b9fed4 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
04 02b9fef0 4a77cb28 USER32!MsgWaitForMultipleObjects+0x1f
05 02b9ff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x1f9
06 02b9ffb8 77e64829 msvcrt!_endthreadex+0xa3
07 02b9ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 17
System Thread ID: 88
Kernel Time: 0:0:3.734
User Time: 0:0:3.265
*** WARNING: Unable to verify checksum for C:\Inetpub\Aspexec
\ASPEXEC.DLL
*** ERROR: Symbol file could not be found. Defaulted to export
symbols for C:\Inetpub\Aspexec\ASPEXEC.DLL -
Thread Type: ASP
Executing Page: ??C:\INETPUB\PRODUCTION\TRACKING\REPORTS
\REPORTTEMPLATE.ASP

# ChildEBP RetAddr
00 02bdb118 7c82776b ntdll!KiFastSystemCallRet
01 02bdb11c 77e419d3 ntdll!NtReadFile+0xc
02 02bdb184 06912071 kernel32!ReadFile+0x10d
WARNING: Stack unwind information not available. Following frames may
be wrong.
03 02bdf260 06912951 ASPEXEC!DllUnregisterServer+0x33635
04 02bdf28c 77d05186 ASPEXEC!DllUnregisterServer+0x33f15
05 02bdf2a8 77d14acf OLEAUT32!DispCallFunc+0x16a
06 02bdf338 068dd5c5 OLEAUT32!CTypeInfo2::Invoke+0x234
07 02bdf364 7347d3cb ASPEXEC+0xd5c5
08 02bdf3b8 73468ad1 vbscript!CatchIDispatchInvoke+0x46
09 02bdf3f8 73468a40 vbscript!IDispatchInvoke2+0xaf
0a 02bdf434 734689f2 vbscript!IDispatchInvoke+0x59
0b 02bdf548 7346613b vbscript!InvokeDispatch+0x13a
0c 02bdf56c 73466fee vbscript!InvokeByName+0x42
0d 02bdf848 734633e0 vbscript!CScriptRuntime::Run+0x1331
0e 02bdf940 734637d1 vbscript!CScriptEntryPoint::Call+0x5c
0f 02bdf998 73463b9c vbscript!CSession::Execute+0xb4
10 02bdf9e8 73461849 vbscript!COleScript::ExecutePendingScripts+0x13e
11 02bdfa04 709e2ada vbscript!COleScript::SetScriptState+0x150
12 02bdfa30 709e2a9c asp!CActiveScriptEngine::TryCall+0x19
13 02bdfa6c 709e26d0 asp!CActiveScriptEngine::Call+0x31
14 02bdfa88 709e25d4 asp!CallScriptFunctionOfEngine+0x5b
15 02bdfadc 709e24ff asp!ExecuteRequest+0x17e
16 02bdfb44 709e23f7 asp!Execute+0x249
17 02bdfb98 709e2753 asp!CHitObj::ViperAsyncCallback+0x3f3
18 02bdfbb4 4a77b5ea asp!CViperAsyncRequest::OnCall+0x92
19 02bdfbd0 77720d30 comsvcs!CSTAActivityWork::STAActivityWorkHelper
+0x32
1a 02bdfc1c 777217dc ole32!EnterForCallback+0xc4
1b 02bdfd7c 776f03b4 ole32!SwitchForCallback+0x1a3
1c 02bdfda8 7769c194 ole32!PerformCallback+0x54
1d 02bdfe40 7772433a ole32!CObjectContext::InternalContextCallback
+0x159
1e 02bdfe60 4a77b78c ole32!CObjectContext::DoCallback+0x1c
1f 02bdfecc 4a77bcf2 comsvcs!CSTAActivityWork::DoWork+0x12d
20 02bdfee4 4a77c7de comsvcs!CSTAThread::DoWork+0x18
21 02bdff04 4a77cabf comsvcs!CSTAThread::ProcessQueueWork+0x37
22 02bdff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x190
23 02bdffb8 77e64829 msvcrt!_endthreadex+0xa3
24 02bdffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 18
System Thread ID: e70
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 0315ff88 7c827cfb ntdll!KiFastSystemCallRet
01 0315ff8c 71c51108 ntdll!NtWaitForMultipleObjects+0xc
02 0315ffb8 77e64829 NETAPI32!NetbiosWaiter+0x73
03 0315ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 19
System Thread ID: 1f94
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Idle ASP thread
# ChildEBP RetAddr
00 0319fd00 7c827d0b ntdll!KiFastSystemCallRet
01 0319fd04 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 0319fd74 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 0319fd88 4a756c3b kernel32!WaitForSingleObject+0x12
04 0319ffb8 77e64829 comsvcs!PingThread+0xf6
05 0319ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 20
System Thread ID: 2c8
Kernel Time: 0:0:1.984
User Time: 0:0:1.484
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 0332eb80 7c827d0b ntdll!KiFastSystemCallRet
01 0332eb84 71b21af5 ntdll!NtWaitForSingleObject+0xc
02 0332ebc0 71b2c517 mswsock!SockWaitForSingleObject+0x19d
03 0332ec38 71c02fee mswsock!WSPRecv+0x203
04 0332ec80 6a6b8cce WS2_32!recv+0x83
WARNING: Stack unwind information not available. Following frames may
be wrong.
05 0332ed3c 01916798 isapi_redirect+0x8cce
06 01916780 061be940 0x1916798
07 01916764 00000058 0x61be940




Thread ID: 21
System Thread ID: 13f4
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Managed Thread. Possible ASP.Net page or other .Net
worker
SaveModuleToMemory failed
Failed to load SOS data.
No valid SOS data table found.


Begin System Thread Information

# ChildEBP RetAddr
00 0384fea8 00000000 ntdll!KiFastSystemCallRet
GetContextState failed, 0x8007001F
GetContextState failed, 0x8007001F
GetContextState failed, 0x8007001F




Thread ID: 22
System Thread ID: 0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
GetContextState failed, 0x8007001F

Re: IIS6 ASP requests increasing, recycle WP and sessions lost, 2262

The reason information seems "hard" to find is because almost
everyone's web application is unique. Thus, platforms like Windows/IIS/
ASP/ASP.Net can only provide generic events of things going awry, but
the root cause is often specific to the application. Therefore,
"solutions" from one person rarely apply to another automatically --
one must debug the application to figure out the corrective action and
only if the cause is deemed identical through debugging will the
solution work -- one cannot just search for EventLog IDs and match
symptoms to resolutions because those IDs are generic and do not
uniquely identify the issue. This is a common fallacy.

In your case, it seems like you are happy with IIS6 in IIS5
Compatibility Mode, so why are you changing modes?


> Is it possible to clear distinct requests from within the
> ASP queue in the WP?

You cannot nuke requests from the ASP queue. And if the ASP
application is problem-free, you wouldn't need to do this.


> Does the Thread count settings for ASP.NEt affect
> ASP threads as well?

ASP.Net and ASP are completely different technologies which do not
share any common configuration. Thus, changing thread counts for
ASP.Net does not affect ASP.

FYI: Almost all of the time, when someone wants to change thread
count, it means the web application has a performance flaw. A common
fallacy is that adding threads improve performance. Adding threads
just increase concurrency (simultaneous execution of work), not
improve throughput (quicker completion of work). Thus, properly
performing and scaling applications typically use a fixed number of
threads asynchronously, and that number tends to be small but can
depend on the application.

Which means if one wants to increase # of threads, it either means one
doesn't know the fixed number for threads, or the application has
contention/latency which one tries to work around with concurrency.
Most people think the former is their problem, but the latter tends to
be reality.


> Should IISState cause a recycle?

No, but it is possible that your AppPool configuration is misreacting
to IIS State and prematurely recycling the application pool.


> How can I stop the queuing/recycling?

Determine the cause of the queuing in your application and fixing it.


For example, ASP changes to MTA mode in IIS6 Worker Process Isolation
Mode and STA in IIS5 Compatibility mode. Perhaps your application
actually works better in STA mode because ASPEXEC needs it and is
currently serializing ASP.

http://technet2.microsoft.com/windowsserver/en/library/4fd0b 346-c1d7-4042-9b6d-d6d7ce5810511033.mspx?mfr=true


//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//







On Nov 28, 9:09 pm, sh...@smarttrack.com.au wrote:
> This seems to be difficult to find information on!
>
> I am getting event 2262 :
> ISAPI 'C:\WINDOWS\system32\inetsrv\asp.dll' reported itself as
> unhealthy for
> the following reason: 'ASP unhealthy because 100% of executing
> requests are
> hung and 8% of the request queue is full.'.
>
> After recently moving a website ( mainly classic ASP pages) from from
> IIS6 running in IIS5 mode, to IIS6 native mode, I am getting sessions
> lost and users needing to log in again due to Worker Process
> recycling. This is happening several times during the day, and was not
> a problem in iis5 mode.
> Watching the requests in IIS6 Request Viewer, and the ASP request
> Queued count in performance monitor, it is easy to see the requests
> slowly queuing up, and not releasing. Sometimes a single client(IP
> addr) shows as having multiple requests running against them for
> different ASP pages, all with a status of processing.
>
> Recycling happens anywhere between 40 and 140 queued requests, it is
> hard to replicate the exact cause, but only seems to happen under
> load. CPU and Memory usage of the Worker Process seems to be fairly
> stable.
>
> I have gone through the pages displayed as being processing in the IIS
> request viewer. I have been through the code, and closed of recordsets
> and database connections ( and set to nothing) and that have not been
> explicitly closed. This has made no difference or reduced the
> frequency of the WP recycling. These are common pages that all users
> load.
> I do have an occasional ASPEXEC call which gets stuck in Task Manager
> from a report run, and there can sometime be multiple instances of
> this "stuck" in task manager. However, when I have tried to recreate
> this situation, the ASPEXEC call works correctly, and clears itself
> from both task manager and the ASP Request queue as expected.
> Pages that show as being stuck in the queue seem to work correctly
> most of the time.
>
> There is also an ISAPI filter to a mapping engine which shows in the
> IIS6 ASP queues, but this does not enter into the ASP queues in
> perfmon, and should not be related to the 2262 error message.
> I have run IISState several time against the WP, sometimes it does not
> complete, and each time it causes a recylce of the process, causing
> another logoff, so I try to anticipate a recycle, and do it before the
> recycle occurs (is it supposed to cause session loss and recycle of
> WP?)
>
> Several questions arise:
> Is it possible to clear distinct requests from within the ASP queue in
> the WP?
> Does the Thread count settings for ASP.NEt affect ASP threads as well?
> Should IISState cause a recycle?
> How can I stop the queuing/recycling?
>
> Attached is the last IISState dump, at the time, the ASP request Queue
> in perfmon showed around 160 queued requests.
>
> Opened log file 'C:\iisstate\output\IISState-7396.log'
>
> ***********************
> Starting new log output
> IISState version 3.3.1
>
> Thu Nov 29 12:59:42 2007
>
> OS = Windows 2003 Server
> Executable: w3wp.exe
> PID = 7396
>
> Note: Thread times are formatted as HH:MM:SS.ms
>
> ***********************
>
> Thread ID: 0
> System Thread ID: 2b8
> Kernel Time: 0:0:0.62
> User Time: 0:0:0.62
> Thread Status: Thread is in a WAIT state.
> Thread Type: HTTP Compression Thread
> # ChildEBP RetAddr
> 00 0007fc08 7c827d0b ntdll!KiFastSystemCallRet
> 01 0007fc0c 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 0007fc7c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> 03 0007fc90 5a364662 kernel32!WaitForSingleObject+0x12
> 04 0007fca0 5a366e3f w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
> 05 0007fca8 5a3af42d w3dt!UlAtqStartListen+0x2d
> 06 0007fcb8 5a3bc335 w3core!W3_SERVER::StartListen+0xbd
> 07 0007ff0c 0100187c w3core!UlW3Start+0x26e
> 08 0007ff44 01001a27 w3wp!wmain+0x22a
> 09 0007ffc0 77e6f23b w3wp!wmainCRTStartup+0x12f
> 0a 0007fff0 00000000 kernel32!BaseProcessStart+0x23
>
> Thread ID: 1
> System Thread ID: 2d0
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Other
> # ChildEBP RetAddr
> 00 00ccff9c 7c826f4b ntdll!KiFastSystemCallRet
> 01 00ccffa0 7c83d424 ntdll!NtDelayExecution+0xc
> 02 00ccffb8 77e64829 ntdll!RtlpTimerThread+0x47
> 03 00ccffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 2
> System Thread ID: 1990
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Possible ASP page. Possible DCOM activity
> Executing Page: Unable to locate ASP page
>
> No remote call being made
>
> # ChildEBP RetAddr
> 00 00d7fe18 7c82783b ntdll!KiFastSystemCallRet
> 01 00d7fe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
> 02 00d7ff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
> 03 00d7ff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
> 04 00d7ffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
> 05 00d7ffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
> 06 00d7ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 3
> System Thread ID: dcc
> Kernel Time: 0:0:0.390
> User Time: 0:0:0.359
> *** WARNING: Unable to verify checksum for
> *** ERROR: Symbol file could not be found. Defaulted to export
> symbols for -
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 00f7eb80 7c827d0b ntdll!KiFastSystemCallRet
> 01 00f7eb84 71b21af5 ntdll!NtWaitForSingleObject+0xc
> 02 00f7ebc0 71b2c517 mswsock!SockWaitForSingleObject+0x19d
> 03 00f7ec38 71c02fee mswsock!WSPRecv+0x203
> 04 00f7ec80 6a6b8cce WS2_32!recv+0x83
> WARNING: Stack unwind information not available. Following frames may
> be wrong.
> 05 00f7ed3c 01908568 isapi_redirect+0x8cce
> 06 01908550 06062ff0 0x1908568
> 07 01908534 00000058 0x6062ff0
>
> Thread ID: 4
> System Thread ID: 1d38
> Kernel Time: 0:0:0.125
> User Time: 0:0:0.171
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 00fbeb80 7c827d0b ntdll!KiFastSystemCallRet
> 01 00fbeb84 71b21af5 ntdll!NtWaitForSingleObject+0xc
> 02 00fbebc0 71b2c517 mswsock!SockWaitForSingleObject+0x19d
> 03 00fbec38 71c02fee mswsock!WSPRecv+0x203
> 04 00fbec80 6a6b8cce WS2_32!recv+0x83
> WARNING: Stack unwind information not available. Following frames may
> be wrong.
> 05 00fbed3c 019044c8 isapi_redirect+0x8cce
> 06 019044b0 06075258 0x19044c8
> 07 01904494 00000058 0x6075258
>
> Thread ID: 5
> System Thread ID: 1ffc
> Kernel Time: 0:0:0.968
> User Time: 0:0:0.968
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 00ffeb80 7c827d0b ntdll!KiFastSystemCallRet
> 01 00ffeb84 71b21af5 ntdll!NtWaitForSingleObject+0xc
> 02 00ffebc0 71b2c517 mswsock!SockWaitForSingleObject+0x19d
> 03 00ffec38 71c02fee mswsock!WSPRecv+0x203
> 04 00ffec80 6a6b8cce WS2_32!recv+0x83
> WARNING: Stack unwind information not available. Following frames may
> be wrong.
> 05 00ffed3c 019126f8 isapi_redirect+0x8cce
> 06 019126e0 06185e98 0x19126f8
> 07 019126c4 00000048 0x6185e98
>
> Thread ID: 6
> System Thread ID: f20
> Kernel Time: 0:0:0.593
> User Time: 0:0:0.328
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 0104eb80 7c827d0b ntdll!KiFastSystemCallRet
> 01 0104eb84 71b21af5 ntdll!NtWaitForSingleObject+0xc
> 02 0104ebc0 71b2c517 mswsock!SockWaitForSingleObject+0x19d
> 03 0104ec38 71c02fee mswsock!WSPRecv+0x203
> 04 0104ec80 6a6b8cce WS2_32!recv+0x83
> WARNING: Stack unwind information not available. Following frames may
> be wrong.
> 05 0104ed3c 0190c608 isapi_redirect+0x8cce
> 06 0190c5f0 06095010 0x190c608
> 07 0190c5d4 00000058 0x6095010
>
> Thread ID: 7
> System Thread ID: 174
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 0110fcec 7c827cfb ntdll!KiFastSystemCallRet
> 01 0110fcf0 7c83c78e ntdll!NtWaitForMultipleObjects+0xc
> 02 0110ffb8 77e64829 ntdll!RtlpWaitThread+0x161
> 03 0110ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 8
> System Thread ID: bf8
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: HTTP Compression Thread
> # ChildEBP RetAddr
> 00 01adfa84 7c827d0b ntdll!KiFastSystemCallRet
> 01 01adfa88 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 01adfaf8 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> 03 01adfb0c 5a3b820f kernel32!WaitForSingleObject+0x12
> 04 01adffb8 77e64829 w3core!HTTP_COMPRESSION::CompressionThread+0x126
> 05 01adffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 9
> System Thread ID: 2cc
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: ASP
> Executing Page: Unable to locate ASP page
>
> # ChildEBP RetAddr
> 00 0293ff0c 7c827d0b ntdll!KiFastSystemCallRet
> 01 0293ff10 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 0293ff80 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> 03 0293ff94 709fc803 kernel32!WaitForSingleObject+0x12
> 04 0293ffb4 709fc7db asp!CApplnCleanupMgr::ApplnCleanupDoWork+0x1e
> 05 0293ffb8 77e64829 asp!CApplnCleanupMgr::ApplnCleanupThread+0xa
> 06 0293ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 10
> System Thread ID: 1af4
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: ASP
> Executing Page: Unable to locate ASP page
>
> # ChildEBP RetAddr
> 00 029ffe1c 7c827cfb ntdll!KiFastSystemCallRet
> 01 029ffe20 77e6202c ntdll!NtWaitForMultipleObjects+0xc
> 02 029ffec8 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
> 03 029fff24 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
> 04 029fff40 709fb8be USER32!MsgWaitForMultipleObjects+0x1f
> 05 029fff84 77bcb530 asp!CMTACallbackThread::Thread+0x4f
> 06 029fffb8 77e64829 msvcrt!_endthreadex+0xa3
> 07 029fffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 11
> System Thread ID: f64
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: ASP
> Executing Page: Unable to locate ASP page
>
> # ChildEBP RetAddr
> 00 02a3fef4 7c827d0b ntdll!KiFastSystemCallRet
> 01 02a3fef8 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 02a3ff68 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> 03 02a3ff7c 709fe8ef kernel32!WaitForSingleObject+0x12
> 04 02a3ffb8 77e64829 asp!CViperReqManager::WatchThread+0x68
> 05 02a3ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 12
> System Thread ID: 1858
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: Idle ASP thread
> # ChildEBP RetAddr
> 00 02a9fee8 7c827d0b ntdll!KiFastSystemCallRet
> 01 02a9feec 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 02a9ff5c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> 03 02a9ff70 4a77921b kernel32!WaitForSingleObject+0x12
> 04 02a9ff84 77bcb530 comsvcs!
> CSTAThreadPool::LoadBalanceThreadControlLoop+0x29
> 05 02a9ffb8 77e64829 msvcrt!_endthreadex+0xa3
> 06 02a9ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 13
> System Thread ID: 18c
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: Idle ASP thread
> # ChildEBP RetAddr
> 00 02adfee0 7c827d0b ntdll!KiFastSystemCallRet
> 01 02adfee4 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 02adff54 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> 03 02adff68 4a778cb6 kernel32!WaitForSingleObject+0x12
> 04 02adff84 77bcb530 comsvcs!CSTAThreadPool::KillThreadControlLoop
> +0x25
> 05 02adffb8 77e64829 msvcrt!_endthreadex+0xa3
> 06 02adffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 14
> System Thread ID: 1744
> Kernel Time: 0:0:1.562
> User Time: 0:0:2.78
> Thread Status: Thread is in a WAIT state.
> Thread Type: Idle ASP thread
> # ChildEBP RetAddr
> 00 02b1fdcc 7c827cfb ntdll!KiFastSystemCallRet
> 01 02b1fdd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
> 02 02b1fe78 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
> 03 02b1fed4 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
> 04 02b1fef0 4a77cb28 USER32!MsgWaitForMultipleObjects+0x1f
> 05 02b1ff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x1f9
> 06 02b1ffb8 77e64829 msvcrt!_endthreadex+0xa3
> 07 02b1ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 15
> System Thread ID: 1bd8
> Kernel Time: 0:0:2.937
> User Time: 0:0:2.296
> Thread Status: Thread is in a WAIT state.
> Thread Type: Idle ASP thread
> # ChildEBP RetAddr
> 00 02b5fdcc 7c827cfb ntdll!KiFastSystemCallRet
> 01 02b5fdd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
> 02 02b5fe78 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
> 03 02b5fed4 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
> 04 02b5fef0 4a77cb28 USER32!MsgWaitForMultipleObjects+0x1f
> 05 02b5ff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x1f9
> 06 02b5ffb8 77e64829 msvcrt!_endthreadex+0xa3
> 07 02b5ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 16
> System Thread ID: 1760
> Kernel Time: 0:0:2.640
> User Time: 0:0:3.140
> Thread Status: Thread is in a WAIT state.
> Thread Type: Idle ASP thread
> # ChildEBP RetAddr
> 00 02b9fdcc 7c827cfb ntdll!KiFastSystemCallRet
> 01 02b9fdd0 77e6202c ntdll!NtWaitForMultipleObjects+0xc
> 02 02b9fe78 7739bbd1 kernel32!WaitForMultipleObjectsEx+0x11a
> 03 02b9fed4 7739ce36 USER32!RealMsgWaitForMultipleObjectsEx+0x141
> 04 02b9fef0 4a77cb28 USER32!MsgWaitForMultipleObjects+0x1f
> 05 02b9ff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x1f9
> 06 02b9ffb8 77e64829 msvcrt!_endthreadex+0xa3
> 07 02b9ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 17
> System Thread ID: 88
> Kernel Time: 0:0:3.734
> User Time: 0:0:3.265
> *** WARNING: Unable to verify checksum for C:\Inetpub\Aspexec
> \ASPEXEC.DLL
> *** ERROR: Symbol file could not be found. Defaulted to export
> symbols for C:\Inetpub\Aspexec\ASPEXEC.DLL -
> Thread Type: ASP
> Executing Page: ??C:\INETPUB\PRODUCTION\TRACKING\REPORTS
> \REPORTTEMPLATE.ASP
>
> # ChildEBP RetAddr
> 00 02bdb118 7c82776b ntdll!KiFastSystemCallRet
> 01 02bdb11c 77e419d3 ntdll!NtReadFile+0xc
> 02 02bdb184 06912071 kernel32!ReadFile+0x10d
> WARNING: Stack unwind information not available. Following frames may
> be wrong.
> 03 02bdf260 06912951 ASPEXEC!DllUnregisterServer+0x33635
> 04 02bdf28c 77d05186 ASPEXEC!DllUnregisterServer+0x33f15
> 05 02bdf2a8 77d14acf OLEAUT32!DispCallFunc+0x16a
> 06 02bdf338 068dd5c5 OLEAUT32!CTypeInfo2::Invoke+0x234
> 07 02bdf364 7347d3cb ASPEXEC+0xd5c5
> 08 02bdf3b8 73468ad1 vbscript!CatchIDispatchInvoke+0x46
> 09 02bdf3f8 73468a40 vbscript!IDispatchInvoke2+0xaf
> 0a 02bdf434 734689f2 vbscript!IDispatchInvoke+0x59
> 0b 02bdf548 7346613b vbscript!InvokeDispatch+0x13a
> 0c 02bdf56c 73466fee vbscript!InvokeByName+0x42
> 0d 02bdf848 734633e0 vbscript!CScriptRuntime::Run+0x1331
> 0e 02bdf940 734637d1 vbscript!CScriptEntryPoint::Call+0x5c
> 0f 02bdf998 73463b9c vbscript!CSession::Execute+0xb4
> 10 02bdf9e8 73461849 vbscript!COleScript::ExecutePendingScripts+0x13e
> 11 02bdfa04 709e2ada vbscript!COleScript::SetScriptState+0x150
> 12 02bdfa30 709e2a9c asp!CActiveScriptEngine::TryCall+0x19
> 13 02bdfa6c 709e26d0 asp!CActiveScriptEngine::Call+0x31
> 14 02bdfa88 709e25d4 asp!CallScriptFunctionOfEngine+0x5b
> 15 02bdfadc 709e24ff asp!ExecuteRequest+0x17e
> 16 02bdfb44 709e23f7 asp!Execute+0x249
> 17 02bdfb98 709e2753 asp!CHitObj::ViperAsyncCallback+0x3f3
> 18 02bdfbb4 4a77b5ea asp!CViperAsyncRequest::OnCall+0x92
> 19 02bdfbd0 77720d30 comsvcs!CSTAActivityWork::STAActivityWorkHelper
> +0x32
> 1a 02bdfc1c 777217dc ole32!EnterForCallback+0xc4
> 1b 02bdfd7c 776f03b4 ole32!SwitchForCallback+0x1a3
> 1c 02bdfda8 7769c194 ole32!PerformCallback+0x54
> 1d 02bdfe40 7772433a ole32!CObjectContext::InternalContextCallback
> +0x159
> 1e 02bdfe60 4a77b78c ole32!CObjectContext::DoCallback+0x1c
> 1f 02bdfecc 4a77bcf2 comsvcs!CSTAActivityWork::DoWork+0x12d
> 20 02bdfee4 4a77c7de comsvcs!CSTAThread::DoWork+0x18
> 21 02bdff04 4a77cabf comsvcs!CSTAThread::ProcessQueueWork+0x37
> 22 02bdff84 77bcb530 comsvcs!CSTAThread::WorkerLoop+0x190
> 23 02bdffb8 77e64829 msvcrt!_endthreadex+0xa3
> 24 02bdffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 18
> System Thread ID: e70
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 0315ff88 7c827cfb ntdll!KiFastSystemCallRet
> 01 0315ff8c 71c51108 ntdll!NtWaitForMultipleObjects+0xc
> 02 0315ffb8 77e64829 NETAPI32!NetbiosWaiter+0x73
> 03 0315ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 19
> System Thread ID: 1f94
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Status: Thread is in a WAIT state.
> Thread Type: Idle ASP thread
> # ChildEBP RetAddr
> 00 0319fd00 7c827d0b ntdll!KiFastSystemCallRet
> 01 0319fd04 77e61d1e ntdll!NtWaitForSingleObject+0xc
> 02 0319fd74 77e61c8d kernel32!WaitForSingleObjectEx+0xac
> 03 0319fd88 4a756c3b kernel32!WaitForSingleObject+0x12
> 04 0319ffb8 77e64829 comsvcs!PingThread+0xf6
> 05 0319ffec 00000000 kernel32!BaseThreadStart+0x34
>
> Thread ID: 20
> System Thread ID: 2c8
> Kernel Time: 0:0:1.984
> User Time: 0:0:1.484
> Thread Status: Thread is in a WAIT state.
> Thread Type: Other
> # ChildEBP RetAddr
> 00 0332eb80 7c827d0b ntdll!KiFastSystemCallRet
> 01 0332eb84 71b21af5 ntdll!NtWaitForSingleObject+0xc
> 02 0332ebc0 71b2c517 mswsock!SockWaitForSingleObject+0x19d
> 03 0332ec38 71c02fee mswsock!WSPRecv+0x203
> 04 0332ec80 6a6b8cce WS2_32!recv+0x83
> WARNING: Stack unwind information not available. Following frames may
> be wrong.
> 05 0332ed3c 01916798 isapi_redirect+0x8cce
> 06 01916780 061be940 0x1916798
> 07 01916764 00000058 0x61be940
>
> Thread ID: 21
> System Thread ID: 13f4
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Managed Thread. Possible ASP.Net page or other .Net
> worker
> SaveModuleToMemory failed
> Failed to load SOS data.
> No valid SOS data table found.
>
> Begin System Thread Information
>
> # ChildEBP RetAddr
> 00 0384fea8 00000000 ntdll!KiFastSystemCallRet
> GetContextState failed, 0x8007001F
> GetContextState failed, 0x8007001F
> GetContextState failed, 0x8007001F
>
> Thread ID: 22
> System Thread ID: 0
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> GetContextState failed, 0x8007001F

Re: IIS6 ASP requests increasing, recycle WP and sessions lost, 2262

Hi David
Thank you for your quick response and excellent service to this group.

We are currently in a rewrite of the application into .Net 2.0. The
web server runs around 4 sites, including web services and an apache
based web mapping engine ( idelve). Although we did not get the
session timeouts in iis5 mode, there were still occasional crashes
which required IIS restarting.
Our anticipation was that we would achieve more stability, and
eventually be able to phase out the old ASP code and run the new
application as a parallel website, by moving to IIS6 mode. Our test
server, without the load, runs happily in iis6 mode.

I perhaps needed to reword and redirect some of my questions to make
them more meaningful in the context .

The biggest problem I am having, is identitfying a single ( or any
part of a multiple) reason as to why the ASP queues are blocking. I
suppose this is the information I referred to when I said it is hard
to find. -- What Causes a queue to block????
For instance, how does a single client have many ( up to 20 I have
seen) outstanding requests flagged as processing in the request
viewer? When a request has a status of processing, what is the state
of the request in terms of the client browser and the server
processing queue. What makes one user able to successfully view a
commonly used asp page, and others get stuck in the queue on the same
page. I have had no feedback from users to say "i have this error on
my screen" that I can match up with an unhealthy request queue.

When you state that
"No, but it is possible that your AppPool configuration is
misreacting
to IIS State and prematurely recycling the application pool. "

could this also mean that the IIS installation is misreacting to
normal ASP requests and causing a termination/queue hang in the same
way. This is obviously hard for you to make any call on, but have you
experienced or heard of any similar events?

The IISState and dumps I have taken have not shown any consistency or
patterns that I have been able to identify. The ASPEXEC featured in
the iisstate dump I provided, but this is the only one iisstate dumped
that showed it. It seems to run well 99% of the time, running the same
report query against a standard set of MSSql tables.

The reason I asked about threads ( poorly worded question) was to
determine whether making more threads available may help in the short
term by increasing the time before a recycle occurs. I am assuming
that the ASP requests queued relate to the threads available, so when
all threads are blocked no more request can be processed and a recyle
happens. Am i mistaken in this?

I would love to determine the cause of queuing in the application, but
am finding little to get to grips with.
As a general question, a simple ASP page running a sql query and
displaying results in a table will be queued under what circumstances?

Apart from IISState, which causes grief to my users each time I use
it, what are the tools you would suggest I use to try to determine
this queuing. How many pipelines are there for ASP requests to be
processed.


Once again thanks for your time

Shane

Re: IIS6 ASP requests increasing, recycle WP and sessions lost, 2262

> What Causes a queue to block????

Queues don't block, and there's no such thing as "pipelines". Here's
how it works logically.

Your application is able to process a certain number of requests per
second and generate responses for each (i.e. throughput). The server
experiences a certain input load in terms of requests per second. If
the application throughput is higher than the input load throughput,
there will be no queue -- work is finished as soon as it can. If the
application throughput is lower than the input, then the ASP queue
will grow to hold onto those delivered requests as ASP waits for each
thread of the application to finish the prior request before giving it
the next one.

Thus, growing queue indicates that the application throughput is not
high enough. This either means:
1. Hardware limitation has been reached. It is physically not possible
to process the input load with the hardware. Only solution is to buy/
use better hardware
2. Software bottleneck has been reached. Something is lowering the
theoretical maximum throughput

You have to first determine if #1 or #2 is happening. Monitor CPU
load, Network Bandwidth, Hard disk queues, and RAM consumption.

Assuming that we determine it's software issue, we need to start
looking at whether throughput is lowered due to time spent on:
1. Contention on locks (for synchronization)
2. Latency - Actual CPU cycles spent doing work or waiting

#1 can be addressed by adding concurrency in the form of threads to
work around the locks, but incresing threads also increases contention
and decreases throughput -- thus it is a balancing game between not
enough threads and too many threads. The right number needs to be
experimentally determined because it is specific to the application's
algorithmic profile.

#2 can be addressed with code-change to alter the algorithms. For
example, suppose the page is waiting on a SQL query to complete. The
only way to make it run faster is to make the SQL query faster.


> could this also mean that the IIS installation is misreacting to
> normal ASP requests and causing a termination/queue hang in the same
> way.

It can, in the case of degenerate Application Pool Health Monitoring
configuration. Default values don't do that. Personally, I would start
suspecting the application or its configuration and not IIS itself as
the culprit. The fact that your application runs in IIS5 Compatibility
Mode gives no indication of its correctness.



To address queuing, you're basically looking at either decreasing
latency or contention.

The reason I suspect ASPExec is because it is an STA object, which has
devastating serialization impact (i.e. contention) on IIS6 w3wp
threads running in MTA by default. Try configuring w3wp.exe to run in
STA in IIS6 worker process isolation mode.

If it's not ASPExec, then time to move onto looking for other sources
of latency or contention. This is best done with software profiling
for the hardcore, though code review and frequency analysis can be
just as effective in the right hands.


//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//






On Nov 28, 11:13 pm, sh...@smarttrack.com.au wrote:
> Hi David
> Thank you for your quick response and excellent service to this group.
>
> We are currently in a rewrite of the application into .Net 2.0. The
> web server runs around 4 sites, including web services and an apache
> based web mapping engine ( idelve). Although we did not get the
> session timeouts in iis5 mode, there were still occasional crashes
> which required IIS restarting.
> Our anticipation was that we would achieve more stability, and
> eventually be able to phase out the old ASP code and run the new
> application as a parallel website, by moving to IIS6 mode. Our test
> server, without the load, runs happily in iis6 mode.
>
> I perhaps needed to reword and redirect some of my questions to make
> them more meaningful in the context .
>
> The biggest problem I am having, is identitfying a single ( or any
> part of a multiple) reason as to why the ASP queues are blocking. I
> suppose this is the information I referred to when I said it is hard
> to find. -- What Causes a queue to block????
> For instance, how does a single client have many ( up to 20 I have
> seen) outstanding requests flagged as processing in the request
> viewer? When a request has a status of processing, what is the state
> of the request in terms of the client browser and the server
> processing queue. What makes one user able to successfully view a
> commonly used asp page, and others get stuck in the queue on the same
> page. I have had no feedback from users to say "i have this error on
> my screen" that I can match up with an unhealthy request queue.
>
> When you state that
> "No, but it is possible that your AppPool configuration is
> misreacting
> to IIS State and prematurely recycling the application pool. "
>
> could this also mean that the IIS installation is misreacting to
> normal ASP requests and causing a termination/queue hang in the same
> way. This is obviously hard for you to make any call on, but have you
> experienced or heard of any similar events?
>
> The IISState and dumps I have taken have not shown any consistency or
> patterns that I have been able to identify. The ASPEXEC featured in
> the iisstate dump I provided, but this is the only one iisstate dumped
> that showed it. It seems to run well 99% of the time, running the same
> report query against a standard set of MSSql tables.
>
> The reason I asked about threads ( poorly worded question) was to
> determine whether making more threads available may help in the short
> term by increasing the time before a recycle occurs. I am assuming
> that the ASP requests queued relate to the threads available, so when
> all threads are blocked no more request can be processed and a recyle
> happens. Am i mistaken in this?
>
> I would love to determine the cause of queuing in the application, but
> am finding little to get to grips with.
> As a general question, a simple ASP page running a sql query and
> displaying results in a table will be queued under what circumstances?
>
> Apart from IISState, which causes grief to my users each time I use
> it, what are the tools you would suggest I use to try to determine
> this queuing. How many pipelines are there for ASP requests to be
> processed.
>
> Once again thanks for your time
>
> Shane

Re: IIS6 ASP requests increasing, recycle WP and sessions lost, 2262

The problem seems to be entirely related to ASPEXEC.

I removed this functionality from the website ( better a stable system
with 1 feature less than an unstable system) and I am no longer having
the request queue buildup and recycling problem so far!
Initially, I had gone through the website ensuring recordsets and
database connections were closed, so this may have helped in some
degree, But removing the ASPEXEC calls seems to be the solution.

The reason this was so hard to debug is that the website continued
normally, even though the ASP request queues built up, until IIS got
to the end of its rope and recycled at an apparently arbitrary number
of requests.

Always the trouble having an inherited site is having to adopt the
tools and methodologies that your predecessor used, and having to
adapt them to new environments as hardware and software changes.

The functions requiring ASPEXEC will be rewritten using .Net, and we
will have a better system for that effort anyway.

David, thanks for your assistance and insights. I hope this post can
assist others having similar problems.

Shane