w3wp.exe hangs while serving ASP sites

Hi,

We've recently migrated to a new production webserver running Windows
Server 2003 R2 SP2 and it's giving us a headache: worker processes
processing 'old' ASP sites hang unexpectedly and IIS doesn't notice.

The request queue fills up and CPU load is maxed out to the limit as
configured in the application pool. The number of CurrentConnections,
Current Users and Current Request climbs sky-high and doesn't come
down. IIS doesn't notice a problem (but does disable the application
if CPU monitoring is enabled). The HTTPERR log shows
Connection_Abandoned_By_AppPool errors.

When this happens the application needs to be recycled by hand, after
that it hums along nicely until the next hang.

Load testing the websites using the Web Application Stress tool on our
local Windows 2003 Server SP2 isn't able to reproduce the problem.
Even when testing multiple 'old' ASP sites at the same time. I've
checked the versions of w3wp.exe, asp.dll and they are identical on
both servers. The ASP sites use external objects extensively, like
ADODB (SQL Server), MSXML3, and ASPupload.

The system monitor shows some characteristics that look like blocking
(low CPU usage, request queue that fills up) but we don't know *what*
is blocking *what*. It doesn't happen on our development servers.

Does anybody have a clue?

Re: w3wp.exe hangs while serving ASP sites

On Nov 23, 6:26 am, sir.fred...@gmail.com wrote:
> Hi,
>
> We've recently migrated to a new production webserver running Windows
> Server 2003 R2 SP2 and it's giving us a headache: worker processes
> processing 'old' ASP sites hang unexpectedly and IIS doesn't notice.
>
> The request queue fills up and CPU load is maxed out to the limit as
> configured in the application pool. The number of CurrentConnections,
> Current Users and Current Request climbs sky-high and doesn't come
> down. IIS doesn't notice a problem (but does disable the application
> if CPU monitoring is enabled). The HTTPERR log shows
> Connection_Abandoned_By_AppPool errors.
>
> When this happens the application needs to be recycled by hand, after
> that it hums along nicely until the next hang.
>
> Load testing the websites using the Web Application Stress tool on our
> local Windows 2003 Server SP2 isn't able to reproduce the problem.
> Even when testing multiple 'old' ASP sites at the same time. I've
> checked the versions of w3wp.exe, asp.dll and they are identical on
> both servers. The ASP sites use external objects extensively, like
> ADODB (SQL Server), MSXML3, and ASPupload.
>
> The system monitor shows some characteristics that look like blocking
> (low CPU usage, request queue that fills up) but we don't know *what*
> is blocking *what*. It doesn't happen on our development servers.
>
> Does anybody have a clue?


Use IISState or DebugDiag during the "hang" and report on the stack
backtrace of the problematic w3wp.exe. That is the correct (and only)
way to determine what is blocking what (amongst many other things).


//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//

Re: w3wp.exe hangs while serving ASP sites

Thanks for the hint, I used DebugDiag to monitor the process, and of
course it hanged immediately. IIS feels our pain :)

Anyway, the report generated indicated long wait times due to
communication with other servers (mainly SQL queries with large
datasets) but the *real* culprit was revealed by a little hint
concerning 'ASP server-side script debugging'. This was enabled by
default and created an awful mess.

ASP Server-side script debugging forces the thread to operate serially
so the worker process handles one thing at the time. Other requests
end up in the request queue and wait for their turn, so a misbehaving
thread manages to hang the entire worker process.

I disabled the debugging and now the worker processes do their job
flawlessly. But I'll enable debugging on our local development server
to see why blocking occurs when the thread operates serially. If we
can fix that it will probably be better anyway :)