Troubleshooting IIS 5

Hi there,

we have a windows 2000 machine with IIS on it. It has 2 websites, one of
which has around 25 subfolders. Each subfolder is an ASP application.

Lately, about 4 times a day, IIS becomes unresponsive. Nothing in the
logging (event nor IIS access logs) seems to give me an indication of
what is going wrong.

Just guessing, but I think one of the ASP programs might get into an
infinite loop, or be waiting for data forever. As these are 2 most
likely issues the ASP applications could have. If that's the case I'm
wondering why:

a) IIS doesn't terminate the process after x time
b) Why IIS doesn't serve other pages. I'm not very familiar with IIS,
but it would seem to me any webserver of this size should be able to
handle multiple requests at once.

Any ideas on where to start troubleshooting?

Unfortunately we can't just move the applications to another server,
they depend on an application that runs on windows 2000 of which we do
not have the source code and we're unable to get it working on windows 2003.

Kind regards

Re: Troubleshooting IIS 5

On Mar 31, 12:30=A0am, Freaky wrote:
> Hi there,
>
> we have a windows 2000 machine with IIS on it. It has 2 websites, one of
> which has around 25 subfolders. Each subfolder is an ASP application.
>
> Lately, about 4 times a day, IIS becomes unresponsive. Nothing in the
> logging (event nor IIS access logs) seems to give me an indication of
> what is going wrong.
>
> Just guessing, but I think one of the ASP programs might get into an
> infinite loop, or be waiting for data forever. As these are 2 most
> likely issues the ASP applications could have. If that's the case I'm
> wondering why:
>
> a) IIS doesn't terminate the process after x time
> b) Why IIS doesn't serve other pages. I'm not very familiar with IIS,
> but it would seem to me any webserver of this size should be able to
> handle multiple requests at once.
>
> Any ideas on where to start troubleshooting?
>
> Unfortunately we can't just move the applications to another server,
> they depend on an application that runs on windows 2000 of which we do
> not have the source code and we're unable to get it working on windows 200=
3.
>
> Kind regards



a. Architectural design. IIS5 is the last version which assumed user
applications are trustworthy enough to run within IIS itself (which
turned out to be a bad assumption because most user applications are
far more poorly written than IIS, but due to IIS's trusting nature, it
gets itself into trouble). Starting with IIS6, IIS assumes user
applications are badly written and malicious and runs defensively by
default, which has resulted in dramatically better security,
reliability, and performance that has been proven.

Both of those issues you point out are problems dealt with in ASP on
IIS6 and remain flaws in ASP on IIS5.

b. With IIS5, it is possible for user applications to monopolize the
system because IIS foolishly allowed it. This would not be possible
with IIS6 and how it treats ASP and user applications in general.

With properly written applications, IIS can handle tens of thousands
of concurrent requests at once. With improperly written applications,
it is possible for one single application to tank the application pool
by chewing up all threads that handle incoming requests. But this is
about as good as it gets -- you certainly cannot expect IIS to work
miracles with badly written applications. At some point, the
application developer has to take full responsibility for a hanging
server and not complain why IIS does not stay responsive regardless of
the junk being shoved at it.

http://blogs.msdn.com/david.wang/archive/2005/12/31/HOWTO_Ba sics_of_IIS6_Tro=
ubleshooting.aspx

Use IIS State or DebugDiag during the "unresponsive" times and see
what's going on.


//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//

Re: Troubleshooting IIS 5

Thanks a bunch for the re', I'll be digging into the tools you mentioned.

> a. Architectural design. IIS5 is the last version which assumed user
> applications are trustworthy enough to run within IIS itself (which
> turned out to be a bad assumption because most user applications are
> far more poorly written than IIS, but due to IIS's trusting nature, it
> gets itself into trouble). Starting with IIS6, IIS assumes user
> applications are badly written and malicious and runs defensively by
> default, which has resulted in dramatically better security,
> reliability, and performance that has been proven.

Do have some options, it's at medium (pooled) now, I can put it in high
(isolated) too, but that still doesn't spawn a lesser privileged process
then I presume? Also, found some settings on ASP that _should_ only
allow scripts to run for 90 seconds.

> Both of those issues you point out are problems dealt with in ASP on
> IIS6 and remain flaws in ASP on IIS5.
>
> b. With IIS5, it is possible for user applications to monopolize the
> system because IIS foolishly allowed it. This would not be possible
> with IIS6 and how it treats ASP and user applications in general.
>
> With properly written applications, IIS can handle tens of thousands
> of concurrent requests at once. With improperly written applications,
> it is possible for one single application to tank the application pool
> by chewing up all threads that handle incoming requests. But this is
> about as good as it gets -- you certainly cannot expect IIS to work
> miracles with badly written applications. At some point, the
> application developer has to take full responsibility for a hanging
> server and not complain why IIS does not stay responsive regardless of
> the junk being shoved at it.

These are really poorly written apps mainly without documentation
whatsoever. The people that wrote them however were blissfully unaware
of what a thread even is, so I doubt it's soaking threads. But I'll have
a look with the tools you provided. CPU load is extremely low btw, even
during hangs.

Anyways thanks a bunch for the re'.

Re: Troubleshooting IIS 5

Hi,

this morning it was stuck again. Used the IISState tool, it outputs a
_lot_ of threads (around 200 or so). Can't find a single ASP page, that
is never have something in the lines of:

xx hhhhhhhh hhhhhhhh ASP*

The vast majority of the threads look similar like the one below. As
there is a lot of code on the server it would be nice to have some hint
on where to start looking. Appearantly it's doing a lot of OLE/RPCRT4
stuff. If I could take a wild guess I'd figure it are attempts to
connect to the SQL database that fail or something, but I don't get why
they don't close, or why they spawn this many.

Anyways thanks a bunch so far :).

Thread ID: 60
System Thread ID: 544
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.

DCOM call being made to Process ID: 2972
Waiting on thread id: 0

# ChildEBP RetAddr
00 02b0ea80 77d4f404 ntdll!NtRequestWaitReplyPort+0xb
01 02b0eaac 77d3b96c RPCRT4!LRPC_CCALL::SendReceive+0x124
02 02b0eab8 7cef6bee RPCRT4!I_RpcSendReceive+0x2c
03 02b0ead8 7cef6ab9 ole32!ThreadSendReceive+0xef
04 02b0eaf0 7cef3ab6
ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0x14f
05 02b0eb30 7cef692d ole32!CRpcChannelBuffer::SendReceive2+0x96
06 02b0eb40 7ce3cc2d ole32!CRpcChannelBuffer::SendReceive+0x11
07 02b0eba0 7ce87ef3 ole32!CAptRpcChnl::SendReceive+0xa9
08 02b0ebf8 77d91320 ole32!CCtxComChnl::SendReceive+0x98
09 02b0ec14 77d93b47 RPCRT4!NdrProxySendReceive+0x4c
0a 02b0ee5c 77d96f9c RPCRT4!NdrClientCall2+0x4f5
0b 02b0ee78 77d7998b RPCRT4!ObjectStublessClient+0x76
0c 02b0ee88 65f369f2 RPCRT4!ObjectStubless+0xf
0d 02b0f49c 65f04c38 w3svc!CWamInfoOutProc::DoProcessRequestCall+0x163
0e 02b0f4c4 65f03d71 w3svc!CWamInfo::ProcessWamRequest+0xb9
0f 02b0fcfc 65f03c49 w3svc!WAM_DICTATOR::ProcessWamRequest+0x1e5
10 02b0fd20 65f03aa2 w3svc!HTTP_REQUEST::DoWamRequest+0x75
11 02b0fd44 65f0249e w3svc!HTTP_REQUEST::ProcessBGI+0x166
12 02b0fec4 65f01d97 w3svc!HTTP_REQUEST::DoWork+0x43f
13 02b0fee4 65f06be5 w3svc!CLIENT_CONN::DoWork+0x1aa
14 02b0ff08 65f06b58 w3svc!CreateClient+0x7b
15 02b0ff4c 6d701ad2 w3svc!W3OnConnectEx+0x118
16 02b0ff80 6d7029a6 ISATQ!AtqpProcessContext+0x23e
17 02b0ffb4 7c57b3bc ISATQ!AtqPoolThread+0x1a8
18 02b0ffec 00000000 KERNEL32!BaseThreadStart+0x52

David Wang wrote:
> On Mar 31, 12:30 am, Freaky wrote:
>> Hi there,
>>
>> we have a windows 2000 machine with IIS on it. It has 2 websites, one of
>> which has around 25 subfolders. Each subfolder is an ASP application.
>>
>> Lately, about 4 times a day, IIS becomes unresponsive. Nothing in the
>> logging (event nor IIS access logs) seems to give me an indication of
>> what is going wrong.
>>
>> Just guessing, but I think one of the ASP programs might get into an
>> infinite loop, or be waiting for data forever. As these are 2 most
>> likely issues the ASP applications could have. If that's the case I'm
>> wondering why:
>>
>> a) IIS doesn't terminate the process after x time
>> b) Why IIS doesn't serve other pages. I'm not very familiar with IIS,
>> but it would seem to me any webserver of this size should be able to
>> handle multiple requests at once.
>>
>> Any ideas on where to start troubleshooting?
>>
>> Unfortunately we can't just move the applications to another server,
>> they depend on an application that runs on windows 2000 of which we do
>> not have the source code and we're unable to get it working on windows 2003.
>>
>> Kind regards
>
>
>
> a. Architectural design. IIS5 is the last version which assumed user
> applications are trustworthy enough to run within IIS itself (which
> turned out to be a bad assumption because most user applications are
> far more poorly written than IIS, but due to IIS's trusting nature, it
> gets itself into trouble). Starting with IIS6, IIS assumes user
> applications are badly written and malicious and runs defensively by
> default, which has resulted in dramatically better security,
> reliability, and performance that has been proven.
>
> Both of those issues you point out are problems dealt with in ASP on
> IIS6 and remain flaws in ASP on IIS5.
>
> b. With IIS5, it is possible for user applications to monopolize the
> system because IIS foolishly allowed it. This would not be possible
> with IIS6 and how it treats ASP and user applications in general.
>
> With properly written applications, IIS can handle tens of thousands
> of concurrent requests at once. With improperly written applications,
> it is possible for one single application to tank the application pool
> by chewing up all threads that handle incoming requests. But this is
> about as good as it gets -- you certainly cannot expect IIS to work
> miracles with badly written applications. At some point, the
> application developer has to take full responsibility for a hanging
> server and not complain why IIS does not stay responsive regardless of
> the junk being shoved at it.
>
> http://blogs.msdn.com/david.wang/archive/2005/12/31/HOWTO_Ba sics_of_IIS6_Troubleshooting.aspx
>
> Use IIS State or DebugDiag during the "unresponsive" times and see
> what's going on.
>
>
> //David
> http://w3-4u.blogspot.com
> http://blogs.msdn.com/David.Wang
> //

Re: Troubleshooting IIS 5

If debuggers knew where to start looking for problems, we'd have
programs writing programs. And since we obviously are not at that
state of Artificial Intelligence, human experience is the only
alternative.

You're looking at inetinfo.exe. This looks like something hanging in
the the IIS dllhost.exe handling medium/high isolation applications.
Is Process ID 2972 dllhost.exe and if so, what's going on in that
process?


//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//



On Apr 1, 1:10=A0am, Freaky wrote:
> Hi,
>
> this morning it was stuck again. Used the IISState tool, it outputs a
> _lot_ of threads (around 200 or so). Can't find a single ASP page, that
> is never have something in the lines of:
>
> xx hhhhhhhh hhhhhhhh ASP*
>
> The vast majority of the threads look similar like the one below. As
> there is a lot of code on the server it would be nice to have some hint
> on where to start looking. Appearantly it's doing a lot of OLE/RPCRT4
> stuff. If I could take a wild guess I'd figure it are attempts to
> connect to the SQL database that fail or something, but I don't get why
> they don't close, or why they spawn this many.
>
> Anyways thanks a bunch so far :).
>
> Thread ID: 60
> System Thread ID: 544
> Kernel Time: 0:0:0.0
> User Time: 0:0:0.0
> Thread Type: Possible ASP page. =A0Possible DCOM activity
> Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
> Continuing with other analysis.
>
> DCOM call being made to Process ID: 2972
> Waiting on thread id: 0
>
> =A0 # ChildEBP RetAddr
> 00 02b0ea80 77d4f404 ntdll!NtRequestWaitReplyPort+0xb
> 01 02b0eaac 77d3b96c RPCRT4!LRPC_CCALL::SendReceive+0x124
> 02 02b0eab8 7cef6bee RPCRT4!I_RpcSendReceive+0x2c
> 03 02b0ead8 7cef6ab9 ole32!ThreadSendReceive+0xef
> 04 02b0eaf0 7cef3ab6
> ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0x14f
> 05 02b0eb30 7cef692d ole32!CRpcChannelBuffer::SendReceive2+0x96
> 06 02b0eb40 7ce3cc2d ole32!CRpcChannelBuffer::SendReceive+0x11
> 07 02b0eba0 7ce87ef3 ole32!CAptRpcChnl::SendReceive+0xa9
> 08 02b0ebf8 77d91320 ole32!CCtxComChnl::SendReceive+0x98
> 09 02b0ec14 77d93b47 RPCRT4!NdrProxySendReceive+0x4c
> 0a 02b0ee5c 77d96f9c RPCRT4!NdrClientCall2+0x4f5
> 0b 02b0ee78 77d7998b RPCRT4!ObjectStublessClient+0x76
> 0c 02b0ee88 65f369f2 RPCRT4!ObjectStubless+0xf
> 0d 02b0f49c 65f04c38 w3svc!CWamInfoOutProc::DoProcessRequestCall+0x163
> 0e 02b0f4c4 65f03d71 w3svc!CWamInfo::ProcessWamRequest+0xb9
> 0f 02b0fcfc 65f03c49 w3svc!WAM_DICTATOR::ProcessWamRequest+0x1e5
> 10 02b0fd20 65f03aa2 w3svc!HTTP_REQUEST::DoWamRequest+0x75
> 11 02b0fd44 65f0249e w3svc!HTTP_REQUEST::ProcessBGI+0x166
> 12 02b0fec4 65f01d97 w3svc!HTTP_REQUEST::DoWork+0x43f
> 13 02b0fee4 65f06be5 w3svc!CLIENT_CONN::DoWork+0x1aa
> 14 02b0ff08 65f06b58 w3svc!CreateClient+0x7b
> 15 02b0ff4c 6d701ad2 w3svc!W3OnConnectEx+0x118
> 16 02b0ff80 6d7029a6 ISATQ!AtqpProcessContext+0x23e
> 17 02b0ffb4 7c57b3bc ISATQ!AtqPoolThread+0x1a8
> 18 02b0ffec 00000000 KERNEL32!BaseThreadStart+0x52
>
>
>
> David Wang wrote:
> > On Mar 31, 12:30 am, Freaky wrote:
> >> Hi there,
>
> >> we have a windows 2000 machine with IIS on it. It has 2 websites, one o=
f
> >> which has around 25 subfolders. Each subfolder is an ASP application.
>
> >> Lately, about 4 times a day, IIS becomes unresponsive. Nothing in the
> >> logging (event nor IIS access logs) seems to give me an indication of
> >> what is going wrong.
>
> >> Just guessing, but I think one of the ASP programs might get into an
> >> infinite loop, or be waiting for data forever. As these are 2 most
> >> likely issues the ASP applications could have. If that's the case I'm
> >> wondering why:
>
> >> a) IIS doesn't terminate the process after x time
> >> b) Why IIS doesn't serve other pages. I'm not very familiar with IIS,
> >> but it would seem to me any webserver of this size should be able to
> >> handle multiple requests at once.
>
> >> Any ideas on where to start troubleshooting?
>
> >> Unfortunately we can't just move the applications to another server,
> >> they depend on an application that runs on windows 2000 of which we do
> >> not have the source code and we're unable to get it working on windows =
2003.
>
> >> Kind regards
>
> > a. Architectural design. IIS5 is the last version which assumed user
> > applications are trustworthy enough to run within IIS itself (which
> > turned out to be a bad assumption because most user applications are
> > far more poorly written than IIS, but due to IIS's trusting nature, it
> > gets itself into trouble). Starting with IIS6, IIS assumes user
> > applications are badly written and malicious and runs defensively by
> > default, which has resulted in dramatically better security,
> > reliability, and performance that has been proven.
>
> > Both of those issues you point out are problems dealt with in ASP on
> > IIS6 and remain flaws in ASP on IIS5.
>
> > b. With IIS5, it is possible for user applications to monopolize the
> > system because IIS foolishly allowed it. This would not be possible
> > with IIS6 and how it treats ASP and user applications in general.
>
> > With properly written applications, IIS can handle tens of thousands
> > of concurrent requests at once. With improperly written applications,
> > it is possible for one single application to tank the application pool
> > by chewing up all threads that handle incoming requests. But this is
> > about as good as it gets -- you certainly cannot expect IIS to work
> > miracles with badly written applications. At some point, the
> > application developer has to take full responsibility for a hanging
> > server and not complain why IIS does not stay responsive regardless of
> > the junk being shoved at it.
>
> >http://blogs.msdn.com/david.wang/archive/2005/12/31/HOWTO_B asics_of_I...
>
> > Use IIS State or DebugDiag during the "unresponsive" times and see
> > what's going on.
>
> > //David
> >http://w3-4u.blogspot.com
> >http://blogs.msdn.com/David.Wang
> > //- Hide quoted text -
>
> - Show quoted text -