IIS Threads Blocked

Under heavy load, our webserver (IIS6) can get into a state whereby it
cannot communicate with the filer to retrieve content. The redirector
stops transmitting packets to the remote filer even though the remote
filer's server service is still active and accepting requests from
other servers. The IIS threads appear to hang, and many of them (with
a thread state of 5... "waiting") go from a "thread wait reason" of 15
to a "thread wait reason" of 27. Then, after some period of time,
such as a minute or two, the system recovers for a while. The
documentation I see for "thread wait reason" suggests that any value
of 20+ is reserved. Is this just a situation where perfmon is not
able to detect the accurate state of the IIS threads, or is a value of
27 for "thread wait reason" meaningful in the context of IIS?

Thanks,
Brian

Re: IIS Threads Blocked

Are the remote requests to the filer authenticated or unauthenticated
(i.e. anonymous). How many SMB control blocks are used by serverside
application to send traffic to the remote filer, and how many SMB
control blocks are available on the filer to handle incoming requests.

I'm guessing that you are saying that the filer is not throttled
because it is still handling requests from other servers, so my
question is whether the server itself is throttled from making any
more outbound SMB requests and thus blocking IIS threads being
requested by an HTTP client to retrieve resources from the filer.


//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//




On Aug 3, 11:29 am, Brian Hicks wrote:
> Under heavy load, our webserver (IIS6) can get into a state whereby it
> cannot communicate with the filer to retrieve content. The redirector
> stops transmitting packets to the remote filer even though the remote
> filer's server service is still active and accepting requests from
> other servers. The IIS threads appear to hang, and many of them (with
> a thread state of 5... "waiting") go from a "thread wait reason" of 15
> to a "thread wait reason" of 27. Then, after some period of time,
> such as a minute or two, the system recovers for a while. The
> documentation I see for "thread wait reason" suggests that any value
> of 20+ is reserved. Is this just a situation where perfmon is not
> able to detect the accurate state of the IIS threads, or is a value of
> 27 for "thread wait reason" meaningful in the context of IIS?
>
> Thanks,
> Brian

Re: IIS Threads Blocked

David,

Thanks for the reply. The remote requests to the filer are
authenticated. However, there are very few user accounts used for
accessing the remote content (~10), and each of those logon sessions
is cached, much in the way that IIS caches the anonymous web user for
each site. Therefore, there is not much logon/logoff activity
reported in the security event log. When I look at the shared folders
-> sessions section of the filer via the manage tool, there are only
those 10 users' sessions showing throughout the test. Redirector ->
current commands is low on the IIS server, at around 0 or 1, but can
climb as high as 6 or 7 under heavy load. However manipulating the
MaxCmds registry setting seems to have no effect on eliminating the
hangs. Filer "Current Sessions" metric is at 10. IISState logs taken
during the hang show that most of the IIS threads are blocked on IO
operations, such as CloseHandle(), and sometimes CreateFile() and
ReadFile(). During a typical hang, IISState logs taken at each end of
a 30-second interval show nearly identical output, indicating that
none of the IIS threads are doing any work during that interval.
During the hang, Redirector -> Current Commands will flatline at some
value, such as 7, and Redirector -> Packets Transmitted/sec will drop
and flatline at 0. Eventually, the block clears, perhaps due to IIS
dropping connections, though I am not certain why. There are a number
of Connection_Dropped messages in the httperr.log. It would seem that
the server's redirector is hung (IIS server, not file server), though
I do not know how to verify that. Do you have any suggestions on how
to verify that?

Thanks,
Brian

Re: IIS Threads Blocked

On Aug 6, 3:14 pm, Brian Hicks wrote:
> David,
>
> Thanks for the reply. The remote requests to the filer are
> authenticated. However, there are very few user accounts used for
> accessing the remote content (~10), and each of those logon sessions
> is cached, much in the way that IIS caches the anonymous web user for
> each site. Therefore, there is not much logon/logoff activity
> reported in the security event log. When I look at the shared folders
> -> sessions section of the filer via the manage tool, there are only
> those 10 users' sessions showing throughout the test. Redirector ->
> current commands is low on the IIS server, at around 0 or 1, but can
> climb as high as 6 or 7 under heavy load. However manipulating the
> MaxCmds registry setting seems to have no effect on eliminating the
> hangs. Filer "Current Sessions" metric is at 10. IISState logs taken
> during the hang show that most of the IIS threads are blocked on IO
> operations, such as CloseHandle(), and sometimes CreateFile() and
> ReadFile(). During a typical hang, IISState logs taken at each end of
> a 30-second interval show nearly identical output, indicating that
> none of the IIS threads are doing any work during that interval.
> During the hang, Redirector -> Current Commands will flatline at some
> value, such as 7, and Redirector -> Packets Transmitted/sec will drop
> and flatline at 0. Eventually, the block clears, perhaps due to IIS
> dropping connections, though I am not certain why. There are a number
> of Connection_Dropped messages in the httperr.log. It would seem that
> the server's redirector is hung (IIS server, not file server), though
> I do not know how to verify that. Do you have any suggestions on how
> to verify that?
>
> Thanks,
> Brian



I don't really know about the redirector so I don't have any more
suggestions.

I can only say that this scenario has been tested and verified against
a WS03 acting as the UNC server.


//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//