Devs think we"re hitting worker thread issues, I can"t find any evidence

Over the past month (a couple of weeks after one our monthly code pushes), a
few of the webservers in our farm have hit situations where they've popped
up the ASP.Net "Server Application Unavailable" page and won't serve any
asp.net pages until we recycle the app pool. What I've noticed when we have
these problems is very high memory usage by the app pool. When the pool is
recycled, memory usage drops to normal, and everything is fine.

Nothing is logged to the eventlog, and only errors we see from our
application has been the occasional "Exception of type
System.OutOfMemoryException was thrown". The same page doesn't error every
time, and the function I've seen throw exceptions is incredibly generic and
used heavily throughout our entire application (or so our devs have told
me). Our application doesn't even log an error every time this has
happened.

The http error log doesn't seem to have anything that corresponds directly
with the errors we see, although we are seeing a fair amount of
Connection_Dropped entries.

So, where I'm going with all of this is that I'm leaning more towards of a
code issue, but one of our developers is adamant that we're seeing worker
thread issues as described here:

http://support.microsoft.com/default.aspx?scid=kb;en-us;8212 68

We're not seeing any of the error messages described there, is there some
other way to confirm or reject his suspicions? Will making the changes
described there cause any issues?

I did run debugdiag, and it found a 3rd party component it rated with a 89%
probability of a leak which they're addressing, but our devs have said that
it's basically unrelated to the issue at hand.

Can anyone offer some advice here?

Re: Devs think we"re hitting worker thread issues, I can"t find anyevidence

Daniel Peterson wrote:
> Over the past month (a couple of weeks after one our monthly code
> pushes), a few of the webservers in our farm have hit situations where
> they've popped up the ASP.Net "Server Application Unavailable" page and
> won't serve any asp.net pages until we recycle the app pool. What I've
> noticed when we have these problems is very high memory usage by the app
> pool. When the pool is recycled, memory usage drops to normal, and
> everything is fine.
>
> Nothing is logged to the eventlog, and only errors we see from our
> application has been the occasional "Exception of type
> System.OutOfMemoryException was thrown". The same page doesn't error
> every time, and the function I've seen throw exceptions is incredibly
> generic and used heavily throughout our entire application (or so our
> devs have told me). Our application doesn't even log an error every
> time this has happened.
>
> The http error log doesn't seem to have anything that corresponds
> directly with the errors we see, although we are seeing a fair amount of
> Connection_Dropped entries.
>
> So, where I'm going with all of this is that I'm leaning more towards of
> a code issue, but one of our developers is adamant that we're seeing
> worker thread issues as described here:
>
> http://support.microsoft.com/default.aspx?scid=kb;en-us;8212 68
>
> We're not seeing any of the error messages described there, is there
> some other way to confirm or reject his suspicions? Will making the
> changes described there cause any issues?
>
> I did run debugdiag, and it found a 3rd party component it rated with a
> 89% probability of a leak which they're addressing, but our devs have
> said that it's basically unrelated to the issue at hand.
>
> Can anyone offer some advice here?

I would suspect a memory leak with the application being at fault, but
that's just me (simplest explanation is usually the right one, etc).

Have you tried tweaking the application pool recycling settings to
release the memory before it has a chance to becomes a problem?

It's not a solution, but if this problem is occuring on your production
servers then hopefully it'll keep your users from seeing the problems.

--
Chris M.

Remove pants to email me.

Re: Devs think we"re hitting worker thread issues, I can"t find any evidence

Hello,

I did forget to mention that. Right now we're recycling the pool every 29
hours, we're going to play around with those settings as well.

"Chris M" wrote in message
news:fhv2fd$dve$1@aioe.org...
> Daniel Peterson wrote:
>> Over the past month (a couple of weeks after one our monthly code
>> pushes), a few of the webservers in our farm have hit situations where
>> they've popped up the ASP.Net "Server Application Unavailable" page and
>> won't serve any asp.net pages until we recycle the app pool. What I've
>> noticed when we have these problems is very high memory usage by the app
>> pool. When the pool is recycled, memory usage drops to normal, and
>> everything is fine.
>>
>> Nothing is logged to the eventlog, and only errors we see from our
>> application has been the occasional "Exception of type
>> System.OutOfMemoryException was thrown". The same page doesn't error
>> every time, and the function I've seen throw exceptions is incredibly
>> generic and used heavily throughout our entire application (or so our
>> devs have told me). Our application doesn't even log an error every time
>> this has happened.
>>
>> The http error log doesn't seem to have anything that corresponds
>> directly with the errors we see, although we are seeing a fair amount of
>> Connection_Dropped entries.
>>
>> So, where I'm going with all of this is that I'm leaning more towards of
>> a code issue, but one of our developers is adamant that we're seeing
>> worker thread issues as described here:
>>
>> http://support.microsoft.com/default.aspx?scid=kb;en-us;8212 68
>>
>> We're not seeing any of the error messages described there, is there some
>> other way to confirm or reject his suspicions? Will making the changes
>> described there cause any issues?
>>
>> I did run debugdiag, and it found a 3rd party component it rated with a
>> 89% probability of a leak which they're addressing, but our devs have
>> said that it's basically unrelated to the issue at hand.
>>
>> Can anyone offer some advice here?
>
> I would suspect a memory leak with the application being at fault, but
> that's just me (simplest explanation is usually the right one, etc).
>
> Have you tried tweaking the application pool recycling settings to release
> the memory before it has a chance to becomes a problem?
>
> It's not a solution, but if this problem is occuring on your production
> servers then hopefully it'll keep your users from seeing the problems.
>
> --
> Chris M.
>
> Remove pants to email me.

Re: Devs think we"re hitting worker thread issues, I can"t find any

On Nov 20, 8:24 am, "Daniel Peterson" wrote:
> Over the past month (a couple of weeks after one our monthly code pushes), a
> few of the webservers in our farm have hit situations where they've popped
> up the ASP.Net "Server Application Unavailable" page and won't serve any
> asp.net pages until we recycle the app pool. What I've noticed when we have
> these problems is very high memory usage by the app pool. When the pool is
> recycled, memory usage drops to normal, and everything is fine.
>
> Nothing is logged to the eventlog, and only errors we see from our
> application has been the occasional "Exception of type
> System.OutOfMemoryException was thrown". The same page doesn't error every
> time, and the function I've seen throw exceptions is incredibly generic and
> used heavily throughout our entire application (or so our devs have told
> me). Our application doesn't even log an error every time this has
> happened.
>
> The http error log doesn't seem to have anything that corresponds directly
> with the errors we see, although we are seeing a fair amount of
> Connection_Dropped entries.
>
> So, where I'm going with all of this is that I'm leaning more towards of a
> code issue, but one of our developers is adamant that we're seeing worker
> thread issues as described here:
>
> http://support.microsoft.com/default.aspx?scid=kb;en-us;8212 68
>
> We're not seeing any of the error messages described there, is there some
> other way to confirm or reject his suspicions? Will making the changes
> described there cause any issues?
>
> I did run debugdiag, and it found a 3rd party component it rated with a 89%
> probability of a leak which they're addressing, but our devs have said that
> it's basically unrelated to the issue at hand.
>
> Can anyone offer some advice here?


Normally, if you see ASP.Net return "503 Server Application
Unavailable", it is a problem with the ASP.net pages, not ASP.Net
itself. Of course, most developers are adamant that their code is
perfect, so they will blame anything else -- this is expected.

However, when you see 503s when there is also high memory usage, there
could be another reason related to the high memory usage. So, let's
not blame the devs just yet. But, the dev being adamant about 821268
is unwarranted, especially if you are running ASP.Net 2.0.

But first, let me explain the 503. Then the high memory usage. And
finally how the two *could* be correlated and what to do.

Basically, the 503 is returned when a request arrives and ASP.Net's
request queue has reached its configurable maximum. Since ASP.Net
cannot put the request work item on its queue (it has reached its
limit), ASP.Net can only reject it with 503.

You can look at the ASP.Net queue with the perf counter "\ASP.Net
\Requests Queued". And the number of rejections with "\ASP.Net
\Requests Rejected".

Now, the reason that the queue grew and reached its limit is because
more requests are arriving than the dev's ASP.Net code is completing.
When requests are arriving at a faster pace than the dev's ASP.Net
code can complete, ASP.Net has to put those requests into its queue
and wait for the dev's code to finish each request before giving it
another.

The dev can never blame that "there's not enough threads" because of
the following fact -- if the dev's ASP.Net code cannot complete
requests faster than the incoming rate, NO AMOUNT OF THREADS will be
sufficient. The ASP.Net app will eventually bog down with requests and
queue up. Having more threads simply stall this fate, but it does not
deny the fact that queuing will happen and eventually reach 503.

Another way to view this is: Suppose you've got a swimming pool
(Queue) and it's getting filled with water (Requests) at a rate faster
than you can scoop (Dev ASP.Net code) it out with a bucket (thread).
You can add more threads (more people with SMALLER buckets whose total
volume matches that of the original bucket), but unless the people
scoop faster (have faster Dev ASP.Net code) with the smaller buckets,
the swimming pool will eventually overflow.

Common misconception is that adding more threads means adding more
buckets of the original size. It doesn't. The total volume of the
buckets is fixed to the CPU speed, no matter the number of buckets.

Now, I'm not saying that application should be happy with very few
threads. Some applications perf better with more threads -- for
example, instead of 4 threads, it wants 10. However, devs should tell
you this requirement up front, not when the app goes kaput. And
remember, you can't just keep increasing the number of threads because
it just stalls the inevitable queuing that I described earlier.
Basically, every app should be happy with some FIXED number of
threads. It is the app that keeps wanting MORE threads that is bad and
will get into this error condition.

Now, how high memory usage can impact this scenario is this -- the CLR
Threadpool will start throttling its threads (amongst many other
things) when it is in out-of-memory situation. This basically
increases latency of your dev's code, which exacerbates the pace at
which the queue fills up due to requests being processed more slowly.
It is more complicated to figure this out unless you turn on IIS6 ETW
Tracing, scan the IIS log for a spurt of throttling, and then confirm
with IIS6 ETW Tracing that the throttling happened when AppDomain is
being Entered.


Personally, I would look at why the "out of memory" is happening, to
remove the memory pressure. Then, if the queuing is still happening,
the dev needs to look through code to figure out how to decrease the
latency of the ASP.Net app. Or you may start throttling requests in
front of the IIS server, but that is a last resort hack. Recycling is
also a reasonable temporary hack until you figure out the real issue.


//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//

Re: Devs think we"re hitting worker thread issues, I can"t find any

On Nov 20, 8:50 am, "Daniel Peterson" wrote:
> Hello,
>
> I did forget to mention that. Right now we're recycling the pool every 29
> hours, we're going to play around with those settings as well.
>
> "Chris M" wrote in message
>
> news:fhv2fd$dve$1@aioe.org...
>
>
>
> > Daniel Peterson wrote:
> >> Over the past month (a couple of weeks after one our monthly code
> >> pushes), a few of the webservers in our farm have hit situations where
> >> they've popped up the ASP.Net "Server Application Unavailable" page and
> >> won't serve any asp.net pages until we recycle the app pool. What I've
> >> noticed when we have these problems is very high memory usage by the app
> >> pool. When the pool is recycled, memory usage drops to normal, and
> >> everything is fine.
>
> >> Nothing is logged to the eventlog, and only errors we see from our
> >> application has been the occasional "Exception of type
> >> System.OutOfMemoryException was thrown". The same page doesn't error
> >> every time, and the function I've seen throw exceptions is incredibly
> >> generic and used heavily throughout our entire application (or so our
> >> devs have told me). Our application doesn't even log an error every time
> >> this has happened.
>
> >> The http error log doesn't seem to have anything that corresponds
> >> directly with the errors we see, although we are seeing a fair amount of
> >> Connection_Dropped entries.
>
> >> So, where I'm going with all of this is that I'm leaning more towards of
> >> a code issue, but one of our developers is adamant that we're seeing
> >> worker thread issues as described here:
>
> >>http://support.microsoft.com/default.aspx?scid=kb;en-us;82 1268
>
> >> We're not seeing any of the error messages described there, is there some
> >> other way to confirm or reject his suspicions? Will making the changes
> >> described there cause any issues?
>
> >> I did run debugdiag, and it found a 3rd party component it rated with a
> >> 89% probability of a leak which they're addressing, but our devs have
> >> said that it's basically unrelated to the issue at hand.
>
> >> Can anyone offer some advice here?
>
> > I would suspect a memory leak with the application being at fault, but
> > that's just me (simplest explanation is usually the right one, etc).
>
> > Have you tried tweaking the application pool recycling settings to release
> > the memory before it has a chance to becomes a problem?
>
> > It's not a solution, but if this problem is occuring on your production
> > servers then hopefully it'll keep your users from seeing the problems.
>
> > --
> > Chris M.
>
> > Remove pants to email me.- Hide quoted text -
>
> - Show quoted text -



Normally, if you see ASP.Net return "503 Server Application
Unavailable", it is a problem with the ASP.net pages, not ASP.Net
itself. Of course, most developers are adamant that their code is
perfect, so they will blame anything else -- this is expected.

However, when you see 503s when there is also high memory usage, there
could be another reason related to the high memory usage. So, let's
not blame the devs just yet. But, the dev being adamant about 821268
is unwarranted, especially if you are running ASP.Net 2.0.

But first, let me explain the 503. Then the high memory usage. And
finally how the two *could* be correlated and what to do.

Basically, the 503 is returned when a request arrives and ASP.Net's
request queue has reached its configurable maximum. Since ASP.Net
cannot put the request work item on its queue (it has reached its
limit), ASP.Net can only reject it with 503.

You can look at the ASP.Net queue with the perf counter "\ASP.Net
\Requests Queued". And the number of rejections with "\ASP.Net
\Requests Rejected".

Now, the reason that the queue grew and reached its limit is because
more requests are arriving than the dev's ASP.Net code is completing.
When requests are arriving at a faster pace than the dev's ASP.Net
code can complete, ASP.Net has to put those requests into its queue
and wait for the dev's code to finish each request before giving it
another.

The dev can never blame that "there's not enough threads" because of
the following fact -- if the dev's ASP.Net code cannot complete
requests faster than the incoming rate, NO AMOUNT OF THREADS will be
sufficient. The ASP.Net app will eventually bog down with requests and
queue up. Having more threads simply stall this fate, but it does not
deny the fact that queuing will happen and eventually reach 503.

Another way to view this is: Suppose you've got a swimming pool
(Queue) and it's getting filled with water (Requests) at a rate faster
than you can scoop (Dev ASP.Net code) it out with a bucket (thread).
You can add more threads (more people with SMALLER buckets whose total
volume matches that of the original bucket), but unless the people
scoop faster (have faster Dev ASP.Net code) with the smaller buckets,
the swimming pool will eventually overflow.

Common misconception is that adding more threads means adding more
buckets of the original size. It doesn't. The total volume of the
buckets is fixed to the CPU speed, no matter the number of buckets.

Now, I'm not saying that application should be happy with very few
threads. Some applications perf better with more threads -- for
example, instead of 4 threads, it wants 10. However, devs should tell
you this requirement up front, not when the app goes kaput. And
remember, you can't just keep increasing the number of threads because
it just stalls the inevitable queuing that I described earlier.
Basically, every app should be happy with some FIXED number of
threads. It is the app that keeps wanting MORE threads that is bad and
will get into this error condition.

Now, how high memory usage can impact this scenario is this -- the CLR
Threadpool will start throttling its threads (amongst many other
things) when it is in out-of-memory situation. This basically
increases latency of your dev's code, which exacerbates the pace at
which the queue fills up due to requests being processed more slowly.
It is more complicated to figure this out unless you turn on IIS6 ETW
Tracing, scan the IIS log for a spurt of throttling, and then confirm
with IIS6 ETW Tracing that the throttling happened when AppDomain is
being Entered.


Personally, I would look at why the "out of memory" is happening, to
remove the memory pressure. Then, if the queuing is still happening,
the dev needs to look through code to figure out how to decrease the
latency of the ASP.Net app. Or you may start throttling requests in
front of the IIS server, but that is a last resort hack.


//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//