AppPool/W3WP Crashes.

Hi,

Our production server - w3wp/AppPool crashes intermittenly. When crashes it
leaves the following messages in the logs.

HTTP.exe log has statments as
1. "Connection_Dropped DefaultAppPool"
2. "Connection_Abandoned_By_AppPool DefaultAppPool"

Event Logs has statements as
1. "A process serving application pool 'DefaultAppPool' exceeded time
limits during shut down. The process id was '3180'. "
2. "A process serving application pool 'DefaultAppPool' suffered a fatal
communication error with the World Wide Web Publishing Service. The process
id was '1376'. The data field contains the error number. "
3. "A process serving application pool 'DefaultAppPool' terminated
unexpectedly. The process id was '216'. The process exit code was '0x1'."
4. "A process serving application pool 'DefaultAppPool' failed to respond
to a ping. The process id was '3908'."
5. "Application pool 'DefaultAppPool' is being automatically disabled due
to a series of failures in the process(es) serving that application pool."

Our production System is running with
1. win 2003 server SP2
2. IIS 6.0
3. 3 GB RAM

But when many hits comes to the site w3wp keeps on increasing its size and
some times if it goes just beyond 350+MB of physical memory usage it is
crashed. But some times it is crashed when its size is 600+MB.

And our production server has the following limitations as well.
1. Cannot Auto recycle W3WP.exe during operating hours:
Because INLIS maintains Session - InProc mode as ePayment is involved.

2. Cannot use Web Garden [More than one W3WP.exe on IIS]:
All the Searches on the site communicates to a single VC++ utility.
And this utility gets data from ADABAS which is running on different network
by utilizing a leased line channel.

My queries regarding the above issue:
1. What could be reason the w3wp/apppool crashes?
2. How can I avoid w3wp/apppool crashes?
3. How can I maximize the RAM usage of w3wp/apppool to use GB of RAM to
cater to serve more no of hits on the site?

--
Thanks & Regards,

Velan.R

Re: AppPool/W3WP Crashes.

On May 12, 3:59 am, velanwins
wrote:
> Hi,
>
> Our production server - w3wp/AppPool crashes intermittenly. When crashes it
> leaves the following messages in the logs.
>
> HTTP.exe log has statments as
> 1. "Connection_Dropped DefaultAppPool"
> 2. "Connection_Abandoned_By_AppPool DefaultAppPool"
>
> Event Logs has statements as
> 1. "A process serving application pool 'DefaultAppPool' exceeded time
> limits during shut down. The process id was '3180'. "
> 2. "A process serving application pool 'DefaultAppPool' suffered a fatal
> communication error with the World Wide Web Publishing Service. The process
> id was '1376'. The data field contains the error number. "
> 3. "A process serving application pool 'DefaultAppPool' terminated
> unexpectedly. The process id was '216'. The process exit code was '0x1'."
> 4. "A process serving application pool 'DefaultAppPool' failed to respond
> to a ping. The process id was '3908'."
> 5. "Application pool 'DefaultAppPool' is being automatically disabled due
> to a series of failures in the process(es) serving that application pool."
>
> Our production System is running with
> 1. win 2003 server SP2
> 2. IIS 6.0
> 3. 3 GB RAM
>
> But when many hits comes to the site w3wp keeps on increasing its size and
> some times if it goes just beyond 350+MB of physical memory usage it is
> crashed. But some times it is crashed when its size is 600+MB.
>
> And our production server has the following limitations as well.
> 1. Cannot Auto recycle W3WP.exe during operating hours:
> Because INLIS maintains Session - InProc mode as ePayment is involved.
>
> 2. Cannot use Web Garden [More than one W3WP.exe on IIS]:
> All the Searches on the site communicates to a single VC++ utility.
> And this utility gets data from ADABAS which is running on different network
> by utilizing a leased line channel.
>
> My queries regarding the above issue:
> 1. What could be reason the w3wp/apppool crashes?
> 2. How can I avoid w3wp/apppool crashes?
> 3. How can I maximize the RAM usage of w3wp/apppool to use GB of RAM to
> cater to serve more no of hits on the site?
>
> --
> Thanks & Regards,
>
> Velan.R


The entries in HTTP log and Event log simply indicate that something
crashed the w3wp.exe. You will need to debug to determine what is
crashing it -- it is most likely caused by the server-side
application, and debugging and getting a stack trace will confirm.
Memory size MAY NOT have any correlation with logical bugs causing a
crash - which is what you observed.

I imagine that the crashes are bad for you because it is just like an
unscheduled auto-recycle -- and you said that you cannot auto-recycle
during operating hours due to ePayments, so the crashes must be
causing problems for your ePayments.

All your questions are really about the server-side application
software you run on IIS and not IIS itself.
1. Usually crashes are caused by logical bugs in server-software
running on IIS6. See this URL for info (
http://blogs.msdn.com/david.wang/archive/2005/08/29/HOWTO_Un derstand_and_Diagnose_an_AppPool_Crash.aspx
)
2. You avoid crashes by either having the bug fixed or avoid doing
things that trigger the bug. I suggest you read through the following
URLs on how to troubleshoot and get the crash diagnosed (
http://blogs.msdn.com/david.wang/archive/2005/12/31/HOWTO_Ba sics_of_IIS6_Troubleshooting.aspx
http://blogs.msdn.com/david.wang/archive/2005/08/29/HOWTO_Un derstand_and_Diagnose_an_AppPool_Crash.aspx
)
3. IIS allows the application maximum usage of RAM by default, so the
question is whether the server-side application can be configured or
otherwise take advantage of the available resources. Often, the server-
side application has performance or contention problems which prevent
it from using all available memory or CPU -- which is best addressed
by fixing the server-side application. IIS6 configuration can only
work around *some* of the issues like a bandaid, but if you want a
real solution you have to fix the application.

The limitations on your server are also caused by the application:
1. If the application stored session state outside of w3wp.exe, you
can auto-recycle it. But the application does not; hence you cannot
auto-recycle w3wp.exe (
http://blogs.msdn.com/david.wang/archive/2005/09/19/Why-do-I -lose-ASP-Session-State-on-IIS6.aspx
)
2. Sounds like the VC++ utility does not support multiple instances of
itself simultaneously, so it won't work with Web Garden. And unless
you can articulate the reason you need Web Garden, you should not use
it ( http://blogs.msdn.com/david.wang/archive/2006/03/14/Thoughts -on-Application-Pools-running-out-of-threads.aspx
)

Unfortunately, most of the things you are asking for can never be
addressed with any feature within IIS6, as sometimes server-side
applications have deficiencies that are such gaping wounds that IIS
bandaids cannot address.


//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//

Re: AppPool/W3WP Crashes.

Hi David,

Thanks for your valuable inputs on our production server’s W3WP/AppPool
issue.

1. On production server I have installed IIS Diagnostics (32bit) tools and
used the Debug Diagnostics Tool1.0 to get the stack trace. The following
rules I have configured against W3WP / inetinfo:
a. A Crash Rule on W3WP.exe with Stack Trace
b. A Crash Rule on inetinfo.exe with Access Violation exception with
Stack trace
c. IIS Hang Rule on inetinfo.exe
d. Leak Rule on inetinfo.exe

Today (14/May/2007 03:09 PM), we had faced the same kind of crash on the
site and we got “Server Unavailable” response from the server.
In that point the RAM usage of the W3WP was 997+ MB and CPU usage was 15-20%.

I have expected that the Crash Rule of W3WP might have executed; but the
Hang Rule was executed and completed. I was trying to analyze the logs, but I
couldn’t understand.

Could I seek your help in a way that you can check the stack trace for us
and help to identify the root cause? If so, can I share these logs as a
cabinet file with HTTP err/Event logs/IIS logs/Stack Trace?

2. As per the following link
http://support.microsoft.com/Default.aspx?id=885654, I had tried to use
“RegMon.exe” on our server but it couldn’t. So, I have used MS Authentication
& Access Control Diagnostics Tool1.0 to get to know the User Rights or
Privileges on the server. The following user accounts had error message as
“Not found privileges: Adjust memory quotas for a process; Replace a process
level token”
1. NT Authority\Local Services
2. NT Authority\Network Services.

--
Thanks & Regards,

Velan.R


"David Wang" wrote:

> On May 12, 3:59 am, velanwins
> wrote:
> > Hi,
> >
> > Our production server - w3wp/AppPool crashes intermittenly. When crashes it
> > leaves the following messages in the logs.
> >
> > HTTP.exe log has statments as
> > 1. "Connection_Dropped DefaultAppPool"
> > 2. "Connection_Abandoned_By_AppPool DefaultAppPool"
> >
> > Event Logs has statements as
> > 1. "A process serving application pool 'DefaultAppPool' exceeded time
> > limits during shut down. The process id was '3180'. "
> > 2. "A process serving application pool 'DefaultAppPool' suffered a fatal
> > communication error with the World Wide Web Publishing Service. The process
> > id was '1376'. The data field contains the error number. "
> > 3. "A process serving application pool 'DefaultAppPool' terminated
> > unexpectedly. The process id was '216'. The process exit code was '0x1'."
> > 4. "A process serving application pool 'DefaultAppPool' failed to respond
> > to a ping. The process id was '3908'."
> > 5. "Application pool 'DefaultAppPool' is being automatically disabled due
> > to a series of failures in the process(es) serving that application pool."
> >
> > Our production System is running with
> > 1. win 2003 server SP2
> > 2. IIS 6.0
> > 3. 3 GB RAM
> >
> > But when many hits comes to the site w3wp keeps on increasing its size and
> > some times if it goes just beyond 350+MB of physical memory usage it is
> > crashed. But some times it is crashed when its size is 600+MB.
> >
> > And our production server has the following limitations as well.
> > 1. Cannot Auto recycle W3WP.exe during operating hours:
> > Because INLIS maintains Session - InProc mode as ePayment is involved.
> >
> > 2. Cannot use Web Garden [More than one W3WP.exe on IIS]:
> > All the Searches on the site communicates to a single VC++ utility.
> > And this utility gets data from ADABAS which is running on different network
> > by utilizing a leased line channel.
> >
> > My queries regarding the above issue:
> > 1. What could be reason the w3wp/apppool crashes?
> > 2. How can I avoid w3wp/apppool crashes?
> > 3. How can I maximize the RAM usage of w3wp/apppool to use GB of RAM to
> > cater to serve more no of hits on the site?
> >
> > --
> > Thanks & Regards,
> >
> > Velan.R
>
>
> The entries in HTTP log and Event log simply indicate that something
> crashed the w3wp.exe. You will need to debug to determine what is
> crashing it -- it is most likely caused by the server-side
> application, and debugging and getting a stack trace will confirm.
> Memory size MAY NOT have any correlation with logical bugs causing a
> crash - which is what you observed.
>
> I imagine that the crashes are bad for you because it is just like an
> unscheduled auto-recycle -- and you said that you cannot auto-recycle
> during operating hours due to ePayments, so the crashes must be
> causing problems for your ePayments.
>
> All your questions are really about the server-side application
> software you run on IIS and not IIS itself.
> 1. Usually crashes are caused by logical bugs in server-software
> running on IIS6. See this URL for info (
> http://blogs.msdn.com/david.wang/archive/2005/08/29/HOWTO_Un derstand_and_Diagnose_an_AppPool_Crash.aspx
> )
> 2. You avoid crashes by either having the bug fixed or avoid doing
> things that trigger the bug. I suggest you read through the following
> URLs on how to troubleshoot and get the crash diagnosed (
> http://blogs.msdn.com/david.wang/archive/2005/12/31/HOWTO_Ba sics_of_IIS6_Troubleshooting.aspx
> http://blogs.msdn.com/david.wang/archive/2005/08/29/HOWTO_Un derstand_and_Diagnose_an_AppPool_Crash.aspx
> )
> 3. IIS allows the application maximum usage of RAM by default, so the
> question is whether the server-side application can be configured or
> otherwise take advantage of the available resources. Often, the server-
> side application has performance or contention problems which prevent
> it from using all available memory or CPU -- which is best addressed
> by fixing the server-side application. IIS6 configuration can only
> work around *some* of the issues like a bandaid, but if you want a
> real solution you have to fix the application.
>
> The limitations on your server are also caused by the application:
> 1. If the application stored session state outside of w3wp.exe, you
> can auto-recycle it. But the application does not; hence you cannot
> auto-recycle w3wp.exe (
> http://blogs.msdn.com/david.wang/archive/2005/09/19/Why-do-I -lose-ASP-Session-State-on-IIS6.aspx
> )
> 2. Sounds like the VC++ utility does not support multiple instances of
> itself simultaneously, so it won't work with Web Garden. And unless
> you can articulate the reason you need Web Garden, you should not use
> it ( http://blogs.msdn.com/david.wang/archive/2006/03/14/Thoughts -on-Application-Pools-running-out-of-threads.aspx
> )
>
> Unfortunately, most of the things you are asking for can never be
> addressed with any feature within IIS6, as sometimes server-side
> applications have deficiencies that are such gaping wounds that IIS
> bandaids cannot address.
>
>
> //David
> http://w3-4u.blogspot.com
> http://blogs.msdn.com/David.Wang
> //
>
>