IIS 5 + ASP Pages crash after memory fragmentation

Hi All,

I'm having an annoying problem every month on a customer application. This
is the scenario:

Hardware (for both, WEB and DB server):
* Intel based server, with two Pentium 2GHz, HT
* 2GB RAM memory
* Plenty of hard disk space

Software (WEB server):
* Windows 2000 SP4 with IIS 5
* ASP pages accessing DB server directly (no components layer)

Software (DB server):
* Windows 2000 SP4
* SQL Server 2000 SP2

ASP Application:
* APP Isolation level Medium
* +- 600 ASP pages
* +- 3000 users

Problem:
* Every end of month, when the workload is heavier, i start to receive
errors like "Out_of_memory" and
"Not_enough_storage_is_available_to_complete_this_operation" , along with
several "500_Server_Error" in the website logfile.
* At the time the error occurs, the server memory is still about 50% free.
The rundll32 process associated with the website is about 300~500MB in size.
* The problem gets worse in time, until the server ceases to serve ASP
pages. Then a IIS restart is needed, a simple unload in the website
application doesn't works.
* Even though the server receives almost no requests during the night, the
problem persists from one day to another.

What i found so far, is that there is an issue related to memory
fragmentation, and the use of certain functions that need contiguous memory
to work (inspecting the error page that the user receives, it's frequently
pointed to a line with an ADODB.Recordset.GetRows() function). I think that
after a period of heavy load, the memory of the ASP process become too
fragmented, and then the GetRows function can't alocate the contiguous
memory it needs to perform successfully. There is also some other commands
that needs contiguous memory too.
The developer team already searched (and fixed) connections and recordsets
not being properly closed, but that didn't solved the problem. Wherever it
was possible, the GetRows functions were changed for Do-While loops in the
recordset, but in some cases it's mandatory.

Please someone, help me to see if i'm going in the right direction, and if
there is something that can be done, because migrating the application to
..Net (and enjoy the benefits of the app recycle feature of the application
pools) isn't an option for me at this time.

Thanks,
Yesca

Re: IIS 5 + ASP Pages crash after memory fragmentation

The memory you are running short on is the 2GB Virtual Memory map that each
process on a 32bit system is provided with. In this 2GB space, all dll's,
threads, memory allocations for web pages, etc. must exist. Over time, this
space can become fragmented, eventually preventing large allocations from
succeeding.

You can gauge the level of fragmentation by looking at Process:Virtual Bytes
& Process:Private Bytes in perfmon for the .exe in question. The two curves
should track in parallel more-or-less. If you see them diverging, then
fragmentation is occurring.

Unloading of a website is just a 'logical' unloading - it doesn't
necessarily flush the memory out. But restarting of the hosting process
will.

Probably the easiest thing to do is just to restart the processes
periodically (IIS6 does this automatically). Tracking down all of the
causes of fragmentation can take a very long time, each fix/change resulting
in a very small improvement. Only when you look at fixing all of them is a
material improvement made. You don't need to reboot the machine since the
issue is unique to the process.

Another method is to spread the site(s) into multiple high-isolation
processes (each process gets a separate 2GB space). You can also look at
the use of local vs global variables Local (to the page) variables that are
re-allocated every time to the same value (e.g. the ADO constants) would
drive more fragmentation than just setting them with a global value (single
allocation) or even better would be to use the Type-Lib (values are embedded
in a dll vs scattered around the VM map).

Pat


"Marcello Yesca" wrote in message
news:%23R2NdIqBIHA.3716@TK2MSFTNGP03.phx.gbl...
> Hi All,
>
> I'm having an annoying problem every month on a customer application. This
> is the scenario:
>
> Hardware (for both, WEB and DB server):
> * Intel based server, with two Pentium 2GHz, HT
> * 2GB RAM memory
> * Plenty of hard disk space
>
> Software (WEB server):
> * Windows 2000 SP4 with IIS 5
> * ASP pages accessing DB server directly (no components layer)
>
> Software (DB server):
> * Windows 2000 SP4
> * SQL Server 2000 SP2
>
> ASP Application:
> * APP Isolation level Medium
> * +- 600 ASP pages
> * +- 3000 users
>
> Problem:
> * Every end of month, when the workload is heavier, i start to receive
> errors like "Out_of_memory" and
> "Not_enough_storage_is_available_to_complete_this_operation" , along with
> several "500_Server_Error" in the website logfile.
> * At the time the error occurs, the server memory is still about 50% free.
> The rundll32 process associated with the website is about 300~500MB in
> size.
> * The problem gets worse in time, until the server ceases to serve ASP
> pages. Then a IIS restart is needed, a simple unload in the website
> application doesn't works.
> * Even though the server receives almost no requests during the night, the
> problem persists from one day to another.
>
> What i found so far, is that there is an issue related to memory
> fragmentation, and the use of certain functions that need contiguous
> memory to work (inspecting the error page that the user receives, it's
> frequently pointed to a line with an ADODB.Recordset.GetRows() function).
> I think that after a period of heavy load, the memory of the ASP process
> become too fragmented, and then the GetRows function can't alocate the
> contiguous memory it needs to perform successfully. There is also some
> other commands that needs contiguous memory too.
> The developer team already searched (and fixed) connections and recordsets
> not being properly closed, but that didn't solved the problem. Wherever it
> was possible, the GetRows functions were changed for Do-While loops in the
> recordset, but in some cases it's mandatory.
>
> Please someone, help me to see if i'm going in the right direction, and if
> there is something that can be done, because migrating the application to
> .Net (and enjoy the benefits of the app recycle feature of the application
> pools) isn't an option for me at this time.
>
> Thanks,
> Yesca
>
>
>

Re: IIS 5 + ASP Pages crash after memory fragmentation

Hi Pat,

I agree with you, the best workaround for now is to periodically restart
the www service. We are negotiating with the customer's datacenter an
upgrade to the IIS6.

Thank you and regards,
Yesca


"Pat [MSFT]" wrote in message
news:A7DBE0D2-3980-4B7A-B02B-18389E960703@microsoft.com...
> The memory you are running short on is the 2GB Virtual Memory map that
> each process on a 32bit system is provided with. In this 2GB space, all
> dll's, threads, memory allocations for web pages, etc. must exist. Over
> time, this space can become fragmented, eventually preventing large
> allocations from succeeding.
>
> You can gauge the level of fragmentation by looking at Process:Virtual
> Bytes & Process:Private Bytes in perfmon for the .exe in question. The
> two curves should track in parallel more-or-less. If you see them
> diverging, then fragmentation is occurring.
>
> Unloading of a website is just a 'logical' unloading - it doesn't
> necessarily flush the memory out. But restarting of the hosting process
> will.
>
> Probably the easiest thing to do is just to restart the processes
> periodically (IIS6 does this automatically). Tracking down all of the
> causes of fragmentation can take a very long time, each fix/change
> resulting in a very small improvement. Only when you look at fixing all
> of them is a material improvement made. You don't need to reboot the
> machine since the issue is unique to the process.
>
> Another method is to spread the site(s) into multiple high-isolation
> processes (each process gets a separate 2GB space). You can also look at
> the use of local vs global variables Local (to the page) variables that
> are re-allocated every time to the same value (e.g. the ADO constants)
> would drive more fragmentation than just setting them with a global value
> (single allocation) or even better would be to use the Type-Lib (values
> are embedded in a dll vs scattered around the VM map).
>
> Pat
>
>
> "Marcello Yesca" wrote in message
> news:%23R2NdIqBIHA.3716@TK2MSFTNGP03.phx.gbl...
>> Hi All,
>>
>> I'm having an annoying problem every month on a customer application.
>> This is the scenario:
>>
>> Hardware (for both, WEB and DB server):
>> * Intel based server, with two Pentium 2GHz, HT
>> * 2GB RAM memory
>> * Plenty of hard disk space
>>
>> Software (WEB server):
>> * Windows 2000 SP4 with IIS 5
>> * ASP pages accessing DB server directly (no components layer)
>>
>> Software (DB server):
>> * Windows 2000 SP4
>> * SQL Server 2000 SP2
>>
>> ASP Application:
>> * APP Isolation level Medium
>> * +- 600 ASP pages
>> * +- 3000 users
>>
>> Problem:
>> * Every end of month, when the workload is heavier, i start to receive
>> errors like "Out_of_memory" and
>> "Not_enough_storage_is_available_to_complete_this_operation" , along with
>> several "500_Server_Error" in the website logfile.
>> * At the time the error occurs, the server memory is still about 50%
>> free. The rundll32 process associated with the website is about 300~500MB
>> in size.
>> * The problem gets worse in time, until the server ceases to serve ASP
>> pages. Then a IIS restart is needed, a simple unload in the website
>> application doesn't works.
>> * Even though the server receives almost no requests during the night,
>> the problem persists from one day to another.
>>
>> What i found so far, is that there is an issue related to memory
>> fragmentation, and the use of certain functions that need contiguous
>> memory to work (inspecting the error page that the user receives, it's
>> frequently pointed to a line with an ADODB.Recordset.GetRows() function).
>> I think that after a period of heavy load, the memory of the ASP process
>> become too fragmented, and then the GetRows function can't alocate the
>> contiguous memory it needs to perform successfully. There is also some
>> other commands that needs contiguous memory too.
>> The developer team already searched (and fixed) connections and
>> recordsets not being properly closed, but that didn't solved the problem.
>> Wherever it was possible, the GetRows functions were changed for Do-While
>> loops in the recordset, but in some cases it's mandatory.
>>
>> Please someone, help me to see if i'm going in the right direction, and
>> if there is something that can be done, because migrating the application
>> to .Net (and enjoy the benefits of the app recycle feature of the
>> application pools) isn't an option for me at this time.
>>
>> Thanks,
>> Yesca
>>
>>
>>
>