Why is OutOfMemoryException thrown?

I can't figure out why once or twice per week, my C# server application
throws OutOfMemoryExceptions. There is plenty of memory in the machine. As
far as I can tell, when the exception is thrown, only 400 megs of 1gig total
is in use.

The exception is thrown while trying to update List<>. Here's a bit of the
stack trace:

===
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException'
was thrown.
at System.Collections.Generic.List`1.set_Capacity(Int32 value)
at System.Collections.Generic.List`1.EnsureCapacity(Int32 min)
at System.Collections.Generic.List`1.Add(T item)
at Mosaic.Core.StatUtils.ExtractRORs(MiniPerfItem[] perfItems, Int32
startIdx, Int32 endIdx, Boolean useNAOnNonPosBal) in
C:\Development\Mosaic\Mosaic-AM\Mosaic.Core.Shared\Code\Util ities\Statistical\StatUtils.cs:line
38
===

I have a lot of small objects in memory at the time of the exception but I'm
nowhere close to exhausting available RAM. Any ideas?

Re: Why is OutOfMemoryException thrown?

Lamont Sanford wrote:
> I can't figure out why once or twice per week, my C# server application
> throws OutOfMemoryExceptions. There is plenty of memory in the machine. As
> far as I can tell, when the exception is thrown, only 400 megs of 1gig total
> is in use.
>
> The exception is thrown while trying to update List<>. Here's a bit of the
> stack trace:
>
> ===
> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException'
> was thrown.
> at System.Collections.Generic.List`1.set_Capacity(Int32 value)
> at System.Collections.Generic.List`1.EnsureCapacity(Int32 min)
> at System.Collections.Generic.List`1.Add(T item)
> at Mosaic.Core.StatUtils.ExtractRORs(MiniPerfItem[] perfItems, Int32
> startIdx, Int32 endIdx, Boolean useNAOnNonPosBal) in
> C:\Development\Mosaic\Mosaic-AM\Mosaic.Core.Shared\Code\Util ities\Statistical\StatUtils.cs:line
> 38
> ===
>
> I have a lot of small objects in memory at the time of the exception but I'm
> nowhere close to exhausting available RAM. Any ideas?

Do you have any idea how big the list is at the point of failure?

--
Jon Skeet -
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

Re: Why is OutOfMemoryException thrown?

OutOfMemoryException really means "out of resources". Are you opening files
and not disposing them or something?

http://mrpmorris.blogspot.com/2006/12/cf-outofmemory-or-mayb e-not.html

Re: Why is OutOfMemoryException thrown?

> Do you have any idea how big the list is at the point of failure?
>

At max it could get up to the tens of thousands, but certainly never over
100k...

Re: Why is OutOfMemoryException thrown?

> OutOfMemoryException really means "out of resources". Are you opening
> files and not disposing them or something?
>
> http://mrpmorris.blogspot.com/2006/12/cf-outofmemory-or-mayb e-not.html
>

Interesting. I'll check out the article.

No, I don't make use of files to any major extent, but I do use a lot of
database connections. Maybe I'm not closing them properly?

Re: Why is OutOfMemoryException thrown?

Most likely - improperly closed / disposed connection objects means you run
out of Connection Pool connections, which can cause exceptions similar to
what you're seeing - especially if combined with other code.

1) use the same connection string througout your app.
2) open a connection just before you need to do database call.
3) close the connection immediately after - this returns it to the pool
4) be very careful with DataReaders as they hold open a connection.
-- Peter
Recursion: see Recursion
site: http://www.eggheadcafe.com
unBlog: http://petesbloggerama.blogspot.com
BlogMetaFinder: http://www.blogmetafinder.com



"Lamont Sanford" wrote:

> > OutOfMemoryException really means "out of resources". Are you opening
> > files and not disposing them or something?
> >
> > http://mrpmorris.blogspot.com/2006/12/cf-outofmemory-or-mayb e-not.html
> >
>
> Interesting. I'll check out the article.
>
> No, I don't make use of files to any major extent, but I do use a lot of
> database connections. Maybe I'm not closing them properly?
>
>
>

Re: Why is OutOfMemoryException thrown?

Lamont Sanford wrote:
> > Do you have any idea how big the list is at the point of failure?
>
> At max it could get up to the tens of thousands, but certainly never over
> 100k...

That sounds unlikely to be the issue then.

Can I suggest that you use perfmon to look at both the heap and the
handle count of your process?

--
Jon Skeet -
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

Re: Why is OutOfMemoryException thrown?

Lamont Sanford pisze:
> I can't figure out why once or twice per week, my C# server application
> throws OutOfMemoryExceptions. There is plenty of memory in the machine. As
> far as I can tell, when the exception is thrown, only 400 megs of 1gig total
> is in use.
>
> The exception is thrown while trying to update List<>. Here's a bit of the
> stack trace:
>
> ===
> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException'
> was thrown.
> at System.Collections.Generic.List`1.set_Capacity(Int32 value)
> at System.Collections.Generic.List`1.EnsureCapacity(Int32 min)
> at System.Collections.Generic.List`1.Add(T item)
> at Mosaic.Core.StatUtils.ExtractRORs(MiniPerfItem[] perfItems, Int32
> startIdx, Int32 endIdx, Boolean useNAOnNonPosBal) in
> C:\Development\Mosaic\Mosaic-AM\Mosaic.Core.Shared\Code\Util ities\Statistical\StatUtils.cs:line
> 38
> ===
>
> I have a lot of small objects in memory at the time of the exception but I'm
> nowhere close to exhausting available RAM. Any ideas?

Maybe due to extensively allocating and freeing memory, your memory is
too much fragmentized ?

Regards

Re: Why is OutOfMemoryException thrown?

>
> That sounds unlikely to be the issue then.
>
> Can I suggest that you use perfmon to look at both the heap and the
> handle count of your process?
>

Sure, I'll check it out. I've never used permon before. What signs/symptoms
should I be looking for?

Re: Why is OutOfMemoryException thrown?

> Maybe due to extensively allocating and freeing memory, your memory is
> too much fragmentized ?
>

How would I go about defragmenting it? Should I manually run a garbage
collection?

Re: Why is OutOfMemoryException thrown?

On Oct 12, 2:03 pm, "Lamont Sanford" wrote:
> > Can I suggest that you use perfmon to look at both the heap and the
> > handle count of your process?
>
> Sure, I'll check it out. I've never used permon before. What signs/symptoms
> should I be looking for?

Handles or heap memory rising constantly would be generally bad - but
you can also add performance traces for various .NET aspects, which
can be handy.

Jon

Re: Why is OutOfMemoryException thrown?

"Lamont Sanford" wrote in message
news:MNWdnaM7ReBx7ZLanZ2dnUVZ_jSdnZ2d@giganews.com...
>
>> Maybe due to extensively allocating and freeing memory, your memory is
>> too much fragmentized ?
>>
>
> How would I go about defragmenting it? Should I manually run a garbage
> collection?
>


No, there is nothing you can do about a fragmented heap (both managed and
process heap), other than restart the server.
Of course it's up to you to prevent possible heap fragmentation as much as
possible.
Therefore, I would like to know what version of the Framework and OS you
are running this on?
How large are the Lists in bytes, if you say up to 100k, do you mean 100kb
or something else, how did you measure this size?
How many of these list do you possibly create simultaneously?
Are you using Sockets connections?
Do you explicitly pin objects, or do you call into unmanaged code?

Willy.

Re: Why is OutOfMemoryException thrown?

I got the same problem before. Then I rewrited the application by replacing
DataSet by SqlDataReader.

"Lamont Sanford" wrote in message
news:b_WdnUmyg6El9pPanZ2dnUVZ_uWlnZ2d@giganews.com...
>I can't figure out why once or twice per week, my C# server application
>throws OutOfMemoryExceptions. There is plenty of memory in the machine. As
>far as I can tell, when the exception is thrown, only 400 megs of 1gig
>total is in use.
>
> The exception is thrown while trying to update List<>. Here's a bit of the
> stack trace:
>
> ===
> System.OutOfMemoryException: Exception of type
> 'System.OutOfMemoryException' was thrown.
> at System.Collections.Generic.List`1.set_Capacity(Int32 value)
> at System.Collections.Generic.List`1.EnsureCapacity(Int32 min)
> at System.Collections.Generic.List`1.Add(T item)
> at Mosaic.Core.StatUtils.ExtractRORs(MiniPerfItem[] perfItems, Int32
> startIdx, Int32 endIdx, Boolean useNAOnNonPosBal) in
> C:\Development\Mosaic\Mosaic-AM\Mosaic.Core.Shared\Code\Util ities\Statistical\StatUtils.cs:line
> 38
> ===
>
> I have a lot of small objects in memory at the time of the exception but
> I'm nowhere close to exhausting available RAM. Any ideas?
>
>
>
>
>

Re: Why is OutOfMemoryException thrown?

Try using LinkedList<> instead of List<> if you can. It won't be as
efficient at stopping memory fragmentation, but it will not call the
EnsureCapacity function, which, in essence, momentarily triples the
amount of memory used so that it can double the amount of memory used.
Okay, so it's not really "memory used" unless you're using structs.
It's memory used for pointers, which on a 32bit system for 100k should
400kilobytes -- not much! (Tripling that should be fine which makes me
wonder if it's a list of structs.)

Re: Why is OutOfMemoryException thrown?

> Therefore, I would like to know what version of the Framework and OS you
> are running this on?

I'm running .NET Framework 2.0 (VS 2005) on Win2003.

> How large are the Lists in bytes, if you say up to 100k, do you mean 100kb
> or something else, how did you measure this size?

I just meant that the size of the list (List.Count) would never exceed
100,000. If type 'T' is a double and double requires 8 bytes, the total
required memory shouldn't exceed 800,000 bytes... and I have hundreds of
gigs free.

> How many of these list do you possibly create simultaneously?

Only one can exist at a time, but these lists will be created over and over
again with different values.When a list is no longer needed, it isn't
explicitly Clear()'d, but it does fall out of scope so it should be GCd.

> Are you using Sockets connections?

Yes, sockets are used. No more than 30 inbound connections are active at any
given time.

> Do you explicitly pin objects, or do you call into unmanaged code?

What does "pin" mean? No, I don't make any calls into unmanaged code.

Re: Why is OutOfMemoryException thrown?

"Lamont Sanford" wrote in message
news:dP-dnSP5rd3T9I7anZ2dnUVZ_v-hnZ2d@giganews.com...
>> Therefore, I would like to know what version of the Framework and OS you
>> are running this on?
>
> I'm running .NET Framework 2.0 (VS 2005) on Win2003.
>
>> How large are the Lists in bytes, if you say up to 100k, do you mean
>> 100kb or something else, how did you measure this size?
>
> I just meant that the size of the list (List.Count) would never exceed
> 100,000. If type 'T' is a double and double requires 8 bytes, the total
> required memory shouldn't exceed 800,000 bytes... and I have hundreds of
> gigs free.
>

No you don't? The virtual memory space attributed to your process is only
2GB (supposing W2K3 32 bit). That means that you cannot allocate more than
this space for managed objects minus the space allocated by the application
the runtime and the Framework data and code, this leaves you with less than
1.6GB (non contiguous) in the most optimal case at process creation time.
Now, 800 Kb is not a big deal, but as these objects are larger than 85Kb,
they end on the LOH and as this one is never compacted, you may end with a
highly fragmented heap, if you don't tput an eye on object life time.

>> How many of these list do you possibly create simultaneously?
>
> Only one can exist at a time, but these lists will be created over and
> over again with different values.When a list is no longer needed, it isn't
> explicitly Clear()'d, but it does fall out of scope so it should be GCd.
>
Yes, but that doesn't mean that only one can exist at a time, the GC doesn't
run deterministically.

>> Are you using Sockets connections?
>
> Yes, sockets are used. No more than 30 inbound connections are active at
> any given time.
>
>> Do you explicitly pin objects, or do you call into unmanaged code?
>
> What does "pin" mean? No, I don't make any calls into unmanaged code.

Yes, you do (indirectly), there is no single managed application in the
world that don't call in unmanaged code. The socket classes need to pin the
send/reeive buffers when calling into the native Winsock API's. Pinning is
there to prevent the heap allocated object to move in memory when the GC
runs, however, pinning also prevents compactation of the GC heap.
Does it happen that these buffers have something in common with the above
mentioned List's?

Anyway, I would suggest you to inspect your memory allocation pattern by
running your code against a memory profiler.

Willy.

Re: Why is OutOfMemoryException thrown?

>> I just meant that the size of the list (List.Count) would never exceed
>> 100,000. If type 'T' is a double and double requires 8 bytes, the total
>> required memory shouldn't exceed 800,000 bytes... and I have hundreds of
>> gigs free.
>>
>
> No you don't?

Ooops, I meant hundreds of megs, not hundreds of gigs... sorry for the
typeo.

Thanks for the hints. I'll take your advice and hopefully I can get to the
bottom of this.