DBD::Oracle / Instant Client segfaults

hi

I'm having problems with DBD::Oracle built with Oracle Instant Client
on Linux (RHEL 3 U6). Occasionally when i run my application from
the command line the process receives a segmentation fault and
sometimes it terminates with an 'Out of memory' message.

I've been able to reproduce the issue with a simple script that just
connects to the database and does nothing else:

use strict;
use DBI;
my $dbdsn;
#use Digest::SHA1;
$dbdsn = 'dbi:Oracle:host=foo.bar.com;sid=devel5';
my $dbuser = 'baz';
my $dbpass = 'baz';
my $dbh = DBI->connect("$dbdsn", "$dbuser/$dbpass", '');
print "dbh: $dbh\n";
1;

When i run the script with the 'use Digest::SHA1;' line commented the
process occasionally terminates with a message like the following:

Out of memory during request for 964 bytes, total sbrk() is
1320960 bytes!

but segmentation faults don't seem to occur. The process seems to be
reserving quite a bit of memory for a simple script according to the
above message. The strace includes over 69000 brk() calls (the strace
attachment files don't seem to come through to this list).

When I load the Digest::SHA1 module segmentation faults occur once in
a while. The frequency changes but here's how many times i had to run
the script until i received the segmentation fault on 5 separate
runs: 83, 47, 51, 173, 5.

I've tested this on another machine with Red Hat Enterprise Linux 4
U2 and after over 4000 runs i haven't seen a single out of memory
error or segfault.

I've built DBD::Oracle 1.16 (DBI 1.42) with Oracle Instant Client
10gR2 on Red Hat Enterprise Linux 3.0 (update 6). The build procedure
followed the build recipe described by Tim Barrass and posted on dbi-
users earlier (the recipe includes setting up the IC directory
structure in a similar manner as with normal Oracle client
installation and a Makefile.PL patch). I've installed the following
IC packages:

instantclient-basic-linux32-10.2.0.1-20050713.zip
instantclient-sdk-linux32-10.2.0.1-20050713.zip
instantclient-sqlplus-linux32-10.2.0.1-20050713.zip

The problem doesn't appear on the same RHEL 3 machine with
DBD::Oracle built with the full Oracle10g R2 client (tested with over
3000 runs).

Is anyone having similar problems with DBD::Oracle built with the
Oracle Instant Client? Any ideas on how to resolve this problem?


br. aspa

Re: DBD::Oracle / Instant Client segfaults

hi

I've been doing some more tests concerning this issue and it seems
that it only appears with the combination of Oracle Instant client
libraries, Perl configured with -Dusemymalloc on RHEL 3. When I
configure Perl without -Dusemymalloc everything works fine. The issue
doesn't appear with Oracle10g full client on RHEL 3 or with OIC on
RHEL 4, Solaris 8 or HP-UX 11i v1.

I've reported this issue to Oracle but does anyone have clues on what
is causing this or suggestions on how to get this fixed on the Perl
side? I'd really need to use -Dusemymalloc and Oracle Instant Client.

br. aspa



On 2005-12-29, at 14.26, Marko Asplund wrote:

> I'm having problems with DBD::Oracle built with Oracle Instant
> Client on Linux (RHEL 3 U6). Occasionally when i run my
> application from the command line the process receives a
> segmentation fault and sometimes it terminates with an 'Out of
> memory' message.
>
> I've been able to reproduce the issue with a simple script that
> just connects to the database and does nothing else:
>
> use strict;
> use DBI;
> my $dbdsn;
> #use Digest::SHA1;
> $dbdsn = 'dbi:Oracle:host=foo.bar.com;sid=devel5';
> my $dbuser = 'baz';
> my $dbpass = 'baz';
> my $dbh = DBI->connect("$dbdsn", "$dbuser/$dbpass", '');
> print "dbh: $dbh\n";
> 1;
>
> When i run the script with the 'use Digest::SHA1;' line commented
> the process occasionally terminates with a message like the following:
>
> Out of memory during request for 964 bytes, total sbrk() is
> 1320960 bytes!
>
> but segmentation faults don't seem to occur. The process seems to
> be reserving quite a bit of memory for a simple script according to
> the above message. The strace includes over 69000 brk() calls (the
> strace attachment files don't seem to come through to this list).
>
> When I load the Digest::SHA1 module segmentation faults occur once
> in a while. The frequency changes but here's how many times i had
> to run the script until i received the segmentation fault on 5
> separate runs: 83, 47, 51, 173, 5.
>
> I've tested this on another machine with Red Hat Enterprise Linux 4
> U2 and after over 4000 runs i haven't seen a single out of memory
> error or segfault.
>
> I've built DBD::Oracle 1.16 (DBI 1.42) with Oracle Instant Client
> 10gR2 on Red Hat Enterprise Linux 3.0 (update 6). The build
> procedure followed the build recipe described by Tim Barrass and
> posted on dbi-users earlier (the recipe includes setting up the IC
> directory structure in a similar manner as with normal Oracle
> client installation and a Makefile.PL patch). I've installed the
> following IC packages:
>
> instantclient-basic-linux32-10.2.0.1-20050713.zip
> instantclient-sdk-linux32-10.2.0.1-20050713.zip
> instantclient-sqlplus-linux32-10.2.0.1-20050713.zip
>
> The problem doesn't appear on the same RHEL 3 machine with
> DBD::Oracle built with the full Oracle10g R2 client (tested with
> over 3000 runs).
>
> Is anyone having similar problems with DBD::Oracle built with the
> Oracle Instant Client? Any ideas on how to resolve this problem?
>
>
> br. aspa

Re: DBD::Oracle / Instant Client segfaults

On Sat, Jan 28, 2006 at 09:48:17PM +0200, Marko Asplund wrote:
> hi
>
> I've been doing some more tests concerning this issue and it seems
> that it only appears with the combination of Oracle Instant client
> libraries, Perl configured with -Dusemymalloc on RHEL 3. When I
> configure Perl without -Dusemymalloc everything works fine.

By which I presume you mean it appears to work fine because it doesn't
segfault. An underlying problem, such as memory corruption, may still
be present, just not causing any visible problem in you application.
(Sorry to be pessimistic.)

There are memory allocation debugging tools that may help to isolate
the original problem and so give some confidence that it's not present
in the other configurations.

> The issue doesn't appear with Oracle10g full client on RHEL 3 or with
> OIC on RHEL 4, Solaris 8 or HP-UX 11i v1.

Thanks for doing the legwork to narrow it down.

> I've reported this issue to Oracle but does anyone have clues on what
> is causing this or suggestions on how to get this fixed on the Perl
> side? I'd really need to use -Dusemymalloc and Oracle Instant Client.

Why?

The Perl -DPERL_POLLUTE_MALLOC option may be of use. But you might just
be trading one can of worms for another.

Also, try http://www.data-plan.com/public/DBD-Oracle-1.17-RC6.tar.gz
I don't think there are any relevant changes but I'd be interested to
know if you get similar behaviour on the various systems.

Tim.

Re: DBD::Oracle / Instant Client segfaults

On 2006-02-01, at 00.45, Tim Bunce wrote:

> By which I presume you mean it appears to work fine because it doesn't
> segfault. An underlying problem, such as memory corruption, may still
> be present, just not causing any visible problem in you application.
> (Sorry to be pessimistic.)

You're right, i may have been too optimistic. The issue has typically
required around 50 to 150 script runs to reproduce on RHEL 3. On RHEL
4 i've done more than 4000 runs without seeing segfaults, but as you
point out this still doesn't rule out the possibility of memory
corruption.

> There are memory allocation debugging tools that may help to isolate
> the original problem and so give some confidence that it's not present
> in the other configurations.

I'am not really familiar with such tools, can you recommend a good
tool for this? How about Valgrind?

>> I've reported this issue to Oracle but does anyone have clues on what
>> is causing this or suggestions on how to get this fixed on the Perl
>> side? I'd really need to use -Dusemymalloc and Oracle Instant Client.
>
> Why?

I don't have empirical data on this but several sources (among others
Perl distribution INSTALL file, mod_perl Cookbook) hint that -
Dusemymalloc may give better performance and use less memory.

> Also, try http://www.data-plan.com/public/DBD-Oracle-1.17-RC6.tar.gz
> I don't think there are any relevant changes but I'd be interested to
> know if you get similar behaviour on the various systems.

I'll give it a try when i have the chance (possibly during next week).


aspa

Re: DBD::Oracle / Instant Client segfaults

On Wed, Feb 01, 2006 at 01:52:02PM +0200, Marko Asplund wrote:
> On 2006-02-01, at 00.45, Tim Bunce wrote:
>
> >There are memory allocation debugging tools that may help to isolate
> >the original problem and so give some confidence that it's not present
> >in the other configurations.
>
> I'am not really familiar with such tools, can you recommend a good
> tool for this? How about Valgrind?

Try http://www.google.com/search?q=malloc+debug+tools

> >>I've reported this issue to Oracle but does anyone have clues on what
> >>is causing this or suggestions on how to get this fixed on the Perl
> >>side? I'd really need to use -Dusemymalloc and Oracle Instant Client.
> >
> >Why?
>
> I don't have empirical data on this but several sources (among others
> Perl distribution INSTALL file, mod_perl Cookbook) hint that -
> Dusemymalloc may give better performance and use less memory.

That doesn't sound like a good enough reason. More like premature
optimization. I'd only return to this option once you've proven you
have performance problems and you've then exhausted all higher-level
optimizations.

Tim.