DBI memory leaks?

DBI memory leaks?

am 24.05.2007 04:03:07 von chris.wagner

Hi, I'm trying to troubleshoot a massive memory leak problem. I wanted
to see if anybody knew of any leak problems in the module versions I'm
using. Or had any ideas on how to troubleshoot it. I have a non
database enabled version of this script and it doesn't show the
problem. The script collects snmp information from devices and writes
it to a database. It's multithreaded and runs ok with 100 threads until
the leak kills the process. The non db version runs fine with 100
threads. The process starts off at about 500 MB of memory usage but
after a short while the usage starts increasing almost like watching a
clock. After a few hours the usage is up around 2.5 to 3 GB and the
script either dies or self terminates. It doesn't save but a few megs of
data in memory. Here's what I'm running. Thanks.

[user@host ~]$ perl
use DBI;
use DBD::mysql;
print $DBI::VERSION, "\n";
print $DBD::mysql::VERSION;
1.51
3.0006

[user@host ~]$ uname -a
SunOS netdisc 5.8 Generic_117350-36 sun4u sparc SUNW,Sun-Fire-V240

[user@host ~]$ perl -V
Set up gcc environment - 3.4.2
Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
Platform:
osname=solaris, osvers=2.6, archname=sun4-solaris-thread-multi
uname='sunos axe 5.6 generic_105181-33 sun4u sparc sunw,ultra-60 '
config_args='-ders -Dcc=cc -Dusethreads -Duseithreads -Ud_sigsetjmp
-Uinstal
lusrbinperl -Ulocincpth= -Uloclibpth= -Accflags=-DUSE_SITECUSTOMIZE
-Duselargefi
les -Accflags=-DNO_HASH_SEED -Dinc_version_list=5.8.7/$archname 5.8.7
5.8.6/$arc
hname 5.8.6 5.8.4/$archname 5.8.4 5.8.3/$archname 5.8.3 5.8.2/$archname
5.8.2 5.
8.1/$archname 5.8.1 5.8.0/$archname 5.8.0
-Dconfig_heavy=Config_static.pl -Dpref
ix=/opt/ActivePerl-5.8 -Dcf_by=ActiveState
-Dcf_email=support@ActiveState.com'
hint=recommended, useposix=true, d_sigaction=define
usethreads=define use5005threads=undef useithreads=define
usemultiplicity=de
fine
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='gcc', ccflags ='-D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT
-DUSE_SITECUSTOM
IZE -DNO_HASH_SEED -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
-fno-strict-aliasi
ng -pipe',
optimize='-O',
cppflags='-D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT
-DUSE_SITECUSTOMIZE -DNO_H
ASH_SEED -fno-strict-aliasing -pipe'
ccversion='', gccversion='3.4.2', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize
=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='gcc', ldflags =' -L/usr/lib -L/usr/ccs/lib
-L/opt/SUNWspro/WS6U2/lib '
libpth=/usr/lib /usr/ccs/lib /opt/SUNWspro/WS6U2/lib /usr/local/lib
libs=-lsocket -lnsl -ldl -lm -lpthread -lc
perllibs=-lsocket -lnsl -ldl -lm -lpthread -lc
libc=, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
cccdlflags='-fPIC', lddlflags='-G -L/usr/lib -L/usr/ccs/lib
-L/opt/SUNWspro/
WS6U2/lib'


Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY PERL_IMPLICIT_CONTEXT
PERL_MALLOC_WRAP USE_ITHREADS USE_LARGE_FILES
USE_PERLIO USE_REENTRANT_API USE_SITECUSTOMIZE
Locally applied patches:
ActivePerl Build 817 [257965]
Iin_load_module moved for compatibility with build 806
PerlEx support in CGI::Carp
Less verbose ExtUtils::Install and Pod::Find
Patch for CAN-2005-0448 from Debian with modifications
Partly reverted 24733 to preserve binary compatibilty
27528 win32_pclose() error exit doesn't unlock mutex
27527 win32_async_check() can loop indefinitely
27515 ignore directories when searching @INC
27359 Fix -d:Foo=bar syntax
27210 Fix quote typo in c2ph
27203 Allow compiling swigged C++ code
27200 Make stat() on Windows handle trailing slashes correctly
27194 Get perl_fini() running on HP-UX again
27133 Initialise lastparen in the regexp structure
27034 Avoid "Prototype mismatch" warnings with autouse
26970 Make Passive mode the default for Net::FTP
26921 Avoid getprotobyname/number calls in IO::Socket::INET
26897,26903 Make common IPPROTO_* constants always available
26670 Make '-s' on the shebang line parse -foo=bar switches
26379 Fix alarm() for Windows 2003
26087 Storable 0.1 compatibility
25861 IO::File performace issue
25084 long groups entry could cause memory exhaustion
24699 ICMP_UNREACHABLE handling in Net::Ping
Built under solaris
Compiled at Mar 20 2006 20:36:33
@INC:
/opt/ActivePerl-5.8/lib/5.8.8/sun4-solaris-thread-multi
/opt/ActivePerl-5.8/lib/5.8.8
/opt/ActivePerl-5.8/lib/site_perl/5.8.8/sun4-solaris-thread- multi
/opt/ActivePerl-5.8/lib/site_perl/5.8.8
/opt/ActivePerl-5.8/lib/site_perl
.

MySQL Server is 5.0.21-standard



--
Chris Wagner
CBTS
GE Aircraft Engines
Chris.Wagner@ae.ge.com

--
MySQL Perl Mailing List
For list archives: http://lists.mysql.com/perl
To unsubscribe: http://lists.mysql.com/perl?unsub=gcdmp-msql-mysql-modules@m .gmane.org

Re: DBI memory leaks?

am 24.05.2007 05:16:49 von Ruben I Safir

This sounds like an OS problem.

Ruben

On Wed, May 23, 2007 at 10:03:07PM -0400, Wagner, Chris (GEAE, CBTS) wrote:
> Hi, I'm trying to troubleshoot a massive memory leak problem. I wanted
> to see if anybody knew of any leak problems in the module versions I'm
> using. Or had any ideas on how to troubleshoot it. I have a non
> database enabled version of this script and it doesn't show the
> problem. The script collects snmp information from devices and writes
> it to a database. It's multithreaded and runs ok with 100 threads until
> the leak kills the process. The non db version runs fine with 100
> threads. The process starts off at about 500 MB of memory usage but
> after a short while the usage starts increasing almost like watching a
> clock. After a few hours the usage is up around 2.5 to 3 GB and the
> script either dies or self terminates. It doesn't save but a few megs of
> data in memory. Here's what I'm running. Thanks.
>
> [user@host ~]$ perl
> use DBI;
> use DBD::mysql;
> print $DBI::VERSION, "\n";
> print $DBD::mysql::VERSION;
> 1.51
> 3.0006
>
> [user@host ~]$ uname -a
> SunOS netdisc 5.8 Generic_117350-36 sun4u sparc SUNW,Sun-Fire-V240
>
> [user@host ~]$ perl -V
> Set up gcc environment - 3.4.2
> Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
> Platform:
> osname=solaris, osvers=2.6, archname=sun4-solaris-thread-multi
> uname='sunos axe 5.6 generic_105181-33 sun4u sparc sunw,ultra-60 '
> config_args='-ders -Dcc=cc -Dusethreads -Duseithreads -Ud_sigsetjmp
> -Uinstal
> lusrbinperl -Ulocincpth= -Uloclibpth= -Accflags=-DUSE_SITECUSTOMIZE
> -Duselargefi
> les -Accflags=-DNO_HASH_SEED -Dinc_version_list=5.8.7/$archname 5.8.7
> 5.8.6/$arc
> hname 5.8.6 5.8.4/$archname 5.8.4 5.8.3/$archname 5.8.3 5.8.2/$archname
> 5.8.2 5.
> 8.1/$archname 5.8.1 5.8.0/$archname 5.8.0
> -Dconfig_heavy=Config_static.pl -Dpref
> ix=/opt/ActivePerl-5.8 -Dcf_by=ActiveState
> -Dcf_email=support@ActiveState.com'
> hint=recommended, useposix=true, d_sigaction=define
> usethreads=define use5005threads=undef useithreads=define
> usemultiplicity=de
> fine
> useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
> use64bitint=undef use64bitall=undef uselongdouble=undef
> usemymalloc=n, bincompat5005=undef
> Compiler:
> cc='gcc', ccflags ='-D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT
> -DUSE_SITECUSTOM
> IZE -DNO_HASH_SEED -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
> -fno-strict-aliasi
> ng -pipe',
> optimize='-O',
> cppflags='-D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT
> -DUSE_SITECUSTOMIZE -DNO_H
> ASH_SEED -fno-strict-aliasing -pipe'
> ccversion='', gccversion='3.4.2', gccosandvers=''
> intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321
> d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
> ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
> lseeksize
> =8
> alignbytes=8, prototype=define
> Linker and Libraries:
> ld='gcc', ldflags =' -L/usr/lib -L/usr/ccs/lib
> -L/opt/SUNWspro/WS6U2/lib '
> libpth=/usr/lib /usr/ccs/lib /opt/SUNWspro/WS6U2/lib /usr/local/lib
> libs=-lsocket -lnsl -ldl -lm -lpthread -lc
> perllibs=-lsocket -lnsl -ldl -lm -lpthread -lc
> libc=, so=so, useshrplib=false, libperl=libperl.a
> gnulibc_version=''
> Dynamic Linking:
> dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
> cccdlflags='-fPIC', lddlflags='-G -L/usr/lib -L/usr/ccs/lib
> -L/opt/SUNWspro/
> WS6U2/lib'
>
>
> Characteristics of this binary (from libperl):
> Compile-time options: MULTIPLICITY PERL_IMPLICIT_CONTEXT
> PERL_MALLOC_WRAP USE_ITHREADS USE_LARGE_FILES
> USE_PERLIO USE_REENTRANT_API USE_SITECUSTOMIZE
> Locally applied patches:
> ActivePerl Build 817 [257965]
> Iin_load_module moved for compatibility with build 806
> PerlEx support in CGI::Carp
> Less verbose ExtUtils::Install and Pod::Find
> Patch for CAN-2005-0448 from Debian with modifications
> Partly reverted 24733 to preserve binary compatibilty
> 27528 win32_pclose() error exit doesn't unlock mutex
> 27527 win32_async_check() can loop indefinitely
> 27515 ignore directories when searching @INC
> 27359 Fix -d:Foo=bar syntax
> 27210 Fix quote typo in c2ph
> 27203 Allow compiling swigged C++ code
> 27200 Make stat() on Windows handle trailing slashes correctly
> 27194 Get perl_fini() running on HP-UX again
> 27133 Initialise lastparen in the regexp structure
> 27034 Avoid "Prototype mismatch" warnings with autouse
> 26970 Make Passive mode the default for Net::FTP
> 26921 Avoid getprotobyname/number calls in IO::Socket::INET
> 26897,26903 Make common IPPROTO_* constants always available
> 26670 Make '-s' on the shebang line parse -foo=bar switches
> 26379 Fix alarm() for Windows 2003
> 26087 Storable 0.1 compatibility
> 25861 IO::File performace issue
> 25084 long groups entry could cause memory exhaustion
> 24699 ICMP_UNREACHABLE handling in Net::Ping
> Built under solaris
> Compiled at Mar 20 2006 20:36:33
> @INC:
> /opt/ActivePerl-5.8/lib/5.8.8/sun4-solaris-thread-multi
> /opt/ActivePerl-5.8/lib/5.8.8
> /opt/ActivePerl-5.8/lib/site_perl/5.8.8/sun4-solaris-thread- multi
> /opt/ActivePerl-5.8/lib/site_perl/5.8.8
> /opt/ActivePerl-5.8/lib/site_perl
> .
>
> MySQL Server is 5.0.21-standard
>
>
>
> --
> Chris Wagner
> CBTS
> GE Aircraft Engines
> Chris.Wagner@ae.ge.com
>
> --
> MySQL Perl Mailing List
> For list archives: http://lists.mysql.com/perl
> To unsubscribe: http://lists.mysql.com/perl?unsub=ruben@mrbrklyn.com

--
http://www.mrbrklyn.com - Interesting Stuff
http://www.nylxs.com - Leadership Development in Free Software

So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998

http://fairuse.nylxs.com DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002

"Yeah - I write Free Software...so SUE ME"

"The tremendous problem we face is that we are becoming sharecroppers to our own cultural heritage -- we need the ability to participate in our own society."

"> I'm an engineer. I choose the best tool for the job, politics be damned.<
You must be a stupid engineer then, because politcs and technology have been attacted at the hip since the 1st dynasty in Ancient Egypt. I guess you missed that one."

--
MySQL Perl Mailing List
For list archives: http://lists.mysql.com/perl
To unsubscribe: http://lists.mysql.com/perl?unsub=gcdmp-msql-mysql-modules@m .gmane.org