mysqld crash due to innoDB problems?
mysqld crash due to innoDB problems?
am 17.05.2004 17:11:58 von rusty.shanklin
DESCRIPTION:
Mysqld appears to crash every few days or so. The .err file shows
that it may be an innodb problem. (See below for detail)
How-To-Repeat
I am not able to repeat the problem. The query that runs when the
mysqld server dies, is automatically re-run when the server comes back
up. The query always appears to run successfully after the restart.
Fix:
The automatic restart of mysqld solves the problem, however the
server going down is now my problem.
Submitter-ID: RustyShanklin
Organization: Pikeville Medical Center
MySQL support: NONE
Synopsis: mysqld crash - Possible InnoDB corruption?
Severity: serious
Severity: medium
Category: mysql
Class: sw-bug
Release: mysql-4.0.18-standard (Official MySQL-standard binary)
C compiler: 2.95.3
C++ compiler: 2.95.3
Environment:
System: Linux mish02.pikevillehospital.org 2.4.18-14smp
#1 SMP Wed Sep 4 12:34:47 EDT 2002 i686 i686 i386 GNU/Linux
(4 processor system) 2GB Ram.
Architecture: i686
Some paths: /usr/bin/perl /usr/bin/make /usr/bin/gmake /usr/bin/gcc
/usr/bin/cc
GCC: Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/3.2/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/u
sr/share/info --enable-shared --enable-threads=posix --disable-checking
--host=i
386-redhat-linux --with-system-zlib --enable-__cxa_atexit
Thread model: posix
gcc version 3.2 20020903 (Red Hat Linux 8.0 3.2-7)
Compilation info: CC='gcc' CFLAGS='-O2 -mcpu=pentiumpro' CXX='gcc'
CXXFLAGS='
-O2 -mcpu=pentiumpro -felide-constructors' LDFLAGS='' ASFLAGS=''
LIBC:
lrwxrwxrwx 1 root root 14 Mar 4 2003 /lib/libc.so.6
-> libc-2
..2.93.so
-rwxr-xr-x 1 root root 1235468 Sep 5 2002
/lib/libc-2.2.93.so
-rw-r--r-- 1 root root 2233342 Mar 10 2003 /usr/lib/libc.a
-rw-r--r-- 1 root root 178 Mar 10 2003 /usr/lib/libc.so
Configure command: ./configure '--prefix=/usr/local/mysql'
'--localstatedir=/usr
/local/mysql/data' '--libexecdir=/usr/local/mysql/bin'
'--with-comment=Official
MySQL-standard binary' '--with-extra-charsets=complex'
'--with-server-suffix=-st
andard' '--enable-thread-safe-client' '--enable-local-infile'
'--enable-assemble
r' '--disable-shared' '--with-client-ldflags=-all-static'
'--with-mysqld-ldflags
=-all-static' '--with-embedded-server' '--with-innodb' 'CFLAGS=-O2
-mcpu=pentium
pro' 'CXXFLAGS=-O2 -mcpu=pentiumpro -felide-constructors' 'CXX=gcc'
----------mysql.err file:
InnoDB: Error: Mem area size is 0. Possibly a memory overrun of the
InnoDB: previous allocated area!
InnoDB: Apparent memory corruption: mem dump len 500; hex
4e43303b5350433a4c414
23b3133343a3a3a3a3a3a3a4e3a3a22223a3a3a31333439383b5354414d5 045523b44414
e49454c3
b483b3b3b4d443a3a22223a22223a3a3a3230303430353135323230343a3 b20203a3a4f7
26465726
5643a3a313b22223b22223b3230303430353136303230323b22223b333b3 b3b3a3a3a222
23a3a3a3
a3a3a0d4447313a39393a49393a3a3a3a574f524b494e473a3a3a3a3a3a3 a3a0d5a43413
a22223a2
2223a3b3b3b0d3506001e05353738303708535441525f574542133230303 42d30352d313
52032323
a30343a3133016e03413038fc09064d53483a3b7e5c263a0000000000000 00000000000a
90100000
0000000b010a808a0010000f63d775f9268a520773073656c2e6300a2080 000000000000
00000000
000000000000000000000009001000000000000000000009001000040000 00000000000f
00e41084
d53483a3b7e5c263a48424f433a413a534f46544c41423a413a323030343 035313530373
33830363
a3a4f524d3b4f30313b30313a373239303430343a503a322e313a3732393 03430343a3a0
d5049443
a3a3a3031303030353b3b3b413a3a5455524e45523b4b4154484c45454e3 a3a313933303
03631333
a463a3a3a3a3a3a3a3a3a3a303431333130333336373b3b3b413a3430302 d33382d38393
1380d4f5
2433a4e413a31353630; asc
NC0;SPC:LAB;134:::::::N::"":::13498;STAMPER;DANIEL;H;;;
MD::"":"":::200405152204:;
::Ordered::1;"";"";200405160202;"";3;;;:::""::::::.D
G1:99:I9::::WORKING::::::::.ZCA:"":"":;;;.5....57807.STAR_WE B.2004-05-15
22:04:1
3.n.A08...MSH:;~\&:.............................=w_.h.
w0sel.c..................
........................@.........A.MSH:;~\&:HBOC:A:SOFTLAB: A:20040515073
806::ORM
;O01;01:7290404:P:2.1:7290404::.PID:::010005;;;A::TURNER;KAT HLEEN::19300
613:F:::
:::::::0413103367;;;A:400-38-8918.ORC:NA:1560;
InnoDB: Scanning backward trying to find previous allocated mem blocks
Freed mem block at - 52160, file w0sel.c, line 2210
Freed mem block at - 53016, file w0sel.c, line 2210
Freed mem block at - 54584, file w0sel.c, line 2210
Freed mem block at - 117720, file w0sel.c, line 2210
Mem block at - 117912, file 0pcur.c, line 28
Mem block at - 118128, file w0upd.c, line 289
Mem block at - 118232, file m0rec.c, line 443
Freed mem block at - 124928, file w0sel.c, line 2210
Mem block at - 125216, file w0sel.c, line 2273
Mem block at - 126704, file w0sel.c, line 2210
InnoDB: Scanning forward trying to find next allocated mem blocks
Freed mem block at + 16, file w0sel.c, line 2210
Mem block at + 2448, file nodb.cc, line 1920
Freed mem block at + 2736, file w0sel.c, line 2210
Mem block at + 3832, file m0rec.c, line 443
Mem block at + 3936, file w0upd.c, line 289
Mem block at + 4152, file 0pcur.c, line 28
Freed mem block at + 4344, file w0sel.c, line 2210
Mem block at + 15560, file w0sel.c, line 2210
Mem block at + 17520, file w0sel.c, line 2210
Freed mem block at + 17912, file w0sel.c, line 2210
040517 9:32:12 InnoDB: Assertion failure in thread 110620 in file
mem0pool.c l
ine 493
InnoDB: Failing assertion: 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Send a detailed bug report to mysql@lists.mysql.com
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this
binary
or one of the libraries it was linked against is corrupt, improperly
built,
or misconfigured. This error can also be caused by malfunctioning
hardware.
We will try our best to scrape up some info that will hopefully help
diagnose
the problem, but since we have already crashed, something is definitely
wrong
and this may fail.
key_buffer_size=268435456
read_buffer_size=1044480
max_used_connections=70
max_connections=100
threads_connected=38
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections
= 466543
K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd=0x50e28b08
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbfc3e728, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x80705d0
0x8297878
0x822837c
0x82271d1
0x81322fd
0x8137047
0x80cbe07
0x80cbeb3
0x809dd75
0x809d109
0x809cdc0
0x8095402
0x8093876
0x807afd0
0x807e9ba
0x807a0c3
0x8079b1d
0x8079359
0x829502c
0x82be82a
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://www.mysql.com/doc/en/Using_stack_trace.html and
follow instru
ctions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x8a7b610 = select * from Inbound where status = 'n' order
by id
thd->thread_id=18105
thd->thread_id=18105
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
Number of processes running now: 0
040517 09:32:12 mysqld restarted
040517 9:32:13 InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 1 2535462740
InnoDB: Doing recovery: scanned up to log sequence number 1 2535485165
040517 9:32:13 InnoDB: Starting an apply batch of log records to the
database.
...
InnoDB: Progress in percents: 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
20 21 22
23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46
47 48 4
9 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72
73 74 75
76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 460531001, file name
../mish02-bin.046
040517 9:32:14 InnoDB: Flushing modified pages from the buffer pool...
040517 9:32:14 InnoDB: Started
/usr/local/mysql-4.0.18/bin/mysqld: ready for connections.
Version: '4.0.18-standard-log' socket: '/var/lib/mysql/mysql.sock'
port: 3306
Rusty W. Shanklin, Director-IT
Pikeville Medical Center
(606) 218 - 4940
Note: The information transmitted is intended only for the person or
entity to which it is
addressed and may contain confidential and/or privileged material. Any
review,
retransmission, dissemination or other use of, or taking any action in
reliance upon,
this information by persons or entities other than the intended
recipient is prohibited.
If you received this in error, please contact the sender and delete this
material from any
computer.
--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/mysql?unsub=gcdmg-mysql@m.gmane.org
Re: mysqld crash due to innoDB problems?
am 17.05.2004 22:19:59 von Heikki Tuuri
Rusty,
please send the WHOLE mysqld .err log to me. Please also send also the
output of
SHOW CREATE TABLE Inbound;
I am starting to suspect there might be an InnoDB memory corruption bug
associated with the blob_heap of the table handle. You are the second to
report a crash in the same place.
Though it is hard to see what might corrupt exactly 4, or maybe 16, bytes at
the start of an InnoDB additional mem pool block. It does not look like an
ordinary memory buffer overrun. Anyway, it is easy to add diagnostic code to
4.0.2x, so that we can study this in detail.
Best regards,
Heikki Tuuri
Innobase Oy
Foreign keys, transactions, and row level locking for MySQL
InnoDB Hot Backup - a hot backup tool for InnoDB which also backs up MyISAM
tables
http://www.innodb.com/order.php
Order MySQL technical support from https://order.mysql.com/
0x80705d0 handle_segfault + 420
0x8297878 pthread_sighandler + 184
0x822837c mem_area_free + 548
0x82271d1 mem_heap_block_free + 385
0x81322fd row_sel_store_mysql_rec + 65
0x8137047 row_search_for_mysql + 11491
0x80cbe07 general_fetch__11ha_innobasePcUiUi + 359
0x80cbeb3 index_next__11ha_innobasePc + 35
0x809dd75 join_read_next__FP14st_read_record + 33
0x809d109 sub_select__FP4JOINP13st_join_tableb + 337
0x809cdc0 do_select__FP4JOINPt4List1Z4ItemP8st_tableP9Procedure + 412
0x8095402
mysql_select__FP3THDP13st_table_listRt4List1Z4ItemP4ItemP8st _orderT4T3
T4UlP13select_result + 6994
0x8093876 handle_select__FP3THDP6st_lexP13select_result + 102
0x807afd0 mysql_execute_command__Fv + 1004
0x807e9ba mysql_parse__FP3THDPcUi + 146
0x807a0c3 dispatch_command__F19enum_server_commandP3THDPcUi + 1435
0x8079b1d do_command__FP3THD + 157
0x8079359 handle_one_connection + 641
0x829502c pthread_start_thread + 220
0x82be82a thread_start + 4
void
row_sel_store_mysql_rec(
/*====================*/
byte* mysql_rec, /* out: row in the MySQL format */
row_prebuilt_t* prebuilt, /* in: prebuilt struct */
rec_t* rec) /* in: Innobase record in the index
which was described in prebuilt's
template */
{
mysql_row_templ_t* templ;
mem_heap_t* extern_field_heap = NULL;
byte* data;
ulint len;
byte* blob_buf;
ulint i;
ut_ad(prebuilt->mysql_template);
if (prebuilt->blob_heap != NULL) {
mem_heap_free(prebuilt->blob_heap); <--- crash here
prebuilt->blob_heap = NULL;
}
..................
List:MySQL Bugs« Previous Message
From:Rusty W. ShanklinDate:May 17 2004 5:11pm
Subject:mysqld crash due to innoDB problems?
DESCRIPTION:
Mysqld appears to crash every few days or so. The .err file shows
that it may be an innodb problem. (See below for detail)
How-To-Repeat
I am not able to repeat the problem. The query that runs when the
mysqld server dies, is automatically re-run when the server comes back
up. The query always appears to run successfully after the restart.
Fix:
The automatic restart of mysqld solves the problem, however the
server going down is now my problem.
Submitter-ID: RustyShanklin
Organization: Pikeville Medical Center
MySQL support: NONE
Synopsis: mysqld crash - Possible InnoDB corruption?
Severity: serious
Severity: medium
Category: mysql
Class: sw-bug
Release: mysql-4.0.18-standard (Official MySQL-standard binary)
C compiler: 2.95.3
C++ compiler: 2.95.3
Environment:
System: Linux mish02.pikevillehospital.org 2.4.18-14smp
#1 SMP Wed Sep 4 12:34:47 EDT 2002 i686 i686 i386 GNU/Linux
(4 processor system) 2GB Ram.
Architecture: i686
Some paths: /usr/bin/perl /usr/bin/make /usr/bin/gmake /usr/bin/gcc
/usr/bin/cc
GCC: Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/3.2/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/u
sr/share/info --enable-shared --enable-threads=posix --disable-checking
--host=i
386-redhat-linux --with-system-zlib --enable-__cxa_atexit
Thread model: posix
gcc version 3.2 20020903 (Red Hat Linux 8.0 3.2-7)
Compilation info: CC='gcc' CFLAGS='-O2 -mcpu=pentiumpro' CXX='gcc'
CXXFLAGS='
-O2 -mcpu=pentiumpro -felide-constructors' LDFLAGS='' ASFLAGS=''
LIBC:
lrwxrwxrwx 1 root root 14 Mar 4 2003 /lib/libc.so.6
-> libc-2
..2.93.so
-rwxr-xr-x 1 root root 1235468 Sep 5 2002
/lib/libc-2.2.93.so
-rw-r--r-- 1 root root 2233342 Mar 10 2003 /usr/lib/libc.a
-rw-r--r-- 1 root root 178 Mar 10 2003 /usr/lib/libc.so
Configure command: ./configure '--prefix=/usr/local/mysql'
'--localstatedir=/usr
/local/mysql/data' '--libexecdir=/usr/local/mysql/bin'
'--with-comment=Official
MySQL-standard binary' '--with-extra-charsets=complex'
'--with-server-suffix=-st
andard' '--enable-thread-safe-client' '--enable-local-infile'
'--enable-assemble
r' '--disable-shared' '--with-client-ldflags=-all-static'
'--with-mysqld-ldflags
=-all-static' '--with-embedded-server' '--with-innodb' 'CFLAGS=-O2
-mcpu=pentium
pro' 'CXXFLAGS=-O2 -mcpu=pentiumpro -felide-constructors' 'CXX=gcc'
----------mysql.err file:
InnoDB: Error: Mem area size is 0. Possibly a memory overrun of the
InnoDB: previous allocated area!
InnoDB: Apparent memory corruption: mem dump len 500; hex
4e43303b5350433a4c414
23b3133343a3a3a3a3a3a3a4e3a3a22223a3a3a31333439383b5354414d5 045523b44414
e49454c3
b483b3b3b4d443a3a22223a22223a3a3a3230303430353135323230343a3 b20203a3a4f7
26465726
5643a3a313b22223b22223b3230303430353136303230323b22223b333b3 b3b3a3a3a222
23a3a3a3
a3a3a0d4447313a39393a49393a3a3a3a574f524b494e473a3a3a3a3a3a3 a3a0d5a43413
a22223a2
2223a3b3b3b0d3506001e05353738303708535441525f574542133230303 42d30352d313
52032323
a30343a3133016e03413038fc09064d53483a3b7e5c263a0000000000000 00000000000a
90100000
0000000b010a808a0010000f63d775f9268a520773073656c2e6300a2080 000000000000
00000000
000000000000000000000009001000000000000000000009001000040000 00000000000f
00e41084
d53483a3b7e5c263a48424f433a413a534f46544c41423a413a323030343 035313530373
33830363
a3a4f524d3b4f30313b30313a373239303430343a503a322e313a3732393 03430343a3a0
d5049443
a3a3a3031303030353b3b3b413a3a5455524e45523b4b4154484c45454e3 a3a313933303
03631333
a463a3a3a3a3a3a3a3a3a3a303431333130333336373b3b3b413a3430302 d33382d38393
1380d4f5
2433a4e413a31353630; asc
NC0;SPC:LAB;134:::::::N::"":::13498;STAMPER;DANIEL;H;;;
MD::"":"":::200405152204:;
::Ordered::1;"";"";200405160202;"";3;;;:::""::::::.D
G1:99:I9::::WORKING::::::::.ZCA:"":"":;;;.5....57807.STAR_WE B.2004-05-15
22:04:1
3.n.A08...MSH:;~\&:.............................=w_.h.
w0sel.c..................
........................@.........A.MSH:;~\&:HBOC:A:SOFTLAB: A:20040515073
806::ORM
;O01;01:7290404:P:2.1:7290404::.PID:::010005;;;A::TURNER;KAT HLEEN::19300
613:F:::
:::::::0413103367;;;A:400-38-8918.ORC:NA:1560;
InnoDB: Scanning backward trying to find previous allocated mem blocks
Freed mem block at - 52160, file w0sel.c, line 2210
Freed mem block at - 53016, file w0sel.c, line 2210
Freed mem block at - 54584, file w0sel.c, line 2210
Freed mem block at - 117720, file w0sel.c, line 2210
Mem block at - 117912, file 0pcur.c, line 28
Mem block at - 118128, file w0upd.c, line 289
Mem block at - 118232, file m0rec.c, line 443
Freed mem block at - 124928, file w0sel.c, line 2210
Mem block at - 125216, file w0sel.c, line 2273
Mem block at - 126704, file w0sel.c, line 2210
InnoDB: Scanning forward trying to find next allocated mem blocks
Freed mem block at + 16, file w0sel.c, line 2210
Mem block at + 2448, file nodb.cc, line 1920
Freed mem block at + 2736, file w0sel.c, line 2210
Mem block at + 3832, file m0rec.c, line 443
Mem block at + 3936, file w0upd.c, line 289
Mem block at + 4152, file 0pcur.c, line 28
Freed mem block at + 4344, file w0sel.c, line 2210
Mem block at + 15560, file w0sel.c, line 2210
Mem block at + 17520, file w0sel.c, line 2210
Freed mem block at + 17912, file w0sel.c, line 2210
040517 9:32:12 InnoDB: Assertion failure in thread 110620 in file
mem0pool.c l
ine 493
InnoDB: Failing assertion: 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Send a detailed bug report to mysql@lists.mysql.com
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this
binary
or one of the libraries it was linked against is corrupt, improperly
built,
or misconfigured. This error can also be caused by malfunctioning
hardware.
We will try our best to scrape up some info that will hopefully help
diagnose
the problem, but since we have already crashed, something is definitely
wrong
and this may fail.
key_buffer_size=268435456
read_buffer_size=1044480
max_used_connections=70
max_connections=100
threads_connected=38
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections
= 466543
K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd=0x50e28b08
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbfc3e728, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x80705d0
0x8297878
0x822837c
0x82271d1
0x81322fd
0x8137047
0x80cbe07
0x80cbeb3
0x809dd75
0x809d109
0x809cdc0
0x8095402
0x8093876
0x807afd0
0x807e9ba
0x807a0c3
0x8079b1d
0x8079359
0x829502c
0x82be82a
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://www.mysql.com/doc/en/Using_stack_trace.html
and
follow instru
ctions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x8a7b610 = select * from Inbound where status = 'n' order
by id
thd->thread_id=18105
thd->thread_id=18105
The manual page at http://www.mysql.com/doc/en/Crashing.html
contains
information that should help you find out what is causing the crash.
Number of processes running now: 0
040517 09:32:12 mysqld restarted
040517 9:32:13 InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 1 2535462740
InnoDB: Doing recovery: scanned up to log sequence number 1 2535485165
040517 9:32:13 InnoDB: Starting an apply batch of log records to the
database.
...
InnoDB: Progress in percents: 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
20 21 22
23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46
47 48 4
9 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72
73 74 75
76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 460531001, file name
../mish02-bin.046
040517 9:32:14 InnoDB: Flushing modified pages from the buffer pool...
040517 9:32:14 InnoDB: Started
/usr/local/mysql-4.0.18/bin/mysqld: ready for connections.
Version: '4.0.18-standard-log' socket: '/var/lib/mysql/mysql.sock'
port: 3306
Rusty W. Shanklin, Director-IT
Pikeville Medical Center
(606) 218 - 4940
--
MySQL Bugs Mailing List
For list archives: http://lists.mysql.com/bugs
To unsubscribe: http://lists.mysql.com/bugs?unsub=gcdmb-bugs@m.gmane.org
Re: mysqld crash due to innoDB problems?
am 18.05.2004 01:31:04 von Heikki Tuuri
Rusty,
I found and fixed now the following bug:
" mem0pool.c:
Fix a memory corruption bug: in 32-bit computers, every 4 billionth
malloc outside
innodb_additional_mem_pool_size was mistreated when freeing the memory; this
could
corrupt the InnoDB additional mem pool and could have caused crashes
anywhere, also
inside MySQL, or even database corruption! the bug exists also in 3.23 and
4.1;
workaround: configure innodb_additional_mem_pool_size big enough
"
How big had you set innodb_additional_mem_pool_size?
You can use
SHOW INNODB STATUS\G
to look at the memory usage in the additional mem pool. Note that the real
usage is maybe 2 - 4 times bigger than what it reports. If you are using
BLOBs, that will temporarily increase memory usage, and can explain why the
crashes were associated with the blob_heap of a table handle.
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 84966343; in additional pool allocated 1402624
Regards,
Heikki
----- Original Message -----
From: "Heikki Tuuri"
To: ;
Sent: Monday, May 17, 2004 11:19 PM
Subject: Re: mysqld crash due to innoDB problems?
> Rusty,
>
> please send the WHOLE mysqld .err log to me. Please also send also the
> output of
>
> SHOW CREATE TABLE Inbound;
>
> I am starting to suspect there might be an InnoDB memory corruption bug
> associated with the blob_heap of the table handle. You are the second to
> report a crash in the same place.
>
> Though it is hard to see what might corrupt exactly 4, or maybe 16, bytes
at
> the start of an InnoDB additional mem pool block. It does not look like an
> ordinary memory buffer overrun. Anyway, it is easy to add diagnostic code
to
> 4.0.2x, so that we can study this in detail.
>
> Best regards,
>
> Heikki Tuuri
> Innobase Oy
> Foreign keys, transactions, and row level locking for MySQL
> InnoDB Hot Backup - a hot backup tool for InnoDB which also backs up
MyISAM
> tables
> http://www.innodb.com/order.php
>
> Order MySQL technical support from https://order.mysql.com/
>
> 0x80705d0 handle_segfault + 420
> 0x8297878 pthread_sighandler + 184
> 0x822837c mem_area_free + 548
> 0x82271d1 mem_heap_block_free + 385
> 0x81322fd row_sel_store_mysql_rec + 65
> 0x8137047 row_search_for_mysql + 11491
> 0x80cbe07 general_fetch__11ha_innobasePcUiUi + 359
> 0x80cbeb3 index_next__11ha_innobasePc + 35
> 0x809dd75 join_read_next__FP14st_read_record + 33
> 0x809d109 sub_select__FP4JOINP13st_join_tableb + 337
> 0x809cdc0 do_select__FP4JOINPt4List1Z4ItemP8st_tableP9Procedure + 412
> 0x8095402
> mysql_select__FP3THDP13st_table_listRt4List1Z4ItemP4ItemP8st _orderT4T3
> T4UlP13select_result + 6994
> 0x8093876 handle_select__FP3THDP6st_lexP13select_result + 102
> 0x807afd0 mysql_execute_command__Fv + 1004
> 0x807e9ba mysql_parse__FP3THDPcUi + 146
> 0x807a0c3 dispatch_command__F19enum_server_commandP3THDPcUi + 1435
> 0x8079b1d do_command__FP3THD + 157
> 0x8079359 handle_one_connection + 641
> 0x829502c pthread_start_thread + 220
> 0x82be82a thread_start + 4
>
> void
> row_sel_store_mysql_rec(
> /*====================*/
> byte* mysql_rec, /* out: row in the MySQL format */
> row_prebuilt_t* prebuilt, /* in: prebuilt struct */
> rec_t* rec) /* in: Innobase record in the
index
> which was described in prebuilt's
> template */
> {
> mysql_row_templ_t* templ;
> mem_heap_t* extern_field_heap = NULL;
> byte* data;
> ulint len;
> byte* blob_buf;
> ulint i;
>
> ut_ad(prebuilt->mysql_template);
>
> if (prebuilt->blob_heap != NULL) {
> mem_heap_free(prebuilt->blob_heap); <--- crash here
> prebuilt->blob_heap = NULL;
> }
>
> .................
>
> List:MySQL Bugs« Previous Message
> From:Rusty W. ShanklinDate:May 17 2004 5:11pm
> Subject:mysqld crash due to innoDB problems?
>
>
>
> DESCRIPTION:
> Mysqld appears to crash every few days or so. The .err file shows
> that it may be an innodb problem. (See below for detail)
>
> How-To-Repeat
> I am not able to repeat the problem. The query that runs when the
> mysqld server dies, is automatically re-run when the server comes back
> up. The query always appears to run successfully after the restart.
>
> Fix:
> The automatic restart of mysqld solves the problem, however the
> server going down is now my problem.
>
> Submitter-ID: RustyShanklin
> Organization: Pikeville Medical Center
> MySQL support: NONE
> Synopsis: mysqld crash - Possible InnoDB corruption?
> Severity: serious
> Severity: medium
> Category: mysql
> Class: sw-bug
> Release: mysql-4.0.18-standard (Official MySQL-standard binary)
> C compiler: 2.95.3
> C++ compiler: 2.95.3
> Environment:
> System: Linux mish02.pikevillehospital.org 2.4.18-14smp
> #1 SMP Wed Sep 4 12:34:47 EDT 2002 i686 i686 i386 GNU/Linux
> (4 processor system) 2GB Ram.
> Architecture: i686
> Some paths: /usr/bin/perl /usr/bin/make /usr/bin/gmake /usr/bin/gcc
> /usr/bin/cc
> GCC: Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/3.2/specs
> Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
> --infodir=/u
> sr/share/info --enable-shared --enable-threads=posix --disable-checking
> --host=i
> 386-redhat-linux --with-system-zlib --enable-__cxa_atexit
> Thread model: posix
> gcc version 3.2 20020903 (Red Hat Linux 8.0 3.2-7)
> Compilation info: CC='gcc' CFLAGS='-O2 -mcpu=pentiumpro' CXX='gcc'
> CXXFLAGS='
> -O2 -mcpu=pentiumpro -felide-constructors' LDFLAGS='' ASFLAGS=''
> LIBC:
> lrwxrwxrwx 1 root root 14 Mar 4 2003 /lib/libc.so.6
> -> libc-2
> .2.93.so
> -rwxr-xr-x 1 root root 1235468 Sep 5 2002
> /lib/libc-2.2.93.so
> -rw-r--r-- 1 root root 2233342 Mar 10 2003 /usr/lib/libc.a
> -rw-r--r-- 1 root root 178 Mar 10 2003 /usr/lib/libc.so
> Configure command: ./configure '--prefix=/usr/local/mysql'
> '--localstatedir=/usr
> /local/mysql/data' '--libexecdir=/usr/local/mysql/bin'
> '--with-comment=Official
> MySQL-standard binary' '--with-extra-charsets=complex'
> '--with-server-suffix=-st
> andard' '--enable-thread-safe-client' '--enable-local-infile'
> '--enable-assemble
> r' '--disable-shared' '--with-client-ldflags=-all-static'
> '--with-mysqld-ldflags
> =-all-static' '--with-embedded-server' '--with-innodb' 'CFLAGS=-O2
> -mcpu=pentium
> pro' 'CXXFLAGS=-O2 -mcpu=pentiumpro -felide-constructors' 'CXX=gcc'
>
> ----------mysql.err file:
> InnoDB: Error: Mem area size is 0. Possibly a memory overrun of the
> InnoDB: previous allocated area!
> InnoDB: Apparent memory corruption: mem dump len 500; hex
> 4e43303b5350433a4c414
> 23b3133343a3a3a3a3a3a3a4e3a3a22223a3a3a31333439383b5354414d5 045523b44414
> e49454c3
> b483b3b3b4d443a3a22223a22223a3a3a3230303430353135323230343a3 b20203a3a4f7
> 26465726
> 5643a3a313b22223b22223b3230303430353136303230323b22223b333b3 b3b3a3a3a222
> 23a3a3a3
> a3a3a0d4447313a39393a49393a3a3a3a574f524b494e473a3a3a3a3a3a3 a3a0d5a43413
> a22223a2
> 2223a3b3b3b0d3506001e05353738303708535441525f574542133230303 42d30352d313
> 52032323
> a30343a3133016e03413038fc09064d53483a3b7e5c263a0000000000000 00000000000a
> 90100000
> 0000000b010a808a0010000f63d775f9268a520773073656c2e6300a2080 000000000000
> 00000000
> 000000000000000000000009001000000000000000000009001000040000 00000000000f
> 00e41084
> d53483a3b7e5c263a48424f433a413a534f46544c41423a413a323030343 035313530373
> 33830363
> a3a4f524d3b4f30313b30313a373239303430343a503a322e313a3732393 03430343a3a0
> d5049443
> a3a3a3031303030353b3b3b413a3a5455524e45523b4b4154484c45454e3 a3a313933303
> 03631333
> a463a3a3a3a3a3a3a3a3a3a303431333130333336373b3b3b413a3430302 d33382d38393
> 1380d4f5
> 2433a4e413a31353630; asc
> NC0;SPC:LAB;134:::::::N::"":::13498;STAMPER;DANIEL;H;;;
> MD::"":"":::200405152204:;
> ::Ordered::1;"";"";200405160202;"";3;;;:::""::::::.D
> G1:99:I9::::WORKING::::::::.ZCA:"":"":;;;.5....57807.STAR_WE B.2004-05-15
> 22:04:1
> 3.n.A08...MSH:;~\&:.............................=w_.h.
> w0sel.c..................
> .......................@.........A.MSH:;~\&:HBOC:A:SOFTLAB:A :20040515073
> 806::ORM
> ;O01;01:7290404:P:2.1:7290404::.PID:::010005;;;A::TURNER;KAT HLEEN::19300
> 613:F:::
> :::::::0413103367;;;A:400-38-8918.ORC:NA:1560;
> InnoDB: Scanning backward trying to find previous allocated mem blocks
> Freed mem block at - 52160, file w0sel.c, line 2210
> Freed mem block at - 53016, file w0sel.c, line 2210
> Freed mem block at - 54584, file w0sel.c, line 2210
> Freed mem block at - 117720, file w0sel.c, line 2210
> Mem block at - 117912, file 0pcur.c, line 28
> Mem block at - 118128, file w0upd.c, line 289
> Mem block at - 118232, file m0rec.c, line 443
> Freed mem block at - 124928, file w0sel.c, line 2210
> Mem block at - 125216, file w0sel.c, line 2273
> Mem block at - 126704, file w0sel.c, line 2210
> InnoDB: Scanning forward trying to find next allocated mem blocks
> Freed mem block at + 16, file w0sel.c, line 2210
> Mem block at + 2448, file nodb.cc, line 1920
> Freed mem block at + 2736, file w0sel.c, line 2210
> Mem block at + 3832, file m0rec.c, line 443
> Mem block at + 3936, file w0upd.c, line 289
> Mem block at + 4152, file 0pcur.c, line 28
> Freed mem block at + 4344, file w0sel.c, line 2210
> Mem block at + 15560, file w0sel.c, line 2210
> Mem block at + 17520, file w0sel.c, line 2210
> Freed mem block at + 17912, file w0sel.c, line 2210
> 040517 9:32:12 InnoDB: Assertion failure in thread 110620 in file
> mem0pool.c l
> ine 493
> InnoDB: Failing assertion: 0
> InnoDB: We intentionally generate a memory trap.
> InnoDB: Send a detailed bug report to mysql@lists.mysql.com
> mysqld got signal 11;
> This could be because you hit a bug. It is also possible that this
> binary
> or one of the libraries it was linked against is corrupt, improperly
> built,
> or misconfigured. This error can also be caused by malfunctioning
> hardware.
> We will try our best to scrape up some info that will hopefully help
> diagnose
> the problem, but since we have already crashed, something is definitely
> wrong
> and this may fail.
> key_buffer_size=268435456
> read_buffer_size=1044480
> max_used_connections=70
> max_connections=100
> threads_connected=38
> It is possible that mysqld could use up to
> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections
> = 466543
> K
> bytes of memory
> Hope that's ok; if not, decrease some variables in the equation.
>
> thd=0x50e28b08
> Attempting backtrace. You can use the following information to find out
> where mysqld died. If you see no messages after this, something went
> terribly wrong...
> Cannot determine thread, fp=0xbfc3e728, backtrace may not be correct.
> Stack range sanity check OK, backtrace follows:
> 0x80705d0
> 0x8297878
> 0x822837c
> 0x82271d1
> 0x81322fd
> 0x8137047
> 0x80cbe07
> 0x80cbeb3
> 0x809dd75
> 0x809d109
> 0x809cdc0
> 0x8095402
> 0x8093876
> 0x807afd0
> 0x807e9ba
> 0x807a0c3
> 0x8079b1d
> 0x8079359
> 0x829502c
> 0x82be82a
> New value of fp=(nil) failed sanity check, terminating stack trace!
> Please read http://www.mysql.com/doc/en/Using_stack_trace.html
> and
> follow instru
> ctions on how to resolve the stack trace. Resolved
> stack trace is much more helpful in diagnosing the problem, so please do
> resolve it
> Trying to get some variables.
> Some pointers may be invalid and cause the dump to abort...
> thd->query at 0x8a7b610 = select * from Inbound where status = 'n' order
> by id
> thd->thread_id=18105
> thd->thread_id=18105
> The manual page at http://www.mysql.com/doc/en/Crashing.html
> contains
> information that should help you find out what is causing the crash.
>
> Number of processes running now: 0
> 040517 09:32:12 mysqld restarted
> 040517 9:32:13 InnoDB: Database was not shut down normally.
> InnoDB: Starting recovery from log files...
> InnoDB: Starting log scan based on checkpoint at
> InnoDB: log sequence number 1 2535462740
> InnoDB: Doing recovery: scanned up to log sequence number 1 2535485165
> 040517 9:32:13 InnoDB: Starting an apply batch of log records to the
> database.
> ..
> InnoDB: Progress in percents: 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
> 20 21 22
> 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46
> 47 48 4
> 9 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72
> 73 74 75
> 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
> InnoDB: Apply batch completed
> InnoDB: Last MySQL binlog file position 0 460531001, file name
> ./mish02-bin.046
> 040517 9:32:14 InnoDB: Flushing modified pages from the buffer pool...
> 040517 9:32:14 InnoDB: Started
> /usr/local/mysql-4.0.18/bin/mysqld: ready for connections.
> Version: '4.0.18-standard-log' socket: '/var/lib/mysql/mysql.sock'
> port: 3306
>
> Rusty W. Shanklin, Director-IT
> Pikeville Medical Center
> (606) 218 - 4940
>
--
MySQL Bugs Mailing List
For list archives: http://lists.mysql.com/bugs
To unsubscribe: http://lists.mysql.com/bugs?unsub=gcdmb-bugs@m.gmane.org
RE: mysqld crash due to innoDB problems?
am 18.05.2004 15:33:38 von Heikki Tuuri
Hi!
Analysis shows that this is probably the bug that I fixed last night. It is
present in all 3.23, 4.0 and 4.1 versions.
"
ChangeSet@1.1836, 2004-05-18 01:53:06+03:00, heikki@hundin.mysql.fi
mem0pool.c:
Fix a memory corruption bug: in 32-bit computers, every 4 billionth
malloc
outside innodb_additional_mem_pool_size was mistreated when freeing the
memory;
this could corrupt the InnoDB additional mem pool and could have caused
crashes
anywhere, also inside MySQL, or even database corruption! the bug exists
also in
3.23 and 4.1; workaround: configure innodb_additional_mem_pool_size big
enough
"
Best regards,
Heikki
Innobase Oy
InnoDB - transactions, row level locking, and foreign keys for MySQL
InnoDB Hot Backup - a hot backup tool for InnoDB which also backs up MyISAM
tables
http://www.innodb.com/order.php
--
MySQL Bugs Mailing List
For list archives: http://lists.mysql.com/bugs
To unsubscribe: http://lists.mysql.com/bugs?unsub=gcdmb-bugs@m.gmane.org