MySQL db died / got corrupted / ???

mysql-5.0.77-4.el5_4.1 on RHEL 5.4

At some point, mysqld died. When i try to restart, mysqld.log says:

;InnoDB: End of
page dump
100104 16:31:14 InnoDB: Page checksum 1575996416, prior-to-4.0.14-form
checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
100104 16:31:14InnoDB: Error: trying to access a stray pointer
0x80002aaaabae3ff8
InnoDB: buf pool start is at 0x2aaaabad4000, end at 0x2aab4bad4000
InnoDB: Probable reason is database corruption or memory
InnoDB: corruption. If this happens in an InnoDB database recovery, see
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: how to force recovery.
100104 16:31:14InnoDB: Assertion failure in thread 47130669341296 in
file ./../include/buf0buf.ic line 268
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
100104 16:31:14 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this
binary
or one of the libraries it was linked against is corrupt, improperly
built,
or misconfigured. This error can also be caused by malfunctioning
hardware.
We will try our best to scrape up some info that will hopefully help
diagnose
the problem, but since we have already crashed, something is definitely
wrong
and this may fail.

key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_connections=100
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections
= 217599 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
frame pointer is NULL, did you compile with
-fomit-frame-pointer? Aborting backtrace!
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html
contains
information that should help you find out what is causing the crash.
100104 16:31:15 mysqld ended














Trying innodb_force_recovery = 4 did not help. Googling isn't really
helping me. I don't know what I can do about a "stray pointer".

--
************************************************************ ***********
* John Oliver http://www.john-oliver.net/ *
* *
************************************************************ ***********

--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/mysql?unsub=gcdmg-mysql-2@m.gmane.org

Re: MySQL db died / got corrupted / ???

I didn't notice that there was a **LOT** more junk dumped into the log:

100104 16:43:47 mysqld started
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
100104 16:43:49 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Page directory corruption: supremum not pointed to
100104 16:43:50 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 0<>

;InnoDB: End of
page dump
100104 16:43:51 InnoDB: Page checksum 1575996416, prior-to-4.0.14-form
checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page directory corruption: supremum not pointed to
100104 16:43:51 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 0<>

;InnoDB:
End of page dump
100104 16:43:51 InnoDB: Page checksum 1575996416, prior-to-4.0.14-form
checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
100104 16:43:51InnoDB: Error: trying to access a stray pointer
0x80002aaaabae3ff8
InnoDB: buf pool start is at 0x2aaaabad4000, end at 0x2aab4bad4000
InnoDB: Probable reason is database corruption or memory
InnoDB: corruption. If this happens in an InnoDB database recovery, see
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: how to force recovery.
100104 16:43:51InnoDB: Assertion failure in thread 47242840046192 in
file ./../include/buf0buf.ic line 268
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
100104 16:43:51 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this
binary
or one of the libraries it was linked against is corrupt, improperly
built,
or misconfigured. This error can also be caused by malfunctioning
hardware.
We will try our best to scrape up some info that will hopefully help
diagnose
the problem, but since we have already crashed, something is definitely
wrong
and this may fail.

key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_connections=100
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections
= 217599 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
frame pointer is NULL, did you compile with
-fomit-frame-pointer? Aborting backtrace!
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html
contains
information that should help you find out what is causing the crash.
100104 16:43:51 mysqld ended

--
************************************************************ ***********
* John Oliver http://www.john-oliver.net/ *
* *
************************************************************ ***********

--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/mysql?unsub=gcdmg-mysql-2@m.gmane.org

RE: MySQL db died / got corrupted / ???

Interesting. Usually innodb corruption is the result of hardware failure; i=
t looks from the log like you need to recover your database, and remove the=
existing data at the filesystem level, then re-import.

How to get data out of a corrupted innodb is here:
http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html

Make sure to make backups, even of your corrupted data. You might want to =
check your hardware, it could be faulty. Good luck!

Regards,
Gavin Towey

-----Original Message-----
From: John Oliver [mailto:joliver@john-oliver.net]
Sent: Monday, January 04, 2010 4:47 PM
To: mysql@lists.mysql.com
Subject: Re: MySQL db died / got corrupted / ???

I didn't notice that there was a **LOT** more junk dumped into the log:

100104 16:43:47 mysqld started
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
100104 16:43:49 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Page directory corruption: supremum not pointed to
100104 16:43:50 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 0<>

;InnoDB: End of
page dump
100104 16:43:51 InnoDB: Page checksum 1575996416, prior-to-4.0.14-form
checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >=3D MySQL-4.1.1 and stored already) 0
InnoDB: Page directory corruption: supremum not pointed to
100104 16:43:51 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 0<>

=
;InnoDB:
End of page dump
100104 16:43:51 InnoDB: Page checksum 1575996416, prior-to-4.0.14-form
checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >=3D MySQL-4.1.1 and stored already) 0
100104 16:43:51InnoDB: Error: trying to access a stray pointer
0x80002aaaabae3ff8
InnoDB: buf pool start is at 0x2aaaabad4000, end at 0x2aab4bad4000
InnoDB: Probable reason is database corruption or memory
InnoDB: corruption. If this happens in an InnoDB database recovery, see
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: how to force recovery.
100104 16:43:51InnoDB: Assertion failure in thread 47242840046192 in
file ./../include/buf0buf.ic line 268
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
100104 16:43:51 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this
binary
or one of the libraries it was linked against is corrupt, improperly
built,
or misconfigured. This error can also be caused by malfunctioning
hardware.
We will try our best to scrape up some info that will hopefully help
diagnose
the problem, but since we have already crashed, something is definitely
wrong
and this may fail.

key_buffer_size=3D0
read_buffer_size=3D131072
max_used_connections=3D0
max_connections=3D100
threads_connected=3D0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections
=3D 217599 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=3D(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
frame pointer is NULL, did you compile with
-fomit-frame-pointer? Aborting backtrace!
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html
contains
information that should help you find out what is causing the crash.
100104 16:43:51 mysqld ended

--
************************************************************ ***********
* John Oliver http://www.john-oliver.net/ *
* *
************************************************************ ***********

--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/mysql?unsub=3Dgtowey@ffn.com


This message contains confidential information and is intended only for the=
individual named. If you are not the named addressee, you are notified th=
at reviewing, disseminating, disclosing, copying or distributing this e-mai=
l is strictly prohibited. Please notify the sender immediately by e-mail i=
f you have received this e-mail by mistake and delete this e-mail from your=
system. E-mail transmission cannot be guaranteed to be secure or error-fre=
e as information could be intercepted, corrupted, lost, destroyed, arrive l=
ate or incomplete, or contain viruses. The sender therefore does not accept=
liability for any loss or damage caused by viruses or errors or omissions =
in the contents of this message, which arise as a result of e-mail transmis=
sion. [FriendFinder Networks, Inc., 220 Humbolt court, Sunnyvale, CA 94089,=
USA, FriendFinder.com

--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/mysql?unsub=3Dgcdmg-mysql-2@m.gmane.o rg