Vacuumdb error

Hi,

We have our production postgres 8.0.10 database running on linux x86_64
machine. Recently we have started getting an error from one of our
database while running vacuumdb. We are not getting this error during
backups just only during vacuuming of a database. Can anyone please help
us figure out why we are getting this error and how we can get rid of
it. Any help will be appreciated.=20

Here is the error:
tst_021 ERROR: invalid page header in block 8 of relation
"securityevent_pkey"


Thanks
Paramjeet Bhella
=20

--=20
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Vacuumdb error

"Bhella Paramjeet-PFCW67" writes:
> We have our production postgres 8.0.10 database running on linux x86_64
> machine. Recently we have started getting an error from one of our
> database while running vacuumdb. We are not getting this error during
> backups just only during vacuuming of a database. Can anyone please help
> us figure out why we are getting this error and how we can get rid of
> it. Any help will be appreciated.

> Here is the error:
> tst_021 ERROR: invalid page header in block 8 of relation
> "securityevent_pkey"

Well, since it's just an index, you should be able to fix it by
reindexing. But I'd worry a bit about what caused the corruption.
8.0.10 is not exactly current --- you should update to 8.0.latest.
And running some memory and disk diagnostics might not be wasted
effort.

regards, tom lane

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Vacuumdb error - corruption

=20
Hi Tom,

We recreated the index and that fixed it last week. We also had similar
failure few days back on a table but fortunately we had the luxury of
recreating it as the data was not so important, so we survived.
Yesterday we upgraded our postgres databases to 8.0.15 as suggested and
again we got error while vacuum analayze was run on the database and
this time it is on a different database and different table. I am not
able to access the table in question. Here is the error I get while
accessing the table. How do we recover this table from the error.

ectest=3D# select count(*) from securityevent;
ERROR: could not access status of transaction 33554431
DETAIL: could not open file "/pgdata/ec/data/pg_clog/001F": No such
file or directory

Error in the database vacuum log.
INFO: vacuuming "public.securityevent"
WARNING: relation "securityevent" TID 21/3: OID is invalid
vacuumdb: vacuuming of database "ectest" failed: ERROR: could not
access status of transaction 33554431
DETAIL: could not open file "/pgdata/ec/data/pg_clog/001F": No such
file or directory

Here are the machine specifics on which this database is running:=20
Platform: Linux x86_64
OS: Red Hat Enterprise Linux ES release 4 Kernel version: 2.6.9-34

According to our sysadmin there are no bad disks on the emc storage. So
what do you suggest we should do to narrow out the problem. Any help
will be highly appreciated. I have looked into the postgres archives and
people have reported this problem but there were no responses as to how
they resolved this issue. Do you still think it is due to block
corruption on disk? Please advice.

Thanks
Paramjeet Kaur=20

-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]=20
Sent: Thursday, April 17, 2008 5:18 PM
To: Bhella Paramjeet-PFCW67
Cc: pgsql-admin@postgresql.org; Subbiah Stalin-XCGF84
Subject: Re: [ADMIN] Vacuumdb error=20

"Bhella Paramjeet-PFCW67" writes:
> We have our production postgres 8.0.10 database running on linux=20
> x86_64 machine. Recently we have started getting an error from one of=20
> our database while running vacuumdb. We are not getting this error=20
> during backups just only during vacuuming of a database. Can anyone=20
> please help us figure out why we are getting this error and how we can

> get rid of it. Any help will be appreciated.

> Here is the error:
> tst_021 ERROR: invalid page header in block 8 of relation=20
> "securityevent_pkey"

Well, since it's just an index, you should be able to fix it by
reindexing. But I'd worry a bit about what caused the corruption.
8.0.10 is not exactly current --- you should update to 8.0.latest.
And running some memory and disk diagnostics might not be wasted effort.

regards, tom lane

--=20
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Vacuumdb error - corruption

"Bhella Paramjeet-PFCW67" writes:
> Error in the database vacuum log.
> INFO: vacuuming "public.securityevent"
> WARNING: relation "securityevent" TID 21/3: OID is invalid

That smells like a data corruption problem ...

> vacuumdb: vacuuming of database "ectest" failed: ERROR: could not
> access status of transaction 33554431

and so does that, particularly since the value equates to hex 01FFFFFF.
It's a lot easier to believe a hardware-ish fault stuffing such a value
than a software bug.

> Here are the machine specifics on which this database is running:
> Platform: Linux x86_64
> OS: Red Hat Enterprise Linux ES release 4 Kernel version: 2.6.9-34

If I'm reading the Red Hat CVS correctly, that kernel is two years old
next week. Perhaps a newer kernel would help your problems. A quick
troll through the changelog reveals a number of x86_64-specific fixes
that sound like they could have resulted in userspace data corruption.

regards, tom lane

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Vacuumdb error - corruption

> -----Original Message-----
> From: Bhella Paramjeet-PFCW67 [mailto:PBhella@Motorola.com]
> Sent: Thursday, April 24, 2008 1:13 PM
> To: Tom Lane
> Cc: pgsql-admin@postgresql.org; Subbiah Stalin-XCGF84
> Subject: Re: Vacuumdb error - corruption
>=20
>=20
> According to our sysadmin there are no bad disks on the emc storage.
So

EMC Storage? Is the Database sitting on NFS storage?


> what do you suggest we should do to narrow out the problem. Any help
> will be highly appreciated. I have looked into the postgres archives
and
> people have reported this problem but there were no responses as to
how
> they resolved this issue. Do you still think it is due to block
> corruption on disk? Please advice.
>=20
> Thanks
> Paramjeet Kaur
>=20
> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> Sent: Thursday, April 17, 2008 5:18 PM
> To: Bhella Paramjeet-PFCW67
> Cc: pgsql-admin@postgresql.org; Subbiah Stalin-XCGF84
> Subject: Re: [ADMIN] Vacuumdb error
>=20
> "Bhella Paramjeet-PFCW67" writes:
> > We have our production postgres 8.0.10 database running on linux
> > x86_64 machine. Recently we have started getting an error from one
of
> > our database while running vacuumdb. We are not getting this error
> > during backups just only during vacuuming of a database. Can anyone
> > please help us figure out why we are getting this error and how we
can
>=20
> > get rid of it. Any help will be appreciated.
>=20
> > Here is the error:
> > tst_021 ERROR: invalid page header in block 8 of relation
> > "securityevent_pkey"
>=20
> Well, since it's just an index, you should be able to fix it by
> reindexing. But I'd worry a bit about what caused the corruption.
> 8.0.10 is not exactly current --- you should update to 8.0.latest.
> And running some memory and disk diagnostics might not be wasted
effort.
>=20
> regards, tom lane

--=20
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Vacuumdb error - corruption

Hi Eric,

No database is not sitting on NFS storage. We are using emc storage and
the file system is fibre attached to storage.=20
In last few days we have had lots of errors on the database. Last night
while taking a pg_dump we got invalid memory alloc request error "ecdemo
2008-04-25 00:40:31 CDTFATAL: invalid memory alloc request size
936749196708242529" and also "invalid page header" errors.

Thanks
Paramjeet Kaur

-----Original Message-----
From: Eric Comeau [mailto:Eric.Comeau@signiant.com]=20
Sent: Friday, April 25, 2008 2:39 PM
To: Bhella Paramjeet-PFCW67; Tom Lane
Cc: pgsql-admin@postgresql.org; Subbiah Stalin-XCGF84
Subject: RE: Vacuumdb error - corruption



> -----Original Message-----
> From: Bhella Paramjeet-PFCW67 [mailto:PBhella@Motorola.com]
> Sent: Thursday, April 24, 2008 1:13 PM
> To: Tom Lane
> Cc: pgsql-admin@postgresql.org; Subbiah Stalin-XCGF84
> Subject: Re: Vacuumdb error - corruption
>=20
>=20
> According to our sysadmin there are no bad disks on the emc storage.
So

EMC Storage? Is the Database sitting on NFS storage?


> what do you suggest we should do to narrow out the problem. Any help=20
> will be highly appreciated. I have looked into the postgres archives
and
> people have reported this problem but there were no responses as to
how
> they resolved this issue. Do you still think it is due to block=20
> corruption on disk? Please advice.
>=20
> Thanks
> Paramjeet Kaur
>=20
> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> Sent: Thursday, April 17, 2008 5:18 PM
> To: Bhella Paramjeet-PFCW67
> Cc: pgsql-admin@postgresql.org; Subbiah Stalin-XCGF84
> Subject: Re: [ADMIN] Vacuumdb error
>=20
> "Bhella Paramjeet-PFCW67" writes:
> > We have our production postgres 8.0.10 database running on linux
> > x86_64 machine. Recently we have started getting an error from one
of
> > our database while running vacuumdb. We are not getting this error=20
> > during backups just only during vacuuming of a database. Can anyone=20
> > please help us figure out why we are getting this error and how we
can
>=20
> > get rid of it. Any help will be appreciated.
>=20
> > Here is the error:
> > tst_021 ERROR: invalid page header in block 8 of relation=20
> > "securityevent_pkey"
>=20
> Well, since it's just an index, you should be able to fix it by=20
> reindexing. But I'd worry a bit about what caused the corruption.
> 8.0.10 is not exactly current --- you should update to 8.0.latest.
> And running some memory and disk diagnostics might not be wasted
effort.
>=20
> regards, tom lane

--=20
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Vacuumdb error - corruption

On Fri, Apr 25, 2008 at 06:03:12PM -0400, Bhella Paramjeet-PFCW67 wrote:

> No database is not sitting on NFS storage. We are using emc storage and
> the file system is fibre attached to storage.

What's the filesystem? Are you sure you don't have any bad memory in
the box? I'm suspicious of the hardware first.

A

--
Andrew Sullivan
ajs@commandprompt.com
+1 503 667 4564 x104
http://www.commandprompt.com/

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin