linux standard layout

Dear list,

I have about 20 postgresql databases, about 3-4 GB in total.

We are moving them from Solaris/SPARC to a linux based virtual machine.

I don't like the VMWare environment, but it's not my choice, and assuming
the cpu load is ok, will there be any benefits if I put each database on
separate partitions, vs. simply using the one data directory?


Also, how is using standard rpm, with its standard layout (/var/lib/pgsql,
/usr/lib/pgsql, ...), generally regarded? ( vs. compiling everything ?)
Does anyone think using the rpm is unprofessional or something that only
beginners will do?

I have someone who opposes the use of standard rpms (even yums) for this
reason. I thought I'd check out how it is received professionally.

I ask the question because sometimes I feel uneasy mixing rpms and source
compilation.

If I compile something from the source, sometimes I see a boundary
condition - like, if I already have DBI from a standard rpm, it expects
postgresql library at a certain location - making me wonder whether I
should remove the DBI rpm and compile it also from the source, or whether
I should use standard rpms for postgresql as well. (DBI may not be a good
example.)

In general I didn't have any problems yet with standard rpms and I can
make the rpms work if there's a problem, but I may be missing something.




Any advice or reference to a relevant article on this issue will be
appreciated.



Thanks.

Ben Kim

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: linux standard layout

On Tue, Mar 9, 2010 at 1:18 PM, Jan-Ivar Mellingen
wrote:
> Regarding 'pulling the plug' on the servers: Physical or virtual, always use
> a UPS. You can pull the plug as much as you like. When the power is about to
> run out it signals the server, which shuts down cleanly.
> Our servers have dual powersupplies, connected to separate UPS'es on
> separate power sources...

I've watched three redundant UPSes, three redundant power
conditioners, and the switch for the diesel generator all fry when the
perfect storm of events happened in a job 7 or 8 years ago. Every
single machine in the hosting center lost power. Of the hundred or
so database servers, mine was the only one that came up. The others
all had to rely on off site backups to get up and running. Not one
other DBA at that company had performed a power failure test.

> In a nutshell, I am heartly recommending virtualization.

In a nutshell, you are relying on luck that both heavy iron machines
can't lose power at the same time. Sure, it's a low possibility, but
it's still a real one.

> And - I do not want to start a discussion about it. Just sharing my opinion.

Well, you can't throw the post you threw out there and not expect it
to start a discussion really. I understand a lot of the reasoning for
virtualization. My DB servers run at 75 to 100% capacity during
midday, there'd be no real advantage to buying an eve bigger piece of
iron to run them on.

I see the advantages of virtualization for certain load types, and
allowing to easily move services as a single disk image instead of
installing the service and configuring it on a new machine. Where I
work all the servers (except the nagios box) work hard and there'd be
no real advantage to me in putting all my eggs in the virtualization
basket there. I do use KVM to run multiple servers on my laptop for
testing. It's great for that. But hope is not a method I use when
installing my servers.

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: linux standard layout

And Jan-Ivar, please don't think I'm saying your way is not a valid
way to do things, it just requires things like read recoverable dbs
via PITR or something like that. Without a read to recover separate
machine for your big db you could be facing downtime measured in hours
or even days.

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: linux standard layout

On Tue, 2010-03-09 at 13:35 -0700, Scott Marlowe wrote:

> > In a nutshell, I am heartly recommending virtualization.
>
> In a nutshell, you are relying on luck that both heavy iron machines
> can't lose power at the same time. Sure, it's a low possibility, but
> it's still a real one.
>

Not luck. Percentage of risk.

Joshua D. Drake


--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.


--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: linux standard layout

On Tue, Mar 9, 2010 at 2:06 PM, Joshua D. Drake wrot=
e:
> On Tue, 2010-03-09 at 13:35 -0700, Scott Marlowe wrote:
>
>> > In a nutshell, I am heartly recommending virtualization.
>>
>> In a nutshell, you are relying on luck that both heavy iron machines
>> can't lose power at the same time. =A0Sure, it's a low possibility, but
>> it's still a real one.
>>
>
> Not luck. Percentage of risk.

They're both ways of saying you're rolling the dice. And in every
situation we're rolling the dice, it's just a question of how many and
how unlikely a particular outcome it. It's why we all have off-site
backups, and so on.

--=20
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: linux standard layout

On Tue, 2010-03-09 at 14:25 -0700, Scott Marlowe wrote:
> On Tue, Mar 9, 2010 at 2:06 PM, Joshua D. Drake wrote:
> > On Tue, 2010-03-09 at 13:35 -0700, Scott Marlowe wrote:
> >
> >> > In a nutshell, I am heartly recommending virtualization.
> >>
> >> In a nutshell, you are relying on luck that both heavy iron machines
> >> can't lose power at the same time. Sure, it's a low possibility, but
> >> it's still a real one.
> >>
> >
> > Not luck. Percentage of risk.
>
> They're both ways of saying you're rolling the dice. And in every
> situation we're rolling the dice, it's just a question of how many and

Well my point was all about risk versus reward. For many, a 3% risk is
more than appropriate. That isn't luck, it is a calculation of risk.


> how unlikely a particular outcome it. It's why we all have off-site
> backups, and so on.

Yes.

Joshua D. Drake


>


--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.


--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: linux standard layout

On Tue, Mar 09, 2010 at 01:28:20PM -0800, Joshua D. Drake wrote:
> On Tue, 2010-03-09 at 14:25 -0700, Scott Marlowe wrote:
> > On Tue, Mar 9, 2010 at 2:06 PM, Joshua D. Drake wrote:
> > > On Tue, 2010-03-09 at 13:35 -0700, Scott Marlowe wrote:
> > >
> > >> > In a nutshell, I am heartly recommending virtualization.
> > >>
> > >> In a nutshell, you are relying on luck that both heavy iron machines
> > >> can't lose power at the same time. Sure, it's a low possibility, but
> > >> it's still a real one.
> > >>
> > >
> > > Not luck. Percentage of risk.
> >
> > They're both ways of saying you're rolling the dice. And in every
> > situation we're rolling the dice, it's just a question of how many and
>
> Well my point was all about risk versus reward. For many, a 3% risk is
> more than appropriate. That isn't luck, it is a calculation of risk.
>
True, but in many cases the analysis of risk/reward is flawed by not
including the true cost of a protracted outage. Some of the second
order effects can be nasty if not included originally. I would also
recommend that the analysis and implementation be signed-off at the
highest levels -- that is where the head-hunting will start.

Cheers,
Ken

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: linux standard layout

On Tue, 2010-03-09 at 15:43 -0600, Kenneth Marshall wrote:
> On Tue, Mar 09, 2010 at 01:28:20PM -0800, Joshua D. Drake wrote:
> > On Tue, 2010-03-09 at 14:25 -0700, Scott Marlowe wrote:
> > > On Tue, Mar 9, 2010 at 2:06 PM, Joshua D. Drake wrote:
> > > > On Tue, 2010-03-09 at 13:35 -0700, Scott Marlowe wrote:
> > > >
> > > >> > In a nutshell, I am heartly recommending virtualization.
> > > >>
> > > >> In a nutshell, you are relying on luck that both heavy iron machines
> > > >> can't lose power at the same time. Sure, it's a low possibility, but
> > > >> it's still a real one.
> > > >>
> > > >
> > > > Not luck. Percentage of risk.
> > >
> > > They're both ways of saying you're rolling the dice. And in every
> > > situation we're rolling the dice, it's just a question of how many and
> >
> > Well my point was all about risk versus reward. For many, a 3% risk is
> > more than appropriate. That isn't luck, it is a calculation of risk.
> >
> True, but in many cases the analysis of risk/reward is flawed by not
> including the true cost of a protracted outage. Some of the second
> order effects can be nasty if not included originally. I would also
> recommend that the analysis and implementation be signed-off at the
> highest levels -- that is where the head-hunting will start.

I concur with that... Always have a CYA document.


Joshua D. Drake


>
> Cheers,
> Ken
>


--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.


--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: linux standard layout

* Ben Kim wrote:

Hi,

> I don't like the VMWare environment, but it's not my choice, and assuming
> the cpu load is ok, will there be any benefits if I put each database on
> separate partitions, vs. simply using the one data directory?

Depending on your underlying storage, it might be advisable to put
each instance on an separate spindle/enclosure, maybe even on an
different bus. But there's no universal answer to this.

You could start putting each instance on an separate partition or
lvm volume, record your real disk workload w/ blktrace and try it
out on different storage configurations w/ blkreplay.

> Also, how is using standard rpm, with its standard layout (/var/lib/pgsql,
> /usr/lib/pgsql, ...), generally regarded?

When working w/ some mainline distribution, always try to use that
distro's packages (not distro-independent binpkgs!), unless you've
got a valid reason for doing otherwise. When building your own
packages, use your distro's build machinery for that.

If your (virtual) box really only runs the server and you'd like to
do specific optimizations (eg. processor specific, etc), you could
create your own micro-distro. I've got some tools to ease that -
just mail me directly if you're interested.

> I have someone who opposes the use of standard rpms (even yums) for this
> reason. I thought I'd check out how it is received professionally.

Well, that depends on his specific reasons. For examples, several
distros tend to do strange things, you might not want in your
environment. Perhaps you need certain patches your distro doenst
provide. But even in this case you should try to use your distro's
build machinery for creating your own package.

Please ask your collegue for his concrete reasons and report back :)

> If I compile something from the source, sometimes I see a boundary
> condition - like, if I already have DBI from a standard rpm, it expects
> postgresql library at a certain location - making me wonder whether I
> should remove the DBI rpm and compile it also from the source, or whether
> I should use standard rpms for postgresql as well. (DBI may not be a good
> example.)

Pathes should be configurable, or you could use symlinks or bind mounts.


cu
--
------------------------------------------------------------ ---------
Enrico Weigelt == metux IT service - http://www.metux.de/
------------------------------------------------------------ ---------
Please visit the OpenSource QM Taskforce:
http://wiki.metux.de/public/OpenSource_QM_Taskforce
Patches / Fixes for a lot dozens of packages in dozens of versions:
http://patches.metux.de/
------------------------------------------------------------ ---------

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin