Fresh build on OS X not working (memory)

--Boundary_(ID_9z93N5jLjcYCKQYHyAEGxA)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT

Mac OS X 10.6.3
Built and installed PG 8.4.3 from source
Trying to start the server, I get "could not create shared memory segment: Invalid argument".

I've bumped my shmmax/shmall and related parameters as suggested in the README for the one-click installer (which I installed and then uninstalled before building from source), and then even higher as suggested by users on IRC. No luck.

This is for a dev laptop (2GB RAM), so I don't care about tuning performance. I just want pg to get up and running at all.

Any suggestions for what to try next are welcome. Relevant details follow:

Slim2:pgsql _postgres$ sysctl -a | grep shm
kern.sysv.shmmax: 2147483648
kern.sysv.shmmin: 1
kern.sysv.shmmni: 32
kern.sysv.shmseg: 8
kern.sysv.shmall: 2147483648
machdep.pmap.hashmax: 12
security.mac.posixshm_enforce: 1
security.mac.sysvshm_enforce: 1

Slim2:pgsql _postgres$ cat data/postgresql.conf | grep -E "max_conn|shared_buff"
max_connections = 10 # (change requires restart)
# Note: Increasing max_connections costs ~400 bytes of shared memory per
shared_buffers = 16MB # min 128kB
# max_locks_per_transaction * (max_connections + max_prepared_transactions)

Slim2:pgsql _postgres$ /usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data start
server starting
Slim2:pgsql _postgres$ FATAL: could not create shared memory segment: Invalid argument
DETAIL: Failed system call was shmget(key=5432001, size=18366464, 03600).
HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter. You can either reduce the request size or reconfigure the kernel with larger SHMMAX. To reduce the request size (currently 18366464 bytes), reduce PostgreSQL's shared_buffers parameter (currently 2048) and/or its max_connections parameter (currently 13).
If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for.
The PostgreSQL documentation contains more information about shared memory configuration.


--Boundary_(ID_9z93N5jLjcYCKQYHyAEGxA)
Content-type: text/html; charset=us-ascii
Content-transfer-encoding: quoted-printable

-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Mac =
OS X 10.6.3

margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: =

0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; font: =

normal normal normal 12px/normal Monaco, 'Courier New', 'DejaVu Sans =

Mono', 'Bitstream Vera Sans Mono', monospace; line-height: 1.4em; =

font-family: 'Bitstream Vera Sans Mono', Courier, monospace; ">

class=3D"line" id=3D"LC1" style=3D"margin-top: 0px; margin-right: 0px; =

margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: =

0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; =

">Slim2:pgsql _postgres$ sysctl -a | grep shm

id=3D"LC2" style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: =

0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; =

padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; =

">kern.sysv.shmmax: 2147483648

style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =

margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: =

0px; padding-left: 1em; line-height: 1.4em; ">kern.sysv.shmmin: =

1

margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: =

0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; =

line-height: 1.4em; ">kern.sysv.shmmni: 32

id=3D"LC5" style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: =

0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; =

padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; =

">kern.sysv.shmseg: 8

style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =

margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: =

0px; padding-left: 1em; line-height: 1.4em; ">kern.sysv.shmall: =

2147483648

margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: =

0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; =

line-height: 1.4em; ">machdep.pmap.hashmax: 12

id=3D"LC8" style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: =

0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; =

padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; =

">security.mac.posixshm_enforce: 1

style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =

margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: =

0px; padding-left: 1em; line-height: 1.4em; =

">security.mac.sysvshm_enforce: 1

style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =

margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: =

0px; padding-left: 1em; line-height: 1.4em; ">

0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =

padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: =

0px; line-height: 1.4em; ">

style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =

margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: =

0px; padding-left: 1em; line-height: 1.4em; ">Slim2:pgsql _postgres$ cat =

data/postgresql.conf | grep -E "max_conn|shared_buff"

class=3D"line" id=3D"LC12" style=3D"margin-top: 0px; margin-right: 0px; =

margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: =

0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; =

">max_connections =3D 10			# (change requires =

restart)

margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: =

0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; =

line-height: 1.4em; "># Note:  Increasing max_connections costs ~400 =

bytes of shared memory per 

style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =

margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: =

0px; padding-left: 1em; line-height: 1.4em; ">shared_buffers =3D 16MB		=

	# min 128kB

style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =

margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: =

0px; padding-left: 1em; line-height: 1.4em; "># =

max_locks_per_transaction * (max_connections + =

max_prepared_transactions)

style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =

margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: =

0px; padding-left: 1em; line-height: 1.4em; ">

0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =

padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: =

0px; line-height: 1.4em; ">

style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =

margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: =

0px; padding-left: 1em; line-height: 1.4em; ">Slim2:pgsql _postgres$ =

/usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data start

class=3D"line" id=3D"LC18" style=3D"margin-top: 0px; margin-right: 0px; =

margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: =

0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; =

">server starting

style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =

margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: =

0px; padding-left: 1em; line-height: 1.4em; ">Slim2:pgsql _postgres$ =

FATAL:  could not create shared memory segment: Invalid =

argument

margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: =

0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; =

line-height: 1.4em; ">DETAIL:  Failed system call was =

shmget(key=3D5432001, size=3D18366464, 03600).

id=3D"LC21" style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: =

0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; =

padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">HINT:  =

This error usually means that PostgreSQL's request for a shared memory =

segment exceeded your kernel's SHMMAX parameter.  You can either reduce =

the request size or reconfigure the kernel with larger SHMMAX.  To =

reduce the request size (currently 18366464 bytes), reduce PostgreSQL's =

shared_buffers parameter (currently 2048) and/or its max_connections =

parameter (currently 13).

style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =

margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: =

0px; padding-left: 1em; line-height: 1.4em; ">	If the request size is =

already small, it's possible that it is less than your kernel's SHMMIN =

parameter, in which case raising the request size or reconfiguring =

SHMMIN is called for.

style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =

margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: =

0px; padding-left: 1em; line-height: 1.4em; ">	The PostgreSQL =

documentation contains more information about shared memory =

configuration.

Re: Fresh build on OS X not working (memory)

Gavin Kistner writes:
> Mac OS X 10.6.3
> Built and installed PG 8.4.3 from source
> Trying to start the server, I get "could not create shared memory segment: Invalid argument".

OSX is *notoriously* unfriendly about configuring shm parameters.
You look at it sidewise, it doesn't work, and generally won't tell
you why it's refusing to talk to you. We document a few of the known
constraints in our manual, but I'm sure there are more.

> Slim2:pgsql _postgres$ sysctl -a | grep shm
> kern.sysv.shmmax: 2147483648
> kern.sysv.shmmin: 1
> kern.sysv.shmmni: 32
> kern.sysv.shmseg: 8
> kern.sysv.shmall: 2147483648

Hm. That shmall setting is not sane at all (remember the units are
pages). I wonder whether you've forced some internal overflow inside
the kernel. For comparison, I have these settings on my laptop:

pro:~ tgl$ cat /etc/sysctl.conf
kern.sysv.shmmax=33554432
kern.sysv.shmmin=1
kern.sysv.shmmni=32
kern.sysv.shmseg=8
kern.sysv.shmall=8192

and PG starts fine for me.

regards, tom lane

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Fresh build on OS X not working (memory)

On Apr 29, 2010, at 11:42 PM, Tom Lane wrote:
> For comparison, I have these settings on my laptop:
>
> pro:~ tgl$ cat /etc/sysctl.conf
> kern.sysv.shmmax=33554432
> kern.sysv.shmmin=1
> kern.sysv.shmmni=32
> kern.sysv.shmseg=8
> kern.sysv.shmall=8192
>
> and PG starts fine for me.

I'm not sure what magic you have in there, but this incantation copied verbatim worked for me. Thanks!

The first sysctl.conf I tried that didn't work from from the one-click installer's README. Do you see anything about the following settings that jumps out at you as bad? Is it notable that yours don't set the maxproc*?

kern.sysv.shmmax=1610612736
kern.sysv.shmall=393216
kern.sysv.shmmin=1
kern.sysv.shmmni=32
kern.sysv.shmseg=8
kern.maxprocperuid=512
kern.maxproc=2048


--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Fresh build on OS X not working (memory)

Gavin Kistner writes:
> On Apr 29, 2010, at 11:42 PM, Tom Lane wrote:
>> For comparison, I have these settings on my laptop:
>> pro:~ tgl$ cat /etc/sysctl.conf
>> kern.sysv.shmmax=33554432
>> kern.sysv.shmmin=1
>> kern.sysv.shmmni=32
>> kern.sysv.shmseg=8
>> kern.sysv.shmall=8192
>>
>> and PG starts fine for me.

> I'm not sure what magic you have in there, but this incantation copied verbatim worked for me. Thanks!

> The first sysctl.conf I tried that didn't work from from the one-click installer's README. Do you see anything about the following settings that jumps out at you as bad? Is it notable that yours don't set the maxproc*?

> kern.sysv.shmmax=1610612736
> kern.sysv.shmall=393216
> kern.sysv.shmmin=1
> kern.sysv.shmmni=32
> kern.sysv.shmseg=8
> kern.maxprocperuid=512
> kern.maxproc=2048

I have no idea why EDB recommend changing the maxproc settings, but
I doubt that's related to shared memory. I see that they have shmmax
equal to exactly 4096 times shmall, so that's good, but there must be
some other OSX peculiarity that this is tripping over. Maybe it's too
large ... do you actually have 1.6GB of RAM available that could
reasonably be dedicated to PG shared memory? Another possibility is
that maybe only exact powers of 2 work well. I'm just guessing though.

regards, tom lane

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Fresh build on OS X not working (memory)

I wrote:
> I have no idea why EDB recommend changing the maxproc settings, but
> I doubt that's related to shared memory. I see that they have shmmax
> equal to exactly 4096 times shmall, so that's good, but there must be
> some other OSX peculiarity that this is tripping over. Maybe it's too
> large ... do you actually have 1.6GB of RAM available that could
> reasonably be dedicated to PG shared memory? Another possibility is
> that maybe only exact powers of 2 work well. I'm just guessing though.

I poked around in the OS X kernel sources (let's hear it for open
source) and found that there really isn't a lot of special magic around
the shm control parameters anymore: as of OS X 10.6, they're all 64 bits
and the comparisons are entirely straightforward. If you can get the
values into the system, which you did since sysctl was reporting them,
they should work as-expected.

So why were you getting an EINVAL failure? The only theory that seems
to hold water after looking at the kernel source code is that there was a
pre-existing shm segment with the same key, and you got to this bit in
shmget_existing():

if (uap->size && uap->size > shmseg->u.shm_segsz)
return EINVAL;

if ((uap->shmflg & (IPC_CREAT | IPC_EXCL)) == (IPC_CREAT | IPC_EXCL))
return EEXIST;

IOW, if there's an existing shm segment of the same key but it's too
small, you get EINVAL. Not EEXIST, which is what our code in
InternalIpcMemoryCreate is expecting to see for a collision. So our
code fails to recognize the case as a collision and spits out a
misleading error message.

If memory serves, we've seen this issue before [ ... digs in archives
.... ] ah-hah:

http://archives.postgresql.org/pgsql-hackers/2008-10/msg0089 6.php

I argued in that message that returning EINVAL when EEXIST would apply
is a kernel bug, and I still think that. But what seems likely at this
point is that many BSD-derived kernels behave this way, and we're never
gonna get 'em all fixed. So it would behoove us to work around it.

Does the theory of a pre-existing smaller shmem segment make sense from
your end? In particular, had you previously had another Postgres server
running on that machine, and perhaps killed it ungracefully? If this
theory is correct, the issue was "fixed" as a result of rebooting (thus
making the old segment go away), not as a result of any changes of the
shm parameters. OTOH I'd have expected you to have to reboot multiple
times while experimenting with the shm parameters, so I'm not entirely
convinced I've hit on the right explanation.

regards, tom lane

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Fresh build on OS X not working (memory)

I wrote:
> ... OTOH I'd have expected you to have to reboot multiple
> times while experimenting with the shm parameters, so I'm not entirely
> convinced I've hit on the right explanation.

Oh ... were you using sysctl -w rather than rebooting to adjust the
shmmem parameters? I realized after experimenting that 10.6 lets you
do that. It didn't use to be possible to change these parameters in
OS X without rebooting --- I wonder when they fixed that.

regards, tom lane

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Fresh build on OS X not working (memory)

On Apr 30, 2010, at 3:35 PM, Tom Lane wrote:
> Does the theory of a pre-existing smaller shmem segment make sense from
> your end? In particular, had you previously had another Postgres server
> running on that machine, and perhaps killed it ungracefully? If this
> theory is correct, the issue was "fixed" as a result of rebooting (thus
> making the old segment go away), not as a result of any changes of the
> shm parameters. OTOH I'd have expected you to have to reboot multiple
> times while experimenting with the shm parameters, so I'm not entirely
> convinced I've hit on the right explanation.

It seems probably that you have hit the nail squarely on the head. I had installed from the one-click (which was working with the memory changes supplied in the README), and IIRC I tried to kill a few processes in preparation for a manual uninstall before I found the supplied uninstall app. I then compiled and installed from source. When that failed as described earlier, all changes I made to the settings were (as you guessed) through the sysctl -w option without a reboot. The first reboot I had was after I replaced my sysctl.conf wholesale with yours, and all was well.

Nice sleuthing. If I didn't have so many things going on at the moment I'd revert to the supplied settings and test to ensure it works and provide a confirmation. I'll try to remember to do that later.

Thanks again for your help and good information.

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Fresh build on OS X not working (memory)

Gavin Kistner writes:
> On Apr 30, 2010, at 3:35 PM, Tom Lane wrote:
>> Does the theory of a pre-existing smaller shmem segment make sense from
>> your end? In particular, had you previously had another Postgres server
>> running on that machine, and perhaps killed it ungracefully?

> It seems probably that you have hit the nail squarely on the head. I had installed from the one-click (which was working with the memory changes supplied in the README), and IIRC I tried to kill a few processes in preparation for a manual uninstall before I found the supplied uninstall app. I then compiled and installed from source. When that failed as described earlier, all changes I made to the settings were (as you guessed) through the sysctl -w option without a reboot. The first reboot I had was after I replaced my sysctl.conf wholesale with yours, and all was well.

Yeah, I was able to reproduce the problem locally by kill -9'ing the
postmaster and then enlarging max_connections in postgresql.conf before
trying to restart. In your case it might've been caused by the new
version needing a bit more shared memory rather than by any change in
parameter settings, but the basic issue seems pretty clear now.

I've applied a patch to our CVS to make PG behave more sanely in this
case. It does do the right thing if it recognizes that the shmget
failure is due to a pre-existing segment, but we weren't being
sufficiently careful about distinguishing that from a bad size request.

Thanks for the report!

regards, tom lane

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Fresh build on OS X not working (memory)

On Thu, Apr 29, 2010 at 11:42 PM, Tom Lane wrote:
> Gavin Kistner writes:
>> Mac OS X 10.6.3
>> Built and installed PG 8.4.3 from source
>> Trying to start the server, I get "could not create shared memory segmen=
t: Invalid argument".
>
> OSX is *notoriously* unfriendly about configuring shm parameters.
> You look at it sidewise, it doesn't work, and generally won't tell
> you why it's refusing to talk to you. =A0We document a few of the known
> constraints in our manual, but I'm sure there are more.

The fact that they seem to change the needed magical incantations
every point release doesn't help things either.

--=20
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Fresh build on OS X not working (memory)

Scott Marlowe writes:
> On Thu, Apr 29, 2010 at 11:42 PM, Tom Lane wrote:
>> OSX is *notoriously* unfriendly about configuring shm parameters.
>> You look at it sidewise, it doesn't work, and generally won't tell
>> you why it's refusing to talk to you.  We document a few of the known
>> constraints in our manual, but I'm sure there are more.

> The fact that they seem to change the needed magical incantations
> every point release doesn't help things either.

Actually, it appears they've finally reached a point of reasonable
sanity in 10.6. It's still true that they reject non-page-multiple
SHMMAX, instead of rounding to the next page boundary like everybody
else does; but the old kluges whereby you could only set the values by
waving a dead chicken in a specific fashion during boot seem to be
mostly gone. I wasn't aware of this myself, not having experimented
recently.

regards, tom lane

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin