Ubuntu 10.04 - Cannot Create TCP/IP Sockets

Good evening.

I've recently provisioned a new Ubuntu 10.04 server. I ran "apt-get
install postgresql-8.4", just like I normally do when I'm setting up a
new server. However, this time I got an error about an inability to
create TCP/IP sockets. The full error is below.

* Starting PostgreSQL 8.4 database server
* The PostgreSQL server failed to start. Please check the log output:
2010-05-03 03:36:06 UTC LOG: could not bind IPv4 socket: Cannot assign
requested address
2010-05-03 03:36:06 UTC HINT: Is another postmaster already running on
port 5432? If not, wait a few seconds and retry.
2010-05-03 03:36:06 UTC WARNING: could not create listen socket for
"localhost"
2010-05-03 03:36:06 UTC FATAL: could not create any TCP/IP sockets

However, there is nothing on port 5432...

root@coolnewserver:~# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:ssh *:* LISTEN
tcp 0 0 localhost:smtp *:* LISTEN
tcp6 0 0 [::]:ssh [::]:* LISTEN

....and no postgresql processes...

root@coolnewserver:~# ps -ef | grep postgresql
root 7738 3403 0 03:37 pts/0 00:00:00 grep --color=auto postgresql
root@coolnewserver:~# ps -ef | grep postmaster
root 7740 3403 0 03:37 pts/0 00:00:00 grep --color=auto postmaster

Possibly pertinent lines from /etc/postgresql/8.4/main/postgresql.conf...

listen_addresses = 'localhost'
port = 5432
max_connections = 100
unix_socket_directory = '/var/run/postgresql'

....permissions on that last directory are...

drwxrwsr-x 2 postgres postgres 40 2010-05-03 03:10 postgresql

....and localhost is resolving...

root@coolnewserver:~# nslookup localhost
Server: [IP]
Address: [IP]#53

Name: localhost
Address: 127.0.0.1

Using search engines, I've seen this error, and my question, asked on
occasion; what I didn't find was what the actual solution was. Anyone
remember? I'll be sure to post what I find, so that maybe others can
benefit a few years from now. :)

Thanks...


Daniel

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Ubuntu 10.04 - Cannot Create TCP/IP Sockets

Couple of things to try:

- The default process is not postmaster any more, rather is it postgres:

http://www.postgresql.org/docs/8.4/interactive/server-start. html

- Try manually starting up the postgres with the debug cranked all the way =
up, see what that gives you in the server logs

- Perhaps try moving the port to somewhere else with the postgres command

HTH,
Tino

________________________________________
From: pgsql-admin-owner@postgresql.org [pgsql-admin-owner@postgresql.org] O=
n Behalf Of Daniel J. Summers [daniel.lists@djs-consulting.com]
Sent: Sunday, May 02, 2010 11:45 PM
To: Dai, Tino; pgsql-admin@postgresql.org
Subject: [ADMIN] Ubuntu 10.04 - Cannot Create TCP/IP Sockets

Good evening.

I've recently provisioned a new Ubuntu 10.04 server. I ran "apt-get
install postgresql-8.4", just like I normally do when I'm setting up a
new server. However, this time I got an error about an inability to
create TCP/IP sockets. The full error is below.

* Starting PostgreSQL 8.4 database server
* The PostgreSQL server failed to start. Please check the log output:
2010-05-03 03:36:06 UTC LOG: could not bind IPv4 socket: Cannot assign
requested address
2010-05-03 03:36:06 UTC HINT: Is another postmaster already running on
port 5432? If not, wait a few seconds and retry.
2010-05-03 03:36:06 UTC WARNING: could not create listen socket for
"localhost"
2010-05-03 03:36:06 UTC FATAL: could not create any TCP/IP sockets

However, there is nothing on port 5432...

root@coolnewserver:~# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:ssh *:* LISTEN
tcp 0 0 localhost:smtp *:* LISTEN
tcp6 0 0 [::]:ssh [::]:* LISTEN

...and no postgresql processes...

root@coolnewserver:~# ps -ef | grep postgresql
root 7738 3403 0 03:37 pts/0 00:00:00 grep --color=3Dauto postgre=
sql
root@coolnewserver:~# ps -ef | grep postmaster
root 7740 3403 0 03:37 pts/0 00:00:00 grep --color=3Dauto postmas=
ter

Possibly pertinent lines from /etc/postgresql/8.4/main/postgresql.conf...

listen_addresses =3D 'localhost'
port =3D 5432
max_connections =3D 100
unix_socket_directory =3D '/var/run/postgresql'

...permissions on that last directory are...

drwxrwsr-x 2 postgres postgres 40 2010-05-03 03:10 postgresql

...and localhost is resolving...

root@coolnewserver:~# nslookup localhost
Server: [IP]
Address: [IP]#53

Name: localhost
Address: 127.0.0.1

Using search engines, I've seen this error, and my question, asked on
occasion; what I didn't find was what the actual solution was. Anyone
remember? I'll be sure to post what I find, so that maybe others can
benefit a few years from now. :)

Thanks...


Daniel

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

--=20
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Ubuntu 10.04 - Cannot Create TCP/IP Sockets

Maybe SElinux is in the way?

Dai, Tino wrote:
> Couple of things to try:
>
> - The default process is not postmaster any more, rather is it postgres:
>
> http://www.postgresql.org/docs/8.4/interactive/server-start. html
>
> - Try manually starting up the postgres with the debug cranked all the way up, see what that gives you in the server logs
>
> - Perhaps try moving the port to somewhere else with the postgres command
>
> HTH,
> Tino
>
> ________________________________________
> From: pgsql-admin-owner@postgresql.org [pgsql-admin-owner@postgresql.org] On Behalf Of Daniel J. Summers [daniel.lists@djs-consulting.com]
> Sent: Sunday, May 02, 2010 11:45 PM
> To: Dai, Tino; pgsql-admin@postgresql.org
> Subject: [ADMIN] Ubuntu 10.04 - Cannot Create TCP/IP Sockets
>
> Good evening.
>
> I've recently provisioned a new Ubuntu 10.04 server. I ran "apt-get
> install postgresql-8.4", just like I normally do when I'm setting up a
> new server. However, this time I got an error about an inability to
> create TCP/IP sockets. The full error is below.
>
> * Starting PostgreSQL 8.4 database server
> * The PostgreSQL server failed to start. Please check the log output:
> 2010-05-03 03:36:06 UTC LOG: could not bind IPv4 socket: Cannot assign
> requested address
> 2010-05-03 03:36:06 UTC HINT: Is another postmaster already running on
> port 5432? If not, wait a few seconds and retry.
> 2010-05-03 03:36:06 UTC WARNING: could not create listen socket for
> "localhost"
> 2010-05-03 03:36:06 UTC FATAL: could not create any TCP/IP sockets
>
> However, there is nothing on port 5432...
>
> root@coolnewserver:~# netstat -l
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> tcp 0 0 *:ssh *:* LISTEN
> tcp 0 0 localhost:smtp *:* LISTEN
> tcp6 0 0 [::]:ssh [::]:* LISTEN
>
> ..and no postgresql processes...
>
> root@coolnewserver:~# ps -ef | grep postgresql
> root 7738 3403 0 03:37 pts/0 00:00:00 grep --color=auto postgresql
> root@coolnewserver:~# ps -ef | grep postmaster
> root 7740 3403 0 03:37 pts/0 00:00:00 grep --color=auto postmaster
>
> Possibly pertinent lines from /etc/postgresql/8.4/main/postgresql.conf...
>
> listen_addresses = 'localhost'
> port = 5432
> max_connections = 100
> unix_socket_directory = '/var/run/postgresql'
>
> ..permissions on that last directory are...
>
> drwxrwsr-x 2 postgres postgres 40 2010-05-03 03:10 postgresql
>
> ..and localhost is resolving...
>
> root@coolnewserver:~# nslookup localhost
> Server: [IP]
> Address: [IP]#53
>
> Name: localhost
> Address: 127.0.0.1
>
> Using search engines, I've seen this error, and my question, asked on
> occasion; what I didn't find was what the actual solution was. Anyone
> remember? I'll be sure to post what I find, so that maybe others can
> benefit a few years from now. :)
>
> Thanks...
>
>
> Daniel
>
> --
> Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-admin
>
>


--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Ubuntu 10.04 - Cannot Create TCP/IP Sockets

"Daniel J. Summers" wrote:

> 2010-05-03 03:36:06 UTC HINT: Is another postmaster already
> running on port 5432? If not, wait a few seconds and retry.

Just to be thorough, you got the same errors in the log when you
tried the start later?

-Kevin

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Ubuntu 10.04 - Cannot Create TCP/IP Sockets

Yeb Havinga writes:
>> 2010-05-03 03:36:06 UTC LOG: could not bind IPv4 socket: Cannot assign
>> requested address

Does it work if you change listen_addresses to '*' or '127.0.0.1' ?

>> ..and localhost is resolving...
>> root@coolnewserver:~# nslookup localhost

nslookup isn't a terribly reliable way to check that. Try "dig".

regards, tom lane

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Ubuntu 10.04 - Cannot Create TCP/IP Sockets

On 05/03/2010 11:55 AM, Dai, Tino wrote:
> Couple of things to try:
>
> - The default process is not postmaster any more, rather is it postgres:
>
> http://www.postgresql.org/docs/8.4/interactive/server-start. html
>

Any idea where this would be located? It's not in any of the usual
places. I found how to increase the debug level, but I can't find the
"postgres" executable.

@Kevin - yes, I'm still getting that problem, even when I ran it again
this morning.

@Yeb - I don't think Ubuntu enables SELinux by default. There aren't
any processes running for it. :)


Thanks again...


Daniel

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Ubuntu 10.04 - Cannot Create TCP/IP Sockets

On 05/03/2010 01:52 PM, Tom Lane wrote:
> Yeb Havinga writes:
>
>>> 2010-05-03 03:36:06 UTC LOG: could not bind IPv4 socket: Cannot assign
>>> requested address
>>>
> Does it work if you change listen_addresses to '*' or '127.0.0.1' ?
>

Bingo - changing it to "*" got it started. :)

>
>>> ..and localhost is resolving...
>>> root@coolnewserver:~# nslookup localhost
>>>
> nslookup isn't a terribly reliable way to check that. Try "dig".
>

Dig seemed to return "localhost" OK...

me@coolnewserver:~$ dig localhost

; <<>> DiG 9.7.0-P1 <<>> localhost
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18467
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; QUESTION SECTION:
;localhost. IN A

;; ANSWER SECTION:
localhost. 604800 IN A 127.0.0.1

;; AUTHORITY SECTION:
localhost. 604800 IN NS localhost.

;; ADDITIONAL SECTION:
localhost. 604800 IN AAAA ::1

;; Query time: 0 msec
;; SERVER: 173.203.4.8#53(173.203.4.8)
;; WHEN: Mon May 3 13:57:54 2010
;; MSG SIZE rcvd: 85

Could this possibly be related to IPv6? I know with this version of
Ubuntu (10.04), there's a known issue with LightTPD if you include the
IPv6 library in the configuration file (which is on by default - heh);
if you comment them out, it works just fine.

Thanks for the help!


Daniel

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Ubuntu 10.04 - Cannot Create TCP/IP Sockets

"Daniel J. Summers" writes:
> On 05/03/2010 01:52 PM, Tom Lane wrote:
>> Does it work if you change listen_addresses to '*' or '127.0.0.1' ?

> Bingo - changing it to "*" got it started. :)

OK, what about 127.0.0.1? Given that "dig localhost" returns that,
it *should* behave the same, but obviously something here is not
behaving as expected. Also, after you start it with '*', does
"psql -h localhost" work?

I'm guessing that there is something funny about the way your local
loopback is configured. It might be useful to look at "ifconfig lo".
On a Fedora machine I get

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:621786 errors:0 dropped:0 overruns:0 frame:0
TX packets:621786 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:149095072 (142.1 MiB) TX bytes:149095072 (142.1 MiB)


regards, tom lane

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Ubuntu 10.04 - Cannot Create TCP/IP Sockets

On 05/03/2010 02:20 PM, Tom Lane wrote:
> "Daniel J. Summers" writes:
>
>> On 05/03/2010 01:52 PM, Tom Lane wrote:
>>
>>> Does it work if you change listen_addresses to '*' or '127.0.0.1' ?
>>>
>
>> Bingo - changing it to "*" got it started. :)
>>
> OK, what about 127.0.0.1? Given that "dig localhost" returns that,
> it *should* behave the same, but obviously something here is not
> behaving as expected. Also, after you start it with '*', does
> "psql -h localhost" work?
>

Yes on both accounts. When I had 127.0.0.1 or *, it both started up,
and psql -h localhost gave me a password prompt.

> I'm guessing that there is something funny about the way your local
> loopback is configured. It might be useful to look at "ifconfig lo".
> On a Fedora machine I get
>
> lo Link encap:Local Loopback
> inet addr:127.0.0.1 Mask:255.0.0.0
> inet6 addr: ::1/128 Scope:Host
> UP LOOPBACK RUNNING MTU:16436 Metric:1
>

Mine's a carbon copy of that. It's strange.

Let me know if there are any other commands you'd like for me to run.
The original problem is solved, but I'd be happy to help find the
disconnect (if there is one - and pardon the pun). :)


Daniel

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Ubuntu 10.04 - Cannot Create TCP/IP Sockets

"Daniel J. Summers" writes:
> On 05/03/2010 02:20 PM, Tom Lane wrote:
>> OK, what about 127.0.0.1? Given that "dig localhost" returns that,
>> it *should* behave the same, but obviously something here is not
>> behaving as expected. Also, after you start it with '*', does
>> "psql -h localhost" work?

> Yes on both accounts. When I had 127.0.0.1 or *, it both started up,
> and psql -h localhost gave me a password prompt.

Huh. Well, there seems to be no other possible conclusion except that
getaddrinfo() is returning something valid for "127.0.0.1" but not
for "localhost". And it's not failing outright for "localhost", but
returning a sockaddr that bind() won't actually accept.

I'm guessing that there's some sort of IPv4-vs-IPv6 confusion
involved here, but it's hard to say more.

We might be able to learn something if you strace the postmaster
while it's trying to start up with "localhost" as the setting.
There will be a lot of output, so send it to a file. Look for
the bind() calls --- on my Linux box the relevant part of the
trace looks like this:

socket(PF_INET6, SOCK_STREAM, IPPROTO_IP) = 3
setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
setsockopt(3, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0
bind(3, {sa_family=AF_INET6, sin6_port=htons(5432), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
listen(3, 208) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 4
setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
bind(4, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
listen(4, 208) = 0
getppid() = 12399
open("/tmp/.s.PGSQL.5432.lock", O_RDWR|O_CREAT|O_EXCL, 0600) = 5
write(5, "12400\n/var/lib/pgsql/data"..., 33) = 33
close(5) = 0
unlink("/tmp/.s.PGSQL.5432") = -1 ENOENT (No such file or directory)
socket(PF_FILE, SOCK_STREAM, 0) = 5
bind(5, {sa_family=AF_FILE, path="/tmp/.s.PGSQL.5432"}, 110) = 0
chmod("/tmp/.s.PGSQL.5432", 0777) = 0
listen(5, 208) = 0

which shows the postmaster opening up both IPv6 and IPv4 sockets
as well as a Unix-domain socket in /tmp.

regards, tom lane

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Ubuntu 10.04 - Cannot Create TCP/IP Sockets

On 05/04/2010 02:24 PM, Tom Lane wrote:
>
> We might be able to learn something if you strace the postmaster
> while it's trying to start up with "localhost" as the setting.
> There will be a lot of output, so send it to a file. Look for
> the bind() calls --- on my Linux box the relevant part of the
> trace looks like this:
>

I did an "strace -o strace.txt /etc/init.d/postgresql-8.4 start" *, but
the only "socket" that was in it was in the error message. I modified
/usr/share/postgresql-common/init.d-functions to add strace to the
pg_ctlcluster call **, and there are lots of references to "socket", but
it doesn't look like it's having much success.

* Output is at http://djs-consulting.com/strace.txt
** Output is at http://djs-consulting.com/pg_trace.txt


Daniel

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Ubuntu 10.04 - Cannot Create TCP/IP Sockets

"Daniel J. Summers" writes:
> I did an "strace -o strace.txt /etc/init.d/postgresql-8.4 start" *, but
> the only "socket" that was in it was in the error message. I modified
> /usr/share/postgresql-common/init.d-functions to add strace to the
> pg_ctlcluster call **, and there are lots of references to "socket", but
> it doesn't look like it's having much success.

Just for the record, this wouldn't have yielded anything interesting
because you didn't use -f, so the tracing wouldn't have followed the
actions of anything but the directly invoked script. What we need to
see is strace of the postmaster itself.

regards, tom lane

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

Re: Ubuntu 10.04 - Cannot Create TCP/IP Sockets

I wrote:
> Huh. Well, there seems to be no other possible conclusion except that
> getaddrinfo() is returning something valid for "127.0.0.1" but not
> for "localhost". And it's not failing outright for "localhost", but
> returning a sockaddr that bind() won't actually accept.

For the archives: some off-list investigation revealed that Daniel's
machine is resolving "localhost" as "localhost.us", which resolves as
216.240.187.169, which of course is not a local address on his machine
so the bind() fails.

It's not immediately clear to me why his machine is resolving it like
that --- there's no search directive in /etc/resolv.conf, and even more
interestingly nslookup doesn't react that way --- but maybe Ubuntu has
some other configuration setting for DNS search path. The other
interesting question is what in the world made the owners of the .us
domain think it'd be a good idea to offer a translation of "localhost"
that's not 127.0.0.1. But anyway, this seems to be a DNS configuration
problem and not a Postgres bug.

regards, tom lane

--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin