Problems Connecting with SSL

I have three mySQL server, all of which are SSL-enable and all of which are generally accessible from remote clients over SSL.

However, one of my FreeBSD mySQL clients is having problems connecting to any of the servers. Running mySQL under truss reports the following:

----------
clock_gettime(13,{1298593067.000000000 }) = 0 (0x0)
open("/dev/urandom",O_RDONLY,00) = 4 (0x4)
read(4,"\a\^V\M-)\^C\M-L{`\M^O\M^@\M-v"...,32) = 32 (0x20)
sendto(2,"\^V\^C\^A\0Y\^A\0\0U\^C\^A\M-m"...,94,0x0,NULL,0x0 ) = 94 (0x5e)
recvfrom(2,"\^V",1,0x2,NULL,0x0) = 1 (0x1)
ioctl(2,FIONREAD,0xffffd784) = 0 (0x0)
recvfrom(2,"\^V\^C\^A\0J\^B\0\0F\^C\^AMf\M-u"...,1819,0x0,NU LL,0x0) = 1819 (0x71b)
clock_gettime(13,{1298593067.000000000 }) = 0 (0x0)
clock_gettime(13,{1298593067.000000000 }) = 0 (0x0)
gettimeofday({1298593067.844592 },0x0) = 0 (0x0)
sendto(2,"\^V\^C\^A\0F\^P\0\0B\0@\M-B\M^Y"...,134,0x0,NULL,0 x0) = 134 (0x86)
recvfrom(2,"\^U",1,0x2,NULL,0x0) = 1 (0x1)
ioctl(2,FIONREAD,0xffffd784) = 0 (0x0)
recvfrom(2,"\^U\^C\^A\0\^B\^B\n\^V\0\0\^B"...,33,0x0,NULL,0x 0) = 33 (0x21)
close(3) = 0 (0x0)
shutdown(2,SHUT_RDWR) = 0 (0x0)
close(2) = 0 (0x0)
fstat(1,{ mode=crw--w---- ,inode=113,size=0,blksize=4096 }) = 0 (0x0)
ioctl(1,TIOCGETA,0xffffe390) = 0 (0x0)
write(2,"ERROR 2026 (HY000): ",20) ERR#9 'Bad file descriptor'
write(2,"SSL connection error",20) ERR#9 'Bad file descriptor'
write(2,"\n",1) ERR#9 'Bad file descriptor'
write(1,"\a",1) = 1 (0x1)
close(4) = 0 (0x0)
process exit, rval = 1
----------

Can anyone tell me, based on the truss output above, what might be going wrong? I'm using the exact same command line options and ssl-ca file that I am using on other clients, which can connect successfully.

(As an aside: feature request: better SSL connection debugging, please!)

Tim Gustafson
Baskin School of Engineering
UC Santa Cruz
tjg@soe.ucsc.edu
831-459-5354



--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/mysql?unsub=gcdmg-mysql-2@m.gmane.org

Re: Problems Connecting with SSL

Hi Tim, all!


Sorry, but I have to start with a negative remark:
Why, oh why, don't you mention the version number you are using?

I hope it is some roughly current 5.1, but I'd rather like you to tell it=



Tim Gustafson wrote:
> I have three mySQL server, all of which are SSL-enable and all of which=
are generally accessible from remote clients over SSL.
>=20
> However, one of my FreeBSD mySQL clients is having problems connecting =
to any of the servers. Running mySQL under truss reports the following:
>=20

Looking at this output tells something went terribly wrong:
file descriptor 2 is mishandled.

> ----------
> clock_gettime(13,{1298593067.000000000 }) =3D 0 (0x0)
> open("/dev/urandom",O_RDONLY,00) =3D 4 (0x4)
> read(4,"\a\^V\M-)\^C\M-L{`\M^O\M^@\M-v"...,32) =3D 32 (0x20)
> sendto(2,"\^V\^C\^A\0Y\^A\0\0U\^C\^A\M-m"...,94,0x0,NULL,0x0 ) =3D 94 (0=
x5e)
> recvfrom(2,"\^V",1,0x2,NULL,0x0) =3D 1 (0x1)
> ioctl(2,FIONREAD,0xffffd784) =3D 0 (0x0)
> recvfrom(2,"\^V\^C\^A\0J\^B\0\0F\^C\^AMf\M-u"...,1819,0x0,NU LL,0x0) =3D=
1819 (0x71b)

In the above lines, socket calls 'sendto()' and 'recvfrom()' are applied
to FD 2. Whatever the transmitted data may be, this should not be done:
FD 2 is "standard error", so it is a file and no socket, and "receiving"
from it doesn't make any sense at all.

As the problem happens only for one of your combinations, I suspect that
for some unknown reason FD 2 was closed in that run, and then this
number got returned when a socket was to be created.
Again, this should not happen: The only reason to close FD 2 should be
to re-open standard error to some other destination, which is unlikely
for the client to do.

> [[...]]
> shutdown(2,SHUT_RDWR) =3D 0 (0x0)
> close(2) =3D 0 (0x0)

So FD 2 is closed now.

> fstat(1,{ mode=3Dcrw--w---- ,inode=3D113,size=3D0,blksize=3D4096 }) =3D=
0 (0x0)
> ioctl(1,TIOCGETA,0xffffe390) =3D 0 (0x0)
> write(2,"ERROR 2026 (HY000): ",20) ERR#9 'Bad file descriptor'
> write(2,"SSL connection error",20) ERR#9 'Bad file descriptor'
> write(2,"\n",1) ERR#9 'Bad file descriptor'

.. but with a closed FD 2 there is no chance to output an error messag=
e.
(The operations on FD 1 also look suspicious to me.)

> write(1,"\a",1) =3D 1 (0x1)
> close(4) =3D 0 (0x0)
> process exit, rval =3D 1
> ----------
>=20
> Can anyone tell me, based on the truss output above, what might be goin=
g wrong? I'm using the exact same command line options and ssl-ca file t=
hat I am using on other clients, which can connect successfully.

I can't really tell what happened. It might be that file descriptor 2
(and probably also 1) were closed without good reason, and then creating
a socket (or opening some file) returned these descriptors.

You might check the preceding "truss" output for a "close(2)", IMO that
would likely be an error - but I have no idea why it happened.

>=20
> (As an aside: feature request: better SSL connection debugging, please!=
)

I assume your SSL problems might just be a consequence of inappropriate
actions on FDs 1 and 2. Did you do any redirection, are you running into
"disk full", or is there any similar rsk on your stdout and stderr?


HTH,
Jörg

--=20
Joerg Bruehe, MySQL Build Team, joerg.bruehe@oracle.com
ORACLE Deutschland B.V. & Co. KG, Komturstrasse 18a, D-12099 Berlin
Geschaeftsfuehrer: Juergen Kunz, Marcel v.d. Molen, Alexander v.d. Ven
Amtsgericht Muenchen: HRA 95603


--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/mysql?unsub=3Dgcdmg-mysql-2@m.gmane.o rg

Re: Problems Connecting with SSL

> Sorry, but I have to start with a negative remark:
> Why, oh why, don't you mention the version number you
> are using? I hope it is some roughly current 5.1, but
> I'd rather like you to tell it.

Doh! Right you are! Sorry, that was totally my bad.

mysql> select version();
+------------+
| version() |
+------------+
| 5.1.51-log |
+------------+

The server and the client are both running FreeBSD as follows:

FreeBSD 8.1-RELEASE-p2

Tim Gustafson
Baskin School of Engineering
UC Santa Cruz
tjg@soe.ucsc.edu
831-459-5354

--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/mysql?unsub=gcdmg-mysql-2@m.gmane.org

Re: Problems Connecting with SSL

> Interesting that you can request advice from finland vs
> calling from redwood shores CA for technical advice
>
> first take a look at:
> http://dev.mysql.com/doc/refman/5.0/en/secure-basics.html
>
> once you've determined the algorithm you will want to implement
> take a look at SSL 1,2,3 and TLS 1.0,1.1,1.2 and determine which
> protocol works best for you
>
> http://en.wikipedia.org/wiki/Transport_Layer_Security
>
> feel free to ping me offline for implementation details

In my original post, I stated that I had SSL connections up and running on all but this one client. I'm intimately familiar with the inner workings of SSL and TLS, and I'm positive that the servers are configured correctly, and that in general my clients are all able to connect using SSL. It's just this one client that is failing, and absent any useful debugging information, I'm having a hard time understanding why.

As to why I didn't call Redwood Shores: I didn't realize that Oracle was in the business of giving out free tech support. I just e-mailed the mySQL general support list, which is usually what I do when I need help from an Open Source software group. :)

Tim Gustafson
Baskin School of Engineering
UC Santa Cruz
tjg@soe.ucsc.edu
831-459-5354

--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/mysql?unsub=gcdmg-mysql-2@m.gmane.org

Re: Problems Connecting with SSL

> perhaps a misconfiguration on the failing client ..
> can you go to successful client, enable DEBUG level
> debugging and log the connection and then go to the
> failing client..enable DEBUG level debugging and log
> the connection to the failing client this would determine
> deltas between the failing and successful connections and
> you compare each log side-by-side to ascertain the point
> of failure. joerg ..any thoughts on this matter??

Well that's exactly my question: how does one enable debugging of this kind of connection on the mySQL client? I get no more information when I use "--verbose --verbose --verbose" than when I don't.

Tim Gustafson
Baskin School of Engineering
UC Santa Cruz
tjg@soe.ucsc.edu
831-459-5354

--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/mysql?unsub=gcdmg-mysql-2@m.gmane.org

Re: Problems Connecting with SSL

Hi!


Martin Gainty wrote:
> [[...]]
>=20
> joerg ..any thoughts on this matter??

Sorry - none.
I never dealt with the inner workings of the SSL library.

Just seeing file descriptor 2 (standard error) used for socket
operations makes me assume *very* strongly that something went wrong
well before that part:
For some (until now: unknown) reason standard error must have been
closed, so that FD 2 became available for the socket creation, and in a
MYSQL client application I don't see any reason to ever close standard
error.

I propose Tim should check the full truss output for some preceding
close(2). Hopefully, the surroundings of that call will give a hint what
went wrong.
(I hope it was caused by some write error, disk full, permission
problem, ... which Tim can fix. I of course agree such an error should
not make the program do a close(2), but it might happen nonetheless.)

That's all I can say here.


Regards,
Jörg

--=20
Joerg Bruehe, MySQL Build Team, joerg.bruehe@oracle.com
ORACLE Deutschland B.V. & Co. KG, Komturstrasse 18a, D-12099 Berlin
Geschaeftsfuehrer: Juergen Kunz, Marcel v.d. Molen, Alexander v.d. Ven
Amtsgericht Muenchen: HRA 95603


--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe: http://lists.mysql.com/mysql?unsub=3Dgcdmg-mysql-2@m.gmane.o rg