crash in Apache core with mod_proxy_fcgi in 2.2 and trunk

Hi list,

I posted this issue already to the dev-list, but got no reply. I apologize, if I misinterpreted the intention of the dev-list.


So first, let me describe my scenario:

I run Apache 2.2.14 on a Debian (2.6.28-2-amd64) box. As I have a urgent need for mod_proxy_fcgi, I tried a "backport" by copying mod_proxy_fcgi.c and fcgi_protocol.h to modules/proxy from the current Apache trunk, modified the Makefile.am, ran buildconf and so on - everything's fine. mod_proxy_fcgi seems to work out of the box without any source-changes. By the way - it would be really great to have mod_proxy_fcgi in 2.2. As mod_proxy_scgi did it, I hope mod_proxy_fcgi will come soon? :)

For your reference, this is my configure line:
../configure --with-mpm=worker --enable-maintainer-mode --enable-exception-hook --enable-so --enable-proxy --enable-proxy-fcgi
I added --enable-maintainer-mode --enable-exception-hook to aid me in debugging. The scenario crashes with and without these options.


Now if my FCGI-backend sends an invalid header in a response, Apache segfaults. As I saw in the sourcecode, HTTP-Headerlines in the format name=value are expected followed by a double CRLF. If just a HTTP status line is sent along, this is enough to tear the complete child process down.

This is the stack trace:
#0 0x0000000000ad9630 in ?? ()
#1 0x000000000044b2d3 in getsfunc_BRIGADE (buf=0x4cf85a20 "", len=8191, arg=0xae1ee0) at util_script.c:636
#2 0x000000000044af2a in ap_scan_script_header_err_core (r=0xadb6e8, buffer=0x0, getsfunc=0x44b267 , getsfunc_data=0xae1ee0) at util_script.c:540
#3 0x000000000044b465 in ap_scan_script_header_err_brigade (r=0xadb6e8, bb=0xae1ee0, buffer=0x0) at util_script.c:669
#4 0x0000000000478e87 in handle_headers (r=0xadb6e8, state=0x4cf89b7c,
readbuf=0x4cf87b40 "HTTP/1.1 200 OK\r\nContent-Type: text/html;\r\nSet-Cookie: TestCookie=\"FCGI TestCookie\"; Comment=\"Test-Cookie ueber FCGI\"; Path=/; Version=1;\r\n\r\n\n", ob=0xae1ee0)
at mod_proxy_fcgi.c:479
#5 0x00000000004795c6 in dispatch (conn=0xa49b58, r=0xadb6e8, request_id=1) at mod_proxy_fcgi.c:750
#6 0x0000000000479909 in fcgi_do_request (p=0xadb668, r=0xadb6e8, conn=0xa49b58, origin=0x0, conf=0xa73600, uri=0xae0db8, url=0xae0e40 "/hallo", server_portstr=0x4cf89d30 ":8080") at mod_proxy_fcgi.c:890
#7 0x0000000000479c79 in proxy_fcgi_handler (r=0xadb6e8, worker=0xa77f78, conf=0xa73148, url=0xae0e40 "/hallo", proxyname=0x0, proxyport=0) at mod_proxy_fcgi.c:981
#8 0x00000000004676a4 in proxy_run_scheme_handler (r=0xadb6e8, worker=0xa77f78, conf=0xa73148, url=0xae0cde "fcgi://proto-web1:80/hallo", proxyhost=0x0, proxyport=0) at mod_proxy.c:2380
#9 0x0000000000463d7a in proxy_handler (r=0xadb6e8) at mod_proxy.c:996
#10 0x0000000000443508 in ap_run_handler (r=0xadb6e8) at config.c:158
#11 0x0000000000443da1 in ap_invoke_handler (r=0xadb6e8) at config.c:372
#12 0x00000000004861bc in ap_process_request (r=0xadb6e8) at http_request.c:282
#13 0x0000000000483194 in ap_process_http_connection (c=0xad57b0) at http_core.c:190
#14 0x000000000044c94a in ap_run_process_connection (c=0xad57b0) at connection.c:43
#15 0x000000000044cd78 in ap_process_connection (c=0xad57b0, csd=0xad5598) at connection.c:178
#16 0x00000000004a0e52 in process_socket (p=0xad5518, sock=0xad5598, my_child_num=0, my_thread_num=23, bucket_alloc=0xad9658) at worker.c:544
#17 0x00000000004a1728 in worker_thread (thd=0xa78e50, dummy=0xa389c0) at worker.c:894
#18 0x00007f7d0bf7befd in dummy_worker (opaque=0xa78e50) at threadproc/unix/thread.c:142
#19 0x00007f7d0b8fcfc7 in start_thread () from /lib/libpthread.so.0
#20 0x00007f7d0b46e5ad in clone () from /lib/libc.so.6
#21 0x0000000000000000 in ?? ()

While eating up the response in getsfunc_BRIGADE, apr_bucket_read hits the brigade sentinel (the (e)->type->read pointer of a sentinel is obviously bogus) and segfaults.

I can't judge if mod_proxy_fcgi doesn't meet the preconditions to call ap_scan_script_header_err_brigade, getsfunc_BRIGADE should be prepared to see brigade sentinels.

I tried it out with the current Apache 2.3 trunk and it shows the same behaviour, so I assume this isn't an issue of mod_proxy_fcgi being incompatible with Apache 2.2.

So, is this a bug in mod_proxy_fcgi? How can it be fixed? Adding detection for brigade-sentinels in getsfunc_BRIGADE seems to leave the connection garbled when it is reused, as not all data is read.

Regards,
Edgar Frank

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: crash in Apache core with mod_proxy_fcgi in 2.2 and trunk

Hi everyone,

it's been two days now without a reply (or any reply on the dev list).
It's not that I'm inpatient, but I can't imagine that nobody is
interested in this issue, especially with Apache developers on both
lists.

I'm relatively new to mailing lists and wonder if I did something wrong.
So if I violated the etiquette, I'm sorry, but please let me know. Or if
I'm still on the wrong list, please let me know, too.

Greetings & Regards,
Edgar


--- original message ---

Hi list,

I posted this issue already to the dev-list, but got no reply. I
apologize, if I misinterpreted the intention of the dev-list.

So first, let me describe my scenario:

I run Apache 2.2.14 on a Debian (2.6.28-2-amd64) box. As I have a urgent
need for mod_proxy_fcgi, I tried a "backport" by copying
mod_proxy_fcgi.c and fcgi_protocol.h to modules/proxy from the current
Apache trunk, modified the Makefile.am, ran buildconf and so on -
everything's fine. mod_proxy_fcgi seems to work out of the box without
any source-changes. By the way - it would be really great to have
mod_proxy_fcgi in 2.2. As mod_proxy_scgi did it, I hope mod_proxy_fcgi
will come soon? :)

For your reference, this is my configure line:
../configure --with-mpm=worker --enable-maintainer-mode
--enable-exception-hook --enable-so --enable-proxy --enable-proxy-fcgi
I added --enable-maintainer-mode --enable-exception-hook to aid me in
debugging. The scenario crashes with and without these options.

Now if my FCGI-backend sends an invalid header in a response, Apache
segfaults. As I saw in the sourcecode, HTTP-Headerlines in the format
name=value are expected followed by a double CRLF. If just a HTTP status
line is sent along, this is enough to tear the complete child process
down.

This is the stack trace:
#0 0x0000000000ad9630 in ?? ()
#1 0x000000000044b2d3 in getsfunc_BRIGADE (buf=0x4cf85a20 "",
len=8191, arg=0xae1ee0) at util_script.c:636
#2 0x000000000044af2a in ap_scan_script_header_err_core (r=0xadb6e8,
buffer=0x0, getsfunc=0x44b267 ,
getsfunc_data=0xae1ee0) at util_script.c:540
#3 0x000000000044b465 in ap_scan_script_header_err_brigade (r=0xadb6e8,
bb=0xae1ee0, buffer=0x0) at util_script.c:669
#4 0x0000000000478e87 in handle_headers (r=0xadb6e8, state=0x4cf89b7c,
readbuf=0x4cf87b40 "HTTP/1.1 200 OK\r\nContent-Type:
text/html;\r\nSet-Cookie: TestCookie=\"FCGI TestCookie\";
Comment=\"Test-Cookie ueber FCGI\"; Path=/; Version=1;\r\n\r\n\n",
ob=0xae1ee0)
at mod_proxy_fcgi.c:479
#5 0x00000000004795c6 in dispatch (conn=0xa49b58, r=0xadb6e8,
request_id=1) at mod_proxy_fcgi.c:750
#6 0x0000000000479909 in fcgi_do_request (p=0xadb668, r=0xadb6e8,
conn=0xa49b58, origin=0x0, conf=0xa73600, uri=0xae0db8, url=0xae0e40
"/hallo", server_portstr=0x4cf89d30 ":8080") at mod_proxy_fcgi.c:890
#7 0x0000000000479c79 in proxy_fcgi_handler (r=0xadb6e8,
worker=0xa77f78, conf=0xa73148, url=0xae0e40 "/hallo", proxyname=0x0,
proxyport=0) at mod_proxy_fcgi.c:981
#8 0x00000000004676a4 in proxy_run_scheme_handler (r=0xadb6e8,
worker=0xa77f78, conf=0xa73148, url=0xae0cde
"fcgi://proto-web1:80/hallo", proxyhost=0x0, proxyport=0) at
mod_proxy.c:2380
#9 0x0000000000463d7a in proxy_handler (r=0xadb6e8) at mod_proxy.c:996
#10 0x0000000000443508 in ap_run_handler (r=0xadb6e8) at config.c:158
#11 0x0000000000443da1 in ap_invoke_handler (r=0xadb6e8) at config.c:372
#12 0x00000000004861bc in ap_process_request (r=0xadb6e8) at
http_request.c:282
#13 0x0000000000483194 in ap_process_http_connection (c=0xad57b0) at
http_core.c:190
#14 0x000000000044c94a in ap_run_process_connection (c=0xad57b0) at
connection.c:43
#15 0x000000000044cd78 in ap_process_connection (c=0xad57b0,
csd=0xad5598) at connection.c:178
#16 0x00000000004a0e52 in process_socket (p=0xad5518, sock=0xad5598,
my_child_num=0, my_thread_num=23, bucket_alloc=0xad9658) at worker.c:544
#17 0x00000000004a1728 in worker_thread (thd=0xa78e50, dummy=0xa389c0)
at worker.c:894
#18 0x00007f7d0bf7befd in dummy_worker (opaque=0xa78e50) at
threadproc/unix/thread.c:142
#19 0x00007f7d0b8fcfc7 in start_thread () from /lib/libpthread.so.0
#20 0x00007f7d0b46e5ad in clone () from /lib/libc.so.6
#21 0x0000000000000000 in ?? ()

While eating up the response in getsfunc_BRIGADE, apr_bucket_read hits
the brigade sentinel (the (e)->type->read pointer of a sentinel is
obviously bogus) and segfaults.

I can't judge if mod_proxy_fcgi doesn't meet the preconditions to call
ap_scan_script_header_err_brigade, getsfunc_BRIGADE should be prepared
to see brigade sentinels.

I tried it out with the current Apache 2.3 trunk and it shows the same
behaviour, so I assume this isn't an issue of mod_proxy_fcgi being
incompatible with Apache 2.2.

So, is this a bug in mod_proxy_fcgi? How can it be fixed? Adding
detection for brigade-sentinels in getsfunc_BRIGADE seems to leave the
connection garbled when it is reused, as not all data is read.

Regards,
Edgar Frank


------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: crash in Apache core with mod_proxy_fcgi in 2.2 andtrunk

On 20 Nov 2009, at 10:54, Edgar Frank wrote:

> it's been two days now without a reply (or any reply on the dev list).

Don't recollect seeing it there (not that I'd necessarily have taken
any notice of it).

> I run Apache 2.2.14 on a Debian (2.6.28-2-amd64) box. As I have a urgent
> need for mod_proxy_fcgi, I tried a "backport" by copying

mod_proxy_fcgi is not really ready for primetime. You should only
use it if you're expecting to hack/debug it, and preferably feed back
your experiences/fixes.

Reading what you've sent, it would be a useful bug report to
someone interested in mod_proxy_fcgi, *IF* that module had
reached the stage where its expected to work (e.g. an alpha
test release).

As it stands, there's no great likelihood of that happening unless
someone makes it happen. Since mod_fcgid has been donated
to apache, that's now the focus of fastcgi work among the devs.

My best suggestions:
(1) Enter a report in bugzilla
(2) If you really want proxy_fcgi and not mod_fcgid, tell us what
you need that the one offers but the other doesn't.

--
Nick Kew

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: crash in Apache core with mod_proxy_fcgi in 2.2 and trunk

Hi Nick,

thanks for you instant reply.

> mod_proxy_fcgi is not really ready for primetime. You should only
> use it if you're expecting to hack/debug it, and preferably feed back
> your experiences/fixes.
That's good to know. I saw mod_proxy_fcgi being relatively old. When it
was added into the SVN trunk it looked much like a copy of the formerly
existing mod_proxy_fcgi. So I expected it to be kinda stable.

What would be then the right place/person to report feedback to?

> Reading what you've sent, it would be a useful bug report to
> someone interested in mod_proxy_fcgi, *IF* that module had
> reached the stage where its expected to work (e.g. an alpha
> test release).
Okay thanks, I'll keep that in mind.

> As it stands, there's no great likelihood of that happening unless
> someone makes it happen. Since mod_fcgid has been donated
> to apache, that's now the focus of fastcgi work among the devs.
>
> My best suggestions:
> (1) Enter a report in bugzilla
> (2) If you really want proxy_fcgi and not mod_fcgid, tell us what
> you need that the one offers but the other doesn't.

For (2), what would be again the right place to report?

The prime reason for me to use mod_proxy_fcgi is that mod_fcgid doesn't
offer a possibility to specify an externally spawned FCGI process on
a remote host. The other reason is that I need the env-var interpolation
respectivly mod_rewrite-capabilities to change the target FCGI server
depening on request data (e.g. forward a well-known user to his "home"
server).
mod_proxy_fcgi is perfect for me, because it's small, doesn't come with
a process manager and allows dynamic configuration of the target server.

If I missed that features in mod_fcgid, I'd be glad if you could let me
know.

There is still mod_fastcgi on my mind, but I'd have to modify it to
work for my purpose and it's code is horrible in terms that it still
has support for Apache 1.x.

Regards,
Edgar


------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: crash in Apache core with mod_proxy_fcgi in 2.2 andtrunk

Edgar Frank wrote:
> Hi Nick,
>
> thanks for you instant reply.
>
>> mod_proxy_fcgi is not really ready for primetime. You should only
>> use it if you're expecting to hack/debug it, and preferably feed back
>> your experiences/fixes.
> That's good to know. I saw mod_proxy_fcgi being relatively old. When it
> was added into the SVN trunk it looked much like a copy of the formerly
> existing mod_proxy_fcgi. So I expected it to be kinda stable.

Hmmm. I may have been wrong; it's just my impression that
mod_proxy_fcgi had essentially been abandoned.

> What would be then the right place/person to report feedback to?

dev@httpd and/or bugzilla. But that depends on someone who's
actually worked on it being motivated to respond, and mod_fcgid
reduces the likelihood of that happening.

>> As it stands, there's no great likelihood of that happening unless
>> someone makes it happen. Since mod_fcgid has been donated
>> to apache, that's now the focus of fastcgi work among the devs.
>>
>> My best suggestions:
>> (1) Enter a report in bugzilla
>> (2) If you really want proxy_fcgi and not mod_fcgid, tell us what
>> you need that the one offers but the other doesn't.
>
> For (2), what would be again the right place to report?

dev@httpd.

> The prime reason for me to use mod_proxy_fcgi is that mod_fcgid doesn't
> offer a possibility to specify an externally spawned FCGI process on
> a remote host. The other reason is that I need the env-var interpolation
> respectivly mod_rewrite-capabilities to change the target FCGI server
> depening on request data (e.g. forward a well-known user to his "home"
> server).
> mod_proxy_fcgi is perfect for me, because it's small, doesn't come with
> a process manager and allows dynamic configuration of the target server.

Now that makes sense - and indeed ties into something I'm trying to
work towards (but don't expect anything exciting anytime soon).
Posting something based on that to dev@ might be more productive than
a long/rambling report that you have to read in detail before confirming
it looks like a bug (as opposed to a user error).

> If I missed that features in mod_fcgid, I'd be glad if you could let me
> know.
>
> There is still mod_fastcgi on my mind, but I'd have to modify it to
> work for my purpose and it's code is horrible in terms that it still
> has support for Apache 1.x.

If mod_fastcgi has capabilities that are missing in mod_fcgid and
that you have a use for, that may be worth turning into a feature
request.

--
Nick Kew

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org