I want to send all error output to a process for specialized logging.
I know that you can use CustomLog with a pipe. However, will that
capture output from things like 'print STDERR ...' or if an XS module
directly writes to stderr? I need that output to also go to my logging
process.
Does anyone know if there an Apache module which does this?
On Fri 03 Apr 2009, E R wrote:
> I want to send all error output to a process for specialized logging.
> I know that you can use CustomLog with a pipe. However, will that
> capture output from things like 'print STDERR ...' or if an XS module
> directly writes to stderr? I need that output to also go to my
> logging process.
>
> Does anyone know if there an Apache module which does this?
Give a named pipe's name to ErrorLog outside any virtual host. Then have
your process read from that pipe.
At shell level:
mkfifo /path/to/PIPE
your_program
in httpd.conf:
ErrorLog /path/to/PIPE
I'd very much recommend to monitor "your_program" and have it restarted
automatically if it crashes. The simplest way to do that is to start it
via /etc/inittab.
Torsten
--
Need professional mod_perl support?
Just hire me: torsten.foertsch@gmx.net
One operational issue with this is that if the logger process dies,
existing httpd processes will get a SIGPIPE signal on subsequent
writes. Restarting the logger process does not restore the lost
connections.
On Fri, Apr 3, 2009 at 12:38 PM, Torsten Foertsch
> On Fri 03 Apr 2009, E R wrote:
>> I want to send all error output to a process for specialized logging.
>> I know that you can use CustomLog with a pipe. However, will that
>> capture output from things like 'print STDERR ...' or if an XS module
>> directly writes to stderr? I need that output to also go to my
>> logging process.
>>
>> Does anyone know if there an Apache module which does this?
>
> Give a named pipe's name to ErrorLog outside any virtual host. Then have
> your process read from that pipe.
>
> At shell level:
>
> =A0mkfifo /path/to/PIPE
> =A0your_program
>
> in httpd.conf:
>
> =A0ErrorLog /path/to/PIPE
>
> I'd very much recommend to monitor "your_program" and have it restarted
> automatically if it crashes. The simplest way to do that is to start it
> via /etc/inittab.
>
> Torsten
>
> --
> Need professional mod_perl support?
> Just hire me: torsten.foertsch@gmx.net
>
On Sat 04 Apr 2009, E R wrote:
> One operational issue with this is that if the logger process dies,
> existing httpd processes will get a SIGPIPE signal on subsequent
> writes. Restarting the logger process does not restore the lost
> connections.
>
>
> On Fri, Apr 3, 2009 at 12:38 PM, Torsten Foertsch
>
>
> > On Fri 03 Apr 2009, E R wrote:
> >> I want to send all error output to a process for specialized
> >> logging. I know that you can use CustomLog with a pipe. However,
> >> will that capture output from things like 'print STDERR ...' or if
> >> an XS module directly writes to stderr? I need that output to also
> >> go to my logging process.
> >>
> >> Does anyone know if there an Apache module which does this?
> >
> > Give a named pipe's name to ErrorLog outside any virtual host. Then
> > have your process read from that pipe.
At least with prefork and worker MPMs apache ignores SIGPIPE:
httpd-2.2.11/server/mpm/prefork/prefork.c:406: (similar in worker.c)
#ifdef SIGPIPE
sa.sa_handler = SIG_IGN;
if (sigaction(SIGPIPE, &sa, NULL) < 0)
ap_log_error(APLOG_MARK, APLOG_WARNING, errno, ap_server_conf, "sigaction(SIGPIPE)");
#endif
And later on it does not check the return code when writing the log:
httpd-2.2.11/server/log.c:682:
apr_file_puts(errstr, logf);
apr_file_flush(logf);
Back in 1999-2000 apache had a reliability problem with piped loggers.
Ever since I log to named pipes.
Torsten
--
Need professional mod_perl support?
Just hire me: torsten.foertsch@gmx.net
On Fri, Apr 3, 2009 at 1:22 PM, E R
> I want to send all error output to a process for specialized logging.
> I know that you can use CustomLog with a pipe. However, will that
> capture output from things like 'print STDERR ...' or if an XS module
> directly writes to stderr? I need that output to also go to my logging
> process.
In addition to what Torsten said about the pipes, you should know that
it STDERR is directed to the error_log (or pipe if you configure
that), so it should catch the cases you're asking about.
- Perrin
Hi Torsten,
On Sun, Apr 5, 2009 at 4:13 AM, Torsten Foertsch
>
> And later on it does not check the return code when writing the log:
>
> httpd-2.2.11/server/log.c:682:
> =A0 =A0 =A0 =A0apr_file_puts(errstr, logf);
> =A0 =A0 =A0 =A0apr_file_flush(logf);
My concern is about what happens in this scenario:
1. Set everything up (named pipe, logger process, etc.)
2. Now suppose the logger process dies (it crashes or I need to restart it)
3. Any writes through existing connections to the named pipe will just
fail, and I'll lose those writes.
I'm pretty sure about #3, but I might be wrong.
Perhaps I can't do what I want to do at the application level and I
need to go to the kernel level - like write my own device driver or
something, especially since I want to capture all output sent through
stderr (ie. from Apache and from mod_perl and from any XS modules.)
On Sun 05 Apr 2009, E R wrote:
> My concern is about what happens in this scenario:
>
> 1. Set everything up (named pipe, logger process, etc.)
> 2. Now suppose the logger process dies (it crashes or I need to
> restart it) 3. Any writes through existing connections to the named
> pipe will just fail, and I'll lose those writes.
>
> I'm pretty sure about #3, but I might be wrong.
A (named) pipe is a ring buffer within the kernel. A writer can write to
it up to a certain level. Then it is "full" and the writer blocks or
gets an EAGAIN for nonblocking writes.
So there is a certain period of time when the data resides in the kernel
buffer. All that data is lost when the kernel panics, loses power or
similar.
Then the reader reads a piece of data from the pipe and does something
with it.
If that process crashes between reading the data from the pipe and
finishing its processing that piece is lost.
So there are at least 2 periods of time when data can get lost. But is
that all? It depends on your definition of "sending to stderr". For
example lets look at httpd. The event it wants to log happens at a
certain "point" (in quotes because it is a short period in reality) in
time. Before it goes out of stderr there is a period of time when a
string is made from the event in ap_log_error(), log_error_core() or
somewhere else. That perhaps requires some malloc()ing to extend a pool
etc. And of course there can be a stray SIGKILL on its way to the
process that arrives just in the moment before the formatted string
hits stderr.
That means there is a period of time when the event that is to be logged
has happend but has not arrived on stderr.
All I want to say here is you can't make 100% sure that every event that
has to be logged is really logged. You can achieve quite a good level
but never 100%.
The good thing about a named pipe is that it can be connected by
unrelated processes. So if the reader dies only the piece of data it is
currently processing is lost. The writer can still write until the pipe
is full. So, have your reader restarted fast enough. That's why I
mentioned /etc/inittab.
> Perhaps I can't do what I want to do at the application level and I
> need to go to the kernel level - like write my own device driver or
> something, especially since I want to capture all output sent through
> stderr (ie. from Apache and from mod_perl and from any XS modules.)
As I said before you can't achieve 100% even at kernel level. And
imagine you'd want to write to a data base from kernel space. I don't
say its impossible but you'd have to reimplement the whole protocol and
the protocol specs may be not even published. Happy hacking!
On Linux the ErrorLog of the base server is always opened on file
descriptor 2 that means stderr. So, a reader that reads this stream
gets everything (under normal conditions) that Apache, Perl or any XS
module writes to stderr.
Torsten
--
Need professional mod_perl support?
Just hire me: torsten.foertsch@gmx.net
Torsten Foertsch wrote:
[...]
>
> On Linux the ErrorLog of the base server is always opened on file
> descriptor 2 that means stderr. So, a reader that reads this stream
> gets everything (under normal conditions) that Apache, Perl or any XS
> module writes to stderr.
>
Hi Torsten, just jumping in here and hooking into the above.
How does it work with Apache children/threads that may be acting as a
VirtualHost's having their own ErrorLog ?
My experience is that anything logged by mod_perl modules running "in" a
VirtualHost using explicit warn() or $r->log_error() or $r->log->info()
etc.. ends up in the VirtualHost's ErrorLog, but some run-time messages like
"Use of uninitialized value in concatenation (.) or string at .." end up
in the "main" Apache error log. How does that come about ?
On Mon 06 Apr 2009, Andr=E9 Warnier wrote:
> How does it work with Apache children/threads that may be acting as a
> VirtualHost's having their own ErrorLog ?
> My experience is that anything logged by mod_perl modules running
> "in" a VirtualHost using explicit warn() or $r->log_error() or
> $r->log->info() etc.. ends up in the VirtualHost's ErrorLog, but some
> run-time messages like "Use of uninitialized value in concatenation
> (.) or string at .." end up in the "main" Apache error log. =A0How does
> that come about ?
$r or $s based logging goes to the VHosts ErrorLog, yes. But the=20
original poster worried about "all of STDERR" come it from perl, xs or=20
whereever else.
Torsten
=2D-=20
Need professional mod_perl support?
Just hire me: torsten.foertsch@gmx.net