Centralized logging.

So how are people out there in mailing list land handling centralized
logging?

I'd like to mirror my web farm's logfiles on a central server, but want
to be sure to preserve the W3C-ness so it's easy to run awstats, etc
against them from off this central box.

The options I see so far are:

- Piping to syslog via external process (perl script, etc).
* Downside being that we end up with syslog timestamps
- Pipe to something like netcat
* Custom-ish / hacky
- Fire off some sort of synchronization process from central box to
pull in logs (rsync) periodically.
* Asynchronous -- there's a somewhat large window of time where
logs will be missing on the internal server.
- Log directly to shared, remote file server (via NFS)
* Log server network issues or technical problems could affect
Apache servers

How do you guys typically deal with this?

Ray

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Centralized logging.

I'm using syslog-ng. It does the job.

A nice little rsync script is nice, but you're still storing log files
on the individual servers until you run some rsync and delete script.
This can be risky if resources are limited or machines are ephemeral.

NFS has its own issues as well. You've got to ask yourself "What's my
plan B" and write that into your logging config/scripts.

On 4/8/10 3:52 PM, Ray Van Dolson wrote:
> So how are people out there in mailing list land handling centralized
> logging?
>
> I'd like to mirror my web farm's logfiles on a central server, but want
> to be sure to preserve the W3C-ness so it's easy to run awstats, etc
> against them from off this central box.
>
> The options I see so far are:
>
> - Piping to syslog via external process (perl script, etc).
> * Downside being that we end up with syslog timestamps
> - Pipe to something like netcat
> * Custom-ish / hacky
> - Fire off some sort of synchronization process from central box to
> pull in logs (rsync) periodically.
> * Asynchronous -- there's a somewhat large window of time where
> logs will be missing on the internal server.
> - Log directly to shared, remote file server (via NFS)
> * Log server network issues or technical problems could affect
> Apache servers
>
> How do you guys typically deal with this?
>
> Ray
>
> ------------------------------------------------------------ ---------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>

--

Jason A. Nunnelley
+1 2562971652

http://www.google.com/profiles/imjasonn

[Member Tekany, LLC]


------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Centralized logging.

On Thu, Apr 08, 2010 at 02:18:28PM -0700, Jason Nunnelley wrote:
> I'm using syslog-ng. It does the job.
>
> A nice little rsync script is nice, but you're still storing log files
> on the individual servers until you run some rsync and delete script.
> This can be risky if resources are limited or machines are ephemeral.
>
> NFS has its own issues as well. You've got to ask yourself "What's my
> plan B" and write that into your logging config/scripts.

Thanks Jason. We're using syslog-ng as well.

So how exactly are you implementing this? Apache procs piping to a
perl process which shunts things out to your syslog server?

Do you do any post/pre-processing for timestamp duplication? We want
to try and have the logs on the syslog server look exactly like they
did on the origin server (ie no syslog timestamps, no syslog
facility/priority stuff and no "hostname").

Ray

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Centralized logging.

It was thus said that the Great Ray Van Dolson once stated:
> On Thu, Apr 08, 2010 at 02:18:28PM -0700, Jason Nunnelley wrote:
> > I'm using syslog-ng. It does the job.
> >
> > A nice little rsync script is nice, but you're still storing log files
> > on the individual servers until you run some rsync and delete script.
> > This can be risky if resources are limited or machines are ephemeral.
> >
> > NFS has its own issues as well. You've got to ask yourself "What's my
> > plan B" and write that into your logging config/scripts.
>
> Thanks Jason. We're using syslog-ng as well.
>
> So how exactly are you implementing this? Apache procs piping to a
> perl process which shunts things out to your syslog server?

I just tested this, and if you are on a Unix-like system, see if you have
the logger program. This is working for me:


ServerName lua.roswell.area51
ServerAdmin sean@conman.org
DocumentRoot /home/spc/web/lua/htdocs
CustomLog "|/usr/bin/logger" combined
ErrorLog syslog:daemon
LogLevel debug

# ...


What gets logged is (and this is using a custom syslogd I wrote, so if the
format looks a bit odd to you, that's why):

/dev/log | logger | user notice | Apr 08 20:55:18 | ::1 - - [08/Apr/2010:20:55:18 -0400] "GET / HTTP/1.1" 304 - "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
/dev/log | logger | user notice | Apr 08 20:55:38 | ::1 - - [08/Apr/2010:20:55:38 -0400] "GET /quot.html HTTP/1.1" 404 207 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
/dev/log | logger | user notice | Apr 08 20:55:43 | ::1 - - [08/Apr/2010:20:55:43 -0400] "GET /quote.html HTTP/1.1" 200 395 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
/dev/log | logger | user notice | Apr 08 20:55:43 | ::1 - - [08/Apr/2010:20:55:43 -0400] "GET /quote.css HTTP/1.1" 200 177 "http://lua.roswell.area51/quote.html" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"

> Do you do any post/pre-processing for timestamp duplication? We want
> to try and have the logs on the syslog server look exactly like they
> did on the origin server (ie no syslog timestamps, no syslog
> facility/priority stuff and no "hostname").

I don't use syslog-ng, so I can't say for sure, but since the actual
Apache logline is logged, it looks like it would be easy to strip the syslog
specific stuff and use just the Apache generated data.

-spc


------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Centralized logging.

On Apr 8, 2010, at 1:52 PM, Ray Van Dolson wrote:

> The options I see so far are:

Anyone using mod_log_spread and a spread ring to distribute logs?

S.

--
Sander Temme
sctemme@apache.org
PGP FP: 51B4 8727 466A 0BC3 69F4 B7B8 B2BE BC40 1529 24AF




------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Centralized logging.

On 04/08/2010 10:52 PM, Ray Van Dolson wrote:
> So how are people out there in mailing list land handling centralized
> logging?
>

We are considering using Scribe[1] to send logs to a remote log server
and aggregate/elaborate them in real-time.

[1] http://en.wikipedia.org/wiki/Scribe_%28log_server%29

Regards,
Sandro

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Centralized logging.

On 09/04/2010 03:38, Sander Temme wrote:

>> The options I see so far are:
>
> Anyone using mod_log_spread and a spread ring to distribute logs?

I used that at my previous job which I left a few years ago. I'm
assuming they still have it in place. IIRC there were eight shared
hosting web servers using it to log to a central location. It just worked.

You can embed perl in the spread log daemon to process/modify log
entries before writing them out. I used this functionality to change the
log format to W3C Extended so the logs were in the same format as those
created by IIS on our Windows based web servers.

--
Mike Cardwell - Perl/Java/Web developer, Linux admin, Email admin
Read my tech Blog - https://secure.grepular.com/
Follow me on Twitter - http://twitter.com/mickeyc
Hire me - http://cardwellit.com/ http://uk.linkedin.com/in/mikecardwell

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Centralized logging.

On 08/04/2010 22:52, Ray Van Dolson wrote:
> So how are people out there in mailing list land handling centralized
> logging?
>
> I'd like to mirror my web farm's logfiles on a central server, but want
> to be sure to preserve the W3C-ness so it's easy to run awstats, etc
> against them from off this central box.
>
> The options I see so far are:
>
> - Piping to syslog via external process (perl script, etc).
> * Downside being that we end up with syslog timestamps
> - Pipe to something like netcat
> * Custom-ish / hacky
> - Fire off some sort of synchronization process from central box to
> pull in logs (rsync) periodically.
> * Asynchronous -- there's a somewhat large window of time where
> logs will be missing on the internal server.
> - Log directly to shared, remote file server (via NFS)
> * Log server network issues or technical problems could affect
> Apache servers
>
> How do you guys typically deal with this?
>
>

Hiya

Why not use splunk.

HTH

Brent Clark

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Centralized logging.

On Fri, Apr 09, 2010 at 10:44:54PM -0700, Brent Clark wrote:
> On 08/04/2010 22:52, Ray Van Dolson wrote:
> > So how are people out there in mailing list land handling centralized
> > logging?
> >
> > I'd like to mirror my web farm's logfiles on a central server, but want
> > to be sure to preserve the W3C-ness so it's easy to run awstats, etc
> > against them from off this central box.
> >
> > The options I see so far are:
> >
> > - Piping to syslog via external process (perl script, etc).
> > * Downside being that we end up with syslog timestamps
> > - Pipe to something like netcat
> > * Custom-ish / hacky
> > - Fire off some sort of synchronization process from central box to
> > pull in logs (rsync) periodically.
> > * Asynchronous -- there's a somewhat large window of time where
> > logs will be missing on the internal server.
> > - Log directly to shared, remote file server (via NFS)
> > * Log server network issues or technical problems could affect
> > Apache servers
> >
> > How do you guys typically deal with this?
> >
> >
>
> Hiya
>
> Why not use splunk.
>
> HTH
>
> Brent Clark

Way too expensive. I think this space could use some competition. :)

(Splunk is pretty awesome however though it wouldn't help me in this
particular case really...).

Ray

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Centralized logging.

On 4/10/10 10:04 AM, Ray Van Dolson wrote:
> Why not use splunk.

--
> Way too expensive. I think this space could use some competition. :)
>
> (Splunk is pretty awesome however though it wouldn't help me in this
> particular case really...).
>

I keep hearing about folks and Splunk, and would love to try it out.
But, I'm pulling 5-50G a day in log files.

I'd love some conversation here about better log reading techniques and
reporting packages.

--

Jason A. Nunnelley
+1 2562971652

http://www.google.com/profiles/imjasonn

[Member Tekany, LLC]


------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org