Buffered Logging?

--001636426f43572c3b0470950267
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hey PHPers,

We've been doing sampled logging to the database in our application for
awhile, and now I'm hoping eventually to blow that out to a larger scale.
I'm worried about the performance implications of logging to our database on
every single page load, though, so I was wondering if anyone's found a
solution that does buffered logging. Essentially it would log to memory
(memcached/apc/etc), and then periodically dump to a database in a
structured format, preferrably user-defined. It's not essential that we get
every signle hit, so I'd be fine if there was some data loss on a restart.
I started writing my own solution, and then thought I'd ask the list to see
if anyone has any experience with other tools that do this.

My Google searches around "buffered logging" have mainly found error logging
packages, like PEAR's Log package, or log4php. Those all seem to write to
only one particular stream at a time, with no real support for buffering it
in memory and then moving it to database.

Thanks for any help!

Waynn

--001636426f43572c3b0470950267--

Re: Buffered Logging?

--00163691fedc6aeb66047095fd5d
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

You don't mention what database you're using, but mySQL supports memory
based tables. You can use this to insert your single page loads and then
have a job that periodically inserts in bulk.

Memory based tables:
http://dev.mysql.com/doc/refman/5.0/en/memory-storage-engine .html


Jerry Wilborn
jerrywilborn@gmail.com


On Fri, Aug 7, 2009 at 5:46 PM, Waynn Lue wrote:

> Hey PHPers,
>
> We've been doing sampled logging to the database in our application for
> awhile, and now I'm hoping eventually to blow that out to a larger scale.
> I'm worried about the performance implications of logging to our database
> on
> every single page load, though, so I was wondering if anyone's found a
> solution that does buffered logging. Essentially it would log to memory
> (memcached/apc/etc), and then periodically dump to a database in a
> structured format, preferrably user-defined. It's not essential that we
> get
> every signle hit, so I'd be fine if there was some data loss on a restart.
> I started writing my own solution, and then thought I'd ask the list to see
> if anyone has any experience with other tools that do this.
>
> My Google searches around "buffered logging" have mainly found error
> logging
> packages, like PEAR's Log package, or log4php. Those all seem to write to
> only one particular stream at a time, with no real support for buffering it
> in memory and then moving it to database.
>
> Thanks for any help!
>
> Waynn
>

--00163691fedc6aeb66047095fd5d--

Re: Buffered Logging?

On Sat, 08 Aug 2009 01:46:38 +0300, Waynn Lue wrote:

> Hey PHPers,
>
> We've been doing sampled logging to the database in our application for
> awhile, and now I'm hoping eventually to blow that out to a larger scale.
> I'm worried about the performance implications of logging to our
> database on
> ...
>

If you are using mysql and MyISAM tables, you can try using "insert
DELAYED " method.

http://dev.mysql.com/doc/refman/5.1/en/insert-delayed.html

This will bulk all your inserts for writes.


--

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: Buffered Logging?

--001636427084cd44090470973a60
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

>
> Hey PHPers,
>>
>> We've been doing sampled logging to the database in our application for
>> awhile, and now I'm hoping eventually to blow that out to a larger scale.
>> I'm worried about the performance implications of logging to our database
>> on
>> ...
>>
>>
> If you are using mysql and MyISAM tables, you can try using "insert DELAYED
> " method.
>
> http://dev.mysql.com/doc/refman/5.1/en/insert-delayed.html
>
> This will bulk all your inserts for writes.
>

Thanks for the suggestions! Those both look great for what I was going to
do. One other thought I had after reading those suggestions, if we're doing
web server logging, we can also parse the logs using webalizer or awstats.
I know apache provides file size and the URL that's being hit, but what if I
want to do custom referral tracking? We append ref=foo to our links to
track where people are coming from, should I look at building my own
solution for that, or do existing tools like awstats suffice for that as
well?

--001636427084cd44090470973a60--

Re: Re: Buffered Logging?

I did some very complete logging for two major german companies on their
intranet pages. an application with something like 23000 registered users
and more then 50000 hits a day. I did none of any kind of buffering, just
raw table inserts. it never gave any problem on performance HOWEVER we did a
DAILY backup AND reset of the logging tables. so the tables where never much
bigger then 50000 records. so if u use mySQL and the buffer technices
mentioned earlier I would go with it because of the benefit not to maintain
another different tool.

ralph


"Waynn Lue" wrote in message
news:d29bea5e0908071825k73920480g598af559383ffa99@mail.gmail .com...
> >
> > Hey PHPers,
> >>
> >> We've been doing sampled logging to the database in our application for
> >> awhile, and now I'm hoping eventually to blow that out to a larger
scale.
> >> I'm worried about the performance implications of logging to our
database
> >> on
> >> ...
> >>
> >>
> > If you are using mysql and MyISAM tables, you can try using "insert
DELAYED
> > " method.
> >
> > http://dev.mysql.com/doc/refman/5.1/en/insert-delayed.html
> >
> > This will bulk all your inserts for writes.
> >
>
> Thanks for the suggestions! Those both look great for what I was going to
> do. One other thought I had after reading those suggestions, if we're
doing
> web server logging, we can also parse the logs using webalizer or awstats.
> I know apache provides file size and the URL that's being hit, but what if
I
> want to do custom referral tracking? We append ref=foo to our links to
> track where people are coming from, should I look at building my own
> solution for that, or do existing tools like awstats suffice for that as
> well?
>



--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: Buffered Logging?

On Aug 7, 2009, at 9:25 PM, Waynn Lue wrote:

>>
>> Hey PHPers,
>>>
>>> We've been doing sampled logging to the database in our
>>> application for
>>> awhile, and now I'm hoping eventually to blow that out to a larger
>>> scale.
>>> I'm worried about the performance implications of logging to our
>>> database
>>> on
>>> ...
>>>
>>>
>> If you are using mysql and MyISAM tables, you can try using "insert
>> DELAYED
>> " method.
>>
>> http://dev.mysql.com/doc/refman/5.1/en/insert-delayed.html
>>
>> This will bulk all your inserts for writes.
>>
>
> Thanks for the suggestions! Those both look great for what I was
> going to
> do. One other thought I had after reading those suggestions, if
> we're doing
> web server logging, we can also parse the logs using webalizer or
> awstats.
> I know apache provides file size and the URL that's being hit, but
> what if I
> want to do custom referral tracking? We append ref=foo to our links
> to
> track where people are coming from, should I look at building my own
> solution for that, or do existing tools like awstats suffice for
> that as
> well?


You may want to investigate facebook's scribe as a logging mechanism.
It's highly scalable

http://www.facebook.com/note.php?note_id=32008268919

Bastien

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: Buffered Logging?

--0016364ef507c004fd0470d700d9
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Thanks for all the help! I'm going to try just writing to a database table
first with INSERT DELAYED, and if there ends up being a performance hit,
I'll use a MEMORY table that gets archived off every 5 minutes or so.

Waynn

--0016364ef507c004fd0470d700d9--