"grep" a file - output to browser not working
"grep" a file - output to browser not working
am 01.11.2007 00:45:45 von phillip.corchary
I cannot figure why this works fine at the command-line of the linux
server, but will not output anything to the browser, and no errors in
the error_log (syslog). The goal here is to have a page to enter a
search term and grep or zgrep pattern matches in the file (will be a
log file). Probably there is a better way to do this using php, but
this is what I was able to come up with using my limited php-
knowledge. Can anyone help me debug this?
Search Log
//$test=$_POST['sstr'];
$test="foo";
if (! empty($test))
{
$cmdstr = "grep -i $test ";
$fp = popen($cmdstr, 'r');
while ($buffer = fread($fp, 8096)) {
echo "
";
print($buffer);
echo "
";
flush(); }
pclose($fp);
}
else
{ echo "Please enter a search-term and try again!"; }
?>
Re: "grep" a file - output to browser not working
am 01.11.2007 04:52:35 von phillip.corchary
I should point out that I know this has something to do with the file-
size. With a small file it works OK. With a large file it fails to the
browser. In this case, a large file is up to 3.5GB of uncompressed
ASCII text. There may be up to 100 files to search - I can do that
part I think, once I get it to actually search the file! :-) Any help
GREATLY appreciated.
Probably I should mention that I am not a full-time developer ... I'm
quite good with shell scripts, REGEX and procedural languages ... I'm
a hacker at best with PHP :-) I'm more of a systems person.
On Oct 31, 5:45 pm, Phil wrote:
> I cannot figure why this works fine at the command-line of the linux
> server, but will not output anything to the browser, and no errors in
> the error_log (syslog). The goal here is to have a page to enter a
> search term and grep or zgrep pattern matches in the file (will be a
> log file). Probably there is a better way to do this using php, but
> this is what I was able to come up with using my limited php-
> knowledge. Can anyone help me debug this?
Re: "grep" a file - output to browser not working
am 01.11.2007 04:58:07 von Jerry Stuckle
Phil wrote:
> I should point out that I know this has something to do with the file-
> size. With a small file it works OK. With a large file it fails to the
> browser. In this case, a large file is up to 3.5GB of uncompressed
> ASCII text. There may be up to 100 files to search - I can do that
> part I think, once I get it to actually search the file! :-) Any help
> GREATLY appreciated.
>
> Probably I should mention that I am not a full-time developer ... I'm
> quite good with shell scripts, REGEX and procedural languages ... I'm
> a hacker at best with PHP :-) I'm more of a systems person.
>
> On Oct 31, 5:45 pm, Phil wrote:
>> I cannot figure why this works fine at the command-line of the linux
>> server, but will not output anything to the browser, and no errors in
>> the error_log (syslog). The goal here is to have a page to enter a
>> search term and grep or zgrep pattern matches in the file (will be a
>> log file). Probably there is a better way to do this using php, but
>> this is what I was able to come up with using my limited php-
>> knowledge. Can anyone help me debug this?
>
>
> I should point out that I know this has something to do with the file-
> size. With a small file it works OK. With a large file it fails to the
> browser. In this case, a large file is up to 3.5GB of uncompressed
> ASCII text. There may be up to 100 files to search - I can do that
> part I think, once I get it to actually search the file! :-) Any help
> GREATLY appreciated.
>
> Probably I should mention that I am not a full-time developer ... I'm
> quite good with shell scripts, REGEX and procedural languages ... I'm
> a hacker at best with PHP :-) I'm more of a systems person.
>
(Top posting fixed)
Well, some code would help. But are you possibly running out of memory
and/or execution time? Those are the two main things which cause
problems with large amounts of data but not small.
Anything in your PHP error log?
>
>
--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================
Re: "grep" a file - output to browser not working
am 01.11.2007 04:59:56 von Jerry Stuckle
Phil wrote:
P.S. Please don't top post. Thanks.
>
--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================
Re: "grep" a file - output to browser not working
am 01.11.2007 06:04:14 von phillip.corchary
On Oct 31, 9:58 pm, Jerry Stuckle wrote:
> Phil wrote:
> > I should point out that I know this has something to do with the file-
> > size. With a small file it works OK. With a large file it fails to the
> > browser. In this case, a large file is up to 3.5GB of uncompressed
> > ASCII text. There may be up to 100 files to search - I can do that
> > part I think, once I get it to actually search the file! :-) Any help
> > GREATLY appreciated.
>
> > Probably I should mention that I am not a full-time developer ... I'm
> > quite good with shell scripts, REGEX and procedural languages ... I'm
> > a hacker at best with PHP :-) I'm more of a systems person.
>
> > On Oct 31, 5:45 pm, Phil wrote:
> >> I cannot figure why this works fine at the command-line of the linux
> >> server, but will not output anything to the browser, and no errors in
> >> the error_log (syslog). The goal here is to have a page to enter a
> >> search term and grep or zgrep pattern matches in the file (will be a
> >> log file). Probably there is a better way to do this using php, but
> >> this is what I was able to come up with using my limited php-
> >> knowledge. Can anyone help me debug this?
>
> > I should point out that I know this has something to do with the file-
> > size. With a small file it works OK. With a large file it fails to the
> > browser. In this case, a large file is up to 3.5GB of uncompressed
> > ASCII text. There may be up to 100 files to search - I can do that
> > part I think, once I get it to actually search the file! :-) Any help
> > GREATLY appreciated.
> >
> > Probably I should mention that I am not a full-time developer ... I'm
> > quite good with shell scripts, REGEX and procedural languages ... I'm
> > a hacker at best with PHP :-) I'm more of a systems person.
> >
> (Top posting fixed)
>
> Well, some code would help. But are you possibly running out of memory
> and/or execution time? Those are the two main things which cause
> problems with large amounts of data but not small.
>
> Anything in your PHP error log?
there is squat from logs :(
This really seems to be an issue of file size.
Even this simple read/echo for a file of only 9200 lines it fails, say
nothing of my other
files of over 2million lines.
I have max_execution set to 3000 sec, so i don't think that is it. I
don't see too much in
php.ini that controls memory handling. "memory limit" is 32MB and post
limit is 8PM (default).
This is Apache/2.0.52 and PHP 4.3.9 by the way on a dual dual-core P4
3Ghz with 4GB RAM and CentOS 4.4 (32bit) with a 2.6.9-42.0.3.ELsmp
kernel
$file="/var/log/httpd/access_log";
$handle = @fopen("$file", "r");
if ($handle) {
echo "";
while (!feof($handle)) {
$buffer = fgets($handle, 2048);
print_r ($buffer);
ob_flush();
}
echo "
";
fclose($handle);
}
Sorry for Top Post. See GP for original code snip.
Re: "grep" a file - output to browser not working
am 01.11.2007 06:25:07 von Jackie Silva
top posting is fine. it's really a matter of individual preference. here are
some posting tips for you:
http://www.robertnyman.com/2007/03/29/posting-styles-and-why -top-posting-is-just-fine/
http://en.wikipedia.org/wiki/Posting_style
and the quote in your message is missing completely.
"Jerry Stuckle" wrote in message
news:2tqdnSFVda_L0rTanZ2dnUVZ_tDinZ2d@comcast.com...
> Phil wrote:
>
> P.S. Please don't top post. Thanks.
>>
>
>
> --
> ==================
> Remove the "x" from my email address
> Jerry Stuckle
> JDS Computer Training Corp.
> jstucklex@attglobal.net
> ==================
>
Re: "grep" a file - output to browser not working
am 01.11.2007 12:30:10 von Jerry Stuckle
Phil wrote:
> On Oct 31, 9:58 pm, Jerry Stuckle wrote:
>> Phil wrote:
>>> I should point out that I know this has something to do with the file-
>>> size. With a small file it works OK. With a large file it fails to the
>>> browser. In this case, a large file is up to 3.5GB of uncompressed
>>> ASCII text. There may be up to 100 files to search - I can do that
>>> part I think, once I get it to actually search the file! :-) Any help
>>> GREATLY appreciated.
>>> Probably I should mention that I am not a full-time developer ... I'm
>>> quite good with shell scripts, REGEX and procedural languages ... I'm
>>> a hacker at best with PHP :-) I'm more of a systems person.
>>> On Oct 31, 5:45 pm, Phil wrote:
>>>> I cannot figure why this works fine at the command-line of the linux
>>>> server, but will not output anything to the browser, and no errors in
>>>> the error_log (syslog). The goal here is to have a page to enter a
>>>> search term and grep or zgrep pattern matches in the file (will be a
>>>> log file). Probably there is a better way to do this using php, but
>>>> this is what I was able to come up with using my limited php-
>>>> knowledge. Can anyone help me debug this?
>> > I should point out that I know this has something to do with the file-
>> > size. With a small file it works OK. With a large file it fails to the
>> > browser. In this case, a large file is up to 3.5GB of uncompressed
>> > ASCII text. There may be up to 100 files to search - I can do that
>> > part I think, once I get it to actually search the file! :-) Any help
>> > GREATLY appreciated.
>> >
>> > Probably I should mention that I am not a full-time developer ... I'm
>> > quite good with shell scripts, REGEX and procedural languages ... I'm
>> > a hacker at best with PHP :-) I'm more of a systems person.
>> >
>> (Top posting fixed)
>>
>> Well, some code would help. But are you possibly running out of memory
>> and/or execution time? Those are the two main things which cause
>> problems with large amounts of data but not small.
>>
>> Anything in your PHP error log?
>
> there is squat from logs :(
>
> This really seems to be an issue of file size.
> Even this simple read/echo for a file of only 9200 lines it fails, say
> nothing of my other
> files of over 2million lines.
>
> I have max_execution set to 3000 sec, so i don't think that is it. I
> don't see too much in
> php.ini that controls memory handling. "memory limit" is 32MB and post
> limit is 8PM (default).
>
> This is Apache/2.0.52 and PHP 4.3.9 by the way on a dual dual-core P4
> 3Ghz with 4GB RAM and CentOS 4.4 (32bit) with a 2.6.9-42.0.3.ELsmp
> kernel
>
> $file="/var/log/httpd/access_log";
> $handle = @fopen("$file", "r");
> if ($handle) {
> echo "";
> while (!feof($handle)) {
> $buffer = fgets($handle, 2048);
> print_r ($buffer);
> ob_flush();
> }
> echo "
";
> fclose($handle);
> }
>
> Sorry for Top Post. See GP for original code snip.
>
>
OK, in your php.ini file, ensure you have
error_reporting=E_ALL
display_errors=On
for testing purposes.
If this is a production system, I don't recommend having
display_errors=On. Rather, I use log_errors=On and set a log file.
P.S. Don't mind the trolls here. Someone is starved for attention.
--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================
Re: "grep" a file - output to browser not working
am 01.11.2007 13:39:55 von Captain Paralytic
On 1 Nov, 05:25, "Jackie Silva" wrote:
> top posting is fine. it's really a matter of individual preference. here are
> some posting tips for you:
>
> http://www.robertnyman.com/2007/03/29/posting-styles-and-why -top-post...http://en.wikipedia.org/wiki/Posting_style
>
> and the quote in your message is missing completely.
>
> "Jerry Stuckle" wrote in message
>
> news:2tqdnSFVda_L0rTanZ2dnUVZ_tDinZ2d@comcast.com...
>
>
>
> > Phil wrote:
>
> > P.S. Please don't top post. Thanks.
>
> > --
> > ==================
> > Remove the "x" from my email address
> > Jerry Stuckle
> > JDS Computer Training Corp.
> > jstuck...@attglobal.net
> > ==================- Hide quoted text -
>
> - Show quoted text -
I have a posting tip for you.
Don't bother!
Re: "grep" a file - output to browser not working
am 01.11.2007 14:29:20 von phillip.corchary
On Nov 1, 5:30 am, Jerry Stuckle wrote:
> Phil wrote:
> > On Oct 31, 9:58 pm, Jerry Stuckle wrote:
> >> Phil wrote:
> >>> I should point out that I know this has something to do with the file-
> >>> size. With a small file it works OK. With a large file it fails to the
> >>> browser. In this case, a large file is up to 3.5GB of uncompressed
> >>> ASCII text. There may be up to 100 files to search - I can do that
> >>> part I think, once I get it to actually search the file! :-) Any help
> >>> GREATLY appreciated.
> >>> Probably I should mention that I am not a full-time developer ... I'm
> >>> quite good with shell scripts, REGEX and procedural languages ... I'm
> >>> a hacker at best with PHP :-) I'm more of a systems person.
> >>> On Oct 31, 5:45 pm, Phil wrote:
> >>>> I cannot figure why this works fine at the command-line of the linux
> >>>> server, but will not output anything to the browser, and no errors in
> >>>> the error_log (syslog). The goal here is to have a page to enter a
> >>>> search term and grep or zgrep pattern matches in the file (will be a
> >>>> log file). Probably there is a better way to do this using php, but
> >>>> this is what I was able to come up with using my limited php-
> >>>> knowledge. Can anyone help me debug this?
> >> > I should point out that I know this has something to do with the file-
> >> > size. With a small file it works OK. With a large file it fails to the
> >> > browser. In this case, a large file is up to 3.5GB of uncompressed
> >> > ASCII text. There may be up to 100 files to search - I can do that
> >> > part I think, once I get it to actually search the file! :-) Any help
> >> > GREATLY appreciated.
>
> >> > Probably I should mention that I am not a full-time developer ... I'm
> >> > quite good with shell scripts, REGEX and procedural languages ... I'm
> >> > a hacker at best with PHP :-) I'm more of a systems person.
>
> >> (Top posting fixed)
>
> >> Well, some code would help. But are you possibly running out of memory
> >> and/or execution time? Those are the two main things which cause
> >> problems with large amounts of data but not small.
>
> >> Anything in your PHP error log?
>
> > there is squat from logs :(
>
> > This really seems to be an issue of file size.
> > Even this simple read/echo for a file of only 9200 lines it fails, say
> > nothing of my other
> > files of over 2million lines.
>
> > I have max_execution set to 3000 sec, so i don't think that is it. I
> > don't see too much in
> > php.ini that controls memory handling. "memory limit" is 32MB and post
> > limit is 8PM (default).
>
> > This is Apache/2.0.52 and PHP 4.3.9 by the way on a dual dual-core P4
> > 3Ghz with 4GB RAM and CentOS 4.4 (32bit) with a 2.6.9-42.0.3.ELsmp
> > kernel
>
> > $file="/var/log/httpd/access_log";
> > $handle = @fopen("$file", "r");
> > if ($handle) {
> > echo "";
> > while (!feof($handle)) {
> > $buffer = fgets($handle, 2048);
> > print_r ($buffer);
> > ob_flush();
> > }
> > echo "
";
> > fclose($handle);
> > }
>
> > Sorry for Top Post. See GP for original code snip.
>
> OK, in your php.ini file, ensure you have
>
> error_reporting=E_ALL
> display_errors=On
>
> for testing purposes.
>
> If this is a production system, I don't recommend having
> display_errors=On. Rather, I use log_errors=On and set a log file.
>
> P.S. Don't mind the trolls here. Someone is starved for attention.
I already have those options set.
Actually, I partly found the answer, and it was a beginners
mistake ... file permissions, as the logs are root-owned and apache/
php runs as apache - a little 'is_readable' helped me out there. Now I
have to decide if i want to have apache run as root, or modify system
log files. It is 'production' system, but it's an enterprise internal
only single-purpose and with Change Control I can do anything I need
for this.
I'd still be interested in quid-pro-quo on using external grep versus
some internal preg_grep (and how to make that one work ... I've been
stumped so far).
Thanks! phil
Re: "grep" a file - output to browser not working
am 01.11.2007 16:28:13 von luiheidsgoeroe
On Thu, 01 Nov 2007 06:25:07 +0100, Jackie Silva
wrote:
> top posting is fine. it's really a matter of individual preference. here
> are
> some posting tips for you:
>
> http://www.robertnyman.com/2007/03/29/posting-styles-and-why -top-posting-is-just-fine/
Yes, topposting is fine, and actually my preferred method, for email
(sometimes interleaved, though rarely). This however is usenet, not
personal email.
--
Rik Wasmus
Re: "grep" a file - output to browser not working
am 01.11.2007 16:38:26 von Ivan Marsh
On Thu, 01 Nov 2007 00:25:07 -0500, Jackie Silva wrote:
> top posting is fine. it's really a matter of individual preference.
No... it's not.
If you top post you can assume your query will end up in a thread about
top posting rather than being answered... just as this thread has.
--
I told you this was going to happen.
Re: "grep" a file - output to browser not working
am 01.11.2007 21:02:25 von Jerry Stuckle
Phil wrote:
> On Nov 1, 5:30 am, Jerry Stuckle wrote:
>> Phil wrote:
>>> On Oct 31, 9:58 pm, Jerry Stuckle wrote:
>>>> Phil wrote:
>>>>> I should point out that I know this has something to do with the file-
>>>>> size. With a small file it works OK. With a large file it fails to the
>>>>> browser. In this case, a large file is up to 3.5GB of uncompressed
>>>>> ASCII text. There may be up to 100 files to search - I can do that
>>>>> part I think, once I get it to actually search the file! :-) Any help
>>>>> GREATLY appreciated.
>>>>> Probably I should mention that I am not a full-time developer ... I'm
>>>>> quite good with shell scripts, REGEX and procedural languages ... I'm
>>>>> a hacker at best with PHP :-) I'm more of a systems person.
>>>>> On Oct 31, 5:45 pm, Phil wrote:
>>>>>> I cannot figure why this works fine at the command-line of the linux
>>>>>> server, but will not output anything to the browser, and no errors in
>>>>>> the error_log (syslog). The goal here is to have a page to enter a
>>>>>> search term and grep or zgrep pattern matches in the file (will be a
>>>>>> log file). Probably there is a better way to do this using php, but
>>>>>> this is what I was able to come up with using my limited php-
>>>>>> knowledge. Can anyone help me debug this?
>>>> > I should point out that I know this has something to do with the file-
>>>> > size. With a small file it works OK. With a large file it fails to the
>>>> > browser. In this case, a large file is up to 3.5GB of uncompressed
>>>> > ASCII text. There may be up to 100 files to search - I can do that
>>>> > part I think, once I get it to actually search the file! :-) Any help
>>>> > GREATLY appreciated.
>>>> > Probably I should mention that I am not a full-time developer ... I'm
>>>> > quite good with shell scripts, REGEX and procedural languages ... I'm
>>>> > a hacker at best with PHP :-) I'm more of a systems person.
>>>> (Top posting fixed)
>>>> Well, some code would help. But are you possibly running out of memory
>>>> and/or execution time? Those are the two main things which cause
>>>> problems with large amounts of data but not small.
>>>> Anything in your PHP error log?
>>> there is squat from logs :(
>>> This really seems to be an issue of file size.
>>> Even this simple read/echo for a file of only 9200 lines it fails, say
>>> nothing of my other
>>> files of over 2million lines.
>>> I have max_execution set to 3000 sec, so i don't think that is it. I
>>> don't see too much in
>>> php.ini that controls memory handling. "memory limit" is 32MB and post
>>> limit is 8PM (default).
>>> This is Apache/2.0.52 and PHP 4.3.9 by the way on a dual dual-core P4
>>> 3Ghz with 4GB RAM and CentOS 4.4 (32bit) with a 2.6.9-42.0.3.ELsmp
>>> kernel
>>> $file="/var/log/httpd/access_log";
>>> $handle = @fopen("$file", "r");
>>> if ($handle) {
>>> echo "";
>>> while (!feof($handle)) {
>>> $buffer = fgets($handle, 2048);
>>> print_r ($buffer);
>>> ob_flush();
>>> }
>>> echo "
";
>>> fclose($handle);
>>> }
>>> Sorry for Top Post. See GP for original code snip.
>> OK, in your php.ini file, ensure you have
>>
>> error_reporting=E_ALL
>> display_errors=On
>>
>> for testing purposes.
>>
>> If this is a production system, I don't recommend having
>> display_errors=On. Rather, I use log_errors=On and set a log file.
>>
>> P.S. Don't mind the trolls here. Someone is starved for attention.
>
> I already have those options set.
> Actually, I partly found the answer, and it was a beginners
> mistake ... file permissions, as the logs are root-owned and apache/
> php runs as apache - a little 'is_readable' helped me out there. Now I
> have to decide if i want to have apache run as root, or modify system
> log files. It is 'production' system, but it's an enterprise internal
> only single-purpose and with Change Control I can do anything I need
> for this.
>
Ah, didn't think about the file not being readable. :-)
You NEVER want to run Apache as root. If someone hacks your site, they
have access to EVERYTHING.
> I'd still be interested in quid-pro-quo on using external grep versus
> some internal preg_grep (and how to make that one work ... I've been
> stumped so far).
>
> Thanks! phil
>
>
>
Depends on what you're going to do. If all you're doing is searching an
array, just use preg_grep(). Or, you can read a small file into an
array with file() and use preg_grep().
If you need to do more than one file or a large file, then an external
grep is probably better.
But if I'm searching, I generally have what I'm looking for in a database.
--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================
Re: "grep" a file - output to browser not working
am 01.11.2007 22:26:49 von phillip.corchary
On Nov 1, 2:02 pm, Jerry Stuckle wrote:
> Phil wrote:
> > On Nov 1, 5:30 am, Jerry Stuckle wrote:
> >> Phil wrote:
> >>> On Oct 31, 9:58 pm, Jerry Stuckle wrote:
> >>>> Phil wrote:
> >>>>> I should point out that I know this has something to do with the file-
> >>>>> size. With a small file it works OK. With a large file it fails to the
> >>>>> browser. In this case, a large file is up to 3.5GB of uncompressed
> >>>>> ASCII text. There may be up to 100 files to search - I can do that
> >>>>> part I think, once I get it to actually search the file! :-) Any help
> >>>>> GREATLY appreciated.
> >>>>> Probably I should mention that I am not a full-time developer ... I'm
> >>>>> quite good with shell scripts, REGEX and procedural languages ... I'm
> >>>>> a hacker at best with PHP :-) I'm more of a systems person.
> >>>>> On Oct 31, 5:45 pm, Phil wrote:
> >>>>>> I cannot figure why this works fine at the command-line of the linux
> >>>>>> server, but will not output anything to the browser, and no errors in
> >>>>>> the error_log (syslog). The goal here is to have a page to enter a
> >>>>>> search term and grep or zgrep pattern matches in the file (will be a
> >>>>>> log file). Probably there is a better way to do this using php, but
> >>>>>> this is what I was able to come up with using my limited php-
> >>>>>> knowledge. Can anyone help me debug this?
> >>>> > I should point out that I know this has something to do with the file-
> >>>> > size. With a small file it works OK. With a large file it fails to the
> >>>> > browser. In this case, a large file is up to 3.5GB of uncompressed
> >>>> > ASCII text. There may be up to 100 files to search - I can do that
> >>>> > part I think, once I get it to actually search the file! :-) Any help
> >>>> > GREATLY appreciated.
> >>>> > Probably I should mention that I am not a full-time developer ... I'm
> >>>> > quite good with shell scripts, REGEX and procedural languages ... I'm
> >>>> > a hacker at best with PHP :-) I'm more of a systems person.
> >>>> (Top posting fixed)
> >>>> Well, some code would help. But are you possibly running out of memory
> >>>> and/or execution time? Those are the two main things which cause
> >>>> problems with large amounts of data but not small.
> >>>> Anything in your PHP error log?
> >>> there is squat from logs :(
> >>> This really seems to be an issue of file size.
> >>> Even this simple read/echo for a file of only 9200 lines it fails, say
> >>> nothing of my other
> >>> files of over 2million lines.
> >>> I have max_execution set to 3000 sec, so i don't think that is it. I
> >>> don't see too much in
> >>> php.ini that controls memory handling. "memory limit" is 32MB and post
> >>> limit is 8PM (default).
> >>> This is Apache/2.0.52 and PHP 4.3.9 by the way on a dual dual-core P4
> >>> 3Ghz with 4GB RAM and CentOS 4.4 (32bit) with a 2.6.9-42.0.3.ELsmp
> >>> kernel
> >>> $file="/var/log/httpd/access_log";
> >>> $handle = @fopen("$file", "r");
> >>> if ($handle) {
> >>> echo "";
> >>> while (!feof($handle)) {
> >>> $buffer = fgets($handle, 2048);
> >>> print_r ($buffer);
> >>> ob_flush();
> >>> }
> >>> echo "
";
> >>> fclose($handle);
> >>> }
> >>> Sorry for Top Post. See GP for original code snip.
> >> OK, in your php.ini file, ensure you have
>
> >> error_reporting=E_ALL
> >> display_errors=On
>
> >> for testing purposes.
>
> >> If this is a production system, I don't recommend having
> >> display_errors=On. Rather, I use log_errors=On and set a log file.
>
> >> P.S. Don't mind the trolls here. Someone is starved for attention.
>
> > I already have those options set.
> > Actually, I partly found the answer, and it was a beginners
> > mistake ... file permissions, as the logs are root-owned and apache/
> > php runs as apache - a little 'is_readable' helped me out there. Now I
> > have to decide if i want to have apache run as root, or modify system
> > log files. It is 'production' system, but it's an enterprise internal
> > only single-purpose and with Change Control I can do anything I need
> > for this.
>
> Ah, didn't think about the file not being readable. :-)
>
> You NEVER want to run Apache as root. If someone hacks your site, they
> have access to EVERYTHING.
>
> > I'd still be interested in quid-pro-quo on using external grep versus
> > some internal preg_grep (and how to make that one work ... I've been
> > stumped so far).
>
> > Thanks! phil
>
> Depends on what you're going to do. If all you're doing is searching an
> array, just use preg_grep(). Or, you can read a small file into an
> array with file() and use preg_grep().
>
> If you need to do more than one file or a large file, then an external
> grep is probably better.
>
> But if I'm searching, I generally have what I'm looking for in a database.
Well, what I'm searching is sendmail logs on a central syslog for a
cluster of sendmail servers. I think someday I should look into a
syslog->DB gateway, but currently what I have is 4 months (rotated)
archived syslog of about 3.5 million lines per file. Instead of have a
CustSvc guy place an Ops request to have some one grep logs, I want to
let them do it themselves with a web page.
Sounds like you think that the php calling grep (or zgrep in this
case) is the best solution ... and given what I've seen of php
performance supporting 10x and 100x thousands of connections, I think
it's agree. This is pretty low volume, but I trust *nix to balance the
loads with grep commands running with a pipe back to php more than I
think i trust php to do it all internally.
The server is internal only, but I understand your comment about
apache as root. What I did was changed the maillog files to 755, and
added 'apache' user to the 'root' group in *nix. Works good.
Thanks for everything. Here is my current code, lemme know if you see
anything I've done grossly wrong...
Search Mail Logs
// since we use POST method, get global value into 'test' string
$test=$_POST['sstr'];
// check if submit button pressed and some search value entered
if (isset($_POST['submit']) && ! empty($test) )
{
// set file and path to search
$file="/var/log/" . $_POST[d_file];
// check the file is readable
if (! is_readable ($file)) {
print "ERROR accessing file $file, please contact the
admin.
"; exit;}
// build the search command
$cmdstr="zgrep -i $test $file";
echo "Return To The Search
Form
\n";
echo "Search results for $test in $file ...
";
flush();
// search the file, display the result
$fp = popen($cmdstr . ' 2>&1', 'r'); // open proc pointer
echo "";
while ($buffer = fgets($fp, 4096))
{
echo("$buffer");
}
echo "
";
echo "END
\n";
pclose($fp);
flush();
exit;
}
?>
Mail Log Search
Re: "grep" a file - output to browser not working
am 02.11.2007 03:02:10 von Jerry Stuckle
Phil wrote:
> On Nov 1, 2:02 pm, Jerry Stuckle wrote:
>> Phil wrote:
>>> On Nov 1, 5:30 am, Jerry Stuckle wrote:
>>>> Phil wrote:
>>>>> On Oct 31, 9:58 pm, Jerry Stuckle wrote:
>>>>>> Phil wrote:
>>>>>>> I should point out that I know this has something to do with the file-
>>>>>>> size. With a small file it works OK. With a large file it fails to the
>>>>>>> browser. In this case, a large file is up to 3.5GB of uncompressed
>>>>>>> ASCII text. There may be up to 100 files to search - I can do that
>>>>>>> part I think, once I get it to actually search the file! :-) Any help
>>>>>>> GREATLY appreciated.
>>>>>>> Probably I should mention that I am not a full-time developer ... I'm
>>>>>>> quite good with shell scripts, REGEX and procedural languages ... I'm
>>>>>>> a hacker at best with PHP :-) I'm more of a systems person.
>>>>>>> On Oct 31, 5:45 pm, Phil wrote:
>>>>>>>> I cannot figure why this works fine at the command-line of the linux
>>>>>>>> server, but will not output anything to the browser, and no errors in
>>>>>>>> the error_log (syslog). The goal here is to have a page to enter a
>>>>>>>> search term and grep or zgrep pattern matches in the file (will be a
>>>>>>>> log file). Probably there is a better way to do this using php, but
>>>>>>>> this is what I was able to come up with using my limited php-
>>>>>>>> knowledge. Can anyone help me debug this?
>>>>>> > I should point out that I know this has something to do with the file-
>>>>>> > size. With a small file it works OK. With a large file it fails to the
>>>>>> > browser. In this case, a large file is up to 3.5GB of uncompressed
>>>>>> > ASCII text. There may be up to 100 files to search - I can do that
>>>>>> > part I think, once I get it to actually search the file! :-) Any help
>>>>>> > GREATLY appreciated.
>>>>>> > Probably I should mention that I am not a full-time developer ... I'm
>>>>>> > quite good with shell scripts, REGEX and procedural languages ... I'm
>>>>>> > a hacker at best with PHP :-) I'm more of a systems person.
>>>>>> (Top posting fixed)
>>>>>> Well, some code would help. But are you possibly running out of memory
>>>>>> and/or execution time? Those are the two main things which cause
>>>>>> problems with large amounts of data but not small.
>>>>>> Anything in your PHP error log?
>>>>> there is squat from logs :(
>>>>> This really seems to be an issue of file size.
>>>>> Even this simple read/echo for a file of only 9200 lines it fails, say
>>>>> nothing of my other
>>>>> files of over 2million lines.
>>>>> I have max_execution set to 3000 sec, so i don't think that is it. I
>>>>> don't see too much in
>>>>> php.ini that controls memory handling. "memory limit" is 32MB and post
>>>>> limit is 8PM (default).
>>>>> This is Apache/2.0.52 and PHP 4.3.9 by the way on a dual dual-core P4
>>>>> 3Ghz with 4GB RAM and CentOS 4.4 (32bit) with a 2.6.9-42.0.3.ELsmp
>>>>> kernel
>>>>> $file="/var/log/httpd/access_log";
>>>>> $handle = @fopen("$file", "r");
>>>>> if ($handle) {
>>>>> echo "";
>>>>> while (!feof($handle)) {
>>>>> $buffer = fgets($handle, 2048);
>>>>> print_r ($buffer);
>>>>> ob_flush();
>>>>> }
>>>>> echo "
";
>>>>> fclose($handle);
>>>>> }
>>>>> Sorry for Top Post. See GP for original code snip.
>>>> OK, in your php.ini file, ensure you have
>>>> error_reporting=E_ALL
>>>> display_errors=On
>>>> for testing purposes.
>>>> If this is a production system, I don't recommend having
>>>> display_errors=On. Rather, I use log_errors=On and set a log file.
>>>> P.S. Don't mind the trolls here. Someone is starved for attention.
>>> I already have those options set.
>>> Actually, I partly found the answer, and it was a beginners
>>> mistake ... file permissions, as the logs are root-owned and apache/
>>> php runs as apache - a little 'is_readable' helped me out there. Now I
>>> have to decide if i want to have apache run as root, or modify system
>>> log files. It is 'production' system, but it's an enterprise internal
>>> only single-purpose and with Change Control I can do anything I need
>>> for this.
>> Ah, didn't think about the file not being readable. :-)
>>
>> You NEVER want to run Apache as root. If someone hacks your site, they
>> have access to EVERYTHING.
>>
>>> I'd still be interested in quid-pro-quo on using external grep versus
>>> some internal preg_grep (and how to make that one work ... I've been
>>> stumped so far).
>>> Thanks! phil
>> Depends on what you're going to do. If all you're doing is searching an
>> array, just use preg_grep(). Or, you can read a small file into an
>> array with file() and use preg_grep().
>>
>> If you need to do more than one file or a large file, then an external
>> grep is probably better.
>>
>> But if I'm searching, I generally have what I'm looking for in a database.
>
> Well, what I'm searching is sendmail logs on a central syslog for a
> cluster of sendmail servers. I think someday I should look into a
> syslog->DB gateway, but currently what I have is 4 months (rotated)
> archived syslog of about 3.5 million lines per file. Instead of have a
> CustSvc guy place an Ops request to have some one grep logs, I want to
> let them do it themselves with a web page.
>
> Sounds like you think that the php calling grep (or zgrep in this
> case) is the best solution ... and given what I've seen of php
> performance supporting 10x and 100x thousands of connections, I think
> it's agree. This is pretty low volume, but I trust *nix to balance the
> loads with grep commands running with a pipe back to php more than I
> think i trust php to do it all internally.
>
No, I think a database is the best solution. Once a day I'd parse the
logs and add them to the database. And of older entries need to be
deleted, no problem.
3.5M lines is nothing to a good RDB.
> The server is internal only, but I understand your comment about
> apache as root. What I did was changed the maillog files to 755, and
> added 'apache' user to the 'root' group in *nix. Works good.
>
Remember that the majority of hacks come from INSIDE - not outside.
Disgruntled employees, etc.
> Thanks for everything. Here is my current code, lemme know if you see
> anything I've done grossly wrong...
>
>
>
>
> Search Mail Logs
>
>
>
>
> // since we use POST method, get global value into 'test' string
> $test=$_POST['sstr'];
>
> // check if submit button pressed and some search value entered
> if (isset($_POST['submit']) && ! empty($test) )
> {
> // set file and path to search
> $file="/var/log/" . $_POST[d_file];
>
You didn't verify the file name. *NEVER* trust user input. It could be
something like "../../var/log/syslog".
> // check the file is readable
> if (! is_readable ($file)) {
> print "ERROR accessing file $file, please contact the
> admin.
"; exit;}
>
> // build the search command
> $cmdstr="zgrep -i $test $file";
> echo "Return To The Search
> Form
\n";
> echo "Search results for $test in $file ...
";
> flush();
>
> // search the file, display the result
> $fp = popen($cmdstr . ' 2>&1', 'r'); // open proc pointer
> echo "";
> while ($buffer = fgets($fp, 4096))
> {
> echo("$buffer");
> }
> echo "
";
> echo "END
\n";
> pclose($fp);
> flush();
> exit;
> }
>
> ?>
>
I don't see anything else seriously wrong. But as I indicated before,
it isn't the way I'd do it.
>
> Mail Log Search
>
>
>
>
>
>
>
>
--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================