Sendmail with high load averages.

We recently have been having issues with sendmail. For some reason our
RedHat Linux machine has been spiking high load averages. The load
average can get as high as 40-45 but usually around 10-15. Not sure
where to start troubleshooting. The top command does not shed any
light on this issue. I have also looked at vmstat and iostat but
cannot tell what would be causing the spikes. Any suggestions/
questions would be greatly appreciated. Thanks!

Re: Sendmail with high load averages.

On Mon, 12 Nov 2007, dlhirsch wrote:

>
> We recently have been having issues with sendmail. For some reason our
> RedHat Linux machine has been spiking high load averages. The load
> average can get as high as 40-45 but usually around 10-15. Not sure
> where to start troubleshooting. The top command does not shed any
> light on this issue. I have also looked at vmstat and iostat but
> cannot tell what would be causing the spikes. Any suggestions/
> questions would be greatly appreciated. Thanks!


This is better of on a redhat list, since they like to
(butcher|hack_the_crap_out_of|customize_beyond_belief) most things that go
into their distro.

Not to mention you've not told us any versions, how many processes etc..


--
Cheers
Res

Re: Sendmail with high load averages.

On Nov 12, 4:21 pm, Res wrote:
> On Mon, 12 Nov 2007, dlhirsch wrote:
>
> > We recently have been having issues with sendmail. For some reason our
> > RedHat Linux machine has been spiking high load averages. The load
> > average can get as high as 40-45 but usually around 10-15. Not sure
> > where to start troubleshooting. The top command does not shed any
> > light on this issue. I have also looked at vmstat and iostat but
> > cannot tell what would be causing the spikes. Any suggestions/
> > questions would be greatly appreciated. Thanks!
>
> This is better of on a redhat list, since they like to
> (butcher|hack_the_crap_out_of|customize_beyond_belief) most things that go
> into their distro.
>
> Not to mention you've not told us any versions, how many processes etc..
>
> --
> Cheers
> Res

I could have guessed what information would have been helpful but it
would have been only a guess. Asking for specifics like version
numbers is still kind of vague. The version of sendmail is
"sendmail-8.9.3-20" on RedHat 7.3. As far as number of processes, at
what time do you want to know the number of processes? Or do you want
to know what those processes are and how many of each? Can you be more
specific? - Thanks in advance!

Re: Sendmail with high load averages.

On Mon, 12 Nov 2007, dlhirsch wrote:

>> Not to mention you've not told us any versions, how many processes etc..
>
> I could have guessed what information would have been helpful but it
> would have been only a guess. Asking for specifics like version
> numbers is still kind of vague. The version of sendmail is


> "sendmail-8.9.3-20" on RedHat 7.3. As far as number of processes, at

WTF!@!!@!
Ok, the first thing I am going to tell you to do is, copy all your
sendmail config files (especially a .mc file if you have one)....

rpm -qa | grep sendmail
rpm -e every-package it says

go to www.sendmail.org and get 8.14.2, and install it, dont just copy your
..cf file, get a modernish MC file, one can be got from here:
http://support.ausics.net/sendmail/newsendmail.mc
(save it as sendmail.mc in the /sendmail-source-dir/cf/cf directory
and rebuild and install the new cf file

The version you are using is so old and so full of security exploits I'm
not surprised you have problems with it.

and since its RH 7.3 thats only the start of your problems, chances are
sendmail is just relaying what some other exploited package has sent to it
to send out. Seriously, 7.3 is what, umm 7 years old and unsupported for 3
4 or more years by RH. If you can install something new, Slackware 12.0 is
good and solid.


--
Cheers
Res

Re: Sendmail with high load averages.

In article <1194909354.433970.174370@57g2000hsv.googlegroups.com>,
dlhirsch wrote:

> On Nov 12, 4:21 pm, Res wrote:
> > On Mon, 12 Nov 2007, dlhirsch wrote:
> >
> > > We recently have been having issues with sendmail. For some reason our
> > > RedHat Linux machine has been spiking high load averages. The load
> > > average can get as high as 40-45 but usually around 10-15. Not sure
> > > where to start troubleshooting. The top command does not shed any
> > > light on this issue. I have also looked at vmstat and iostat but
> > > cannot tell what would be causing the spikes. Any suggestions/
> > > questions would be greatly appreciated. Thanks!
> >
> > This is better of on a redhat list, since they like to
> > (butcher|hack_the_crap_out_of|customize_beyond_belief) most things that go
> > into their distro.
> >
> > Not to mention you've not told us any versions, how many processes etc..
> >
> > --
> > Cheers
> > Res
>
> I could have guessed what information would have been helpful but it
> would have been only a guess. Asking for specifics like version
> numbers is still kind of vague. The version of sendmail is
> "sendmail-8.9.3-20" on RedHat 7.3.

That makes the top guess very simple: you no longer are in control of
this system, and whatever you can see about what is happening on it is
limited to what the functional owner of the system wants to allow you to
see.

A machine running that software has been sitting for many years with
security flaws that make it possible for the entire system to be taken
over remotely. That state is survivable for a short time on pure luck,
but a machine neglected like that for years with any exposure to the net
has almost certainly been found at least once by someone eager and able
to exploit those flaws. If that has happened, then the tools on the
system that you would normally use to analyze a normal performance
problem are very likely to have been replaced with programs of the same
names in the same places which are modified so as to hide the details of
what is really going on. You cannot trust what you are told by the
versions of top, ps, ls, vmstat, iostat, netstat, or anything else on
that host. A highly proficient sysadmin might be able to reclaim control
and salvage such a system from whoever has taken it over, but the fact
that this system is in this neglected state argues that it has not been
touched by even a minimally competent sysadmin in a long time.

Your best chance at this point is not to attempt to fix this system, but
to wipe it and build something fresh.

--
Now where did I hide that website...

Re: Sendmail with high load averages.

On Nov 12, 10:01 pm, Bill Cole wrote:
> In article <1194909354.433970.174...@57g2000hsv.googlegroups.com>,
>
>
>
>
>
> dlhirsch wrote:
> > On Nov 12, 4:21 pm, Res wrote:
> > > On Mon, 12 Nov 2007, dlhirsch wrote:
>
> > > > We recently have been having issues with sendmail. For some reason our
> > > > RedHat Linux machine has been spiking high load averages. The load
> > > > average can get as high as 40-45 but usually around 10-15. Not sure
> > > > where to start troubleshooting. The top command does not shed any
> > > > light on this issue. I have also looked at vmstat and iostat but
> > > > cannot tell what would be causing the spikes. Any suggestions/
> > > > questions would be greatly appreciated. Thanks!
>
> > > This is better of on a redhat list, since they like to
> > > (butcher|hack_the_crap_out_of|customize_beyond_belief) most things that go
> > > into their distro.
>
> > > Not to mention you've not told us any versions, how many processes etc..
>
> > > --
> > > Cheers
> > > Res
>
> > I could have guessed what information would have been helpful but it
> > would have been only a guess. Asking for specifics like version
> > numbers is still kind of vague. The version of sendmail is
> > "sendmail-8.9.3-20" on RedHat 7.3.
>
> That makes the top guess very simple: you no longer are in control of
> this system, and whatever you can see about what is happening on it is
> limited to what the functional owner of the system wants to allow you to
> see.
>
> A machine running that software has been sitting for many years with
> security flaws that make it possible for the entire system to be taken
> over remotely. That state is survivable for a short time on pure luck,
> but a machine neglected like that for years with any exposure to the net
> has almost certainly been found at least once by someone eager and able
> to exploit those flaws. If that has happened, then the tools on the
> system that you would normally use to analyze a normal performance
> problem are very likely to have been replaced with programs of the same
> names in the same places which are modified so as to hide the details of
> what is really going on. You cannot trust what you are told by the
> versions of top, ps, ls, vmstat, iostat, netstat, or anything else on
> that host. A highly proficient sysadmin might be able to reclaim control
> and salvage such a system from whoever has taken it over, but the fact
> that this system is in this neglected state argues that it has not been
> touched by even a minimally competent sysadmin in a long time.
>
> Your best chance at this point is not to attempt to fix this system, but
> to wipe it and build something fresh.
>
> --
> Now where did I hide that website...- Hide quoted text -
>
> - Show quoted text -

To compound the issue we are running Spamassassin and Kaspersky
Anitvirus. I guess the best bet would be to start from scratch on a
different machine since the machine with the issue may be overloaded.
It is a web server, mail server, fax server (Hylafax), proxy server
(Squid), and many others. I wanted to basically get a second opinion
in case there was something simple. It appears that most emails are
from "User unknown" which seems to flood the server. Below is the
output from a script we wrote to count different processes.

Sendmail Server Stats for Tue Nov 13 08:36:18 CST 2007

output from the "w" command

8:36am up 3:50, 2 users, load average: 7.50, 6.45, 5.48
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root pts/1 130.0.1.67 8:12am 5.00s 0.12s 0.12s -bash
root pts/2 130.0.1.54 8:12am 0.00s 0.20s 0.01s /bin/
sh /usr/bi

***********************************************************
ps -e | grep | wc -l

The number of:
Sendmail processes running is: 745
Procmail processes running is: 166
Kavkeeper processes running is: 146
Spamd proccesses running is: 3
Spamassassin processes running is: 0

***********************************************************
ls -1