local mail problem after FC4->FC5 upgrade

Hi.

We have a cluster of Intel machines, which we are upgrading from Fedora
Core 4 to Fedora Core 5. Our cluster consists of various Dell machines,
along with three HP Compaq dc5000 MT machines.

Prior to this upgrade, mail sent to a local user on any of these (FC4)
machines went through without any problems.

The problem: After the upgrade, mail sent to a local user on any of the HP
machines is bouncing with a "550 User unknown" error msg. However, mail
sent to a local user on any of the Dell machines works fine.

Some background:

(1) MX records have been set up so that dsm.dsm.fordham.edu is the mail
exchanger for dsm.fordham.edu.

(2) dsm.dsm.fordham.edu's mail-spool is NFS-mounted on all Linux hosts
in the dsm.fordham.edu domain.

(3) Prior to the FC4->FC5 upgrade, mail to local users worked fine on all
the machines.

(4) The sendmail.mc files are the same on all machines.

The details:

(1) If mail is sent to joeuser@dsm.fordham.edu from any of these machines,
then it goes through.

(2) If mail is sent to joeuser (as a local user) on any of the HP machines,
(say, sobolev.dsm.fordham.edu), then root gets an email msg from
mailer-daemon
procmail: Unknown user "joeuser"
550 5.1.1 ... User unknown

(3) If mail is sent to joeuser (as a local user) on any of the Dell machines,
then it goes through.

If anybody has an explanation for why sendmail is having this problem on
some (but not all) of our FC5 boxes, I would greatly appreciate knowing the
reason for the problem, as well as the solution.

Many thanks!!

--
Art Werschulz (8-{)} "Metaphors be with you." -- bumper sticker
GCS/M (GAT): d? -p+ c++ l u+(-) e--- m* s n+ h f g+ w+ t++ r- y?
Internet: agw STRUDEL cs.columbia.edu
ATTnet: Columbia U. (212) 939-7060, Fordham U. (212) 636-6325

Re: local mail problem after FC4->FC5 upgrade

Art Werschulz wrote:

> Hi.
>
> We have a cluster of Intel machines, which we are upgrading from Fedora
> Core 4 to Fedora Core 5. Our cluster consists of various Dell machines,
> along with three HP Compaq dc5000 MT machines.
>
> Prior to this upgrade, mail sent to a local user on any of these (FC4)
> machines went through without any problems.
>
> The problem: After the upgrade, mail sent to a local user on any of the HP
> machines is bouncing with a "550 User unknown" error msg. However, mail
> sent to a local user on any of the Dell machines works fine.
>
> Some background:
>
> (1) MX records have been set up so that dsm.dsm.fordham.edu is the mail
> exchanger for dsm.fordham.edu.
>
> (2) dsm.dsm.fordham.edu's mail-spool is NFS-mounted on all Linux hosts
> in the dsm.fordham.edu domain.
>
> (3) Prior to the FC4->FC5 upgrade, mail to local users worked fine on all
> the machines.
>
> (4) The sendmail.mc files are the same on all machines.
>
> The details:
>
> (1) If mail is sent to joeuser@dsm.fordham.edu from any of these machines,
> then it goes through.
>
> (2) If mail is sent to joeuser (as a local user) on any of the HP machines,
> (say, sobolev.dsm.fordham.edu), then root gets an email msg from
> mailer-daemon
> procmail: Unknown user "joeuser"
> 550 5.1.1 ... User unknown
>
> (3) If mail is sent to joeuser (as a local user) on any of the Dell machines,
> then it goes through.
>
> If anybody has an explanation for why sendmail is having this problem on
> some (but not all) of our FC5 boxes, I would greatly appreciate knowing the
> reason for the problem, as well as the solution.
>
> Many thanks!!
>


Presumably the user information is stored in an LDAP directory? Are the
HP's configured to consult the LDAP directory and do they store the
message or do they suppose to forward the email on?

You need to look at the mail servers logs to see what is going on.


You might have to define in the mail service configuration that it needs
to forward any and all emails to the MX. Or make sure that the mail
server knows the *.dsm.fordham.edu is a local domain or if there is an
email address map that it needs to be consulted.

Which mail server are you using?

AK

Re: local mail problem after FC4->FC5 upgrade

Hi.

AK writes:

> Presumably the user information is stored in an LDAP directory? Are the
> HP's configured to consult the LDAP directory and do they store the message
> or do they suppose to forward the email on?

All the machines on the network are using NIS for user information.
There's one master NIS server, and no slave servers. IOW, all the machines
have the same user info.

> You need to look at the mail servers logs to see what is going on.

When I sent mail to a local user on one of the affected machines, the
following appeared in /var/log/maillog:

May 31 09:30:24 dsm spamd[2532]: prefork: child states: II
May 31 09:30:25 dsm sendmail[3121]: k4VDUOuW003120:
to=, delay=00:00:01, xdelay=00:00:01, mailer=local,
pri=34002, dsn=2.0.0, stat=Sent

(I have replaced "@" with " AT ", to make things a bit harder for naive
simpleminded spambots.)

When I sent mail to joeuser AT dsm.fordham.edu (again, on one of the affected
machines), the following appeared in /var/log/maillog:

May 31 09:31:48 dsm sendmail[3153]: k4VDVmag003153: from=<>, size=3775,
class=0, nrcpts=1,
msgid=<200605311331.k4VDVmPX028099 AT sobolev.dsm.fordham.edu>,
proto=ESMTP, daemon=MTA, relay=sobolev.dsm.fordham.edu [150.108.64.57]
May 31 09:31:48 dsm spamd[2554]: spamd: connection from localhost [127.0.0.1]
at port 41721
May 31 09:31:48 dsm spamd[2554]: spamd: setuid to root succeeded
May 31 09:31:48 dsm spamd[2554]: spamd: still running as root: user not
specified with -u, not found, or set to root, falling back to nobody at
/usr/bin/spamd line 1152, line 4.
May 31 09:31:48 dsm spamd[2554]: spamd: processing message
<200605311331.k4VDVmPX028099 AT sobolev.dsm.fordham.edu> for root:99
May 31 09:31:49 dsm spamd[2554]: mkdir /root/.spamassassin: Permission denied
at /usr/lib/perl5/vendor_perl/5.8.8/Mail/SpamAssassin.pm line 1469
May 31 09:31:49 dsm spamd[2554]: locker: safe_lock: cannot create tmp
lockfile /root/.spamassassin/auto-whitelist.lock.dsm.dsm.fordham.edu. 2554 for
/root/.spamassassin/auto-whitelist.lock: Permission denied
May 31 09:31:49 dsm spamd[2554]: auto-whitelist: open of auto-whitelist
file failed: locker: safe_lock: cannot create tmp lockfile
/root/.spamassassin/auto-whitelist.lock.dsm.dsm.fordham.edu. 2554 for
/root/.spamassassin/auto-whitelist.lock: Permission denied
May 31 09:31:49 dsm spamd[2554]: bayes: locker: safe_lock: cannot create tmp
lockfile /root/.spamassassin/bayes.lock.dsm.dsm.fordham.edu.2554 for
/root/.spamassassin/bayes.lock: Permission denied
May 31 09:31:49 dsm spamd[2554]: spamd: clean message (-1.4/5.0) for root:99
in 0.1 seconds, 4060 bytes.
May 31 09:31:49 dsm spamd[2554]: spamd: result: . -1 - ALL_TRUSTED
scantime=0.1,size=4060,user=root,uid=99,required_score=5.0,r host=localhost,
raddr=127.0.0.1,rport=41721,
mid=<200605311331.k4VDVmPX028099 AT sobolev.dsm.fordham.edu>,autolearn=failed
May 31 09:31:49 dsm spamd[2532]: prefork: child states: II
May 31 09:31:49 dsm sendmail[3154]: k4VDVmag003153:
to=, delay=00:00:01, xdelay=00:00:01, mailer=local,
pri=33993, dsn=2.0.0, stat=Sent
May 31 09:32:07 dsm sendmail[3169]: k4VDW74M003169:
from=, size=590, class=0, nrcpts=1,
msgid=<200605311332.k4VDW7S9028110 AT sobolev.dsm.fordham.edu>,
proto=ESMTP, daemon=MTA, relay=sobolev.dsm.fordham.edu [150.108.64.57]
May 31 09:32:07 dsm sendmail[3170]: k4VDW74M003169:
to=, ctladdr= (201/150),
delay=00:00:00, xdelay=00:00:00, mailer=local, pri=30838, dsn=2.0.0,
stat=Sent

> You might have to define in the mail service configuration that it needs to
> forward any and all emails to the MX. Or make sure that the mail server
> knows the *.dsm.fordham.edu is a local domain or if there is an email
> address map that it needs to be consulted.

Exactly how should this be done?

Please recall that mail worked on all the machines before the FC4->FC5
upgrade, and it is still working on all the non-HP machines. It's only on
the HP machines that mail to local users has stopped working. We haven't
changed any of the sendmail configuration files.

> Which mail server are you using?

Our mail server is dsm.dsm.fordham.edu.

Many thanks for your response. I look forward to your next suggestions.

--
Art Werschulz (8-{)} "Metaphors be with you." -- bumper sticker
GCS/M (GAT): d? -p+ c++ l u+(-) e--- m* s n+ h f g+ w+ t++ r- y?
Internet: agw STRUDEL cs.columbia.edu
ATTnet: Columbia U. (212) 939-7060, Fordham U. (212) 636-6325

Re: local mail problem after FC4->FC5 upgrade

Art Werschulz wrote:
> Hi.
>
> AK writes:
>
>> Presumably the user information is stored in an LDAP directory? Are the
>> HP's configured to consult the LDAP directory and do they store the message
>> or do they suppose to forward the email on?
>
> All the machines on the network are using NIS for user information.
> There's one master NIS server, and no slave servers. IOW, all the machines
> have the same user info.
>
>> You need to look at the mail servers logs to see what is going on.
>
> When I sent mail to a local user on one of the affected machines, the
> following appeared in /var/log/maillog:
>
> May 31 09:30:24 dsm spamd[2532]: prefork: child states: II
> May 31 09:30:25 dsm sendmail[3121]: k4VDUOuW003120:
> to=, delay=00:00:01, xdelay=00:00:01, mailer=local,
> pri=34002, dsn=2.0.0, stat=Sent

This says that a spam daemon exited, then the email was delivered to
root. ie: handed off to procmail for "root"

>
> (I have replaced "@" with " AT ", to make things a bit harder for naive
> simpleminded spambots.)
>
> When I sent mail to joeuser AT dsm.fordham.edu (again, on one of the affected
> machines), the following appeared in /var/log/maillog:
>
> May 31 09:31:48 dsm sendmail[3153]: k4VDVmag003153: from=<>, size=3775,
> class=0, nrcpts=1,
> msgid=<200605311331.k4VDVmPX028099 AT sobolev.dsm.fordham.edu>,
> proto=ESMTP, daemon=MTA, relay=sobolev.dsm.fordham.edu [150.108.64.57]

It appears that the above message was fed to spamd at this point which
processed the message on a child process.

> May 31 09:31:48 dsm spamd[2554]: spamd: connection from localhost [127.0.0.1]
> at port 41721
> May 31 09:31:48 dsm spamd[2554]: spamd: setuid to root succeeded
> May 31 09:31:48 dsm spamd[2554]: spamd: still running as root: user not
> specified with -u, not found, or set to root, falling back to nobody at
> /usr/bin/spamd line 1152, line 4.

I see an issue here. your spamd (presumably SpamAssassin running in
daemon mode as a sendmail milter) successfully set it's user id to root,
then immediately dropped privileges again to the "nobody" account.

Subsequently because it lacked permissions the kernel denied access to
spamd when it attempted to access it's files in the /root account.

IMO, based on the information that you've provided thus far, this issue
is more likely a permissions issue on some of your servers than a
hardware issue.

> May 31 09:31:48 dsm spamd[2554]: spamd: processing message
> <200605311331.k4VDVmPX028099 AT sobolev.dsm.fordham.edu> for root:99
> May 31 09:31:49 dsm spamd[2554]: mkdir /root/.spamassassin: Permission denied
> at /usr/lib/perl5/vendor_perl/5.8.8/Mail/SpamAssassin.pm line 1469
> May 31 09:31:49 dsm spamd[2554]: locker: safe_lock: cannot create tmp
> lockfile /root/.spamassassin/auto-whitelist.lock.dsm.dsm.fordham.edu. 2554 for
> /root/.spamassassin/auto-whitelist.lock: Permission denied
> May 31 09:31:49 dsm spamd[2554]: auto-whitelist: open of auto-whitelist
> file failed: locker: safe_lock: cannot create tmp lockfile
> /root/.spamassassin/auto-whitelist.lock.dsm.dsm.fordham.edu. 2554 for
> /root/.spamassassin/auto-whitelist.lock: Permission denied
> May 31 09:31:49 dsm spamd[2554]: bayes: locker: safe_lock: cannot create tmp
> lockfile /root/.spamassassin/bayes.lock.dsm.dsm.fordham.edu.2554 for
> /root/.spamassassin/bayes.lock: Permission denied
> May 31 09:31:49 dsm spamd[2554]: spamd: clean message (-1.4/5.0) for root:99
> in 0.1 seconds, 4060 bytes.
> May 31 09:31:49 dsm spamd[2554]: spamd: result: . -1 - ALL_TRUSTED
> scantime=0.1,size=4060,user=root,uid=99,required_score=5.0,r host=localhost,
> raddr=127.0.0.1,rport=41721,
> mid=<200605311331.k4VDVmPX028099 AT sobolev.dsm.fordham.edu>,autolearn=failed
> May 31 09:31:49 dsm spamd[2532]: prefork: child states: II

The spamd process finishes returning it's results

> May 31 09:31:49 dsm sendmail[3154]: k4VDVmag003153:
> to=, delay=00:00:01, xdelay=00:00:01, mailer=local,
> pri=33993, dsn=2.0.0, stat=Sent

And the DSN (Delivery Status Notification) says that the message was
successfully delivered to root. (ie: handed off to procmail for root)

> May 31 09:32:07 dsm sendmail[3169]: k4VDW74M003169:
> from=, size=590, class=0, nrcpts=1,
> msgid=<200605311332.k4VDW7S9028110 AT sobolev.dsm.fordham.edu>,
> proto=ESMTP, daemon=MTA, relay=sobolev.dsm.fordham.edu [150.108.64.57]

New message

> May 31 09:32:07 dsm sendmail[3170]: k4VDW74M003169:
> to=, ctladdr= (201/150),
> delay=00:00:00, xdelay=00:00:00, mailer=local, pri=30838, dsn=2.0.0,
> stat=Sent

Which was successfully delivered to joeuser. (ie: handed off to procmail
for joeuser)

If procmail is not configured to do anything special for those accounts,
it would just deliver the message to the inbox for the account.

>
>> You might have to define in the mail service configuration that it needs to
>> forward any and all emails to the MX. Or make sure that the mail server
>> knows the *.dsm.fordham.edu is a local domain or if there is an email
>> address map that it needs to be consulted.
>
> Exactly how should this be done?

I would double check your config files in /etc/mail
Specifically make sure all your local host names are defined in

/etc/mail/local-host-names

for each of your machines that processes mail.
Look for typos, syntax errors, etc.

Also double check the entries in

/etc/mail/access

Again, look for typos

>
> Please recall that mail worked on all the machines before the FC4->FC5
> upgrade, and it is still working on all the non-HP machines. It's only on
> the HP machines that mail to local users has stopped working. We haven't
> changed any of the sendmail configuration files.

If you ran an upgrade instead of an install, look for *.rpmsave files
which may have been replaced by distribution default files.

find / -type f -name '*.rpmsave'

Sometimes this happens when you modify distribution config files, then
run an upgrade. If an old config file was overwritten, it could
potentially cause all kinds of havoc on a custom installation.

>
>> Which mail server are you using?
>
> Our mail server is dsm.dsm.fordham.edu.
>
> Many thanks for your response. I look forward to your next suggestions.
>

I'm not seeing any problem with the logs you supplied that addresses
your original issue of mail to local host user names not being
delivered. Do those accounts actually exist as such on those hosts? (ie:
are they present in /etc/passwd and do each of them have home
directories on those machines?) If not, then you could just add email
aliases to /etc/aliases for those problem user names which will
intercept and forward to the correct fully qualified email address.

spamd immediately dropping privileges when it's acquired root level
access isn't surprising, and is probably normal. I don't run
SpamAssassin here so am not certain of this.

--
Garen