Procmail problem

Have had an email setup that has worked perfectly for a long time, using
Fetchmail, Sendmail, and Procmail. Now, Procmail has begun to
misbehave, and I can't puzzle out why. It is version 3.15.2 (running
Slackware 10.0).

For quite a while now, I've gotten nothing in my inbox, but didn't
recognize that as a problem. A couple of days ago I was looking for an
email that should have shown up there, and didn't. Checked the logfile,
and it had been chucked in the dustbin!

Here's the original $HOME/.procmailrc file:

-----------------------------------------------------------
PATH=$HOME/bin:/usr/bin:/usr/ucb:/bin:/usr/local/bin:.
MAILDIR=/data/mail
DEFAULT=$MAILDIR/inbox
LOGFILE=$MAILDIR/procmaillog
LOCKFILE=$HOME/.lockmail
BITBUCKET=/dev/null
BLACKFILE=$MAILDIR/filters/black
GREYFILE=$MAILDIR/filters/grey

# Process the whitelist
INCLUDERC=$MAILDIR/filters/white

# Test if the email's sender is in the blacklist
:0
* ? formail -x"From" -x"From:" -x"Sender:" \
-x"Reply-To:" -x"Return-Path:" -x"To:" \
| egrep -f $BLACKFILE
$BITBUCKET

# Test if the email's sender is in the greylist
:0
* ? formail -x"From" -x"From:" -x"Sender:" \
-x"Reply-To:" -x"Return-Path:" -x"To:" \
| egrep -is -f $GREYFILE
junk

# Anything that has not been delivered at the end will go to $DEFAULT
# using LOCKFILE=$DEFAULT$LOCKEXT
-----------------------------------------------------------

The first thing I did was to `touch /data/mail/black` and change the
above file: BITBUCKET=/data/mail/black. No more lost emails at least!

The result was that stuff that should have past the blacklist wound up
there anyway. So I moved the BLACKFILE and replaced it with an empty
file. Now, anything not in the whitelist winds up /data/mail/junk.

At least I'm not loosing emails, but for whatever reason the filters are
not working, and I can't figure out why. And of course I can't figure
out what changed: no log files going back to when that occured (13 Oct
2006).

Anyone have any ideas?

Thanks,

Longfellow

Re: Procmail problem

Longfellow wrote:
> Have had an email setup that has worked perfectly for a long time, using
> Fetchmail, Sendmail, and Procmail. Now, Procmail has begun to
> misbehave, and I can't puzzle out why. It is version 3.15.2 (running
> Slackware 10.0).
>
> For quite a while now, I've gotten nothing in my inbox, but didn't
> recognize that as a problem. A couple of days ago I was looking for an
> email that should have shown up there, and didn't. Checked the logfile,
> and it had been chucked in the dustbin!
>
> Here's the original $HOME/.procmailrc file:
>
> -----------------------------------------------------------
> PATH=$HOME/bin:/usr/bin:/usr/ucb:/bin:/usr/local/bin:.
> MAILDIR=/data/mail
> DEFAULT=$MAILDIR/inbox
> LOGFILE=$MAILDIR/procmaillog
> LOCKFILE=$HOME/.lockmail
> BITBUCKET=/dev/null
> BLACKFILE=$MAILDIR/filters/black
> GREYFILE=$MAILDIR/filters/grey
>
> # Process the whitelist
> INCLUDERC=$MAILDIR/filters/white
>
> # Test if the email's sender is in the blacklist
> :0
> * ? formail -x"From" -x"From:" -x"Sender:" \
> -x"Reply-To:" -x"Return-Path:" -x"To:" \
> | egrep -f $BLACKFILE
> $BITBUCKET

# This recipe should use a procmail lockfile since it's potentially
# delivering either to a file or to /dev/null

:0: ${LOCKFILE}
* ? formail -x"From" -x"From:" -x"Sender:" \
-x"Reply-To:" -x"Return-Path:" -x"To:" \
| egrep -f $BLACKFILE
$BITBUCKET

Other than that the only thing I can suggest here is to look in your
$BLACKFILE for any empty lines since an empty line in this file will
cause this recipe to always match. I ran into that problem a few years
ago, and it drove me nuts for a while trying to figure it out. Also keep
in mind that the contents of your $BLACKFILE should be regular
expressions, one per line, instead of fixed strings since you are using
egrep instead of fgrep

>
> # Test if the email's sender is in the greylist
> :0
> * ? formail -x"From" -x"From:" -x"Sender:" \
> -x"Reply-To:" -x"Return-Path:" -x"To:" \
> | egrep -is -f $GREYFILE
> junk

Same here, check for any empty lines in $GREYFILE and use at least an
implied procmail lockfile (Done with the 2nd semi-colon) since it's a
delivering recipe.

:0:
* ? formail -x"From" -x"From:" -x"Sender:" \
-x"Reply-To:" -x"Return-Path:" -x"To:" \
| egrep -is -f $GREYFILE
junk

>
> # Anything that has not been delivered at the end will go to $DEFAULT
> # using LOCKFILE=$DEFAULT$LOCKEXT
> -----------------------------------------------------------
> [SNIP]

--
Garen

Re: Procmail problem

On 2006-12-23, Garen Erdoisa wrote:
> Longfellow wrote:
>> Have had an email setup that has worked perfectly for a long time, using
>> Fetchmail, Sendmail, and Procmail. Now, Procmail has begun to
>> misbehave, and I can't puzzle out why. It is version 3.15.2 (running
>> Slackware 10.0).
>>
>> For quite a while now, I've gotten nothing in my inbox, but didn't
>> recognize that as a problem. A couple of days ago I was looking for an
>> email that should have shown up there, and didn't. Checked the logfile,
>> and it had been chucked in the dustbin!
>>
>> Here's the original $HOME/.procmailrc file:
>>
>> -----------------------------------------------------------
>> PATH=$HOME/bin:/usr/bin:/usr/ucb:/bin:/usr/local/bin:.
>> MAILDIR=/data/mail
>> DEFAULT=$MAILDIR/inbox
>> LOGFILE=$MAILDIR/procmaillog
>> LOCKFILE=$HOME/.lockmail
>> BITBUCKET=/dev/null
>> BLACKFILE=$MAILDIR/filters/black
>> GREYFILE=$MAILDIR/filters/grey
>>
>> # Process the whitelist
>> INCLUDERC=$MAILDIR/filters/white
>>
>> # Test if the email's sender is in the blacklist
>> :0
>> * ? formail -x"From" -x"From:" -x"Sender:" \
>> -x"Reply-To:" -x"Return-Path:" -x"To:" \
>> | egrep -f $BLACKFILE
>> $BITBUCKET
>
> # This recipe should use a procmail lockfile since it's potentially
> # delivering either to a file or to /dev/null
>
>:0: ${LOCKFILE}
> * ? formail -x"From" -x"From:" -x"Sender:" \
> -x"Reply-To:" -x"Return-Path:" -x"To:" \
> | egrep -f $BLACKFILE
> $BITBUCKET

Aha!! I was just reading about this. Don't understand it yet, but I'm
on the right track.

> Other than that the only thing I can suggest here is to look in your
> $BLACKFILE for any empty lines since an empty line in this file will
> cause this recipe to always match. I ran into that problem a few years
> ago, and it drove me nuts for a while trying to figure it out. Also keep
> in mind that the contents of your $BLACKFILE should be regular
> expressions, one per line, instead of fixed strings since you are using
> egrep instead of fgrep

Oops, that means an empty file is an "always match" file?

Okay, what that suggests is that a file should contain no '\n' after the
last entry. Is that correct?

>>
>> # Test if the email's sender is in the greylist
>> :0
>> * ? formail -x"From" -x"From:" -x"Sender:" \
>> -x"Reply-To:" -x"Return-Path:" -x"To:" \
>> | egrep -is -f $GREYFILE
>> junk
>
> Same here, check for any empty lines in $GREYFILE and use at least an
> implied procmail lockfile (Done with the 2nd semi-colon) since it's a
> delivering recipe.
>
>:0:
> * ? formail -x"From" -x"From:" -x"Sender:" \
> -x"Reply-To:" -x"Return-Path:" -x"To:" \
> | egrep -is -f $GREYFILE
> junk

Here is where there is a new line after the last entry. Have to get rid
of that new line, then.

Okay, just vimed the black and grey files, and they both had blank lines
at the end. Both are removed, so that there is no new line at the end
of the last entry.

It seems to me that I may have added an entry to the black list on or
about Oct 13, which would account for the change. Since all worked
perfectly before then, I'm going to leave the lock file changes alone
for the present.

And I best be looking at egrep vs fgrep, lest I cause myself grief on
that account at some point.

Going to give all this a try, and will post the results as soon as I can
determine them.

Thanks very much for this response; you've been most helpful!

Longfellow


>>
>> # Anything that has not been delivered at the end will go to $DEFAULT
>> # using LOCKFILE=$DEFAULT$LOCKEXT
>> -----------------------------------------------------------
>> [SNIP]
>

Re: Procmail problem

Longfellow wrote:
> On 2006-12-23, Garen Erdoisa wrote:
>> Longfellow wrote:
>>> Have had an email setup that has worked perfectly for a long time, using
>>> Fetchmail, Sendmail, and Procmail. Now, Procmail has begun to
>>> misbehave, and I can't puzzle out why. It is version 3.15.2 (running
>>> Slackware 10.0).
>>>
>> [SNIP]
> Aha!! I was just reading about this. Don't understand it yet, but I'm
> on the right track.

Basically the idea behind a lock file is to prevent race conditions
where more than one procmail process wants to write to the same file at
the same time. If you use a lock file, one process will wait until the
other process removes it's lock file before applying it's own lock file
so it can write to the file. If you don't use lock file where
appropriate, then you may end up with partial email messages embedded in
the middle of other email messages in the file. I've actually seen this
happen.

In the special case where the file is /dev/null, lockfiles really don't
matter. But in the normal cases where the file is a real file, then the
lockfile does matter.

procmail will try to "guess" at what lockfile name to use if you don't
give it an explicit lockfile name. So in the case of the lockfile being
something like /dev/null.lock the username owning that procmail process
may not have write privileges for the /dev directory in order to create
a /dev/null.lock so in that case you can tell procmail explicitly what
lockfile name to use in a directory it does have write permissions on.

>
>> Other than that the only thing I can suggest here is to look in your
>> $BLACKFILE for any empty lines since an empty line in this file will
>> cause this recipe to always match. I ran into that problem a few years
>> ago, and it drove me nuts for a while trying to figure it out. Also keep
>> in mind that the contents of your $BLACKFILE should be regular
>> expressions, one per line, instead of fixed strings since you are using
>> egrep instead of fgrep
>
> Oops, that means an empty file is an "always match" file?

yep. Well actually not a zero byte file. But a file with just a single
\n would always match.

>
> Okay, what that suggests is that a file should contain no '\n' after the
> last entry. Is that correct?

correct. No blank lines is what you want in this case.

> [SNIP]
>> junk
>
> Here is where there is a new line after the last entry. Have to get rid
> of that new line, then.
>
> Okay, just vimed the black and grey files, and they both had blank lines
> at the end. Both are removed, so that there is no new line at the end
> of the last entry.
>
> It seems to me that I may have added an entry to the black list on or
> about Oct 13, which would account for the change. Since all worked
> perfectly before then, I'm going to leave the lock file changes alone
> for the present.

I would still do the lockfiles. They won't hurt anything and may save
you some grief later. It is how procmail is intended to be used.

>
> And I best be looking at egrep vs fgrep, lest I cause myself grief on
> that account at some point.

About the only think with email addresses to worry about there is that a
dot matches any character, so it's possible that an egrep file containing

someuser@example.com

would match with strings like

someuser@examplekcom

in an egrep file you'd want to write the address as:

someuser@example\.com

where the dot is escaped.

in an fgrep file you don't have to worry about escaping the dot, since
it's fixed string being used for the pattern match.

Usually it's not a problem unless you are doing some heavy duty pattern
matching. Also the egrep on a file containing a lot of regular
expression patterns uses a lot more CPU time than fgrep on a file using
a list of fixed strings. I've actually tested both approaches in past
experiments.

If you need to do regular expression pattern matching in procmail, it's
best for CPU load to use the built in procmail regular expressions
instead of relying on external programs for that job.

Remember that every time you have procmail use an external program to do
a job that can be done entirely in procmail, it adds unnecessarily to
the CPU and memory overhead for handling your email.

On a small system handling only a few 10's or 100's of emails an hour
it's not that big a deal. But when you your system is handling thousands
or hundreds of thousands of email events an hour, your CPU budget starts
becoming a critical factor in procmail recipe design.

>
> Going to give all this a try, and will post the results as soon as I can
> determine them.
>
> Thanks very much for this response; you've been most helpful!

Quite welcome.

>>> [SNIP]

--
Garen

Re: Procmail problem

On 2006-12-23, Garen Erdoisa wrote:
> Longfellow wrote:
>> On 2006-12-23, Garen Erdoisa wrote:
>>> Longfellow wrote:
>>>> Have had an email setup that has worked perfectly for a long time, using
>>>> Fetchmail, Sendmail, and Procmail. Now, Procmail has begun to
>>>> misbehave, and I can't puzzle out why. It is version 3.15.2 (running
>>>> Slackware 10.0).
>>>>
>>> [SNIP]
>> Aha!! I was just reading about this. Don't understand it yet, but I'm
>> on the right track.
>
> Basically the idea behind a lock file is to prevent race conditions
> where more than one procmail process wants to write to the same file at
> the same time. If you use a lock file, one process will wait until the
> other process removes it's lock file before applying it's own lock file
> so it can write to the file. If you don't use lock file where
> appropriate, then you may end up with partial email messages embedded in
> the middle of other email messages in the file. I've actually seen this
> happen.
>
> In the special case where the file is /dev/null, lockfiles really don't
> matter. But in the normal cases where the file is a real file, then the
> lockfile does matter.
>
> procmail will try to "guess" at what lockfile name to use if you don't
> give it an explicit lockfile name. So in the case of the lockfile being
> something like /dev/null.lock the username owning that procmail process
> may not have write privileges for the /dev directory in order to create
> a /dev/null.lock so in that case you can tell procmail explicitly what
> lockfile name to use in a directory it does have write permissions on.

Okay, on further reading (third time through) this is what I gathered;
you confirm it. In my case, a single fetchmail process runs in an
orderly fashion, so I haven't had occasion to experience a race
condition. Nevertheless, I'll follow good form here and add these lock
files. If (when?) I add other email fetch processes, that will save me
from potential disaster.

>>
>>> Other than that the only thing I can suggest here is to look in your
>>> $BLACKFILE for any empty lines since an empty line in this file will
>>> cause this recipe to always match. I ran into that problem a few years
>>> ago, and it drove me nuts for a while trying to figure it out. Also keep
>>> in mind that the contents of your $BLACKFILE should be regular
>>> expressions, one per line, instead of fixed strings since you are using
>>> egrep instead of fgrep
>>
>> Oops, that means an empty file is an "always match" file?
>
> yep. Well actually not a zero byte file. But a file with just a single
> \n would always match.

Ah, right. It does take a newline to create a line.

>>
>> Okay, what that suggests is that a file should contain no '\n' after the
>> last entry. Is that correct?
>
> correct. No blank lines is what you want in this case.
>
>> [SNIP]
>>> junk
>>
>> Here is where there is a new line after the last entry. Have to get rid
>> of that new line, then.
>>
>> Okay, just vimed the black and grey files, and they both had blank lines
>> at the end. Both are removed, so that there is no new line at the end
>> of the last entry.
>>
>> It seems to me that I may have added an entry to the black list on or
>> about Oct 13, which would account for the change. Since all worked
>> perfectly before then, I'm going to leave the lock file changes alone
>> for the present.
>
> I would still do the lockfiles. They won't hurt anything and may save
> you some grief later. It is how procmail is intended to be used.

Noted, and thanks.
>>
>> And I best be looking at egrep vs fgrep, lest I cause myself grief on
>> that account at some point.
>
> About the only think with email addresses to worry about there is that a
> dot matches any character, so it's possible that an egrep file containing
>
> someuser@example.com
>
> would match with strings like
>
> someuser@examplekcom
>
> in an egrep file you'd want to write the address as:
>
> someuser@example\.com
>
> where the dot is escaped.
>
> in an fgrep file you don't have to worry about escaping the dot, since
> it's fixed string being used for the pattern match.
>
> Usually it's not a problem unless you are doing some heavy duty pattern
> matching. Also the egrep on a file containing a lot of regular
> expression patterns uses a lot more CPU time than fgrep on a file using
> a list of fixed strings. I've actually tested both approaches in past
> experiments.
>
> If you need to do regular expression pattern matching in procmail, it's
> best for CPU load to use the built in procmail regular expressions
> instead of relying on external programs for that job.
>
> Remember that every time you have procmail use an external program to do
> a job that can be done entirely in procmail, it adds unnecessarily to
> the CPU and memory overhead for handling your email.
>
> On a small system handling only a few 10's or 100's of emails an hour
> it's not that big a deal. But when you your system is handling thousands
> or hundreds of thousands of email events an hour, your CPU budget starts
> becoming a critical factor in procmail recipe design.

Yep, `man grep` explained this difference. `fgrep` == `grep -F`. But
it didn't explain why this might be useful, so thanks for this added
information! What I've done is add this information to the .procmailrc
file. I suppose it's not surprising that this information is not
available in any of the Procmail documentation; I wonder if passing this
on to the procmail people might be worthwhile.

As for `formail`, it is apparently a part of the Procmail package,
although it can be called separately. I assume that it is not treated
as an external utility.

You know, having this exchange on Usenet insures that it is seen by
anyone who may read this group and be interested in Procmail. That is a
Good Thing (as we all know...): others may benefit as well. In any
case, unless you've further info to add, I'm done here.

And again, thanks very much!

Longfellow

Re: Procmail problem

Longfellow wrote:
> On 2006-12-23, Garen Erdoisa wrote:
>> Longfellow wrote:
>>> On 2006-12-23, Garen Erdoisa wrote:
>>>> Longfellow wrote:
>>>>> Have had an email setup that has worked perfectly for a long time, using
>>>[SNIP...]
>> procmail will try to "guess" at what lockfile name to use if you don't
>> give it an explicit lockfile name. So in the case of the lockfile being
>> something like /dev/null.lock the username owning that procmail process
>> may not have write privileges for the /dev directory in order to create
>> a /dev/null.lock so in that case you can tell procmail explicitly what
>> lockfile name to use in a directory it does have write permissions on.
>
> Okay, on further reading (third time through) this is what I gathered;
> you confirm it. In my case, a single fetchmail process runs in an
> orderly fashion, so I haven't had occasion to experience a race
> condition. Nevertheless, I'll follow good form here and add these lock
> files. If (when?) I add other email fetch processes, that will save me
> from potential disaster.

With a fetchmail process getting mail from an external mailbox then
handing off those emails to procmail, fetchmail will launch a separate
procmail process to handle each incoming mail in turn. So in this
particular case a lockfile wouldn't be much of an issue since the
procmail process are running successively instead of running concurrently.

In the case of running a mailer daemon such as sendmail which hands off
emails to procmail, you can end up with potentially hundreds of procmail
processes running concurrently. So in that case lockfiles are much more
important to avoid the race conditions.

>
>> [SNIP]
>> On a small system handling only a few 10's or 100's of emails an hour
>> it's not that big a deal. But when you your system is handling thousands
>> or hundreds of thousands of email events an hour, your CPU budget starts
>> becoming a critical factor in procmail recipe design.
>
> Yep, `man grep` explained this difference. `fgrep` == `grep -F`. But
> it didn't explain why this might be useful, so thanks for this added
> information! What I've done is add this information to the .procmailrc
> file. I suppose it's not surprising that this information is not
> available in any of the Procmail documentation; I wonder if passing this
> on to the procmail people might be worthwhile.
>
> As for `formail`, it is apparently a part of the Procmail package,
> although it can be called separately. I assume that it is not treated
> as an external utility.

formail is called the same way any other external program would be
called from within a procmail recipe. It's a simple mail filter in it's
own right and, you are correct that it is part of the procmail package.

grep is not part of procmail, but as you noted, it can be called from
within procmail. Any program can that can read from STDIN and give some
sort of text based output that can be run in a shell can potentially be
used from within a procmail recipe.

--
Garen

Re: Procmail problem

On 2006-12-23, Garen Erdoisa wrote:


> In the case of running a mailer daemon such as sendmail which hands off
> emails to procmail, you can end up with potentially hundreds of procmail
> processes running concurrently. So in that case lockfiles are much more
> important to avoid the race conditions.


Aha. Yes I'm running sendmail as the MTA, so the lockfiles are
relevant. Don't believe I've ever had a race condition, but multiple
procmail processes make that possible, as you note.

Lockfiles are now part of the .procmailrc, so I'm good to go.

Thanks again,

Longfellow

Re: Procmail problem

Longfellow wrote:
> # Test if the email's sender is in the blacklist
> :0
> * ? formail -x"From" -x"From:" -x"Sender:" \
> -x"Reply-To:" -x"Return-Path:" -x"To:" \
> | egrep -f $BLACKFILE
> $BITBUCKET

Not quite. The expansion token $ for environment variables is missing.
Try "* $ ? formail ..." and see if it helps.

For more examples see the last line of my signature.

All the best, Timo

--
Prof. Timo Salmi ftp & http://garbo.uwasa.fi/ archives 193.166.120.5
Department of Accounting and Business Finance ; University of Vaasa
mailto:ts@uwasa.fi ; FIN-65101, Finland
Timo's procmail tips at http://www.uwasa.fi/~ts/info/proctips.html

Re: Procmail problem

Timo Salmi wrote:
> Longfellow wrote:
>> # Test if the email's sender is in the blacklist
>> :0
>> * ? formail -x"From" -x"From:" -x"Sender:" \
>> -x"Reply-To:" -x"Return-Path:" -x"To:" \
>> | egrep -f $BLACKFILE
>> $BITBUCKET
>
> Not quite. The expansion token $ for environment variables is missing.
> Try "* $ ? formail ..." and see if it helps.
>

In this particular case the $ doesn't matter because variables in lines
that are executing commands are expanded by the shell after they are
passed to the shell to be executed. You won't see the variable expansion
take place with verbose output in the procmail log but it does happen.

It won't hurt to have it there and have procmail expand the variable
before passing it to the shell, it just won't make any difference in the
overall behavior of the recipe.

It does matter in lines where the line is a regular expression
containing variables because in that case those variables are not passed
through a shell.

Here is a simple example to illustrate this behavior.

Test message piped to a procmail test recipe:

-----${HOME}/testmessage.txt----
To: nobody@example.com
From: nobody@example.com
Subject: procmail test message

Test message body
-----end of file-----


Data file used by the procmail test recipe:

-----${HOME}/greylist.txt----
nobody@example.com
-----end of file-----


Procmail test recipe:

-----${HOME}/test.rc-----
NL="
"

VERBOSE=yes
GREYLIST=${HOME}/greylist.txt

:0
* ? formail -cx "To:" |grep -Fif ${GREYLIST}
{
VERBOSE=no
LOG="[$$]$_: Test1 matched${NL}"
}
VERBOSE=no
:0 E
{
LOG="[$$]$_: Test1 did not match${NL}"
}

VERBOSE=yes
:0
* $ ? formail -cx "To:" |grep -Fif ${GREYLIST}
{
VERBOSE=no
LOG="[$$]$_: Test2 matched${NL}"
}
VERBOSE=no
:0 E
{
LOG="[$$]$_: Test2 did not match${NL}"
}

:0
/dev/null
----end of file----

procmail ./test.rc
procmail: [31395] Sat Dec 30 15:56:15 2006
procmail: Assigning "GREYLIST=/tmp/example/greylist.txt"
procmail: Executing " formail -cx "To:" |grep -Fif ${GREYLIST}"
procmail: Match on " formail -cx "To:" |grep -Fif ${GREYLIST}"
procmail: Assigning "VERBOSE=no"
[31395]./temp.rc: Test1 matched
procmail: Executing " formail -cx To: |grep -Fif /tmp/example/greylist.txt"
procmail: Match on " formail -cx To: |grep -Fif /tmp/example/greylist.txt"
procmail: Assigning "VERBOSE=no"
[31395]./temp.rc: Test2 matched


I used to think the same thing until someone else pointed this
particular aspect of procmail behavior out to me a few years ago.

--
Garen