using .procmail to compare "To" addresses.

I have a "whitelist" type question for any of you procmail gurus out
there who have a bit of time to help me out.

I have a few hundred valid "To" email addresses for my domain in a text
file. I'd like to compare all inbound email against this list, passing
all messages on (for now), but recording the "to" addresses of anybody
not in the main list in an "UnknownAddresses.txt" file, and send
everything on to $DEFAULT. I read somewhere that it's better to use ^TO
instead of $TO for the test, but I'm not sure I understand why...

Eventually I'd like to simply bit-bucket any message who's "To" address
is not in ValidAddresses.txt, but until I've run the "collector" for a
while, I don't want to turn it on.

I'm will also be creating a single regex rule for "To" addresses that
are *.com@mydomain.com" that will pass through as valid, but that can
be after I understand procmail a little better... In other words, when
I "register" with companies today, I give each company their own email
address using their domain name, and I want to allow pretty much
anything.com@mydomain.com to come through $DEFAULT

----------------------------------------------------

# .procmailrc
# Any "To" addresses in the text file "ValidAddresses.txt" are passed
# All other "to" addresses are allowed (for now) and recorded in
# "UnknownAddresses.txt". Eventually we'll kill all email to addresses
# that are not in ValidAddresses.txt
#
FGREP=/bin/fgrep
TO=`formail -x To:`
FROM=`formail -x From:`
LOGFILE=$HOME/ProcMail.log

# :0 - Start of a rule
# hc - h-Look at header, c-continue after rule
# * What to "analyze"
# ? Use Return code of following program
# If "To" not found in ValidAddresses.txt
# | Pipe the header to "UnknownAddresses.txt".

:0hc
* ? (! ( echo ^TO | $FGREP -i -f $HOME/ValidAddresses.txt) )
| echo ^TO >> $HOME/UnknownAddresses.txt
# /dev/null Eventually, we'll turn this on

Re: using .procmail to compare "To" addresses.

SRP wrote:
> I have a "whitelist" type question for any of you procmail gurus out
> there who have a bit of time to help me out.
>
> I have a few hundred valid "To" email addresses for my domain in a text
> file. I'd like to compare all inbound email against this list, passing
> all messages on (for now), but recording the "to" addresses of anybody
> not in the main list in an "UnknownAddresses.txt" file, and send
> everything on to $DEFAULT. I read somewhere that it's better to use ^TO
> instead of $TO for the test, but I'm not sure I understand why...

It's because the "^TO" or "^TO_" strings in procmail regular expressions
are actually macros that expand to a fairly complex regular expression
that should match just about every possible recipient field in the headers.

QUOTE - From man procmailrc under the MISCELLANEOUS section:

If the regular expression contains ‘^TO_’ it will be substituted by
‘(^((Original-)?(Resent-)?(To|Cc|Bcc)|(X-Envelope
|Apparently(-Resent)?)-To):(.*[^-a-zA-Z0-9_.])?)’, which should catch
all destination specifications containing a specific address.

If the regular expression contains ‘^TO’ it will be substituted by
‘(^((Original-)?(Resent-)?(To|Cc|Bcc)|(X-Envelope
|Apparently(-Resent)?)-To):(.*[^a-zA-Z])?)’, which should catch all
destination specifications containing a specific word.

ENDQUOTE

Note the difference between the two macros.

>
> Eventually I'd like to simply bit-bucket any message who's "To" address
> is not in ValidAddresses.txt, but until I've run the "collector" for a
> while, I don't want to turn it on.
>
> I'm will also be creating a single regex rule for "To" addresses that
> are *.com@mydomain.com" that will pass through as valid, but that can
> be after I understand procmail a little better... In other words, when
> I "register" with companies today, I give each company their own email
> address using their domain name, and I want to allow pretty much
> anything.com@mydomain.com to come through $DEFAULT
>
> ----------------------------------------------------
>
> # .procmailrc
> # Any "To" addresses in the text file "ValidAddresses.txt" are passed
> # All other "to" addresses are allowed (for now) and recorded in
> # "UnknownAddresses.txt". Eventually we'll kill all email to addresses
> # that are not in ValidAddresses.txt
> #
> FGREP=/bin/fgrep
> TO=`formail -x To:`
> FROM=`formail -x From:`
> LOGFILE=$HOME/ProcMail.log
>
> # :0 - Start of a rule
> # hc - h-Look at header, c-continue after rule
> # * What to "analyze"
> # ? Use Return code of following program
> # If "To" not found in ValidAddresses.txt
> # | Pipe the header to "UnknownAddresses.txt".
>
> :0hc
> * ? (! ( echo ^TO | $FGREP -i -f $HOME/ValidAddresses.txt) )
> | echo ^TO >> $HOME/UnknownAddresses.txt
> # /dev/null Eventually, we'll turn this on
>

# This is an example of how I would approach this problem using just the
# To: header and perhaps the Cc: header. This method could be adapted
# for the other cases you might want to look at.

# Define a newline variable for use in procmail LOG lines
NL="
"

FORMAIL=formail
LOGFILE=${HOME}/procmail.log
MAILDIR=${HOME}/example_mail_directory

# Define a variable pointing to the file containing a list of valid
# recipient email addresses.
VALIDADDRESSEFILE=${HOME}/ValidAddresses.txt

# pipe the contents of the To: header through grep to see if anything in
# the To: header matches with any fixed string in the file.
# This file should contain email address, one per line, with no leading
# or trailing spaces.
# If so, capture the matching results in the LOCALBUFFER variable.
# Note the backtics used on this line which launch an embedded shell
# script. The contents of the header and body of the message in the
# procmail pipe are available to formail on STDIN during execution
# of this embedded shell script.

LOCALBUFFER=`${FORMAIL} -cx "To:" |grep -Fxif ${VALIDADDRESSEFILE}`

# Test to see if the LOCALBUFFER variable is not empty, in other words
# it contains a match. If so, log the event in the procmail log file,
# and log the contents of the LOCALBUFFER in the LOG entry.
# Also use a procmail filter rule to add an X-header to the email to
# stamp the event onto the email in the procmail pipeline.

:0
* ! LOCALBUFFER ?? ^^^^
{

# Log the event in the procmail.log
LOG="[$$]$_: To: header contains a valid recipient address.
${LOCALBUFFER}${NL}"

# Add an X- header line to the email noting the event.
:0 f
|${FORMAIL} -A "X-recipient-test: To: header contains a valid
recipient address"

}

# Else LOCALBUFFER is empty
:0 E
{

# Log the event in the procmail.log
LOG="[$$]$_: To: header did not contain any valid recipient address${NL}"

# Add and X- header line to the email noting the event.
:0 f
|${FORMAIL} -A "X-recipient-test: To: header did not contain any valid
recipient address."

# Trashcan the email, Note: procmail will stop processing any further
# rules beyond this point when the message is delivered to a file.
# You could change this to /dev/null later when you are happy with the
# way it's working.
:0:
${MAILDIR}/non-valid-recipient
}

# ... do whatever else you want

# deliver the message to the default inbox.
:0:
${DEFAULT}

Hope this helps

--
Garen

Re: using .procmail to compare "To" addresses.

On Dec 20, 8:41 pm, Garen Erdoisa wrote:
> SRP wrote:

> > I have a few hundred valid "To" email addresses for my domain in a text
> > file. I'd like to compare all inbound email against this list, passing
> > all messages on (for now), but recording the "to" addresses of anybody
> > not in the main list in an "UnknownAddresses.txt" file, and send

<<<<>>>>

> LOCALBUFFER=`${FORMAIL} -cx "To:" |grep -Fxif ${VALIDADDRESSEFILE}`
>
> :0
> * ! LOCALBUFFER ?? ^^^^
> {
>
<<<<>>>>
>
> }# Else LOCALBUFFER is empty
> :0 E
> {
>
> # Log the event in the procmail.log
> LOG="[$$]$_: To: header did not contain any valid recipient address${NL}"
>
> # Add and X- header line to the email noting the event.
> :0 f
> |${FORMAIL} -A "X-recipient-test: To: header did not contain any valid recipient address."
> :0:
> ${MAILDIR}/non-valid-recipient

Garen,

Thanks VERY much for the help. I've got this installed and with a few
tweaks (and minor adjustments to line wrapping I didn't catch at the
beginning), it appears to be working almost exactly as I wanted. One
minor thing I did want that's not there (and I'm not sure how to do) is
in the else block, I wanted to write the "To" addresses found into the
LOG (so I can scan it occasionally and decide whether to manually move
those addresses to the valid list.)

if it will help, here's the EXACT script I have installed right now:

Thanks (in advance and in the past) for your help....

Steve

----- Begin.procmailrc -----
# This is an example of how I would approach this problem using just
the
# To: header and perhaps the Cc: header. This method could be adapted
# for the other cases you might want to look at.

# Define a newline variable for use in procmail LOG lines
NL="
"

FORMAIL=formail
LOGFILE=${HOME}/procmail.log
#MAILDIR=${HOME}/example_mail_directory

# Define a variable pointing to the file containing a list of valid
# and bad recipient email addresses.
VALIDADDRESSEFILE=${HOME}/ValidAddresses.txt
BADADDRESSEFILE=${HOME}/BadAddresses.txt

# pipe the contents of the To: header through grep to see if anything
in
# the To: header matches with any fixed string in the file.
# This file should contain email address, one per line, with no leading
# or trailing spaces.
# If so, capture the matching results in the LOCALBUFFER variable.
# Note the backtics used on this line which launch an embedded shell
# script. The contents of the header and body of the message in the
# procmail pipe are available to formail on STDIN during execution
# of this embedded shell script.

LOCALBUFFER=`${FORMAIL} -cx "To:" |grep -Fxif ${VALIDADDRESSEFILE}`

# Test to see if the LOCALBUFFER variable is not empty, in other words
# it contains a match. If so, log the event in the procmail log file,
# and log the contents of the LOCALBUFFER in the LOG entry.
# Also use a procmail filter rule to add an X-header to the email to
# stamp the event onto the email in the procmail pipeline.

:0
* ! LOCALBUFFER ?? ^^^^
{

# Log the event in the procmail.log
LOG="[$$]$_: To: header is VALID. ${LOCALBUFFER}${NL}"

# Add an X- header line to the email noting the event.
:0 f
|${FORMAIL} -A "X-recipient-test: To: header contains a valid
recipient address"

}

# Else LOCALBUFFER is empty
:0 E
{

# Log the event in the procmail.log
LOG="[$$]$_: To: header did not contain any valid recipient
address${NL}"

# Add and X- header line to the email noting the event.
:0 f
|${FORMAIL} -A "X-recipient-test: To: header did not contain any
valid recipient address."

# Trashcan the email, Note: procmail will stop processing any further
# rules beyond this point when the message is delivered to a file.
# You could change this to /dev/null later when you are happy with
the
# way it's working.
:0:
${BADADDRESSFILE}

}

# ... do whatever else you want

# deliver the message to the default inbox.
:0:
${DEFAULT}

Re: using .procmail to compare "To" addresses.

SRP wrote:
>
> On Dec 20, 8:41 pm, Garen Erdoisa wrote:
>> SRP wrote:
>
>>>[SNIP]
>> ${MAILDIR}/non-valid-recipient
>
> Garen,
>
> Thanks VERY much for the help. I've got this installed and with a few
> tweaks (and minor adjustments to line wrapping I didn't catch at the
> beginning), it appears to be working almost exactly as I wanted. One
> minor thing I did want that's not there (and I'm not sure how to do) is
> in the else block, I wanted to write the "To" addresses found into the
> LOG (so I can scan it occasionally and decide whether to manually move
> those addresses to the valid list.)

There are a number of methods you can use to extract the contents of a
header field into a buffer variable, then write that to a file. I prefer
using formail to do this task because formail has switches you can use
to unwrap header lines all onto one line which makes it much easier to
edit and/or do pattern matching on the edited contents of the buffer
variable than it is on the original header line which often wraps across
several lines.

So something like this:

SOMEBUFFER=`${FORMAIL} -cx "To:"`

Will capture the contents of the To: header. However in this case you
also have to be careful because variables are limited in how many
characters they can contain. You can set that limit in procmail recipes
with the LINEBUF variable. I.E:

LINEBUF=65000

would let you capture up to 64999 characters into any variables used as
a buffer. If you exceed that limit, procmail will complain about it.

If you do this instead:

LOG=`${FORMAIL} -cx "To: >>${HOME}/somefilename.txt`

It can avoid this problem and let you capture pretty much anything you
want directly into a data file. You might in some cases want to also
pipe the output through sed first to make formating adjustments before
appending the contents of the header to your file.

LOG=`${FORMAIL} -cx "To:" |sed -e 's/[ \t]\+/ /g'
>>${HOME}/somefilename.txt`

The above GNU sed edit will replace multiple tabs and spaces with just
single spaces. If you are using a version of sed that is non GNU, then
you may have to use the actual tab character instead of the \t in that
position, and replace the \+ with \{1,\} thus:

LOG=`${FORMAIL} -cx "To:" |sed -e 's/[ ]\{1,\}/ /g'
>>${HOME}/somefilename.txt`

The above would be for a POSIX version of sed but does the same thing.

Another method in procmail you can use is to capture parts of a regular
expression that match a portion of your pattern into the MATCH variable
using the procmail macro \/ which is embedded in a regular expression.

example:

:0
* ^To: \/.*
{
LOCALBUFFER=${MATCH}
LOG="[$$]$_: LOCALBUFFER=${LOCALBUFFER}${NL}"
}

The above would also write the contents of the To: header into your
procmail log file, but it would be a bit more messy than the first
method if the header line wraps. Which method I would choose depends a
lot on what I'm trying to accomplish in the recipe. Quite often I do sed
edits and other operations on the string before sending it to a data file.

>
> if it will help, here's the EXACT script I have installed right now:

Ok, I made a couple of adjustments. (see below) You seem to be confusing
procmail delivery of an email to a mail folder with just writing some
pieces of extracted data to a .txt file. So, I went ahead and made that
correction as well.

Using a trashcan folder will allow you the opportunity to recover from
any possible errors. On my system I do this with a backup cache
directory that saves like the last 500 messages or so on my main
account. There is an example in man procmailex that shows how to setup
procmail recipe to do a backup cache. It could just as easily be done
with a mail folder that you periodically delete messages from.

>
> Thanks (in advance and in the past) for your help....

Quite welcome.

>
> Steve
>
> [SNIP]

----- Begin.procmailrc -----
# This is an example of how I would approach this problem using just
the
# To: header and perhaps the Cc: header. This method could be adapted
# for the other cases you might want to look at.

# Define a newline variable for use in procmail LOG lines
NL="
"

FORMAIL=formail
LOGFILE=${HOME}/procmail.log

# Set a large value to reduce the likelihood that capturing
# text into variables will exceed this character limit.
LINEBUF=65000

#MAILDIR=${HOME}/example_mail_directory

# Define a variable pointing to the file containing a list of valid
# and bad recipient email addresses.
VALIDADDRESSEFILE=${HOME}/ValidAddresses.txt
BADADDRESSEFILE=${HOME}/BadAddresses.txt
TRASHCAN=${HOME}/trashcan

# pipe the contents of the To: header through grep to see if anything
in
# the To: header matches with any fixed string in the file.
# This file should contain email address, one per line, with no leading
# or trailing spaces.
# If so, capture the matching results in the LOCALBUFFER variable.
# Note the backtics used on this line which launch an embedded shell
# script. The contents of the header and body of the message in the
# procmail pipe are available to formail on STDIN during execution
# of this embedded shell script.

LOCALBUFFER=`${FORMAIL} -cx "To:" |grep -Fxif ${VALIDADDRESSEFILE}`

# Test to see if the LOCALBUFFER variable is not empty, in other words
# it contains a match. If so, log the event in the procmail log file,
# and log the contents of the LOCALBUFFER in the LOG entry.
# Also use a procmail filter rule to add an X-header to the email to
# stamp the event onto the email in the procmail pipeline.

:0
* ! LOCALBUFFER ?? ^^^^
{

# Log the event in the procmail.log
LOG="[$$]$_: To: header is VALID. ${LOCALBUFFER}${NL}"

# Add an X- header line to the email noting the event.
:0 f
|${FORMAIL} -A "X-recipient-test: To: header contains a valid
recipient address"

}

# Else LOCALBUFFER is empty
:0 E
{

# Log the event in the procmail.log
LOG="[$$]$_: To: header did not contain any valid recipient
address${NL}"

# Unfold the To: header and append it's contents
# to the BADADDRESSFILE

LOG=`${FORMAIL} -cx "To:" >>${BADADDRESSFILE}`

# Another way to do this directly to the procmail log.
LOCALBUFFER=`${FORMAIL} -cx "To:"`
LOG="[$$]$_:$ To: ${LOCALBUFFER}{NL}"

# Add and X- header line to the email noting the event.
:0 f
|${FORMAIL} -A "X-recipient-test: To: header did not contain any
valid recipient address."

# Trashcan the email, Note: procmail will stop processing any further
# rules beyond this point when the message is delivered to a file.
# You could change this to /dev/null later when you are happy with
the
# way it's working.
:0:
${TRASHCAN}

}

# ... do whatever else you want

# deliver the message to the default inbox.
:0:
${DEFAULT}