Procmail: Why are these users getting multiple copies?

Hope this is a good place for this ... the procmail list didn't help
....

Problem: Messages matching rule are delivered multiple times

Scenario:
- UserA is subscribed to a newsletter ("PATTERN" is always in the
Subject)
- UserB and UserC want a copy whenever it comes in (they can't
subscribe to it)
- Here's my /etc/procmailrc rule for appending these messages to
UserB/C mailboxes:

:0
* ^Subject.*PATTERN
{
:0c:
/var/spool/mail/UserB
:0c:
/var/spool/mail/UserC
}

- UserA gets their single copy just fine
- UserB and UserC each get 2 copies

Can anyone explain why UserB and UserC get multiple copies?

I have tried this rule with and without locking and with and without
cloning.
- Without locking is exactly the same as with locking ... multiple
copies.
- Without cloning halts distribution after appending to UserB's mailbox
(of course).

I appreciate any insight into this. Thanks.

Re: Procmail: Why are these users getting multiple copies?

stupidscript@hotmail.com wrote:
> Hope this is a good place for this ... the procmail list didn't help
> ...
>
> Problem: Messages matching rule are delivered multiple times
>
> Scenario:
> - UserA is subscribed to a newsletter ("PATTERN" is always in the
> Subject)
> - UserB and UserC want a copy whenever it comes in (they can't
> subscribe to it)
> - Here's my /etc/procmailrc rule for appending these messages to
> UserB/C mailboxes:
>
> :0
> * ^Subject.*PATTERN
> {
> :0c:
> /var/spool/mail/UserB
> :0c:
> /var/spool/mail/UserC
> }
>
> - UserA gets their single copy just fine
> - UserB and UserC each get 2 copies
>
> Can anyone explain why UserB and UserC get multiple copies?

I suspect that you have UserB and/or UserC either aliased to receive a
copy in /etc/aliases (sendmail), or being forwarded by UserA to UserB
and UserC to via a ${HOME}/.forward mechanism.

Either of those would have the following happen:

Email sent to UserA

/etc/procmailrc (called by root) delivers courtesy copies directly to
the inboxes of UserB and UserC That's the first instance of the copy.

/etc/procmailrc then delivers one copy to UserA

UserA is aliased to, or told to forward to, UserB, and UserC

/etc/procmailrc is called again to handle the delivery to UserB and
UserC (once for each user)

/etc/procmailrc (UserB) delivers copies to UserB and UserC then
Delivers a Default to UserB

At this point, UserA has one copy
UserB has 3 copies
UserC has 2 copies

/etc/procmailrc (UserC) delivers copies to UserB and UserC then
delivers a default to UserC

At this point UserA has one copy
UserB has 4 copies
UserC has 4 copies

LOL

That would be my best guess based on what you described. I was able to
reproduce this effect here by adding email addresses to ${HOME}/.forward
for UserA or adding email addresses to /etc/aliases for UserA

>[snip]

--
Garen

Re: Procmail: Why are these users getting multiple copies?

Aaaaaaa....HA!

Indeed I do have both UserB and UserC aliased to a common ("staff")
mailbox for archival purposes. Excellent deduction, and DOH!

Reading through the mailing list seems to indicate that there is
current activity trying to figure out a way to "tag" messages already
processed by any rule, and that it is an exceedingly difficult
proposition. (i.e. process a message, stash its id, on subsequent
iterations of the rule check for id and bypass if already ...
yaddayadda).

Hmmm. I really appreciate your insight, Garen. I'm off to figure out a
different way to get mail to the right people. Thank you!

James

Re: Procmail: Why are these users getting multiple copies?

stupidscript@hotmail.com wrote:
> Aaaaaaa....HA!
>
> Indeed I do have both UserB and UserC aliased to a common ("staff")
> mailbox for archival purposes. Excellent deduction, and DOH!
>
> Reading through the mailing list seems to indicate that there is
> current activity trying to figure out a way to "tag" messages already
> processed by any rule, and that it is an exceedingly difficult
> proposition. (i.e. process a message, stash its id, on subsequent
> iterations of the rule check for id and bypass if already ...
> yaddayadda).

Pretty simple to do really. The idea you need is just to create an sha1
hash of the message body and combine that with the first external IP
number for your id string, then rotate that through a cache file saving
the last, say 500 of them.

Here's an example (using procmail on a Linux system)
This is intended to be run out of a ${HOME}/.procmailrc but could be
adapted for a system wide use.

-=-=-=-=-
# You need to have first extracted the FIRST_EXTERNAL_IP before
# using this procmail recipe.

# Set a large procmail linebuffer size so the DIGESTCACHE variable
# can hold 500 lines worth of cache information.
# if 60000 doesn't work on your system, try lowering the size, but also
# only save about 250 lines instead of 500 in the cache.

LINEBUF=60000

# Define a newline character for use in procmail LOG lines.
NL="
"

# Define the command to use to generate the message body digest
DIGEST='/usr/bin/openssl sha1'

# Note the backticks that launch an embedded shell script here.
# This creates a string like this:
# <4404ff446015133ca9972023a5b1af9876f788c8@[192.168.10.100]>
# ie:

MESSAGEBODYDIGEST=`${FORMAIL} -I "" |${DIGEST} |sed -e "s/[ -]\+//g ;
s/^/ /"`

# create the cache file if it doesn't exist then
# see if the above string is in the cache.
:0
* ? touch ${HOME}/.digestcache
* ? grep -Fx "${MESSAGEBODYDIGEST}" ${HOME}/.digestcache
{
DIGESTDUPLICATE=yes
LOG="[$$]$_: Found ${MESSAGEBODYDIGEST} cached.
Message-ID:${MESSAGEID}${NL}"
}
# Else
:0 E
{
DIGESTDUPLICATE=no

# Alternate way to save only the last 500 strings.
# DIGESTCACHE=`(echo "${MESSAGEBODYDIGEST}" ; cat ${HOME}/.digestcache
)|sed -e '501,$d'`

# save the last 500 strings.
DIGESTCACHE=` echo "${MESSAGEBODYDIGEST}" ; head -qn 499
${HOME}/.digestcache `

:0 Wic: ${HOME}/.digestcache.lock
|( ${FORMAIL} -X "" -I "" ; echo "${DIGESTCACHE}" ) >${HOME}/.digestcache

# also do this if the last recipe succeeded
:0 a
{ LOG="[$$]$_: Wrote ${MESSAGEBODYDIGEST} to ${HOME}/.digestcache ${NL}" }

# else
:0 E
{ LOG="[$$]$_: Error updating ${HOME}/.digestcache with
${MESSAGEBODYDIGEST}${NL}" }
}

# Test the variable. If it contains yes, then bitbucket the duplicate
# message.
:0
* DIGESTDUPLICATE ?? ^yes$
{
LOG="[$$]$_: Digest Cache - Duplicate Message detected. Filed in
/dev/null${NL}"
LOGABSTRACT=no
:0
/dev/null
}

>
> Hmmm. I really appreciate your insight, Garen. I'm off to figure out a
> different way to get mail to the right people. Thank you!
Quite welcome. Glad it helped.

>
> James
>

--
Garen

Re: Procmail: Why are these users getting multiple copies?

Aannnd this brings up the question (whose answer may be self-evident to
someone brighter than me):

When the rule appends a message to a mailbox, is this being treated as
a "forward", and hence the multiple deliveries?

I had thought:

Mail comes into system -> Hits rule -> Appends to UserB -> Appends to
UserC -> Continues to UserA -> End of delivery

The aliasing issue seems to imply:

Mail comes into system -> Hits rule -> Appends to UserB -> Hits rule ->
Appends to UserB -> Appends to UserC -> Hits rule -> Appends to UserB
-> Appends to UserC -> Continues to UserA -> (possibly more
rule-hitting) -> End of delivery

Note that this is not being "delivered" to UserB and UserC, rather it
is being appended as text to their mailbox files. Does that trigger
another round of rule-hitting?

I'm just trying to figure out why the aliases are being considered at
all. Thanks again.

(I think I'm going to add a "To" condition so if the copy is being
"delivered" to UserB and UserC the rule will only trigger when the
original message comes in "To" the original receiver ... hopefully. ;)

Re: Procmail: Why are these users getting multiple copies?

stupidscript@hotmail.com wrote:
> [snip]
> I'm just trying to figure out why the aliases are being considered at
> all. Thanks again.

It's because the aliasing is a function of sendmail which is an MTA
(mail transport agent) while /etc/procmailrc is a function of procmail
which is an MDA (Mail delivery agent) that is called by sendmail. (or
other MTAs)

So, that means that there are two different programs that are
interacting in this particular scenario, not one.

Each time sendmail is called, it in turn hands off to a new instance of
procmail, which runs the /etc/procmailrc (as root) for each instance
that it's called.

> [snip]

--
Garen

Re: Procmail: Why are these users getting multiple copies?

Lovely! I get it ... finally. And your hash code is inspiring! I'll be
giving that a whack ...

I really appreciate your help, Garen. Have a great day!

James