Filtering text embedded in HTML with procmail

(I may have accidently posted this twice - once before the posting was
finished. Sorry if there is a duplicate.)

Hello. I'm receiving spam from a spammer who embeds text in HTML. For
some reason, I can't filter anything embedded in HTML using procmail.
As a test, I wrote a few recipies with the following results (I am
trying to filter out the word snoozlewacker in my test):

If snoozlewacker is not embedded in html, then:

:0B
* .*snoozlewacker.*
/dev/null

or

:0
* .*snoozlewacker.*
/dev/null

filter the mail out fine. But if I embed the word in HTML like this:


Then neither of those recipies work.

Any idea why my recipies don't work, and how I can fix this?

Thanks for your help!
Michael

Re: Filtering text embedded in HTML with procmail

On 21 Jan 2006 08:45:04 -0800, nooneinparticular314159@yahoo.com wrote:
> (I may have accidently posted this twice - once before the posting was
> finished. Sorry if there is a duplicate.)
>
> Hello. I'm receiving spam from a spammer who embeds text in HTML. For
> some reason, I can't filter anything embedded in HTML using procmail.
> As a test, I wrote a few recipies with the following results (I am
> trying to filter out the word snoozlewacker in my test):
>
> If snoozlewacker is not embedded in html, then:
>
>:0B
> * .*snoozlewacker.*
> /dev/null
>
> or
>
>:0
> * .*snoozlewacker.*
> /dev/null
>
> filter the mail out fine. But if I embed the word in HTML like this:
>
>
> Then neither of those recipies work.
>
> Any idea why my recipies don't work, and how I can fix this?

Have you looked at the raw email? Not as your MUA renders it. In pine
use the "H" option when reading the email. Poke around in your MUA to
find how you do it in your case. Absent that, "S"ave the email to a
test email folder -- then, at a command line, dump the file (folder) and
inspect the payload. My guess is your "snoozlewacker" is encoded thus:
"snoozlewacker" , or the somesuch.

Then you start tediouly making procmail recipes for all the html
obsfucation you catch the slimeballs using.

HTH
Jonesy
--
Marvin L Jones | jonz | W3DHJ | linux
Pueblo, Colorado | @ | Jonesy | OS/2 __
38.24N 104.55W | config.com | DM78rf | SK

Re: Filtering text embedded in HTML with procmail

Yes. I do use the h option in pine to do just that. The html in there
is nasty. But shouldn't procmail ignore everything except for
snoozlewacker?

In this case, it isn't that words are being broken up. The words in
the html are being ignored, as in my example above. (I constructed the
html precisely to test this.) When I wrote snoozlewacker outside of
html, it was filtered properly. Wrapping it in html as shown above
rendered procmail useless.

Thanks,
Michael

Re: Filtering text embedded in HTML with procmail

On 21 Jan 2006 09:59:16 -0800, nooneinparticular314159@yahoo.com wrote:
> Yes. I do use the h option in pine to do just that. The html in there
> is nasty. But shouldn't procmail ignore everything except for
> snoozlewacker?

Why would you believe that procmail is some sort of hyper-text markup
language decoder/processor?

> In this case, it isn't that words are being broken up. The words in
> the html are being ignored, as in my example above.

WHAT "ABOVE"!? You're posting -- from Google, sigh... -- without
quoting. You wrongly assume everybody see's what you see. Hint: NOBODY
on usenet sees exactly what ANYBODY ELSE sees. I only pull un-read
posts with slrn. I filter and kill extensively with my score file. I
only have so much time that I can (should) spend in these waters. In
many other ng's I've started killfiling all followups posted from Google.
I'll probably start doing that here. The noise-to-signal ratio
has started to grow a little too large here.

Go ahead, keep posting from Google. Pretty soon the Fed's will collect
all the records on your _usenet_ activity, too.

Jonesy

Re: Filtering text embedded in HTML with procmail

On comp.mail.misc, in <1137861904.893019.193230@g47g2000cwa.googlegroups.com>, "nooneinparticular314159@yahoo.com" wrote:



I'm with Allodoxaphobia: I usually don't even read the articles
of people posting through GG unless the subject indicates that
they are asking about setting up networking on Linux or how to
use a real newsreader.

comp.os.linux.networking
comp.os.linux.misc
news.software.readers

This is an interesting challenge you have here. I can think of
one way to solve it.

And when you have decided to actually join the Usenet, instead of
just attempting to exploit it, I'll give you all the help I can.

If you immediately start posting with a newsreader, and I don't
see any posts on the above groups from this very same alias
first, I'm going to know that you are a troll, and won't help
you to save your life.


[Note: I don't read the posts of "Sam" or his numerous
sockpuppets or his 'friends', nor any responses to them.]

Alan


--
http://home.earthlink.net/~alanconnor/contact.html
see also: links.html and newsfilter.html
Other URLs of possible interest in my headers.

Re: Filtering text embedded in HTML with procmail

This is a MIME GnuPG-signed message. If you see this text, it means that
your E-mail or Usenet software does not support MIME signed messages.
The Internet standard for MIME PGP messages, RFC 2015, was published in 1996.
To open this message correctly you will need to install E-mail or Usenet
software that supports modern Internet standards.

--=_mimegpg-commodore.email-scan.com-5996-1137898696-0001
Content-Type: text/plain; format=flowed; charset="US-ASCII"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit

Usenet Beavis writes:

> On comp.mail.misc, in <1137861904.893019.193230@g47g2000cwa.googlegroups.com>, "nooneinparticular314159@yahoo.com" wrote:
>
>
>
> I'm with Allodoxaphobia: I usually don't even read the articles
> of people posting through GG unless the subject indicates that

Beavis, it looks like you don't read any articles in this newsgroup at all.
Yet, somehow, you somehow figured out what they all say.

> And when you have decided to actually join the Usenet, instead of
> just attempting to exploit it, I'll give you all the help I can.

That'll work only if you remember the cardinal rule of comp.mail.misc:
whatever advice Beavis gives, the correct answer always lies 180 degrees to
the opposite.

> If you immediately start posting with a newsreader, and I don't
> see any posts on the above groups from this very same alias
> first, I'm going to know that you are a troll, and won't help
> you to save your life.

Pop quiz, Beavis: what am I using to post messages to this newsfroup?

> [Note: it's not my fault that I'm a complete dumbass. I was dropped on my
> head as a child. See http://www.pearlgates.net/nanae/kooks/alanconnor for
> more information]
>
> Beavis


--=_mimegpg-commodore.email-scan.com-5996-1137898696-0001
Content-Type: application/pgp-signature
Content-Transfer-Encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.7 (GNU/Linux)

iD8DBQBD0vTIx9p3GYHlUOIRAk8+AJ9kzeqfdJZgP8g5PSk0JRhLBi3maQCf SQK3
gXBHbyd5yd3F2EFwn2uSBzs=
=IBfB
-----END PGP SIGNATURE-----

--=_mimegpg-commodore.email-scan.com-5996-1137898696-0001--

Re: Filtering text embedded in HTML with procmail

Ok. That's just silly. I'm not asking a question about Linux or news
readers, I'm asking about procmail filtering. So posting on those
newsgroups would be off topic. Why would you care what news reader I
use to post? FYI, I use google for news postings because it provides a
convenient way to send spam to a spam account, and has excellent search
capabilities. But that shouldn't matter to you. Whether or not you
know the answer to my question should. If you know a solution to my
procmail problem, please post it. That would be very helpful.
Otherwise, I'm sure there is an alt.flame.wrong.newsreader group
somewhere.

Re: Filtering text embedded in HTML with procmail

nooneinparticular314159@yahoo.com wrote:
> (I may have accidently posted this twice - once before the posting was
> finished. Sorry if there is a duplicate.)
>
> Hello. I'm receiving spam from a spammer who embeds text in HTML. For
> some reason, I can't filter anything embedded in HTML using procmail.
> As a test, I wrote a few recipies with the following results (I am
> trying to filter out the word snoozlewacker in my test):
>
> If snoozlewacker is not embedded in html, then:
>
> :0B
> * .*snoozlewacker.*
> /dev/null
>
> or
>
> :0
> * .*snoozlewacker.*
> /dev/null
>
> filter the mail out fine. But if I embed the word in HTML like this:
>
>
> Then neither of those recipies work.

Your example worked fine as written when I tested it on a Fedora Linux
system.

One thing you might check that I've run into before is to make sure when
you write a procmail recipe that it doesn't have any CR's (carriage
return) characters at the end of the lines. The end of line characters
in procmail recipes should be just a linefeed character.

Also, make sure there are no trailing spaces in your regular expression
pattern matches unless they are intended. That can cause a pattern match
to fail. If you do intend to have a trailing space in a pattern match,
enclose it in parentheses thus. ( )
This way you'll know the next time you look at it that the trailing
space was intended.

When you are testing a recipe try setting VERBOSE=yes and monitor the
procmail log. That can help to identify issues.

>
> Any idea why my recipies don't work, and how I can fix this?
>
> Thanks for your help!
> Michael
>

Garen

[OT-netiquette] Re: Filtering text embedded in HTML with procmail

* nooneinparticular314159@yahoo.com wrote:
> Ok. That's just silly. I'm not asking a question about Linux or news
> readers, I'm asking about procmail filtering. So posting on those
> newsgroups would be off topic. Why would you care what news reader I
> use to post? FYI, I use google for news postings because it provides a
> convenient way to send spam to a spam account, and has excellent search
> capabilities. But that shouldn't matter to you. Whether or not you
> know the answer to my question should. If you know a solution to my
> procmail problem, please post it. That would be very helpful.
> Otherwise, I'm sure there is an alt.flame.wrong.newsreader group
> somewhere.

It's not so much about your newsreader, it's about how it doesn't play
well with everyone else. For example, you've posted the above with
absolutely no quoting of the post you are following up. Many readers
will read the above and wonder what the hell you are talking about and
to whom. You did the same to Allodaxaphobia.

Quote correctly, comply with common usenet netiquette guidelines, and
you'll get much better responses.

--
Troy Piggins
Ubuntu 5.10 pkgs : kernel 2.6.12-9-386, postfix 2.2.4, procmail 3.22
Compiled from src : slrn 0.9.8.1/rt (score_color patch), mutt 1.5.10i
vim 6.4

Re: Filtering text embedded in HTML with procmail

On 2006-01-22, nooneinparticular314159@yahoo.com wrote:
> Ok. That's just silly.

What's "just silly"? Please read .

...

Ah! You're replying to AC. I can understand not wanting to repost
anything of his.

> I'm not asking a question about Linux or news readers, I'm asking
> about procmail filtering. So posting on those newsgroups would be
> off topic. Why would you care what news reader I use to post? FYI, I
> use google for news postings because it provides a convenient way to
> send spam to a spam account, and has excellent search capabilities.

The Google posting interface to Usenet newgroups is broken, but can
be fixed. If you follow the instructions in the URL above, there's
nothing wring with posting from Google.

> But that shouldn't matter to you. Whether or not you know the answer
> to my question should.

Most people just killfile AC.

> If you know a solution to my procmail problem, please post it. That
> would be very helpful. Otherwise, I'm sure there is an
> alt.flame.wrong.newsreader group somewhere.


--
Chris F.A. Johnson, author |
Shell Scripting Recipes: | My code in this post, if any,
A Problem-Solution Approach | is released under the
2005, Apress | GNU General Public Licence

Re: Filtering text embedded in HTML with procmail

I agree that google should implement quoting (and as you can see, I've
just implemented a different quoting method below). But don't confuse
the people who use it to post (for very good reasons) with google.
That's like saying that anyone who uses IE shoudn't be able to see a
website (or conversely, people who use Firefox) because you have an
issue with one of those browsers. I'd personally like everyone to stop
sending me HTML mail because I use a text based mail reader, but I
won't reject mail from people who do. As for a killfile for AC, sounds
like a great idea. :-)

Quoting Chris Johnson :-)

On 2006-01-22, nooneinparticular314...@yahoo.com wrote:
> Ok. That's just silly.

What's "just silly"? Please read .

...

Ah! You're replying to AC. I can understand not wanting to repost
anything of his.

> I'm not asking a question about Linux or news readers, I'm asking
> about procmail filtering. So posting on those newsgroups would be
> off topic. Why would you care what news reader I use to post? FYI, I
> use google for news postings because it provides a convenient way to
> send spam to a spam account, and has excellent search capabilities.

The Google posting interface to Usenet newgroups is broken, but can
be fixed. If you follow the instructions in the URL above, there's
nothing wring with posting from Google.

> But that shouldn't matter to you. Whether or not you know the answer
> to my question should.

Most people just killfile AC.

> If you know a solution to my procmail problem, please post it. That
> would be very helpful. Otherwise, I'm sure there is an
> alt.flame.wrong.newsreader group somewhere.

Re: Filtering text embedded in HTML with procmail

On 2006-01-23, nooneinparticular314159@yahoo.com wrote:
> I agree that google should implement quoting (and as you can see, I've
> just implemented a different quoting method below).

[please don't top post]

You can quote correctly with Google. Just follow the instructions
in the URL I posted.

Your method does not work in most newsreaders.

> But don't confuse the people who use it to post (for very good
> reasons) with google.

Why would anyone by confused by simple instructions?

> That's like saying that anyone who uses IE shoudn't be able to see a
> website (or conversely, people who use Firefox) because you have an
> issue with one of those browsers.

> I'd personally like everyone to stop sending me HTML mail because I
> use a text based mail reader, but I won't reject mail from people
> who do.

I do reject it. I scan the rejected list every few days; there have
only been 2 or 3 legit. HTML e-mails in the last 3 years.


--
Chris F.A. Johnson, author |
Shell Scripting Recipes: | My code in this post, if any,
A Problem-Solution Approach | is released under the
2005, Apress | GNU General Public Licence

Re: Filtering text embedded in HTML with procmail

nooneinparticular314159@yahoo.com wrote:
> (I may have accidently posted this twice - once before the posting was
> finished. Sorry if there is a duplicate.)
>
> Hello. I'm receiving spam from a spammer who embeds text in HTML. For
> some reason, I can't filter anything embedded in HTML using procmail.
> As a test, I wrote a few recipies with the following results (I am
> trying to filter out the word snoozlewacker in my test):
>
> If snoozlewacker is not embedded in html, then:
>
> :0B
> * .*snoozlewacker.*
> /dev/null
>
> or
>
> :0
> * .*snoozlewacker.*
> /dev/null
>
> filter the mail out fine. But if I embed the word in HTML like this:
>
>
> Then neither of those recipies work.
>
> Any idea why my recipies don't work, and how I can fix this?

Try just
* snoozlewacker
without '.*'. Also, examine the raw email to make sure it's not
encoded, say, in base64 or print-quotable.

--
William Park , Toronto, Canada
ThinFlash: Linux thin-client on USB key (flash) drive
http://home.eol.ca/~parkw/thinflash.html
BashDiff: Super Bash shell
http://freshmeat.net/projects/bashdiff/

Re: Filtering text embedded in HTML with procmail

nooneinparticular314159@yahoo.com wrote:

> Yes. I do use the h option in pine to do just that. The html in there
> is nasty. But shouldn't procmail ignore everything except for
> snoozlewacker?
>
> In this case, it isn't that words are being broken up. The words in
> the html are being ignored, as in my example above. (I constructed the
> html precisely to test this.) When I wrote snoozlewacker outside of
> html, it was filtered properly. Wrapping it in html as shown above
> rendered procmail useless.
>
> Thanks,
> Michael
>
Procmail tries to match the rule exactly as you listed it. When you
looked at the mailing, you saw the pattern because you remember what you
saw just before, procmail is not capable of that.
i.e. if your rule says look for snooz, while to you s n o o z looks the
same, it would not match the pattern.
same with s<>n<>o<>o<>z.

You can use procmail filtering to strip all <.+> entries and see if your
pattern is matched, but you might be better off using scoring to see if
the number of HTML entries is excesive to a mailing based on its length.

The problem you might run into is false positive if legitamate mailings
you receive have HTML content.

AK

Re: Filtering text embedded in HTML with procmail

Alan Connor wrote:

> On comp.mail.misc, in <1137861904.893019.193230@g47g2000cwa.googlegroups.com>, "nooneinparticular314159@yahoo.com" wrote:
>
>
>
> I'm with Allodoxaphobia: I usually don't even read the articles
> of people posting through GG unless the subject indicates that
> they are asking about setting up networking on Linux or how to
> use a real newsreader.
>
> comp.os.linux.networking
> comp.os.linux.misc
> news.software.readers
>
> This is an interesting challenge you have here. I can think of
> one way to solve it.
>
> And when you have decided to actually join the Usenet, instead of
> just attempting to exploit it, I'll give you all the help I can.
>
> If you immediately start posting with a newsreader, and I don't
> see any posts on the above groups from this very same alias
> first, I'm going to know that you are a troll, and won't help
> you to save your life.
>
>
> [Note: I don't read the posts of "Sam" or his numerous
> sockpuppets or his 'friends', nor any responses to them.]
>
> Alan
>
>

Alan,

I think you just proved that all your claims about what you do not read
are a bunch of junk. When I saw Allodoxaphobia's post, I thought someone
was mocking you. And then you posted affirming the rediculous notion you
aspouse that USENET needs to conform to your idea of what news reader
one can or should use to post.

AK

Re: Filtering text embedded in HTML with procmail

On comp.mail.misc, in , "AK" wrote:
> Path: newsspool1.news.pas.earthlink.net!stamper.news.pas.earthlink .net!stamper.news.atl.earthlink.net!elnk-atl-nf2!newsfeed.ea rthlink.net!nx01.iad01.newshosting.com!newshosting.com!216.1 96.98.140.MISMATCH!border1.nntp.dca.giganews.com!nntp.gigane ws.com!local01.nntp.dca.giganews.com!nntp.comcast.com!news.c omcast.com.POSTED!not-for-mail
> NNTP-Posting-Date: Tue, 24 Jan 2006 22:36:58 -0600
> Date: Tue, 24 Jan 2006 23:37:05 -0500
> From: AK
> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.12) Gecko/20050915
> X-Accept-Language: en-us, en
> MIME-Version: 1.0
> Newsgroups: comp.mail.misc
> Subject: Re: Filtering text embedded in HTML with procmail
> References: <1137861904.893019.193230@g47g2000cwa.googlegroups.com>
> In-Reply-To:
> Content-Type: text/plain; charset=us-ascii; format=flowed
> Content-Transfer-Encoding: 7bit
> Message-ID:
> Lines: 46
> NNTP-Posting-Host: 68.81.191.195
> X-Trace: sv3-ShQyVxcCveM8OYzof0s7H7CxzI1X1Qxuyo2isjIEj84Cu4ZHzIv4rSE4 vZD22k5kFDH/S+BbqgAzKLq!dY0+o+l8YLeel0kf6Rx8QnENFK5YSWTeeGnD LB+DqFHfq2UM2N2OyNsexZIJ+YOWZ5XlKD8ESTIJ!81z8VHWs0EyJAg==
> X-Complaints-To: abuse@comcast.net
> X-DMCA-Complaints-To: dmca@comcast.net
> X-Abuse-and-DMCA-Info: Please be sure to forward a copy of ALL headers
> X-Abuse-and-DMCA-Info: Otherwise we will be unable to process your complaint properly
> X-Postfilter: 1.3.32
> Xref: news.earthlink.net comp.mail.misc:76225
> X-Received-Date: Tue, 24 Jan 2006 20:36:59 PST (newsspool1.news.pas.earthlink.net)
>
>
>
> Alan Connor wrote:
>
>> On comp.mail.misc, in
>> <1137861904.893019.193230@g47g2000cwa.googlegroups.com>,
>> "nooneinparticular314159@yahoo.com" wrote:
>>
>>
>>
>> I'm with Allodoxaphobia: I usually don't even read the
>> articles of people posting through GG unless the subject
>> indicates that they are asking about setting up networking on
>> Linux or how to use a real newsreader.
>>
>> comp.os.linux.networking comp.os.linux.misc
>> news.software.readers
>>
>> This is an interesting challenge you have here. I can think of
>> one way to solve it.
>>
>> And when you have decided to actually join the Usenet, instead
>> of just attempting to exploit it, I'll give you all the help I
>> can.
>>
>> If you immediately start posting with a newsreader, and I
>> don't see any posts on the above groups from this very same
>> alias first, I'm going to know that you are a troll, and won't
>> help you to save your life.
>>
>>
>> [Note: I don't read the posts of "Sam" or his numerous
>> sockpuppets or his 'friends', nor any responses to them.]
>>
>> Alan
>>
>

>
> Alan,
>
> I think you just proved that all your claims about what you
> do not read are a bunch of junk. When I saw Allodoxaphobia's
> post, I thought someone was mocking you. And then you posted
> affirming the rediculous notion you aspouse that USENET needs
> to conform to your idea of what news reader one can or should
> use to post.
>

I said that _I_ don't help people who post through GG.

You can do what you want. While you live with the fact that
I don't give a tinker's dam_ what you think.

I don't help people who post through GG because they almost never
help anyone else. They just take and take and take.

People who use real newsreaders give and take, as a rule.
They have a commitment to the Usenet.

> AK
>
>
>

I see. An ignorant windoze-weenie who just doesn't realize what a
pathetic operating system he is using, and thinks that all other
operating systems have the same limitations.

http://slrn.sourceforge.net/docs/README.offline

Welcome to my killfile.

[Note: I don't read the posts of "Sam" or his numerous
sockpuppets or his 'friends', nor any responses to them.]

Alan

--
http://home.earthlink.net/~alanconnor/contact.html
see also: links.html and newsfilter.html
Other URLs of possible interest in my headers.

Re: Filtering text embedded in HTML with procmail

This is a MIME GnuPG-signed message. If you see this text, it means that
your E-mail or Usenet software does not support MIME signed messages.
The Internet standard for MIME PGP messages, RFC 2015, was published in 1996.
To open this message correctly you will need to install E-mail or Usenet
software that supports modern Internet standards.

--=_mimegpg-commodore.email-scan.com-9175-1138190397-0003
Content-Type: text/plain; format=flowed; charset="US-ASCII"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit

Usenet Beavis writes:

> I said that _I_ don't help people who post through GG.

So what? You also said that you've met Bigfoot, and got molested by Xena
the Warrior Princess.

> You can do what you want. While you live with the fact that
> I don't give a tinker's dam_ what you think.

Obviously you do, at least enough to make a reply to a post that you did not
read.

> People who use real newsreaders give and take, as a rule.
> They have a commitment to the Usenet.

And you have a commitment to being our village idiot.

> I see. An ignorant windoze-weenie who just doesn't realize what a
> pathetic operating system he is using, and thinks that all other
> operating systems have the same limitations.

Speaking of that, Beavis, have you yet managed to "hack" my C: drive?

> Welcome to my killfile.

A fate worse than death. *shudder*


> [Note: it's not my fault that I'm a complete dumbass. I was dropped on my
> head as a child. See http://www.pearlgates.net/nanae/kooks/alanconnor for
> more information]
>
> Beavis



--=_mimegpg-commodore.email-scan.com-9175-1138190397-0003
Content-Type: application/pgp-signature
Content-Transfer-Encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.7 (GNU/Linux)

iD8DBQBD12g9x9p3GYHlUOIRAj3XAJ9xwXPYRIWy8pU3EZ2JXJU7Hq26zQCc CMYz
hnIZAhyAx3vOhpWPlWaoGco=
=3QKM
-----END PGP SIGNATURE-----

--=_mimegpg-commodore.email-scan.com-9175-1138190397-0003--

Re: Filtering text embedded in HTML with procmail

Thanks, William! Your solution appears to be the correct one! Why
does that work? What is the difference between

* snoozlewacker
and
* .*snoozlewacker.*

?

Why doesn't * .*snoozlewacker.* work?

Thanks!

Michael

Re: Filtering text embedded in HTML with procmail

nooneinparticular314159@yahoo.com wrote:

> Thanks, William! Your solution appears to be the correct one! Why
> does that work? What is the difference between
>
> * snoozlewacker
> and
> * .*snoozlewacker.*
>
> ?
>
> Why doesn't * .*snoozlewacker.* work?
>
> Thanks!
>
> Michael
>

In short, the preceding .* and the following .* means that it might
match accross several lines.

prepend the condition with \/ and as long as you log, you will see what
matches when it is assigned to the variable MATCh.

AK