Line ends with /n or /r/n ?

am 07.05.2005 13:17:37 von siliconmike

I'm writing a script that will split an incoming email into lines and
store each line in consecutive indexes of an array.

So what are the line ending possibilities?

\n ?
\r\n ?
\r ?

Mike

Re: Line ends with /n or /r/n ?

am 07.05.2005 15:19:13 von Sam

Re: Line ends with /n or /r/n ?

am 07.05.2005 15:52:40 von siliconmike

> E-mail is text. If you are using a brain-dead operating system from
> Microsoft, lines of text are terminated by \r\n. The rest of the
world uses
> text files where each line of text is terminated by \n.

Please read this from RFC 822. I still think it is \r\n. But I'm still
doubtful.

3.4.8. FOLDING LONG HEADER FIELDS

Each header field may be represented on exactly one line con-
sisting of the name of the field and its body, and terminated
by a CRLF; this is what the parser sees.

Re: Line ends with /n or /r/n ?

am 07.05.2005 15:53:11 von Frank Slootweg

Sam wrote:
> siliconmike writes:
>
> > I'm writing a script that will split an incoming email into lines and
> > store each line in consecutive indexes of an array.
> >
> > So what are the line ending possibilities?
> >
> > \n ?
> > \r\n ?
> > \r ?
>
> E-mail is text. If you are using a brain-dead operating system from
> Microsoft, lines of text are terminated by \r\n. The rest of the world uses
> text files where each line of text is terminated by \n.

Well, sorry to rain on your parade, but the [1] method
*predates* Microsoft and UNIX. There's alot you can blame on Microsoft,
but not this one.

I wonder what (current) non-MS/non-UNIX(-like) OSs use, or is, in your
world, an OS brain-dead if it's not UNIX(-like)? :-)

[1] Carriage-Return, Line-Feed, i.e. teletype (and earlier?) era, i.e.
the early 60s (or earlier?).

Re: Line ends with /n or /r/n ?

am 07.05.2005 16:03:55 von siliconmike

Well, after some research, I found the answer to my question:

siliconmike writes:
> I'm writing a script that will split an incoming email into lines and
> store each line in consecutive indexes of an array.

> So what are the line ending possibilities?

> \n ?
> \r\n ?
> \r ?

Any field in an email message must end with a \r\n
(This is including the message field)
Any field my contain bare \r or bare \n inside.

So, a typical email message field may look like:

message: hello how are you ? \n
we're on next line \n
but since message field ends here: \r\n

Comments?
Mike

Re: Line ends with /n or /r/n ?

am 07.05.2005 16:26:30 von Mark Crispin

On Sat, 7 May 2005, siliconmike wrote:
> I'm writing a script that will split an incoming email into lines and
> store each line in consecutive indexes of an array.
> So what are the line ending possibilities?
> \n ?
> \r\n ?
> \r ?

Lines in Internet standard texts end with CRLF; that is, \r\n.

Traditionally, Internet standard texts treated bare \r and bare \n as
being distinct from newline. Bare \r was used to indicate overstriking a
line, and bare \n was used to move to the same position at the next line
(as opposed to the start of the next line), consistant with how old
hardcopy terminals worked.

There were only a handful of UNIX systems on the net until the mid-1980s,
consequently two very different traditions of newlines developed and had
to merge.

Texts are almost always modified in transit between the Internet and the
local disk to suit your local operating system. In particular, UNIX
systems use \n without \r as an end of line character. Old Macintosh
systems (prior to OS X) use \r without \n as end of line. Other systems
(Microsoft, VMS, TOPS-20, etc.) use \r\n unmodified.

So, unless you know what platform your code will run on, you need to
consider all possibilities.

Over the years, there have been *many* bugs over the years related to
newline conversion between UNIX \n and Internet standard \r\n newlines.
These bugs continue to pop up; consequently no prudent individual depends
upon newline conversion working properly.

As UNIX has become the majority operating system on the Internet in the 20
years since then, the Internet approach to newlines has changed. Bare \r
and bare \n are no longer recognized in Internet standard texts.

It's probably best to treat bare \r, bare \n, and \r\n as all being end of
line. If you're certain that you'll never encounter old Macintosh texts,
you can ignore \r (discarding \r whenever encountered in input) and only
consider \n.

You need to do this even on systems (such as Windows) which use the
traditional \r\n definition of newline. There are too many cases in which
UNIX-style bare \n newlines end up in Windows filesystems without being
converted properly.

Amusingly, there is a school of thought that says that strict adherance to
correct newlines is a good thing -- for anti-spam and anti-virus purposes.
Many spam and virus generating programs are written by novice programmers
who don't understand the issues of newlines fully; consequently they tend
to transmit bare \n newlines over the Internet. An SMTP server which only
recognizes \r\n newlines will not interoperate with the spam/virus sender.

I would not count on that as the sole defense against evil email...

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Re: Line ends with /n or /r/n ?

am 07.05.2005 16:38:20 von Mark Crispin

On Sat, 7 May 2005, Frank Slootweg wrote:
> Well, sorry to rain on your parade, but the [1] method
> *predates* Microsoft and UNIX. There's alot you can blame on Microsoft,
> but not this one.

You beat me to the punch...

> I wonder what (current) non-MS/non-UNIX(-like) OSs use, or is, in your
> world, an OS brain-dead if it's not UNIX(-like)? :-)

Probably it's best not to go there. A large number of individuals
consider all non-UNIX(-like) systems to be brain-dead, and the subsequent
discussion will be much flame and little light.

> [1] Carriage-Return, Line-Feed, i.e. teletype (and earlier?) era, i.e.
> the early 60s (or earlier?).

Indeed, this was influenced by the behavior of teletypes. The PDP-1 did
something else, but subsequent DEC operating systems (most notably on the
PDP-10) all used CR LF.

In the early ARPAnet days, PDP-10 systems were the majority, especially
those running BBN Tenex. It isn't surprising that their native newline
convention became the network newline convention.

I am aware of the following newlines used by various operating systems:
CR LF
LF
CR
US (that is, CTRL/_, 037, 0x1f)
NUL

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Re: Line ends with /n or /r/n ?

am 07.05.2005 17:19:09 von Karl Kleinpaste

"siliconmike" writes:
> I'm writing a script that will split an incoming email into lines and
> store each line in consecutive indexes of an array.
> So what are the line ending possibilities?

You must distinguish between the text as it travels in SMTP between
MTAs and the text as it is delivered to a user-accessible mailbox.

During SMTP, lines are terminated with CR+LF. But it is extremely
common that, as stored in a mailbox file, the local OS-typical line
termination convention is adopted, so that in (e.g.) UNIX-style
systems, the line termination is then just what we call "\n".

Bear in mind that "\n" really means just "newline," whose physical
manifestation on a screen or on paper is supposed to mean "carriage
return back to left margin, then line feed down once," and under UNIX
this happens only merely to coincide with the literal ASCII LF
character. That is, "newline" is a concept, as a standard mechanism
for depicting line termination. The concept "newline" in SMTP
conversation is by convention CR+LF. The concept "newline" in other
OSes is by convention just CR.

Is your script speaking SMTP as an MTA? Split on CR+LF. (Same for POP.)
Is your script mangling post-delivery mail? Split on whatever the
local OS convention uses.

Re: Line ends with /n or /r/n ?

am 07.05.2005 19:57:24 von Sam

Re: Line ends with /n or /r/n ?

am 07.05.2005 20:01:22 von Sam

Re: Line ends with /n or /r/n ?

am 07.05.2005 20:42:10 von Mark Crispin

On Sat, 7 May 2005, Karl Kleinpaste wrote:
> Bear in mind that "\n" really means just "newline," whose physical
> manifestation on a screen or on paper is supposed to mean "carriage
> return back to left margin, then line feed down once," and under UNIX
> this happens only merely to coincide with the literal ASCII LF
> character. That is, "newline" is a concept, as a standard mechanism
> for depicting line termination.

Actually,...it depends.

Most people think of \n as being a concept defined by the C programming
language. In C, \n *usually* means the same thing as \012, and thus the
expression ('\n' == '\012') is true. However, the C programming language
definition itself makes no such assertion.

*Most* C compilers on *most* systems with other newline conventions retain
C's convention of \n being \012, and translate to the system newline in
the stdio library. In these systems, the "b" mode character is fopen() is
significant (it is ignored on UNIX system) since it controls whether or
not this happens. You may also have to deal with O_BINARY in open(). The
Windows, VAX/VMS, and TOPS-20 C compilers all behave this way.

I have, however, seen systems in which \n has a value other than \012.

> The concept "newline" in SMTP
> conversation is by convention CR+LF. The concept "newline" in other
> OSes is by convention just CR.

You're confusing operating systems and protocols. The standard newline in
the Internet protocol suite is CRLF; it isn't just SMTP.

> Is your script speaking SMTP as an MTA? Split on CR+LF. (Same for POP.)
> Is your script mangling post-delivery mail? Split on whatever the
> local OS convention uses.

Or, more simply, you need to know the context of the data in question.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Re: Line ends with /n or /r/n ?

am 07.05.2005 21:07:24 von Frank Slootweg

Sam wrote:
> Frank Slootweg writes:
>
> > Sam wrote:
> >> siliconmike writes:
> >>
> >> > I'm writing a script that will split an incoming email into lines and
> >> > store each line in consecutive indexes of an array.
> >> >
> >> > So what are the line ending possibilities?
> >> >
> >> > \n ?
> >> > \r\n ?
> >> > \r ?
> >>
> >> E-mail is text. If you are using a brain-dead operating system from
> >> Microsoft, lines of text are terminated by \r\n. The rest of the world uses
> >> text files where each line of text is terminated by \n.
> >
> > Well, sorry to rain on your parade, but the [1] method
> > *predates* Microsoft and UNIX.
>
> So?
>
> In other words: "this stupid way of doing things has been the case for so
> long, it's not our fault for continuing to do the same stupid thing, over
> and over again."

The *point* is that it's *not* stupid. There were good reasons for
doing it (as in: the *only* possible way). See also Mark Crispin's
response(s). (Example) Hint: ASR.

Bottom line: Your not knowing/understanding something doesn't make it
stupid.

> > There's alot you can blame on Microsoft,
> > but not this one.
>
> Of course I can.

If you want to be silly, then indeed you can.

> > I wonder what (current) non-MS/non-UNIX(-like) OSs use, or is, in your
> > world, an OS brain-dead if it's not UNIX(-like)? :-)
>
> I wonder how many baby seals had to be killed, in order to butcher that
> sentence.

Well, you're the one who's always complaining about people who can't
parse simple sentences. So what's up?

Re: Line ends with /n or /r/n ?

am 07.05.2005 21:27:49 von Frank Slootweg

Sam wrote:
> siliconmike writes:
>
> >> E-mail is text. If you are using a brain-dead operating system from
> >> Microsoft, lines of text are terminated by \r\n. The rest of the
> > world uses
> >> text files where each line of text is terminated by \n.
> >
> > Please read this from RFC 822. I still think it is \r\n. But I'm still
> > doubtful.
>
> Read it yourself.
>
> That document discusses the format of E-mail messages when transmitted over
> the Internet.
>
> Which has absolutely nothing to do with the format of E-mail messages stored
> on the mail server.
>
> After receiving a message, formatted accordingly, the mail server typically
> converts it to the local convention for storing text files, since, after
> all, an E-mail message is a text file.

All true, but - as others noted - if any of it is relevant depends on
what siliconmike means with "incoming email". I.e. "incoming" from
*what*, and - if relevant - on which platform?

Re: Line ends with /n or /r/n ?

am 07.05.2005 21:42:36 von Mark Crispin

On Sat, 7 May 2005, Frank Slootweg wrote:
> (Example) Hint: ASR.

And especially the difference between 33, 35, and 37.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Re: Line ends with /n or /r/n ?

am 07.05.2005 21:51:21 von Sam

This is a MIME GnuPG-signed message. If you see this text, it means that
your E-mail or Usenet software does not support MIME signed messages.

--=_mimegpg-commodore.email-scan.com-20503-1115495487-0001
Content-Type: text/plain; format=flowed; charset="US-ASCII"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit

Frank Slootweg writes:

> Sam wrote:
>> Frank Slootweg writes:
>>
>> > Sam wrote:
>> >> siliconmike writes:
>> >>
>> >> > I'm writing a script that will split an incoming email into lines and
>> >> > store each line in consecutive indexes of an array.
>> >> >
>> >> > So what are the line ending possibilities?
>> >> >
>> >> > \n ?
>> >> > \r\n ?
>> >> > \r ?
>> >>
>> >> E-mail is text. If you are using a brain-dead operating system from
>> >> Microsoft, lines of text are terminated by \r\n. The rest of the world uses
>> >> text files where each line of text is terminated by \n.
>> >
>> > Well, sorry to rain on your parade, but the [1] method
>> > *predates* Microsoft and UNIX.
>>
>> So?
>>
>> In other words: "this stupid way of doing things has been the case for so
>> long, it's not our fault for continuing to do the same stupid thing, over
>> and over again."
>
> The *point* is that it's *not* stupid.

Of course it is. Why use two characters to do one's job?

Just forget all the voluminous tomes of historical documents at your
disposal (I'm sure they're very exciting and informative, and make for
compelling reading). Just focus your mind on today, Saturday, some time in
the year 2005. You're reading something (or writing, or whatever). You've
come to the end of the line. You want to express your will to start the
next line of the text (or read it, or whatever). Do you use:

A) One character, or

B) Two characters

to express your will and desire? Explain your choice logically, so that
everyone reading your magnificent explanation would just slap their heads
"Of course! That's the only way it must be!!!!".

Avoid the temptation to include in your argument things like "well, fourty
years ago in someone's basement in New Jersey there was some jury-rigged
metallic beast that had to do this, that, or the other, in order to
communicate with its handlers. That was fourty years ago. Ancient history.
Long dead, and forgotten.

> There were good reasons for
> doing it (as in: the *only* possible way).

The key word there is "were". We're not in the '60s any more. Things move
on, and are replaced by bigger and better things. It's called "evolution",
and "progress".

> See also Mark Crispin's
> response(s). (Example) Hint: ASR.

Age/Sex/Religion?

>> > I wonder what (current) non-MS/non-UNIX(-like) OSs use, or is, in your
>> > world, an OS brain-dead if it's not UNIX(-like)? :-)
>>
>> I wonder how many baby seals had to be killed, in order to butcher that
>> sentence.
>
> Well, you're the one who's always complaining about people who can't
> parse simple sentences. So what's up?

You have me confused with someone else.

Searched all groups

Your search - sentence parse group:comp.mail.misc author:sam - did not match
any documents.

--=_mimegpg-commodore.email-scan.com-20503-1115495487-0001
Content-Type: application/pgp-signature
Content-Transfer-Encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQBCfRw/x9p3GYHlUOIRAvQYAJ9rBdOPdVCBxqfXEY+s9yBCBFwbuACe PfNe
Ljk6doCQ8iFysC/WKjH/1O4=
=f2xZ
-----END PGP SIGNATURE-----

--=_mimegpg-commodore.email-scan.com-20503-1115495487-0001--

Re: Line ends with /n or /r/n ?

am 07.05.2005 22:09:46 von Frank Slootweg

Sam wrote:
> [-- text/plain, encoding 7bit, charset: US-ASCII, 84 lines --]
>
> Frank Slootweg writes:
>
> > Sam wrote:
> >> Frank Slootweg writes:
> >>
> >> > Sam wrote:
> >> >> siliconmike writes:
> >> >>
> >> >> > I'm writing a script that will split an incoming email into lines and
> >> >> > store each line in consecutive indexes of an array.
> >> >> >
> >> >> > So what are the line ending possibilities?
> >> >> >
> >> >> > \n ?
> >> >> > \r\n ?
> >> >> > \r ?
> >> >>
> >> >> E-mail is text. If you are using a brain-dead operating system from
> >> >> Microsoft, lines of text are terminated by \r\n. The rest of the world uses
> >> >> text files where each line of text is terminated by \n.
> >> >
> >> > Well, sorry to rain on your parade, but the [1] method
> >> > *predates* Microsoft and UNIX.
> >>
> >> So?
> >>
> >> In other words: "this stupid way of doing things has been the case
> >> for so long, it's not our fault for continuing to do the same
> >> stupid thing, over and over again."
> >
> > The *point* is that it's *not* stupid.
>
> Of course it is. Why use two characters to do one's job?

As I said: Because it was the only possible way. See the hint.

> Just forget all the voluminous tomes of historical documents at your
> disposal

Why would I need documents when I have my personal experience?

[irrelevant stuff deleted]

> > There were good reasons for
> > doing it (as in: the *only* possible way).
>
> The key word there is "were". We're not in the '60s any more. Things move
> on, and are replaced by bigger and better things. It's called "evolution",
> and "progress".

Yes, the keyword is "were". Just as the keywords are "has been" in
your:

> >> In other words: "this stupid way of doing things has been the case
> >> for so long, it's not our fault for continuing to do the same
> >> stupid thing, over and over again."

I.e. you refered to the *past* being "stupid". I explained why it
wasn't.

> > See also Mark Crispin's
> > response(s). (Example) Hint: ASR.
>
> Age/Sex/Religion?

Not quite. Try again. See Mark's recent response for more hints.

> >> > I wonder what (current) non-MS/non-UNIX(-like) OSs use, or is, in your
> >> > world, an OS brain-dead if it's not UNIX(-like)? :-)
> >>
> >> I wonder how many baby seals had to be killed, in order to butcher that
> >> sentence.
> >
> > Well, you're the one who's always complaining about people who can't
> > parse simple sentences. So what's up?
>
> You have me confused with someone else.

Possibly. I that's the case, I apologize.

[deleted]

Re: Line ends with /n or /r/n ?

am 07.05.2005 22:48:41 von Mark Crispin

On Sat, 7 May 2005, Sam wrote:
> Avoid the temptation to include in your argument things like "well, fourty
> years ago in someone's basement in New Jersey there was some jury-rigged
> metallic beast that had to do this, that, or the other, in order to
> communicate with its handlers. That was fourty years ago. Ancient history.
> Long dead, and forgotten.

In 2005, stty on Linux and FreeBSD offers the onlcr flag, however it seems
that "stty -onlcr" does not, in fact, turn off the onlcr flag; at least
not on an xterm. Either stty or the shell is keeping it from happening.
I guess that they don't want novice UNIX users to be confused by what
happens if onlcr isn't set.

However, using that xterm to connect to a system that treats CR and LF
separately shows that xterm implements CR as "move the cursor to the start
of the current line" and LF as "drop the cursor to the next line but do
not move it to the start of the line."

What this all means is that the quoted text is a red herring. The
necessity of sending CR LF to a terminal to a accomplish a newline is
still with us. UNIX and the C library just choose to hide that detail.

Actually, there is an exception: the ASR 37, which was used heavily by the
developers of UNIX in the early days, did a newline when it received a
line feed.

Consequently, it can be argued with equal (in)validity that UNIX is the
way it is because of a hardcopy terminal that nobody has used for at least
25 years.

On UNIX, all the world is a glorified ASR 37. On other operating systems,
all the world is a glorified ASR 35 (or LA120). Neither is "right" or
"wrong", except to religious bigots.

Both sides point selective readings of ASCII to bolster their weak cases.
The fact is that *nobody* today fully upholds ASCII.

If you find that LF for newline is more convenient to you, then fine. If
you find that CR LF for newline is more convenient to you, then fine too.
But most of the time, you'll find that you don't have a choice in the
matter, and instead must follow the choice that others made for you.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Re: Line ends with /n or /r/n ?

am 08.05.2005 19:13:42 von siliconmike

> All true, but - as others noted - if any of it is relevant depends
on
> what siliconmike means with "incoming email". I.e. "incoming" from
> *what*, and - if relevant - on which platform?

When mail comes in it is received by Sendmail and diverted to the
appropriate user's inbox (or offsite email address). Sendmail will
check the user account to look for a .forward file. If one is found,
Sendmail will deliver the mail to each recipient in the file. This is
where the mailbot gets called. The mail is "delivered" to the mailbot,
which reads it in from stdin. The mailbot explodes the email into
lines.

For now, I'm just using \n as new line character to extract lines. Then
I trim away the \r from the edge of the line, if present.

Mike

Re: Line ends with /n or /r/n ?

am 08.05.2005 19:38:47 von Mark Crispin

On Sun, 8 May 2005, siliconmike wrote:
> For now, I'm just using \n as new line character to extract lines. Then
> I trim away the \r from the edge of the line, if present.

That's a suitable algorithm. Whether or not the \r is present depends
upon the setting of E= in the sendmail configuration rule for Mlocal.
Doing it the way you are doing it will work either way.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Re: Line ends with /n or /r/n ?

am 14.05.2005 19:30:38 von Macallan

Try to respond in text format please.

"Sam" a écrit dans le message de news:
cone.1115471951.458223.15595.500@commodore.email-scan.com...

Re: Line ends with /n or /r/n ?

am 14.05.2005 20:17:14 von Sam

Re: Line ends with /n or /r/n ?

am 15.05.2005 09:49:18 von Frank Slootweg

Macallan wrote:
>
> Try to respond in text format please.
>
> "Sam" a ?crit dans le message de news:
> cone.1115471951.458223.15595.500@commodore.email-scan.com...

As Sam mentioned, don't blame him, but your newsreader software. FWIW,
even in your newsreader, Outlook Express (OE), you can still see his
text by opening the .txt 'attachment'.

To see what's wrong with your newsreader, do (while displaying the
article): File -> Properties -> Details -> Message Source...

The very *first* lines of the article body say:

> This is a MIME GnuPG-signed message. If you see this text, it means that
> your E-mail or Usenet software does not support MIME signed messages.

If OE would only display this information (by default) instead of
hiding it, you would have known who (not) to blame.

As to "respond in text format": His response *is* text. The very
*first* MIME part is:

> Content-Type: text/plain; format=flowed; charset="US-ASCII"

Again OE decides to hide (as an attachment) the text/plain body,
instead of just displaying it.

As a reference: My first, 1983, newsreader would have no problem with
Sam's article, so OE is at least several decades out-of-date, i.e. when
it was *created* it was already broken.

Personally, I think PGP signing Usenet postings adds no to very little
value and is somewhat of a nuisance (mostly ecause of problems like
yours), but, as said, the main problem lies with your newsreader, not
with Sam's use of PGP.