Large Mailboxes

We are currently running MS-Exchange with about 250 mailboxes on our
network. One of the challenges we have faced is that several people
have accumulated very large mailboxes (over 1gB) which is slowing the
system down, especially for those people with the large mailboxes. We
have multiple locations connected to the central data centre via a WAN
as well as a fairly large mobile work force who connect via VPN - often
from hotels.

We have attempted to reduce mailbox size by having people purge and
archive their mail with only limited success. Also, when people
archive their mail, it is being saved to their local computer and we've
had several instances where users (senior managers) have lost their
archives when their laptops have failed or been stolen.

We're not married to the idea of using Exchange although we are
commited to using the Outlook client. Are there other servers out
there that could handle these large mailboxes that we could use?

andrew@floatingbear.ca

Re: Large Mailboxes

I would suggest an open source solution using IMAP.

You also need to pay careful attention to what sort of mail store to use.
Some mail stores can handle large mailboxes much better than others.

Also, there is large and large. Is the 1GB caused by many small messages,
or is it caused by fewer large messages? A solution that handles
godzillagrams well may not handle 6-digit message counts, and vice versa.

It's rarely as simple as "install program XYZZY and problem solved."

When you say "250 mailboxes", I think that you mean that you have 250
users. In IMAP, a user can have multiple mailboxes; and typically has an
INBOX for incoming messages and some number of secondary mailboxes on the
server in which messages are filed away. The server performance cost is
only on whichever mailbox is open at the time; so a 1GB user who has a
20MB INBOX and 980MB split between a few dozen secondary mailboxes is
never going to hit the server performance cost of 1GB.

If your users insist upon having 1GB in a single mailbox, there are mail
stores which handle that well. Some servers (such as UW imapd) support
multiple mail stores, and selection of a mail store that is good for you
is critical.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Re: Large Mailboxes

Mark Crispin wrote:
> I would suggest an open source solution using IMAP.
>
> You also need to pay careful attention to what sort of mail store to use.
> Some mail stores can handle large mailboxes much better than others.
>
> Also, there is large and large. Is the 1GB caused by many small messages,
> or is it caused by fewer large messages? A solution that handles
> godzillagrams well may not handle 6-digit message counts, and vice versa.
>
> It's rarely as simple as "install program XYZZY and problem solved."
>
> When you say "250 mailboxes", I think that you mean that you have 250
> users. In IMAP, a user can have multiple mailboxes; and typically has an
> INBOX for incoming messages and some number of secondary mailboxes on the
> server in which messages are filed away. The server performance cost is
> only on whichever mailbox is open at the time; so a 1GB user who has a
> 20MB INBOX and 980MB split between a few dozen secondary mailboxes is
> never going to hit the server performance cost of 1GB.
>
> If your users insist upon having 1GB in a single mailbox, there are mail
> stores which handle that well. Some servers (such as UW imapd) support
> multiple mail stores, and selection of a mail store that is good for you
> is critical.
>
> -- Mark --
>

Mark:

Thank you for your quick reply. I don't know a lot about IMAP but it
looks that I'll need to learn.

Yes - you are correct, I mean 250 users. We have been pretty
successful in getting users to file their mail away in separate
folders. Without doing an analysis, I would suggest that your guess of
the split would describe the bulk of our "problem" users and yes, they
would in general be made up of a large number of small messages.

Do you know if there are any resources out there that I would be able
to consult to see what IMAP solutions there are? I've found imap.org
which looks like a good starting point.

Andrew B.

Re: Large Mailboxes

On Mon, 18 Sep 2006, andrew@floatingbear.ca wrote:
> Do you know if there are any resources out there that I would be able
> to consult to see what IMAP solutions there are? I've found imap.org
> which looks like a good starting point.

That is, indeed, a good starting point. So are the comp.mail.imap
newsgroup and the imap-use@u.washington.edu mailing list.

One possibility is UW imapd with the new mix format as the mail store.
mix may be better for you since what you described has very large
mailboxes (which tend to hurt performance in a flat file format such as
traditional UNIX or UW imapd's former preferred mbx format), and very many
messages (which, depending upon your choice of operating system, may hurt
performance in a one-message-per-file format).

mix is a hybrid format which represents the mailbox as multiple files but
not at the extreme of one-message-per-file. It also has an index,
separate of static and dynamic data, and a cache of thread data (so it is
not necessary to read/parse messages in order to thread them).

I admit to bias (being the author of UW imapd and mix); and thus recommend
that you research a wide range of alternatives so you ultimately feel
comfortable with the choice which you have made. mix is certainly not the
only mail store that indexes and/or caches, and you may have needs that
would be better addressed by an alternative solution. Only you can make
that determination.

One other point: whatever you do, deploy your server on a free,
open-source, operating system such as Linux or one of the BSDs. In other
words, do NOT use proprietary systems such as Windows (good for desktops,
not for servers) or commercial UNIX systems (such as Solaris, AIX, HP-UX,
etc.).

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Re: Large Mailboxes

This is a MIME GnuPG-signed message. If you see this text, it means that
your E-mail or Usenet software does not support MIME signed messages.
The Internet standard for MIME PGP messages, RFC 2015, was published in 1996.
To open this message correctly you will need to install E-mail or Usenet
software that supports modern Internet standards.

--=_mimegpg-commodore.email-scan.com-3463-1158618712-0001
Content-Type: text/plain; format=flowed; charset="US-ASCII"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit

andrew@floatingbear.ca writes:

> We're not married to the idea of using Exchange although we are
> commited to using the Outlook client.

You're screwed. Outlook only works with Exchange. Microsoft won't have it
any other way.

There are occasional claims that Outlook will work with generic IMAP
servers. Those brave souls that ventured in that territory found that
Outlook quickly castrates itself, and 95% of its functions only work with
Exchange. With IMAP you'll only be able to read your mail messages, and
that's about it. No calendaring. No shared folders. Nothing else, but
random error messages with Outlook's lying error messages repeatedly blaming
the IMAP server for some imaginary transgressions, when the truth of the
matter is that Outlook's IMAP support is broken, but, Microsoft wouldn't
have it any other way except to blame the server for Outlook's own bugs.

> Are there other servers out
> there that could handle these large mailboxes that we could use?

Oh, there are plenty of IMAP servers that will easily handle huge volumes of
mail.

But you'll have to use a standard IMAP mail client to use them, instead of
Microsoft's virus distribution and trojan delivery software.

Having said that:

There are some third party plugins that claim the ability to fool Outlook
into thinking that it's talking to Exchange, when it's really connecting to
a garden-variety IMAP server. I've not used them myself, so I can't vouch
for them. What I hear is that they work for some people; for other people
they don't.



--=_mimegpg-commodore.email-scan.com-3463-1158618712-0001
Content-Type: application/pgp-signature
Content-Transfer-Encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQBFDx5Yx9p3GYHlUOIRAomUAJ9YJsxhKRCPk6LZRf0cQ5Tok5M1FwCe L1cf
5ZxwW4Mo26X8bz5Ka7dS6N8=
=IGeh
-----END PGP SIGNATURE-----

--=_mimegpg-commodore.email-scan.com-3463-1158618712-0001--

Re: Large Mailboxes

Mark Crispin wrote:

[SNIP sensible suggestoins about IMAP server...]

> One other point: whatever you do, deploy your server on a free,
> open-source, operating system such as Linux or one of the BSDs. In
> other words, do NOT use proprietary systems such as Windows (good for
> desktops, not for servers) or commercial UNIX systems (such as Solaris,
> AIX, HP-UX, etc.).
Why in the seven hells is this a problem?

They are all Operating Systems, some better for some purposes than
others, but why are "Linux or one of the BSDs" the only suitable solutions?

BTW - Solaris 10 is free as in "free to use", and OpenSolaris is free as
in "the source code is freely available".

Cheers,
Gary B-)

--
____________________________________________________________ __________________
Armful of chairs: Something some people would not know
whether you were up them with or not
- Barry Humphries

Re: Large Mailboxes

Mark Crispin wrote:

> I admit to bias (being the author of UW imapd and mix); and thus recommend
> that you research a wide range of alternatives so you ultimately feel
> comfortable with the choice which you have made. mix is certainly not the
> only mail store that indexes and/or caches, and you may have needs that
> would be better addressed by an alternative solution. Only you can make
> that determination.

I have successfully used UW imapd, Cyrus IMAP, and Dovecot. My personal
opinions of these programs:

UW imapd: Easy to set up, but somewhat inflexible. (At least, this was my
experience several years ago -- things may have changed.) For example,
it insisted on putting folders under my users' home directories, which
was not at all what I wanted.

Cyrus IMAP: More difficult to set up. Somewhat idiosyncratic: Requires
special administrative tools to set up accounts, rather than using normal
UNIX accounts. Definitely designed to scale, though; on a good UNIX
system with the appropriate file system, you could have mailboxes containing
hundreds of thousands of messages without any trouble.

Dovecot: Easier to set up than Cyrus; somewhat more flexible than UW imapd.
That's what I use now.

Regards,

David.

UW imapd: http://www.washington.edu/imap/
Cyrus IMAP: http://cyrusimap.web.cmu.edu/
Dovecot: http://www.dovecot.org/
Courier IMAP: http://www.courier-mta.org/imap/ (Never used this myself, but
have heard good things.)

Re: Large Mailboxes

Following up on myself:

While any of the open-source/free-software IMAP servers is likely
to meet your needs, your biggest problem is likely to be migration.
Getting mail for 250 people (some of whom have >1GB of mail each) out
of Exchange and into an open-source IMAP server is going to be a
challenge.

There are tools out there that claim to do IMAP-to-IMAP moves between
servers, but I never found one that worked properly. In fact, the
main reason I chose Dovecot was that I could migrate my UW imap
mailboxes simply by copying files.

Good luck!

Regards,

David.

Re: Large Mailboxes

On Tue, 19 Sep 2006, Gary R. Schmidt wrote:
>> One other point: whatever you do, deploy your server on a free,
>> open-source, operating system such as Linux or one of the BSDs. In other
>> words, do NOT use proprietary systems such as Windows (good for desktops,
>> not for servers) or commercial UNIX systems (such as Solaris, AIX, HP-UX,
>> etc.).
> Why in the seven hells is this a problem?
> They are all Operating Systems, some better for some purposes than others,
> but why are "Linux or one of the BSDs" the only suitable solutions?

Windows: There are not very many IMAP servers available for Windows; and
none of the options are particularly compelling.

Proprietary UNIX: Solaris and other proprietary UNIX systems are
technically far behind the state of the art. I can't speak for other IMAP
servers; but I can say that UW imapd runs MUCH better on Linux and BSD
than on proprietary UNIX due to deficiencies in proprietary UNIX.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Re: Large Mailboxes

Thanks everyone for the input - the more discussion on some of the
challenges we may face the better. After my manager gets back from
vacation in a week or two he wants to review the options with me.

One of the challenges that we probably would have would be the lack of
UNIX experience in our shop. That may lead us to looking at the
possibility of just outsourcing everything. That and moving our
existing message store I can see as the bigger hurdles. There is some
scripting that I know that I can do against Outlook to retrieve
messages but what I could do with that I'm not sure. I've done some
reading and there seem to be some IMAP to Exchange conversion routines,
but nothing going the other way.

Andrew B

Re: Large Mailboxes

In article <1158681751.373450.281560@k70g2000cwa.googlegroups.com>,
andrew@floatingbear.ca wrote:

> That and moving our
> existing message store I can see as the bigger hurdles. There is some
> scripting that I know that I can do against Outlook to retrieve
> messages but what I could do with that I'm not sure. I've done some
> reading and there seem to be some IMAP to Exchange conversion routines,
> but nothing going the other way.

See http://www.emailman.com/conversion/ -- you might contact one of the consultants listed there.

If you do, let me know how it works out.

-Andrew

--
eMailman, LLC - Andrew Starr, President
eMailman(r):
NewsReaders(sm):
New discussion forums (via web or newsreader) at both of the above sites.

Re: Large Mailboxes

andrew@floatingbear.ca wrote:
[deleted]

> We're not married to the idea of using Exchange although we are
> commited to using the Outlook client. Are there other servers out
> there that could handle these large mailboxes that we could use?

You may want to have a look at the OpenMail server software. OpenMail
was originally developed by HP (Hewlett Packard) and later sold to/by
Samsung (Don't get me started on *that* one! :-(). While OpenMail does
not (or at least did not) implement all the Outlook/Exchange/MS-specific
stuff (see Sam's response), it implements some of it (in addition to
IMAP/POP/SMTP). And BTW, OpenMail runs on UNIX, not (MS-)Windows.

I have no current pointer to OpenMail (at Samsung), but can probably
find/get some if you can't find any.

Re: Large Mailboxes

Frank Slootweg wrote:

> You may want to have a look at the OpenMail server software. OpenMail
> was originally developed by HP (Hewlett Packard) and later sold to/by
> Samsung (Don't get me started on *that* one! :-(). While OpenMail does
> not (or at least did not) implement all the Outlook/Exchange/MS-specific
> stuff (see Sam's response), it implements some of it (in addition to
> IMAP/POP/SMTP). And BTW, OpenMail runs on UNIX, not (MS-)Windows.
>
> I have no current pointer to OpenMail (at Samsung), but can probably
> find/get some if you can't find any.

It seems to be "Samsung Contact" - http://www.samsungcontact.com/

I don't think we want to go towards software that isn't currently being
marketed. We have enough issues here around getting support for old
applications without adding more.

Andrew B

Re: Large Mailboxes

andrew@floatingbear.ca wrote:
>
> Frank Slootweg wrote:
>
> > You may want to have a look at the OpenMail server software. OpenMail
> > was originally developed by HP (Hewlett Packard) and later sold to/by
> > Samsung (Don't get me started on *that* one! :-(). While OpenMail does
> > not (or at least did not) implement all the Outlook/Exchange/MS-specific
> > stuff (see Sam's response), it implements some of it (in addition to
> > IMAP/POP/SMTP). And BTW, OpenMail runs on UNIX, not (MS-)Windows.
> >
> > I have no current pointer to OpenMail (at Samsung), but can probably
> > find/get some if you can't find any.
>
> It seems to be "Samsung Contact" - http://www.samsungcontact.com/
>
> I don't think we want to go towards software that isn't currently being
> marketed. We have enough issues here around getting support for old
> applications without adding more.

Sorry, I didn't know that it was being phased out.

To be [f|F]rank, I'm not really suprised. Microsoft's 'business
practices' caused the death#####sale of the product in the first place.
With no reputable (IT) company behind it, it's no wonder that it's now
nearly dead. Just another case of inferior software 'beating' vastly
superior software. Suprised? No. Sad/disgusted/etc.? Yes.

Re: Large Mailboxes

In article <1159372140.056801.309110@i3g2000cwc.googlegroups.com>, andrew@floatingbear.ca writes:
>
>Frank Slootweg wrote:
>
>> You may want to have a look at the OpenMail server software. OpenMail
>> was originally developed by HP (Hewlett Packard) and later sold to/by
>> Samsung (Don't get me started on *that* one! :-(). While OpenMail does
>> not (or at least did not) implement all the Outlook/Exchange/MS-specific
>> stuff (see Sam's response), it implements some of it (in addition to
>> IMAP/POP/SMTP). And BTW, OpenMail runs on UNIX, not (MS-)Windows.
>>
>> I have no current pointer to OpenMail (at Samsung), but can probably
>> find/get some if you can't find any.
>
>It seems to be "Samsung Contact" - http://www.samsungcontact.com/
>
>I don't think we want to go towards software that isn't currently being
>marketed. We have enough issues here around getting support for old
>applications without adding more.
>
For a supported application you might want to look at Communigate Pro
see

http://www.stalker.com/content/cgp_groupware.html


David Webb
Security team leader
CCSS
Middlesex University


>Andrew B
>