Backing up a mail message store (full/incremental).

When your mailboxes have mbox/mbx format, dumps and restores
involve relatively few but large files.
If your message store is relatively busy, incremental backups
will cost you nearly as much as full ones, in terms of tapes/
cassettes and time needed.

When operating a Courier or Cyrus server, dumps and restores
involve a large number of relatively small files. The cost of
an incremental backup will closely resemble the amount of new
mail that has arrived since your previous backup run.
Restoring the contents of a complete mail spool partition (in
the event of the need for recreating following a major h/w
failure or similar cockup) will require too much downtime for
your users to remain happy.

Has anyone already thought of modifying the MDA (procmail, Cyrus
deliver, ...) to generate an additional copy of each incoming
message to a separate area (together with its local recipient
list) and periodically backup that area ?

All suggestions/comments/hints/remarks/pointers welcomed.


Eric.

Re: Backing up a mail message store (full/incremental).

On 13 Jan 2006 14:00:50 GMT, Eric Luyten wrote:
> Has anyone already thought of modifying the MDA (procmail, Cyrus
> deliver, ...) to generate an additional copy of each incoming
> message to a separate area (together with its local recipient
> list) and periodically backup that area ?
>
> All suggestions/comments/hints/remarks/pointers welcomed.

I'd consider running a 2-leg or 3-leg mirror (depending if I was willing
to ever run in an un-mirrored state). I would detach one leg
of the mirror when I needed to run the backup, backup that
then re-attach the mirror and let it rebuild.

Cheers, Liam
>
>
> Eric.

Re: Backing up a mail message store (full/incremental).

This is a MIME GnuPG-signed message. If you see this text, it means that
your E-mail or Usenet software does not support MIME signed messages.
The Internet standard for MIME PGP messages, RFC 2015, was published in 1996.
To open this message correctly you will need to install E-mail or Usenet
software that supports modern Internet standards.

--=_mimegpg-commodore.email-scan.com-28620-1137194650-0005
Content-Type: text/plain; format=flowed; charset="US-ASCII"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit

Eric Luyten writes:

> When operating a Courier or Cyrus server, dumps and restores
> involve a large number of relatively small files. The cost of
> an incremental backup will closely resemble the amount of new
> mail that has arrived since your previous backup run.
> Restoring the contents of a complete mail spool partition (in
> the event of the need for recreating following a major h/w
> failure or similar cockup) will require too much downtime for
> your users to remain happy.

That may not be true in all cases. Native Linux filesystems are fairly
speedy even when creating thousands of small files in the same directory.


--=_mimegpg-commodore.email-scan.com-28620-1137194650-0005
Content-Type: application/pgp-signature
Content-Transfer-Encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.7 (GNU/Linux)

iD8DBQBDyDaax9p3GYHlUOIRAtt1AJ9QffxXfE/qJaVQi/FKy1NlJ5zIXwCf Yu0m
LlToTVUdQSpT/DjVLC4Yx0w=
=Vpwd
-----END PGP SIGNATURE-----

--=_mimegpg-commodore.email-scan.com-28620-1137194650-0005--

Re: Backing up a mail message store (full/incremental).

Eric Luyten wrote:
> When your mailboxes have mbox/mbx format, dumps and restores
> involve relatively few but large files.
> If your message store is relatively busy, incremental backups
> will cost you nearly as much as full ones, in terms of tapes/
> cassettes and time needed.
>
> When operating a Courier or Cyrus server, dumps and restores
> involve a large number of relatively small files. The cost of
> an incremental backup will closely resemble the amount of new
> mail that has arrived since your previous backup run.
> Restoring the contents of a complete mail spool partition (in
> the event of the need for recreating following a major h/w
> failure or similar cockup) will require too much downtime for
> your users to remain happy.
>
> Has anyone already thought of modifying the MDA (procmail, Cyrus
> deliver, ...) to generate an additional copy of each incoming
> message to a separate area (together with its local recipient
> list) and periodically backup that area ?
>
> All suggestions/comments/hints/remarks/pointers welcomed.
>
>

Backups are normally scheduled by rules of thumb, not analysis. A
moments thought suggests that backing up /var/mqueue once a day is not
very helpfull, since the average file lifetime in that directory is
measured in seconds. /var/mail files live a bit longer - hours usually,
so a backup will have a few usefull files on it. But a daily backup is
not a very effective way to protect users from either disk failures or
inadvertent deletions.

Keeping an archive of all received messages on a separate disk, seems
much more likely to be usefull. Of course, when it comes time to do a
restore, the archive won't know which messages have already been
received by the user MUA, and which haven't, so the user will be
subjected to a 2nd coming of messages. In our experience that hasn't
been a big problem. grepmail can process our archives (in mbox format)
to extract messages after a certain date.

Daniel Feenberg

> Eric.

Re: Backing up a mail message store (full/incremental).

feenberg@gmail.com wrote:
....
> Keeping an archive of all received messages on a separate disk, seems
> much more likely to be usefull. Of course, when it comes time to do a
> restore, the archive won't know which messages have already been
> received by the user MUA, and which haven't, so the user will be
> subjected to a 2nd coming of messages. ...

Hmm. You certainly do have a point here.

The original message data are one thing,
information as "seen" or "deleted" another.


Eric.

Re: Backing up a mail message store (full/incremental).

Liam Greenwood wrote:
> On 13 Jan 2006 14:00:50 GMT, Eric Luyten wrote:
>> Has anyone already thought of modifying the MDA (procmail, Cyrus
>> deliver, ...) to generate an additional copy of each incoming
>> message to a separate area (together with its local recipient
>> list) and periodically backup that area ?

<...>

> I'd consider running a 2-leg or 3-leg mirror (depending if I was willing
> to ever run in an un-mirrored state). I would detach one leg
> of the mirror when I needed to run the backup, backup that
> then re-attach the mirror and let it rebuild.

If your OS/filesystem has snapshot capability, that may also be a
viable solution for creating a coherent filesystem backup.
It still makes you perform a "restore" operation at an OS/filesystem
level and not at an application level, which is what I was aiming at.

Eric.