When is What ... Deleted, Expired, Expunged or Purged?

The terminology sometimes plays administrators for fools -- which they are obviously not -- but an article clarifying what it is that is meant by either term of Deleted, Expired, Expunged or Purged goes a long way.

Deleted

A message has been flagged as \Deleted.

In the context of folders, Deleted only really applies to a folder having been removed (from the user's (IMAP client) view), as opposed to having been renamed to a hierarchy in a trash folder.

Expired

The index records of a message have been expired, and usually this means the message file has been purged as well. However the message file could be purged prior to index records being expired.

Expunged

The message (which has been flagged as \Deleted) is also expunged, meaning that the user can in no way retrieve the message autonomously.

Purged

The message's index record may still exist (until they are expired), but the message file is removed from the filesystem, or in the context of folders, the mail folder is removed from the filesystem.

Users and IMAP Clients Deleting Messages

When a message is deleted by a user, this means that the user's IMAP client has in fact flagged the message with \Deleted, or alternatively, the IMAP client has moved the message to a trash folder (and has at least flagged the original copy as \Deleted).

What is the exact behavior depends on the IMAP client software, and if so allowed, the user's preferences specified within the IMAP client software, and the Cyrus IMAP server configuration.

Flagged As \Deleted

When a message is merely flagged with \Deleted, the message itself as such remains available to the IMAP client, but the IMAP client used may not make it possible for the user to view a list of messages flagged with \Deleted. As such, the user may interpret the message as removed and unavailable -- if the removal was accidental, a support call may be on its way.

The message in fact still resides in the Cyrus IMAP mail spool, still resides in the same IMAP folder, and still resides on the filesystem.

Only when the user (or the user's IMAP client as is often the case) issues an EXPUNGE against the folder, or a UID EXPUNGE against the message [1], will the message be actually removed -- at least from the user's perspective. It then becomes irretrievable even if the IMAP client allows the listing of messages flagged with \Deleted.

Issuing an EXPUNGE may come in the form of a button to "compact" the folder, or an IMAP client routine that is executed periodically or at the end of a session (e.g. as the client application is closed), such as an "Empty Trash folder" kind of option.

It is here that the Cyrus IMAP server settings come in to play, most prominently the expunge_mode setting, which has three possible values:

delayed (the default since version 2.5.0)

The message files as well as the index records are retained for an undetermined period of time -- possibly indefinitely.

A separate job (using cyr_expire) is responsible for actually removing index records and message files.

default

The message files are removed at the first opportunity, while the index records remain available to facilitate QRESYNC.

In this context also, when we say "message files are removed", we mean "purged from the filesystem".

immediate

The message files as well as the index records are removed at the earliest opportunity.

In this context, when we say "message files are removed", we mean "purged from the filesystem".

Exceptional circumstances aside, when immediate or default is the configured expunge_mode, message files are often purged from the filesystem too quickly for anyone to recover.

Note

One such exceptional circumstance is a mailbox with multiple sessions keeping the mailbox open. Cyrus IMAP ensures no mailbox records disappear from underneath an existing open mailbox session.

Moved to Trash Folder

Should the IMAP client normally, or allow the user to specify through preferences, that messages being deleted should be moved in to a trash folder, then the user will usually be able to recover from accidental deletion autonomously, for as long as a copy of the deleted message(s) resides in such trash folder.

However, the trash folder would typically continue to grow and grow, and usually counts towards the user's resource usage (a.k.a. Quota); many IMAP clients therefore allow the user to specify a preference to empty the trash folder at the end of a session, or otherwise periodically.

If the IMAP client does not support RFC 6851 (for UID MOVE), the client may choose to COPY the message then flag the original with \Deleted, then EXPUNGE the folder or UID EXPUNGE (RFC 4315) the message.

This does not fare well in situations where the user is over quota, though, and (other) messages will need to be flagged as \Deleted and expunged, and/or folders within the quota root hierarchy will need to be deleted.

Expunged Messages

Messages in expunged folders, or messages that have been expunged individually, can not autonomously be restored by a user, and are gone permanently unless expunge_mode: delayed is used.

Recovering expunged messages requires administrator assistance, who can use command-line tools such as unexpunge to list and restore messages expunged. See the documentation on unexpunge for a walk-through on how that works.

With the use of expunge_mode: delayed, a regular EVENT (see cyrus.conf(5)) is responsible for triggering cyr_expire. This utility takes a parameter -X <days> to delete from the filesystem any messages that had been expunged (by the user or the IMAP client) more than <days> days ago.

In other words, using expunge_mode: delayed and cyr_expire allows an administrator to recover messages that have been deleted by the user less than <days> ago.

Note

This also offers a backup program the chance to obtain all message files. For a monthly full cycle, for example, one could choose to purge message files from the filesystem only after 69 days: two months plus the maximum margin for a first Saturday to Sunday night of the week.

Deleting Folders

When folders are deleted the IMAP client tends to either delete the folder, or rename the folder to a hierarchy in a trash folder.

Note

Note that deleting a folder A/B in a hierarchy A/B/C also deletes the folder A/B/C.

If the folder is not renamed to a hierarchy in a trash folder but instead removed directly, then the user has no way to autonomously recover from such event.

This is where the Cyrus IMAP server settings come in to play, most prominently delete_mode.

The setting holds two values:

delayed (the default since version 2.5.0)

Mailboxes that are being deleted are not deleted from the filesystem, but instead renamed to a special mailbox hierarchy under the deleted prefix, to be removed later by cyr_expire.

immediate

In immediate mode, the mailbox is removed from the filesystem immediately. Note that for large folders, this can be a comparatively expensive operation.

Where are the Messages?

This part of the documentation assumes that you have run with the default settings of delete_mode: delayed and expunge_mode: delayed.

The result of a message having deleted in either of the former ways, or an entire folder having been deleted, is one of the following stages;

  • The message has only been flagged as \Deleted and the message nor the folder has been expunged.

    Result: The message resides in the original folder.

  • The message has only been flagged as \Deleted and either the message individually or the entire folder as a whole has been expunged.

    Result: The message resides in the original folder and can be retrieved using unexpunge.

  • The message has been copied to the trash folder and at least flagged \Deleted in the source folder, and the original message or the entire folder in which the original message resided may or may not have been expunged.

    Similarly, the trash folder may or may not have been "emptied".

    Result: A copy of the message still exists in the original folder and can be retrieved using unexpunge.

  • The message was moved in to the trash folder, implying the original message is expunged from the source folder -- through UID MOVE or RFC 6851 support since version 2.5.0.

    The trash folder may or may not have been "emptied".

    Result: A copy of the message still exists in the original folder and can be retrieved using unexpunge.

  • The folder was moved to a hierarchy in the trash folder, and the trash folder has not yet been "emptied".

    Result: A copy of the message exists in the trash folder's hierarchy.

  • The folder was moved to a hierarchy in the trash folder, and the trash folder as subsequently been emptied.

    Result: The folder hierarchy has been renamed to the deleted namespace.

Footnotes