Replication: Installation and Administration

Architecture

Overall structure of replication.

Terminology

Host

A computer chassis, slot or cabinet. A machine.

Server or Service

A program or process which "serves" a particular protocol. Typically a server will listen on a network port or Unix socket for communications from a client (local or remote).

Client

A program or process which talks to a server or servers for a given protocol. The initiator of a client/server session.

Instance

A particular instance or iteration of a program, which may be one, or one of many, providing similar services to different consumers.

Master

In this document, Master always means the source of data to be replicated.

Replica

The target of data replication is the Replica, which refers both to an instance of sync_server and to the resultant dataset.

Operating Modes

Cyrus replication supports two modes of operation: Rolling and Periodic. The difference is that rolling replication is a more or less continuous process whereas periodic replication occurs on demand, triggered by some manual or automated process such as cron(8).

Rolling Replication

Rolling replication is enabled by setting sync_log to True in imapd.conf(5). With sync_log: true, any process which alters the mail spool will update the sync_log files with details as to which mailbox(es) or users have been affected by their actions. In this way the sync_log acts as a command file for the sync_client(8) process(es).

The log files are stored in {configdirectory}/sync/log for single channel systems (see Channels for more information) and are rotated on a regular basis by Cyrus. Multi-channel deployments will have a separate sync_log file for each, stored as {configdirectory}/sync/<channel>/log.

Upon completing a log file, sync_client will go to sleep, or, if processing took longer than sync_repeat_interval seconds, will start over again on the next log.

Note

Any unsuccessful run of sync_client will result in the incomplete remains of the original log file being left behind as "log-<$PID>". This may be re-run as needed.

Note

Please also see below for other uses of sync_log.

Periodic Replication

With sync_log set to the default False, replication must be triggered either by manually running sync_client(8), or by doing so via cron or an entry in cyrus.conf(5).

In either event, command line switches control the operation of sync_client.

Once the process completes its work, it will exit.

Idempotency

Synchronization itself is idempotent in either mode, so log files may be "replayed" without concern of damage to the replica's mail spools.

Sync Chains

Cyrus supports chained replication, in which one replica replicates to another. I.e. A replicates to B; B replicates to C. If you wish to use this approach, please see the sync_log_chain setting:

sync_log_chain: 0

Enable replication action logging by sync_server as well, allowing chaining of replicas. Use this on 'B' for A => B => C replication layout

Note that sync_log_chain is to be set on the middle server(s) in a chain, not on the first or last.

Transport

Older versions (pre-3.0) of Cyrus used the dedicated csync transport -- typically over TCP port 2005 -- and server process -- sync_server(8) -- for replication. This is no longer necessary.

From v3.0 forward, the sync_client(8) will default to using IMAP protocol for transport, and an IMAP instance on the replica will process the synchronization instructions. If you wish, you may override this by setting the sync_try_imap setting in imapd.conf(5) to False.

sync_try_imap: 1

Whether sync_client should try to perform an IMAP connection before falling back to csync. If this is set to "no", sync_client will only use csync. Prefix with a channel name to apply only for that channel

Installation

One must build Cyrus IMAPd with the --enable-replication configure option. This builds the replication client/server applications and utilities.

Note

Those using their distribution's packages may need to install a separate package for replication support. For example, on Debian and derived distros, install the cyrus-replication package.

Requirements

1. At least one Cyrus IMAP server instance to be the master.

2. At least one Cyrus IMAP server instance to be the replica.

Note

Sample configurations for both "master" and "replica" instances are included in the standard distribution.

Replica server configuration

The replica is a standalone server instance which listens for and processes synchronization messages from a single master server. The replica server needs to be configured to accept synchronization messages via IMAP or the (deprecated) sync_server(8) process.

Important

Within a Cyrus Murder environment, replicas must not be configured to invoke ctl_mboxlist(8) on startup (pushing the local mailbox list to the Mupdate Master). This may only be done on the Master instance.

Important

It is critically important that all partitions existing on a master server instance also be present on all replica instances. Failure to ensure this will result in crashes and could lead to lost data on the replicas.

1. Configure a standalone server.

2. If using the deprecated sync_server scheme, add the following line to the /etc/services file. Note that the port number is arbitrary as long as its not being used by any other services on the network.

   csync     2005/tcp
   

3. If using the deprecated sync_server scheme, add a line similar to the following in the SERVICES section of cyrus.conf(5):

   syncserver       cmd="/usr/cyrus/bin/sync_server" listen="csync"
   

4. Start/restart /usr/cyrus/bin/master.

Master server configuration

The master server is a standalone or backend Cyrus IMAP server instance which is actively serving mailboxes to clients. This server needs to be configured to synchronize its mailstore with a replica server via an instance of sync_client(8).

If using the deprecated sync_server scheme, add the following line to the /etc/services file.

csync     2005/tcp

Note

The port number MUST be the same as that used on the replica server.

Specify the hostname of the replica server and how to authenticate to it in imapd.conf(5) using these options:

• sync_host

• sync_port

• sync_authname

• sync_realm

• sync_password

Note

sync_authname MUST be an admin user on the replica.

Note

sync_realm and sync_password may not be necessary depending on the SASL mechanism used for authentication.

Note

See Channels, below, for details on how to use these settings to control syncing to multiple replicas.

Important

If using sync_log_channels for any other purpose, such as specifying the sync_log used by squatter(8) command, you must also either specify a sync_log channel for replication, or specify the default "" (the two-character string U+22 U+22).

Add invocation specifications to cyrus.conf(5) to spawn sync_client(8) as desired (for each channel used) as described below in Rolling Replication or Periodic Replication.

Compression

If one runs replication over a WAN link, the trade-off between bandwidth and CPU usage will tilt strongly in favour of enabling compression to save bandwidth at a slight increase in CPU cost. Set the sync_compress value in imapd.conf(5):

sync_compress: On

or pass the -z flag to sync_client(8) in the service spec in cyrus.conf(5):

syncclient       cmd="/usr/cyrus/bin/sync_client -r -z"

Rolling Replication Configuration

Rolling Replication means that the master instance continuously synchronizes itself with a replica.

To configure rolling replication, perform the following:

1. Enable the sync_log option in imapd.conf(5). This allows the imapd, pop3d, nntpd, and lmtpd services to log synchronization actions which will be periodically serviced by sync_client:

   sync_log: On
   

2. Optionally, adjust the sync_repeat_interval in imapd.conf(5):

   sync_repeat_interval: 300
   

3. Add a line similar to the following in the STARTUP section of cyrus.conf(5):

   syncclient       cmd="/usr/cyrus/bin/sync_client -r"
   

Start/restart /usr/cyrus/bin/master.

Hint

In a multi-channel mesh, the channel to be used by a given sync_client must be specified via the "-n <channel>" argument on the command line:

syncclient       cmd="/usr/cyrus/bin/sync_client -r -n channel1"

Terminating Rolling Replication

To be able to stop rolling replication at any time, configure the sync_shutdown_file option in imapd.conf(5) to point to a non-existant file, the appearance of this file will trigger a shutdown of a sync_client(8) instance:

sync_shutdown_file: /var/lib/imap/syncstop

Tweaking Rolling Replication

The default frequency of replication runs is 3 seconds. Lengthening this produces higher efficiency at the cost of slightly more stale data on the replica. Alter this via the sync_repeat_interval in imapd.conf(5) or by using the "-d" argument in the invocation of sync_client(8).

Periodic Replication Configuration

In Periodic Replication the sync_client instance must be spawned from time to time, causing replication to start at that time. This may be handled via a cron(8) job, or by adding an entry to the EVENTS section of cyrus.conf(5) like any of these:

EVENTS {
    <...>
    # Periodically sync ALL user mailboxes every 4 hours
    syncclient       cmd="/usr/cyrus/bin/sync_client -A" period=240

    # Periodically sync changes at specific times
    syncclient       cmd="/usr/cyrus/bin/sync_client -A" at=0800
    syncclient       cmd="/usr/cyrus/bin/sync_client -A" at=1200
    syncclient       cmd="/usr/cyrus/bin/sync_client -A" at=1800
    <...>
}

Note

When using the "-A" flag (sync all users) no non-user mailboxes are synced. As the man page imapd.conf(5) notes, "... this could be considered a bug and maybe it should do those mailboxes independently."

Tweaking Replication

You may control the number of messages replicated in each batch, via the sync_batchsize setting:

sync_batchsize: 8192

the number of messages to upload in a single mailbox replication. Default is 8192. If there are more than this many messages appended to the mailbox, generate a synthetic partial state and send that.

Channels

The Cyrus replication scheme is very flexible, and supports meshes in which masters running on various hosts may replicate to instances on other hosts. This is achieved by use of the Channels feature of the replication system.

To employ channels, prefix any of the following sync_ configuration options in imapd.conf(5) with the channel name and an underscore "_" character as needed:

sync_authname
sync_password
sync_realm
sync_host
sync_port
sync_repeat_interval
sync_shutdown_file

Then add the setting sync_log_channels with a list of the channels:

sync_log_channels: chan1 chan2 chan3

For example, a site using the same auth credentials for all servers has no need to specify unique per-channel settings for sync_authname, sync_password or sync_realm, but might do the following for the rest of the sync related settings in imapd.conf(5):

sync_authname: replman
sync_password: <secret>
sync_log_channels: repl1 repl2 offsite
##
# The main replica
repl1_sync_host: mailrepl1.example.org
repl1_sync_repeat_interval: 180
repl1_shutdown_file: /run/cyrus/sync/repl1_shutdown
##
# A second replica used to feed the tape backup system
repl2_sync_host: mailrepl2.example.org
repl2_sync_repeat_interval: 180
repl2_shutdown_file: /run/cyrus/sync/repl2_shutdown
##
# An offsite replica which needs a different port and uses a slower
# cycle rate
offsite_sync_port: 19205
offsite_sync_host: mailoffsite.example.org
offsite_sync_repeat_interval: 360
offsite_shutdown_file: /run/cyrus/sync/offsite_shutdown

Then these entries in cyrus.conf(5) would complete the exercise:

repl1sync       cmd="/usr/cyrus/bin/sync_client -r -n repl1"
repl2sync       cmd="/usr/cyrus/bin/sync_client -r -n repl2"
offsitesync     cmd="/usr/cyrus/bin/sync_client -r -n offsite"

Again, this is just an example for illustration. The system provides so much flexibility, and one can combine channels with chaining to achieve even more.

Other Considerations

Important

This section is currently under development. If you believe you are impacted by these considerations, please check back with each release and follow the mailing list.

The infrastructure provided by sync_log has now been leveraged by the Rolling Indexing capability introduced in v3.0. See squatter(8) for more details (see the fourth mode synopsis).

Specifically, the following new settings have been added to imapd.conf(5) in support of this new use of sync_log:

sync_log_unsuppressable_channels: squatter

If specified, the named channels are exempt from the effect of setting sync_log_chain:off, i.e. they are always logged to by the sync_server process. This is only really useful to allow rolling search indexing on a replica.

Administration

Manual replication

To manually synchronize any part of the mailstore, run sync_client(8) with the appropriate command line options. Note that manual synchronization DOES NOT interfere with rolling replication.

For example:

[root@skynet ~]# /usr/lib/cyrus-imapd/sync_client -S cyrus-replica.example.org -v -u john.doe@example.org
USER john^doe@example.org

One can run cyr_synclog(8) instead, which will insert the record into the rolling replication log.

Failover