Upgrading to ZCP 7.0 with MSR
From Zarafa wiki
With the MSR (MailStore Relocator) it is possible to upgrade from ZCP 6.40 to ZCP 7 with almost no downtime. This document will explain the prerequisites, best practices, tips, etc.
Disclaimer Upgrading using the MSR is a complex procedure and has some limitations and is considered as the least favorite option to use for upgrades. Please contact firstname.lastname@example.org before you try to use this procedure.
Migrating users with MSR is a way of migrating from 6.40 to 7 with almost no downtime at all but is a complex and labor intensive way of upgrading. The MSR is mainly used to migrate users between servers with the same software version in a multi-server setup.
A proven concept is to migrate users in small batches, as the MSR sets up a notification after the migration is done, it not advised to migrate thousands of users at once using one MSR. So it is advised to migrate within small batches of 50-200 users. You can run multiple MSRs in parallel but never run two MSRs on the same store because that can break stores.
IMPORTANT When delegates(send as) are used, it's recommended to migrate per department as delegates will mostly be used between users within that department. Alternatively you can run a second separate spooler on the 6.40 server that only picks up the outgoing mail on the new 7.0 server. The 7.0 components can not connect to the 6.40 components, so you will not be running a spooler on the 7.0 server during the migration.
- It is mandatory to upgrade the 6.40 node first to 6.40.14 or later. There are some fixes in there which make the migration seem more seamless, e.g.: WebAccess settings are kept after a store is migrated to 7.
- It is mandatory to install ZCP 7.0.4 or later on the new 7 node. Some fixes are in there to make the migration more seamless.
- The zarafa-msr needs to be from a ZCP 6.40.14 or later. It is advised to run the zarafa-msr on a linux distribution with python version 2.6. If your current 6.40 node has python 2.6 installed then you can easily run the zarafa-msr from your 6.40 node. If your current 6.40 node does NOT have python 2.6, then it is advised to install an extra linux distribution which has python 2.6 and do a man in the middle migration from that node: ZCP 6.40 ---> MSR node --> ZCP 7 node.
- You'll need to keep running with 6.40 clients (WebAccess, Outlook) until ALL stores have been migrated to ZCP 7, this has to do with backwards compatibility.
- You'll need to keep delivering ALL emails to the dagent on the 6.40 node until ALL stores have been migrated to ZCP 7. The 6.40 dagent is the only dagent which can both deliver to ZCP 6.40 stores and ZCP 7 stores. (Use the dagent with SSL so that it can open an admin session on both nodes, see the administrator manual http://doc.zarafa.com/7.1/Administrator_Manual/en-US/ on how to do this).
- You'll need at least zarafa 6.40 for this to work. The MSR upgrade to 7 will NOT work directly from ZCP 6.30 to ZCP 7.
- You'll need to have a 6.40 node which is configured as a multi server node. If you do not have this, then you can migrate your node to a multi server node according to this article: http://www.zarafa.com/wiki/index.php/Migrating_to_MultiServer_setup.
- You'll need a Zarafa Enterprise subscription in order to run a ZCP Multi-Server environment. Please contact Zarafa to get a temporary subscription for the migration.
- For python you'll need the following dependencies which is not installed by default on most distro's: threadpool. If your distribution (eg. RHEL5) doesn't have threadpool, you can do:
# yum install python-setuptools # easy_install threadpool
In order to migrate from ZCP 6.40 to ZCP 7, you'll need an empty ZCP 7 installation:
- Make sure you'll install the ZCP 7 node correctly with multi server enabled.
- The ZCP 7 node needs to be in the same multi-sever cluster as the ZCP 6.40 node.
- Make sure that SSL has been correctly setup on the ZCP 6.40 node for dagent and spooler. Check the ZCP Adminstrator manual on how to do this.
You'll only need to setup SSL for the 6.40 node (dagent, spooler), this is because:
- ZCP 6.40 components can connect to ZCP 7 (ZCP 7 is backwards compatible with 6.40)
- ZCP 7 components can NOT connect to ZCP 6.40
When your 6.40 is a multi server node and you have checked all the versions and dependencies we'll first need to change a value in the server.cfg. On both the ZCP 6.40 node and the ZCP 7 make sure the following value is set to yes:
enable_enhanced_ics = yes
Restart both zarafa-server processes after this change.
It is advised NOT to configure any quotas on the new 7 node while the migration is still taking place. Zarafa 7 counts store sizes slightly different and it could happen that an MSR fails, if you MSR a store which is almost on the quota limit.
To disable default quota of users the quota options in the server.cfg on the ZCP7 server must be set to 0. If users have a personal quota setting in Active Directory or OpenLDAP, this quota can be temporarily disabled by setting the ldap attribute ldap_quotaoverride_attribute to a non existing attribute, like "noquota".
quota_warn = 0 quota_soft = 0 quota_hard =0
ldap_quotaoverride_attribute = noquota
Make sure that the zarafa-spooler on the ZCP 7 node is configured to relay ALL emails to the MTA of the ZCP 6.40 node. This way you can make sure that mail delivery is being done ONLY by the ZCP 6.40 dagent. The ZCP 6.40 dagent is the only dagent which is able to deliver to mail stores on both ZCP 6.40 and ZCP 7.
If you are using a completely different mail gateway then you can have the ZCP 7 spooler relay ALL emails to that mail gateway. But you'll need to make sure that your mail gateway only delivers to the ZCP 6.40 node until ALL stores have been migrated to ZCP 7.
The following option need to be changed in the spooler.cfg file on the ZCP7 node:
smtp_server = <ip-address 6.40 server>
Now we have a working multi server cluster with one ZCP 6.40 node (with all stores) and one empty ZCP 7 node. We can now use the MSR to migrate the stores to the ZCP 7 node.
Note: Make sure you are using zarafa-msr from a 6.40 installation, and it is advised to use msr with python version 2.6.
You'll need to use the SSL certificates in the MSR config, so that zarafa-msr is able to open an admin session on both ZCP nodes. The following is an example msr config:
[Connection] serverpath: file:///var/run/zarafa sslkey_file: /etc/zarafa/ssl/client.pem sslkey_pass: zarafa [Mapping] user1: https://ip_of_zcp7_node:237/zarafa [Logging] log_file: /var/log/zarafa/msr.log
This config file first sets up SSL, so that it can open admin sessions on both nodes (check our administrator manual in the multi server section for more information about ZCP and SSL).
Next you'll get the user mapping: user1 will be migrated to the ZCP 7 node.
Now we can start the MSR to migrate user1 to the ZCP 7 node. The MSR only takes one argument, which is the location of the msr config file:
# zarafa-msr /etc/zarafa/msr.cfg
Note: While the MSR is running, users can just keep working, they won't notice anything.
The MSR tool uses state files to keep track of the synchronization. These files are located in a hidden directory inside the home directory of the user who executed the MSR tool. For example, for the root user this files would be located in:
The MSR tool will clone the complete user store. When it's finished it will keep the store in sync. This is some sample output of a zarafa-msr run:
INFO : Starting to relocate user 'user1' to 'https://ip_of_zcp7_node:237/zarafa' INFO : Maintaining sync for user 'user1'... INFO : Initial migration completed for user 'user1' INFO : All migrations have completed successfully, maintaining sync.
Now first check the msr log file if any errors occurred.
Now you can switch the the zarafaUserServer in LDAP to the new ZCP 7 node. When switched it is advised to force a sync on both nodes:
# zarafa-admin --sync
When the sync is done you can check if zarafa now sees user1 on the new ZCP 7 node:
# zarafa-admin -l username fullname --------------------------------------------- SYSTEM SYSTEM user1 User One (zcp7)
The MSR tool kept the store in sync while we switched the user to the new 7 node. This is now finished, so we can break of the MSR with CTRL-C:
INFO : Starting to relocate user 'user1' to 'https://ip_of_zcp7_node:237/zarafa' INFO : Maintaining sync for user 'user1'... INFO : Initial migration completed for user 'user1' INFO : All migrations have completed successfully, maintaining sync. ^C INFO : User requested exit...
User1 has now been migrated to ZCP 7 without any downtime.
The order of migration is very important! Please read these sections and in case of questions please contact email@example.com.
Previously there was already explained that only ZCP 6.40 components (dagent, spooler, etc..) are able to connect to both ZCP 6.40 and ZCP 7. The ZCP 7 components are not able not connect to a ZCP 6.40 server.
The preferred order of migration is:
- Shared folders
- User folders per department in batches of approx 100 users at a time.
- The public store can be done when the stability of the ZCP7 is proven.(eg. ZCP7 is generally newer and better hardware)
- Last are power user(think secretary) Try to migrate users with lots of delegate rights to send-on-behalve of other users. Do not confuse this with sending-on-behalve-of shared store.
Public store migration order
As long as the LDAP attribute zarafaContainsPublic is set to true for one ZCP server then there is no problem when to migrate the public store.
ldap_server_contains_public_attribute = zarafaContainsPublic
Shared stores should be migrated first because of the "sent-on-behalve-of" functionality needed by the Outlook/WebAccess clients. A user can not log into a shared store but the d-agent on the ZCP7 server needs to be able to put messages into both ZCP 6.40 and ZCP7 sent-items folders of the shared-store. Because the d-agent on the ZCP7 can not do this on the ZCP6.4 server, this dictates that all "shared-stores" should be stored first on the ZCP7 server.
Delegates migration order
This will bring some issues, e.g.:
- There is a shared mailbox called MAILBOX1
- USER1 has delegate rights on MAILBOX1
- USER1 has been moved to ZCP 7
- MAILBOX1 is still on ZCP 6.40
When USER1 now tries to "send on behalf of" MAILBOX1, the following will happen:
- USER1 sends an email "on behalf of" MAILBOX1
- USER1 is on the ZCP 7 node, so this email will be picked up by zarafa-spooler on ZCP 7
- zarafa-spooler needs to check the delegate rights on MAILBOX1 (which is still on ZCP 6.40)
- The ZCP 7 zarafa-spooler is NOT able to connect to ZCP 6.40 --> ZCP 6.40 server does NOT allow ZCP 7 components to communicate
- The zarafa-spooler will NOT allow USER1 to send "on behalf of" MAILBOX1
The workaround for this is to keep USER1 on ZCP 6.40 and to FIRST migrate MAILBOX1 to ZCP 7. So if delegate rights are really important in your company, you'll probably need to make an inventory of all delegate rights and first move all the stores which have delegate rights set. It is advised to FIRST migrate all the shared stores to the ZCP 7 node.
IMPORTANT When delegates are used, it's recommended to migrate per department as delegates will mostly be used between users within that department.
If you are using Public Folders, then do NOT forget to migrate the Public Store to the new ZCP 7 node. There is a special name for the Public Store which you can use in the msr config file. For a single tenant environment use:
[Connection] serverpath: file:///var/run/zarafa sslkey_file: /etc/zarafa/ssl/client.pem sslkey_pass: zarafa [Mapping] __public__: https://ip_of_zcp7_node:237/zarafa [Logging] log_file: /var/log/zarafa/msr.log
If you are using a multi tenant environment, please check our administrator manual on how to migrate the public stores.
When you migrated the Public Store (do not forget to check the log file), you can switch the Public folder in LDAP to the new node.
We keep running ZCP 6.40 clients (Outlook, WebAccess) until ALL stores have been migrated. As explained many times earlier in this article: ONLY 6.40 clients are able to open stores on both ZCP 6.40 and ZCP 7.
When ALL stores have been migrated to the ZCP 7 node, you must upgrade your Outlook clients and Webaccess clients to the 7 version.
IMPORTANT When Z-Push 1.5.x is still installed, users with mobile devices have to reinitialize their Activesync profile, as the sync states aren't migrated by the msr. This means all existing Activesync profiles have to be deleted and recreated on all mobile devices.
In case Z-Push 2.0.x is installed, users will be automatically resynced after they are switched to the ZCP7 server node.
We advise to upgrade to z-push2 first before using the MSR for upgrading.
Blackberry users should be removed in the BES admin interface and do the enterprise activation again.
Change the connection string in all your clients (WebAccess, Z-Push, etc..) to connect to the new ZCP 7 node. This is only relevant if different FQDNs are being used to connect to the ZCP7 server. In the multi server protocol the Outlook client version 7.x will automatically update it's connection string once it detects that the user has been moved to a new server. This only works if the ZCP6 node is still running so keep all ZCP6 nodes running until no sessions are created.
The spooler on the ZCP 7 node was configured to relay ALL emails to the MTA on the 6.40 node. Because we now have migrated ALL stores we can set the spooler to use it's own MTA.
If you are using a completely different mail gateway then you'll probably have not changed the smtp server in the spooler.cfg. You can now have the mail gateway deliver directly to the ZCP 7 node.
Before shuttinf down your ZCP 6 nodes
Before you can shutdown the old ZCP6 nodes make sure no OUTLOOK sessions are active on the ZCP6 nodes.
zarafa-stats --sessions | grep OUTLOOK
Also make sure only zarafa-outlook-plugins version 7 are used because the Outlook profiles need to be updated with the new servername of the ZCP 7 nodes and the new location of the public stores.
Outlook 2003 and older versions
If a user has openend shared stores or shared calendars (other users stores) they need to remove those shared stores and close Outlook to make sure all old references are removed. An alternative is to start outlook with the commandline switch
"C:\Program Files\Microsoft Office\OfficeXX\Outlook.exe" /resetsharedfolders
Shutdown ZCP 6.40
Finally you will be able to shutdown your ZCP 6.40 node. It is really advised not to immediately delete all data on the old ZCP 6.40 node, instead you should shutdown the node (or all zarafa related processes on that node) and wait for some time. This way you will find out if you have configured all components in the network to communicate only with the ZCP 7 node.
While the old ZCP 6.40 node is shutdown, you should check your ZCP 7 node:
- Spooler logs -> if mail is being send out
- Dagent logs -> if mail is being delivered to Zarafa users
- MTA logs/queues -> if mail is being processed by the MTA
When trouble arises, and you find out that some things have been overlooked, you will still be able to fire up the old ZCP 6.40 and fix all the overlooked issues accordingly.